Masterface

The target and the rules

The target is one image per subject, or an honest "not feasible" with a reason. Sunglasses in nearly every photo cannot be calculated away, and the engine has to say so instead of inventing eyes. The output is a bald head with the brows kept, plus five hair variants with a free forehead, because one uniform hairstyle did not suit everyone.

Three rules came from me and shaped everything.

• The embedding drives the swapper, but it is never the yardstick. It was built from a part of the same photos, so measuring against it would be circular. The check uses AdaFace IR-101, which is not part of the embedding, against the source photos.
• An algorithm, not a selection sheet. All decisions follow fixed rules and I only see the result.
• The last verdict is my eye. A score can pass and the image can still be wrong.

How the chain works

engine profile takes a subject folder and runs read, embedding, card, masterface, assets and profile. Each step remembers a fingerprint of its sources, so a second run only repeats what changed.

The read step removes duplicates and finds the subject by face, never by the meaning of a file name. The embedding step averages up to 12 selected faces into a vector in the format VisoMaster Fusion reads. The card step describes the photos. The masterface step then does the main work:

The output is a set of clean images, a labelled overview with the score under each image, and a run log. The order is the point of the design: hair and light change first in an instruction editor, and identity comes back afterwards through the swapper.

The tools around the chain

• Identity and assignment: buffalo_l, AdaFace IR-101 with mirror test-time augmentation, and CR-FIQA for face quality. The setup came from a benchmark on 136 identities. CR-FIQA stays out of the read and embedding steps.
• Embedding format: w600k_r50 for Inswapper, plus SimSwap ArcFace and Ghost ArcFace vectors, so the file loads directly in the program.
• Photo description: Qwen3-VL-8B and Qwen3.5-9B fine-tunes through llama.cpp, four questions per photo. The final model is still undecided. Benchmarks of the base models favour the second, and a Qwen3.6-35B-A3B variant was only researched.
• Photo records: ExifTool for metadata, Tesseract for text in images, GeoNames offline for places. Each photo gets an .xmp sidecar, and the photo itself is never changed.
• Runtime: ComfyUI on its own port, onnxruntime on the GPU, TensorRT as provider in the program, nvidia-smi for free VRAM.
• Program: VisoMaster Fusion 3.12.3, third-party, its code unchanged, with six presets of my own built on Inswapper, InStyleSwapper 256 or AlphaFace, GPEN restorers, XSeg masks, RetinaFace or Yolov8 detection and LivePortrait expression tools.

Seven places it pushed back

Identity against editing

This was the core conflict. Everything that creates a clean frontal image pulls away from the subject, and everything that keeps identity keeps the bad pose and light as well. Generators with an identity adapter start from noise and a vector, so they produced a plausible stranger. Starting from an existing photo of the subject beat every generated image I saw. The way out was to split the work. An editor changes what must change, then the swapper brings the inner face back with a parser mask, which raised identity clearly in my tests. The risk that remains is drift in the editor, which is why every image is measured.

The forehead

A fringe looks like a small detail, but it blocked the classic route. SDXL inpainting in Krita paints the fringe back from its own context, however the mask is drawn, and it took 22 minutes per run. Stable-Hair and similar bald converters were an option on paper. The turning point was a research round that pointed at instruction editors with open weights, because they can do "fringe away, forehead free" as one step. Qwen-Image-Edit-2511 gave the first usable bald head at 0.753 and hair variants at 0.78 to 0.80, measured with AdaFace against the original.

A check that cannot grade itself

The early engine set aside every fifth good photo as a hold-out and scored against those on a scale where 1.0 meant a typical source photo. It still passed results I rejected, because with little material the comparison is weak. So below five usable photos that engine reported "not checkable". The embedding left the scoring entirely. One recogniser and my eye are all I have, and I name that limit openly.

Mouth and hair colour

Two faults came from averaging rather than from the editor. The embedding is a weighted average of up to 12 faces, and the swapper puts that average on at the end, so pursed lips in the set came back in the result. Now mouth width over eye distance is compared to the subject's top quarter, and under 90 % or over 115 % drops out before the embedding is built. Mouth opening and smile come from 68 points.

Hair colour was measured in a fixed box above the forehead on one photo. That box could hit a picture frame or miss the hair altogether. Now it uses hair pixels from the face parser, the median over all usable photos, with finer wording for tone, warmth and lighter strands, and the instruction says the colour must not be darkened. Hair colour from the profile card only counts from ten photos on, since a single filtered photo could shift it. Iris colour is a median too. Collages get a penalty as base image, and face height outliers beyond 12 % stay out of the five-point average, because head tilt distorts it.

Some sources of error cannot be detected reliably: beauty filters, ages mixed across photos, mirrored partings, heavy make-up. They stay on the list as known risks.

Cropped input

Tight face crops were the hardest input. With 51 crops of 512 pixels, the base image lay 15.5 % outside the frame, and the grey fill turned into a veil and a strip on the bald head. The chain needs the whole head with a margin. The value is in the run log, but it does not warn yet.

A 20B editor on a 12 GB laptop GPU

The editor runs quantised with about half of it offloaded, next to 32 GB of system memory. The GGUF alone is 13.2 GB and the text encoder 9.4 GB. Pinned memory first held 12.9 GB fixed, and one of the memory options pushed ComfyUI to 37 GB, so the right start options took a while to find. ComfyUI also had to start as its own process, because it crashed inside the engine's job object. Free memory still fell as low as 0.06 GB during an edit, and a start at 10 GB stopped, so the start rule is now 11 GB free. torch.cuda.mem_get_info does not see other processes under Windows, which made nvidia-smi necessary.

All variants in one pass was the speed lever. Six images take 892 seconds, 128 to 180 seconds each, and a whole masterface 965 to 1,079 seconds per subject.

Settings that were not active

The program under the fit stage had its own trap. Stacking every quality module looked like an alien, and research across 89 issues and 266 pull requests of Fusion, 120 issues of VisoMaster, 21 video transcripts, about 2,000 forum posts and blogs in three languages said there is no magic setting. Restorers belong under 100 %, and profile distortion is a limit of Inswapper 128. The rule became one change per test. Then it turned out that Apply Settings is off after every start, so control values had not been reaching the program. Controls are now written by a start script while the program is closed, and the saved workspace is checked afterwards. For speed, Inswapper 128 at higher resolutions took up to about 32 passes per face and frame, so four of the five main presets moved to InStyleSwapper 256, which works natively in one pass, and the head-turn preset uses AlphaFace.

Results that held

• Worked: Qwen-Image-Edit-2511 with the Lightning LoRA through ComfyUI, under Apache 2.0.
• Worked: LivePortrait for rotation and gaze. Yaw came down from 14, 13 and 11 degrees to 4, 2 and 2.
• Worked: Inswapper with the embedding after the edit. In the earlier chain, fitting raised the AdaFace value from 1.07 to 1.26.
• Worked: eyes taken from the source, not repainted.
• Worked: GPEN at half strength for sharpness without a plastic look.
• Worked: AdaFace as an independent check.
• Worked: Lanczos instead of AI upscaling on assets.
• Worked: a full run on a 660-photo folder with the subject in 535 photos. 0.73 to 0.82.
• Worked: a run on 29 photos with the subject in 24. Bald 0.791 and clean, hair 0.835 to 0.887. The best numbers so far.
• Partly: 51 tight crops. Bald 0.718 with the veil, hair 0.780 to 0.821.
• Partly: the audit fixes for lips and hair. They are built in, with thresholds derived from one set.

Routes that stayed closed

• Did not work: PhotoMaker V2 and IP-Adapter FaceID Plus V2. AdaFace 0.56 to 0.70, and the faces looked artificial and staged.
• Did not work: InstantID and PuLID on SDXL. Identity near 0.
• Did not work: Arc2Face. 0.77 to 0.87 and too little of the subject. It also needs an antelopev2 vector, so a stored w600k embedding cannot feed it directly, and it had to be patched for transformers 5.
• Did not work: SDXL templates chosen by geometry, the passport canvas chain. Replaced by the editor route.
• Did not work: repainting eyes with SDXL, EVA-CLIP eyes, CodeFormer. The results were garbage.
• Did not work: HivisionIDPhotos. It only crops and changes the background.
• Did not work: a forehead hair filter on the embedding. There was 0.00 forehead hair in the swap anyway, so it only lowered identity.
• Did not work: larger images first, five workers, pitch from five points. Weaker hits, too little memory, unusable angles.
• Did not work: Krita with SDXL hair fill.
• Did not work: Real-ESRGAN on the assets. It distorted eyes and skin.
• Did not work: fixed tag lists for describing photos. The model filled them with nonsense, so tags are free and limited to what is visible.
• Did not work: stacking all quality modules in the program.

Models I read about and never ran

• NextFace: a 3DMM fitter that gives a smooth mask without hair, mouth interior or eyes, on a renderer with old CUDA wheels.
• FaceAnything: point clouds from a 15 GB checkpoint, not a new portrait.
• GAN frontalisers such as TP-GAN, FNM, Rotate-and-Render and FFWM: low resolution and unmaintained.
• 3D head routes, FaceLift, PanoHead inversion, CAP4D, SMIRK, Pixel3DMM with MICA, LAM, MeshLAM, FreeUV and DiffLocks: struck or left behind, since the goal is an image and my filter was use, Windows support and available weights.
• Other editors and identity models: FLUX.1 Kontext dev, FLUX.2 klein and dev, Qwen-Image 2.0 and 2.1, HunyuanImage 3, InfiniteYou, PuLID-FLUX II, USO, WithAnyone, OmniGen2, Bagel, Step1X-Edit. Too large, API only, or not worth an A/B test yet.
• Restoration and hair: RestorerID, ReF-LDM, Stable-Hair, HairMapper, AnyBald. On the research list.
• Measurement alternatives: MediaPipe expressions, OpenFace, LibreFace, MiVOLO v2, SegFace, TopoFR, LVFace, ViT-FIQA, Audio2Face, GEM-X, WiLoR. No clear gain, or no use for this goal.
• Florence-2, OWLv2, RT-DETR, DINOv2, CLIP: in the model pool and unused, because a vision language model covers them.
• The Multiple Angles LoRA for Qwen: wired in code and switched off.

New models now come in only when they are better or bring new data, and Masterface is only replaced after an A/B test.

Agents, rules and my eye

Claude Code agents researched, planned, coded, ran and measured this, in one project chat with sub-agents for studies and fact checks. The face engine is 7,307 lines of Python. I set the goal and the conventions, made the rules above, gave the go for each direction, and judged the results with my own eyes. A decisions file holds who decided what and why, and every rejected approach with its lesson and the condition for trying it again. That file comes first before any new model.

What is still open

• The bald head on tight crops carries a grey veil. I decided to repair it with the existing material rather than swap the images.
• There is no warning for cropped heads yet.
• A full run with a visual comparison against the best earlier result is still open.
• Thresholds need recalibrating on five or more sets.
• Lips taken from the editor are built behind a switch and off, because the mouth edge is a risk.
• Measured iris and hair colours are not stored yet, and in small folders with two people the main subject is sometimes wrong.
• Scores come from one recogniser. The only ground truth is the source photos and my eye.

The face engine

A folder of images, and only faces to go on

Everything in my work on digital humans starts the same way. There is a folder of images, and somewhere in it is the person the profile is about. The images come in every form: group shots, crops, duplicates, upscaled copies, artwork. Before anything can be built, a program has to find out which faces in that folder belong together, and which of them is the main person.

Two conditions made this harder than it sounds, and both were mine from the first day. Folder and file names are never evaluated, because the assignment must not depend on any name. And the source files stay untouched. The tools only read them, and everything they write goes into one hidden .profile folder per person.

The goal: one command-line engine that takes a person folder and returns a clean result, JSON for machines and HTML for humans, ready for the steps that follow. Masterface, the body work and the voice profiles all build on what it delivers.

Choosing the assigner

The core question is when two faces are the same person. That question got a benchmark and not a guess. On 136 profiles, combinations were compared, and the winner was three models working together:

The threshold is 0.25. Faces get grouped by AdaFace, and the main person is decided by anchors, the folder picture and the background picture if they exist, otherwise by the largest group. A second recogniser cross-checks the result, and outliers against the core of the group are removed. The same assignment feeds every later step, so nothing downstream rebuilds it its own way.

Not every folder has enough material. So there is no hard refusal, only levels: full, few, fallback, none. A profile from few photos is still useful, as long as it says what it is.

One command

engine profile runs read, embedding, card, masterface, assets and profile for a person. Each step remembers a fingerprint of its sources, so a second run only repeats what has changed. Several folders make a batch, with an overview page at the end.

• Read finds duplicates and near-duplicates by content, and prefers an original over its upscaled copy. Artwork files are recognised and left out.
• Embedding averages selected faces into one vector, in the format VisoMaster Fusion can load.
• Card evaluates all photos and writes the profile card.
• Assets and profile write everything into a fixed layout, with validation against JSON schemas.
• A separate command writes an .xmp record next to each photo, if wanted.

File formats and names follow a written convention that Face, Body and Voice share. A machine file is named after its own format field, because files with equal names get mixed up as soon as they leave their folder.

The profile card

The card is the human view: one HTML file per person with a gallery of all photos, in German and English through a shared glossary. An image model answers fixed questions about each photo, and the values are stored in English. I tested two image models side by side and measured how often they agreed per field. A blind comparison was prepared to decide between them, and it was never evaluated, so that decision is still open.

161 profiles in six minutes

This part went better than expected. Four worker processes share the GPU, with persons spread evenly by image count. A pool of 161 profiles took 362 seconds. Another pool of 16 profiles with 12,527 images took 55 seconds. All of it runs under a hard memory limit, a Windows job object at 85 % of the RAM, so one run can never take the machine down.

Six lessons from the engine

• A check that was not independent. An early version reported a pass, and I looked at the result and asked: who is that? With little material the check had only confirmed itself. Since then the check has to be independent of the material it was built from.
• Single photos that lie. One photo with a pink filter made the hair too light, so hair colour from the card only counts from ten photos on.
• Filters on a hunch. Filtering out forehead hair for the embedding sounded sensible. It made no difference to the swap and only cost identity. The lesson: measure first, then filter.
• Too much structure. Fixed tag lists for the photo record made the model fill every slot with nonsense. Free tags limited to what is visible worked.
• Too much parallelism. Five workers ran out of RAM and ONNX memory. Four is the proven number.
• Shared hardware. Models from the earlier steps are not released before the body step, and there is no GPU check before a run yet.

Kept and dropped

• Worked: assignment by face alone, without reading a single name.
• Worked: levels instead of a hard stop.
• Worked: fingerprints per step, which make repeat runs fast and safe.
• Worked: the split into JSON for machines and HTML for humans.
• Partly: small folders with two people. The main person sometimes comes out wrong.
• Partly: two command options that are not fully built yet.
• Did not work: "big images first" as a selection rule. It gave weaker matches.

What it still owes

The engine is about 7,000 lines of Python, written by Claude Code agents to my rules and verdicts. It carries the work for Masterface, and the body and voice work build on it too. Still open besides the two options: faces are sometimes detected twice, a cut-off head gives no warning yet, measured iris and hair colour are not stored, and the image model for the card has to be chosen.

The video field study

The question

Can a face in a video be replaced, on one laptop with 12 GB of graphics memory and 32 GB of RAM, so that the person changes and everything else stays? Background, head pose, expression, lighting and motion should all remain the original. It is the video side of my work on digital humans, and it had to run locally.

Over time a lot was tried, in many separate rounds. So at the end one study pulled all of it together, from the project's own records, with nothing newly calculated. Every statement carries one of four labels: measured, seen, sourced or assumed. That labelling turned out to be as valuable as the results, because it shows how much of what we believed was actually proven.

The short answer: only the classic route does what the question asks, and nothing became perfect.

How results were measured

A single number misled us more than once, so the test bench grew into a set of measures that check each other.

The target was a mean of at least 0.5 and at least 90 % of frames above 0.4.

The classic route: twenty steps, one change each

The base is Inswapper128 at 512 pixels through 4 by 4 tiling, an averaged face embedding, a face parser mask and a restorer. On top of that, a recipe grew through twenty numbered versions. The rule was strict: exactly one change per round, kept only if neither of two different test videos got worse.

Some of the steps that stayed:

• An embedding averaged from six reference images, which lifted identity from 0.717 to 0.799.
• A mask limited to where the parser finds skin, eyes and mouth. That removed streaks and veils at steep angles, even though the identity score went down. The score rewarded the wrong thing there.
• Original eyes only when the eyes are closed, and the original mouth only at large openings. Protecting them all the time kept the expression but cost identity. The dosed version was the compromise, because I could only recognise the person with the full swap.
• GPEN-1024 limited to the face classes of the parser. Otherwise it sharpened hair in a visible square.
• Turn templates only when the head is not nodding, and no swap at all beyond about 60 degrees of tilt, where the model paints eyes that do not exist.
• A crop scale of 0.9, the best value on both videos.

The best recipe reached 0.787 against 0.708 for the VisoMaster default on the Masterface reference, 97 % of frames above 0.4, a background of 41.5 dB and a median pose deviation of about 1 degree. But honesty needs the second number: on the reference photo it reached 0.750 against 0.811 for the default. The averaged embedding and the Masterface reference are the same six images, so part of that good number is probably self-confirmation.

The generative routes

Generative video models can invent what is not in the picture, which is exactly what a classic swapper cannot do. They ran at 16 frames per second and about 480p, in clips of a few seconds, with Q4 builds where a model was too large, because that is what fits.

• Did not work: Wan2.2-Animate with a swapped first frame. A still with 0.8 identity turned into 0.5 to 0.6 in the video, the whole image was redrawn at 19 dB, and the eyes went soft.
• Did not work: Animate with a head mask. 0.39 on a frontal clip and 0.13 on a steep one.
• Did not work: DreamID-V. Identity about 0.48, weak expression, and the last frame of every run washed out. On a larger head crop it reached 0.64, but with no expression at all, which at least showed that face size matters.
• Did not work: VACE with a masked head and a trained LoRA. Without an expression channel the face stays frozen, and anything that carries expression also carries the old identity.
• Partly: FLUX.2 Klein on single frames, at about 0.7 to 0.85 when frontal. It is useful for stills, but at extreme angles it turns the head to the front, with 0.08 to 0.29.

Other swappers and an online service

• Did not work: GhostFace. It scored a high 0.77 to 0.79 and was wrong by 47 to 53 degrees of pose. It simply set a frontal face on top, and the recogniser loves frontal faces.
• Did not work: SimSwap 512, with 0.53 and crumpled faces when the head leans back.
• Did not work: AlphaFace, which neutralised the expression.
• Partly: an online swap service, compared frame by frame on the same video. It reached 0.566 against 0.672 for my recipe at that stage, and it broke down at the same angles.

Training an identity

A person LoRA for Wan2.1 14B was trained with musubi-tuner: 20 epochs, 2,040 steps, on 51 images. The best epoch reached 0.63, the last one 0.57, so more training did not mean better. The run also crashed working sessions by eating the RAM. A memory-safe version was prepared and never started. Its effect inside Animate was never validly tested, so this route rests.

Four reasons nothing became perfect

Keeping or inventing, nobody does both. The classic swap keeps background, pose and expression, but it has a resolution of 128 pixels, keeps the geometry of the target face, and has nothing to swap where no face is visible. Generative models invent, and pay with identity or background. Every attempt to join them inherited the weak half of the other side.

Errors add up. Each stage leaves a rest that the next one cannot repair. An anchor of 0.8 ends at 0.5 to 0.6.

The material sets a limit too. In the extreme test clips there is simply no face in the picture after a certain frame, only chin and neck. Better face detection gained 2 to 4 points. Looking at the frames first would have saved blaming the method.

Measuring lagged behind. Scales got mixed, undetected frames fell out of the mean, reference and embedding were the same images, and the run log had no column for quality. Several routes were dropped after one single run, which is a decision and not a measurement.

The hardware sits on top of all four: 12 GB forces Q4 models, 480p, short clips and few seeds.

More findings

• Upscaling before the swap does harm: 0.799 fell to 0.770, with rectangular seams, at 22 seconds per frame. Frame interpolation with RIFE at the very end does no harm.
• More restoration brings no identity. GPEN, GFPGAN and RestoreFormer all land at 0.70 to 0.72, and GPEN invents details on small faces.
• TensorRT only in fp32. It cut the time from 209 to 87 ms, while fp16 dropped identity from 0.616 to 0.11.
• A 106-point alignment was worse than 5 points at steep angles, and an occlusion mask on its own did not fix the glitches, since their cause sits in the swapper.
• Claims go stale. "Pixel-identical" turned out to mean re-encoded, so a statement has to be checked against the latest measurement before it is quoted.

The test bench in one table

Claude Code agents built the tools, the test bench and the study, and audit and counter-audit went through the results. I judged the clips.

The route that stays

The classic recipe is the route that stays, with known weaknesses: frames with the head far back (13 % of frames clearly worse than the default, cause unknown), darker lips, and a slight jitter that no measure captures. Open questions are written down: how it behaves at a true profile angle of 60 to 90 degrees, since no test material had one, how large the self-confirmation really is, and whether Animate holds up at 720p, which was never tested. A next round would need reference photos that never entered the embedding, and a blind comparison by eye.

Connected

• VisoMaster Bridge
• Masterbody
• Voice Cloning & Voice Morphing
• Project Everything