Tree / Human Digitalizing
Masterbody
Research into building a rigged 3D body from photos in minutes.
- Status
- Red
- Type
- workflow
- Hours
- about 80
- Tokens
- about 25 M
Goal not reached yet. The fast route has no accepted body so far, its last run at outline IoU 0.827 against a target of 0.85, and the slower route that gave my best result is not automatic.
In this article
What I wanted, and why no tool delivered it
Masterbody has the same purpose as the models that can do this kind of work: digital humans for film, games and research. The goal: from a folder of ordinary photos to a rigged 3D body in standing pose, fully automatic, with no AI chat involved, in about five minutes the first time and seconds from cache after that. Output: a GLB with a skeleton, measurements, a report and an overview sheet. I wanted a believable body, not a 1:1 copy. Few photos are the normal case, and I never pick them by hand.
The gap turned out to be wider than I thought. Paid services seem to do this in minutes, so I had their methods studied, without uploading or paying for anything. None of them reconstructs a body from casual photos.
- Avaturn, MetaPerson and Headshot put a face on a template body set by sliders.
- Meshcapade, Bodygram and 3DLOOK fit a few measurements into a scan-trained body model, and they ask for height, weight, and a front and a side photo in a fixed pose.
- Rodin, Tripo and Meshy generate image-to-3D that looks good but is not measured.
What makes their results convincing is the finished asset: skin, face, hair and light. A grey mesh never convinces, even if its shape is right.
Open research did not close the gap either. For "one subject, several casual photos" no ready method turned up, and almost every model that estimates shape from one image pulls towards an average body.
The approach
The basic problem is depth. Bodies differ mostly front to back, an outline from the front does not see that, and regressors guess it. What carries depth is the direction of the surface, so normal maps and side views matter more than any further outline.
Two routes came out of this.
Way D lifts each photo into 3D. TripoSG gives a fine surface per photo in the pose of that photo. A subdivided SMPL-X mesh is laid onto that surface and bent to T-pose with its own skeleton. Then all fits are fused point by point by reliability and view angle, and what nobody saw is filled from an IDOL back source. It gave my best body, but it is slow and manual, and an audit named its weakness: every photo invents the back anew.
The fast route fits one shared shape for all photos and one pose per photo together, by analysis by synthesis, in the manner of SMPLify-X over several images. Each photo contributes only where it sees something. What no photo sees stays with the body model and is not invented. I chose this principle over per-photo lifting and averaging.
The fast route, stage by stage
One orchestrator with one settings file runs the stages, each in its own Python environment as a subprocess, with a heartbeat and a loud stop on any missing output.
| Stage | What it does | Tool |
|---|---|---|
| Photos | Assignment by face, prefilter | Face engine |
| Pose | Pose and camera per photo | NLF-L, TorchScript |
| Candidates | 12 to 16 candidates by fixed rules | Own scoring |
| Level | Straightens photos along a line from ankle to neck | Own step |
| Masks | Segmentation and normals | Sapiens2 0.8B |
| Select | 6 to 8 photos over six view angles, with anchor and zone roles | Own scoring |
| Shape | One shared body against outlines and normals of all chosen photos, two starts | Own fit, Anny 0.6.1 on SMPL-X topology, nvdiffrast |
| Zone target | A helper fit for one torso zone | TripoSG, own SMPL-X fit |
| Holdout | Outline check on photos not used, one swap allowed, pass at median IoU 0.85 | Own check |
| Finish | T-pose through SMPL-X skinning, subdivision, fusing | Own fusing |
| Check | Outline, movement and crease checks, report | Three own checks |
| Export | GLB, mesh, embedding, report |
The photo rules are strict and automatic: right subject and alone in the frame, whole body visible, standing upright, arms a little away from the body, close-fitting clothes, large and sharp in the image, no backlight, no wide angle up close, no duplicates. Then the best one or two per view group.
Measured times: pose 103 seconds for 129 photos and 197 for 745, masks 151 to 218 seconds, shape about 200 to 360 seconds, TripoSG about a minute per photo, a whole run about 11 to 12 minutes.
Tools and models by job
- Body models: SMPL-X and Anny. Anny went from 0.3.1 to 0.6.1, which brought AnnyInverter and 254 fine controls.
- Pose and camera: NLF-L. CameraHMR, Multi-HMR 1 and 2, SAM 3D Body and PromptHMR as candidates.
- Masks and normals: Sapiens2 0.8B with 29 body parts. The first silhouette work used SAM 2.1 masks.
- Rendering in the fit: nvdiffrast, in PyTorch.
- 3D surfaces: TripoSG, under MIT, which fits in 12 GB on Windows. IDOL for unseen areas.
- Face in the body photos: the shared face engine, and YuNet with SFace for group shots in Way D.
- Attributes: SHAPY, as a hint only.
- Measurements: one shared module for every route.
- Export: glTF with a skin material.
Where it got hard
The pull to the middle
The decisive test came from a simple idea: take two opposite bodies. If a tool cannot tell them apart, it cannot be a shape source. NLF, Multi-HMR 2 and CameraHMR all stayed under that bar in the same way, and the two bodies came apart by only plus or minus 1 cm. NLF's SMPL-X output in practice used only two shape values. Refitting after a regressor did not help either, because it inherits the errors of its start.
One hurdle sat in my own chain. Converting bodies into Anny through six basic values missed by about 12 mm per point and pulled them to the middle on its own. With all phenotypes, 254 fine controls and a refinement pass, the error dropped to 2.5 to 5.2 mm.
Outline does not fix depth
A silhouette fit from the front stays near its start. Two starts gave two equally good bodies. Point-to-outline comparison could not settle the shape, and a flexible pose per photo made the outline fit without making the shape certain. Rendered silhouettes through nvdiffrast were the first step that told slim from strong, and side photos were then weighted higher. Normals from Sapiens2 then brought surface direction into the same fit.
Pose and shape in one
Fitting a generated surface back into the body model through controls and pose at the same time did not hold. The model bent itself and looked bloated, at 13 to 17 mm. Bending the surface back directly left skins between arm and trunk, and cracks. Way D solved this by laying the mesh on in the photo's pose first and only then using the skeleton.
Clothing
Jackets and dresses pull a fit wider. The fit got a skin-edge rule: the body only has to touch where skin edges are visible, and it must stay inside the person mask. A band of plus or minus 1.5 cm around clothing edges does not count as shape evidence, and loose clothing removes the zones it covers from that photo.
Few photos
One test set held only four usable full-body photos, side views of the torso were missing, and the form came out too round. So the selection split into two roles. Up to three anchors with knees and feet visible set height and proportions, and up to six zone photos, torso crops preferably from the side or at an angle, carry the form. In the fit, an outline at the image edge counts as unknown. Before that, an audit had found 74 % of the shape estimated in one earlier result and only about 20 % of each photo reliable.
Fusing and smoothing
Averaging deletes detail. A median over 20 photos wiped forms and blurred the head. A mean across fits flattened zones, so Way D took an upper quantile there. Outliers per zone are cut at 1.6 times the median, and weights are smoothed over the surface and then sharpened, so the best view dominates with soft transitions. These became 20 proven rules. Smoothing with a hard mask flattened forms and made new edges, so relaxing became soft. A dedicated form stage brought a step at one zone edge from 1.94 to 0.29 cm and a kink from 51 to 13 degrees in Way D. Many separate smoothing steps remain a weak spot.
No yardstick
There is no body with known truth here. An audit named that as its first finding and proposed rendering bodies with known values as test photos. I said no: the standard is photos and the visual check, and a few finished bodies come first. The automatic substitute is the outline check, which puts the master into each photo's pose and compares outlines, plus a holdout on unused photos. Measuring tools needed checking themselves, as a polygon fill error once reported 2 to 4 cm too narrow. Numbers alone also missed visible regressions. So every check now means 3D from all angles over all photos next to the best body, a visible shape fault blocks a version number, and open is not solved.
Five minutes against twelve
The target of five minutes met TripoSG at about a minute per photo, Sapiens2 at 3.5 to 6 seconds per photo with about a minute of loading, and a shape fit of several minutes. Way D took about 100 minutes plus hand work, with 5 to 7 scripts in 4 Python environments. The fast route needs no hand work and about 12 minutes, so it is far closer, though still over the target.
Parts that held
- Worked: Sapiens2 masks and normals.
- Worked: NLF-L for pose and camera.
- Worked: fine controls in the conversion to Anny.
- Worked: two opposite bodies as a first test for any shape source.
- Worked: the orchestrator, which runs from photos to report without a chat.
- Worked: the automatic photo selection with fixed rules.
- Worked: the outline check and the holdout as checks without an eye. Way D scored 0.878 IoU, with widths within plus or minus 1 cm.
- Partly: Way D. Best body so far, V2.1. Way D bodies reached 88 to 89 % outline cover. Too slow and manual to be the standard. One variant had side bulges, so the other remained the reference.
- Partly: the fast route. No run accepted so far. The last was smooth without edges, but reached IoU 0.827 against 0.85, kept a visible crease and a kink of 26.9 degrees, and one torso zone came out clearly smaller than in V2.1.
- Partly: SHAPY, wired in as a hint. Its S2A and A2S are small second-degree polynomial ridge regressions on 10 betas, runnable in numpy. Fifteen word ratings alone give about 11 mm in the paper, height only 12.8 mm, height with circumferences about 6.3 mm. Projecting a body to 10 betas left 7.4 mm RMS, and one circumference differed by about 8 cm from its definition alone.
Approaches that did not hold
- Did not work: Multi-HMR 1 into Anny 0.3.1. Mixed body with a gender value of 0.57 and an age estimate of 64, refit at its limit.
- Did not work: NLF, Multi-HMR 2, CameraHMR as shape sources. All pull to the middle. CameraHMR needed a self-built Detectron2. SAM 3D Body and PromptHMR were ruled out without a run here: a published study shows the same pull for the first, and the second's text part is not in its code.
- Did not work: Anny-Fit by NAVER. 531 seconds for five photos, and it moved the wrong way. The cause was a brake on the controls, not the principle.
- Did not work: conversion through only six basic values.
- Did not work: the silhouette fit from the front alone.
- Did not work: fitting TripoSG surfaces into Anny, and repairing the meshes afterwards.
- Did not work: Qwen re-posing as a shape source. The prompt decides the figure, so it may only ever fill missing views, without shape adjectives.
- Did not work: IDOL as main route. Outline IoU 0.825, details gone. It stayed as a filler for unseen areas.
- Did not work: adjusting the silhouette afterwards and a separate neutral pose step. Hips came out too wide, and skinning through SMPL-X replaced the second one.
Body methods I read about and never ran
- Clothed or generative humans: PSHuman, ECON, ICON, SIFU, SiTH, Human3Diffusion, MagicMan, AdaHuman, DynaAvatar. Mostly Linux, pytorch3d or kaolin, 16 GB or more, or a focus on clothing rather than body shape.
- General image-to-3D: TRELLIS and TRELLIS.2 at 16 to 24 GB, SAM 3D Objects at 32 GB, Hunyuan3D, whose licence excludes the EU, including the multi-view Hunyuan3D-2mv. Direct3D-S2, Step1X-3D at 27 GB, Sparc3D in the cloud, TripoSF as a VAE.
- LHM and LHM++: LHM needs 14 to 16 GB of VRAM against my 12. LHM++ has CC BY-NC weights, gives Gaussians in T-pose with no mesh, and has Linux wheels only. Its shape comes from SMPL-X anyway, so it was deferred.
- Depth and normals: MoGe-2, UniDepth v2, Depth Anything 3, Metric3D v2, DSINE, StableNormal, Marigold. Kept as reserve or passed over, since Sapiens2 is built for humans.
- Other regressors: SMPLer-X, SMPLest-X, AiOS, HMR 2.0, TokenHMR, BEDLAM-CLIFF, ShapeBoost, PIXIE, OSX. Pose-focused, hard to install, or the same pull to the average.
- Words to shape: BodyShapeGPT. Too coarse.
- Multi-view reconstruction: RealityScan, COLMAP, Meshroom, VGGT. They need one rigid scene, and separate photos are not one.
- Video avatars: GaussianAvatar, ExAvatar, Vid2Avatar, GVHMR, GEM-X. The shape stays on the body model, or they only help with video.
- Hands: WiLoR, HaMeR. Not needed, since hands come from SMPL-X.
- Size: PERSONA at 48 GB, Human3R for video, UP2You on Linux only, Sapiens2 5B beyond 12 GB.
Who steered and who built
This was a heavily agent-run attempt. I set the goal, chose shared-shape fitting over per-photo lifting, rejected the artificial yardstick, decided that few photos are normal, and made every stop. Claude Code did the research, the code and the runs: a supervisor chat steered, audited and handed out the GPU, a sub-session built the chain, and helper sessions took one module each, with interfaces agreed first and fixed ownership of files.
Best body so far, and the next try
- Most tools regress to an average, and few casual photos do not fix depth.
- Best body: V2.1 from the slow route. Best automatic run: not accepted.
- Height should be a fixed input per subject, because every image model estimates it anew. The services ask for it.
- Believability has to be judged on the finished asset with skin and face, not on a grey mesh.
The next run is prepared with a soft volume target for the weak zone. If that is not enough, plan B takes the zone directly from the TripoSG meshes, the way V2.1 did. New and untested in the model pool are Sapiens2 1B with pointmap, normals and segmentation, and GR-Lite. Neither is wired in. The limit for normals in the holdout check is still uncalibrated, and the many smoothing steps should become one equation system.