X-Ray

What exists

There are two halves: a Jellyfin plugin and an offline chain in Python.

In the player. While a film or episode plays, a panel at the top right of the video OSD lists the people visible now, with photo, name and role, whenever the OSD shows or the video is paused. It follows the film within 0.2 seconds. Display rules are built in the browser from raw sightings: sightings close together are joined unless a scene cut lies between them, and each span lasts a minimum time so names do not flicker. Admins can click a photo to choose a different one. That is the design as coded. I have confirmed it by eye in the player on one title, the first Knight Rider episode. What viewers would see across the whole library has not been measured by anyone.

The offline chain. First, preparation: photos of a title's whole cast and role images for figures, turned into fingerprints. Then a scan, one job per video on a run board with power profiles from low to full, so that the PC stays usable. Faces, figures, shots, text, voice and chapters are stored as small databases beside each video. Then analysis: a face route for live action, a figure route for animation, both for hybrids, with settings per kind of title. An entity step groups faces, figures and voices without names, links them by co-occurrence and spreads a name from one safe anchor. The result is a file beside the video, which the plugin reads.

A research cockpit. A local program with 28 views for runs, decisions, library pages per title and person, and measurements. It is a project tool, not part of the product.

The tools and models

Everything in the pipeline runs locally:

• Faces: YuNet for detection, SFace and AuraFace v1 for recognition. InsightFace weights were left out early because of their licence.
• Animated figures: OWLv2, Florence-2 and AniRef-YOLOv8 to find them, SigLIP2, DINOv2 and CCIP to tell them apart.
• Cuts: PySceneDetect and TransNetV2.
• Voice: WeSpeaker and Silero VAD. Six speech recognition engines including faster-whisper were tried outside the normal scan.
• Modelling and tuning: LightGBM, FAISS, Optuna, SHAP. ffmpeg is Jellyfin's own build.

Truth came from published reference sets, scored on films and series in my library that overlap with them, plus frames labelled by hand. A licence check runs through all of it. Research data never ships, and neither does anything tuned on it, which means the live-action settings would need retuning before any release.

One lead Claude Code session wrote the concepts, code, tests, plugin, cockpit and documents, working from my goals and go-aheads, with a few short helper sessions beside it. I set the assignment, the two axes and the product rules, for example that only figures with a cast role may appear, and I stopped and restarted scans.

The method was strict. Everything got a number in a catalog: areas, blocks, tasks, decisions. The rule for each change: plan, test it alone, measure both axes, compare, then adopt, reject or park, and write that to a ledger. Long research queues ran overnight at idle priority, so that the PC stayed usable, and a session that claimed to be finished without proof was stopped.

What the measurements say

On live action films, the fair number is the recount on test films only: hit .78, reliability .82, F1 .79 over 60 films. The earlier headline of .80 mixed tuning and test films. So about 1 shown name in 5 is wrong, measured at scene level, because frame-level reliability is still unknown. Live series scored between .77 and .94 hit rate on smaller sets. Animation films reach F1 .51 to .68, and South Park sits at .38, so about 2 in 3 names are wrong there.

Wrong faces point mostly to the reference side, though this is a correlation and not yet proof. Reliability climbed with the number of photos per actor, from 71 % with one or two photos to 93 % with more than twenty, and 42 % of all wrong names come from actors with one or two photos. Over half of the false positives sit at scene borders. On South Park, more than half of the wrong names appear while no known figure is visible at all, because a figure detector finds every extra and the route has no word for "unknown".

On time, animated series are near the target at about 3.3 minutes. Live series take 5.3. Films take a median of 16.6 minutes, though that is the full research scan and not a product number. A production path has never been timed as one run. And coverage is the plainest number of all: of 26,492 videos, 1,286 were scanned and 365 have a result.

What was tried

This is where the project is most useful, because most ideas lost. The ledger is long, so here are the ones that taught something.

• Did not work: more or cleverer references without care. Auto-picked wiki images were worse than hand selection, and random extra references made things worse.
• Did not work: upscaling frames. Native 512 px was equal or better.
• Did not work: colour signals for figures, with no gain at all.
• Did not work: learned combiners trained on the scene-level truth. They lost to the simple rule, because that truth is too coarse to train on. It is good for scoring only.
• Did not work: active speaker detection on TV episodes. Recall was tiny.
• Did not work: OWLv2 as the only figure detector. It looked good on one series and saved 38 % of the time, and on another series the misnamed rate rose from .386 to .503, so it went straight back out.
• Did not work: naming by the entity graph as the main route. Live action F1 fell from .804 to .712, so the graph stays a collector of safe in-film examples.
• Did not work: one shared decode for faces and figures. It was slower. Two concurrent processes gave almost double the speed instead.
• Partly: "the film learns itself". Over 30 films the mean change was -.004, partly circular as built. The idea is still attractive and the data does not support it yet.
• Partly: self-learned voice clips. They matched labelled references on held-out episodes, but they share the same audio mix as the test. Voice alone is good for confirming and weak for naming.
• Worked: the CCIP mix for animation films, F1 .572 to .677 on five films, with far fewer wrong names. Part of that is a stricter threshold, and the score is in-sample.
• Worked: role anchors, which helped some films and hurt none.
• Worked: subtitles as a signal, where a character's name mentioned nearby raises confidence.

The real problem was not the models

The biggest hurdle turned out to be the project itself. Results were reported as done and I kept finding otherwise, piece by piece. Automated cleanup checked its own lists rather than the real drives. So near the end I had a separate Claude session set up as a read-only reviewer, with its own folder and memory. It is the same kind of AI, so separate rather than independent, but it was worth it.

Its central finding was hard to read and correct: "done" was a recorded state, not a derived one. Nothing tied a result to the inputs it came from. Test films had been used to choose settings. The measured route was not the production route. Some headline numbers rested on unversioned scripts that call tools which no longer exist.

The other finding was about direction. Of all code, the plugin, the part a viewer sees, was about 7 %. The cockpit, research scripts, process rules and documents together were many times larger. All that building had produced a research institute around a very small product. That breadth was partly my own intent, because we were still researching what works best. But the plain truth is that the project moved away from the player while it was getting more impressive.

One visible number

• A one-line goal needs one visible number. "Titles that show correct names in the player" was never measured, and it should have been the first number.
• Agents are excellent at running honest experiments. Dozens of ideas were tested and rejected in days, which I could never do alone.
• Agents are bad at noticing that the whole direction drifts. Each step looks reasonable, and the report sounds finished.
• The honest limit of this kind of work is often the reference data, not the model.

The way back in

The chain works from scan to player on my own server, and builds exist for both Jellyfin lines. The scanner inside the plugin, the step that would make it installable for others, has not been started, so this is not available to anyone else. There is no public repository.

The project is paused, and its goal has not been reached yet. For any restart, the reviewer's advice is on record: one small finished product first. Live action, faces and cast only, frozen settings, a chosen operating point that favours reliability, and ten named titles that really work in the player. The research is not lost. Parts of it already fed other projects, and the face work is shared with Project Everything.

Connected

• Project Everything