Voice Cloning & Voice Morphing

Part one: voice cloning

The chain from raw audio to a dataset

Every stage was researched first, with versions, sizes, licences and alternatives, and several tools I started from were replaced, with reasons.

One thing was left out deliberately: no de-reverb. In a heavily processed voice the reverb and the colouration belong to the voice, so removing them would have removed part of the target.

The pipeline then runs by itself on a folder of videos. It joins the audio, transcribes in blocks and builds speaker groups. It accepts a group as the target only if it speaks in at least 30 % of the videos, or clearly leads with twice the videos and 1.5 times the speech of the runner-up. Otherwise it stops and I choose on a listening page. It separates music only around those passages, cuts sentences of 1.5 to 20 s with similarity of at least 0.6, and removes clips from videos with the same content. Two source files of equal length within 0.5 s and cosine of at least 0.98 count as near-duplicates. A profile counts as "full" from 20 clips and 600 s, and as "few" from 3 clips and 60 s. Under 5 clips or 60 s the pipeline stops, and there is no profile. A wrong voice would be worse than a question, so the pipeline never guesses.

Text-to-speech engines

The thinking behind the choice: if a later step can shape the timbre, the engine must above all deliver good German.

VoiceStudio 0.5.6 was inspected as a possible base and then not used. Its full environment was large, and a slim one around OmniVoice was enough.

A reference is not simply a clip. The agents tested the best long single clips against joined sets of two or three mid-length clips, always with the same test sentences. Joined sets won for some voices at about 16 to 23 s, and a single sentence for another. The score is only a shortlist. For the first voices I heard the final candidates myself, and where a music-separated and an unseparated reference competed, separated won. In the batch runs the best score picked the reference automatically, and I listened to the finished voice. Pocket showed a hard limit here: references of 30 s or more broke it, down to 0.36 and 0.00.

The refinement step

The first clone of the first voice had the right timbre but lost the heavy processing of its source. The first idea was classic audio effects, with 300 effect chains searched automatically. The score only moved from 0.754 to 0.770, while the originals scored 0.866 among themselves. Speaker prints are trained to ignore the channel, so the metric cannot see effects at all.

RVC learned the sound directly instead, from just 28 clips and 121.7 s, where the plan had asked for about ten minutes. The setup was Applio 3.7.0 with RVC v2, ContentVec as embedder, RMVPE, and the pretrained 40 kHz HiFi-GAN base: 300 epochs, 2,700 steps, batch 8, 21.1 minutes, about 5.1 GB of VRAM. Applio replaced the original RVC package because it covers the same ground with a finished Windows build. Generator loss was lowest near epoch 214, but the model in use is epoch 300, tested by ear against epoch 150. Inference runs at index rate 0.75 and protect 0.33. My verdict: perfect.

The same recipe on an unprocessed voice (150 epochs, 10,650 steps, 58 minutes) went the other way. With RVC it sounded artificial, and OmniVoice alone sounded better. So refinement exists for one voice only, and it is optional in the format.

Sixty-three seconds to the first sound

Speech takes about 2 s per sentence, but the first plain script needed 63 s to the first sound, nearly all of it model loading. The agents built a local speech server that holds both models in one process, plays sentence by sentence while the next one is generated, and parks models in RAM when idle or short of GPU memory.

The faster engines lost by ear, so speed went to the warm path, not to a weaker voice. The only way to speech without a GPU is an idea that has not been started: distilling the chain into a small CPU model trained on its own output.

Results for cloning

• Worked: OmniVoice as the single engine for all 15 voices.
• Worked: RVC refinement on a heavily processed voice, from two minutes of audio.
• Worked: the automatic pipeline, including its stop rule. Merged groups, a manual cut and one source with no usable speech were handled as exceptions, and that source got no profile.
• Did not work: effect chains. The metric could not see them, and RVC made them unnecessary.
• Did not work: RVC on unprocessed voices.
• Did not work: Pocket and Piper as replacements. They were faster, but clearly worse by ear.
• Did not work: tuning Pocket towards OmniVoice, with 11 samples and 9 settings. Pocket reached 0.846 to 0.878 against 0.812 for OmniVoice plus RVC, the settings only moved the score within its noise, and I still heard OmniVoice as better.

Part two: voice morphing

Why it needed a different tool

The text chain cannot do this, because a recording has to keep its timing and delivery. RVC was the only converter in the project, and it already had a known weakness. The test bench was strict: 20 clips, 41 s, every output exactly as long as the original within 10 ms, at the original's loudness. Methods ran one at a time, and the results went onto HTML listening pages with the variants side by side.

The candidates are built very differently. RVC is a trained model on ContentVec features. Seed-VC V1 uses a Whisper small encoder, CAM++ style features and a 44 kHz BigVGAN vocoder. Seed-VC V2 uses the ASTRAL tokenizer on HuBERT large features and a 22 kHz BigVGAN. Vevo2 from Amphion uses flow matching with Whisper medium content features. Chatterbox uses its S3Gen stage. OneVoice looked interesting but had no weights available.

Round one: five methods

Similarity comes from WeSpeaker against the target voice. The table separates sentences from non-speech vocal sounds, where nearly everything collapsed.

Round two: 34 variants

Before settling, I wanted every method pushed to its maximum with little material. The target had only 27 clips and 152 s. References were sorted into standard, expressive, soft, loud and long, and a grid ran on 8 typical clips: Seed-VC 12 variants across reference with auto F0 or cfg, Vevo2 6 across reference and pitch, Chatterbox 8 across two models and four references, RVC 8 across index and protect, on a new model from 2.5 minutes. MossFormer2_SR_48K lifted 22 and 24 kHz outputs to 48 kHz.

An expressive reference raised the scores almost everywhere. By ear nothing changed: only Chatterbox hit voice and modulation at all.

Round three: only Chatterbox

Three voices, all 20 clips, v3 with standard and expressive references, each raw and with restored highs. The English original S3Gen was already out, because it mispronounces German, for example the g. The simplest setting won: v3, standard reference, no restoration. So the expressive reference that lifted the scores and the restoration step both stayed out.

Turning a converter into profiles

Chatterbox takes only about 10 s of reference and varies from run to run, so one reference and one take were not enough. Each voice now gets a ranked set:

• Candidates: good sources of 5 to 10 s with similarity of at least 0.60 or the median minus 0.03, whichever is higher, not clipped, not too quiet, at most 60.
• Pre-round: every candidate on 8 clips with 2 takes.
• Final round: the best 3 on 20 clips with 6 takes, fixed seeds, best take chosen automatically.
• Score: similarity plus modulation, divided by two. Modulation is the correlation of log pitch and loudness contours between source and output. Dropouts halve the score.
• Storage: samples under 5 s go, and so do samples that overlap a better one by more than half.

This ended with 5 to 60 ranked samples per voice and a settings file recording source position, measurements, seeds, the ranking method and Chatterbox's implicit Perth watermark.

Results for morphing

• Worked: Chatterbox multilingual v3, zero-shot, about 1.7 GB of VRAM, 0.6 to 2 minutes per run, for all 15 voices.
• Did not work: RVC as a general converter. It had the best number in round one but lost by ear, and it needs training.
• Did not work: Seed-VC V1 and V2. V2 topped round two and still lost by ear. The project also looked unmaintained in the first research.
• Did not work: Vevo2. High scores, the highest VRAM use, rejected by ear.
• Did not work: the English S3Gen model, because of its German.
• Did not work: high-frequency restoration, which the winning setting did not need.
• Did not work: OneVoice, because no weights were available.
• Partly: non-speech vocal sounds. Every method stayed far below its sentence score, and these clips also pull down averages, so the format notes that.

Ear against metric, and a shared card

• The metric against the ear. Three separate tests said the same thing: Pocket against OmniVoice, then both converter rounds. Similarity ranks well enough to shortlist, but it does not predict what sounds better. Converter output normally sits at 0.5 to 0.6, not 0.8 and above. The modulation term exists for this reason, but whether it predicts my verdict is not documented.
• Very little material. Two minutes for a trained model and ten seconds per converter reference meant that selection mattered more than volume.
• Knowing who is speaking. Speaker clustering split one speaker into two groups, or found no clear leader. That required stop rules, listening pages and sometimes a manual cut.
• German. Several strong models are weak or absent here, which ruled out Kokoro and the English S3Gen.
• A shared GPU. Other projects used the same card, so models ran strictly one after another, with VRAM checked first and a guard at 1 GB free. torch.cuda.mem_get_info does not see other processes under WDDM, so nvidia-smi became the source of truth. On this card generation CTranslate2 has switched INT8 off, which conflicts with the standing Whisper setting and still has to be measured.
• Memory on long sources. Computing prints over whole sound tracks exceeded the 31.4 GB of RAM, so all audio work moved to windows, blocks and one source at a time, under a 12 GB cap per process. The largest round peaked at 10.3 GB.
• Precision. OmniVoice cannot run fully in half precision, because its audio tokenizer needs fp32.
• Levels. References ranged from -45.8 to -18.4 LUFS, and engines inherit the level. The format moved to -18 LUFS with peaks at -1 dBFS, and the audit now measures loudness itself. Levelling changed modulation by up to 0.13 on quiet samples, so ranks would turn. A full re-measurement would cost about 8 hours of GPU time. I declined it, and the files say the ranking predates levelling.
• Terms. "Changer" and voice_conversion for the RVC step kept getting mixed up. The format now says conversion for morphing and refinement for RVC after cloning.

A format that outlasts the tests

A voice profile holds a 256-dimensional print averaged over the clips, measured pitch, speech rate, loudness, bandwidth and noise floor, a provenance record by content ID, the clone settings with reference and transcript, four JSON analysis files, the conversion folder, and the optional refinement model. Every setting that changes the sound is stored, so a result can be rebuilt. Model files load only with weights_only, because .pth is a pickle format. Training clips, code and notes stay out. Each export is validated against six JSON schemas. The format was audited before release, with 24 findings on the first version and 17 on the refinement section, and it follows the same rules as the face and body profiles.

All 15 profiles passed with 0 errors and are accepted. Every verdict rests on one listener and every score on one speaker model, which is why I do not count the measuring problem as solved.

Why it stays yellow

The only result that really works needs massive graphics memory in the background. That is fine for generating files: text goes in, and a finished voice file comes out. It does not work for real time. Even real time has to make the files first. So there is still no good real-time approach, for example for a talking chatbot.

Connected

• Masterface
• Masterbody