back to posts
#56 • Part 4 • 2026-06-20 • 11 min

The Voice Clone Hiding in My Dictation App

My dictation app had been quietly saving every recording next to its transcript for months. That is an accidentally-labeled voice corpus - exactly what a cloning model needs. So I cloned my own voice, 100% locally, from data I already owned.

local-first voice-ai csm-1b mlx privacy proof-of-concept
Read on Substack
The Voice Clone Hiding in My Dictation App

The Voice Clone Hiding in My Dictation App

My dictation app had been quietly saving every recording next to its transcript for months. That turns out to be exactly what a voice-cloning model needs. So I cloned my own voice - 100% locally, from data I already owned.

The realization

I build a dictation app called Push to Transcribe. You hold a key, you talk, it types. Nothing exotic.

One night I was poking around its data directory and it hit me: the app keeps everything. Every recording I have ever dictated is sitting in a folder. And every one of those recordings has a transcript - because transcribing is the whole point of the app.

Let me say that more precisely, because the precision is the entire insight:

I had been accumulating a labeled (audio, text) dataset of my own voice for months without realizing it.

~1,402 clips. ~3,632 transcriptions. Around seven hours of me talking, each snippet paired with exactly what I said.

If you have ever looked at how voice cloning actually works, you know why my stomach dropped a little. A cloning model does not need a studio session. It needs reference audio and, ideally, the text of that audio so it can line up sound to words. That pairing - audio next to its transcript - is the expensive part. Companies pay to produce it.

I had produced seven hours of it by accident, just by using my own app.

What “voice cloning” actually needs

Let me be blunt about what modern voice cloning is, because the phrase sounds more sci-fi than the reality.

You are not “training a model on your voice” in the deep-learning-course sense. You are doing reference conditioning. You take a model that already knows how to turn text into speech, and you hand it a few seconds of a target voice as context. It generates new speech that sounds like that context.

So the ingredients are small:

The third ingredient is the one people usually have to manufacture. I had it lying around in a SwiftData store.

The build (nothing left the machine)

I run a local voice stack on my Mac already, so the pieces were close at hand. The whole thing stayed on-device - which, for your own cloned voice, is not a nice-to-have. It is the only version of this I would ever run.

The stack:

The API is almost insultingly simple. You build Segments (a speaker id, the text, and the reference audio), pass them as context, and generate:

from huggingface_hub import hf_hub_download
from mlx_lm.sample_utils import make_sampler
from csm_mlx import CSM, generate, Segment
from csm_mlx.cli.config import MODEL
from csm_mlx.utils import write_audio

# Three reference clips - each one is (my audio, exactly what I said in it)
REFS = [
    ("refs-clean/ref1.wav", "That means we were not able to go through the entirety of the last context window..."),
    ("refs-clean/ref2.wav", "Right now I've got a push to transcribe desktop for Windows and Mac. I'm about to push the iOS app..."),
    ("refs-clean/ref3.wav", "Basically what I'm saying is I tried popping out some academic papers..."),
]

model = CSM(MODEL["1b"]["config"])
model.load_weights(hf_hub_download("senstella/csm-1b-mlx", "ckpt.safetensors"))

ctx = [Segment(speaker=0, text=t, audio_path=p) for p, t in REFS]
sampler = make_sampler(temp=0.6, top_k=40)

audio = generate(model, "Any sentence I want to say in my own voice.",
                 speaker=0, context=ctx, max_audio_length_ms=32000, sampler=sampler)
write_audio(audio, "myron-clone.wav", 24000)

That is the entire happy path. Reference clips in, my voice saying new words out. The first time it worked I just sat there replaying a sentence I had never actually spoken.

Here, listen. This is not a recording of me. It is a clone of my voice, generated on my laptop from my own dictation history, saying a sentence I never said:

(audio clip: my voice clone saying “Hey, this is my voice, but I’m not saying these words.”)

The verdict (told honestly)

Here is where I have to be straight, because the blog is supposed to show the real thing, not the demo-reel version.

It sounds like me. Not “uncanny, indistinguishable from a recording” like me - within my natural range like me. My own reaction at the time: “kind of like me, could be polished, but I sound different every time anyway.”

That last part is the real lesson about how to judge one of these. The success bar is not “does this match one specific recording.” I do not match one specific recording either. I sound different when I am tired, excited, on my third coffee. The right bar is: does the output fall inside the envelope of ways I actually sound? On the first genuine try, it did. That is a wild thing to get from seven hours of accidental data and a 1-billion-parameter model running on a laptop.

Play it again, though, and you will hear the seams - which are the interesting part. It sounds a little dull and a little flat, and neither is the model’s fault. The dullness is my microphone: a laptop mic is tuned to make me intelligible on a call, not to make me sound like me, so it quietly throws away most of what gives a voice its body. The clone can only reproduce what it was handed. The flatness is me: when I dictate I am talking at a machine, not to a person, so I do not inflect the way I would in a real conversation. The clone learned my computer voice, because that is the only voice I ever pointed at the thing. If I actually wanted a rich clone, the fix would not be a bigger model. It would be a better mic and a few minutes of me talking like a human.

And yes, I hear the irony of dropping a clip of my own cloned voice into a post that argues your voice is a credential you should not hand over. A few dull seconds of output is a very different thing from uploading the labeled seven-hour corpus that could reproduce me on demand. But I clocked it, and it is here on purpose - because the entire point is that this is mine. It ran on my machine, and nothing left the building.

The three things that actually went wrong

The happy path above hides a night of failures. These are the durable lessons - the stuff I would tell anyone attempting this.

1. CSM-1B is flaky - and I was making it worse. Roughly three out of every four generations collapsed into silence after the first phrase. Not an error - it just produces a few words and then dead air. Some of that is the model (a 1-billion-parameter model is a coin flip), but a lot of it, I figured out later, was me: I was feeding it references full of my own silence and it was faithfully handing the silence back (that is failure #2). Either way, at the output stage you defend with quantity - best-of-N is not an optimization here, it is a requirement. You generate several takes and you automatically score them (I measured how much of each take was silence) and keep the one that actually finished:

import mlx.core as mx
for i in range(3):
    mx.random.seed(i * 7 + 1)          # vary the seed so takes differ
    s = make_sampler(temp=0.7, top_k=50)
    a = generate(model, TEXT, speaker=0, context=ctx, max_audio_length_ms=16000, sampler=s)
    write_audio(a, f"take{i+1}.wav", 24000)   # then pick the least-silent take

2. The model clones your silence along with your voice - and the obvious fix backfires. Here is how I dictate, and I suspect I am not alone: I hold the key and blast into the mic, then go quiet for a couple of seconds while I think, then blast again. My recordings are full of those thinking-pauses. And CSM-1B does not just clone my timbre from the reference clips - it clones the dead air right along with it. Feed it clips where I go silent a lot and it learns that going silent is a thing this voice does, then does it: trailing off mid-sentence, or collapsing into the silence from failure #1. Dead air in, dead air out.

Obvious fix, right? Strip the silence. Wrong - and this is the nuance that matters. Aggressive silence removal (ffmpeg silenceremove with stop_periods=-1) does not just cut the long thinking-pauses, it chops the tiny word gaps too - the micro-pauses between words and breaths that are my rhythm. What comes out is unnaturally dense, artifact-ridden audio unlike anything the model has heard, and it collapsed every single time. You cannot fix silence by deleting silence, because you take the rhythm out with it.

So the real move is to separate the two kinds of quiet: the dead air (long thinking gaps, leading and trailing silence) you want gone, and the rhythm (short word and breath gaps) you have to keep. In the PoC I did the cheap version of that - selection, not surgery: I measured each candidate clip’s silence ratio with silencedetect and just picked the naturally continuous ones (my best was 5% silence). I did not clean a clip. I found one. The proper version - a voice-activity pass that trims the dead air while preserving the cadence - is exactly what the follow-up build is about.

3. Run the model in the foreground. Background-spawned generation runs got reaped before they finished. Run it foreground, or as a real service. Small thing, cost me an hour.

Why this stayed local, and why that matters

Every commercial “clone your voice” service wants you to upload samples. Read that back slowly: you upload recordings of your own voice, labeled, to someone else’s servers, so they can synthesize you.

For a novelty, maybe you shrug. For your actual voice - the thing that authenticates you to your family on the phone - that should give you pause. A voice clone is a credential. I am not uploading mine.

The entire pipeline above ran on my laptop. The corpus never left the machine. The model weights are a download. There is no account, no API key, no terms of service that can change next year. That is the whole reason I find this version interesting and the cloud version faintly alarming.

Take it further: cloning your voice from any recordings you own

The Push to Transcribe angle is what made mine effortless - the transcripts were already there. But you do not need my app. You almost certainly already have hours of your own voice sitting on disk. Here is the roadmap from “cute proof of concept” to “actually useful voice avatar.”

Step 1 - Assemble a corpus from whatever you have. Voice memos, meeting recordings, podcast appearances, old video. You need (audio, text) pairs, and if you only have the audio, you generate the text yourself: run the clips through a local Whisper model to transcribe them. Now you have the same accidentally-labeled corpus I had, just built on purpose. Filter to clips where it is only you talking.

Step 2 - Automate reference selection instead of hand-picking. The single highest-leverage lesson: continuous clips beat clean clips. Script it. Transcode candidates, run silencedetect, rank by silence ratio, auto-pick the lowest few in the length range the model likes (a handful of seconds each). Never run silenceremove. Let the data pick the references.

Step 3 - Build the real generation loop: sentence-by-sentence + rolling context + best-of-N + auto-scoring + stitch. To generate anything longer than a sentence (a paragraph, a chapter, an audiobook), you do not generate it in one shot - that is where the collapses live. You:

That loop is the difference between “a fun clip” and “a narrated document in my own voice.”

Step 4 - Graduate off the flaky model. CSM-1B is remarkable for its size but it is a coin flip. If you are generating a whole book, ~6 retries per sentence adds up. The roadmap item is to evaluate a sturdier local conditioning model (the open-weights TTS space is moving fast) so the best-of-N count drops from “mandatory survival mechanism” to “occasional insurance.”

Step 5 - Wrap it as a service. Foreground execution as a long-running local service with a tiny queue: text in, chosen-take audio out. Now it is a capability you can call from anything - a reader, an assistant, an accessibility tool.

The product-shaped version. This is the part I keep coming back to. A dictation app is already a labeled-voice-corpus generator. “Clone your voice from your own recordings, entirely on your device” is a feature whose hardest ingredient - the labeled data and the consent, because it is unambiguously your voice - is already handled by the fact that you recorded it yourself. The data moat builds itself while you use the app for something else entirely.

I parked the PoC after one good night. But I keep thinking about that folder full of accidentally-labeled audio, and about how many people are sitting on the same quiet corpus without knowing it.

Built and tested locally on an M3 Mac, June 2026. CSM-1B via csm-mlx. No audio left the machine. Full scripts + reference-selection notes live in the project’s voice-clone-poc/ directory.