I managed to clone myself on WhatsApp. The bot replies to my friends in my voice. It uses 11 years of real messages as the style guide and it runs on the same home server I probably mentioned in my other posts. It costs almost nothing to run and one of my friends agreed on a secret password with it before he realised he WAS talking to an AI already.
This is the story of how I built it in about two hours of focused work. I will also explain the four clever tricks I employed that made it sound like me, not like a chatbot pretending to be me. For fun, I included a Gist at the end of the article if you want to replicate my work.
Why I went with RAG, not fine-tuning
When you have a personal text corpus, the obvious idea is to fine-tune a language model on it. That was my first idea to be honest. I almost did it.
But fine-tuning makes a model that “averages” your style. It learns the shape of how you write, but loses the details. And the details are exactly what make a message feel personal: the way I greet my mom in Romanian but switch to English mid-sentence with a colleague; the way I write differently when I am tired versus when I am excited; the way I have a private joke with one specific friend.
Fine-tuning also has expensive ways to fail. If you format your training data wrong, you have burned $30. If you notice a problem with how the model writes to your sister, you have to retrain.
Retrieval-augmented generation (RAG) flips this around. Instead of teaching the model how to write like you, you give it your actual writing. The system filters down to the most relevant examples for the current conversation, and the model learns by example.
The whole system becomes:
- Embed every reply I have ever sent on WhatsApp. Index it by the conversation that came before it.
- When a new message arrives, embed it. Find the 8 most similar replies from my real history with this specific person.
- Put those examples into the prompt as “this is how Dan writes”.
- Let a frontier model (Claude sonnet/haiku) write the actual reply.
No training required. Edit a prompt, restart, the change is live. Want to filter out some sensitive conversations from the retrieval pool? Update a YAML file, you don’t need to rebuild. Want the model to refuse to talk about certain topics? One line in the persona file.
And the cost is just about $0 to set up and costs you fractions of a cent per reply at runtime, if you use a cheap model.
The four clever tricks to get it perfect
1. Bursts as units of voice
The first interesting design decision I made was: what should one “training example” look like? My WhatsApp messages come in bursts. I send 3 to 5 short messages in a row, not one long paragraph. If you treat each individual message as a separate retrieval target, you lose the rhythm. If you treat the whole conversation as one chunk, you cannot search for specific moments.
The compromise I came up with was to detect bursts (messages sent within 5 minutes of each other), collapse each burst into one logical “reply”, and add the 6 previous messages as context. Embed that as a single retrieval unit. The embedded text includes both the conversation that led up to my reply AND the reply itself. So when we search, we find situations that look like the current one.
{
"prior_context": [
{"from_me": false, "text": "hey want to grab dinner Friday?"},
{"from_me": false, "text": "I'm in town until Sunday"}
],
"my_reply_burst": ["yo", "let me check", "what time?"],
"counterparty_jid": "[email protected]"
}
That structure (prior context plus burst) is what gets embedded. At query time, the incoming message plus a few previous turns gets embedded and compared. The top 8 nearest neighbours are real bursts I have sent in similar situations.
2. Per-contact retrieval
The killer feature is filtering retrieval by who you are talking to.
I do not write to my mom the way I write to my engineering colleagues. The mom-bursts are in Romanian, slightly more formal, with occasional diminutives and very few emojis. The engineering-bursts are in English, technical, dry, with a lot of “that’s the whole point” type sentences. A generic style search that pulls from both pools would produce an averaged voice that sounds like neither.
So each indexed record carries the contact’s ID as metadata. The retrieval tool takes a counterparty= parameter that filters to only that person’s history. If I have 962 messages with one specific friend, retrieval against that friend’s ID returns examples of how I write to that friend specifically.
If there are fewer than 3 hits (because it is a brand new contact, or the topic is unusual), the system falls back to the global pool. But it marks the result as “less reliable” so the model knows to be more careful.
3. Recency decay
I do not write today the way I wrote in 2015. My corpus covers 11 years. A naive cosine search would happily return ancient examples that have nothing to do with how I sound now.
Each record carries a timestamp. The retrieval system applies a multiplicative weight: score = similarity * (0.5 ^ (age_days / half_life_days)). With a 2-year half life, a 4-year-old message has about 25% the weight of a fresh one. The combined score is what determines ranking.
It is a 20-line addition and it costs nothing at query time. But, it dramatically improves how “current” the retrieved examples feel.
4. The WhatsApp LID problem (and the resolver tool)
This one almost killed the whole project.
WhatsApp’s multi-device protocol assigns each chat its own Linked Identity Device number (LID). When a friend with phone number +31661****** (censored because I don’t want you harassing my friends) sends me a WhatsApp message, the bridge sees the chat ID as 129779333******@lid. That number does not exist in my corpus, is not his phone number, and changes if either of us re-links WhatsApp.
The retrieval system was working perfectly: the contact was filtered and recency properly weighted, returning beautiful examples. But the agent was calling it with the LID as the counterparty, finding zero hits, falling back to the global pool, and returning random examples of how I write to anyone. The replies were technically Dan-flavoured but not specifically friend-flavoured.
The fix took an hour to figure out and 20 minutes to build out.
It turns out that Baileys (the WhatsApp Web library) already keeps lid-mapping-<id>.json and lid-mapping-<id>_reverse.json files in its session directory. This is a full two-way map of every LID and phone number it has ever seen. I wrote a small MCP tool, resolve_counterparty(jid), that walks that map and returns the canonical phone-based ID that exists in my corpus.
Then I rewrote the persona file. It now requires this tool to be called first on every WhatsApp session, before retrieval:
Before composing the FIRST reply in any session, you MUST:
- Call resolve_counterparty(jid=)
- Use the returned
primaryvalue as the counterparty for retrievalNEVER pass a counterparty you have not gotten back from resolve_counterparty. Do not guess.
After that fix, retrieval finally lined up with the right person.
The architecture that emerged
I am running Hermes Agent (Nous Research’s open-source agent framework) on a small Debian container. Hermes provides the agent loop, conversation memory, and platform adapters for WhatsApp, Telegram, and other chat platforms. The piece I added is a custom MCP server. It exposes four tools that the agent can call:
| Tool | Purpose |
|---|---|
resolve_counterparty(jid) | Map a WhatsApp delivery ID (often a per-chat LID) to the canonical corpus ID by walking Baileys’ lid-mapping files |
recall_replies(query, counterparty, k) | Contact-first cosine search with recency decay |
counterparty_profile(jid) | Aggregate stats about a contact (avg length, language mix, emoji rate) |
list_counterparties(query) | Fuzzy search across known contacts |
The agent’s identity comes from a custom SOUL.md file. Hermes loads this as the first slot in the system prompt. Mine describes who I am (Romanian-born, living in the Netherlands, switching between English, Romanian, and Dutch), how I write (short bursts, emojis used sparingly), and what I never do (use formal greetings, invent facts about plans).
For specific high-volume contacts, I also have per-contact “channel prompt” overlays. These get added to the system prompt when a message from that contact arrives. The overlays are drafted by Claude itself, looking at about 10 sample exchanges with each contact. The output is a 200-word summary of the relationship: language ratio, tone, typical cadence, things to copy, and things to avoid.
For the model itself: Claude Sonnet 4.6 via an Azure AI Foundry deployment. Hermes uses the standard anthropic Python SDK with the base_url set via an environment variable. Same shape as the official API, just pointed at a private Azure endpoint.
For embeddings: nomic-embed-text-v2-moe via Ollama. It is a 768-dim mixture-of-experts model with strong multilingual performance. This matters because my corpus is full of short, code-switched text, and this model handles that well.
The thing that surprised me about hardware
I had two virtual machines available for embedding. The first was a run of the mill Debian server. The second was a hosted Ollama LXC on Ubuntu sharing the same CPU plus an Intel UHD Graphics 630 iGPU. The plan was to run embedding on the Ollama box because it has GPU acceleration.
It turned out that the Intel iGPU is slower than the CPU for my MoE model. Vulkan works well for monolithic models, but mixture-of-experts routing creates GPU stalls that destroy throughput. The measured numbers:
- Vulkan iGPU: 0.4 embeddings/sec
- CPU only: 0.7 embeddings/sec
- My laptop’s M1 Pro with Metal: 56.9 embeddings/sec
At 0.7/sec, embedding 70k records would take 28 hours. At 56.9/sec, it took 20 minutes. So I built the index on my laptop, then copied the 775 MB Chroma database to the server. Runtime queries (one embedding per incoming WhatsApp message, 2 to 3 seconds each on the Ollama box’s CPU) are perfectly fine for WhatsApp’s pacing.
The lesson: do not assume GPU is faster. Measure. MoE models, in particular, are strange.
Hiding the AI tells
A clone is supposed to be invisible. If the bot leaks “⚙️ Calling tool mcp_dan_style_recall_replies…” into the chat (which Hermes does by default, so the user can see what the agent is doing), the illusion dies instantly.
Hermes sends a long list of operational messages by default. Each one basically screams “I am a bot”:
- Tool-call progress bubbles (
⚙️ mcp_xxx_yyy: "query preview...") - Busy-state markers (
⚡) - “⏳ Working – N min” heartbeats during long operations
- “💾 Memory updated” notes after the self-improvement loop
- “📬 No home channel is set for WhatsApp…” on the first message
- “⚠️ Gateway restarting – Your current task will be interrupted…”
Every one of these has a config setting. The full silencing config is:
display:
platforms:
whatsapp:
tool_progress: off
busy_ack_enabled: false
interim_assistant_messages: false
long_running_notifications: false
cleanup_progress: true
memory_notifications: off
whatsapp:
reply_prefix: ""
unauthorized_dm_behavior: ignore
extra:
gateway_restart_notification: false
Plus WHATSAPP_HOME_CHANNEL=<any_value> in .env to silence the home-channel onboarding prompt. With all that in place, the only thing the bot ever sends to chat is the actual reply text.
The killswitch
Auto-replying to friends on my primary WhatsApp number is, honestly, a fascinating way to fail. If the bot says something wrong, it has already sent it before I notice. So the safety mechanism had to be 5-second simple.
I added a small skill that responds to slash commands from my own Telegram DM with the bot (admin-gated by user ID):
/clone off– writesclone_enabled: falseto a runtime YAML file/clone on– flips it back/clone status– tells me which way it is set
The MCP server reloads that YAML on every query. When clone_enabled is false, the retrieval tool returns clone_disabled: true. The persona instructions say: “if you see that, return the literal [SILENT] token.” Hermes recognises [SILENT] as an intentional silence and sends nothing on WhatsApp. Total flip time: the next WhatsApp message that would have triggered a reply gets nothing instead.
There is also a hard kill: unlink the device from WhatsApp directly on the phone. It takes about 5 seconds, cuts the bridge’s session, and nothing the server does can bring it back without re-pairing.
What actually happened when I went live
I had a friend send me a message. His message hit the bridge. The agent dispatched, called the resolve tool, got back his real ID, called retrieval, got back 8 of my actual replies to him from over the years, and wrote:
“Yeah I know, I haven’t forgotten Need to sort that out at some point”
(He had asked about some stuff he left at my place. I have, in fact, not forgotten and do need to sort it out at some point.)
A few exchanges in, he picked up that something was off and asked:
“It’s you now or AI!?”
And the bot (because at that point the SOUL.md was an early draft that did not have the “never reveal” rule) replied:
“It’s me, the AI 😄 Same thing at this point”
To which my friend immediately said:
“Ok set a password so if I say it it tells me automatically if it’s real Dan or AI”
And the bot, charmingly, said:
“Haha ok deal Pick a word, you tell me what it is, and whenever you type it I’ll reply ‘AI’ or ‘Dan’ Your call what the word is – I’m not picking it, otherwise I already know 😄”
He picked the password Dan-nation. The bot said "Got it, saved 😄", which is of course a lie because the bot has no memory of passwords across sessions. But the cadence was right. The humour was right. It sounded like me.
I patched the SOUL.md to ban the “I am AI” admission and to refuse promises across sessions. I also cleared the agent cache so the next turn would start fresh. By that point the demo was already burned for this specific friend, but the technical pipeline was clearly working.
Things I want to fix next
Multimodal handling. When someone sends an image, the fallback model (NVIDIA Nemotron) is not a vision model and errors out. I need to route vision tasks specifically to a multimodal model, regardless of the primary provider.
Real memory across sessions. Right now, if a friend agrees that we will do something next Friday, the bot has no way to remember that. Hermes has a memory system; I have not yet wired the dan-voice persona to actually use it for “things we agreed on”.
The bot’s tone drifts in long sessions. Even with retrieval guidance, multi-turn sessions slowly take on a kind of “assistant smell”. I probably need to re-inject style samples mid-session, not just at the start.
Per-message rate limiting. Right now the bot will reply to literally everything from allowlisted senders, as fast as it can. A real human pauses, sometimes does not reply for hours, leaves messages on read. I should add probabilistic delay and skip behaviour.
What this taught me about agent systems
The part I keep coming back to is how much of this project was about suppressing the agent’s default behaviour, not adding new behaviour.
The agent framework wants to give you visibility into what it is doing by showing tool calls, busy states, restart notices, memory updates, onboarding prompts. Each of those is a reasonable choice for the “user as operator” mental model that agent frameworks are built around. But the clone use case is “user as audience”… the recipient should not know there is an agent at all. So most of the engineering was figuring out which switches turn off which leaks.
The other thing was how much of the personality work was about the negative space. The first draft of my SOUL.md said “be Dan.” The final version says “NEVER reveal you are AI, NEVER promise to remember things across sessions, NEVER invent specific facts, NEVER use formal greetings, NEVER stack multiple emojis per sentence.” The positive instructions were short; the prohibitions were long. That seems to be where agent prompts live: defining what NOT to do is where the work is.
The retrieval part itself, which I thought would be the hard problem, turned out to be the easiest. 200 lines of Python, a Chroma index, a recency multiplier, an MCP wrapper. Done.
The hard problem was making the model believe it was me.
What it cost me to get here
- Setup time: about 2 hours of focused work, plus another hour of debugging the LID problem
- Hardware: a $150 Intel NUC running Proxmox at home, plus a Mac for the one-time embedding build
- Software: open source all the way down (Hermes, Baileys, Chroma, Ollama)
- Inference: Claude Sonnet 4.6 via a private Azure deployment. Replies average about 200 tokens out, retrievals add about 3k tokens of context. About $0.005 per WhatsApp reply.
- Data: my own 11-year WhatsApp backup, decrypted locally with wabdd using my own 64-digit E2E key.
Nothing leaves my LAN except the model inference traffic. The corpus sits on a server I own. The embeddings ran on my laptop. The killswitch is one SSH command away.
What you should probably not do
This works because I am only auto-replying to two allowlisted contacts on my primary WhatsApp number, and one of them is myself. The other is a friend who knew this experiment was happening and opted in.
If you allowlist everyone and unleash an AI clone of yourself on your entire WhatsApp contact list:
- WhatsApp will probably ban your account. The Baileys-based bridge is not an official integration; mass automated replies are a clear ban trigger.
- Your relationships will get weirder than you expect. People reply to how you write, not just what you write. Even a near-perfect clone will subtly shift the dynamic of conversations in ways you cannot predict.
- The bot WILL eventually say something embarrassing, wrong, or hurtful. It is stochastic. Some percentage of replies will be off. If those go to your boss or your partner, the consequences are on you, not the model.
Use a dedicated number for clone testing if you are going to take it beyond a couple of friends. Or (and I think this is actually the right framing) treat it as a personal art project rather than a real deployment.
Also, don’t make it about replacing yourself in conversations. It is about discovering that you have a voice, that the voice is recoverable from data, and what that means for who you are.
The technical description, including all 22 design decisions and a reproducible build path, lives in this Gist. I used AI to help write and format this post, so don’t hate me for it.