This is the first post in a series sharing what we've learned engineering Betula's agents. I'm logging the key lessons here in case they're useful to others doing the same.
I'll focus on production voice agents, since that's where most of our work has been. Beyond voice, we've prototyped two related agent types for healthcare: avatar agents (voice agents with a face), and multimodal agents that can both see and speak — one of which interviewed Prof. George Church at a healthcare summit in 2025 and was demonstrated at Davos '25.
The ten lessons below are ordered roughly the way the system gets built: infra → models → harness → behavior → memory → UX. They're specific to voice agents that interact directly with humans, so many of them may not transfer to agents working autonomously in the background.
1. Platform choice
There are so many voice agent platforms to choose from now that picking one can feel daunting.
The right choice is driven by which criterion matters most for the agents you're building. For us, that was flexibility all the way down to the telephony layer — we needed it to support the capabilities our agents had to deliver. LiveKit offered that flexibility. The trade-off, of course, is more complexity to manage ourselves. We were comfortable with that.
2. Speech-to-speech vs cascaded pipeline
There's a distinct advantage to a speech-to-speech architecture, where a single model takes in audio (and prompts), makes tool calls, and outputs audio directly. Because it operates in audio space throughout, it preserves prosodic cues — tone, pacing, hesitation — and can modulate its own response in kind. A cascaded pipeline (STT → LLM → TTS) can layer sentiment analysis on top, but only on the transcript text, which loses most of the emotional signal at the STT boundary. Latency is also typically lower in speech-to-speech, since the orchestration overhead of streaming across three components is eliminated.
We chose the cascaded pipeline anyway, primarily for tool-calling reliability. Speech-to-speech models are improving on this front, and when the gap closes, we'll likely switch over — the upside in expressiveness and latency is real.
3. Choosing STT, LLM, and TTS
The choice of speech-to-text (STT) model is a continuous process, driven by both model improvements and the use case. For example, we had to pick a model that performs well in both Arabic and Malayalam (a south Indian language) for a hospital intake deployment in the Middle East. A key factor in our STT decision for production voice agents was support for diarization — knowing who is speaking. This matters for one of our assistants' core capabilities: recognizing different speakers in a face-to-face meeting where the agent is participating, typically on a speakerphone.
A persistent challenge with STT models is correctly transcribing names. They butcher them. We mitigate this where we can: Deepgram Nova-3, for instance, lets us provide textual guidance for known spellings — in our case, the business owner's name. But the model still struggles with the caller's name. And to make matters worse, since our agents have memory, if the agent gets the caller's name wrong, it will keep referring to them by that same wrong name on every future call.
We don't have a clean solution. Our partial mitigation is to automatically add callers to the owner's contact list; the owner can then add a note on how to address the caller, which the agent picks up. It's a workaround, not a fix. Newer models like AssemblyAI Universal-3 Pro support natural-language keyterm prompting mid-stream, which may eventually let us inject the caller's name into the prompt the moment they introduce themselves — a more direct fix worth evaluating.
TTS models are also improving continuously, and choosing one is a similarly ongoing process. Some current models support emotion tags in the TTS stream for more expressive speech. Our experience with this has been mixed; we use it mainly in demo videos rather than in production.
We use different LLMs for different tasks. Small models handle simple input-output transformations — for example, converting a user's spoken day into a specific datetime format. We've also used fine-tuned OpenAI models for specific tasks, but we're phasing that out in favor of small models. For the agents themselves, we run models in non-thinking mode, which fits the real-time nature of the service. We could potentially use thinking models for repetitive background tasks like generating a daily briefing, but we haven't yet — the briefing task is well-defined and runs efficiently with just memory of past briefings to avoid repeating stale items. One key factor in the LLM choice for agents is context length, which we'll revisit in the memory section.
4. How much of the harness to own
A harness, to recap, is the scaffolding around an LLM that turns a stateless next-token predictor into something that can actually do things in the world. An agent runs inside a harness.
A generic agent harness — run a loop, call tools, manage history — is one thing. A voice agent harness has a different set of concerns: real-time audio, interruption handling, turn-taking, partial transcripts, latency budgets, call state, and handoffs. The off-the-shelf options for this layer in 2026 are mainly LiveKit Agents and Pipecat.
The real decision isn't "build vs buy" — it's how much of the harness you take from a framework and how much you own yourself. For us, the answer is to build on top of LiveKit's harness and extend it where it doesn't fit or is lacking. That gives us the loop, the plugin model, and the turn-detection wiring for free, while letting us own the things that genuinely differentiate our agents: persistent memory management, context assembly, iterative and non-iterative behavior, policy enforcement, and the multi-participant scenarios our product depends on.
The trade-off is the same one we made when picking the platform: flexibility over convenience. The cost is responsibility for more of the plumbing. The benefit is that we can implement agent behaviors that an unmodified off-the-shelf framework doesn't directly support.
5. Single agent vs multi-agent
We got this wrong when we started. We began with multiple agents per call, in part because tool calling wasn't reliable yet, and splitting work across agents was a way to reduce the number of tool calls any single agent had to make.
In practice, multi-agent setups didn't hold up in production. This was true on Vapi and stayed true on LiveKit after we migrated. The core problem was simple: the handoff between agents often didn't happen. The agent in control wouldn't release the call, or the next agent wouldn't take it.
Tool-calling reliability improved over time, and that flipped the trade-off. With reliable tool calls, a single agent can do the work that previously had to be split across several, without needing handoffs at all. We consolidated everything into a single agent per call. In hindsight, our initial multi-agent design was the necessary cost of building production-quality agents while the underlying technology itself was still evolving rapidly.
Our current production voice agents are single agents — one per call, handling every action the use case requires. We expect this to hold until our agent capabilities expand to a point where multiple agents on a call become genuinely necessary. By then, we hope, agent handoff reliability will have caught up.
6. Deterministic vs autonomous behavior
Our agents are designed around a collection of skills. The choice of which skills are available depends on who is calling the agent. When the owner calls their own agent, it loads a larger set — journaling, action items, reminders, and so on. When someone else calls the owner's agent, it loads a minimal set — message-taking, appointment scheduling, and support request generation, for example.
Almost all of these skills are deterministic workflows. By deterministic I mean the agent performs a defined sequence of steps, making tool calls along the way, while conversing with the caller to gather what each step needs. The wording the agent uses in that conversation is autonomous — the LLM picks the phrasing in the moment. The only guidance we layer in is high-level: stay professional, pick up on caller agitation, and offer to generate a support ticket when appropriate.
The agent's autonomy, then, lives in two places: the choice of which skill to invoke, and the wording it uses inside that skill. The structure of the skill itself — the sequence of steps, the tool calls, the data captured — is fixed.
This split seems to be roughly the right level of autonomy for agent-human interaction. Too autonomous and the agent becomes unreliable; too scripted and it sounds robotic. Holding the structure deterministic while letting the language be free gives us reliability where it matters and naturalness where the caller notices it.
One exception to this default is iterative autonomous behavior — when the agent decides to act on its own initiative across multiple turns or even multiple calls. We cover that in the next section.
7. Autonomous iterative behavior
We have two categories of autonomous iterative behavior. In the first, no human is involved in any intermediate step. In the second, human interaction is necessary at intermediate steps to converge on the goal.
The daily briefing is an example of the first. The owner sets a directive — for example, "Iran news," and Nvidia, Cerebras, and AMD stock prices — to be delivered at a particular time each day. A background agent takes that directive, makes the individual requests in turn, and assembles the briefing. It also checks that the information is new and hasn't already been reported, so the owner doesn't get the same briefing twice. The process repeats every day at the specified time. These background agents are distinct from the call-handling agents in section 5; the consolidation rule there applied to live calls, not background tasks.
The stakes here are low. If the briefing is wrong, the owner notices and we correct it the next day. No one outside the owner is affected.
The second case is something like finding a plumber for an emergency same-day repair. The agent first prepares a shortlist based on online ratings, then calls each one, and decides who can come. This isn't in production yet — and it's the harder case for a specific reason. Calling a real person with a faulty objective is an irreversible action: it costs their time and damages trust in ways no retry can fix. Our first iteration will involve the owner at two checkpoints: approving the shortlist before calls begin, and confirming the chosen plumber after the calls.
The main takeaway for us is that iterative autonomous behavior in user-facing agents has to be bounded with checkpoints — human- or agent-supervised — to ensure each iteration is actually converging on the goal.
This is also what makes human-facing iterative agents fundamentally different from, say, coding agents. The emerging vocabulary for this distinguishes two kinds of convergence signal: agent self-assessment (the agent checks its own work) and verifiable external checks (something outside the agent confirms the work is correct). A coding agent has cheap, reliable verifiable checks — does the code compile, do the tests pass — and can iterate dozens of times against them. An agent that interacts with humans in intermediate steps doesn't have an equivalent. The "check" is the human's reaction in real time, and a wrong call has already happened by the time the check arrives. There's no compile-and-retry. It has to get it right the first time.
8. Dynamic skill loading and context injection
A note on the word "dynamic." I use it in two related but distinct ways: dynamic prompt assembly per call, and dynamic skill loading on demand. I'll also touch on dynamic context injection mid-conversation at the end of this section.
Per-call prompt assembly
The agent's prompt is constructed afresh at the start of each call. The first thing it knows is what category of caller it's dealing with: a first-time caller, an unambiguous repeat caller, or an ambiguous repeat caller. The distinction between the last two matters for privacy. A phone number can be shared — a household, a business line — so a repeat call from a known number isn't guaranteed to be the same person as last time. If we treated it as unambiguous and greeted them by name, we'd be leaking the prior caller's identity. Unambiguous means we have high confidence it's the same person; ambiguous means we don't, and the agent has to verify before referencing anything specific. This matters because, for an unambiguous repeat caller, the agent would normally check whether the current call is a follow-up to the previous one — and surfacing the previous call's reason to the wrong person would be a real privacy breach.
Dynamic skill loading
What I mean by this is loading a skill on the fly based on what the caller asks for. The prompt includes a list of available skills — names and short descriptions — but not the full implementation of each. When the caller asks for something that maps to one of them, the agent makes a tool call to load the skill's full instructions into context, then proceeds.
The trade-off here is prompt size vs latency. For frequently used skills, we load the full instructions into the initial prompt to avoid the latency of a load-on-demand round trip. For less-used simple skills, there's a small UX trick: the agent can begin servicing the request while the skill is being injected in parallel. For complex skills that require multi-step interaction, the best we can do is execute the first step and let the rest proceed once the skill is loaded.
Dynamic context injection mid-conversation
This is useful beyond skill loading. The most common case for us is the text-relay scenario: the owner gets a call while in a meeting and can't take it, but can send the agent a few sentences by text. For example, a plumber is at the building entrance asking for the gate code. The owner texts the agent the code and a brief message ("tell them I'll be there in half an hour"), and the agent relays it to the plumber in voice. The owner's text is injected into the agent's context mid-call, and the agent's behavior shifts in real time.
9. Agent memory
A voice agent's main memory components are past interaction transcripts, action sequences (both caller-driven and agent-driven), and the effects of those actions.
The central challenge is deciding what needs to be in the prompt when a call comes in versus what can be fetched on demand. This is the same trade-off every memory architecture faces — page in too little and the agent has blind spots; page in too much and you waste context tokens on things the call never needs. Our approach is to keep the minimum possible in the prompt, in the form of summaries, where "summary" covers both conversation summaries and the effects of past actions.
Take journaling as an example. The prompt at call start carries summaries of recent journal entries — enough for the agent to handle most "remind me what I said about X last week" requests directly. For older entries, the agent falls back to a tool-call search over the full journal archive. The summary layer covers the common case; the archive search handles the long tail.
We think of agent memory as being associated with each skill, with the organization of memory for a skill determined by the nature of that skill. So our memory is modular along the skill axis — different from typical harnesses that treat memory as a single uniform store. It is also hierarchical along a second axis: there's an agent-level memory and a skill-level memory.
One example of agent-level memory is the trace of recent interaction. This was essential for the multi-agent case to support handoffs. Our production systems are now single-agent (see section 5), but we kept this layer because we originally built it for multi-agent and it's still useful — for example, when a skill needs to refer back to something said earlier in the call but outside its own scope.
The shape of agent memory should follow the shape of the agent — modular where the work is modular, hierarchical where coordination requires it.
10. The interaction experience
This is where everything above shows up to the user, and it's the area that's improving most constantly. Some of that improvement comes from us — learning from user interaction traces, tightening rough edges. Some of it comes from the platform underneath. For example, LiveKit recently added a semantic turn detector that runs alongside VAD: VAD still handles voice activity, but a small language model on top decides when the user has actually finished a thought. The failure mode it fixes is the obvious one — a user who pauses mid-sentence ("I need to think about that for a moment…") no longer gets interrupted. This is a significant UX gain. It also means staying current is its own job: we have to keep pace with platform releases and incorporate new capabilities without our agent behavior drifting in unexpected ways elsewhere.
Latency is still a work in progress. The 2026 yardstick is sub-500ms perceived latency, and we're not always there. One pragmatic mitigation is a low background score during silent stretches, since the gaps can occasionally run long enough to feel uncomfortable. Some users actually find the score intrusive, so it's a one-line ask to turn off.
Although our agents are primarily voice agents, the owner can also interact with them by text. This turned out to be critical for several use cases — texting the agent from a public area to get a quick answer, for example, or relaying instructions to the agent mid-call as covered in section 8.
Most of the use cases our agents handle are deterministic-outcome workflows (see section 6). But there's at least one case where we deliberately give the agent more room: when the owner is using the agent to think out loud. The agent shifts into an active-guide mode, with the ability to search and prompt back, helping the owner refine the idea and then journal it.
At the other end of the spectrum, there are cases where we force the agent to produce a specific sentence verbatim. The clearest example: when the agent is participating in a call and someone asks it to stay silent — however they phrase it — the agent always responds with "I will stay quiet." That exact phrase serves two purposes. For us, it's a deterministic signal that the underlying tool call was invoked and recognized; for the caller, it's an unambiguous confirmation that the agent will, in fact, stay quiet.
What this post didn't cover
These ten are the architectural decisions. Cutting across all of them are evals, observability, policy, and prompt engineering — the last being where many of the decisions above actually get implemented, including, surprisingly often, as a latency lever. Each deserves its own post, and will get one in this series.
Thanks to Claude for editorial support on this post.