Home Case studies A voice assistant that never leaves the building
Case study · Private AI
A voice assistant that never leaves the building
We built a voice assistant where speech recognition, the language model and the reply voice all run on a server in the room. No cloud service hears a word. And in our tests, its first word came faster than from the cloud models we tried.
The problem
Some data can't go to the cloud
For many Western Australian organisations, the question isn't whether AI is useful. It's whether they are allowed to send the data anywhere. Most AI assistants work by shipping every question, and often every recording, to a provider overseas.
We wanted to show that a useful assistant can run entirely on hardware you own, and that private does not have to mean slow.
- Unannounced results. An explorer's new assays and drilling results can't sit on a third-party service before they are reported to the market.
- Patient and client records. Health services, councils and agencies hold records that must stay inside their own systems.
- Remote sites. A mine or a station with patchy internet needs AI that keeps working when the link drops.
Faster than the cloud
Time to the first word of an answer, same short question. Local runs used the open-weight Qwen3 8B model, except the small Jetson board, which can only fit a 3B model.
Why did a 2080 beat a 5090? For short questions, the first word is limited by start-up overhead, not raw power, and the Windows virtualisation layer adds real delay at this scale. A newer card is not automatically the faster choice.
Where the time goes
From the moment you stop talking to the moment the reply starts. People expect a reply in about 200 ms in conversation, and anything under 500 ms feels natural.
- 193 msWaiting for enough silence to know you've finished. This is the physical floor.
- 18–27 msThe model's first word, on the local server.
- 70–166 msCollect the first sentence and turn it into speech.
- 0.7 msSending a second of audio over a wired office network.
What we built
Small device in the room, real work on the server
Each part runs where it makes sense. Work that is cheap or private stays on the small device. Work that needs a GPU runs on one server on your network that can serve every room.
The assistant can call your own tools, such as a search over internal documents or an orders database. Having tools available costs nothing when a question doesn't need them: 373 ms end to end, the same as without tools. Actually calling one took 567 ms, and the assistant says "let me check" to cover the wait.
- Wake wordon the device, 39 MB model, 21× faster than real time
- End of speechon the device, 193 ms
- Speech to texton the server, streamed while you talk
- Language modelopen-weight 8B, first word in 18–27 ms
- Reply voiceon the server: 52 ms, against 246 ms on the device
- Your toolslocal functions or services on your network
Where it stands
Fast is solved. Accurate in a real room is next.
The numbers above come from repeatable tests that play recorded speech into the device. Our first test with a real person talking in a real room was slower, at 426 to 628 ms, and showed what still needs work before this is ready for daily use.
We publish this because it is the honest picture. What it already proves is the part most people doubt: AI on your own hardware can be fast enough to talk to.
- Proven: the full chain runs locally, from wake word to spoken reply, with no cloud service.
- Proven: a local GPU gave the first word faster than both cloud models we tested.
- Next: speech recognition with room echo and natural speech, not just clean recordings.
- Next: a more reliable wake word, and answers that stick to your tools and data.
Who it helps
The same approach works for any of our applications, not only voice. Every one can run on your own hardware. More on private, on-premise AI.
Miners and explorers
Ask questions of drilling data and reports before they are announced, without them leaving your server.
Health and community services
Assistants that work with client and patient records inside your own systems.
Government and councils
Staff and public assistants that answer from your own records, on your own infrastructure.
Under the hood
Lessons that make local AI fast
Most of the speed came from small engineering choices, not bigger hardware. These are the ones that mattered most.
- Keep the model loadedby default it unloads after 5 minutes; reloading cost 2.8 s
- Speak the first sentence earlystart the voice at the first full stop, not the last
- Separate jobsone model decides you've finished, another transcribes
- Skip hidden reasoningotherwise tool calls slowed from 200 ms to 1.1–2.5 s
- Check the cablebad Wi-Fi took network delay from 1.7 ms to 149 ms
- Test end to endevery deploy speaks a sentence and checks the reply
Keen to have a chat?
Schedule a coffee chat in Perth, an online meeting or an in-person meeting at your office. A rough idea of what you'd like to improve is plenty.