Build a Private Voice Assistant for the Office with OpenJarvis and Ollama

By the end of this post you will have a voice assistant on a machine in your office that transcribes speech, answers with a local model and speaks back, with a scan that shows where every byte goes. Cheapest place to start: a 16 GB laptop and a 4B or 9B model. Comfortable: 24 to 32 GB and a 26B or 35B model.

OpenJarvis is an open-source framework for personal AI agents from Stanford’s Hazy Research and Scaling Intelligence labs, Apache 2.0 licensed. It treats energy, latency and cost as first-class metrics next to accuracy, and it runs models locally by default. The source is Ollama’s post OpenJarvis: local-first personal AI (28 May 2026); we filled in the voice details from the project docs and the repository, checked on 2026-09-09.

What you need

• Linux, macOS, or Windows (WSL2 or the desktop app). Python 3.10 to 3.13; the installer brings uv and a virtual environment (installation docs).
• Ollama, installed by the OpenJarvis installer if missing. We used 0.30.10.
• A microphone and a speaker. A USB conference speakerphone is the easiest office option.
• Memory for the model (table below) plus a few gigabytes for the speech models.
• About 15 minutes; the docs say the installer takes about three on broadband, with larger models downloading in the background.

Step 1: Install with one command

curl -fsSL https://open-jarvis.github.io/OpenJarvis/install.sh | bash
jarvis --version
jarvis doctor

Windows without WSL: irm https://open-jarvis.github.io/OpenJarvis/install.ps1 | iex. The installer sets up Python tooling, Ollama and a starter model, then jarvis starts a chat. jarvis doctor tells you what is missing; on Linux the usual gap is an audio library, which your package manager fixes.

Step 2: Point it at a local model that fits your machine

Ollama’s post uses qwen3.5:35b. Choose by memory:

jarvis model pull qwen3.5:9b          # 16 GB laptop, 6.6 GB download
jarvis model pull gemma4:26b          # 24 to 32 GB machine, 19 GB download
jarvis ask -m qwen3.5:9b "Summarise the three rules of a good meeting agenda."

Then make it the default in ~/.openjarvis/config.toml, exactly as the Ollama post shows:

[engine]
default = "ollama"

[intelligence]
default_model = "qwen3.5:9b"
preferred_engine = "ollama"

The configuration reference lists the rest: [server] binds to 127.0.0.1:8000 by default, [telemetry] writes to a local SQLite file, [traces] is off, and [security] starts in warn mode.

Step 3: Add speech in and speech out

Voice is an extra, and the local engines are open source. In the repository’s src/openjarvis/speech/ module the local pieces are faster_whisper.py (speech to text, default model size base, device auto, language auto-detected) and kokoro_tts.py (text to speech, default voice af_heart, 24 kHz, with a Spanish language code). Cloud engines (OpenAI, Deepgram, Cartesia) exist as options you do not have to touch.

pip install 'OpenJarvis[speech]'     # faster-whisper and audio capture
pip install kokoro                    # local TTS
jarvis                                # in the chat, press Enter on an empty line to record

How it works today: press Enter with nothing typed, speak, and recording stops on silence; the transcript goes to the model and the answer is spoken with Kokoro. The TTS backend order in the CLI is Kokoro first, then the cloud engines only if you configured them. There is no wake word in the repository as of today; we searched the source tree for one and found none, so plan for push-to-talk.

Whisper base has 74 million parameters (OpenAI Whisper model table) and Kokoro 82 million (Kokoro-82M model card), so both run on the CPU of a laptop. For Spanish offices, Whisper auto-detects the language and Kokoro has Spanish voices under language code e; set the voice in the config rather than accepting the English default.

Step 4: Prove where the data goes

This is the step that makes the assistant office-grade. OpenJarvis ships a data-boundary scan that audits where inference, memory, traces and connectors send data (security guide):

jarvis scan --data-boundaries
jarvis scan                           # host posture: disk encryption, open ports, remote access

Expected result with the configuration above: inference local (Ollama), memory local (SQLite), traces off, no cloud connectors. If you later run jarvis connect gdrive for the morning briefing preset, the scan will show Google as a boundary, which is the honest picture.

Ask your assistant to help here too: “Read this scan output and list every destination that is not localhost.”

What we measured

We measured the model side on our NVIDIA GB10 workstation (128 GB unified memory) with /api/generate on 2026-09-09. The speech models are small enough that the language model decides the response time.

Model	Generation speed	Resident memory	Time to a 60-word spoken answer (about 80 tokens)
llama3.1:8b	33.3 tokens/s	4.9 GB file	about 2.4 s of generation
qwen2.5-coder:7b	34.1 tokens/s	6.6 GB	about 2.3 s
gemma4:26b	51.2 tokens/s, thinking on	17 GB at 32k context	about 1.6 s plus thinking
qwen3.6:35b	51.4 tokens/s, thinking off	23 GB at 32k context	about 1.6 s

Add roughly a second for transcription and a second for speech synthesis on CPU, and a spoken answer arrives in three to five seconds on a 16 GB laptop. Third-party guides for the same Whisper + Ollama + Kokoro stack report one to two seconds of latency on a 12 GB GPU (Every Local AI). First loads are slower: gemma4:26b took 116 s to come off disk the first time, so keep the model warm.

Your machine	Model	Notes
16 GB laptop	qwen3.5:4b or qwen3.5:9b	Snappy answers; Whisper and Kokoro on CPU
Mac mini M4 24 GB, about EUR 920 (Compute Market)	gemma4:12b or qwen3.5:9b	Silent box for a meeting room
32 GB and up	gemma4:26b or qwen3.5:35b	The model Ollama’s post uses

Where this fits, and the limits

A private voice assistant fits the tasks people already shout across the office: “what is on the calendar”, “read me the last message from the supplier”, “note that the invoice went out”. The morning-digest preset speaks a daily briefing, and deep-research answers questions with citations from documents you indexed with jarvis memory index ./docs/. The privacy property is simple: audio, transcript, model and telemetry stay on the machine, and the scan proves it.

Honest limits. No wake word means someone presses a key or clicks in the desktop app. Whisper base mishears names and product codes; move to small or medium if that matters, at the cost of a second or two. One microphone in a noisy open office will pick up everyone; a speakerphone near the person asking works better. And the model limits still apply: a 9B model answers office questions well and reasons about your contracts poorly, so give it documents through the memory index rather than expecting it to know your business.

Next steps

• Feed the assistant your documents: n8n + Ollama RAG pipeline for company docs.
• Choose the silent box for the meeting room: Edge AI hardware guide 2026.
• Plan the wider rollout: How to deploy AI locally in your business.

Work with us

We install voice assistants on hardware the client owns, with the data-boundary scan as part of the handover. If you want one in your meeting room, get in touch or see how our consulting works.