Building Wukong Copilot: a real-time AI interview coach

Earlier this year I wanted a side project that would force me past CRUD work and into real-time audio, streaming inference, and desktop app engineering. Wukong Copilot is what came out: a desktop-first AI interview coach that listens to the interviewer, transcribes them live, and surfaces short, resume-grounded hints somewhere the interviewer can't see.

The problem

Browser-based AI interviews (Zara and similar) move fast, and two failure modes hurt the most:

• Recall failure — blanking on a metric, stack, or timeline that is literally on your own resume.
• Structure failure — rambling through a behavioral answer without landing a result.

The tool exists to fix those two, and to stay out of the way while doing it.

How it works

system audio (any browser / Zoom / Teams)
      │  Rust cpal capture, re-routed back to your speakers
      ▼
desktop app ──16 kHz PCM16──► Deepgram Nova-3 (streaming STT)
      │  transcript events
      ▼
coaching brain (runs in the desktop process)
  ├ question detection (regex + session profile)
  ├ context manager (resume index + rolling memory)
  ├ coaching model (OpenRouter → Groq preferred, DeepSeek fallback)
  └ suggestion bus ──┬─► notes window   (second monitor)
                     └─► your phone     (in-process relay over wifi)

Nothing renders until you start listening. When you do, captured audio is re-routed back to your speakers, so the meeting stays fully audible.

Desktop-first architecture

I originally scoped this as a Chrome extension that did everything. That version worked, but hit a hard wall: a Chrome window can't hide itself from a full-screen share. So I pivoted to desktop-first. The Electron app owns the entire coaching brain (audio capture, STT, question detection, context, LLM, and the hint sinks), and the Chrome extension shrank to a slim support tool that captures the interviewer's exact on-screen text and AI-eval requests, then fuses that exact wording over the transcription — so "C#" is never mis-heard as "c hash".

The engineering decisions

Real-time audio, without muting the call

System audio is captured with Rust (cpal) and pushed into the pipeline as 16 kHz PCM16. Capturing without muting isn't a nicety — if you can't hear the interviewer, the tool is useless.

Streaming transcription

Audio streams to Deepgram Nova-3 over a WebSocket, producing interim and final transcripts. Words the transcriber is unsure about are marked in the UI, which turns silent mis-hears into visible ones.

Hitting the latency budget

End-to-end latency — end of question to first hint — targets p95 < 700 ms. Most of that budget is won with speculative generation: when an interim transcript already looks like a complete question, the LLM call starts early; if the final transcript diverges, the in-flight call is aborted and re-fired.

Grounded context (never invent facts)

The coaching prompt is built so resume retrieval is grounded strictly in a structured fact store. The model is instructed to state absence rather than fabricate. A rolling conversation memory compresses old turns into a summary once the window hits about 70%, so a 60-minute interview never overflows.

Provider adapters and cost

STT and LLM sit behind narrow interfaces, so swapping providers is a settings change. Transcription dominates the cost; the whole thing runs around $0.50 per interview-hour.

Off-screen delivery

Hints go to a borderless notes window on the non-primary monitor, with content protection set so the OS excludes it from capture. For sharing your entire screen, the fallback is your phone: the desktop app runs a WebSocket relay in-process, and a token-guarded page reconnects on its own and holds a wake lock. Because the relay binds to your LAN, that token is the only thing between you and a stranger on the same wifi.

What I learned

• Adapter boundaries pay off. Every STT/LLM swap I made was a settings change, not a refactor.
• Speculative generation is where latency is won. Starting the LLM call on a confident interim and aborting on divergence is most of the p95 < 700 ms story.
• Audio passthrough is a correctness requirement, not a feature.
• MV3 has real constraints. A service worker can't hold a MediaStream, so audio lives in an offscreen document.