Building a real-time desktop AI copilot for calls: the hard parts

Half a year ago I asked a simple question: during an online call, could a short, to-the-point hint appear on my screen in a second or two — while the other person is still talking? Not an after-the-fact transcript, but help in the moment. The result is a desktop assistant (macOS + Windows). Below is an honest breakdown of what turned out to be hard, and which solutions worked. Engineering only, no marketing. Architecture in one paragraph On the device there are only two things: audio capture a...