For Shipaton, I Built an AI Voice Memory App That Never Stores Your Data on a Server

I forget things. Not big things — I remember birthdays, deadlines, where I parked. It's the small things. The idea I had in the shower. The exact phrasing my professor used that made a concept finally click. The decision my team made in a five-minute hallway conversation that nobody wrote down.By the time I think to write it down, it's gone.So for RevenueCat's Shipaton 2026 hackathon, I built Memorly — a voice memo app that does the remembering for me. You talk, it transcribes, summarizes, and makes everything searchable. Nothing new there; a dozen apps do that. What makes Memorly different is where the data actually lives: on your phone, and nowhere else. No account. No server database of your voice. No "we value your privacy" fine print that means the opposite.Here's how it's built, and what I learned getting it there.The core loop: record, forget about it, find it laterTap the mic, talk, stop. That's the entire input surface. In the background, Memorly:Transcribes the audio using Groq's Whisper endpoint.Chunks the transcript locally using silence gaps, not an LLM call. There is no reason to pay for something a simple heuristic does just as well.Generates a real semantic embedding for each chunk on-device, using a quantized all-MiniLM-L6-v2 model running through TFLite.Summarizes the whole thing with Llama on Groq, producing a short summary plus extracted topics, decisions, and action items.That's it. No manual tagging, no folders to organize, no "remember to write a summary later." The app does the annoying part immediately, while you've already moved on to the next thing.Why local-first, and why that was the hard partThe easy version of this app ships everything to a server: audio, transcripts, summaries, and a user account tying it all together. It's also the version I didn't want to build. If an app's whole pitch is "tell me your private thoughts," the least I can do is not become the thing users have to trust blindly.So recordings, transcripts, and summaries live in a local Hive database on the device. The only network calls in the entire app are two short-lived ones: audio briefly goes to Groq to get transcribed, and text briefly goes to Groq to get summarized. Neither is stored anywhere I control.Search, including a semantic "find the moment I actually mean" search, runs **entirely on-device** because the embedding model lives in the app bundle, not behind an API.That last part took real engineering, not just a design decision. I hand-verified my Dart tokenizer against Python's `tokenizers` library token-for-token before I trusted a single embedding it produced, because a subtly wrong tokenizer doesn't crash. It just quietly gives you bad search results forever.The feature I'm most proud of: talking to your own memoriesAt some point, "search" stopped feeling like enough. Typing a query into a search bar to find something you said out loud feels backwards.So Search also accepts voice: tap the mic, ask a question, and Memorly transcribes it, searches your memories, and "speaks the answer back" using on-device text-to-speech, so even that stays free and offline.The harder problem hiding inside that feature: what happens when someone asks, "What did I do the last two days?"A normal semantic search fails here. There's no transcript chunk that *means* "the last two days," because that's not content; it's a time filter.I ended up building a small local parser that recognizes date-range phrases such as "today," "yesterday," "the last N days," and "this week." When it finds a match, it skips similarity search entirely, instead pulling every memory from that window and asking the LLM to summarize *across* them.That parser broke in an interesting way during testing: it only matched digits ("last 2 days"), not spoken numbers.Since this query mostly comes in through *voice*, and Whisper transcribes spoken small numbers as words ("last **two** days"), the digit-only version silently failed on exactly the input it was built for.I fixed it by mapping number words to integers before matching, the kind of bug that only shows up once you actually use your own feature the way a real person would.What else is in thereTimeline and Decision Tracking: Every recording grouped by day, plus a running log of every decision extracted across all of them.Action Items Inbox: A real checklist built from every action item the AI has pulled out of a memory, rather than leaving them buried inside individual recording pages.Weekly Recap: An on-demand "here's your week" summary, computed fresh every time you open it.Smart Meeting Detection: Mention a meeting time while recording, and Memorly schedules a local reminder for it without involving a calendar app.Shareable Memory Cards: Turn any memory into a clean image card for sharing, captured entirely client-side through a repaint boundary with no server round trip.A robot mascot: Somehow, this became the feature that receives the most unprompted comments.Monetization, honestlyThe free tier caps recording volume; Pro removes the cap. That's the whole model.I didn't want to lock already-shipped features such as search and timeline behind a paywall after users already had them, so the one thing that's gated is the one thing the plan always called out: how much you can record per month.It's wired through RevenueCat, which turned out to be its own small adventure. Connecting real Google Play billing involves service accounts, IAM permissions, Pub/Sub APIs, and enough moving parts that "add the SDK" undersells the actual setup work by a lot.What building this taught meLocal-first sounds like a simpler architecture until you actually build it. Every feature that would normally be "just query the database" now has to work with a corpus that's small, on one device, and can't lean on a server doing the heavy lifting.It forced better decisions, not worse ones. Brute-force cosine similarity over a personal corpus is plenty fast, and it means there's no infrastructure to maintain, no server bill, and no data breach to worry about because there's no central store to breach.The app is called Memorly. It doesn't forget things.I'm hoping that if you try it, you won't either.