I keep a lot of notes, and I rarely remember the words I used in them. In March I wrote a note called "Rent split with Arjun". In October I searched for "what does my flatmate owe". Keyword search found nothing, because the two share no words.So when I built VB Notes, a markdown notes app where every note is a plain .md file in a folder you choose, search by meaning was the part I cared about most. I also wanted it to run on the device. Notes are private, and I wouldn't use a search box that sends every question to a server for my own notes. This is what it took to run a 350-million-parameter embedding model next to the notes on an iPhone, and the same model on Mac, Windows and Linux. VB Notes on iPhone: searching "what does my flatmate owe" finds the note "Rent split with Arjun", marked "by meaning" Picking the model The phone app is Expo and React Native, and the model runs through react-native-executorch. I wrote 18 questions, each worded differently from the note it should find, and ran them against the models the library ships: LFM2.5-Embedding-350M (about 193 MB): the right note first 17 times out of 18. Unrelated notes stayed under 0.15. all-MiniLM-L6-v2 (44 MB): right first 16 times. Unrelated notes reached 0.21. paraphrase-multilingual-MiniLM-L12-v2 (about 235 MB): on the same test plus Hindi, right first 19 times out of 22, and the only one that matched Hindi notes to English questions and back. Unrelated notes reached 0.25 to 0.45, so it brings more noise. The gap between the right note and an unrelated one mattered more to me than the hit rate. It decides how cleanly you can cut the list off. LFM2.5 became the default, MiniLM is the "Light" choice in Settings, and the multilingual model is there for people who write in more than one language. LFM2.5 also reads a question and a note differently: questions go in with a query: prefix and notes with document: . Chunks, with the title on every one A model reads a limited number of words, so long notes are split into chunks: 200 words each for LFM2.5, each overlapping the previous one by a fifth, at most 16 per note. Every chunk starts with the note's title, so a paragraph deep inside "Trip to Ladakh" still knows it is about the trip. Code blocks and Mermaid diagrams are left out. They are noise for meaning, and keyword search still finds them. Tags lose their # and count as words, so #work/clients still says what the note is about. int8 vectors in SQLite Each chunk becomes 1,024 numbers. As float32 that is 4 KB per chunk. I store each vector as int8 with one float32 scale in front: a quarter of the size, and the cosine similarity barely moves. The vectors live in a SQLite file in the app's caches folder, out of iCloud backups. If iOS clears it, the app embeds the notes again. The .md files are always the source of truth. At launch the app reads only each note's hash, to see what changed. The vectors load on the first search and stay in memory. The model itself waits three seconds after launch, unless you open Search first, so opening the app stays quick. Where to cut the list off Cosine scores from a small model are not probabilities. A fixed floor of 0.3 sounded reasonable and dropped most of the right answers: short questions score between 0.15 and 0.3 even against the note they are about. What works is two rules. Each model gets its own floor, at the level where unrelated notes stop (0.12 for LFM2.5). And a match is kept only if it is within 0.25 of the best match, up to ten matches. Words and meaning in one list Exact words still matter. If I type "Diamox", I want the note that says "Diamox" first. Keyword search is SQLite FTS5 on word starts, so "gro" finds "Groceries". The two lists are merged with reciprocal rank fusion: a note scores 1 / (60 + its rank) in each list, and the scores add up. A note found by both rises to the top, and a note found only by meaning still shows. I never have to compare a keyword score with a cosine score, which would be comparing two different things. On the desktop, a single word only brings in meaning matches when no note contains it. One word gives the model little to go on, and the exact matches are what you want. Getting 193 MB onto a phone The first surprise was the download. I expected an iOS background download to be the right tool. On the simulator, the app's own download ran at 2.4 MB/s, while an iOS background session managed 0.05 to 0.9 MB/s. So the download runs inside the app. The screen stays awake while it runs, and if you leave the app it stops, then carries on from the partial file when you come back. The desktop: the same model through llama.cpp The desktop app is Tauri 2 with Rust, not Electron. It uses the same model, as the official Q8_0 GGUF file (380 MB, checked by SHA-256 and downloaded on first launch), run by llama.cpp through the llama-cpp-2 crate: on Metal on Apple Silicon, on the CPU elsewhere, on its own thread. Notes are chunked by the same TypeScript code as on the phone, and a background thread embeds them newest first, so recent notes become searchable first. The same chunks and the same model mean a question finds the same notes on both. The desktop quick finder: typing "fitness" offers to create a note and finds "Half marathon plan" by meaning What I would tell someone trying this Test on your own questions. Eighteen reworded questions told me more than any benchmark table. Look at the gap, not only the hits. Where unrelated notes stop decides how clean the results look. Put the title on every chunk. Don't use one fixed threshold. Use a floor per model and a window below the best match. Keep keyword search. Merge the two lists by rank instead of replacing one with the other. int8 is enough for vectors you only compare by cosine. Try it: VB Notes is free and open source (MIT), for iPhone, Android, Mac, Windows and Linux. There is no account and no server: notes stay plain files, and search runs on your phone or computer.I'd like to hear how you handle search in your own notes, and what would make you trust a model running next to them.
Running an Embedding Model Next to My Notes on an iPhone
Full Article
Original Source
Read the full article at Hackernoon →KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.