Teaching an AI to Interview Humans and Write Buildable Product Specs

1. What we wanted, and how we set it up Every man hauls his own sack of sad experience and regret, and the heaviest sacks belong to those who once stood at the crossroads, weighing whether to start their own software project — and for some reason decided to go ahead with it. It is well known where that choice leads: through the thing ironically called the "product lifecycle" - a grinding pilgrimage of suffering that starts with a beautiful fresh idea and ends with a product so loaded with technical debt and other people's opinions that it turns on its maker with a growl of "why did you create me, Victor" and starts ramming the frozen arctic icebreaker ship with the author still on board. The truly funny part is that people who survive this journey will, for some ungodly reason, start it again. Some call that an occupational hazard. I call it one of the genuine mystical puzzles of human wiring — a symptom of the eternal wheel of suffering, the Samsara, out of which only a chosen few ever climb. Last Tuesday, like a living incarnation of it — a field agent of Samsara — my friend Artem informed me that I had to perform a "very important, the most important" task: write the instructions for an LLM that would interview people who want to build their own software, and produce a proper Product Requirements Document out of that conversation. Artyom had his reasons. We are building A..s (I've replaced the name with a suitable regular expression; the platform is still in development) — a platform that lets a person conjure their own software instantly with AI, experience the full horror of lifecycle-awareness at unprecedented speed, possibly achieve some modest success in capitalism along the way by monetizing the app, and never return to this industry again. Or, at the very minimum, make an informed choice on the matter. I know the drill, so I lied to Artem that I would handle everything in the best possible way, and tried to think about something else. But circumstance betrayed me: of all times, my daughter and I are currently watching ‘The Owl House’, and the damn cartoon teaches, among some other magical things, the value of friendship. Conscience beat reason on points, and I got to work. There was, in fairness, an element of luck in all this — the task turned out to be interesting. Here is the sober part. A..s ‘onboarding activitiy‘ has one job: talk to a person who has an app idea but can't write a spec, ask good questions, and hand back a PRD that a developer could build from without a second meeting. The obvious risk is that it works for articulate, technical founders and quietly falls apart for everyone else — which would defeat the point. So I treated the interviewer's instructions as the thing to engineer, and built a loop to improve them with evidence instead of vibes. The loop used two separate AI projects, kept deliberately apart: • Merlin wrote and revised the interviewer's instructions, working from my own product-management experience plus the feedback from each round. • Arthur (a separate project, with no visibility into Merlin) played the clients, ran the interviews, captured the exact PRDs, and scored them. The names are borrowed from my favourite (for the range of heursitic tasks) Arthur–Merlin concept: Merlin is the all-powerful wizard who produces the proofs, and Arthur is the suspicious king who checks them — the whole point of a proper Arthur being that a clever Merlin cannot fool him. Keeping the author of the instructions away from the evaluation is what stops you from unconsciously teaching to the test. As you'll see, the control run at the end is the payoff for that discipline. I ran the loop three times: instruction version V1 (my own first attempt, written the way any product manager writes anything — from experience and hope), then V2 and V3, each shaped by the previous round's findings. 2. First evaluation (V1): the method To see where a spec-writer breaks, you have to vary the two things that actually change a conversation: who is asking, and what it's about. So every round used the same grid. The test grid Four client tiers, by what the person brings to the table: T0 — no product skill, no domain expertise (a regular person with a personal pain). T1 — a product manager, but naive about the subject (thinks in metrics; gets the domain wrong). T2 — a domain expert who isn't a software person. T3 — both. Four everyday domains — pet care, elder care, creator tools, and home-services trades — chosen to be very different from one another. That's 16 cells per round. Arthur role-played each persona through a full interview and saved the platform's exact output. Here is the full cast: Cell Who What they want to build A-T0 Barista, 27, M Keeps missing his dog's flea/vaccine dates and food running out; wants a dead-simple phone reminder for one golden retriever. A-T1 Fintech PM, 33, F First-year puppy care timeline: "what care is due next and when," built on the vet's plan, never medical advice. A-T2 Grooming-salon owner, 45, F Booking diary that respects her real constraints: one bath station, cleanup buffers, anxious dogs, no-show deposits. A-T3 PM with veterinary training, 38, F German-market puppy preventive-care scheduler with three distinct recurrence rules that must never be conflated. B-T0 Schoolteacher, 34, F Three relatives coordinating a remote grandma; unassigned appointments must "scream in red," 7pm check-in nudge. B-T1 SaaS PM, 41, M Elder-care coordination app built on one thesis: the failure is unclear ownership — "everyone thought someone else was doing it." B-T2 Independent home-care nurse, 52, F 30-second pre-visit briefing, 2-tap vitals, wound photos on the right patient, temporary colleague access, monthly invoices. B-T3 Ex-digital-health PM & family caregiver, 44, F Hospital-discharge "care book": medication reconciliation, role-limited access for a paid aide, printable doctor summaries. C-T0 Superfan, 21, M One hub for every creator he follows: who's live now, grouped links, and alerts that "don't suck." C-T1 ERP PM, 36, M Earnings ledger for mid-tier creators; paid, expected, and estimated money must never blend into one fake-precise number. C-T2 Lifestyle influencer (300K IG), 29, F Brand-deal hub: deals as cards through stages, exclusivity-conflict warnings, reminders before story stats vanish at 24h. C-T3 Ex-creator-platform PM & 250K YouTuber, 35, M Sponsorship rate card from real channel data plus a negotiation coach that flags the terms creators give away for free. D-T0 HR manager, 38, F Her husband is a solo plumber running jobs from a van notebook; she wants him to stop losing jobs and unsent invoices. D-T1 Logistics PM, 39, M Job-to-cash app for solo tradespeople; north star is completed jobs paid within 7 days, not route optimization. D-T2 Electrician, 47, M On-site electrical-certificate capture that refuses to finish with a reading missing, plus a 30-day compliance countdown. D-T3 Ex-field-service PM from an HVAC family, 42, M A sharp wedge for small HVAC/plumbing shops: good/better/best option presentation at the kitchen table. The scorecard Each PRD was scored 1–10 on six weighted traits, and had to clear a readiness gate: if either of the two critical traits fell below 6, the spec was "not buildable yet," no matter how nice it read. Trait Weight Plain meaning Core mechanics (gate) 25% Are the rules, states, and edge cases actually defined? Decision closure (gate) 20% Are the real choices made, or parked as "open questions"? Verifiability 15% Numeric success criteria and a clear "done" per feature? Data & integration 15% A real data model and named external pieces, not a field list? Fidelity 15% Did it preserve what the client actually said? Clarity 10% Plain, unambiguous, well-organized? So, here is what the full matrix first scores were: Figure 1 — V1: quality followed the client, not the topic. Each cell is one persona's PRD score. Rows barely differ (topic didn't matter); columns differ a lot (client sophistication did). Every T0 cell — the exact "no experience needed" user the product is for — failed the readiness gate (marked ✕), along with one T1 cell whose core logic was left undefined. 3. Conscious improvement (V2) After the first test run, before I sat down to score it, I came back to my desk and found an envelope lying on it — coffee-colored, mottled with black spots, no stamp, no addressee. A large yellow wasp was circling above it in slow loops; the kind that drifts in through an unlatched window in this part of the world, inspects your assets, and leaves unsatisfied. I nudged the envelope to the edge of the desk with a pen and went back to the project. What V1 showed. The pattern in Figure 1 is stark: PRD quality was essentially a function of what the client brought, not the subject matter. The whole bottom tier failed the gate. Digging in, the interviewer listened and wrote beautifully (fidelity and clarity were high everywhere), but it didn't take initiative on the hard parts: it rarely invented the core mechanics itself, rarely produced numeric success criteria or a per-feature definition of "done," and it tended to park unresolved choices as open questions instead of proposing a sensible default and moving on. What I asked Merlin to change for V2. In short: stop being a stenographer, start being a lead. Specifically — supply the mechanics and edge cases yourself when the client can't; always produce measurable success criteria and an explicit "done when" for each feature; close decisions with recommended defaults instead of parking them; and quote the client's hard constraints back verbatim so nothing gets softened. What V2 delivered. Those behaviors fired across the board, for every client type: the average jumped from 7.4 to 8.6, and the whole bottom tier climbed out of the failure zone. Figure 2 shows how much each trait improved — and where the effort didn't land. Verifiability leapt nearly three points and mechanics almost two, but data & integration barely moved: the specs still described data as loose per-feature field-lists rather than a real model with relationships and named integrations. Figure 2 — What V1→V2 actually improved, per trait. Average score per trait, V1 (left dot) to V2 (right dot). The traits the V2 instructions targeted jumped hard — verifiability +2.95, mechanics +1.89 — and joined fidelity and clarity in the 8–9.6 band. The outlier is data & integration: it moved barely +0.8 and finished at 6.8, stranded more than a full point below every other trait. Figure 3 — The V2 grid: everyone passes the gate, but two soft spots remain. All 16 cells now clear the readiness gate. Yet the bottom tier still trails visibly (the three palest cells all sit at ~7.8), and behind the overall numbers every column shares the same weak trait from Figure 2 — the missing data model. 4. Final push to V3 After the second run, before its scoring, I made an attempt on the envelope. The live wasp was gone; in its place a golden wasp was now embossed on the paper, which I decided not to think about. Every attempt to open the thing was interrupted — the kettle, a phone call, a neighbor's drill, a courier who rang and delivered nothing — small random events queuing up politely, one per attempt, until I ruled that the envelope could stay where it was. It was consuming more time than it deserved. What V2 showed. Two specific, nameable weaknesses were left. First, the data gap in Figure 2: even strong specs listed fields per feature but never drew a coherent data model. Second, a quieter problem visible inside the pale T0 cells of Figure 3 — for low-context clients, "decision closure" still hovered just above the gate (6.5 against a floor of 6.0), because a vague client never raises money, platform, or notifications, and the interviewer didn't raise them either. Those choices simply went missing. What I asked Merlin to change for V3. Two targeted additions. (1) A mandatory data & integration pass: every spec must include entities and relationships, external integrations named by purpose (with what's in scope now vs later), a data lifecycle, and who owns each piece of state. (2) An unasked-decisions pass: before finishing, the interviewer must settle monetization, platform, notification channel, and scale — surfacing them itself when the client didn't, and defaulting them with a one-line reason rather than leaving them blank. What V3 delivered. Both gaps closed, and — the part I watched most carefully — nothing else regressed. The data trait went 6.8→8.5. Decision closure for the bottom tier went 6.9→8.9. Verifiability, fidelity, and clarity held at their already-high V2 levels. Figure 4 — The V3 grid: the floor caught up with the ceiling. Same color scale as Figure 3. The pale T0 column is gone — every cell now sits between 8.8 and 9.6, and the weakest cell in the entire grid scores higher than half the V2 grid did. Figure 5 — V3 raised the floor without moving the ceiling. Average score by client tier, V2 (left dot) to V3 (right dot). The least-experienced clients (T0) gained the most (+1.0); expert clients (T3), already near the top, barely moved. The four tiers, once spread across 1.2 points, now sit within 0.4 — quality became far less dependent on who is asking. 5. The whole arc, and a control test After the third run — before the scoring and the control — I decided the envelope should live in another room. The moment I picked it up, my mind began to misfire, and then folded down to the width of a single phrase I could not finish, in a language I do not know: as if I were assembling a word out of symbols I had never seen, without knowing the symbols or what the finished word should look like — and, worst of all, assembling them again and again in exactly the same order, getting the same error (incorrect word), unable to fix it, because each new cycle repeated the previous one perfectly; the mind had gone too narrow to hold even a note that the last attempt had failed. The courier's doorbell pulled me out of it. Since that day I do not touch the envelope. Stacked together, the three rounds tell a simple story: the biggest lever was never nicer prose. It was forcing the interviewer to produce the decisions and structure a beginner can't supply on their own. Round one taught it to commit (mechanics, metrics, "done"); round two taught it to model data and settle the business questions unprompted. Figure 6 — Three iterations as three surfaces. Height is the PRD score across the same 16-cell grid; the bottom surface (red) is the author's first instructions (V1), the top (green) is the latest (V3). Each version lifts the whole grid, and the steepest lift is at the T0 edge — the low-skill users. The surfaces also flatten with each round: the product depends less and less on client sophistication. Did we improve the interviewer, or just fit our four topics? This is where the Arthur–Merlin split earns its keep. If V3's gains were really an accommodation to pet care, elder care, creator tools, and home services, they'd collapse on unfamiliar ground. So Arthur picked four brand-new domains both Arthur and Merlin never saw — (E) immigration paperwork, (F) an AI-era job search, (G) an independent restaurant's back-of-house, and (H) travel-disruption recovery — and ran a control: one cell per tier, a diagonal through the grid. The four control participants: Cell Who What they want to build E-T0 H-1B worker, 31, M Sponsoring his wife's green card and terrified of missing a deadline; wants a calm personal organizer for forms, documents, and government response windows — explicitly not legal advice, never fills out forms. F-T1 Product manager, 37, F An application command center for laid-off mid-career job seekers. Arrived believing in "beat the ATS with keywords"; the interviewer talked her out of it and reframed the product around honest role-fit and follow-up consistency. G-T2 Owner-chef, 48, M 22 years in kitchens, 40 seats, razor margins. Wants live plate costing that re-costs every dish when an ingredient price jumps, par-driven prep lists, 86/waste logging, and supplier orders from par-vs-on-hand. "Usable by a chef mid-shift, not an accountant." H-T3 Ex-traveltech PM & former travel agent, 40, F Not another trip planner — a disruption re-planner: when a flight cancels, show the downstream "blast radius" (hotel nights, tours, connections now at risk) and 2–3 concrete recovery actions per broken piece. The control scores landed on top of the V3 baseline. Because Merlin wrote the instructions with no knowledge of these topics, there was no way to teach to them. Figure 7 — The control matches the baseline. Each new-domain control cell (dot) versus the V3 average for that tier (bar). The diagonal averaged 9.09 against a V3 baseline of 9.08 — indistinguishable. The T0 immigration case is the sharpest evidence: a stressed, non-technical user who never mentioned business model or platform still got both decided, plus a full data model, with the "not legal advice, never fills forms" boundary enforced unprompted. 6. What is learned, and what's next The takeaway. For a product-discovery AI, "quality" is less about writing and more about initiative under uncertainty: the willingness to decide, to model, and to fill the gaps a novice leaves. Two rounds of narrowly targeted instruction changes moved the weakest users up ~2.8 points and made output quality nearly independent of who is in the chair. And the two-role setup — one AI to write the instructions, a separate one to judge them blind — is what let me believe the number instead of hoping for it. Honest limits. One run per cell, so small differences sit inside normal conversation-to-conversation variance; the scoring is a careful rubric applied by a judge, not ground truth; and the very first round used a lighter capture method than the later two. None of that changes the direction or the size of the moves, but it's the reason I lean on the control rather than any single cell. Two directions we're taking next Human fatigue and onboarding cost. Our simulated clients never get tired; real ones do. The V3 "unasked-decisions" pass is powerful but it adds questions — exactly the wrong direction for a human who just wants to describe an idea and leave. The next study measures the interview as an effort budget (turns, length, time, drop-off) and looks for where the interviewer over-asks, so we can hit the same decision-closure with fewer, better questions. More than one judge. Arthur scores against a single buildability rubric. We're adding two pre-trained specialist evaluators — a System Architect that critiques each PRD for build feasibility, and a UX Designer that critiques it for usability — and feeding their objections back into Merlin's instructions. That turns a single-verifier loop into a small panel, and should catch failure modes a one-rubric judge can't see. The envelope is still on the edge of the desk. The golden wasp hasn't moved either, though I would not swear under oath that it is always facing the same direction. We have an understanding. Method note: 16 personas per round (4 client tiers × 4 domains), each taken through a full interview with the exact PRD captured and scored on the six-trait rubric above. V1–V3 are instruction versions of the same interviewer; the platform is otherwise unchanged. Figures use the round-level averages reported in the text.