Diagnosing sentiment in my typo-reporting workflow

This is more of a comment than a question. A few months ago I added a “Report a Typo” feature to Six Colors. The goal was to give people a simple place to go — sixcolors.com/typo — if they found a mistake on the website. That Google form inserts the information into a Google Sheet, and a script attached to the Google Sheet kicks off a simple automation that sends me a notification in a couple different places. This works pretty well. I’m using a very simple Google Apps script to fire off a webhook when there’s a new form submission. The webhook is running on Make.com, which we use for a bunch of simple web automations, but that’s an implementation detail. Make.com posts an item in a private Discord channel and also crafts a custom push notification to my phone using Pushover. However, the moment I turned this feature on, I discovered the problem with asking people to report errors or typos: Some people see any open channel of communication as an open channel for any sort of communication. I immediately started getting more “this is more a comment than a question” items about articles than actual typo submissions. Commentary included personal preferences for writing style, web design, political references (real or imagined), and much more. I’m going to make an admission: This sort of feedback is maddening. I want to be alerted promptly about mistakes on the site so I can fix them promptly, but I am not interested in receiving push notifications regarding your opinion of the professionalism and appropriateness of my writing style. Unfortunately, it’s an open form. There’s nothing to be done. Or is there? This is a challenge that can’t really be solved by looking for a bunch of keywords. You need wholesale sentiment analysis of the entire submission. This is—brace yourself—actually a good use of a large language model. I solved this problem by using a plug-in within Make.com that lets me get a reply back from a cloud LLM that uses a simple model with an API cost of pennies per use, but you could also just use Shortcuts to build a version of this automation that runs on a Mac (Tahoe or Golden Gate) using Apple’s built-in models. It works just fine in Shortcuts on my Mac, via the Use Model action. (See screenshot above.) Here’s the prompt I’m currently using: Your task is binary classification. Input: a user-submitted comment about an article, claiming to report an error. Output: one token only: ALLOW or REJECT. Definition of ALLOW: A concrete, objective, verifiable issue in the article. This includes: * factual mistakes * spelling, grammar, formatting mistakes * broken or incorrect links * wrong names, dates, numbers * technical inaccuracies * omissions of material consequences of a feature or change (e.g., failing to mention that a feature deletes stored Wi-Fi passwords) Definition of REJECT: Anything subjective. This includes: * political disagreement * tone or style complaints * metaphor or comparison preferences * ‘you should have covered X more’ * speculation about motives * emotional reactions or rants * admissions that it’s not actually a mistake Classification rules: 1. If the comment identifies a specific factual, technical, or materially consequential issue in the article, return ACCEPT. 2. If the issue would meaningfully affect reader understanding or use of a product/feature, treat it as an omission and return ACCEPT. 3. If the comment is opinion, taste, politics, vibe, personal criticism, or preference, return REJECT. 4. If there is no concrete, verifiable claim about the article being wrong or incomplete, return REJECT. 5. If it’s off-topic or spam, return REJECT. 6. If it’s a short but simple phrase like “there not their” or “r/sheild/shield”, those are nerdier folks reporting typos and that’s fine, ACCEPT. Respond with exactly one of these two tokens: ALLOW or REJECT. No explanation. Submission: [[insert report text here]] This has worked surprisingly well. And, yes, all messages are logged in the original Google Sheet, so I can occasionally look at to monitor the quality of the filtering and make sure that the system isn’t failing to pass on actual mistakes. If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.