How to Automate Failed Test Triage With Allure TestOps and AI

Hi, I'm Oleg. In the previous article I covered how we generate autotests from Swagger and test cases using OpenAPI Generator plus Cursor AI / Claude Code, and how we measure coverage automatically with Swagger Coverage.This one is about the other end of the pipeline: how we triage failed autotests using Allure TestOps integrated with an issue tracker, an MCP server for TestOps, an MCP server for the tracker, and Cursor AI / Claude Code. But I won't start with AI. First the process itself: why we need defects in TestOps, how we triage a run by hand, why it quickly becomes drudgery without matcher rules, and what exactly we automated afterwards. Two notes for readers outside our stack: Allure TestOps is a test management system that ties manual test cases to automated tests, CI runs, and reporting. Yandex Tracker is an issue tracker, roughly analogous to Jira; TestOps integrates with it the same way it integrates with Jira, so if you're on a different tracker, the shape of this stays the same. The problem: the tests failed — now what? When autotests fail, the mere fact of a red run tells you almost nothing. You need to work out what actually happened: a real bug in the product; the autotest itself broke; the test data went stale; the environment or infrastructure went down; the contract changed but Swagger or the tests weren't updated; or it's a known problem that already has a ticket in progress. If all you ever look at is the list of failed/broken tests, the team quickly starts spending time re-triaging the same failures. Today a QA engineer works out that three tests fail because of one backend error; tomorrow another QA engineer opens the same three tests and walks the same path from scratch. So after a test run we have one simple rule: every failed result must be triaged and linked to a defect. Ideally, after triage the run has no unresolved failed/broken results left. Why defects in TestOps A defect in TestOps isn't just another entity sitting next to a test case and a run. It's how you turn a chaotic list of failures into a usable database of causes. One defect should describe one cause of failure. Not "404", "500" or "AssertionError", but a specific problem — for example, the image-fetching endpoint started returning 404 for a group of scenarios where the contract says 200. That buys several practical things: Fewer duplicates. Before creating a new defect, QA checks whether an open defect with the same cause already exists. Faster repeat runs. If the same error recurs in the next run, TestOps can link the failure to the existing defect automatically. Clearer release risk. Before a release you can see which critical defects are still open and which failures are already known and not blocking. A link to development. When the cause is a product bug, the defect is tied to a tracker issue. Easier to separate concerns. Product bugs don't get mixed up with test data, environment, or authorization problems, or with bugs in the autotests themselves. The main point: defects stop you re-triaging the same failure after every single run. What the manual process looks like The triage procedure is fairly straightforward. 1. Open the run and look at the unresolved results After the autotests finish, go to the run results in TestOps and open the overview. Unresolved results in a run mean either that no defects have been filed for the failed tests, or that defects exist but specific test results aren't linked to them. One important constraint: you can only create a defect and link it to a failed test in a run that isn't closed yet. So triage the failures before closing the launch. 2. Analyse the error in the failed test Next, open the failed test and look at the reason. For example, the test might fail with: Expected status code but was . The error text alone usually isn't enough. You need to expand the stack trace and find the line that belongs to your autotest — the class and method where the failure happened. For example: IntegrationsRegressTests.getIntegrationsProductsAllImagesWithLimitTest Now we have two important pieces: the error text: Expected status code but was . the failure location in the test code: IntegrationsRegressTests.getIntegrationsProductsAllImagesWithLimitTest After that you need to determine the cause. Not every failed autotest equals a product bug. Sometimes the problem is the host, a token, test data, setup, cleanup, the Swagger contract, or local deserialization of the response into a generated model. 3. Create a new defect or reuse an existing one If a suitable open defect already exists, link the failed test to it. But don't stop at a plain manual link — linking the current test result to a defect only works for the current run. For TestOps to link similar failures itself in later runs, the defect needs an automation rule. If this cause hasn't come up before, create a new defect. The title doesn't need to be elaborate at this first step, but it should convey the essence of the error. The description can then be fleshed out so a developer understands it: which endpoint or scenario is failing; what request was made; what response was expected; what response came back; why this looks like a product bug rather than a test problem. 4. Create or link a tracker issue When a defect corresponds to a product bug, it should have a tracker issue attached. You can do this straight from TestOps: either link the defect to an existing issue, or create a new one through the bug tracker integration. This part matters: the issue should be created through the TestOps integration, so the link between run, defect, and tracker issue stays transparent. 5. Configure the automation rule The automation rule is what makes defects actually save time. In the simple case, you put the error text in the matcher: Expected status code but was . But that pattern can be far too broad. A 404 can show up in different places for different reasons. So it's better to constrain the rule with the stack trace as well: IntegrationsRegressTests.getIntegrationsProductsAllImages* A rule like that tells TestOps: if a new run hits "expected 200, got 404" and it happened in the group of tests about fetching images, link that failure to this defect automatically. This doesn't just catch any 404. It catches one specific recurring cause. The matcher doesn't always need to go down to a specific test method. If the problem applies to a whole test class, constraining by the class in the stack trace is enough. And if the error is unique and specific enough on its own, a matcher on just the error message sometimes suffices. 6. Verify the result After creating a defect, check that: the defect is linked to the right failed test results; the defect has an automation rule; if it's a product bug, a tracker issue is created or linked; the run overview shows fewer unresolved results; the matcher isn't so broad that it will hide new, independent errors. On the next run, if these same tests fail — or new tests fail with the same cause — TestOps links them to the defect automatically. And when the tracker issue is closed, the defect in TestOps can close automatically through the integration too. The rules we settled on To keep the defect database from turning into a landfill, we stick to a few rules. One defect — one cause of failure. Before creating a new defect, check whether a suitable open one already exists. Don't merge a product bug, an infrastructure problem, a test data problem, and an autotest bug into a single defect. Every new defect gets an automation rule. The matcher must not be too broad: a bare 404, 500, or AssertionError with no stack trace constraint is almost always a bad idea. Only close a defect once the fix is confirmed and a fresh run has no failures from the same cause. Classifying failures matters separately. If the cause is the autotest, the test data, authorization, the environment, or a Swagger mismatch with no backend bug behind it — don't file a product defect in the tracker. Otherwise the development team gets noise, and trust in the autotests drops. Where AI comes in The manual process works, but it still takes time. Open the run, get all failed/broken results, look at the details, group similar errors, check existing defects, create a matcher, link test results, create or verify a tracker issue, and make sure every field is filled in correctly. So to speed triage up, we wired in AI plus MCP. I've written about our MCP server for TestOps before. In short, MCP lets Cursor / Claude Code reach into TestOps, fetch the entities it needs, and write results back. For this task we use the TestOps MCP together with the tracker's MCP. On top of that I wrote a skill, /testops-run-defects. It takes three parameters: /testops-run-defects For example: /testops-run-defects https://allure.example.com/launch/67737 red loylt-gifts The agent then does roughly what a QA engineer following the procedure would do, only faster and more systematically: extracts the launch id from the link; fetches the unresolved failed/broken test results from TestOps; requests failure details, pulling the stack trace only where it's needed to build a sane matcher; groups failures by shared cause; checks open defects and their matcher rules so it doesn't create duplicates; separates product bugs from test, data, environment, authorization, and Swagger-mismatch problems; creates defects in TestOps with matcher rules; creates a linked tracker issue through the TestOps integration; explicitly links every failed test result to its defect; via the tracker's MCP, verifies and fills in the issue fields: team, component, type, priority, and tag; finishes with a control pass: matchers exist, test results are linked, tracker issues are created, no unresolved results remain. Importantly, the AI must not just file defects for everything red. The skill has guardrails against false defects: if the cause looks like a test data problem, a wrong host, bad authorization, a broken setup, or a bug in the autotest, that result must not become a backend bug. What it looks like in practice Below is an example run of the skill. We passed the launch link, the team red, and the component loylt-gifts. The agent fetched the list of failures, triaged them, created defects, linked them to tracker issues, and verified the result. The output is a short report: how many results were processed, which defects were created, which tracker issues were linked, and whether any unresolved failures remain. In this example the result was: failed/broken unresolved processed: 29 defects created: 1279–1287 issues created and linked: CMA-9551 – CMA-9559 unresolved results remaining: 0 every defect has a matcher all 29 test results are linked to their defects the tracker issues got the right fields: team, component, type, priority, and the autoapi tag What's next Right now I'm actively extending the skill so it fits different teams and different components, rather than one hard-coded mapping. The next logical step is an AI agent that walks the failed autotests of every team, triages the results, files defects and issues, configures matcher rules, and leaves only the ambiguous cases to a human. The ideal picture for me looks like this: the autotests fail, the agent triages the run on its own, groups the causes, separates product bugs from test and data problems, creates the necessary defects and tracker issues — and in the morning QA sees not a heap of red tests, but a clear list of causes and actions. You can't remove the human from this process entirely. But removing most of the drudgery is well within reach. Short version Defects in TestOps aren't a formality. They make the failure database manageable, keep duplicates down, and stop you re-triaging the same errors after every run. Matcher rules turn a defect from a manual annotation into a mechanism for automatically linking recurring failures. Tracker integration connects the autotests to development. And MCP plus Cursor AI / Claude Code automate most of the drudgery around failed results. In practice that means QA spends less time shuttling information between systems and more time on the thing that matters: working out where the product is genuinely broken, and where the tests, the data, or the infrastructure need fixing instead. One caveat worth flagging, since it came up when I published this in Russian: the "one defect — one cause" rule runs into the granularity of what the test actually reports. A single assertion covering ten malformed fields will only ever be split as finely as the report shows it. A soft assertion emitting a message per field can be decomposed properly. Worth knowing before you assume the matcher will do all the work for you. Let’s Work Together I’m open to QA leadership opportunities and collaborations with teams looking to improve their testing strategy, build scalable test automation, introduce AI-assisted QA workflows, and establish meaningful quality metrics. If this sounds relevant to your team, feel free to connect with me on LinkedIn.