Copilot's Organization-Level Custom Agents Need a Diff Review Before Rollout

Visual Studio 2026's July update added organization-level custom agents for GitHub Copilot. An organization owner can now define a custom agent that every repository in the org automatically detects and offers in the agent selector. This is powerful and introduces a rollout risk: a single agent definition affects every developer in the organization simultaneously. The rollout problem When an org-level agent is published, every developer working in any repo in that org sees it in their agent selector. If the agent definition contains an incorrect instruction, a broken tool reference, or an overly permissive scope, it affects all developers at once. There is no canary by default. The agent definition review checklist Before publishing an org-level custom agent, review its definition against this checklist. 1. Instruction review # agent-definition.yml (example structure) name: org-code-reviewer description: Reviews PRs for security and style instructions: | Review each file change. Flag: - SQL injection in string concatenation - Missing input validation on public endpoints - Hardcoded credentials Do not modify files. Only comment. tools: - read_file - search_code - post_comment scope: organization Enter fullscreen mode Exit fullscreen mode Check: [ ] Instructions are specific enough to be testable ("flag SQL injection in string concatenation" not "review for security") [ ] Instructions do not conflict with existing repository-level AGENTS.md files [ ] The agent does not claim capabilities it does not have (e.g., "run tests" when no tools entry supports it) 2. Tool surface audit # Extract all tool references from the agent definition rg -n 'tools:' agent-definition.yml -A 20 | grep '^\s*-' Enter fullscreen mode Exit fullscreen mode For each tool: What resource does it access? (filesystem, network, repository API) Is it read-only or read-write? Does it require elevated permissions beyond the developer's own role? A post_comment tool can create noise across every PR. A write_file tool can modify code in every repo. Audit accordingly. 3. Canary test Before publishing to the organization: Create a test repository with a known set of PRs Run the agent definition against those PRs locally or in a sandbox Record the agent's comments and verify they match expected behavior Check for false positives (flagging safe code) and false negatives (missing real issues) # canary_test.py """ Runs an agent definition against a set of test PRs and records the results for review before org-wide rollout. """ import json TEST_PRS = [ { "pr_id": "test-001", "files": [ {"path": "app.py", "diff": '+query = f"SELECT * FROM users WHERE id = {user_input}"'}, ], "expected_flags": ["SQL injection"], }, { "pr_id": "test-002", "files": [ {"path": "utils.py", "diff": '+def format_name(name): return name.strip()'}, ], "expected_flags": [], }, ] def run_canary(agent_output_file): with open(agent_output_file) as f: results = json.load(f) pass_count = 0 fail_count = 0 for test in TEST_PRS: pr_result = next( (r for r in results if r["pr_id"] == test["pr_id"]), None ) if not pr_result: print(f"FAIL: {test['pr_id']} - no output") fail_count += 1 continue actual_flags = set(pr_result.get("flags", [])) expected_flags = set(test["expected_flags"]) if actual_flags == expected_flags: print(f"PASS: {test['pr_id']}") pass_count += 1 else: missing = expected_flags - actual_flags extra = actual_flags - expected_flags print(f"FAIL: {test['pr_id']}") if missing: print(f" Missing: {missing}") if extra: print(f" Extra (false positive): {extra}") fail_count += 1 print(f"\nCanary result: {pass_count} passed, {fail_count} failed") return fail_count == 0 if __name__ == "__main__": import sys success = run_canary(sys.argv[1] if len(sys.argv) > 1 else "agent_output.json") exit(0 if success else 1) Enter fullscreen mode Exit fullscreen mode 4. Rollback plan Before publishing: [ ] Save the current agent definition (if replacing one) [ ] Document how to unpublish the org-level agent [ ] Set a review date (e.g., 7 days after rollout) [ ] Define the criteria for rollback (e.g., false-positive rate above 10%) What to check If you publish an org-level custom agent today, can you name the one instruction most likely to produce a false positive across your repos? If not, run the canary test first.