# agents.md: Teaching AI Agents How to Scrape (The Future of Web Automation)

You're building a Scrapy scraper. You ask Claude Code for help. "Add pagination to my spider." Claude reads your code. It sees scraper.py, settings.py, the folder structure. Then it makes suggestions. But the suggestions don't match your style. You use specific naming conventions. Your error handling is particular. Your Scrapy middleware is custom. Claude doesn't know any of this. So it writes code that technically works but doesn't fit. You spend 30 minutes rewriting it to match your pattern...