I Gave Up on CSS Selectors: Using LLMs for Web Scraping

A few months ago, I was building a small side project that needed to compare prices across a dozen e-commerce sites. Simple enough, right? I've written scrapers before. BeautifulSoup, a few clever selectors, maybe a regex or two – that's all you need. Except it wasn't. Every site had its own HTML structure. Some loaded content via JavaScript. Others deliberately messed up class names. I spent more time updating selectors than writing actual logic. I was maintaining a fragile tower of CSS select...