Why I ditched regex scrapers for an LLM parser (and when you shouldn't)

Last month I needed to scrape product details from 30 different e-commerce sites. Each site used its own HTML structure, class names changed weekly, and some were just plain inconsistent. I had two options: write a mountain of brittle CSS selectors or try something I’d been avoiding—letting an LLM figure out the extraction. Here’s what I learned the hard way, including the code that actually worked and the cases where I should have just stuck with BeautifulSoup. The problem that broke...