Pretty new to scraping and I’ve hit a wall with reCAPTCHA v2 on a site I’m trying to scrape.

How do you usually approach this? What do you check first to figure out whether something in your setup is triggering it, or whether it’s just a required step on the site?

I’d be interested in any open-source tools or approaches worth learning about. Also curious where you draw the line and decide a site is more trouble than it’s worth.

Any tips or stories about what worked—or what turned out to be a waste of time—would be appreciated. Thanks!

**Not a commercial product:** free and open source (MIT), no paid tier, no sign-up, no affiliate links. I'm the author and I'm sharing it for feedback.

It's a skill for Claude Code aimed at people who don't know selectors. You give it a URL and one sentence ("titles, prices and ratings from this category"), and it:

- opens the page in a real Chrome window and picks the cheapest route to the data: a hidden JSON API, data embedded in the page, responses the page itself receives, or the visible DOM

- shows 5 sample rows plus a screenshot with each value boxed, and asks once whether that's right

- collects at a polite pace and saves the method as a reusable recipe

The part I'd like this sub's opinion on is how it behaves when a site says no. It reads robots.txt and the terms first. If a free official API covers the request, it uses that instead. If the site forbids scraping or shows a bot check, it states the facts and stops for the user to decide. It never solves CAPTCHAs, stealth-patches the browser, rotates proxies, or reimplements request signing.

Things I've had to handle so far: infinite-scroll feeds that delete old posts from the DOM as you scroll, APIs that only answer a real browser, WebSocket/SSE live data, and cursor/GraphQL pagination.

Repo: https://github.com/scrapesmith/scrape-skill

Questions for you:

1. Is "stop and ask" the right default when robots.txt or the ToS forbids scraping, or is it too cautious?

2. Which sites do you think would break it first?