Hey everyone,

I have been scraping a lot of data from a website for the last several months. I have been using 3 phones with unlimited data plans that average around 30 gigs a day of usage per phone.

Recently I just banned from my cell provider due to ToS violation due to automation. Does anyone know of cell providers or somehow I can use this much data without being banned?

A note is that this website will block me after so many requests from a single IP, hence the reason for cell phone plans as I can rotate my IP frequently to avoid being blocked.

I am unable to use VPNs as well as it knows these IPs and will force a log in page.

Any ideas? Maybe starlink lets me rotate IPs?

Or maybe I need to make the phones look more legit to my cell provider?

Hi all, I am building a scraper to scrape articles from a website for research. I have their permission to do so, but they have cloudflare in place, and I keep getting 525 errors (in my browser the site works just fine). Currently I am just using python requests, I was wondering if anyone has any tips on how to mitigate this. Would using selenium and navigating in a browser selenium help? Thanks.

So I shared this about a month ago and got some useful feedback on it from the community here and a few other places. Made some improvements over the last month so it's now able to handle even more different scenarios faster and better.

Basically, just provide a description of the data you want to scrape and where it is, and it will use your Claude or ChatGPT subscription to build a step-by-step automation that opens a browser window, logs in (if needed) and get you the data you need. It can then save to wherever you want too - in this example video it's a Google Sheet, but it can be a local file on your computer or even another third party tool, as long as you can provide an access key.

Have used it extensively on among other places LinkedIn and Reddit without running into issues, but would be interested to hear how others are doing.

It's open source and 100% free, and requires no registration, login or even an email from you, just download the files and get started. All you need is a Claude or ChatGPT subscription. Here's the website if you're interested: https://cryogram.app

Whenever you right-click -> "Copy as cURL" in Chrome, it grabs everything. Your active Authorization: Bearer tokens, your session cookies, your internal headers.

If you paste that into one of the top Google results to convert it to Fetch or Python, you just handed live production credentials to whatever backend is parsing that string.

I finally got paranoid enough about this to write a purely client-side transpiler for myself. It’s just a Next.js static export dumped on Cloudflare Pages. Zero Node backend, no API routes. I wrote a custom AST parser in TypeScript that tokenizes the bash string and generates the code directly in the browser memory, so the tokens never leave the machine.

Also threw in a JWT decoder and a Regex tester running in a Web Worker (with a 2000ms watchdog so catastrophic backtracking doesn't kill the tab).

I need help breaking the bash tokenization. I am trying to bulletproof the parser this weekend. If anyone works with insane multi-line cURL requests, multipart/form-data, or weird escaping rules, can you throw it at the parser and see where it chokes?

Not dropping a bare link here because automods hate new domains, but if you want to try and break it, search SyntaxFiddle on Google or drop a comment and I'll send it to you.

I currently have a group of python scripts which i made to webscrape jobs directly off their website. Without monthly maintenance , theyre constantly breaking and thought i could use Openweb UI and LLM to do the same function.

Its only for my personal use , so what would be the easiest way to achieve this ?

What i want to do is build a weekly database of jobs , and then have AI use my experience and resume to shortlist roles to apply for

Hey folks, I'm pretty new to the scraping scene. I'm building a product which requires scraping social media posts and comments. I've tried some tools but they either are not reliable or are too expensive.

I'd appreciate any suggestions or leads for tools/open-source projects or anything that can help me scrape multiple social media platforms

I am scraping a site for listings. It has around 200 pages.

Somewhere along page 10, i get hit with a basic custom made captcha.

Its quite simple: 5 characters made up of text and digits.

Im rubbing this scraping job twice daily on a cloud environment so no human intervention is possible in my case.

I use playwright to scrape the site.

2 questions: -

1. How do I automate the captcha solving process so that every job goes on without errors?
2. Is playwright the correct approach or should I be using some other tool?

Edit:

I used the Camoufox library to avoid the captcha but that didn’t work. The script still encounters a captcha test even with added noise and human-like pauses in the script.

My preferred solution is a way to either bypass the captcha test or have it solved automatically by the script.

2nd option: The script should pause and launch a browser when it encounters a captcha or an error, expecting human intervention before continuing.

Your help is appreciated.

Good Afternoon Everyone,

This is not promotion, this is a genuine post, and feel free to ignore if this does not concern your web scraping activities.

Some of you might remember adultdatalink from the pornstar stock market index post I made here a while back. It had scrapers that turned 20+ sites into clean JSON, performer and keyword analytics, scraping projects, and some video tools.

After about two years I'm shutting the hosted service down.

Instead of letting the code rot on a hard drive, I'm making it open source under MIT: https://github.com/adultdatalink/adultdatalink-public

First, I owe some people an apology. The API went down at the end of April and I never said anything.

Some of you noticed, and there's even a thread on webdev asking whether it was dead.

I stepped away from it without a proper shutdown notice, and anyone who had built something on top of it deserved better. This post is the notice I should have written back then. Since the code is now public, anyone who depended on it can run it themselves.

That thread also said the user numbers I posted before were probably fake, and that most people just signed up and connected without really using it. That's not correct. 1,635 people signed up, and 1,008 of them generated API keys. Signups started slow, with 17 in March 2025, peaked at 193 in December 2025, and stayed somewhere between 100 and 190 a month after that.

Our own request logs captured about 2.45 million API calls. Almost 2 million of those came between December and April alone, from 303 different API keys, so roughly 430K requests a month. The busiest single day was 25,935 requests. 97.8% of requests came back successful, with a median response time of about a second, which I'm pretty happy with considering every call was scraping a live third party site.

On the Cloudflare side, the peak was 3.9 million requests and 36 GB in one 30 day window back in March 2025, from almost 100 countries. A month and a half before that, the same window showed 195K requests. When I added the video tools, bandwidth went up to 407 GB in a single month, and Brazil, Germany, Canada and Spain suddenly became some of the biggest users. Server errors stayed around 0.05% the whole time. Pornhub was by far the most requested source, followed by Eporner, XVideos, xHamster and XNXX.

So why shut it down?

It just didn't make much money & sunk cost fallacy.

I spent so much time building this that I didn't want to just walk away from it fully, but there comes a where you cut losses on bad investment.

The usage was real but very concentrated. In the latest logs, one integration made about 80% of all requests, and the top 10 keys made 94%. A lot of people tried it, and a handful actually built on it. In this niche, payment processors, ad networks and app stores all make things harder than they need to be, and the scrapers need constant fixing every time a site changes something. Between hosting and my time, there wasn't much left over. I could keep it going as a small side thing, but I'd rather put the code out there and let people build stuff with it than keep it to myself.

The other thing I want to talk about is how much AI has changed this.

I built and ran the whole thing basically by myself. That was twelve FastAPI services covering the scraping API, an analytics engine, accounts, payments, logging, a CDN, storage and an ops dashboard, plus the scrapers, a YOLO annotation pipeline, and the stock style indexes.

A few years ago that would have taken a small team. AI coding tools have made the barrier to entry for SWE alot lower. Building it is not really the hard part anymore, finding people who will pay for it, getting it in front of them, and keeping it maintained is the hard part, and that's where this one came up short.

And also its really not hard for someone to just copy your shit, so software moats are very difficult to maintain IMO nowadays unless paired with hardware.

THE REPO

The repo has 2 pieces. The first is the adult content API & analytics engine.

Everything is configured with environment variable and it runs with pip and uvicorn.

It's archived and I won't be maintaining it, so fork it and make it your own. Some scrapers will break as the sites change, that's just how it goes.

Please read the README part about terms of service and not exposing the proxy routes to the public internet. And obviously it's 18+ only.

Huge thank you to everyone who signed up, built something on the API, sent feedback, reported bugs. I learned a ton doing this and had a lot of fun with it.

Hi, so i'm beginner in scraping, but I need to do this for my (first) job and I need advice. So, you see the kind of website with a search page and each search result is a card with a link to its own page. Obviously each website is structured differently. So, my boss wants me to write a scraper to fill a database using info from those type of lists. So, identify where the list is in the page structure, identify list items, identify the next page pagination button. But for a lot, a lot, a lot of websites with lists.

So i've tried making some heuristics to identify list items and use Scrapling's find_similar() to validate they're part of the same list. And well, sometimes it works but mostly it doesn't and I don't see how I could make something that works everywhere with little to no human input.

Soooo, if you've already done smth like that, or at a smaller scale, what's the best approach ? Should I just continue throwing better heuristics at it? Or are my boss's expectations unrealistic at this scale ?

(Before someone tells me to ask chatgpt, i did and it didn't help)

Thank you

I have just released the Scrapling v0.5 update, and I had to record this!

Here’s Codex using Scrapling MCP to quickly solve hCaptcha, reCAPTCHA, Friendly Captcha, Cloudflare Turnstile, and AWS CAPTCHA 🔥

Watch the video. This is what the new update makes possible. Feel free to cancel your paid subscriptions now; the new Scrapling update enables browser automation for any AI Agent like never before. It pairs perfectly with OpenClaw and Hermes!

It can open a page, inspect it, click, type, run JavaScript, take screenshots, and read network requests and responses. It can perform all browser actions, chain them in one call, and make it look as human as possible while saving tokens.

It can access closed Shadow DOM elements without issues and interact with the website using selectors, references, and coordinates. It can now also save browser state and reload it later so you don't have to repeat anything!

This update also makes it easier than ever to crawl websites automatically from code and convert them to safe Markdown, with filters, crawling strategies (DFS/BFS), and more.

The Cloudflare solver is now faster and more efficient than ever, whenever you want to solve it without AI.

This release includes many changes, fixes, and quality-of-life improvements. The website is now translated into 10 languages!

Check it out here and let me know your thoughts: https://scrapling.readthedocs.io/en/latest/index.html

Check out the video on YouTube: https://www.youtube.com/watch?v=NHwWEiTOoPQ

Full release notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.5

Full migration guide on moving to v0.5 translated to 10 languges: https://scrapling.readthedocs.io/en/latest/migrating-to-v0.5.html

I rewrote Cheerio in Rust as a drop in replacement.

I built Rusteerio to make HTML extraction faster and more memory-efficient in our scraping infrastructure. It’s 4.5× faster than Cheerio in our benchmarks, uses 4× less memory at 16 CPUs, and passes all 777 Cheerio tests.

It’s open source: https://github.com/context-dot-dev/rusteerio

More on why we built it: https://context.dev/blog/introducing-rusteerio

I have a pipeline, which collects factual attributes of listings, tracks price changes, and estimates how long an offer remains online. I don't store personal information, seller-written descriptions, photographs, seller contact details, or other listing content.

The resulting statistics and market insights are aggregated and combined with data from other sources, including paid APIs. The aggregated insights are available in my product for free. There is also a paid subscription plan that includes additional features unrelated to this scraped data.

I have a consultation with a lawyer next week, but before that, I'd like to hear from people who have built similar products or dealt with similar legal questions in the EU because I have read various discussions and legal resources but I am still unclear about how EU database rights apply to this particular model. Expecially could regularly collecting data from most of a marketplace's listings conserns its database rights even if I retain only factual attributes and publish only aggregated analytics?

Hi all,

I’m a filmmaker / photographer based in Paris.
Since few months, my portfolio (Squarespace website) has been getting suspicious recurring visits. 

Pattern :

. It visits my four main pages (Home, Photography, Film, Info), spends a few seconds on each, then leaves.
It never visits an unlinked page that Google has indexed

. The time per page and the page order vary slightly from visit to visit.

. Every visit reports the same machine setup : Mac 10.15.7, Chrome version 142, and the same technical ID (“AppleWebKit/537.36 … Chrome/142.0.0.0 Safari/537.36”).

. The IP address and country change every time (Netherlands, Germany, Austria…).
Squarespace labels the source “Tor exit node”, meaning a network that masks where a visitor connects from.

. It comes every few days, at irregular hours, and never from a link, a search or social media.

. The last visit stayed 18 seconds on my Film page, much longer than usual. My videos are hosted elsewhere, and the logs show none was played.

I already turned on Squarespace’s AI-crawler filter, but the visits continue.

I’m attaching the screenshot of 1 full visit..

Does this look like an automated tool, for example an AI company scraping my work?
Or could it be a person or gallery checking in on my work? Is it worth worrying about, and is there anything I can do to protect my work?

Thanks!

Divergence: formatting a script can turn it into a different program, across 37+ JavaScript processors

hello everyone, each time you got new site to crawl you need to go through numerous gates each one is giving you a score , i made this tool to detect what's blocking me , which providers, which layers in the clipboard of the score and most importantly which direction to choose and what tools/methods to use.

It's Open Source

npm i -g whichwaf

Or from source:

git clone https://github.com/Oussama-Rahmouni/whichwaf.git
cd whichwaf
npm install
npm run build
npm link   # puts `whichwaf` on your PATH

Or run without linking:

npm run whichwaf -- https://example.com

github

For businesses working with e-commerce product data, how do you ensure scraped prices, product details, and availability are accurate?

Is manual checking still a major part of the process, especially when dealing with thousands of products across different websites?

I'm interested in understanding how businesses handle this today and whether reducing manual verification is a real pain point.

Curious to hear from people working in product data, web scraping, or data operations.

Hello everyone, I'm creating a social helping app for undergraduate students and investigators.

So as a core component I need to be able to make a good parser which should be deployed constantly for as an example - to scan for scholarships available to claim. So I am really looking for architecture / frameworks ideas if anyone could share, just not to make another ai slop app :)

hey everyone,

whenever an AI agent or scraper hits a protected page, it usually gets dumped with a massive 50KB wall of challenge HTML. If you're using AI coding agents (Claude Code, Cursor, etc.),

the agent often wastes thousands of context tokens trying to "read" or parse what is actually an obfuscated block page.

To solve this, I built knockknock a fast CLI tool that tells you exactly who's guarding the front door in milliseconds.

What it does:

- Detects 39 antibot, WAF, and CAPTCHA providers (Cloudflare, DataDome, Kasada, Akamai, AWS WAF, etc.).

- Works directly with a URL or by piping raw responses (curl | knockknock).

- Clean machine-readable JSON output so your scripts and AI agents know immediately what they're up against.

- Includes a ready-to-use AI agent skill (npx skills add jeheskielSunloy77/knockknock) so your agents can check protection automatically without burning tokens on challenge pages.

It's a single lightweight binary with zero hassle to install.

check out the repo for installation and docs:

👉 https://github.com/jeheskielSunloy77/knockknock

would love to hear your feedback and PR and is always welcomed

EDIT: I fixed the scraper and it's once again a no-login scraper via a completely new scraping strategy. Reddrix (Open-source, blazing-fast Reddit scraper for posts and comments)

I had a smooth scraper for a website with high-demand data that I built back in September. The scraper was not open source back then. My style of scraping for that website (some may not prefer it, but I really like its simplicity) is to use a browser to load the home page, then make calls to JSON endpoints using JS fetch from within the same browser context. This achieved 2 things: preserved most of the scraping speed and avoided the fingerprint drift that occurs when grabing cookies with a browser and using them with another http client.

THE DECISION I REGRET:

This month, I decided to help people, so I decided to open-source my scraper. A few days before open-sourcing my scraper, I read a comment from someone here about a loophole in the target website that allowed one to get cookies without the browser at all. Knowing that this is a fragile loophole that may stop working any moment, I added this guy's idea as an optional route in my open-source scraper but did not remove my original setup. I allowed users to choose which route to use.

The result:

1. Some people reported my tool to the dev team of the target website.

2. A few days later the target website was updated to specifically break both setups (my style and this guy's style).

3. I ended up making a temporary fix that uses a disposable account to login and continue scraping.

4. When I made a post to ask for collaborators to help maintain this project, that guy made a comment where he claimed that I built a project with his code to gain GitHub stars without giving him credit.

5. Other people blamed me of using the GitHub project as a "bait" to funnel people into my "paid" tool when in fact neither the open source one nor the hosted one gives me a penny of revenue, and I was not planning to monetize either.

6. Another blamed me that I am gathering fake GitHub stars.

7. Another blamed me of lying and that my scraper is not really a "no login" one when in fact it IS a no-login scraper.

Seriously?

How can one share his scraping tools with others? Or should everyone just hide his scraping setups and never help? I seriously don't know if it's just better to be selfish and never help others in anything. I am also considering deleting this open source project for good and never open sourcing a scraper again.

I am just ranting...

Hello everyone! I’m a non-EU citizen, and I’m planning to move to Germany from my home country.

The problem is that the German Embassy opens appointment slots for vocational training visas only on Mondays, and the exact time when the slots become available and close is difficult to predict.

Is there any app or website that can track exactly when the appointment slots open and close? Ideally, I’d also like to view the history of previous openings and see the times when slots became available and were no longer available.

Any suggestions would be greatly appreciated. Thanks!

Hi all, I'm looking for someone to help me scrape the info mainly about literary agents across the world. And maybe in addition also publishers that promote fiction (so, not school books, science, etc). Not only English, but in top 10 languages too. If you could enrich the data with emails, that would be great, if not, I can manage probably. Let me know if interested. 👍

What’s the best way to scrap a subreddit every 20-30s without getting blocked by Reddit anti bot system.

I want to scrap this subreddit every 30s for new posts.