Data Pipeline
X Profile Scanning Pipeline
A vetting workflow that ran for two days unattended and returned 20,509 X profiles with verification status, follower counts, and region data filled in.
The brief
What needed to be solved.
A Web3 marketing team had more than 20,000 X usernames and needed three things for each one: verification status, follower count, and region. Manual checking would have taken weeks.
The hard part was not opening one profile. It was keeping a long run alive, saving progress, and ending with a spreadsheet the team could use right away.
The constraint
What made it interesting.
I used Playwright instead of the paid X API because this was a one-time job and accuracy mattered more than speed.
The pipeline saved progress every 25 profiles, added random delays, retried failures, and skipped media downloads so the run could stay stable on a normal laptop.
The build
What was assembled.
Checkpoint files so the run could restart without losing progress.
Random delays and timed pauses to reduce rate-limit risk.
Retry logic for broken pages and temporary failures.
Spreadsheet output with filtered views for campaign use.
The result
What changed after it ran.
20,509 profiles scanned in one unattended run.
395 verified accounts found and 10,192 dead accounts flagged.
The client saved outreach time because the scan cleaned the list while enriching it.
Stack
Read next
Automation System
Price Update
An n8n workflow that tracked price changes, calculated the deltas, and sent a clean report on schedule. Replaced manual weekly checking with one repeatable reporting flow.
Read case study →Monitoring Pipeline
Twitter to Telegram
A monitoring pipeline that watched selected X accounts and pushed the important updates into Telegram. Moved signal tracking from scattered feeds into one fast channel.
Read case study →