Robots.txt Compliance Audit Pipeline
An async Python pipeline that fetched and classified robots.txt files across roughly 967,000 domains to power a large-scale compliance dataset.
- Role
- Data Pipeline Engineer
- Project type
- Research Data Engineering
- Duration
- 2026
- Tools
- Python, asyncio, aiohttp, Pandas, Plotly, Matplotlib
01
Executive Summary
Built the data pipeline (robots_audit/) behind a large-scale study of robots.txt compliance: asynchronously fetching robots.txt files from roughly 967,000 domains, cleaning a 1.56-million-row URL classification dataset, and producing the visualizations used to analyze compliance gaps by country.
02
The Problem
Studying robots.txt compliance at scale means fetching and parsing files from close to a million live domains without the pipeline stalling on slow or unresponsive servers, then reconciling messy, inconsistently formatted metadata, like country names, across more than a million rows.
03
Users & Stakeholders
- Thesis research (feeds the REP efficacy analysis)
- Course examiner
- Future users of the dataset
04
Discovery and Research
- Benchmarked synchronous vs. async fetching for hundreds of thousands of domains
- Identified inconsistent country-name formats across the source data
- Mapped which fields the downstream classification analysis actually needed
05
Goals
- Fetch robots.txt from ~967K domains without the pipeline stalling
- Clean and standardize a 1.56M-row dataset for downstream analysis
- Produce compliance-gap visualizations by country
- Keep the pipeline resumable and rate-limit friendly
06
Solution
Used aiohttp with asyncio to fetch robots.txt files concurrently across hundreds of thousands of domains, with timeouts and error handling for unresponsive hosts. Cleaned and standardized the resulting dataset, including reconciling country-name variants, and used Plotly and Matplotlib to generate choropleth maps and bar charts showing compliance gaps.
V
Visual analysis (D3)
Every figure below is generated with D3 from this case study’s own data and is annotated using Tamara Munzner’s what / why / how framework: the data abstraction, the abstract task it supports, and the visual idiom with its marks and channels.
V1Risk exposure by impact
- High
- Medium
- Low
What — dataItems are identified project risks; one ordered attribute (impact) and one categorical attribute (mitigation, on hover).
Why — taskRank the risks and identify the extremes that need mitigation first.
How — idiomDot plot: point marks, position on a common ordered scale as the primary channel, colour hue as a redundant encoding.
V3Implementation sequence
What — dataItems are roadmap phases with an ordered key (sequence) and a quantitative attribute (workstreams per phase).
Why — taskSummarise the delivery order and see where the workload concentrates.
How — idiomGantt-style ranged bars: line marks with length encoding magnitude, aligned on a shared horizontal scale, colour separating phases.
V4Solution architecture as a network
What — dataA node-link network: nodes are architecture components, links are data flow, and layer is a categorical attribute.
Why — taskExplore topology — trace a path from data source to governed output and locate dependencies.
How — idiomLayered node-link layout: rectangle marks positioned by layer, connection marks for flow, greyscale luminance separating adjacent layers.
V6Engagement depth profile: radar
- Documented items
What — dataOne item (this project) with seven quantitative attributes: the number of documented artefacts per workstream.
Why — taskSummarise the shape of the engagement and compare workstreams which parts are deep and which are thin.
How — idiomRadar/star plot: line and point marks, angle channel for the attribute key, radial distance for magnitude.
V7Roadmap workload circular barplot
- Workstream items
What — dataItems are roadmap phases with one ordered key (sequence) and one quantitative attribute (workstream items).
Why — taskCompare how much work each phase carries and see the cycle of delivery as a whole.
How — idiomCircular barplot: arc marks, angle for the ordered phase key, radial length for magnitude.
V8Architecture composition: treemap
- Component
- Layer
What — dataA hierarchy: architecture layers containing components, each component counting as one unit.
Why — taskSee where the system's weight sits: which layers hold the most moving parts.
How — idiomTreemap: containment for the hierarchy, area marks sized by component count, nested rectangles for layers.
07
System Architecture
Domain list (~967K)
|
v
+---------------------+ +--------------------------+
| Async fetcher | ---> | robots.txt responses |
| aiohttp + asyncio | | (or timeout / error) |
+---------------------+ +--------------------------+
|
v
+---------------------+
| Cleaning & merge |
| 1.56M-row dataset |
| country standardization |
+---------------------+
|
v
+---------------------+
| Visualization |
| Plotly choropleths |
| Matplotlib bar charts |
+---------------------+Fetching
- aiohttp async client
- asyncio task pool
- Timeout / error handling
Data cleaning
- Pandas transforms
- Country-name standardization
- Row-level classification
Visualization
- Plotly choropleth maps
- Matplotlib bar charts
- Compliance-gap breakdowns
R1
Technical Risks
| Risk | Impact | Mitigation |
|---|---|---|
| Fetching hundreds of thousands of domains can look like abusive traffic | IP blocks or rate-limiting mid-run | Concurrency limits and per-host timeouts to keep request patterns polite |
| Inconsistent or missing robots.txt formatting | Misclassified compliance status | Defensive parsing with explicit handling for malformed or empty responses |
| Country-name variants causing incorrect aggregation | Skewed choropleth results | Built a standardization step before any country-level aggregation |
R2
Implementation Roadmap
Weeks 1–2
Phase 1: Fetch pipeline
- Async fetcher
- Error / timeout handling
- Resumable run state
Weeks 3–4
Phase 2: Data cleaning
- Country-name standardization
- Row-level classification
- Dataset validation
Week 5
Phase 3: Visualization
- Choropleth maps
- Compliance-gap bar charts
- Handoff to thesis analysis
R4
Success Metrics
Scale
- ~967K unique domains fetched
- 1.56M rows cleaned and standardized
Output
- Choropleth maps by country
- Compliance-gap bar charts
Reliability
- Resumable pipeline
- Consistent handling of timeouts and errors
R5
Deliverables
- Async fetcher covering ~967K unique domains
- Cleaned and standardized 1.56M-row URL classification dataset
- Country-name standardization across the dataset
- Choropleth maps and bar charts of compliance gaps (Plotly / Matplotlib)
“At that scale, the pipeline's biggest risk was never the parsing logic, it was staying polite enough not to get blocked before the data was even collected.”
Next project
REP Efficacy Research (BSc Thesis)