NA
Portfolio

A Defense Tier Classification of REP Efficacy Against Generative AI Web Crawlers

Bachelor's thesis classifying robots.txt defenses across 809 European news domains assessing how well the Robots Exclusion Protocol holds up against generative AI crawlers.

ResearchData AnalysisWeb Standards
Role
Thesis Author / Data Analyst
Project type
BSc Thesis (Course 2DV50E)
Duration
2026
Tools
Python, Matplotlib, Statistical Testing, RFC 9309
REP Efficacy Research (BSc Thesis) visual summary

01

Executive Summary

My bachelor's thesis classifies how well the Robots Exclusion Protocol (REP) actually defends against generative AI web crawlers, analyzing robots.txt configurations from 809 European news domains in depth and proposing a defense-tier framework describing how well-configured real-world robots.txt files are.

02

The Problem

Robots.txt is the main mechanism sites use to opt out of AI crawling, but it was unclear how consistently sites actually configure it correctly, and existing analysis tooling hadn't been validated against the current RFC 9309 specification.

03

Users & Stakeholders

  • Course examiner
  • Linnaeus University CS department
  • Future researchers reusing the dataset and tooling

04

Discovery and Research

  • Reviewed RFC 9309 to establish a ground-truth specification
  • Audited the existing Semantic Configuration Analyzer (SCA) codebase against it and found 4 bugs
  • Surveyed prior work on REP compliance and generative AI crawler behavior
  • Defined statistical tests appropriate for the compliance-score data

05

Goals

  • Build a defense-tier classification framework grounded in RFC 9309
  • Validate the SCA tooling before relying on its output
  • Answer RQ1–RQ3 with reproducible statistical analysis
  • Address the Enumeration Fallacy in how compliance is measured
  • Respond fully to examiner feedback on methodology and framing

06

Solution

Fixed 4 bugs in the SCA codebase to bring it in line with RFC 9309, then ran it across 809 European news domains for close analysis computing a compliance-score formula per domain and grouping results into defense tiers. Generated a full set of Matplotlib figures to answer each research question, and revised the compliance-score formula and framing in response to examiner feedback on statistical tests and validation limitations.

V

Visual analysis (D3)

Every figure below is generated with D3 from this case study’s own data and is annotated using Tamara Munzner’s what / why / how framework: the data abstraction, the abstract task it supports, and the visual idiom with its marks and channels.

V1Risk exposure by impact

  • High
  • Medium
  • Low

What — dataItems are identified project risks; one ordered attribute (impact) and one categorical attribute (mitigation, on hover).

Why — taskRank the risks and identify the extremes that need mitigation first.

How — idiomDot plot: point marks, position on a common ordered scale as the primary channel, colour hue as a redundant encoding.

V3Implementation sequence

What — dataItems are roadmap phases with an ordered key (sequence) and a quantitative attribute (workstreams per phase).

Why — taskSummarise the delivery order and see where the workload concentrates.

How — idiomGantt-style ranged bars: line marks with length encoding magnitude, aligned on a shared horizontal scale, colour separating phases.

V4Solution architecture as a network

What — dataA node-link network: nodes are architecture components, links are data flow, and layer is a categorical attribute.

Why — taskExplore topology — trace a path from data source to governed output and locate dependencies.

How — idiomLayered node-link layout: rectangle marks positioned by layer, connection marks for flow, greyscale luminance separating adjacent layers.

V6Engagement depth profile: radar

  • Documented items

What — dataOne item (this project) with seven quantitative attributes: the number of documented artefacts per workstream.

Why — taskSummarise the shape of the engagement and compare workstreams which parts are deep and which are thin.

How — idiomRadar/star plot: line and point marks, angle channel for the attribute key, radial distance for magnitude.

V7Roadmap workload circular barplot

  • Workstream items

What — dataItems are roadmap phases with one ordered key (sequence) and one quantitative attribute (workstream items).

Why — taskCompare how much work each phase carries and see the cycle of delivery as a whole.

How — idiomCircular barplot: arc marks, angle for the ordered phase key, radial length for magnitude.

V8Architecture composition: treemap

  • Component
  • Layer

What — dataA hierarchy: architecture layers containing components, each component counting as one unit.

Why — taskSee where the system's weight sits: which layers hold the most moving parts.

How — idiomTreemap: containment for the hierarchy, area marks sized by component count, nested rectangles for layers.

07

System Architecture

robots.txt corpus (809 domains)
      |
      v
+---------------------+      +--------------------------+
|  SCA parser              | <--- |  RFC 9309 validation       |
|  (4 bugs fixed)           |      |  ground-truth spec check    |
+---------------------+      +--------------------------+
      |
      v
+---------------------+
|  Compliance scoring       |
|  defense-tier grouping     |
+---------------------+
      |
      v
+---------------------+
|  Statistical analysis      |
|  Matplotlib figures         |
|  RQ1–RQ3                     |
+---------------------+

Validation

  • SCA codebase audit
  • 4 bugs fixed against RFC 9309
  • Ground-truth spec compliance

Analysis

  • Compliance-score formula
  • Defense-tier classification
  • Enumeration Fallacy handling

Reporting

  • Statistical tests
  • Matplotlib figure set (RQ1–RQ3)
  • Written thesis and examiner response

10

Dataset scope

Dataset scope
SampleDomainsPurpose
Close analysis809European news domains, in-depth defense-tier classification
Scale analysis809 domain set for statistical patterns

Both samples were parsed with the RFC-9309-validated SCA tooling.

R1

Technical Risks

Risk matrix
RiskImpactMitigation
Relying on unvalidated third-party parsing toolingSystematically wrong compliance classificationsAudited and fixed the SCA codebase against RFC 9309 before running any analysis
Overcounting near-duplicate directives (the Enumeration Fallacy)Inflated apparent complianceExplicitly framed and corrected for the Enumeration Fallacy in the compliance-score formula
Statistical methodology questioned by the examinerFindings not fully defensibleRevised statistical tests and validation limitations in response to formal feedback

R2

Implementation Roadmap

  1. Early thesis

    Phase 1: Tooling validation

    • RFC 9309 review
    • SCA audit and bug fixes

  2. Mid thesis

    Phase 2: Analysis

    • Defense-tier framework
    • Compliance-score formula
    • RQ1–RQ3 figures

  3. Late thesis

    Phase 3: Revision

    • Examiner feedback
    • Statistical test revisions
    • Final submission

R4

Success Metrics

Scale

  • 809 domains analyzed in depth

Tooling

  • 4 SCA bugs fixed
  • Validated against RFC 9309

Output

  • Full defense-tier classification
  • Matplotlib figure set for RQ1–RQ3

R5

Deliverables

  • Full thesis with a defense-tier classification framework
  • Semantic Configuration Analyzer (SCA) validated against RFC 9309 (4 bugs fixed)
  • Matplotlib figure set answering RQ1–RQ3
  • Written response to examiner feedback

“Before trusting any conclusion at scale, I had to trust the tool producing the numbers, and that meant validating it against the spec myself, bug by bug.”

Next project

RedRiver Internal Platform

View Case Study