Monitor the data feed,not only the crawler.

A crawler can finish successfully and still deliver a bad dataset. It may receive valid HTTP responses while parsing an error template, miss a new pagination path, convert unavailable prices into zeroes, or return only the first page after a source redesign. If the monitoring system checks only whether the process exited, every one of those failures can look healthy.

Web scraping monitoring must therefore observe the data outcome as well as the machinery. A dependable operational data feed needs evidence that the expected sources were reached, the agreed records were extracted, the output passed its rules, and the accepted data arrived where it was supposed to go.

This runbook explains what to monitor, how to turn deviations into actions, and what should happen between the first alert and a corrected delivery.

A successful run is not necessarily a successful feed

Treat pipeline health, data health, and delivery health as three separate states.

Pipeline health asks whether the scheduled work started, completed, retried correctly, and stayed inside its expected runtime. This is the conventional operational view.

Data health asks whether the output still represents the agreed source scope. It covers record counts, required fields, schema validity, stable identity, duplicate rates, freshness, and representative comparisons with the source.

Delivery health asks whether the accepted records reached the destination with the expected schema version, timestamp, manifest, and transfer evidence.

These states can disagree. A crawler can complete while the data fails. A dataset can pass validation while delivery to the customer’s storage fails. A delivery can complete while containing yesterday’s output because freshness was never checked.

Google’s Site Reliability Engineering guidance makes a similar distinction between internal system telemetry and user-visible outcomes. Its monitoring chapter describes both white-box monitoring of internals and black-box monitoring of observable service behavior. For a data feed, the black-box question is not merely “did the endpoint respond?” but “did the expected, usable data arrive?” (Google SRE)

The five layers of web scraping monitoring

Monitoring is easier to reason about when each layer has a defined signal and owner.

Scheduling and orchestration

Record when each expected run should start, when it actually starts, its terminal state, duration, retries, and the sources or partitions it attempted. A missing run must be observable; silence is not success.

Periodic pipelines deserve special care. Google’s SRE discussion of data-processing pipelines notes that batch work becomes fragile as volume, duration, and dependencies change, and that monitoring only after completion leaves poor visibility into failures that happen mid-run. (Google SRE)

Source access

Track whether each source returned the expected content state, not only a status code. Detect challenge pages, empty shells, consent screens, maintenance templates, redirects to category roots, and unexpected language or region changes. Save enough non-sensitive evidence to diagnose a deviation without retaining more source content than the project needs.

Coverage should reflect the agreed source scope. A competitor and market intelligence feed might track entities and signal types; a product feed might track categories, product identifiers, sellers, and locales. “Requests completed” is not a substitute for “scope observed.”

Extraction

Measure records discovered and parsed, parser-path usage, required-field population, and the proportion of pages that produced no valid record. Compare these values by source and over time. A single global record count can hide a failed source behind growth somewhere else.

Validation

Run record-level, dataset-level, and source-sample checks before release. Validate schema, types, allowed values, identifiers, duplicates, ranges, freshness, and source fidelity. For feeds with unstable identity—such as real estate listing data—monitor duplicate candidates, relistings, and unexpected changes to matching confidence.

Delivery

Confirm that the destination write succeeded and that the delivered file or object has the expected record count, schema version, timestamp, and checksum. A checksum protects transfer integrity; it does not prove that the values are semantically correct. Amazon S3 documents checksum verification for upload and download integrity, which is useful at this final layer but separate from data-quality validation. (Amazon S3)

The minimum useful monitoring view

A useful view answers a small set of operational questions without requiring the customer to interpret raw crawler logs:

• When did the last expected collection and delivery complete?
• Which sources, categories, or entities were expected and observed?
• How many records were accepted, quarantined, and rejected?
• What proportion of required fields was populated?
• Did the records conform to the agreed schema and field rules?
• Did duplicate, identity, or outlier rates move unexpectedly?
• Is each source inside its freshness window?
• Were retries or access deviations concentrated on one source?
• Did representative source-to-output samples pass?
• Did the destination receive the expected manifest and checksum?

Do not apply one universal threshold to every feed. A 20% record-count drop can be an incident for a stable product catalog and normal seasonality for a job board after a holiday. Job postings and labor-market data often needs lifecycle and stale-record logic that differs from product availability monitoring.

Thresholds belong in the feed specification. Define them from a baseline, the cost of false alarms, the business impact of missed data, and known source behavior. Then revise them when evidence shows that the baseline has changed.

Service objectives should describe the outcome. Google’s SRE guidance for data pipelines highlights throughput, end-to-end latency, and correctness as useful indicators—not only infrastructure uptime. (Google SRE)

Alerts should reflect the feed specification

An alert is useful only when it leads to a proportionate action. Define that action before production.

Use a warning when a deviation deserves investigation but the output remains within an agreed tolerance. Use quarantine when individual records are questionable but the accepted subset remains safe to deliver. Stop delivery when a critical rule fails across the dataset or when the result could mislead downstream systems. Report a known non-critical exception when the customer has agreed that delivery should continue with evidence.

Avoid alerts that page a person for every retry or isolated source timeout. Google’s SRE monitoring guidance recommends paging on symptoms that require immediate human action and handling lower-priority events through non-interrupting channels. The same principle reduces noise in scraper operations. (Google SRE)

Every alert should state:

• the affected feed, run, source, and time window;
• the rule or expectation that failed;
• the observed value and applicable threshold;
• whether records were delivered, held, or quarantined;
• the next owner and expected investigation path;
• the evidence needed to close the incident.

A practical incident runbook

The exact tools differ, but the sequence should remain understandable.

1. Detect

Identify which signal moved and whether the problem affects infrastructure, one source, a parser path, validation, or the destination. Preserve the failed-run evidence and schema version.

2. Contain

Prevent questionable records from entering production. Depending on the rule, pause the delivery, quarantine a subset, or continue with a documented exception.

3. Diagnose

Compare current source samples with the last accepted run. Check source content state, navigation, parser coverage, transformations, and identity rules. Determine the first affected observation window.

4. Repair

Update the relevant collection, parser, mapping, or delivery behavior. Add a regression fixture or validation rule for the failure pattern so the same issue becomes easier to detect.

5. Re-collect or backfill

If the affected source still exposes the required state, re-run the impacted scope. Where historical states cannot be reconstructed, document the gap rather than inventing values.

6. Revalidate

Apply the complete acceptance suite, not only the rule that triggered the incident. A source change can break several fields in different ways.

7. Deliver and communicate

Issue the corrected data, updated manifest, affected window, and a concise incident summary. Tell the customer whether any previously delivered records should be replaced.

8. Learn

Update monitoring thresholds, fixtures, runbook steps, and the source risk profile. Incident response should reduce the time to detect and recover from a similar failure.

Worked example: a daily product and stock feed

Imagine a daily product, price, and stock monitoring feed. The crawler completes, but one retailer’s record count falls 42% and most remaining stock values become null. These figures are hypothetical; they illustrate the decision path rather than a universal threshold.

The scheduler reports success. Source monitoring shows that the retailer returned normal product pages. Extraction monitoring reveals that a redesigned variant selector caused the parser to stop discovering several offers. Required-field completeness fails, and a source-to-output sample confirms that visible stock states are missing.

The runbook quarantines that retailer’s affected records and allows the other sources to continue. Engineering updates the variant path, QA adds the redesigned page to regression coverage, and the affected source is re-collected. The full dataset is revalidated before the corrected retailer subset and new manifest are delivered.

The important point is not the selector fix. It is that coverage, completeness, source sampling, quarantine, correction, and delivery evidence worked together. No customer had to discover the issue in a dashboard.

The same principle applies to public business and directory data: stable identifiers, duplicate candidates, and field population must be monitored separately from the number of pages fetched.

What to ask a managed web scraping provider

Use operational questions that require concrete answers:

1. What exactly constitutes a successful collection and a successful delivery?
2. Which source-coverage and data-quality rules are documented before production?
3. How do you detect a crawler that succeeds but returns incomplete or incorrect data?
4. What happens to records that fail validation—retry, quarantine, rejection, or silent drop?
5. Which incidents block delivery, and which are reported as exceptions?
6. Are correction and backfill included when a source change breaks an agreed field?
7. How is destination delivery verified?
8. What evidence appears in a run or quality report?
9. Who communicates when a source removes data or changes access fundamentally?
10. Who owns the delivered data, and what happens during offboarding?

Ask for a representative artifact, not a generic uptime percentage. WebTruffle publishes a synthetic sample data quality report to show the distinction between run evidence, data rules, quarantined records, and delivery integrity.

Frequently asked questions