Internet Archive Wayback Machine

The Internet Archive Wayback Machine is a public archive of historical snapshots of web pages, captured by crawlers since 1996. For research it is the standard way to observe a website as it was at a past date: page content, update frequency, whether a site existed, and how it changed over time. The data is free and public. Used in, for example, Hu & Ma, where website update frequency over the three years after a pitch is built from Wayback Machine captures as a startup-survival measure.

• Cost: free, public.
• API key: none required (a descriptive User-Agent is courteous; the APIs are rate-limited).
• Coverage: snapshots from 1996 onward; depth varies enormously by site (popular sites captured many times a day, obscure ones rarely).
• Home: https://web.archive.org/

Access

Two no-key JSON APIs serve the archive programmatically.

Availability API answers “is there a snapshot of this URL near this date?”:

# Closest snapshot of sec.gov to 1 Jan 2010
curl -sL "https://archive.org/wayback/available?url=sec.gov&timestamp=20100101"
# -> {"archived_snapshots":{"closest":{"available":true,
#     "url":"http://web.archive.org/web/20100102140201/http://www.sec.gov/", ...}}}

CDX API lists the full capture history, one row per snapshot, and is what you want for update-frequency or change-over-time measures:

# All captures of a URL in 2009, deduplicated to one per day
curl -sL "https://web.archive.org/cdx/search/cdx?url=example.com\
&from=20090101&to=20091231&output=json&collapse=timestamp:8"

CDX returns [urlkey, timestamp, original, mimetype, statuscode, digest, length] per capture. The digest is a content hash: two consecutive captures with the same digest are byte-identical, which is how you separate a recrawl from an actual content change. To fetch the archived page itself, request https://web.archive.org/web/<timestamp>/<original-url> (append id_ to the timestamp, e.g. 20100102140201id_, for the raw capture without the Wayback toolbar).

Load in Python

import requests

def captures(url, year):
    cdx = ("https://web.archive.org/cdx/search/cdx?url=" + url +
           f"&from={year}0101&to={year}1231&output=json&collapse=digest")
    rows = requests.get(cdx, timeout=60,
                        headers={"User-Agent": "research (contact@example.org)"}).json()
    return rows[1:]  # first row is the column header
# len(captures(...)) after collapse=digest = number of distinct content versions

Gotchas (the ones that bite pipelines)

• Coverage is not continuous, and absence is not proof. Snapshot density depends on crawl priority, robots directives, and luck. A gap in captures does not mean the page did not change or did not exist; it means it was not crawled. Treat the archive as an irregular, selection-biased sample, not a complete log. Update-frequency measures should be normalized by crawl frequency or restricted to well-crawled sites.
• A capture is a crawl, not a content change. The raw capture count overstates how often a page changed: the same content is recrawled repeatedly. Use collapse=digest (content hash) or compare the digest field across rows to count distinct versions; counting raw timestamps inflates “activity”.
• Rate limits are real and the APIs return 429. The CDX endpoint in particular throttles aggressively; sustained scraping returns 429 Too Many Requests. Throttle to a few requests per second, back off on 429, and cache responses. For large jobs the Internet Archive publishes bulk CDX/WARC access paths rather than hammering the live API.
• The snapshot is what the crawler saw, not the live page. JavaScript-heavy pages, lazy-loaded content, and resources blocked at crawl time may be missing or rendered differently from the original. The archived HTML can also embed the Wayback toolbar; use the id_ modifier on the timestamp to get the raw capture.
• Timestamps are UTC and 14 digits. The capture timestamp is YYYYMMDDhhmmss in UTC. The Availability API’s closest match can be on either side of your target date and arbitrarily far away if coverage is thin; always check how far timestamp is from what you asked for.
• statuscode matters. CDX rows include captures of redirects (3xx) and errors (4xx/5xx). Filter to statuscode=200 (and the right mimetype) before treating a row as a real page view, or filter=statuscode:200 in the query.

Endpoints

Endpoint	Use
archive.org/wayback/available?url=&timestamp=	Nearest single snapshot to a date (existence check)
web.archive.org/cdx/search/cdx?url=	Full capture history; output=json, collapse=, filter=
web.archive.org/web/<ts>/<url>	The archived page itself (<ts>id_ for the raw capture)

Citation

Cite the Internet Archive, the Wayback Machine, the archived URL and capture timestamp, and the access date, for example: Internet Archive, Wayback Machine, snapshot of http://www.sec.gov/ captured 2010-01-02 (https://web.archive.org/web/20100102140201/http://www.sec.gov/), accessed YYYY-MM-DD. Record the exact capture timestamp so the snapshot is reproducible, since the live site will have changed.