Methods

This page documents how the Business AI Market Observatory collects, stores, interprets and analyses evidence. The guiding rule is that raw evidence and interpretation are stored separately, that history is observed, never reconstructed, and that every number on the site can be traced to a quoted evidence window with a parser version and a confidence score.

1. Unit of analysis and entities

The panel is organised around a normalized entity model (see the schema documentation below):

• Company → Product (a product is the unit of observation; a company may have several, e.g. ChatGPT and the OpenAI API).
• Source: a URL belonging to a product (pricing page, product page, documentation, announcement).
• Snapshot: one immutable capture of a source at one instant (captured_at), with the raw bytes stored gzip-compressed on disk and hashed.
• Observation tables: plan_observation (a plan card as printed on a page), feature_observation (presence of a lexicon feature), provider_observation (a provider or model named on the page), and a generic observation table (description, positioning, unparsed pricing).
• Plan, Feature, Provider, Category: the normalized vocabularies observations point at.
• Events: pricing_event, feature_event, product_event are derived by diffing consecutive snapshots of the same product; curated entry/exit events cite a public announcement.
• Snapshot rows and observation values are protected by database triggers: they cannot be updated or deleted. A correction is a new row whose supersedes_id points at the row it replaces; only review columns are mutable.

2. Panel selection

The pilot panel (data/panel.yaml, panel v1.0) contains 125 products from 121 companies. Inclusion rule: the product (a) sells an AI-native capability to businesses, developers or prosumers, (b) has or had a public pricing or product page, and (c) is prominent in 2023–2026 market coverage across the 18 taxonomy categories. The panel is a purposive sample selected in September 2026, not a census. Because selection conditions on being known in 2026, products that entered and disappeared quietly are under-represented. To partially offset this, seven products with documented exits or acquisitions (Adept, Inflection/Pi, Casetext/CoCounsel, Rockset, MosaicML, Codeium/Windsurf, Weights & Biases) were added deliberately from public announcements. Launch and exit dates are curated with a precision flag (day, month, year) and a source URL; the announcement pages themselves are fetched and stored as announcement snapshots where robots.txt allowed.

3. Collection

Three collection methods feed the same snapshot table; the method is recorded on every row.

1. Archived captures (Internet Archive Wayback Machine). This is the only public source of genuinely historical page states. For every pricing-URL variant of every product we query the CDX index for status-200 captures from 2022 onward, collapsed to at most one per month, and fetch the original bytes with the id_ flag (no toolbar, no URL rewriting). The archive's capture timestamp becomes captured_at. Requests are single-threaded per worker with a 1–1.5 s delay, a descriptive User-Agent, and exponential backoff on 429/5xx.
2. Live collection. For each product's current pricing and product URL we check robots.txt (both our token and *), fetch at most one request per host every 3 s with a descriptive User-Agent, and store the response. HTTP 403/429/503 or challenge markers are logged as anti_bot and never retried or bypassed; those products rely on archived captures or manual ingestion.
3. Manual / archived ingestion. For pages that cannot be collected automatically (JavaScript-only rendering, anti-bot walls, PDFs, press releases), an analyst records a verbatim quote, the URL, the observation time and their identity in data/manual/*.yaml. The quote is stored as a real snapshot (collection_method='manual') and flows through the same extractors, provenance and review machinery.

Every attempt, successful or not, is written to collection_log with its outcome, so collection failures are themselves data (see the data-quality page).

Historical URL variants matter: pricing pages move (openai.com/pricing → openai.com/api/pricing, codeium.com/pricing → windsurf.com/pricing). Each product lists its known variants; Wayback canonicalises www. and trailing-slash variants, and byte-identical captures at the same instant are stored once.

4. Extraction

All extraction is deterministic and versioned. No large language model was used to produce any value in the released dataset; the LLM-assisted path is designed (confidence field, review queue) but the pilot relies on rule-based extractors so that every value is exactly reproducible from the raw snapshot.

• Text: HTML → visible text with block boundaries preserved (text-1.0).
• Pricing parser (price-1.7): finds plan headers (a whitelist of tier words plus inline "Pro Tier: …" headers), attaches the price language that follows each header, classifies the unit from the raw wording (seat, flat, tokens, usage, credits, custom, free), and records the verbatim price string and a 600-character evidence window. Pages that are token-price tables get a synthetic api_usage plan holding the raw list of token prices. Cards whose title is a product name rather than a tier word fall back to a named: plan key at lower confidence and are excluded from price statistics until a human verifies them. A billing-period classifier bug (a later, unrelated price in the same text window could leak its "/month" or "/year" onto an earlier price token) was found during review and fixed in v1.7 — see the Reviewer 2 section of the paper. Confidence is reduced for multiple price tokens in a block, missing billing period, missing unit or missing currency; observations below 0.6 are queued for review.
• Normalization rule: monthly_seat_usd is filled only when the currency is USD, the billing period is monthly (or an explicitly stated annual-billing monthly equivalent, flagged in billing_basis) and the unit is per-seat or flat; annual totals are divided by 12 with an explicit note. Token, usage and credit prices are never converted. Non-USD prices are never converted.
• Feature lexicon (feat-1.1): 31 features, each with a definition, a family (capability, governance, commercial, modality), a fixed confidence reflecting pattern specificity, and regular expressions. 26 are tracked for diffusion. The lexicon is designed to avoid the most common false positives (e.g. "support agents" does not trigger AI agents), but it detects claims on pricing pages, not verified capabilities.
• Provider lexicon (prov-1.3): 19 providers with model-name patterns (higher confidence) and company-name patterns (lower confidence). A mention on a product's own page yields a uses_model relation (or hosts_model when an infrastructure product's page lists three or more third-party models, which is catalogue evidence rather than a dependence claim), or offers_own_model when the page belongs to the provider. Absence of a mention is not evidence of independence.
• Positioning (pos-1.0): meta description; counts of target-customer, enterprise-governance and vertical-industry terms; a vertical specificity index (share of vertical mentions captured by the single most-mentioned vertical, 0–1) and enterprise terms per 1,000 words.

4a. Manual review

The pilot's review pass (scripts/apply_pilot_review.py) is a disclosed, heuristic-assisted human review, not row-by-row verification of all 11,808 plan observations. An analyst read roughly 500 of the ~1,150 latest-snapshot plan cards verbatim (scripts/review.py list) and found two recurring error classes: (1) "worked example" text captured as a plan — illustrative billing calculations, ROI/savings claims, press-release ARR figures, and in one case (Hebbia) a snapshot that captured a contract-analysis case-study page instead of the pricing page; and (2) ambiguous dual-price strings ("$18 $10 per seat/month, billed annually") where it is not certain which number the printed billing qualifier describes. Rule (1) is applied as a content regex across every plan observation, since the same text patterns recur wherever they occur, not only in the sampled rows; a short list of specific rows the analyst individually judged (genuine named-card prices worth promoting, or the one-off Fintool capture of an unrelated Microsoft support page) is applied by row id. Rule (2) flags rows for review rather than rejecting them — the parser's first-token convention matched the analyst's independent knowledge of every product checked, but per-row correctness was not verified. The pass reviewed 11,808 rows: 7,623 verified, 297 rejected (removed from price statistics; the underlying rows remain in the database, immutable, for audit), 1,380 newly flagged as ambiguous, the rest unreviewed. Corrections never edit a value; the script only ever writes to review_status/reviewed_by/reviewed_at/review_note.

The review pass also surfaced two engineering bugs, both fixed and regression-tested: a price parser bug where a later, unrelated price in the same text window could leak its billing period onto an earlier token (fixed in price-1.7), and a provider-lexicon bug where bare 'Nova'/'Titan' patterns matched a demo person's name, a different vendor's own same-named model, and an unrelated deal codename (fixed in prov-1.3). A third, unrelated bug was found independently while re-running the pipeline rather than during content review: a bookkeeping observation was re-inserted on every extraction rerun for snapshots with zero parsed plans. It was fixed, 2,332 stale duplicate rows were removed (scripts/dedup_observation_bug.py), and extraction is now covered by an idempotency regression test.

5. Events

Consecutive pricing-page snapshots of a product (ordered by capture time, at most one per month, preferring the URL variant with more parsed plans) are diffed (diff-1.0). Price changes require the same plan key, unit and currency. Feature additions/removals and provider additions/removals are recorded with the interval (date_lower, date_upper] between the two captures; we never claim a point date from a diff. A feature that flips and flips back at the very next snapshot is flagged is_flicker and excluded from diffusion statistics. Token-price lists do not generate price events because the parser cannot pair a price with a model across page layouts. Event confidence is the minimum of the two observations' confidences.

6. Taxonomy

Taxonomy v1.0 has 18 categories: the 17 in the study design plus voice_audio, added because speech products fit nowhere else without distorting video or infrastructure. Each category has an orientation (horizontal, vertical, infrastructure) and a one-sentence definition; assignment is curated from the product's own stated job-to-be-done, multi-label, with one primary label. Revisions are logged in data/taxonomy.yaml and stored in taxonomy_revision; assignments record method and confidence so that a future lexicon- or LLM-based assignment can coexist and be compared.

7. Analysis

The quarterly panel takes, for each product and quarter, the latest pricing snapshot inside that quarter; no state is carried forward unless an analysis explicitly says so. Summary statistics are computed only for quarters with at least eight observed products.

• RQ1 convergence: mean pairwise Jaccard similarity of tracked-feature sets, with a product-level bootstrap 95% interval, within-category variant, a balanced sub-panel (products observed in every recent quarter) and a first-vs-last prevalence decomposition.
• RQ2 pricing: entry-price distribution (median, IQR, coefficient of variation, modal-point shares), free/custom tier shares, pricing-model mix and its entropy, and the list of confident price changes.
• RQ3 diffusion: adoption share by quarter, first-adopter tables, lag from first adopter, and Kaplan–Meier time-to-adoption from each product's first snapshot with left-censored adopters excluded and reported.
• RQ4 providers: product→provider edges with quoted evidence, provider link counts, an HHI on link counts, multi-homing distribution and quarterly named-provider shares.
• RQ5 specialization: vertical share of entries by year; panel means and within-product first-vs-last change in vertical specificity and enterprise-governance language.
• RQ6 entry: entries by category and year (primary and multi-label), documented exits.
• Market map: TF-IDF (1–2-grams) over product-year page texts → 50-d SVD → cosine similarity, KMeans (k by silhouette) and a t-SNE layout used for visualisation only.
• Regressions: descriptive OLS of tracked-feature counts and log entry price on a quarter index with category or product fixed effects and product-clustered standard errors.

All statistics are descriptive. The panel is unbalanced and purposively selected; nothing here identifies a causal effect.

8. Reproducibility

python -m venv .venv && .venv/bin/pip install -r requirements.txt
python scripts/collect_wayback.py --months monthly     # archive collection (idempotent)
python scripts/collect_live.py                          # robots-respecting live collection
python scripts/collect_announcements.py                 # announcement pages cited for entry/exit
python scripts/run_analysis.py                          # extraction -> events -> analysis JSON
python scripts/review.py list | apply                   # manual review sheet / decisions
python scripts/release.py vX.Y.Z "notes"                # versioned CSV release + checksum
python -m pytest                                        # 25 tests: parser, normalization, diffs, provenance, taxonomy

The site (site/) is a Next.js app that reads the exported JSON; pnpm test runs the filter and API tests. Dataset releases are versioned CSV bundles under data/releases/ with a SHA-256 manifest; the release version is stamped in the site footer.

Data card — Business AI Market Observatory pilot panel

Version: v0.1.2 · Created: 2026-09-07 · Checksum: 4255c131f876f02cbb4c91194c6442d1193e702803591aa22440269d46aa92e8 Parser versions: pricing price-1.7, features feat-1.1, providers prov-1.3, events diff-1.0 · Taxonomy: v1.0 · Panel: v1.0

What it is

A longitudinal panel of public web observations of commercial AI products: immutable page snapshots plus versioned, evidence-quoting interpretations (plans, features, providers, positioning) and derived events.

Composition

• 121 companies, 125 products, 403 sources.
• 4,642 snapshots (live 250, wayback 4,392), 2022-01-01 → 2026-09-06.
• 11,808 plan observations, 148,544 feature observations, 8,433 provider observations.
• 2,283 pricing events, 3,530 feature events, 608 product events (entry/exit/provider changes).
• Category coverage (multi-label): agents 48, infrastructure 28, productivity 25, coding 15, sales 14, foundation_model 13, research 12, data 12, video 11, marketing 11, image 10, customer_support 9, design 8, legal 6, analytics 6, hr 5, finance 5, voice_audio 4.

Collection

Internet Archive CDX + original-bytes fetch (polite, single-threaded per worker); robots-respecting live fetch with descriptive User-Agent; manual/archived ingestion. Every attempt logged (collection_log). Anti-bot controls and robots.txt disallows are never bypassed.

Provenance fields on every observation

source URL, capture timestamp, collection method, archive URL, content hash, extraction method, parser version, verbatim evidence text, confidence, review status/reviewer/time, supersedes_id for corrections.

Intended uses

Descriptive research on product strategy, pricing, feature diffusion and provider dependence; teaching material on provenance-first data engineering; a base for scaling collection.

Not intended for

Market-share or revenue claims (no such data collected); capability benchmarking (features are page claims); causal inference; any use requiring population coverage; personal data (none collected — business pages only).

Known limitations

Purposive panel selected in 2026; uneven archive coverage; lexical extraction with measurable noise (33.1% of plan observations below 0.6 confidence; 7811 rows in review); pricing normalized only for USD subscriptions; provider links are disclosure-based; launch dates curated with declared precision (year 49, day 45, month 31).

Versioning

Releases are CSV bundles under data/releases/<version>/ with a manifest.json (row counts, SHA-256). Observation rows are never edited; a new parser appends rows with a new parser_version. Taxonomy changes are logged in taxonomy_revision.

Licence and ethics

Observations are of public business web pages; no employee or customer personal data is collected. Raw snapshots are stored locally for verification and are not redistributed; the release contains derived tables and evidence quotes. Code: MIT. Data: CC BY 4.0 for derived tables.

Schema documentation

Generated from observatory/schema.sql. SQLite. Raw evidence (snapshot) is separated from interpretations (*_observation); observation tables are append-only and guarded by triggers; events are derived and carry interval-censored dates.

schema_version

company

product

taxonomy_revision

category

product_category

source

snapshot

observation

plan

plan_observation

feature

feature_observation

provider

provider_observation

pricing_event

feature_event

product_event

review_queue

collection_log

dataset_release

Immutability triggers

• snapshot_no_update: blocks UPDATE on snapshot
• snapshot_no_delete: blocks DELETE on snapshot
• plan_obs_freeze: blocks UPDATE OF price_raw, price_amount, currency, billing_period, pricing_unit, monthly_seat_usd, is_free, is_custom, plan_id, snapshot_id, evidence_text on plan_observation
• plan_obs_no_delete: blocks DELETE on plan_observation
• feat_obs_freeze: blocks UPDATE OF present, evidence_text, snapshot_id, feature_id on feature_observation
• feat_obs_no_delete: blocks DELETE on feature_observation
• prov_obs_freeze: blocks UPDATE OF relation, evidence_text, snapshot_id, provider_id on provider_observation
• prov_obs_no_delete: blocks DELETE on provider_observation
• obs_freeze: blocks UPDATE OF value_json, raw_text, snapshot_id, field on observation
• obs_no_delete: blocks DELETE on observation

Review columns (review_status, reviewed_by, reviewed_at, review_note) are the only mutable fields on observation rows. A correction is a new row whose supersedes_id points at the row it replaces.