How to bulk-audit Core Web Vitals for hundreds of URLs without a PageSpeed API key

PageSpeed Insights and the Lighthouse panel in Chrome DevTools are the right tools when you are debugging one URL. They are the wrong shape of tool when you need a spreadsheet of lab Core Web Vitals for a sitemap, every product template after a release, or a monthly report across a client's sites.

The usual next step is Google's PageSpeed Insights API. It works, and it is free up to a quota, but that shared quota is easy to burn through once you start auditing hundreds of URLs, on mobile and desktop, more than once. Local Lighthouse on your laptop also works, if you are happy to install Chrome, walk a sitemap, keep median runs and stitch the JSON together yourself.

This tutorial shows a practical way to run real Google Lighthouse on a list of URLs or a whole sitemap, with no PageSpeed Insights API key and no shared quota, using Bulk Lighthouse Audit on Apify. You can do the same job in the browser in a couple of minutes, or from Python or curl if you want it on a schedule or in CI. Only audit sites you own or are allowed to test.

What you will get

Each row in the dataset is one page on one device. Fields include:

url, finalUrl, strategy (mobile or desktop) and engine
scores: Performance, Accessibility, Best Practices and SEO (0–100)
Lab metrics: largestContentfulPaintMs, cumulativeLayoutShift, totalBlockingTimeMs, plus FCP, Speed Index, TTI, TTFB (serverResponseTimeMs), totalByteWeight, domSize and networkRequests
labCwv (LCP, CLS and TBT each rated good, needs-improvement or poor) and an overall labCwvAssessment
The lcpElement and lcpBreakdownMs (time to first byte vs render delay)
opportunities: the top fixes, with estimated time and bytes saved
failedAudits: failing accessibility, SEO and best-practice checks, listed by name
passedThreshold, if you set a performance budget
Optional reportUrl for a shareable HTML Lighthouse report (local engine only)
Optional fieldData: real-user CrUX p75 LCP, INP, CLS, FCP and TTFB, if you add a free Google API key
lighthouseVersion, runs, error and auditedAt

The dataset has two views: Scores & Core Web Vitals, and Top fixes (one row per recommendation). A SUMMARY record in the key-value store holds site-level averages, the slowest pages and the lab Core Web Vitals pass rate (labCwvPassRate).

Lab Core Web Vitals here are LCP, CLS and TBT. TBT is Lighthouse's lab stand-in for INP. Real-user INP only appears in fieldData, and that needs a Google API key.

Option 1: run it from the Apify Console (no code)

Create a free Apify account if you do not have one. The free plan includes $5 of monthly credit, which covers hundreds of local audits.
Open Bulk Lighthouse Audit and click Try for free.
Paste one or more URLs, or a sitemap URL. The form is prefilled with https://example.com. Sitemap indexes and .gz files are fine.
Keep Devices on mobile for a first run (a simulated mid-range phone on slow 4G, the same method PageSpeed Insights uses). Add desktop when you need both.
Leave Save full HTML Lighthouse reports off unless you need the shareable report. Leave the engine on Auto: with no key, that runs Lighthouse 13 inside the Actor's own Chrome.
Optionally set Max URLs (default 50, counted per device), a Performance budget (for example 80) and Runs per URL. Google recommends 3–5 runs and taking the median when you need stable numbers; each run is billed.
Click Start. A single mobile audit usually finishes in well under a minute.
Open the Dataset tab for scores and Core Web Vitals, and the key-value store for the SUMMARY record. Export CSV or JSON, or pull the results through the API.

That is the whole one-off workflow. Schedule the same input weekly or after each release when you want a history.

Example input

A sitemap, mobile only, capped at 50 pages, with a performance budget:

{
  "sitemapUrls": ["https://crawlee.dev/sitemap.xml"],
  "maxUrls": 50,
  "strategies": ["mobile"],
  "performanceThreshold": 80,
  "runsPerUrl": 1,
  "saveHtmlReports": false
}

A short URL list on mobile and desktop:

{
  "urls": [
    "https://www.python.org",
    "https://www.wikipedia.org",
    "https://example.com"
  ],
  "strategies": ["mobile", "desktop"],
  "performanceThreshold": 80,
  "maxConcurrency": 1
}

You can combine urls and sitemapUrls in one run. The Actor collects the sitemap loc entries, de-duplicates them, and audits each page up to maxUrls per device.

Option 2: run it from Python or curl

Install the official client and call the Actor. Replace the token with your Apify API token (Console → Settings → Integrations). Do not commit the token.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("refillflows/bulk-lighthouse-audit").call(
    run_input={
        "sitemapUrls": ["https://example.com/sitemap.xml"],
        "maxUrls": 50,
        "strategies": ["mobile"],
        "performanceThreshold": 80,
        "saveHtmlReports": False,
    }
)

dataset_id = run["defaultDatasetId"]
items = list(client.dataset(dataset_id).iterate_items())
print(f"Got {len(items)} audits")
for row in items[:5]:
    print(row["url"], row["strategy"], row.get("scores"), row.get("labCwv"))

# Fail a CI job if any page missed the budget
failed = [r for r in items if r.get("passedThreshold") is False]
if failed:
    raise SystemExit(f"{len(failed)} page(s) below the performance budget")

The same call with curl, waiting for the dataset and returning CSV:

curl -X POST \
  "https://api.apify.com/v2/acts/refillflows~bulk-lighthouse-audit/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN&format=csv" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "strategies": ["mobile"],
    "saveHtmlReports": false
  }'

If you already have a DATASET_ID from a finished run, you can also download it with:

https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv&token=YOUR_APIFY_TOKEN

The site-level SUMMARY record lives in the run's default key-value store:

https://api.apify.com/v2/key-value-stores/<VALUE_STORE_ID>/records/SUMMARY?token=YOUR_APIFY_TOKEN

Schedule the Actor daily or weekly, set a performanceThreshold, and post rows where passedThreshold is false to Slack, Sheets or a webhook from a CI pipeline.

Pricing, in practice

The Actor is pay-per-event. You are charged for audits that complete, and platform usage is included in that price. Pages that cannot load are reported with an error field and are not charged.

What	Price
Actor start	$0.005 per run (charged per GB of memory; this Actor defaults to 2 GB, so $0.01)
Lighthouse audit (local engine, no key)	$0.01 per page per device
Audit with your PageSpeed API key	$0.003 per page per device

A 100-page site on mobile is about $1.00, or $2.00 for mobile plus desktop, plus the $0.01 start charge. With your own Google key it is $0.30 per 100 pages. If you set Runs per URL to 3, you pay for three audits per page. The $5 free-plan credit covers hundreds of local audits, or more if you switch to the PSI engine. Set Max URLs or a max cost per run if you are exploring.

Tips that save time

No key is required. The default Auto engine runs local Lighthouse when no Google API key is set, so there is no PSI quota to run out of. Add a key only if you want the cheaper PSI engine and real-user CrUX field data (includeFieldData defaults to on once a key is present; it falls back from URL-level to origin-level data).
To get a free Google API key, open Google Cloud Console, enable the PageSpeed Insights API and the Chrome UX Report API, and create an API key. Paste it into Google API key.
Lab scores vary slightly between machines. They come from a simulated Moto G Power on slow 4G. Use runsPerUrl of 3 or more when you need precise numbers, and remember that each run is billed.
Keep maxConcurrency at 1 for the most accurate local scores (CPU contention skews metrics). Use 2–4 with 4 GB or more of memory when you need speed. The PSI engine can go higher, up to 10.
Fewer Lighthouse categories is faster. The default is all four (performance, accessibility, best-practices, seo). You can drop the ones you do not need.
Leave HTML reports off for routine monitoring. Turn Save full HTML Lighthouse reports on when you want a shareable reportUrl for a specific page (local engine only).
maxUrls is per device. 50 URLs on mobile and desktop is 100 audits.
Failed loads are free. DNS errors, interstitials and pages that never paint are stored with an error and are not billed. The default timeout is 120 seconds per audit (auditTimeoutSecs).

May I audit any website?

The Actor loads only the pages you give it, the same way a normal browser visit does. Only audit sites you own or are allowed to test.

How to bulk-audit Core Web Vitals for hundreds of URLs without a PageSpeed API key

How to bulk-audit Core Web Vitals for hundreds of URLs without a PageSpeed API key

What you will get

Option 1: run it from the Apify Console (no code)

Example input

Option 2: run it from Python or curl

Pricing, in practice

Tips that save time

May I audit any website?

Further reading

Comments

More from this blog

How to track Shopify App Store rankings and pricing over time

How to export all Luma events for a city to CSV