PageSpeed Insights and the Lighthouse panel in Chrome DevTools are the right tools when you are debugging one URL. They are the wrong shape of tool when you need a spreadsheet of lab Core Web Vitals for a sitemap, every product template after a release, or a monthly report across a client's sites.

The usual next step is Google's PageSpeed Insights API. It works, and it is free up to a quota, but that shared quota is easy to burn through once you start auditing hundreds of URLs, on mobile and desktop, more than once. Local Lighthouse on your laptop also works, if you are happy to install Chrome, walk a sitemap, keep median runs and stitch the JSON together yourself.

This tutorial shows a practical way to run real Google Lighthouse on a list of URLs or a whole sitemap, with no PageSpeed Insights API key and no shared quota, using Bulk Lighthouse Audit on Apify. You can do the same job in the browser in a couple of minutes, or from Python or curl if you want it on a schedule or in CI. Only audit sites you own or are allowed to test.

What you will get

Each row in the dataset is one page on one device. Fields include:

• url, finalUrl, strategy (mobile or desktop) and engine
• scores: Performance, Accessibility, Best Practices and SEO (0–100)
• Lab metrics: largestContentfulPaintMs, cumulativeLayoutShift, totalBlockingTimeMs, plus FCP, Speed Index, TTI, TTFB (serverResponseTimeMs), totalByteWeight, domSize and networkRequests
• labCwv (LCP, CLS and TBT each rated good, needs-improvement or poor) and an overall labCwvAssessment
• The lcpElement and lcpBreakdownMs (time to first byte vs render delay)
• opportunities: the top fixes, with estimated time and bytes saved
• failedAudits: failing accessibility, SEO and best-practice checks, listed by name
• passedThreshold, if you set a performance budget
• Optional reportUrl for a shareable HTML Lighthouse report (local engine only)
• Optional fieldData: real-user CrUX p75 LCP, INP, CLS, FCP and TTFB, if you add a free Google API key
• lighthouseVersion, runs, error and auditedAt

The dataset has two views: Scores & Core Web Vitals, and Top fixes (one row per recommendation). A SUMMARY record in the key-value store holds site-level averages, the slowest pages and the lab Core Web Vitals pass rate (labCwvPassRate).

Lab Core Web Vitals here are LCP, CLS and TBT. TBT is Lighthouse's lab stand-in for INP. Real-user INP only appears in fieldData, and that needs a Google API key.

Option 1: run it from the Apify Console (no code)

1. Create a free Apify account if you do not have one. The free plan includes $5 of monthly credit, which covers hundreds of local audits.
2. Open Bulk Lighthouse Audit and click Try for free.
3. Paste one or more URLs, or a sitemap URL. The form is prefilled with https://example.com. Sitemap indexes and .gz files are fine.
4. Keep Devices on mobile for a first run (a simulated mid-range phone on slow 4G, the same method PageSpeed Insights uses). Add desktop when you need both.
5. Leave Save full HTML Lighthouse reports off unless you need the shareable report. Leave the engine on Auto: with no key, that runs Lighthouse 13 inside the Actor's own Chrome.
6. Optionally set Max URLs (default 50, counted per device), a Performance budget (for example 80) and Runs per URL. Google recommends 3–5 runs and taking the median when you need stable numbers; each run is billed.
7. Click Start. A single mobile audit usually finishes in well under a minute.
8. Open the Dataset tab for scores and Core Web Vitals, and the key-value store for the SUMMARY record. Export CSV or JSON, or pull the results through the API.

That is the whole one-off workflow. Schedule the same input weekly or after each release when you want a history.

Example input

A sitemap, mobile only, capped at 50 pages, with a performance budget:

{
  "sitemapUrls": ["https://crawlee.dev/sitemap.xml"],
  "maxUrls": 50,
  "strategies": ["mobile"],
  "performanceThreshold": 80,
  "runsPerUrl": 1,
  "saveHtmlReports": false
}

A short URL list on mobile and desktop:

{
  "urls": [
    "https://www.python.org",
    "https://www.wikipedia.org",
    "https://example.com"
  ],
  "strategies": ["mobile", "desktop"],
  "performanceThreshold": 80,
  "maxConcurrency": 1
}

You can combine urls and sitemapUrls in one run. The Actor collects the sitemap loc entries, de-duplicates them, and audits each page up to maxUrls per device.

Option 2: run it from Python or curl

Install the official client and call the Actor. Replace the token with your Apify API token (Console → Settings → Integrations). Do not commit the token.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("refillflows/bulk-lighthouse-audit").call(
    run_input={
        "sitemapUrls": ["https://example.com/sitemap.xml"],
        "maxUrls": 50,
        "strategies": ["mobile"],
        "performanceThreshold": 80,
        "saveHtmlReports": False,
    }
)

dataset_id = run["defaultDatasetId"]
items = list(client.dataset(dataset_id).iterate_items())
print(f"Got {len(items)} audits")
for row in items[:5]:
    print(row["url"], row["strategy"], row.get("scores"), row.get("labCwv"))

# Fail a CI job if any page missed the budget
failed = [r for r in items if r.get("passedThreshold") is False]
if failed:
    raise SystemExit(f"{len(failed)} page(s) below the performance budget")

The same call with curl, waiting for the dataset and returning CSV:

curl -X POST \
  "https://api.apify.com/v2/acts/refillflows~bulk-lighthouse-audit/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN&format=csv" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "strategies": ["mobile"],
    "saveHtmlReports": false
  }'

If you already have a DATASET_ID from a finished run, you can also download it with:

https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv&token=YOUR_APIFY_TOKEN

The site-level SUMMARY record lives in the run's default key-value store:

https://api.apify.com/v2/key-value-stores/<VALUE_STORE_ID>/records/SUMMARY?token=YOUR_APIFY_TOKEN

Schedule the Actor daily or weekly, set a performanceThreshold, and post rows where passedThreshold is false to Slack, Sheets or a webhook from a CI pipeline.

Pricing, in practice

The Actor is pay-per-event. You are charged for audits that complete, and platform usage is included in that price. Pages that cannot load are reported with an error field and are not charged.

A 100-page site on mobile is about $1.00, or $2.00 for mobile plus desktop, plus the $0.01 start charge. With your own Google key it is $0.30 per 100 pages. If you set Runs per URL to 3, you pay for three audits per page. The $5 free-plan credit covers hundreds of local audits, or more if you switch to the PSI engine. Set Max URLs or a max cost per run if you are exploring.

Tips that save time

• No key is required. The default Auto engine runs local Lighthouse when no Google API key is set, so there is no PSI quota to run out of. Add a key only if you want the cheaper PSI engine and real-user CrUX field data (includeFieldData defaults to on once a key is present; it falls back from URL-level to origin-level data).
• To get a free Google API key, open Google Cloud Console, enable the PageSpeed Insights API and the Chrome UX Report API, and create an API key. Paste it into Google API key.
• Lab scores vary slightly between machines. They come from a simulated Moto G Power on slow 4G. Use runsPerUrl of 3 or more when you need precise numbers, and remember that each run is billed.
• Keep maxConcurrency at 1 for the most accurate local scores (CPU contention skews metrics). Use 2–4 with 4 GB or more of memory when you need speed. The PSI engine can go higher, up to 10.
• Fewer Lighthouse categories is faster. The default is all four (performance, accessibility, best-practices, seo). You can drop the ones you do not need.
• Leave HTML reports off for routine monitoring. Turn Save full HTML Lighthouse reports on when you want a shareable reportUrl for a specific page (local engine only).
• maxUrls is per device. 50 URLs on mobile and desktop is 100 audits.
• Failed loads are free. DNS errors, interstitials and pages that never paint are stored with an error and are not billed. The default timeout is 120 seconds per audit (auditTimeoutSecs).

May I audit any website?

The Actor loads only the pages you give it, the same way a normal browser visit does. Only audit sites you own or are allowed to test.

Further reading

• Bulk Lighthouse Audit on Apify Store
• PageSpeed Insights, which is still the fastest way to inspect a single URL
• Apify API docs
• Apify MCP server, if you want an agent to ask "audit this site's speed" against fresh data

If you also work with Shopify App Store listings or public Luma events, the same publisher has Shopify App Store Scraper and Luma Events Scraper.

Disclosure: I built the Bulk Lighthouse Audit tool this tutorial uses.

If you build, market or invest in Shopify apps, two numbers matter week after week: where you rank for a keyword and what competitors charge. The Shopify App Store shows both, but only one page at a time. There is no official API for search rankings, pricing plans or merchant reviews, and copy-pasting the first page of results into a spreadsheet does not scale past a couple of keywords.

For a one-off check, the manual route is still the right one. Search for subscriptions or reviews, note the organic order, and open a competitor's pricing tab. That takes a few minutes. It falls apart when you want a daily history: "we dropped from position 3 to 7 for upsell on Tuesday, and App X added a cheaper plan."

This tutorial shows a practical way to export those public listings to CSV and to run the same job on a schedule, using the Shopify App Store Scraper on Apify. You can do it in the browser in a couple of minutes, or from Python or curl if you want it to repeat. The Actor only reads pages any visitor can open without logging in.

What you will get

Each app is one record. Useful columns for a ranking or pricing tracker include:

• handle, name and url (use handle as the stable key)
• searchPosition and isSponsored (organic rank; ads are skipped by default)
• rating, reviewCount and, in details mode, ratingDistribution
• pricingSummary, plus pricingPlans (name, price, currency, billing period, features) and planCurrency
• builtForShopify, tagline, launchedDate, languages, worksWith
• developerName, developerWebsite, supportEmail
• Shopify's reviewsAiSummary
• source (which search, category or developer page found the app) and scrapedAt

Merchant reviews are optional. When you turn them on they are nested on the app record (reviewId, rating, date, text, storeName, storeCountry, usageDuration, and any developer reply). The dataset's Reviews view flattens them to one row per review.

Option 1: export from the Apify Console (no code)

1. Create a free Apify account if you do not have one. The free plan includes $5 of monthly credit, which is enough for thousands of apps.
2. Open the Shopify App Store Scraper and click Try for free.
3. Enter a search keyword. The form is prefilled with subscriptions. You can also paste a category URL or slug, a developer page such as https://apps.shopify.com/partners/klaviyo, or specific app URLs.
4. For daily rank tracking, turn Scrape full app details off. That is fast mode: rank, rating, review count, pricing summary and tagline from the listing pages, at the lower price. Leave details on when you want every pricing plan, launch date and integrations.
5. Leave Skip sponsored (ad) results on so searchPosition is organic rank, not a mix of ads and organic results.
6. Leave Scrape reviews off unless you need merchant review text. If you do, set Max reviews per app, Review sort (newest is best for monitoring) and optionally Reviews since or Only these star ratings (for example [1, 2] for complaints).
7. Set Max apps per search/category/developer so a first run stays small. 10–25 is plenty for a test (the form prefills 10).
8. Click Start. About 10 apps with full details usually finish in under a minute.
9. Open the Dataset tab (the Apps view), click Export, and choose CSV (or Excel). Open the file in Google Sheets or Excel.

To watch change over time, open the Actor's Schedules tab and run the same input daily. Keep handle as the join key and compare searchPosition, pricingSummary and, in details mode, pricingPlans.

Example input

Daily organic rank for two keywords, fast mode, first 25 results each:

{
  "searchQueries": ["subscriptions", "upsell"],
  "maxAppsPerSource": 25,
  "scrapeAppDetails": false,
  "excludeSponsored": true,
  "scrapeReviews": false
}

A weekly pricing snapshot: the same keywords, plus one category, with full plan details (no reviews):

{
  "searchQueries": ["subscriptions"],
  "categoryUrls": ["marketing-and-conversion-marketing-email-marketing"],
  "maxAppsPerSource": 50,
  "scrapeAppDetails": true,
  "scrapeReviews": false
}

You can mix sources in one run: searchQueries, categoryUrls, developerUrls and appUrls. Category and developer values can be full URLs or slugs (klaviyo, or https://apps.shopify.com/partners/klaviyo).

Option 2: run it from Python or curl

Install the official client and call the Actor. Replace the token with your Apify API token (Console → Settings → Integrations). Do not commit the token.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("refillflows/shopify-app-store-scraper").call(
    run_input={
        "searchQueries": ["subscriptions"],
        "maxAppsPerSource": 25,
        "scrapeAppDetails": True,
        "scrapeReviews": False,
    }
)

dataset_id = run["defaultDatasetId"]
items = list(client.dataset(dataset_id).iterate_items())
print(f"Got {len(items)} apps")
for app in items[:5]:
    print(app.get("searchPosition"), app["handle"], app.get("pricingSummary"))

The same call with curl, waiting for the dataset and returning CSV:

curl -X POST \
  "https://api.apify.com/v2/acts/refillflows~shopify-app-store-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN&format=csv" \
  -H "Content-Type: application/json" \
  -d '{
    "searchQueries": ["subscriptions"],
    "maxAppsPerSource": 25,
    "scrapeAppDetails": true,
    "scrapeReviews": false
  }'

If you already have a DATASET_ID from a finished run, you can also download it with:

https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv&token=YOUR_APIFY_TOKEN

Schedule the same input daily in Apify Console (or with the API) and send new rows to Google Sheets, Slack or a webhook. Diff yesterday's file against today's on handle. Fast mode is enough to plot searchPosition and to notice a change in the listing's pricingSummary. Use a slower details run when you need every plan's price and feature list.

Pricing, in practice

The Actor is pay-per-event. You are charged for apps and reviews you actually receive, and platform usage is included in that price.

A daily fast-mode run of 25 results for one keyword is a fraction of a cent per day, or roughly $0.25 a month including start charges. A full category of 300 apps with details is about $0.60. 10,000 reviews cost $3. The $5 free-plan credit covers thousands of apps. Set Max apps per search/category/developer, or a max cost per run, if you are exploring.

Tips that save time

• Use fast mode for rank, details mode for pricing. Listing data already includes searchPosition, rating, reviewCount and pricingSummary. Turn Scrape full app details on when you need pricingPlans, launchedDate, worksWith, ratingDistribution or reviewsAiSummary.
• Always store planCurrency. Shopify shows plan prices in the visitor's local currency. On Apify's servers (US) that is USD. If you run the Actor elsewhere, or through a proxy in another country, the numbers will be in that currency. Comparing a USD snapshot to an INR snapshot will look like a price change when it is not.
• handle is the stable key, not the display name. Names and taglines change; the handle in the App Store URL usually does not.
• De-duplication is by handle. If the same app matches both a search keyword and a category in one run, it appears once. The source field tells you which query or page found it first.
• Reviews are optional and cost extra. Leave them off for a rank or pricing tracker. If you want recent 1–2★ feedback for feature ideas, set scrapeReviews: true, reviewRatings: [1, 2], reviewsSort: "newest" and a reviewsSince date. Rating-only reviews (no text) are real and are kept, with text set to null.
• For large crawls (thousands of apps or reviews), enable an Apify datacenter proxy and raise maxConcurrency and maxRequestsPerMinute (defaults are 5 and 120).

Is this allowed?

Shopify does not offer a public App Store API for rankings, plans or reviews. This Actor only reads public App Store pages that any visitor can see without logging in. Review authors are shown as their store names, as Shopify publishes them. You are still responsible for how you use the data, including Shopify's terms and privacy law in your jurisdiction (for example GDPR and CCPA).

Further reading

• Shopify App Store Scraper on Apify Store
• Apify API docs
• Apify MCP server, if you want an agent to ask "what are the top 10 Shopify subscription apps and their cheapest paid plan?" against fresh data

If you also collect public Luma events or need Lighthouse / Core Web Vitals in bulk, the same publisher has Luma Events Scraper and Bulk Lighthouse Audit.

Disclosure: I built the Shopify App Store Scraper this tutorial uses.

Luma does have an official API, but it is aimed at organizers on paid plans and is scoped to calendars you manage. It will not give you "all public AI meetups in Bengaluru next month". Community calendars also offer an iCal subscription, which is perfect when you already know the calendar, and useless for discovering events on calendars you do not follow.

This tutorial shows a practical way to export public Luma events for a city to CSV, using the Luma Events Scraper on Apify. You can do the same job in the browser in a couple of minutes, or from Python if you want it on a schedule. The Actor only reads events that any visitor can see without logging in. It does not collect guest lists or private attendee data.

What you will get

Each row in the CSV is one event, with fields such as:

• Name and Luma URL
• Start and end time, timezone
• Venue, address, city, and coordinates
• Free or paid, price range, sold-out and waitlist flags, remaining spots
• RSVP / guest count
• Organizing calendar and hosts
• A source field that records which city, category or keyword found the event

You can also turn on an optional details mode for the full description and ticket tiers.

Option 1: export from the Apify Console (no code)

1. Create a free Apify account if you do not have one. The free plan includes $5 of monthly credit, which is enough for a few thousand events.
2. Open the Luma Events Scraper and click Try for free.
3. In Locations, enter a city, for example San Francisco, London, or Bengaluru. You can pass "lat,lon" if you prefer. Featured Luma cities resolve immediately; other names are geocoded.
4. Leave Categories and Search keywords empty if you want everything nearby. To narrow the list, add a category such as ai or tech, or a keyword such as hackathon.
5. Optionally set a date window (Start date / End date) and Max events so a first run stays small. 25–100 is plenty for a test.
6. Click Start. A few dozen events usually come back in seconds.
7. Open the Dataset tab, click Export, and choose CSV (or Excel). Open the file in Google Sheets or Excel.

That is the whole one-off workflow. The dataset also has a Location & map view if you want to plot coordinates.

Example input

A week of AI and tech events in two cities, plus one community calendar:

{
  "locations": ["San Francisco", "Bengaluru"],
  "categories": ["ai", "tech"],
  "calendarUrls": ["https://luma.com/genai-sf"],
  "startDate": "2026-11-12",
  "endDate": "2026-11-19",
  "maxItems": 300
}

If you only want one city, all upcoming public events:

{
  "locations": ["San Francisco"],
  "radiusKm": 25,
  "maxItems": 500
}

A 25 km radius around San Francisco is often 500+ upcoming events, so keep maxItems in mind.

Option 2: export to CSV from Python

Install the official client and call the Actor. Replace the token with your Apify API token (Console → Settings → Integrations). Do not commit the token.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("refillflows/luma-events-scraper").call(
    run_input={
        "locations": ["San Francisco"],
        "searchQueries": ["hackathon"],
        "maxItems": 100,
    }
)

# Write a CSV via the dataset API, or iterate in Python
dataset_id = run["defaultDatasetId"]
items = list(client.dataset(dataset_id).iterate_items())
print(f"Got {len(items)} events")
for event in items[:5]:
    print(event["startAt"], event["name"], event["url"])

To download the whole dataset as a CSV file without iterating:

https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv&token=YOUR_APIFY_TOKEN

You can also start a run with curl and then hit that export URL. Schedule the same input daily in Apify Console (or with the API) and send new rows to Google Sheets, Slack or a webhook. Use the eventId field as the stable key when you diff yesterday's file against today's.

Pricing, in practice

The Actor is pay-per-event. You are charged for events you actually receive, and platform usage is included in that price.

A 1,000-event city export is about $1.50. The $5 free-plan credit covers roughly 3,000 events. Set Max events or a run cost limit if you are exploring.

Tips that save time

• De-duplication is built in. If the same event matches both the ai category and a "hackathon" keyword, it appears once. The source field tells you which query found it first.
• Host social links are off by default. Turn them on only if you have a lawful reason to process that data.
• iCal is still better for calendars you already follow. Use the Actor for city and category discovery; subscribe to a calendar's iCal feed when you just want updates from one organizer.
• Luma will rate-limit very large pulls. The Actor retries with backoff. For 10,000+ events or details mode, enable an Apify datacenter proxy.
• This is not a guest-list tool. Public event metadata only.

Is this allowed?

The Actor collects information Luma already shows to any logged-out visitor: title, time, venue, price, public guest count, organizer. It does not access private events or attendee lists. You are still responsible for how you use the data, including Luma's terms and privacy law in your jurisdiction (for example GDPR, if you turn on host social links).

Further reading

• Luma Events Scraper on Apify Store
• Apify API docs
• Apify MCP server, if you want an agent to ask "what is on in this city next week" against fresh data

If you also work with Shopify App Store listings or need Lighthouse / Core Web Vitals in bulk, the same publisher has Shopify App Store Scraper and Bulk Lighthouse Audit.