Static vs. Dynamic Scraping: When You Actually Need a Headless Browser

One of the most common time sinks in web scraping is reaching for a headless browser when a plain HTTP request would have worked fine — or the opposite, spending hours debugging why a simple scraper returns an empty page when the site actually needed JavaScript rendering. Here's how to tell which situation you're in before you write any code.

The quick test

Open the page in your browser, then view the page source (not the "inspect element" DOM view — the actual raw HTML response, usually Ctrl+U or Cmd+Option+U). Search for the data you want.

• If it's there: the server sends fully-rendered HTML. A static HTTP request with requests or curl will get you the same content.
• If it's missing: the page builds itself in the browser using JavaScript after the initial load. You'll need something that executes JavaScript to see the same content a scraper would need.

Static scraping

Static scraping means fetching raw HTML and parsing it directly — no browser involved.

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/blog")
soup = BeautifulSoup(resp.text, "html.parser")

This is fast (a single HTTP round trip), lightweight (no browser process to spin up), and easy to scale — you can run hundreds of these requests concurrently without much memory overhead. Most blogs, documentation sites, and server-rendered e-commerce pages fall into this category.

Dynamic scraping

Dynamic scraping means loading the page in an actual (or headless) browser, waiting for JavaScript to run, and then reading the resulting DOM. Playwright is the most common modern tool for this:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/dashboard")
    page.wait_for_selector(".data-table")
    html = page.content()
    browser.close()

This is necessary for single-page apps, infinite-scroll feeds, sites gated behind client-side auth checks, and anything that fetches its data via background API calls after the initial page load. It's also significantly slower and more resource-intensive — a headless browser instance uses far more CPU and memory than a plain HTTP request, and running dozens of them concurrently requires real infrastructure planning.

A middle ground: reverse-engineering the API call

Before reaching for a full headless browser, check your browser's Network tab while the page loads. Many "dynamic" sites are just calling a JSON API behind the scenes — if you can find and call that endpoint directly, you get the speed of static scraping with the data that would otherwise require a browser. This is often the single highest-leverage move in web scraping, and it's worth checking before writing any browser automation code. See our tool comparison guide for more on picking the right approach per project.

Decision checklist

Situation	Approach
Data visible in page source	Static (requests + parser)
Data loaded via visible JSON API call	Static, call the API directly
Data only appears after JS runs, no visible API call	Dynamic (headless browser)
Site requires login/session state before rendering	Dynamic, or reuse session cookies with a static request
High-volume scraping, cost-sensitive	Prefer static wherever possible

Skipping the decision entirely

If you don't want to maintain both a static scraper and a headless browser fleet for different sites, a scraping API that renders on request — returning clean JSON either way — removes the need to make this call yourself. For Google search results specifically, PrismCrawl handles that rendering and parsing server-side on every search, so you get consistent structured JSON without having to work out which SERP features need a headless browser and which don't.

Frequently asked questions

How do I know if a site is client-side rendered?

View the raw page source (not dev tools' inspected DOM). If your target data isn't in that raw HTML, the page renders it client-side and you'll need a headless browser or the underlying API call.

Is headless browser scraping always slower?

Per-request, yes — spinning up a browser context and waiting for JavaScript execution takes meaningfully longer than a plain HTTP request. At scale, that difference compounds into significant infrastructure cost, which is why finding the underlying API call is almost always worth the extra investigation time.

Can I avoid running my own headless browsers?

Yes — a rendering-capable scraping API runs the browser infrastructure for you and returns structured results. For Google search results specifically, PrismCrawl's API handles rendering server-side, so you get consistent JSON output without managing browser instances yourself.