Static vs. Dynamic Scraping: When You Actually Need a Headless Browser

One of the most common time sinks in web scraping is reaching for a headless browser when a plain HTTP request would have worked fine. The opposite happens too: spending hours debugging why a simple scraper returns an empty page when the site actually needed JavaScript rendering. Here's how to tell which situation you're in before you write any code.

The quick test

Open the page in your browser, then view the page source (the raw HTML response, usually Ctrl+U or Cmd+Option+U, rather than the "inspect element" DOM view). Search for the data you want.

• If it's there: the server sends fully-rendered HTML. A static HTTP request with requests or curl will get you the same content.
• If it's missing: the page builds itself in the browser using JavaScript after the initial load. You'll need something that executes JavaScript to see the same content a scraper would need.

Static scraping

Static scraping means fetching raw HTML and parsing it directly, with no browser involved.

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/blog")
soup = BeautifulSoup(resp.text, "html.parser")

This is fast (a single HTTP round trip), lightweight (no browser process to spin up), and easy to scale. You can run hundreds of these requests concurrently without much memory overhead. Most blogs, documentation sites, and server-rendered e-commerce pages fall into this category.

Dynamic scraping

Dynamic scraping means loading the page in an actual (or headless) browser, waiting for JavaScript to run, and then reading the resulting DOM. Playwright is the most common modern tool for this:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/dashboard")
    page.wait_for_selector(".data-table")
    html = page.content()
    browser.close()

This is necessary for single-page apps, infinite-scroll feeds, sites gated behind client-side auth checks, and anything that fetches its data via background API calls after the initial page load. It's also significantly slower and more resource-intensive. A headless browser instance uses far more CPU and memory than a plain HTTP request, and running dozens of them concurrently requires real infrastructure planning.

A middle ground: reverse-engineering the API call

Before reaching for a full headless browser, check your browser's Network tab while the page loads. Many "dynamic" sites are just calling a JSON API behind the scenes. If you can find and call that endpoint directly, you get the speed of static scraping with the data that would otherwise require a browser. It's often the most useful trick in web scraping, so check for it before writing any browser automation code. See our tool comparison guide for more on picking the right approach per project.

Decision checklist

Skipping the decision entirely

A managed data API can remove the need to operate your own collection stack for supported sources. PrismCrawl provides structured search, product, place, travel, and review data, with automatic CAPTCHA solving and JavaScript challenge handling. Customers receive the documented response without seeing or interacting with those challenges. This is an API for supported data sources, not an arbitrary-URL browser service.

Frequently asked questions

How do I know if a site is client-side rendered?

View the raw page source (not dev tools' inspected DOM). If your target data isn't in that raw HTML, the page renders it client-side and you'll need a headless browser or the underlying API call.

Is headless browser scraping always slower?

Per-request, yes. Spinning up a browser context and waiting for JavaScript execution takes meaningfully longer than a plain HTTP request. At scale, that difference compounds into significant infrastructure cost, which is why finding the underlying API call is almost always worth the extra investigation time.

Can I avoid running my own headless browsers?

Yes. A managed data API can return structured results for the sources it supports. PrismCrawl covers search engines, shopping, maps, travel, reviews, and app stores, with automatic CAPTCHA and JavaScript challenge solving and no browser infrastructure for you to operate.