Scraping Google Search Results with Python: A Step-by-Step Tutorial

Google search results are one of the most useful and most heavily defended datasets on the web. This tutorial walks through the direct approach with Python, where it breaks down at scale, and the API-based alternative that avoids those problems.

If you are deciding between maintaining this scraper and using an API, start with the commercial-intent guide to scraping Google search results programmatically. For a working API integration instead, use the Google Search API Python guide. The Google Search API overview explains targeting and output, while the SERP API hub covers the wider API catalog and production capabilities.

The direct approach

At a small scale, you can request a search results page and parse the HTML directly:

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
query = "best python web scraping libraries"
resp = requests.get(f"https://www.google.com/search?q={query}", headers=headers)

soup = BeautifulSoup(resp.text, "html.parser")
for result in soup.select("div.g"):
    title_el = result.select_one("h3")
    link_el = result.select_one("a")
    if title_el and link_el:
        print(title_el.get_text(), link_el["href"])

This works briefly and inconsistently. A few practical problems show up quickly:

Why this breaks down

• CSS classes change without notice. Google's HTML structure and class names (like div.g above) shift periodically, silently breaking any selector-based parser until you update it.
• CAPTCHAs appear fast. Google is aggressive about rate-limiting automated search traffic, often after single-digit requests in a short window from one IP.
• Layout varies by query type. Featured snippets, "People also ask," local packs, and shopping results all have different HTML structures, so a parser built for plain organic results misses or mishandles them.
• Geographic and personalization differences. Results vary by location, language, and even account state, which is hard to control for consistently with a raw request.

Scaling this reliably

Handling all of the above yourself means: a proxy rotation system (see our rotating proxies guide), a CAPTCHA fallback strategy (see CAPTCHA solving approaches), and an HTML parser you maintain and update every time Google changes its markup. That's a substantial, ongoing engineering commitment for something that's usually a small part of a larger project.

Using a SERP API instead

A SERP API is a service purpose-built for this problem. It handles the request, proxy rotation, and parsing, and returns already-structured JSON. With PrismCrawl's official Python SDK (pip install prismcrawl), the whole scraper above becomes a few lines:

from prismcrawl import PrismCrawl

client = PrismCrawl()  # reads PRISMCRAWL_API_KEY from the environment

response = client.google.search(query="best python web scraping libraries", gl="us")

for item in response["data"]["content"]["results"]:
    print(item["rank"], item["title"], item["url"])

The SDK retries rate limits and transient errors for you. If you'd rather call the REST endpoint directly with requests, the Google Search API Python guide has a complete example, and PrismCrawl's interactive API reference documents the full response schema. The same SDK covers maps, reviews, and app stores too: see how to scrape Google Maps, Yelp, and Tripadvisor reviews with Python.

The output is consistent JSON regardless of layout changes on Google's end, since parsing happens server-side and is maintained centrally rather than by every individual scraper.

Choosing between the two approaches

For a one-off script checking a handful of queries, direct scraping is fine. For anything recurring, like rank tracking, competitive research, or feeding an application, the maintenance cost of direct scraping tends to exceed the cost of a SERP API fairly quickly. See our full SERP API comparison for how PrismCrawl's pricing stacks up against alternatives.

Frequently asked questions

Can I scrape Google search results without getting blocked?

At low volume, often yes for a while. At any meaningful, recurring volume, you'll need proxy rotation and CAPTCHA handling, which is exactly what a SERP API manages for you.

Is scraping Google search results legal?

Search results are publicly visible without a login, which places it on the lower-risk end of the spectrum discussed in our web scraping legality guide, but always review Google's terms of service and consider your specific use case.

How do I get started with a SERP API?

PrismCrawl gives you 100 free credits with no credit card required, so you can test the response format against real queries before committing to a paid plan.