Scraping Google Search Results with Python: A Step-by-Step Tutorial

Google search results are one of the most useful — and most defended — datasets on the web. This tutorial walks through the direct approach with Python, where it breaks down at scale, and the API-based alternative that avoids those problems.

The direct approach

At a small scale, you can request a search results page and parse the HTML directly:

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
query = "best python web scraping libraries"
resp = requests.get(f"https://www.google.com/search?q={query}", headers=headers)

soup = BeautifulSoup(resp.text, "html.parser")
for result in soup.select("div.g"):
    title_el = result.select_one("h3")
    link_el = result.select_one("a")
    if title_el and link_el:
        print(title_el.get_text(), link_el["href"])

This works — briefly, and inconsistently. A few practical problems show up quickly:

Why this breaks down

• CSS classes change without notice. Google's HTML structure and class names (like div.g above) shift periodically, silently breaking any selector-based parser until you update it.
• CAPTCHAs appear fast. Google is aggressive about rate-limiting automated search traffic — often after single-digit requests in a short window from one IP.
• Layout varies by query type. Featured snippets, "People also ask," local packs, and shopping results all have different HTML structures, so a parser built for plain organic results misses or mishandles them.
• Geographic and personalization differences. Results vary by location, language, and even account state, which is hard to control for consistently with a raw request.

Scaling this reliably

Handling all of the above yourself means: a proxy rotation system (see our rotating proxies guide), a CAPTCHA fallback strategy (see CAPTCHA solving approaches), and an HTML parser you maintain and update every time Google changes its markup. That's a substantial, ongoing engineering commitment for something that's usually a small part of a larger project.

Using a SERP API instead

A SERP API is a service purpose-built for this exact problem — it handles the request, proxy rotation, and parsing, and returns already-structured JSON:

import requests

response = requests.post(
    "https://YOUR_PRISMCRAWL_BASE_URL/v1/search",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"query": "best python web scraping libraries", "html": False},
)
results = response.json()

for item in results["results"]:
    print(item["title"], item["link"])

(See PrismCrawl's interactive API reference for your exact base URL, the full response schema, and a live request you can run with your own key.)

The output is consistent JSON regardless of layout changes on Google's end, since parsing happens server-side and is maintained centrally rather than by every individual scraper.

Choosing between the two approaches

	Direct scraping	SERP API
Setup time	Low initially	Low
Maintenance	Ongoing (selectors break)	None on your end
Handles CAPTCHAs/blocks	You build it	Included
Cost at scale	"Free" but engineering time	Per-request, starting at $0.15/1k
Reliability	Degrades over time	Consistent

For a one-off script checking a handful of queries, direct scraping is fine. For anything recurring — rank tracking, competitive research, or feeding an application — the maintenance cost of direct scraping tends to exceed the cost of a SERP API fairly quickly. See our full SERP API comparison for how PrismCrawl's pricing stacks up against alternatives.

Frequently asked questions

Can I scrape Google search results without getting blocked?

At low volume, often yes for a while. At any meaningful, recurring volume, you'll need proxy rotation and CAPTCHA handling — which is exactly what a SERP API manages for you.

Is scraping Google search results legal?

Search results are publicly visible without a login, which places it on the lower-risk end of the spectrum discussed in our web scraping legality guide — but always review Google's terms of service and consider your specific use case.

How do I get started with a SERP API?

PrismCrawl gives you 25 free credits with no credit card required, so you can test the response format against real queries before committing to a paid plan.