Web Scraping with Python: A Beginner's Guide to Your First Scraper

Python is the most common language for web scraping, mostly because two libraries — requests and BeautifulSoup — get you from "empty file" to "working scraper" in about fifteen lines of code. This guide walks through building a small scraper from scratch: fetching a page, parsing it, pulling out the data you actually want, and handling pagination so you're not stuck on page one.

What you'll need

• Python 3.9 or newer
• The requests library, for making HTTP requests
• beautifulsoup4, for parsing HTML

Install both with pip:

pip install requests beautifulsoup4

Step 1: Fetch the page

Every scraper starts the same way — you request a URL and get back raw HTML.

import requests

url = "https://example.com/products"
response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
print(response.status_code)  # 200 means success
html = response.text

Setting a User-Agent header matters even at this beginner stage — some servers reject requests that don't look like they're coming from a browser.

Step 2: Parse the HTML

Raw HTML is not useful on its own. BeautifulSoup turns it into a tree you can search with CSS selectors or tag names.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
titles = soup.select("h2.product-title")

for title in titles:
    print(title.get_text(strip=True))

soup.select() accepts standard CSS selectors, which makes it easy to target exactly the elements you want once you've inspected the page in your browser's dev tools.

Step 3: Extract structured data

Most real scraping tasks need more than one field per item — a product name, price, and link, for example. Loop over a shared parent element instead of each field separately:

products = []
for card in soup.select(".product-card"):
    products.append({
        "title": card.select_one(".product-title").get_text(strip=True),
        "price": card.select_one(".price").get_text(strip=True),
        "url": card.select_one("a")["href"],
    })

Step 4: Handle pagination

Most sites split results across multiple pages. A simple loop with a page counter handles this for URL-based pagination:

all_products = []
page = 1

while True:
    resp = requests.get(f"{url}?page={page}", headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(resp.text, "html.parser")
    cards = soup.select(".product-card")
    if not cards:
        break
    all_products.extend(cards)
    page += 1

Common beginner mistakes

• Not checking response.status_code. A 403 or 429 response still returns a body — often an error page — and parsing it silently produces empty results instead of an obvious error.
• Scraping too fast. Add a small delay (time.sleep(1)) between requests so you don't hammer the target server.
• Ignoring robots.txt and terms of service. Always check what a site allows before scraping it at scale — see our guide on whether web scraping is legal for more detail.
• Not planning for blocks. Sites that don't want to be scraped will eventually rate-limit or block your IP. That's normal, and it's the main reason dedicated scraping APIs exist — they handle proxy rotation, retries, and CAPTCHAs so a request keeps working without you having to build that infrastructure yourself. If Google search results specifically are part of what you're after, PrismCrawl is one example built just for that.

When to move beyond requests + BeautifulSoup

This stack is great for static HTML pages. Once a site loads its content with JavaScript, or starts blocking your requests, you'll need a headless browser or a dedicated scraping API — see our comparison of static vs. dynamic scraping for how to decide which approach fits your project.

Frequently asked questions

Is requests + BeautifulSoup enough for all web scraping?

No — this combination only works for static HTML, where the data you want is present in the initial page source. Sites that render content with JavaScript (many modern web apps) require a headless browser like Playwright, or an API that renders pages for you.

How do I avoid getting blocked as a beginner?

Set a realistic User-Agent, add delays between requests, respect robots.txt, and avoid hitting the same site with a high volume of concurrent requests. For anything beyond small personal projects, a managed scraping API handles IP rotation and blocking automatically — see our full guide on avoiding blocks while scraping.

What's the fastest way to get started without managing infrastructure?

If Google search results are part of what you need rather than a specific site's HTML, PrismCrawl's API returns them as structured JSON directly, with 25 free credits to test on signup — no credit card required.