Sometimes the data you want is on a web page with no API behind it. Scraping means fetching the HTML and pulling the values out of it.

pip install beautifulsoup4

Before you scrape

Scraping touches other people's servers, so there's an etiquette — and sometimes a legal dimension:

• Prefer an API. If one exists, use it. It's faster, more stable, and welcome.
• Check robots.txt (e.g. example.com/robots.txt) and the site's terms of service.
• Rate-limit yourself. Sleep between requests. A tight loop looks like an attack.
• Identify your bot with a real User-Agent and a contact address.
• Never scrape personal data you don't have a right to collect.

Scrapers are also fragile: they depend on page structure, and a redesign breaks them silently. An API is a contract; HTML is not.

BeautifulSoup

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com", timeout=10)
soup = BeautifulSoup(response.text, "html.parser")

print(soup.title.string)
print(soup.find("h1").get_text())

for link in soup.find_all("a"):
    print(link.get("href"))

The core methods:

| Method | Returns | | --- | --- | | soup.find("h1") | the first match, or None | | soup.find_all("a") | a list of every match | | soup.find("div", class_="price") | first match with that CSS class | | soup.select("div.price span") | list, using a CSS selector | | element.get_text() | the text inside | | element.get("href") | an attribute value |

Note class_ with a trailing underscore — class is a reserved Python keyword.

Parsing with the standard library

BeautifulSoup isn't available in the browser, but Python's built-in html.parser is — and the underlying idea is identical. You subclass HTMLParser and react to tags as they're encountered:

from html.parser import HTMLParser

class LinkExtractor(HTMLParser):
    """Collects every href on the page."""

    def __init__(self):
        super().__init__()
        self.links = []

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            for name, value in attrs:
                if name == "href":
                    self.links.append(value)

html = '<p>Hi <a href="/one">One</a> and <a href="/two">Two</a></p>'

parser = LinkExtractor()
parser.feed(html)
print(parser.links)

That's a class with __init__, super(), and an overridden method — Module 5, applied. BeautifulSoup does the same job with a far friendlier interface, which is why it's worth installing.

Extracting text

from html.parser import HTMLParser

class TextExtractor(HTMLParser):
    def __init__(self):
        super().__init__()
        self.parts = []

    def handle_data(self, data):
        text = data.strip()
        if text:
            self.parts.append(text)

parser = TextExtractor()
parser.feed("<div><h1>Title</h1><p>Some <b>bold</b> text</p></div>")
print(parser.parts)
print(" ".join(parser.parts))

Don't parse HTML with regex

You'll be tempted. Don't. HTML nests arbitrarily and regular expressions can't track that — your pattern will work on the example you tested and fail on real pages. Use a parser.

Try it yourself

Build a scraper that pulls structured records out of HTML.

Checkpoint

• Scrape only when there's no API; check robots.txt and the terms first.
• Rate-limit, identify your bot, and never collect personal data you shouldn't.
• BeautifulSoup: find, find_all, select, get_text(), get(attr), and class_.
• html.parser in the standard library does the same job more manually.
• Never parse HTML with regular expressions.
• Scrapers break when pages change — expect maintenance.