Pick this one if you like turning messy real-world data into an answer. This is the shape of an enormous amount of professional Python: get the data, clean the data, say something true about the data.

The one-sentence version

A script that reads a listing page, extracts every product, and prints a report on what's there and what it costs.

What it must do

| Capability | Must-have | | --- | --- | | Parse the page into one record per item | yes | | Clean each field into a real type: price to a number, rating to an int | yes | | Skip and report bad rows without crashing | yes | | Summary statistics: count, mean, median, spread | yes | | Group and compare: mean price per rating | yes | | Save the cleaned data as CSV and JSON | yes | | A text chart in the report | nice | | Follow multiple pages | nice |

Before you scrape anything

Scraping puts you on someone else's server, so:

• Check for an API first. If the site publishes one (Module 8), use it. It's faster, more stable, and explicitly allowed.
• Read /robots.txt — example.com/robots.txt — and honour it. Python's urllib.robotparser can check it for you.
• Read the terms of service. Some sites prohibit scraping outright.
• Rate-limit yourself. time.sleep(1) between requests. A loop with no delay is indistinguishable from an attack.
• Identify yourself with a real User-Agent header, including a contact address.
• Cache the page while developing. Download it once to a file and work on the file. Don't re-request a live site every time you fix a typo.
• Never scrape personal data. Names, emails, profiles — that's a legal problem, not a technical one.

Practise on sites that exist for it: books.toscrape.com and quotes.toscrape.com are built for exactly this.

Milestone 1: parse the page

html.parser is in the standard library, so this runs anywhere. The parser is a state machine: handle_starttag notices where you are, handle_data collects text, handle_endtag closes the record.

Notice £ came out as £ — HTMLParser decodes entities for you.

Notice too that Hyperion has no price. Real pages always have rows like that. The parser's job is to report what's there, not to guess; the next step decides what to do about it.

Milestone 2: clean the data

This is the part nobody writes blog posts about, and it's most of the work. Everything the parser produced is a string. Analysis needs numbers.

The pattern to steal: clean() returns both the records and the problems. It doesn't crash on a bad row and it doesn't silently drop one either. You get your data and a list of what didn't make it, which is exactly what you need to decide whether the scrape is trustworthy.

Each field gets its own small function, so each is separately testable (lesson 9.9) — feed clean_stock an empty string, a weird string, "Out of stock", and check all three.

Prices are floats here because this is read-only analysis. If you were charging anyone, use integer cents — see lesson 10.2.

Milestone 3: analyse and report

statistics is standard library and covers everything you need at this level: mean, median, stdev, mode, quantiles.

Some things worth copying:

• median alongside mean. When they differ, the distribution is skewed. One £200 book drags the mean up and leaves the median where it was.
• n= next to every group average. "5-star books average £11.56" means something different when it's four books versus four hundred.
• A chart made of #. No library, works in any terminal, and it makes the trend visible instantly — here, higher-rated books really are more expensive.
• build_report returns a string. So you can print it, email it, write it to a file, or assert on it in a test.
• Both CSV and JSON. CSV opens in a spreadsheet; JSON keeps the types for the next script.

The real version

Once you're off the canned page, two libraries do the heavy lifting:

import time
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "shelf-report/1.0 (me@example.com)"}


def fetch(url):
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    return response.text


def parse(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []
    for card in soup.select("article.product_pod"):
        products.append(
            {
                "title": card.h3.a["title"],
                "price": card.select_one("p.price_color").get_text(),
                "rating": card.select_one("p.star-rating")["class"][1],
                "stock": card.select_one("p.instock").get_text(),
            }
        )
    return products


def scrape_all(pages):
    rows = []
    for page in range(1, pages + 1):
        url = f"https://books.toscrape.com/catalogue/page-{page}.html"
        rows.extend(parse(fetch(url)))
        time.sleep(1)  # be a good citizen
    return rows

pip install requests beautifulsoup4 (lesson 7.4), inside a virtual environment (lesson 7.5).

soup.select("article.product_pod") is a CSS selector — the same syntax you'd use in a stylesheet. Right-click an element in your browser, Inspect, and read the classes off it. That's the whole workflow.

This needs a network connection, so it can't run in your browser. Everything else in this lesson does — the canned page is there so you can build and test the parsing and analysis without hitting a server at all. That's also how you should develop the real one.

When the page changes

It will. Scrapers break because scraping depends on someone else's HTML, which they change without telling you. Defend yourself:

• Fail loudly. If a page yields 0 products, raise. Silence means you'll find out three weeks of empty reports later.
• Sanity-check the count. Expect 20 per page, got 3? Something changed.
• Keep a saved copy of the HTML as a test fixture — that's how you can test your parser without the network, exactly like the canned page above.

Definition of done

• [ ] Parsing, cleaning, and analysis are three separate functions
• [ ] Bad rows are reported, not crashed on and not silently dropped
• [ ] The report shows count, mean, median, spread, and at least one grouped comparison
• [ ] Cleaned data is written as both CSV and JSON
• [ ] Tests cover the cleaning functions and the parser, using a saved HTML fixture
• [ ] The script says something true and interesting — a number you didn't know before

Stretch goals

1. Multiple pages, with time.sleep(1) between requests.
2. Track over time. Run daily, append to a CSV with the date, and report price changes.
3. pandas. df = pd.DataFrame(records), then df.groupby("rating").price.mean() — one line for the whole grouped analysis.
4. A real chart with matplotlib: plt.bar(...) and plt.savefig("prices.png").
5. argparse for the URL, page count, and output folder.