Python Web Scrapper Project Syllabus

What does the HTTP status code 429 signal during scraping?

$429$ 'Too Many Requests' signals you have exceeded the server's rate limit; you should slow down, add delays, and respect any Retry-After header.

Why set a custom User-Agent header when sending requests?

Many servers block or alter responses for the default Python/requests User-Agent; setting a realistic browser User-Agent reduces the chance of being blocked and is more transparent.

How do you pass custom headers with a requests GET call?

Provide a dictionary to the headers parameter, e.g. $requests.get(url,\ headers={'User\text{-}Agent':\ '...'})$.

What is the difference between an HTTP GET and POST request in scraping?

GET retrieves data and encodes parameters in the URL query string; POST submits data in the request body, used for forms, logins, and search submissions.

How do you create a BeautifulSoup object from a response?

$soup = BeautifulSoup(response.text,\ 'html.parser')$, passing the HTML string and the parser name.

Name three parsers BeautifulSoup can use and a trade-off between them.

'html.parser' (built-in, no install, moderate speed), 'lxml' (very fast, lenient, needs install), and 'html5lib' (most lenient/browser-like, but slowest).

What is the difference between BeautifulSoup's find() and find_all() methods?

$find()$ returns the first matching element (or None); $find\_all()$ returns a list of all matching elements.

How do you select elements by CSS selector in BeautifulSoup?

Use $soup.select(selector)$ for a list of matches or $soup.select\_one(selector)$ for the first match, passing a CSS selector string such as 'div.price > span'.

How do you extract the visible text from a BeautifulSoup element?

Use the element's $.get\_text()$ method (or the $.text$ attribute), often with $.strip()$ to remove surrounding whitespace.

How do you extract an attribute value, such as a link's href, from an element?

Index the element like a dictionary, e.g. $element['href']$, or use $element.get('href')$ which returns None if the attribute is absent.

What is the difference between find_all('a') and selecting by class in extraction?

$find\_all('a')$ matches every anchor tag regardless of attributes; selecting by class (e.g. $find\_all('a',\ class\_='nav')$) narrows to only anchors with that CSS class.

Why is XPath sometimes used instead of CSS selectors for extraction?

XPath can navigate the DOM in any direction (including selecting parents and siblings by relationship and matching on text content), giving more expressive queries than CSS selectors; it is used by lxml, Scrapy, and Selenium.

What technique handles scraping data spread across multiple pages?

Pagination handling: loop through page URLs or follow 'next' links, sending a request and extracting data from each page until no further pages remain.

List three common storage formats for scraped data and a typical use for each.

CSV (simple tabular data, spreadsheet-friendly), JSON (nested/hierarchical records, web APIs), and a database such as SQLite/SQL (large datasets needing querying and relationships).

When is CSV a good choice for storing scraped data, and what is its main limitation?

CSV suits flat, tabular, uniform rows and is widely compatible; its limitation is poor support for nested or hierarchical structures and no built-in typing.

When is JSON preferable to CSV for scraped output?

JSON is preferable when records are nested or have variable fields, since it natively represents hierarchical objects and arrays.

When should you store scraped data in a database instead of a flat file?

Use a database for large or continuously growing datasets, when you need efficient querying, indexing, deduplication, or relationships between records.

Which built-in Python module writes CSV files, and which method writes one row?

The csv module; create a $csv.writer$ and call $writerow(row)$ for a single row or $writerows(rows)$ for many. The $csv.DictWriter$ writes from dictionaries.

How do you serialize a Python object to a JSON file?

Use the json module: $json.dump(data,\ file)$ to write to a file object, or $json.dumps(data)$ to produce a JSON string.

Which popular library structures scraped data into tables and exports to many formats?

pandas: load records into a DataFrame, then export with $df.to\_csv()$, $df.to\_json()$, $df.to\_excel()$, or $df.to\_sql()$.

What should you do immediately after writing the first version of a scraper?

Run it on a small sample and check the console for errors and the output for correctness, confirming requests succeed and selectors return the expected data before scaling up.

Which Python construct gracefully handles request failures during a scraper run?

A try/except block around the request and parsing code (catching exceptions such as requests.exceptions.RequestException), allowing the scraper to log and skip failures instead of crashing.

What does data validation of scraped output involve?

Checking that extracted values are present, correctly typed, within expected ranges/formats, free of duplicates, and not missing (no unexpected None/empty fields) before the data is trusted or used.

Give two examples of validation checks you would apply to scraped records.

Verify required fields are non-empty, and confirm numeric fields parse to numbers and dates match an expected format; also check for and remove duplicate rows.

Why might scraped numeric data need cleaning before validation passes?

Values often arrive as strings with currency symbols, commas, or whitespace (e.g. '$1,299'), so they must be stripped and cast to numeric types before range or arithmetic checks.

What tools schedule a Python scraper to run automatically at intervals?

On Linux/macOS, cron jobs; on Windows, Task Scheduler; cross-platform options include the Python 'schedule' library, APScheduler, or cloud schedulers/Airflow.

What does the cron expression $0\ 2\ *\ *\ *$ schedule?

It runs the job at minute 0 of hour 2 every day — i.e. daily at 2:00 AM. The five fields are minute, hour, day-of-month, month, and day-of-week.

What is incremental (delta) scraping in the context of scheduled runs?

Scraping only new or changed records on each scheduled run instead of re-collecting everything, reducing load and storage by tracking what was already captured.

Why does requests alone often fail to scrape dynamic (JavaScript-rendered) content?

requests only downloads the initial HTML returned by the server and does not execute JavaScript, so content injected into the DOM by client-side scripts is absent from $response.text$.

Name two strategies for scraping dynamic content rendered by JavaScript.

(1) Use a headless browser like Selenium or Playwright to render the page and execute its scripts, then read the DOM; (2) call the underlying API/XHR endpoints (often returning JSON) that the page's JavaScript fetches, found via the Network tab.

What polite practices reduce the risk of overloading a server or getting blocked?

Add delays between requests (e.g. $time.sleep()$), respect robots.txt and rate limits, send a real User-Agent, cache results, and avoid excessive concurrent requests.