Web Crawler Intro
What is a Web Crawler?
HTML Basic
Quick View
Key Words
When you open a web page, your browser sends a request to a server. The server sends back a response, such as an HTML page.
Before Writing a Web Crawler
A typical workflow is:
Web Crawler for Static Web Page
What to Do
Web Crawler with Cookies
Cookies let a website remember information between requests. For example, PTT's Gossiping board uses an age-confirmation page before showing its articles.
Web Crawler with browser_cookie3
This guide demonstrates how to use browser_cookie3 to retrieve cookies from your browser and use them in a web crawler to access the PTT Gossiping board, which requires a over18=1 cookie to bypass the age restriction prompt.
What is a REST API?
A REST API is like a waiter at a restaurant. You tell it what you want (make a request), and it brings you what you asked for (the response). It’s how apps talk to each other.
Selenium Guide: Automating Browser Interaction
Since Selenium 4.6, developers no longer need to manually download ChromeDriver. Selenium automatically detects the local browser version and downloads the appropriate driver.
Scrapy Guide
Scrapy is a Python framework for building web crawlers. Unlike requests + BeautifulSoup, Scrapy handles everything (sending requests, parsing HTML, following links, and saving data) all in one structured project.
Scrapy Pipeline
The Item Pipeline processes and saves data after the Spider yields it. Each item passes through the pipeline stages you define.