ποΈ Web Crawler Intro
What is a Web Crawler?
ποΈ HTML Basic
Quick View
ποΈ Key Words
- Headers
ποΈ What is a REST API?
A REST API is like a waiter at a restaurant. You tell it what you want (make a request), and it brings you what you asked for (the response). Itβs how apps talk to each other.
ποΈ Web Crawler Implementation
Common Libraries
ποΈ Web Crawler for Static Web Page
What to Do
ποΈ Web Crawler with browser_cookie3
This guide demonstrates how to use browser_cookie3 to retrieve cookies from your browser and use them in a web crawler to access the PTT Gossiping board, which requires a over18=1 cookie to bypass the age restriction prompt.
ποΈ Selenium Guide: Automating Browser Interaction
Since Selenium 4.6, developers no longer need to manually download ChromeDriver. Selenium automatically detects the local browser version and downloads the appropriate driver.
ποΈ Scrapy Guide
Scrapy is a Python framework for building web crawlers. Unlike requests + BeautifulSoup, Scrapy handles everything β sending requests, parsing HTML, following links, and saving data β all in one structured project.
ποΈ Scrapy Pipeline
The Item Pipeline processes and saves data after the Spider yields it. Each item passes through the pipeline stages you define.