接着上篇文章（Python爬虫项目实战-使用常用库爬取豆瓣读书 Top 250），我们把它升级成一个“麻雀虽小，五脏俱全”的教学级工程：

基于上个豆瓣读书爬虫项目，同时集成四大进阶方向，并且保持结构清晰、可运行、可扩展。

⚠️ 再次提醒：仅供学习与课程演示，请勿高频请求豆瓣。

项目名称

DoubanBookSpider-Pro

一个集Scrapy 工程化 + 分布式去重 + 反爬策略 + 异步高性能​ 于一体的豆瓣读书爬虫

一、项目结构（重点）

douban_book_spider_pro/ ├── scrapy.cfg ├── requirements.txt ├── Dockerfile ├── docker-compose.yml └── douban_book_spider_pro/ ├── __init__.py ├── settings.py ├── pipelines.py ├── middlewares.py ├── items.py ├── db.py └── spiders/ ├── __init__.py ├── top250_spider.py # Scrapy 主爬虫 ├── async_spider.py # aiohttp 异步爬虫 ├── distributed_spider.py # Redis 分布式爬虫 └── anti_spider_demo.py # 反爬策略演示

二、依赖清单（requirements.txt）

scrapy>=2.11 redis>=5.0 fake-useragent requests beautifulsoup4 lxml pyquery selenium playwright aiohttp aioredis

pip install -r requirements.txt playwright install

三、Item 定义（items.py）

import scrapy class BookItem(scrapy.Item): title = scrapy.Field() author = scrapy.Field() publisher = scrapy.Field() rating = scrapy.Field() detail_url = scrapy.Field()

四、数据库封装（db.py）

import redis REDIS_HOST = "localhost" REDIS_PORT = 6379 def get_redis(): return redis.Redis(host=REDIS_HOST, port=REDIS_PORT, decode_responses=True)

五、方向一：Scrapy 工程化（top250_spider.py）

import scrapy from douban_book_spider_pro.items import BookItem class Top250Spider(scrapy.Spider): name = "top250" allowed_domains = ["book.douban.com"] start_urls = ["https://book.douban.com/top250"] def parse(self, response): for item in response.css(".item"): book = BookItem() book["title"] = item.css(".title a::attr(title)").get() book["author"] = item.css(".author::text").get(default="").strip() book["rating"] = item.css(".rating_nums::text").get() book["detail_url"] = item.css(".title a::attr(href)").get() yield book next_page = response.css(".next a::attr(href)").get() if next_page: yield response.follow(next_page, self.parse)

✅ 体现：

• Spider 规范化

• Item 封装

• Pipeline 可扩展

• 自动翻页

六、方向二：分布式去重（distributed_spider.py）

from scrapy_redis.spiders import RedisSpider from douban_book_spider_pro.items import BookItem class DistributedSpider(RedisSpider): name = "distributed" redis_key = "douban:start_urls" def parse(self, response): for item in response.css(".item"): book = BookItem() book["title"] = item.css(".title a::attr(title)").get() book["rating"] = item.css(".rating_nums::text").get() yield book

settings.py 关键配置

SCHEDULER = "scrapy_redis.scheduler.Scheduler" DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" REDIS_HOST = "localhost" REDIS_PORT = 6379

启动多个爬虫实例即可实现横向扩展。

七、方向三：反爬策略（middlewares.py）

from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware from fake_useragent import UserAgent import random class RandomUserAgentMiddleware(UserAgentMiddleware): def __init__(self, *args, **kwargs): self.ua = UserAgent() def process_request(self, request, spider): request.headers["User-Agent"] = self.ua.random class RandomDelayMiddleware: def process_request(self, request, spider): import time time.sleep(random.uniform(0.5, 1.5))

settings.py 启用

DOWNLOADER_MIDDLEWARES = { 'douban_book_spider_pro.middlewares.RandomUserAgentMiddleware': 400, 'douban_book_spider_pro.middlewares.RandomDelayMiddleware': 500, }

✅ 包含：

• UA 随机化

• 请求间隔

• 可扩展代理池（省略示例）

八、方向四：异步高性能（async_spider.py）

import aiohttp import asyncio from bs4 import BeautifulSoup from douban_book_spider_pro.items import BookItem URL = "https://book.douban.com/top250" async def fetch(session, url): async with session.get(url) as resp: return await resp.text() async def parse_html(html): soup = BeautifulSoup(html, "lxml") for item in soup.select(".item"): book = BookItem() book["title"] = item.select_one(".title a")["title"] book["rating"] = item.select_one(".rating_nums").text print(book) async def main(): async with aiohttp.ClientSession() as session: html = await fetch(session, URL) await parse_html(html) if __name__ == "__main__": asyncio.run(main())

✅ 特点：

• 非阻塞 IO

• 高并发

• 适合大规模抓取

九、Docker 化部署（Dockerfile）

FROM python:3.10-slim WORKDIR /app COPY . . RUN pip install -r requirements.txt && playwright install --with-deps CMD ["scrapy", "crawl", "top250"]

# docker-compose.yml version: "3" services: spider: build: . depends_on: - redis redis: image: redis:7

十、整体数据流总结

aiohttp / Scrapy ↓ UA + 延迟 + 代理 ↓ Redis 去重 ↓ BookItem ↓ Pipeline ↓ JSON / DB

通过这个项目，你了解了：

✅ Scrapy 工程化架构

✅ Redis 分布式爬虫

✅ 常见反爬策略

✅ 异步高性能爬虫

✅ Docker 化部署思路

下一步你可以做什么？

1. ✅ 把Playwright / Selenium​ 无缝接入 Scrapy Downloader Middleware

2. ✅ 把数据写入MySQL / MongoDB / Elasticsearch​

3. ✅ 加一个前端可视化（Flask + ECharts）​

4. ✅ 讲清楚Scrapy vs aiohttp 性能对比与选型