Scrapy
- Category
- Frameworks and libraries
- Docs outline
- 16 sections
Official docs outline
docs.scrapy.org- Scrapy Tutorial
Creating a project, spider and extracting data.
- Architecture Overview
Engine, scheduler, downloader, spiders, pipelines.
- Spiders
Spider classes, start requests and CrawlSpider.
- Selectors
XPath and CSS selection.
- Items
Structured scraped data.
- Item Loaders
Input and output processors.
- Item Pipeline
Processing and storing items.
- Requests and Responses
Callbacks, meta, cb_kwargs and errbacks.
- Downloader Middleware
Request/response hooks, retry, cookies, proxies.
- Spider Middleware
Hooks on spider input and output.
- Settings
Configuring concurrency, delays and user agent.
- Feed Exports
JSON, CSV and storage backends.
- AutoThrottle and Politeness
Adaptive rate limiting.
- Broad Crawls
Scaling large crawls.
- Common Practices
Running multiple spiders and avoiding bans.
- Extensions and Signals
Hooking into Scrapy.
Each section links to the official page. Titles and summaries by Extra Credit; the docs belong to their authors.
