docs: Add Migrating from Scrapy guide - #2013
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a dedicated “Migrating from Scrapy” documentation guide to help Scrapy users translate core concepts and common patterns (routing, data passing, throttling, proxies, error handling, login, and JS rendering) into Crawlee for Python.
Changes:
- Introduces a new
docs/guides/scrapy_migration.mdxguide with a concept mapping table and side-by-side Scrapy vs Crawlee examples. - Adds Scrapy and Crawlee code snippets under
docs/guides/code_examples/scrapy_migration/to support the guide’s runnable and comparison blocks. - Updates Ruff per-file ignores for the Scrapy snippet files used purely for comparison.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Adds Ruff per-file ignores for Scrapy tutorial snippets included as docs-only comparison code. |
| docs/guides/scrapy_migration.mdx | New migration guide content with concept mapping and side-by-side examples. |
| docs/guides/code_examples/scrapy_migration/scrapy_throttling.py | Scrapy AutoThrottle settings snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_quotes.py | Scrapy quotes spider snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_proxy.py | Scrapy rotating proxies/settings snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_playwright.py | Scrapy + scrapy-playwright spider snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_playwright_settings.py | scrapy-playwright settings snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_formrequest.py | Scrapy FormRequest login snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_export.py | Scrapy FEEDS export settings snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_errback.py | Scrapy errback/retry behavior snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_crawlspider.py | Scrapy CrawlSpider/Rules snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_concurrency.py | Scrapy concurrency/download delay settings snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_cb_kwargs.py | Scrapy cb_kwargs snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/scrapy_authors.py | Scrapy callback routing snippet for comparison. |
| docs/guides/code_examples/scrapy_migration/crawlee_user_data.py | Crawlee equivalent showing Request.user_data + labeled handlers. |
| docs/guides/code_examples/scrapy_migration/crawlee_throttling.py | Crawlee equivalent showing ThrottlingRequestManager usage. |
| docs/guides/code_examples/scrapy_migration/crawlee_quotes.py | Crawlee equivalent of the canonical quotes spider. |
| docs/guides/code_examples/scrapy_migration/crawlee_proxy.py | Crawlee equivalent proxy rotation via ProxyConfiguration. |
| docs/guides/code_examples/scrapy_migration/crawlee_post.py | Crawlee equivalent of form submission/login with POST payload + headers. |
| docs/guides/code_examples/scrapy_migration/crawlee_playwright.py | Crawlee PlaywrightCrawler example for JS-rendered pages. |
| docs/guides/code_examples/scrapy_migration/crawlee_labels.py | Crawlee equivalent of callback routing using request labels and router handlers. |
| docs/guides/code_examples/scrapy_migration/crawlee_export.py | Crawlee equivalent dataset export via export_data(). |
| docs/guides/code_examples/scrapy_migration/crawlee_error_handling.py | Crawlee equivalent error + failed-request handlers aligned with retry lifecycle. |
| docs/guides/code_examples/scrapy_migration/crawlee_crawlspider.py | Crawlee equivalent rule-based crawling via enqueue_links() with include globs and labels. |
| docs/guides/code_examples/scrapy_migration/crawlee_concurrency.py | Crawlee equivalent concurrency/rate settings via ConcurrencySettings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
d9cd4ad to
d21b213
Compare
vdusek
left a comment
vdusek
left a comment
There was a problem hiding this comment.
Just a few comments, but overall, this looks great 👍
|
@honzajavorek, would you mind taking a look at this guide? I'd appreciate your feedback. |
Pijukatel
left a comment
Pijukatel
left a comment
There was a problem hiding this comment.
It reads well for someone who has never used Scrapy
|
@Mantisus I'll take a look, thanks for the ping! But don't block on my review. Even if it's merged, I can review it ex-post and note down improvements we could make (if any). |
Alright, let's merge it, thanks everyone 🙂 |
honzajavorek
left a comment
honzajavorek
left a comment
There was a problem hiding this comment.
Trying and understanding Crawlee would be so much better for me if I had this tutorial at hand! Great job! I posted few comments and ideas for further improvement of the document, but I think there's nothing super serious. I didn't check the code itself much, I focused mainly on the text of the document.
I like asyncio more than Twisted, and I don't like classes, but the Scrapy code examples still often feel shorter and easier to my eyes. Not even mentioning the form example, which is just "well do it all yourself, pal". But that's up to designers of the framework to address… 😅 😛 But these days nobody reads the code anyway, it's all just agents, they say, so shorter or longer, I guess it doesn't matter.
|
|
||
| Both frameworks give you a request scheduler, a duplicate filter, retries, and a way to pull data out of pages. The way you wire those pieces together differs. | ||
|
|
||
| - Scrapy runs on the [Twisted](https://twisted.org/) reactor, and your callbacks are synchronous generators that `yield` items and requests. Crawlee runs on `asyncio`, and handlers are coroutines defined with `async def` that `await` helpers like <ApiLink to="class/PushDataFunction">`push_data`</ApiLink> and <ApiLink to="class/EnqueueLinksFunction">`enqueue_links`</ApiLink>. You write straight-line `async`/`await` code without a reactor. |
There was a problem hiding this comment.
I know this difference is the most fundamental, but as a reader considering a switch who doesn't really care about deep technicalities like Twisted reactors, this is quite heavyweight start. I think casual Python users might know that asyncio exists, but the first sentence containing words like reactor, callbacks, synchronous generators, and yield, might be too much for many. I'd at least move this bullet point down the order and start with something more high level and straightforward, like the scrapy crawl, for example.
|
@honzajavorek, thank you for reading the guide and for your suggestions on how to improve it. I'm glad you found it helpful! I opened a new PR in which I implemented your suggestions: #2103
Yeah, Crawlee may seem unusual to those who have worked a lot with Scrapy.
In this regard, I find Crawlee's approach more familiar and convenient 🙂. I hope this won't scare off any Scrapy users who might decide to try our framework. 😈
🙈 |
### Description - Update and polish the `Migrating from Scrapy` guide ### Issues - Follow-up: #2013
|
Thanks! I'll take a look at the PR 🚀 I don't think the form part will scare off anyone, it's just like… coming from Scrapy a thought comes to one's head: "folks, how hard is it to add one helper which does it for me, when it's given what should I do and it's always the same?" I don't remember the mime type from the top of my head, and I don't remember the stdlib import from the top of my head. And if someone needs something special, they can always skip the helper and just do exactly this, but for the most of us mere mortals, we'd have an out of the box tool. |
Description
Issues