Unable to simulate the pull-down action to load products.- Need help

[nanridao2026]

When I used Crawl4AI version 0.9.1 to simulate a pull-down, I initially observed no pull-down behavior in the viewport. I immediately closed the webpage (https://www.bershka.com/es/women/clothes/t-shirts-n4365.html) and used undetected_adapter = UndetectedAdapter().

[SohamKukreti]

Hey @nanridao2026, by pull down do you mean scrolling down the page?

[nanridao2026]

Yes, scroll down the page.

[SohamKukreti]

Hi @nanridao2026, I dug into this on the exact URL you're using and found what's going on. After editing your script a bit, I got the full catalog (198 products) captured. A few separate issues are compounding here:

1. Your scroll JS runs fire-and-forget (the main bug)

Crawl4AI wraps js_code inside its own await (async () => { ... })(). If you pass an IIFE like:

(async () => {
    // scroll loop with awaits
})();

the inner IIFE is started but never awaited — the wrapper returns immediately, and the crawler captures the HTML right away without waiting for your scroll loop. That's why you see no scrolling before the page closes.

Fix: write js_code as plain statements (with a return at the end if you want a value back), not an IIFE.

2. Jumping straight to the bottom doesn't trigger the lazy-load

I verified this live on the Bershka page: a single window.scrollTo(0, document.body.scrollHeight) leaves the product count at 20. The catalog loads in batches of 20 as lazy-load sentinels enter the viewport, so you need incremental scrolling (about one viewport height per step) with waits in between. Batches can also take ~10s to arrive, so the loop needs patience before concluding it's done.

(For reference: there's no inner scroll container on this page — the document itself scrolls — so scrolling the window is correct; it just has to be incremental.)

3. wait_for="js:() => true" waits for nothing

It resolves immediately. The grid is fully client-rendered, so you want to wait for actual product tiles: wait_for='css:[data-qa-anchor="productItem"]'.

4. Drop the user_agent override and enable_stealth when using UndetectedAdapter

Forcing a Windows Chrome 122 UA while the adapter runs a real (different) browser creates a fingerprint mismatch that anti-bot systems specifically look for, and enable_stealth double-patches on top of the adapter. UndetectedAdapter works best on its own.

Working script

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, UndetectedAdapter
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

async def test():
    config = BrowserConfig(headless=False)
    crawler_strategy = AsyncPlaywrightCrawlerStrategy(
        browser_config=config,
        browser_adapter=UndetectedAdapter(),
    )
        let last = count(), stale = 0;
        for (let i = 0; i < 80 && stale < 12; i++) {
            window.scrollBy(0, 900);          // one viewport-height at a time
            await sleep(2000);                // batches can take ~10s to arrive
            const c = count();
            if (c > last) { last = c; stale = 0; }
            else {
                stale += 1;
                window.scrollBy(0, -400);     // jiggle to re-trigger lazy-load
                await sleep(500);
            }
        }
        return last;
    """

    run_config = CrawlerRunConfig(
        js_code=scroll_catalog,
        wait_for='css:[data-qa-anchor="productItem"]',   # grid is client-rendered
        wait_for_timeout=30000,
        page_timeout=240000,                             # the scroll takes a few minutes
        cache_mode=CacheMode.BYPASS,
        delay_before_return_html=2.0,
    )

    async with AsyncWebCrawler(config=config, crawler_strategy=crawler_strategy) as crawler:
        result = await crawler.arun(
            url="https://www.bershka.com/es/women/clothes/t-shirts-n4365.html",
            config=run_config,
        )
        print(f"Success: {result.success}")
        if result.success:
            print(f"Final product count: {result.js_execution_result}")
            with open("test_result.html", "w", encoding="utf-8") as f:
                f.write(result.html)

asyncio.run(test())

Result on my machine:

Success: True
js_execution_result (final product count): {'success': True, 'results': [198]}
Product tiles in captured HTML: 198

Two closing notes:

• scan_full_page=True won't help on this page today: it runs before wait_for (so the client-rendered grid doesn't exist yet when it scrolls) and it exits on the first height plateau instead of waiting out slow batches. Making it handle this case better is on our radar.
• If you want structured product data instead of raw HTML, add an extraction_strategy (e.g. JsonCssExtractionStrategy) to the CrawlerRunConfig — result.extracted_content stays None without one.

[nanridao2026]

Thank you for your detailed explanation, which has resolved a problem that was troubling me.

[SohamKukreti]

No problem!

[nanridao2026]

The previous step extracted the details page link. We then used the UndetectedAdapter and arun_many methods to extract the details content again. However, an anti-bot protection mechanism was detected.

[SohamKukreti]

Can you share the script for the other step with the arun_many() usage? Also can you specify more on what kind of anti-bot protection mechanism is detected? Is it a challenge page or an error message or just result.success=False?

Does your next step run in the same crawl session as the catalog crawl or a fresh run?

These details and the script will help me understand the situation better.
Thanks!

[nanridao2026]

This is the code for the latter part.“ async with AsyncWebCrawler(config=config, crawler_strategy=crawler_strategy) as crawler:
result = await crawler.arun(
url="https://www.bershka.com/es/women/clothes/t-shirts-n4365.html",
config=run_config,
)
print(f"Success: {result.success}")
if result.success:
print(f"Final product count: {result.js_execution_result}")
products = json.loads(result.extracted_content)
links=["https://www.bershka.com/es"+p['link']for p in products][:10]
# print(links)
detail_config = CrawlerRunConfig(
extraction_strategy=JsonCssExtractionStrategy(
schema={
"name": "Product Detail",
"baseSelector": ".product-detail-layout",
"fields": [
{"name": "name", "selector": "h1.product-title","type": "text"},
{"name": "price", "selector": 'span[data-qa-anchor="productItemPrice"]',"type": "text"},
{"name": "description", "selector": ".bsk-drawer__content"，"type": "attribute",
"name": "innerHTML" }, #The result will be in a format similar to this.

...

...

{"name": "images", "selector": ".zoom-modal__swiper-tab img",
"type": "list", "fields": [
{"name": "image",
"type": "attribute",
"attribute": "src"
}]
},
{"name": "colors", "selector": ".color-selector li a",
"type": "list", "fields": [
{"name": "color",
"type": "attribute",
"attribute": "aria-label"
}]
},
{"name": "sizes","selector": ".size-selector span.size-button__label","type": "text"}
]}))

async with AsyncWebCrawler(config=config, crawler_strategy=crawler_strategy) as crawler:
    results = await crawler.arun_many(
        urls= links,
        config=detail_config,
    )
    for result in results:

        if result.success:
            products_info = json.loads(result.extracted_content)
            # print(products_info)
            df = pd.DataFrame(products_info)
            df.to_csv('bershka_products.csv', index=False, encoding='utf-8-sig')”,

The following error message appears after the code runs：
Success: True
Final product count: {'success': True, 'results': [208]}
[INIT].... → Crawl4AI 0.9.1
[FETCH]... ↓ https://www.bershka.com/es/es/camiseta-ajustada-spider-man-c0p228480491.html?colorId=600 | ✓ | ⏱: 6.22s
[SCRAPE].. ◆ https://www.bershka.com/es/es/camiseta-ajustada-spider-man-c0p228480491.html?colorId=600 | ✓ | ⏱: 0.14s
[FETCH]... ↓ https://www.bershka.com/es/es/camiseta-manga-corta-print-the-office-c0p231948402.html?colorId=106 | ✓ | ⏱: 6.36s
[SCRAPE].. ◆ https://www.bershka.com/es/es/camiseta-manga-corta-print-the-office-c0p231948402.html?colorId=106 | ✓ | ⏱: 0.17s
[ANTIBOT]. ℹ https://www.bershka.com...480491.html?colorId=600 | Error: Proxy direct failed: 'type'
[ERROR]... × https://www.bershka.com...480491.html?colorId=600 | Error: Unexpected error in _crawl_web at line 1260 in _extract_item (...venv\Lib\site-packages\crawl4ai\extraction_strategy.py):
Error: 'type'

Code context:
1255 Dict: A dictionary representing the extracted item.
1256 """
1257
1258 item = {}
1259 for field in fields:
1260 → if field["type"] == "computed":
1261 value = self._compute_field(item, field)
1262 else:
1263 value = self._extract_field(element, field)
1264 if value is not None:
1265 item[field["name"]] = value

[SohamKukreti]

Checking it out 👀

[SohamKukreti]

Hello @nanridao2026, Thanks for sharing the second part of the workflow! Good news: you were not blocked by anti-bot protection. The [ANTIBOT] line in your log is misleading (more on that below) — the real failure is a small bug in your extraction schema.

What the log actually shows

Look at the two lines before the error — both detail pages were fetched successfully:

[FETCH]... ↓ https://www.bershka.com/es/es/camiseta-ajustada-spider-man-...  | ✓ | ⏱: 6.22s
[FETCH]... ↓ https://www.bershka.com/es/es/camiseta-manga-corta-...         | ✓ | ⏱: 6.36s

The crash happens after fetching, inside extraction:

Error: 'type'
  at line 1260 in _extract_item (extraction_strategy.py)
  → if field["type"] == "computed":

That's a Python KeyError: 'type' — one of the field dicts in your JsonCssExtractionStrategy schema is missing its "type" key. Internally, the exception bubbles up through our anti-bot retry wrapper, which currently catches any error and logs it as [ANTIBOT] Proxy direct failed: .... That label is a bug on our side (extraction errors have nothing to do with proxies or bot detection) — we'll fix the mislabeling. Sorry it sent you down the wrong path.

The schema bug

Your description field has a duplicate "name" key:

{"name": "description", "selector": ".bsk-drawer__content", "type": "attribute",
 "name": "innerHTML"}   # ← second "name" silently overwrites the first

You almost certainly meant "attribute": "innerHTML". As written, Python turns this into {"name": "innerHTML", "selector": ..., "type": "attribute"} — no "attribute" key, wrong name. Since part of your schema is elided with ..., double-check the hidden fields too: at least one field is missing "type" entirely, which is what triggers the KeyError.

For grabbing HTML content, the cleaner form is:

{"name": "description", "selector": ".bsk-drawer__content", "type": "html"}

Fix the schema and the "anti-bot" error disappears.

Two smaller bugs worth fixing while you're in there

1. Your URLs have a doubled /es/es/ — visible right in your log. The tile links already start with /es, so:

# yours — produces https://www.bershka.com/es/es/...
links = ["https://www.bershka.com/es" + p['link'] for p in products][:10]

# correct
links = ["https://www.bershka.com" + p['link'] for p in products][:10]

The site happens to redirect, but that's an unnecessary redirect (and an extra bot signal) on every request.

2. Your CSV only ever contains the last product. df.to_csv(...) sits inside the for result in results: loop and overwrites the file each iteration. Accumulate first, write once:

all_products = []
for result in results:
    if result.success and result.extracted_content:
        all_products.extend(json.loads(result.extracted_content))
pd.DataFrame(all_products).to_csv('bershka_products.csv', index=False, encoding='utf-8-sig')

About anti-bot on arun_many, for when you scale up

With your current 10 URLs you're fine. When you run all ~200 detail pages, blasting them concurrently from one IP can genuinely trip Akamai. If that happens, slow the batch down with a rate-limited dispatcher, e.g.:

from crawl4ai import SemaphoreDispatcher, RateLimiter

results = await crawler.arun_many(
    urls=links,
    config=detail_config,
    dispatcher=SemaphoreDispatcher(
        semaphore_count=2,
        rate_limiter=RateLimiter(base_delay=(2.0, 5.0)),
    ),
)

A real block looks different from what you saw: result.success is False (or the HTML contains an Akamai challenge page) at the FETCH stage — not a Python KeyError after a successful fetch.

Hope this helps you!

[nanridao2026]

For grabbing HTML content, the cleaner form is:

{"name": "description", "selector": ".bsk-drawer__content", "type": "html"}
For the description field, some responses return an empty list, while others are null.

When the optional key transform is needed, can the following approach be used?
{"name": "description", "selector": ".bsk-drawer__content", "type": "html"，“transform：“def clean_description()”

Another point concerns crawling a specific directory; is there a simple way to crawl an entire website—similar to how Scrapy works? I haven't found a method for this in the documentation so far.

[SohamKukreti]

Hello, answering your second question first, Crawl4AI does have a simple way to crawl an entire website called deep crawling. You attach a deep_crawl_strategy to your CrawlerRunConfig and a single arun() call will discover and follow links by itself. There's BFS, DFS, and a BestFirstCrawlingStrategy (priority-driven with scorers), plus URL/content filters, conceptually very close to Scrapy's link-following spiders. There's also AsyncUrlSeeder if you'd rather discover all URLs up front (from sitemaps / Common Crawl) and then batch-crawl them with arun_many (More efficient method if you want to collect the list of URLs first and then crawl them).

Here's the documentation for Deep Crawling: https://docs.crawl4ai.com/core/deep-crawling
and URL seeding: https://docs.crawl4ai.com/core/url-seeding/

Now for the custom transform field, your approach won't work as it is:
"transform" only accepts a few built-in string values ("lowercase", "uppercase", "strip"), not custom functions:

{"name": "name", "selector": "h1.product-title", "type": "text", "transform": "strip"}

For custom logic there are two options:

Option A: a computed field with a Python callable. Since you're building the schema in Python (not JSON), you can pass a real function. It receives the item extracted so far and returns the computed value:

def clean_description(item):
    raw = item.get("description_raw") or ""
    # ... your cleanup logic ...
    return raw.strip()

schema = {
    "name": "Product Detail",
    "baseSelector": ".product-detail-layout",
    "fields": [
        {"name": "description_raw", "selector": ".bsk-drawer__content", "type": "html"},
        {"name": "description", "type": "computed", "function": clean_description,
         "default": None},
        # ...
    ],
}

Note: field order matters, since computed fields can only see fields extracted before them. (The "expression" string variant of computed fields is disabled for security; use "function" with a callable.)

Option B: just post-process in Python after json.loads(result.extracted_content), before building your DataFrame. For a one-off cleanup this is often the simplest and easiest to debug.

[nanridao2026]

Thank you for your very detailed explanation and suggestions!