[Feature Request]: Support for docker proxy

[maxinfo337]

What needs to be done?

Hi ! when i use your last 9.0 in a docker on a host who have docker proxy variable (host can go out only with proxy) , it doesn't work ... but it works with the version 8.x , please add proxy use to your last version. i use crawl4ai with hermes-agent with a skill.

What problem does this solve?

v9 can bu used in restricted environment.

Target users/beneficiaries

No response

Current alternatives/workarounds

No response

Proposed approach

No response

[SohamKukreti]

Hey! Thanks for reporting this and for the details. Just to make sure I understand correctly: with v0.9.0+ running in Docker on a host that requires an HTTP/HTTPS proxy for outbound access, it doesn’t work, while v0.8.x works fine. Is that right?

If so, could you share a bit more about what exactly fails in v0.9.0+ (for example, any error messages or logs) and how the proxy is configured? That’ll help us understand it better and add a fix

[maxinfo337]

Yes it is exactly that ! here is the logs and dockers var :

Startup LOG:

2026-08-13T16:38:21.070684836Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-13T16:38:22.034057392Z [crawl4ai] 2026-08-13 16:38:22,026 INFO supervisord started with pid 1
2026-08-13T16:38:23.032701013Z [crawl4ai] 2026-08-13 16:38:23,031 INFO spawned: 'redis' with pid 8
2026-08-13T16:38:23.033564454Z [crawl4ai] 2026-08-13 16:38:23,033 INFO spawned: 'gunicorn' with pid 9
2026-08-13T16:38:23.222365584Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-13T16:38:23.222409826Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-13T16:38:23.222416914Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 * Redis version=8.8.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-13T16:38:23.222422393Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 * Configuration loaded
2026-08-13T16:38:23.225108361Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.224 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-13T16:38:23.225141521Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.224 * monotonic clock: POSIX clock_gettime
2026-08-13T16:38:23.238635199Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.238 * Running mode=standalone, port=6379.
2026-08-13T16:38:23.244593897Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.244 * Server initialized
2026-08-13T16:38:23.256029386Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.255 * Ready to accept connections tcp
2026-08-13T16:38:24.257621394Z [crawl4ai] 2026-08-13 16:38:24,257 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-13T16:38:24.257661260Z [crawl4ai] 2026-08-13 16:38:24,257 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-13T16:38:24.344307521Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-13T16:38:24.346211938Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-13T16:38:24.346226880Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-13T16:38:24.349286316Z [crawl4ai] [2026-08-13 16:38:24 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-13T16:38:24.461586279Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-13T16:38:30.102882480Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-13T16:38:30.445625621Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-13T16:38:30.445667754Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-13T16:38:31.746573571Z [crawl4ai] /app/server.py:1010: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-13T16:38:31.746626224Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-13T16:38:31.750931271Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-13T16:38:31.824589062Z [crawl4ai] [2026-08-13 16:38:31 +0000] [15] [INFO] Started server process [15]
2026-08-13T16:38:31.825235325Z [crawl4ai] [2026-08-13 16:38:31 +0000] [15] [INFO] Waiting for application startup.
2026-08-13T16:38:31.826510954Z [crawl4ai] 2026-08-13 16:38:31,826 - server - INFO - Auth gate active (credential configured).
2026-08-13T16:38:31.844207196Z [crawl4ai] 2026-08-13 16:38:31,843 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45119
2026-08-13T16:38:31.847295942Z [crawl4ai] 2026-08-13 16:38:31,846 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-13T16:38:31.854798163Z [crawl4ai] 2026-08-13 16:38:31,854 - monitor - INFO - Started persistence worker
2026-08-13T16:38:31.936707697Z [crawl4ai] 2026-08-13 16:38:31,936 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-13T16:38:35.435144635Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-13T16:38:35.438519691Z [crawl4ai] [2026-08-13 16:38:35 +0000] [15] [INFO] Application startup complete.
2026-08-13T16:38:36.440951601Z [crawl4ai] 2026-08-13 16:38:36,440 INFO reaped unknown pid 23 (exit status 0)

ERROR LOG:

2026-08-13T16:39:05.438196055Z [crawl4ai] 2026-08-13 16:39:05,437 - crawler_pool - INFO - 📊 Pool: hot=0, cold=0, mem=66.4%
2026-08-13T16:39:25.516470871Z [crawl4ai] 2026-08-13 16:39:25,515 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=ac4151cd, mem=67.8%)
2026-08-13T16:39:26.618417943Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-13T16:39:35.439633138Z [crawl4ai] 2026-08-13 16:39:35,438 - crawler_pool - INFO - 📊 Pool: hot=0, cold=1, mem=67.2%
2026-08-13T16:39:59.102887369Z [crawl4ai] [ANTIBOT]. ℹ https://example.com | Error: Proxy direct failed: Failed on navigating ACS-GOTO:
2026-08-13T16:39:59.102916392Z [crawl4ai] Page.goto: net::ERR_TIMED_OUT at https://example.com/
2026-08-13T16:39:59.102924222Z [crawl4ai] Call log:
2026-08-13T16:39:59.102930696Z [crawl4ai] - navigating to "https://example.com/", waiting until "domcontentloaded"
2026-08-13T16:39:59.102937163Z [crawl4ai]
2026-08-13T16:39:59.116143226Z [crawl4ai] [ERROR]... × https://example.com | Error: Unexpected error in _crawl_web at line 778 in _crawl_web
2026-08-13T16:39:59.116282416Z [crawl4ai] (../usr/local/lib/python3.12/site-packages/crawl4ai/async_crawler_strategy.py):
2026-08-13T16:39:59.116291520Z [crawl4ai] Error: Failed on navigating ACS-GOTO:
2026-08-13T16:39:59.116298239Z [crawl4ai] Page.goto: net::ERR_TIMED_OUT at https://example.com/
2026-08-13T16:39:59.116304453Z [crawl4ai] Call log:
2026-08-13T16:39:59.116310521Z [crawl4ai] - navigating to "https://example.com/", waiting until "domcontentloaded"
2026-08-13T16:39:59.116316910Z [crawl4ai]
2026-08-13T16:39:59.117267033Z [crawl4ai]
2026-08-13T16:39:59.117280378Z [crawl4ai] Code context:
2026-08-13T16:39:59.117286909Z [crawl4ai] 773 tag="GOTO",
2026-08-13T16:39:59.117293336Z [crawl4ai] 774 params={"url": url},
2026-08-13T16:39:59.117313620Z [crawl4ai] 775 )
2026-08-13T16:39:59.117319272Z [crawl4ai] 776 response = None
2026-08-13T16:39:59.117342182Z [crawl4ai] 777 else:
2026-08-13T16:39:59.117347675Z [crawl4ai] 778 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2026-08-13T16:39:59.117353175Z [crawl4ai] 779
2026-08-13T16:39:59.117358086Z [crawl4ai] 780 # ──────────────────────────────────────────────────────────────
2026-08-13T16:39:59.117364727Z [crawl4ai] 781 # Walk the redirect chain. Playwright returns only the last
2026-08-13T16:39:59.117369702Z [crawl4ai] 782 # hop, so we trace the request.redirected_from links until the
2026-08-13T16:39:59.117374548Z [crawl4ai] 783 # first response that differs from the final one and surface its
2026-08-13T16:39:59.121423389Z [crawl4ai] 2026-08-13 16:39:59,120 - server - ERROR - server error 500 [cid=e1d04373fe82]: Unexpected error in _crawl_web at line 778 in _crawl_web (../usr/local/lib/python3.12/site-packages/crawl4ai/async_crawler_strategy.py):Error: Failed on navigating ACS-GOTO:Page.goto: net::ERR_TIMED_OUT at https://example.com/Call log: - navigating to "https://example.com/", waiting until "domcontentloaded"Code context: 773 tag="GOTO", 774 params={"url": url}, 775 ) 776 response = None 777 else: 778 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}") 779 780 # ────────────────────────────────────────────────────────────── 781 # Walk the redirect chain. Playwright returns only the last 782 # hop, so we trace the request.redirected_from links until the 783 # first response that differs from the final one and surface its
2026-08-13T16:40:05.441844644Z [crawl4ai] 2026-08-13 16:40:05,440 - crawler_pool - INFO - 📊 Pool: hot=0, cold=1, mem=70.0%
2026-08-13T16:38:21.070684836Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-13T16:38:22.034057392Z [crawl4ai] 2026-08-13 16:38:22,026 INFO supervisord started with pid 1
2026-08-13T16:38:23.032701013Z [crawl4ai] 2026-08-13 16:38:23,031 INFO spawned: 'redis' with pid 8
2026-08-13T16:38:23.033564454Z [crawl4ai] 2026-08-13 16:38:23,033 INFO spawned: 'gunicorn' with pid 9
2026-08-13T16:38:23.222365584Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-13T16:38:23.222409826Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-13T16:38:23.222416914Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 * Redis version=8.8.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-13T16:38:23.222422393Z [crawl4ai] 8:C 13 Aug 2026 16:38:23.221 * Configuration loaded
2026-08-13T16:38:23.225108361Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.224 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-13T16:38:23.225141521Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.224 * monotonic clock: POSIX clock_gettime
2026-08-13T16:38:23.238635199Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.238 * Running mode=standalone, port=6379.
2026-08-13T16:38:23.244593897Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.244 * Server initialized
2026-08-13T16:38:23.256029386Z [crawl4ai] 8:M 13 Aug 2026 16:38:23.255 * Ready to accept connections tcp
2026-08-13T16:38:24.257621394Z [crawl4ai] 2026-08-13 16:38:24,257 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-13T16:38:24.257661260Z [crawl4ai] 2026-08-13 16:38:24,257 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-13T16:38:24.344307521Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-13T16:38:24.346211938Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-13T16:38:24.346226880Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-13T16:38:24.349286316Z [crawl4ai] [2026-08-13 16:38:24 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-13T16:38:24.461586279Z [crawl4ai] [2026-08-13 16:38:24 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-13T16:38:30.102882480Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-13T16:38:30.445625621Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-13T16:38:30.445667754Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-13T16:38:31.746573571Z [crawl4ai] /app/server.py:1010: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-13T16:38:31.746626224Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-13T16:38:31.750931271Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-13T16:38:31.824589062Z [crawl4ai] [2026-08-13 16:38:31 +0000] [15] [INFO] Started server process [15]
2026-08-13T16:38:31.825235325Z [crawl4ai] [2026-08-13 16:38:31 +0000] [15] [INFO] Waiting for application startup.
2026-08-13T16:38:31.826510954Z [crawl4ai] 2026-08-13 16:38:31,826 - server - INFO - Auth gate active (credential configured).
2026-08-13T16:38:31.844207196Z [crawl4ai] 2026-08-13 16:38:31,843 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45119
2026-08-13T16:38:31.847295942Z [crawl4ai] 2026-08-13 16:38:31,846 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-13T16:38:31.854798163Z [crawl4ai] 2026-08-13 16:38:31,854 - monitor - INFO - Started persistence worker
2026-08-13T16:38:31.936707697Z [crawl4ai] 2026-08-13 16:38:31,936 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-13T16:38:35.435144635Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-13T16:38:35.438519691Z [crawl4ai] [2026-08-13 16:38:35 +0000] [15] [INFO] Application startup complete.
2026-08-13T16:38:36.440951601Z [crawl4ai] 2026-08-13 16:38:36,440 INFO reaped unknown pid 23 (exit status 0)
2026-08-13T16:39:05.438196055Z [crawl4ai] 2026-08-13 16:39:05,437 - crawler_pool - INFO - 📊 Pool: hot=0, cold=0, mem=66.4%
2026-08-13T16:39:25.516470871Z [crawl4ai] 2026-08-13 16:39:25,515 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=ac4151cd, mem=67.8%)
2026-08-13T16:39:26.618417943Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-13T16:39:35.439633138Z [crawl4ai] 2026-08-13 16:39:35,438 - crawler_pool - INFO - 📊 Pool: hot=0, cold=1, mem=67.2%
2026-08-13T16:39:59.102887369Z [crawl4ai] [ANTIBOT]. ℹ https://example.com | Error: Proxy direct failed: Failed on navigating ACS-GOTO:
2026-08-13T16:39:59.102916392Z [crawl4ai] Page.goto: net::ERR_TIMED_OUT at https://example.com/
2026-08-13T16:39:59.102924222Z [crawl4ai] Call log:
2026-08-13T16:39:59.102930696Z [crawl4ai] - navigating to "https://example.com/", waiting until "domcontentloaded"
2026-08-13T16:39:59.102937163Z [crawl4ai]
2026-08-13T16:39:59.116143226Z [crawl4ai] [ERROR]... × https://example.com | Error: Unexpected error in _crawl_web at line 778 in _crawl_web
2026-08-13T16:39:59.116282416Z [crawl4ai] (../usr/local/lib/python3.12/site-packages/crawl4ai/async_crawler_strategy.py):
2026-08-13T16:39:59.116291520Z [crawl4ai] Error: Failed on navigating ACS-GOTO:
2026-08-13T16:39:59.116298239Z [crawl4ai] Page.goto: net::ERR_TIMED_OUT at https://example.com/
2026-08-13T16:39:59.116304453Z [crawl4ai] Call log:
2026-08-13T16:39:59.116310521Z [crawl4ai] - navigating to "https://example.com/", waiting until "domcontentloaded"
2026-08-13T16:39:59.116316910Z [crawl4ai]
2026-08-13T16:39:59.117267033Z [crawl4ai]
2026-08-13T16:39:59.117280378Z [crawl4ai] Code context:
2026-08-13T16:39:59.117286909Z [crawl4ai] 773 tag="GOTO",
2026-08-13T16:39:59.117293336Z [crawl4ai] 774 params={"url": url},
2026-08-13T16:39:59.117313620Z [crawl4ai] 775 )
2026-08-13T16:39:59.117319272Z [crawl4ai] 776 response = None
2026-08-13T16:39:59.117342182Z [crawl4ai] 777 else:
2026-08-13T16:39:59.117347675Z [crawl4ai] 778 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}")
2026-08-13T16:39:59.117353175Z [crawl4ai] 779
2026-08-13T16:39:59.117358086Z [crawl4ai] 780 # ──────────────────────────────────────────────────────────────
2026-08-13T16:39:59.117364727Z [crawl4ai] 781 # Walk the redirect chain. Playwright returns only the last
2026-08-13T16:39:59.117369702Z [crawl4ai] 782 # hop, so we trace the request.redirected_from links until the
2026-08-13T16:39:59.117374548Z [crawl4ai] 783 # first response that differs from the final one and surface its
2026-08-13T16:39:59.121423389Z [crawl4ai] 2026-08-13 16:39:59,120 - server - ERROR - server error 500 [cid=e1d04373fe82]: Unexpected error in _crawl_web at line 778 in _crawl_web (../usr/local/lib/python3.12/site-packages/crawl4ai/async_crawler_strategy.py):Error: Failed on navigating ACS-GOTO:Page.goto: net::ERR_TIMED_OUT at https://example.com/Call log: - navigating to "https://example.com/", waiting until "domcontentloaded"Code context: 773 tag="GOTO", 774 params={"url": url}, 775 ) 776 response = None 777 else: 778 → raise RuntimeError(f"Failed on navigating ACS-GOTO:\n{str(e)}") 779 780 # ────────────────────────────────────────────────────────────── 781 # Walk the redirect chain. Playwright returns only the last 782 # hop, so we trace the request.redirected_from links until the 783 # first response that differs from the final one and surface its
2026-08-13T16:40:05.441844644Z [crawl4ai] 2026-08-13 16:40:05,440 - crawler_pool - INFO - 📊 Pool: hot=0, cold=1, mem=70.0%
2026-08-13T16:40:35.444289192Z [crawl4ai] 2026-08-13 16:40:35,442 - crawler_pool - INFO - 🧹 Closing cold browser (sig=ac4151cd, idle=68s)
2026-08-13T16:40:35.638980500Z [crawl4ai] 2026-08-13 16:40:35,637 - crawler_pool - INFO - 📊 Pool: hot=0, cold=0, mem=69.7%
2026-08-13T16:40:35.639520391Z [crawl4ai] 2026-08-13 16:40:35,639 INFO reaped unknown pid 122 (exit status 0)
2026-08-13T16:40:35.639548150Z [crawl4ai] 2026-08-13 16:40:35,639 INFO reaped unknown pid 123 (exit status 0)

ENV VAR DOCKER:

C4AI_VERSION = 0.9.2
CRAWL4AI_API_TOKEN = XXXX
DEBIAN_FRONTEND = noninteractive
GPG_KEY = XXXX
http_proxy = http://192.168.180.254:56560
HTTP_PROXY = http://192.168.180.254:56560
https_proxy = http://192.168.180.254:56560
HTTPS_PROXY = http://192.168.180.254:56560
LANG = C.UTF-8
no_proxy = .dmz.lan,.XXXX.lan,localhost,127.0.0.0/8,192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
NO_PROXY = .dmz.lan,.XXXX.lan,localhost,127.0.0.0/8,192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
PATH = /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PIP_DEFAULT_TIMEOUT = 100
PIP_DISABLE_PIP_VERSION_CHECK = 1
PIP_NO_CACHE_DIR = 1
PYTHON_ENV = production
PYTHON_SHA256 = c08bc65a81971c1dd5783182826503369466c7e67374d1646519adf05207b684
PYTHON_VERSION = 3.12.13
PYTHONDONTWRITEBYTECODE = 1
PYTHONFAULTHANDLER = 1
PYTHONHASHSEED = random
PYTHONUNBUFFERED = 1
REDIS_HOST = localhost
REDIS_PORT = 6379

and docker config :
cat ~/.docker/config.json

{
"proxies":
{
"default":
{
"httpProxy": "http://192.168.180.254:56560",
"httpsProxy": "http://192.168.180.254:56560",
"noProxy": ".dmz.lan,.XXXX.lan,localhost,127.0.0.0/8,192.168.0.0/16,10.0.0.0/8,172.16.0.0/12"
}
}
}

[SohamKukreti]

Hey @maxinfo337, I've opened a PR that adds corporate proxy support to the hardened Docker server.

What was happening: since 0.9.0 the server routes all browser traffic through an internal security proxy (the egress pinning proxy in your logs), and that proxy was dialing target sites directly — ignoring your HTTP_PROXY/HTTPS_PROXY variables. On a host where direct outbound connections are blocked, every crawl timed out with the net::ERR_TIMED_OUT you saw. With the PR, it now chains through your corporate proxy while keeping the security protections intact.

To make this change work no configuration changes are needed. Your existing env vars (injected by your ~/.docker/config.json proxies config) are picked up automatically, and your no_proxy list is honored too.

To try it out, build the image from the PR branch:

git clone --branch fix/egress-proxy-upstream-chaining https://github.com/unclecode/crawl4ai.git
cd crawl4ai
docker buildx build -t crawl4ai-proxy-test:latest --load .

then run it exactly as you do today. On startup you should see a new log line confirming the proxy was detected:

egress pinning proxy chaining through upstream proxy 192.168.180.254:56560

Could you give it a spin in your environment and let me know if crawls work as expected? If anything still fails, the container logs (especially any upstream proxy refused CONNECT warnings) would help a lot.

[maxinfo337]

Yes it's working with this image ;) i will use it for now. here is the full log :

2026-08-15T21:01:38.231913552Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-15T21:01:38.987831484Z [crawl4ai] 2026-08-15 21:01:38,987 INFO supervisord started with pid 1
2026-08-15T21:01:39.992307275Z [crawl4ai] 2026-08-15 21:01:39,991 INFO spawned: 'redis' with pid 8
2026-08-15T21:01:39.995704528Z [crawl4ai] 2026-08-15 21:01:39,995 INFO spawned: 'gunicorn' with pid 9
2026-08-15T21:01:40.112510523Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.111 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-15T21:01:40.112958455Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.112 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-15T21:01:40.113130564Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Redis version=8.10.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-15T21:01:40.113316530Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Configuration loaded
2026-08-15T21:01:40.122727969Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-15T21:01:40.122757085Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * monotonic clock: POSIX clock_gettime
2026-08-15T21:01:40.128378384Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.127 * Running mode=standalone, port=6379.
2026-08-15T21:01:40.128860058Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.128 * Server initialized
2026-08-15T21:01:40.134679347Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.134 * Ready to accept connections tcp
2026-08-15T21:01:41.037954764Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.037990578Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.038008738Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-15T21:01:41.038875278Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-15T21:01:41.039127837Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-15T21:01:41.043075848Z [crawl4ai] [2026-08-15 21:01:41 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-15T21:01:41.130351578Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-15T21:01:45.575770988Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-15T21:01:45.799073558Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-15T21:01:45.799100420Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-15T21:01:46.848005346Z [crawl4ai] /app/server.py:1007: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-15T21:01:46.848032130Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-15T21:01:46.850471581Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-15T21:01:46.906950546Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Started server process [15]
2026-08-15T21:01:46.907355619Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Waiting for application startup.
2026-08-15T21:01:46.908198189Z [crawl4ai] 2026-08-15 21:01:46,908 - server - INFO - Auth gate active (credential configured).
2026-08-15T21:01:46.928375352Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45969
2026-08-15T21:01:46.928516622Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy chaining through upstream proxy 192.168.180.254:56560
2026-08-15T21:01:46.930817729Z [crawl4ai] 2026-08-15 21:01:46,930 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-15T21:01:46.939524189Z [crawl4ai] 2026-08-15 21:01:46,939 - monitor - INFO - Started persistence worker
2026-08-15T21:01:47.017814506Z [crawl4ai] 2026-08-15 21:01:47,017 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-15T21:01:50.747436983Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:01:50.748710952Z [crawl4ai] [2026-08-15 21:01:50 +0000] [15] [INFO] Application startup complete.

TEST:

2026-08-15T21:01:38.231913552Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-15T21:01:38.987831484Z [crawl4ai] 2026-08-15 21:01:38,987 INFO supervisord started with pid 1
2026-08-15T21:01:39.992307275Z [crawl4ai] 2026-08-15 21:01:39,991 INFO spawned: 'redis' with pid 8
2026-08-15T21:01:39.995704528Z [crawl4ai] 2026-08-15 21:01:39,995 INFO spawned: 'gunicorn' with pid 9
2026-08-15T21:01:40.112510523Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.111 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-15T21:01:40.112958455Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.112 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-15T21:01:40.113130564Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Redis version=8.10.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-15T21:01:40.113316530Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Configuration loaded
2026-08-15T21:01:40.122727969Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-15T21:01:40.122757085Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * monotonic clock: POSIX clock_gettime
2026-08-15T21:01:40.128378384Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.127 * Running mode=standalone, port=6379.
2026-08-15T21:01:40.128860058Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.128 * Server initialized
2026-08-15T21:01:40.134679347Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.134 * Ready to accept connections tcp
2026-08-15T21:01:41.037954764Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.037990578Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.038008738Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-15T21:01:41.038875278Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-15T21:01:41.039127837Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-15T21:01:41.043075848Z [crawl4ai] [2026-08-15 21:01:41 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-15T21:01:41.130351578Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-15T21:01:45.575770988Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-15T21:01:45.799073558Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-15T21:01:45.799100420Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-15T21:01:46.848005346Z [crawl4ai] /app/server.py:1007: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-15T21:01:46.848032130Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-15T21:01:46.850471581Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-15T21:01:46.906950546Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Started server process [15]
2026-08-15T21:01:46.907355619Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Waiting for application startup.
2026-08-15T21:01:46.908198189Z [crawl4ai] 2026-08-15 21:01:46,908 - server - INFO - Auth gate active (credential configured).
2026-08-15T21:01:46.928375352Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45969
2026-08-15T21:01:46.928516622Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy chaining through upstream proxy 192.168.180.254:56560
2026-08-15T21:01:46.930817729Z [crawl4ai] 2026-08-15 21:01:46,930 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-15T21:01:46.939524189Z [crawl4ai] 2026-08-15 21:01:46,939 - monitor - INFO - Started persistence worker
2026-08-15T21:01:47.017814506Z [crawl4ai] 2026-08-15 21:01:47,017 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-15T21:01:50.747436983Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:01:50.748710952Z [crawl4ai] [2026-08-15 21:01:50 +0000] [15] [INFO] Application startup complete.
2026-08-15T21:03:00.327397582Z [crawl4ai] 2026-08-15 21:03:00,325 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=442b44b0, mem=54.1%)
2026-08-15T21:03:01.155105112Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:03:03.117785403Z [crawl4ai] [FETCH]... ↓ https://example.com | ✓ | ⏱: 1.91s
2026-08-15T21:03:03.131563810Z [crawl4ai] [SCRAPE].. ◆ https://example.com | ✓ | ⏱: 0.01s
2026-08-15T21:03:03.137049092Z [crawl4ai] [COMPLETE] ● https://example.com | ✓ | ⏱: 1.93s
2026-08-15T21:03:04.369252854Z [crawl4ai] [COMPLETE] ● Database backup created at: /home/appuser/.crawl4ai/crawl4ai.db.backup_20260815_210303
2026-08-15T21:03:04.369286628Z [crawl4ai] [INIT].... → Starting database migration...
2026-08-15T21:03:04.371522114Z [crawl4ai] [COMPLETE] ● Migration completed. 0 records processed.
2026-08-15T21:03:13.887750251Z [crawl4ai] 2026-08-15 21:03:13,887 - crawler_pool - INFO - ❄️ Using cold pool browser (sig=442b44b0)
2026-08-15T21:03:20.452682906Z [crawl4ai] [FETCH]... ↓ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 6.54s
2026-08-15T21:03:22.252476007Z [crawl4ai] [SCRAPE].. ◆ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 1.80s
2026-08-15T21:03:22.278548502Z [crawl4ai] [COMPLETE] ● https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 8.37s
2026-08-15T21:03:22.656568446Z [crawl4ai] 2026-08-15 21:03:22,656 - crawler_pool - INFO - ⬆️ Promoting to hot pool (sig=442b44b0, count=3)
2026-08-15T21:03:23.555156287Z [crawl4ai] [FETCH]... ↓ https://www.wikipedia.org/ | ✓ | ⏱: 0.87s
2026-08-15T21:03:23.697964059Z [crawl4ai] [SCRAPE].. ◆ https://www.wikipedia.org/ | ✓ | ⏱: 0.14s
2026-08-15T21:03:23.717358242Z [crawl4ai] [COMPLETE] ● https://www.wikipedia.org/ | ✓ | ⏱: 1.04s
2026-08-15T21:01:38.231913552Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-15T21:01:38.987831484Z [crawl4ai] 2026-08-15 21:01:38,987 INFO supervisord started with pid 1
2026-08-15T21:01:39.992307275Z [crawl4ai] 2026-08-15 21:01:39,991 INFO spawned: 'redis' with pid 8
2026-08-15T21:01:39.995704528Z [crawl4ai] 2026-08-15 21:01:39,995 INFO spawned: 'gunicorn' with pid 9
2026-08-15T21:01:40.112510523Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.111 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-15T21:01:40.112958455Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.112 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-15T21:01:40.113130564Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Redis version=8.10.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-15T21:01:40.113316530Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Configuration loaded
2026-08-15T21:01:40.122727969Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-15T21:01:40.122757085Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * monotonic clock: POSIX clock_gettime
2026-08-15T21:01:40.128378384Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.127 * Running mode=standalone, port=6379.
2026-08-15T21:01:40.128860058Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.128 * Server initialized
2026-08-15T21:01:40.134679347Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.134 * Ready to accept connections tcp
2026-08-15T21:01:41.037954764Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.037990578Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.038008738Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-15T21:01:41.038875278Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-15T21:01:41.039127837Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-15T21:01:41.043075848Z [crawl4ai] [2026-08-15 21:01:41 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-15T21:01:41.130351578Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-15T21:01:45.575770988Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-15T21:01:45.799073558Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-15T21:01:45.799100420Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-15T21:01:46.848005346Z [crawl4ai] /app/server.py:1007: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-15T21:01:46.848032130Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-15T21:01:46.850471581Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-15T21:01:46.906950546Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Started server process [15]
2026-08-15T21:01:46.907355619Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Waiting for application startup.
2026-08-15T21:01:46.908198189Z [crawl4ai] 2026-08-15 21:01:46,908 - server - INFO - Auth gate active (credential configured).
2026-08-15T21:01:46.928375352Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45969
2026-08-15T21:01:46.928516622Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy chaining through upstream proxy 192.168.180.254:56560
2026-08-15T21:01:46.930817729Z [crawl4ai] 2026-08-15 21:01:46,930 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-15T21:01:46.939524189Z [crawl4ai] 2026-08-15 21:01:46,939 - monitor - INFO - Started persistence worker
2026-08-15T21:01:47.017814506Z [crawl4ai] 2026-08-15 21:01:47,017 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-15T21:01:50.747436983Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:01:50.748710952Z [crawl4ai] [2026-08-15 21:01:50 +0000] [15] [INFO] Application startup complete.
2026-08-15T21:03:00.327397582Z [crawl4ai] 2026-08-15 21:03:00,325 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=442b44b0, mem=54.1%)
2026-08-15T21:03:01.155105112Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:03:03.117785403Z [crawl4ai] [FETCH]... ↓ https://example.com | ✓ | ⏱: 1.91s
2026-08-15T21:03:03.131563810Z [crawl4ai] [SCRAPE].. ◆ https://example.com | ✓ | ⏱: 0.01s
2026-08-15T21:03:03.137049092Z [crawl4ai] [COMPLETE] ● https://example.com | ✓ | ⏱: 1.93s
2026-08-15T21:03:04.369252854Z [crawl4ai] [COMPLETE] ● Database backup created at: /home/appuser/.crawl4ai/crawl4ai.db.backup_20260815_210303
2026-08-15T21:03:04.369286628Z [crawl4ai] [INIT].... → Starting database migration...
2026-08-15T21:03:04.371522114Z [crawl4ai] [COMPLETE] ● Migration completed. 0 records processed.
2026-08-15T21:03:13.887750251Z [crawl4ai] 2026-08-15 21:03:13,887 - crawler_pool - INFO - ❄️ Using cold pool browser (sig=442b44b0)
2026-08-15T21:03:20.452682906Z [crawl4ai] [FETCH]... ↓ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 6.54s
2026-08-15T21:03:22.252476007Z [crawl4ai] [SCRAPE].. ◆ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 1.80s
2026-08-15T21:03:22.278548502Z [crawl4ai] [COMPLETE] ● https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 8.37s
2026-08-15T21:03:22.656568446Z [crawl4ai] 2026-08-15 21:03:22,656 - crawler_pool - INFO - ⬆️ Promoting to hot pool (sig=442b44b0, count=3)
2026-08-15T21:03:23.555156287Z [crawl4ai] [FETCH]... ↓ https://www.wikipedia.org/ | ✓ | ⏱: 0.87s
2026-08-15T21:03:23.697964059Z [crawl4ai] [SCRAPE].. ◆ https://www.wikipedia.org/ | ✓ | ⏱: 0.14s
2026-08-15T21:03:23.717358242Z [crawl4ai] [COMPLETE] ● https://www.wikipedia.org/ | ✓ | ⏱: 1.04s
2026-08-15T21:01:38.231913552Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-15T21:01:38.987831484Z [crawl4ai] 2026-08-15 21:01:38,987 INFO supervisord started with pid 1
2026-08-15T21:01:39.992307275Z [crawl4ai] 2026-08-15 21:01:39,991 INFO spawned: 'redis' with pid 8
2026-08-15T21:01:39.995704528Z [crawl4ai] 2026-08-15 21:01:39,995 INFO spawned: 'gunicorn' with pid 9
2026-08-15T21:01:40.112510523Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.111 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-15T21:01:40.112958455Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.112 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-15T21:01:40.113130564Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Redis version=8.10.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-15T21:01:40.113316530Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Configuration loaded
2026-08-15T21:01:40.122727969Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-15T21:01:40.122757085Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * monotonic clock: POSIX clock_gettime
2026-08-15T21:01:40.128378384Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.127 * Running mode=standalone, port=6379.
2026-08-15T21:01:40.128860058Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.128 * Server initialized
2026-08-15T21:01:40.134679347Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.134 * Ready to accept connections tcp
2026-08-15T21:01:41.037954764Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.037990578Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.038008738Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-15T21:01:41.038875278Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-15T21:01:41.039127837Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-15T21:01:41.043075848Z [crawl4ai] [2026-08-15 21:01:41 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-15T21:01:41.130351578Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-15T21:01:45.575770988Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-15T21:01:45.799073558Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-15T21:01:45.799100420Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-15T21:01:46.848005346Z [crawl4ai] /app/server.py:1007: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-15T21:01:46.848032130Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-15T21:01:46.850471581Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-15T21:01:46.906950546Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Started server process [15]
2026-08-15T21:01:46.907355619Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Waiting for application startup.
2026-08-15T21:01:46.908198189Z [crawl4ai] 2026-08-15 21:01:46,908 - server - INFO - Auth gate active (credential configured).
2026-08-15T21:01:46.928375352Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45969
2026-08-15T21:01:46.928516622Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy chaining through upstream proxy 192.168.180.254:56560
2026-08-15T21:01:46.930817729Z [crawl4ai] 2026-08-15 21:01:46,930 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-15T21:01:46.939524189Z [crawl4ai] 2026-08-15 21:01:46,939 - monitor - INFO - Started persistence worker
2026-08-15T21:01:47.017814506Z [crawl4ai] 2026-08-15 21:01:47,017 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-15T21:01:50.747436983Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:01:50.748710952Z [crawl4ai] [2026-08-15 21:01:50 +0000] [15] [INFO] Application startup complete.
2026-08-15T21:03:00.327397582Z [crawl4ai] 2026-08-15 21:03:00,325 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=442b44b0, mem=54.1%)
2026-08-15T21:03:01.155105112Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:03:03.117785403Z [crawl4ai] [FETCH]... ↓ https://example.com | ✓ | ⏱: 1.91s
2026-08-15T21:03:03.131563810Z [crawl4ai] [SCRAPE].. ◆ https://example.com | ✓ | ⏱: 0.01s
2026-08-15T21:03:03.137049092Z [crawl4ai] [COMPLETE] ● https://example.com | ✓ | ⏱: 1.93s
2026-08-15T21:03:04.369252854Z [crawl4ai] [COMPLETE] ● Database backup created at: /home/appuser/.crawl4ai/crawl4ai.db.backup_20260815_210303
2026-08-15T21:03:04.369286628Z [crawl4ai] [INIT].... → Starting database migration...
2026-08-15T21:03:04.371522114Z [crawl4ai] [COMPLETE] ● Migration completed. 0 records processed.
2026-08-15T21:03:13.887750251Z [crawl4ai] 2026-08-15 21:03:13,887 - crawler_pool - INFO - ❄️ Using cold pool browser (sig=442b44b0)
2026-08-15T21:03:20.452682906Z [crawl4ai] [FETCH]... ↓ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 6.54s
2026-08-15T21:03:22.252476007Z [crawl4ai] [SCRAPE].. ◆ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 1.80s
2026-08-15T21:03:22.278548502Z [crawl4ai] [COMPLETE] ● https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 8.37s
2026-08-15T21:03:22.656568446Z [crawl4ai] 2026-08-15 21:03:22,656 - crawler_pool - INFO - ⬆️ Promoting to hot pool (sig=442b44b0, count=3)
2026-08-15T21:03:23.555156287Z [crawl4ai] [FETCH]... ↓ https://www.wikipedia.org/ | ✓ | ⏱: 0.87s
2026-08-15T21:03:23.697964059Z [crawl4ai] [SCRAPE].. ◆ https://www.wikipedia.org/ | ✓ | ⏱: 0.14s
2026-08-15T21:03:23.717358242Z [crawl4ai] [COMPLETE] ● https://www.wikipedia.org/ | ✓ | ⏱: 1.04s
2026-08-15T21:01:38.231913552Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-15T21:01:38.987831484Z [crawl4ai] 2026-08-15 21:01:38,987 INFO supervisord started with pid 1
2026-08-15T21:01:39.992307275Z [crawl4ai] 2026-08-15 21:01:39,991 INFO spawned: 'redis' with pid 8
2026-08-15T21:01:39.995704528Z [crawl4ai] 2026-08-15 21:01:39,995 INFO spawned: 'gunicorn' with pid 9
2026-08-15T21:01:40.112510523Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.111 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-15T21:01:40.112958455Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.112 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-15T21:01:40.113130564Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Redis version=8.10.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-15T21:01:40.113316530Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Configuration loaded
2026-08-15T21:01:40.122727969Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-15T21:01:40.122757085Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * monotonic clock: POSIX clock_gettime
2026-08-15T21:01:40.128378384Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.127 * Running mode=standalone, port=6379.
2026-08-15T21:01:40.128860058Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.128 * Server initialized
2026-08-15T21:01:40.134679347Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.134 * Ready to accept connections tcp
2026-08-15T21:01:41.037954764Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.037990578Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.038008738Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-15T21:01:41.038875278Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-15T21:01:41.039127837Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-15T21:01:41.043075848Z [crawl4ai] [2026-08-15 21:01:41 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-15T21:01:41.130351578Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-15T21:01:45.575770988Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-15T21:01:45.799073558Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-15T21:01:45.799100420Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-15T21:01:46.848005346Z [crawl4ai] /app/server.py:1007: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-15T21:01:46.848032130Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-15T21:01:46.850471581Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-15T21:01:46.906950546Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Started server process [15]
2026-08-15T21:01:46.907355619Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Waiting for application startup.
2026-08-15T21:01:46.908198189Z [crawl4ai] 2026-08-15 21:01:46,908 - server - INFO - Auth gate active (credential configured).
2026-08-15T21:01:46.928375352Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45969
2026-08-15T21:01:46.928516622Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy chaining through upstream proxy 192.168.180.254:56560
2026-08-15T21:01:46.930817729Z [crawl4ai] 2026-08-15 21:01:46,930 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-15T21:01:46.939524189Z [crawl4ai] 2026-08-15 21:01:46,939 - monitor - INFO - Started persistence worker
2026-08-15T21:01:47.017814506Z [crawl4ai] 2026-08-15 21:01:47,017 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-15T21:01:50.747436983Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:01:50.748710952Z [crawl4ai] [2026-08-15 21:01:50 +0000] [15] [INFO] Application startup complete.
2026-08-15T21:03:00.327397582Z [crawl4ai] 2026-08-15 21:03:00,325 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=442b44b0, mem=54.1%)
2026-08-15T21:03:01.155105112Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:03:03.117785403Z [crawl4ai] [FETCH]... ↓ https://example.com | ✓ | ⏱: 1.91s
2026-08-15T21:03:03.131563810Z [crawl4ai] [SCRAPE].. ◆ https://example.com | ✓ | ⏱: 0.01s
2026-08-15T21:03:03.137049092Z [crawl4ai] [COMPLETE] ● https://example.com | ✓ | ⏱: 1.93s
2026-08-15T21:03:04.369252854Z [crawl4ai] [COMPLETE] ● Database backup created at: /home/appuser/.crawl4ai/crawl4ai.db.backup_20260815_210303
2026-08-15T21:03:04.369286628Z [crawl4ai] [INIT].... → Starting database migration...
2026-08-15T21:03:04.371522114Z [crawl4ai] [COMPLETE] ● Migration completed. 0 records processed.
2026-08-15T21:03:13.887750251Z [crawl4ai] 2026-08-15 21:03:13,887 - crawler_pool - INFO - ❄️ Using cold pool browser (sig=442b44b0)
2026-08-15T21:03:20.452682906Z [crawl4ai] [FETCH]... ↓ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 6.54s
2026-08-15T21:03:22.252476007Z [crawl4ai] [SCRAPE].. ◆ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 1.80s
2026-08-15T21:03:22.278548502Z [crawl4ai] [COMPLETE] ● https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 8.37s
2026-08-15T21:03:22.656568446Z [crawl4ai] 2026-08-15 21:03:22,656 - crawler_pool - INFO - ⬆️ Promoting to hot pool (sig=442b44b0, count=3)
2026-08-15T21:03:23.555156287Z [crawl4ai] [FETCH]... ↓ https://www.wikipedia.org/ | ✓ | ⏱: 0.87s
2026-08-15T21:03:23.697964059Z [crawl4ai] [SCRAPE].. ◆ https://www.wikipedia.org/ | ✓ | ⏱: 0.14s
2026-08-15T21:03:23.717358242Z [crawl4ai] [COMPLETE] ● https://www.wikipedia.org/ | ✓ | ⏱: 1.04s
2026-08-15T21:01:38.231913552Z [crawl4ai] entrypoint: no REDIS_PASSWORD provided; generated an ephemeral one.
2026-08-15T21:01:38.987831484Z [crawl4ai] 2026-08-15 21:01:38,987 INFO supervisord started with pid 1
2026-08-15T21:01:39.992307275Z [crawl4ai] 2026-08-15 21:01:39,991 INFO spawned: 'redis' with pid 8
2026-08-15T21:01:39.995704528Z [crawl4ai] 2026-08-15 21:01:39,995 INFO spawned: 'gunicorn' with pid 9
2026-08-15T21:01:40.112510523Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.111 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2026-08-15T21:01:40.112958455Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.112 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2026-08-15T21:01:40.113130564Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Redis version=8.10.0, bits=64, commit=00000000, modified=1, pid=8, just started
2026-08-15T21:01:40.113316530Z [crawl4ai] 8:C 15 Aug 2026 21:01:40.113 * Configuration loaded
2026-08-15T21:01:40.122727969Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * Increased maximum number of open files to 10032 (it was originally set to 1024).
2026-08-15T21:01:40.122757085Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.122 * monotonic clock: POSIX clock_gettime
2026-08-15T21:01:40.128378384Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.127 * Running mode=standalone, port=6379.
2026-08-15T21:01:40.128860058Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.128 * Server initialized
2026-08-15T21:01:40.134679347Z [crawl4ai] 8:M 15 Aug 2026 21:01:40.134 * Ready to accept connections tcp
2026-08-15T21:01:41.037954764Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.037990578Z [crawl4ai] 2026-08-15 21:01:41,037 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-08-15T21:01:41.038008738Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Starting gunicorn 26.0.0
2026-08-15T21:01:41.038875278Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Listening at: http://[::]:11235 (9)
2026-08-15T21:01:41.039127837Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
2026-08-15T21:01:41.043075848Z [crawl4ai] [2026-08-15 21:01:41 +0000] [15] [INFO] Booting worker with pid: 15
2026-08-15T21:01:41.130351578Z [crawl4ai] [2026-08-15 21:01:41 +0000] [9] [INFO] Control socket listening at /home/appuser/.gunicorn/gunicorn.ctl
2026-08-15T21:01:45.575770988Z [crawl4ai] No SECRET_KEY set. Auto-generated an ephemeral key (changes on restart, invalidating issued tokens). Set SECRET_KEY for any real deployment.
2026-08-15T21:01:45.799073558Z [crawl4ai] /app/job.py:49: UserWarning: Field name "schema" in "LlmJobPayload" shadows an attribute in parent "BaseModel"
2026-08-15T21:01:45.799100420Z [crawl4ai] class LlmJobPayload(BaseModel):
2026-08-15T21:01:46.848005346Z [crawl4ai] /app/server.py:1007: FastAPIDeprecationWarning: regex has been deprecated, please use pattern instead
2026-08-15T21:01:46.848032130Z [crawl4ai] context_type: str = Query("all", regex="^(code|doc|all)$"),
2026-08-15T21:01:46.850471581Z [crawl4ai] MCP server running on 127.0.0.1:11235
2026-08-15T21:01:46.906950546Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Started server process [15]
2026-08-15T21:01:46.907355619Z [crawl4ai] [2026-08-15 21:01:46 +0000] [15] [INFO] Waiting for application startup.
2026-08-15T21:01:46.908198189Z [crawl4ai] 2026-08-15 21:01:46,908 - server - INFO - Auth gate active (credential configured).
2026-08-15T21:01:46.928375352Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy listening on http://127.0.0.1:45969
2026-08-15T21:01:46.928516622Z [crawl4ai] 2026-08-15 21:01:46,927 - crawl4ai.egress - INFO - egress pinning proxy chaining through upstream proxy 192.168.180.254:56560
2026-08-15T21:01:46.930817729Z [crawl4ai] 2026-08-15 21:01:46,930 - crawl4ai.workqueue - INFO - work queue started (maxsize=1000, workers=4, per_principal=unlimited)
2026-08-15T21:01:46.939524189Z [crawl4ai] 2026-08-15 21:01:46,939 - monitor - INFO - Started persistence worker
2026-08-15T21:01:47.017814506Z [crawl4ai] 2026-08-15 21:01:47,017 - crawler_pool - INFO - 🔥 Creating permanent default browser
2026-08-15T21:01:50.747436983Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:01:50.748710952Z [crawl4ai] [2026-08-15 21:01:50 +0000] [15] [INFO] Application startup complete.
2026-08-15T21:03:00.327397582Z [crawl4ai] 2026-08-15 21:03:00,325 - crawler_pool - INFO - 🆕 Creating new browser in cold pool (sig=442b44b0, mem=54.1%)
2026-08-15T21:03:01.155105112Z [crawl4ai] [INIT].... → Crawl4AI 0.9.2
2026-08-15T21:03:03.117785403Z [crawl4ai] [FETCH]... ↓ https://example.com | ✓ | ⏱: 1.91s
2026-08-15T21:03:03.131563810Z [crawl4ai] [SCRAPE].. ◆ https://example.com | ✓ | ⏱: 0.01s
2026-08-15T21:03:03.137049092Z [crawl4ai] [COMPLETE] ● https://example.com | ✓ | ⏱: 1.93s
2026-08-15T21:03:04.369252854Z [crawl4ai] [COMPLETE] ● Database backup created at: /home/appuser/.crawl4ai/crawl4ai.db.backup_20260815_210303
2026-08-15T21:03:04.369286628Z [crawl4ai] [INIT].... → Starting database migration...
2026-08-15T21:03:04.371522114Z [crawl4ai] [COMPLETE] ● Migration completed. 0 records processed.
2026-08-15T21:03:13.887750251Z [crawl4ai] 2026-08-15 21:03:13,887 - crawler_pool - INFO - ❄️ Using cold pool browser (sig=442b44b0)
2026-08-15T21:03:20.452682906Z [crawl4ai] [FETCH]... ↓ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 6.54s
2026-08-15T21:03:22.252476007Z [crawl4ai] [SCRAPE].. ◆ https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 1.80s
2026-08-15T21:03:22.278548502Z [crawl4ai] [COMPLETE] ● https://github.com/NousResearch/hermes-agent | ✓ | ⏱: 8.37s
2026-08-15T21:03:22.656568446Z [crawl4ai] 2026-08-15 21:03:22,656 - crawler_pool - INFO - ⬆️ Promoting to hot pool (sig=442b44b0, count=3)
2026-08-15T21:03:23.555156287Z [crawl4ai] [FETCH]... ↓ https://www.wikipedia.org/ | ✓ | ⏱: 0.87s
2026-08-15T21:03:23.697964059Z [crawl4ai] [SCRAPE].. ◆ https://www.wikipedia.org/ | ✓ | ⏱: 0.14s
2026-08-15T21:03:23.717358242Z [crawl4ai] [COMPLETE] ● https://www.wikipedia.org/

[maxinfo337]

but i had problem to connect the mcp ! it send 127.0.0.1 to the proxy ! so it don"t work ... even if i had in "noproxy" variable 127.0.0.0/8, i had to add 127.0.0.1 ... it's seem don't understand mask ! , after this change the mcp works correctly ;)

[SohamKukreti]

Awesome, thanks for testing it so quickly! 🎉
Regarding the MCP issue, that one's actually not coming from this fix. That connection is your MCP client talking to crawl4ai server
on 127.0.0.1:11235 so it never goes through the code in this PR (it only handles the server's outbound crawl traffic). Your client saw HTTP_PROXY and tried to reach 127.0.0.1 via the corporate proxy. The 127.0.0.0/8 not matching is a classic gotcha: there's no real standard for NO_PROXY, and a lot of tools simply don't understand CIDR masks. Go-based tools usually do, but Python and most Node clients just do plain string matching, so a mask entry silently matches nothing. Adding the literal 127.0.0.1 like you did is the right fix, and honestly the most portable way to write these lists.