Repository navigation
Conversation
Bulk load batches (up to ~10 MB of CSV) upload from the browser through the API server, and upload bandwidth is often the bottleneck. Web batches over 64 KB are now gzipped, and the server forwards the compressed bytes to the Bulk API, which decodes them itself. The server never decompresses the body, so the inflate: false hardening on every other parser stays in place. Desktop, extension and canvas call Salesforce in-process and are unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2191.
Bulk load batches (up to ~10 MB of CSV each) upload from the browser through the API server, and on slow or asymmetric connections the upload is the bottleneck. On web, batches over 64 KB are now gzipped in the browser (
CompressionStream). The server forwards those bytes to Salesforce untouched, and the Bulk API decodes them itself. CSV usually shrinks 5–10× with gzip.The server never decompresses anything.
gzipBodyPassthroughis mounted only onPOST /api/bulk/:jobId, before the global parsers. It handles requests that aretext/csvwithContent-Encoding: gzip: it reads the raw bytes up to the existing 30 MB limit, checks the gzip magic bytes, and setsreq.bodyto the Buffer. The global parsers then skip that request, because body-parser checksisFinished(req)first.inflate: falsefrom fix: reject compressed request bodies in every body parser #2191, so any other compressed body still gets a 415.Content-Encoding: gzipthroughApiBulk.addBatchToJob(new optionalcontentEncodingparam).Desktop / extension / canvas: no change. They set a custom axios adapter and call Salesforce in-process, so the client only compresses when
AxiosAdapterConfig.adapteris unset. Their in-process routers read non-JSON bodies as text, so sending them gzip would corrupt the data.Batch size limits are unchanged: the client still splits batches at ~9.5M characters before compressing, and Salesforce enforces its 10 MB limit on the decompressed batch.
Verified
/job/{id}/batchendpoint directly on the dev org withContent-Encoding: gzip: 201, and the rows were inserted (cleaned up afterwards)gzip-body-passthrough.middleware.spec.ts: gzip passes through byte-for-byte, plain CSV still goes through the regular parser, non-gzip bytes get 400, over-limit gets 413,brstill gets 415Bulk Job - Add gzipped Batch: the batch reachesCompletedwith 1 processed, 0 failed (pnpm e2e:local bulk.api.spec.ts; the other bulk API tests pass too)nx affected -t typecheckand unit tests for api, salesforce-api, shared-utils, shared-data, shared-constants, load-records and load-records-multi-object passNot verified: a manual web Load Records run with a batch over 64 KB. The E2E test calls the API directly, so it doesn't exercise the browser-side
CompressionStreampath.