Skip to content

feat(load): gzip large bulk batches and pass them through to Salesforce - #2192

Open
paustint wants to merge 1 commit into
mainfrom
feat/bulk-gzip-passthrough
Open

paustint wants to merge 1 commit into
mainfrom
feat/bulk-gzip-passthrough

Conversation

@paustint

Copy link
Copy Markdown
Contributor

Follow-up to #2191.

Bulk load batches (up to ~10 MB of CSV each) upload from the browser through the API server, and on slow or asymmetric connections the upload is the bottleneck. On web, batches over 64 KB are now gzipped in the browser (CompressionStream). The server forwards those bytes to Salesforce untouched, and the Bulk API decodes them itself. CSV usually shrinks 5–10× with gzip.

The server never decompresses anything.

  • gzipBodyPassthrough is mounted only on POST /api/bulk/:jobId, before the global parsers. It handles requests that are text/csv with Content-Encoding: gzip: it reads the raw bytes up to the existing 30 MB limit, checks the gzip magic bytes, and sets req.body to the Buffer. The global parsers then skip that request, because body-parser checks isFinished(req) first.
  • Every other parser keeps inflate: false from fix: reject compressed request bodies in every body parser #2191, so any other compressed body still gets a 415.
  • The controller passes Content-Encoding: gzip through ApiBulk.addBatchToJob (new optional contentEncoding param).

Desktop / extension / canvas: no change. They set a custom axios adapter and call Salesforce in-process, so the client only compresses when AxiosAdapterConfig.adapter is unset. Their in-process routers read non-JSON bodies as text, so sending them gzip would corrupt the data.

Batch size limits are unchanged: the client still splits batches at ~9.5M characters before compressing, and Salesforce enforces its 10 MB limit on the decompressed batch.

Verified

  • Probed the Bulk API v1 /job/{id}/batch endpoint directly on the dev org with Content-Encoding: gzip: 201, and the rows were inserted (cleaned up afterwards)
  • New unit spec gzip-body-passthrough.middleware.spec.ts: gzip passes through byte-for-byte, plain CSV still goes through the regular parser, non-gzip bytes get 400, over-limit gets 413, br still gets 415
  • New E2E test Bulk Job - Add gzipped Batch: the batch reaches Completed with 1 processed, 0 failed (pnpm e2e:local bulk.api.spec.ts; the other bulk API tests pass too)
  • nx affected -t typecheck and unit tests for api, salesforce-api, shared-utils, shared-data, shared-constants, load-records and load-records-multi-object pass

Not verified: a manual web Load Records run with a batch over 64 KB. The E2E test calls the API directly, so it doesn't exercise the browser-side CompressionStream path.

Bulk load batches (up to ~10 MB of CSV) upload from the browser through the
API server, and upload bandwidth is often the bottleneck. Web batches over
64 KB are now gzipped, and the server forwards the compressed bytes to the
Bulk API, which decodes them itself.

The server never decompresses the body, so the inflate: false hardening on
every other parser stays in place. Desktop, extension and canvas call
Salesforce in-process and are unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant