TIKA-4809: reconcile server/pipes docs and CHANGES with 4.0.0 behavior

Author: tballison

CHANGES.txt

@@ -17,7 +17,51 @@ Release 4.0.0 - ???
      completing the Markdown default (TIKA-4663); use /tika/xml for XHTML (TIKA-4809).
 
    * tika-server: request bodies are now capped by maxRequestSizeBytes,
-     defaulting to 1 GiB; larger requests are rejected with 413 (TIKA-4809).
+     defaulting to 1 GiB; larger requests are rejected with 413, including
+     over-limit chunked uploads, which previously surfaced as an empty 500
+     (TIKA-4809).
+
+   * tika-server: a /meta request with no Accept header now returns JSON;
+     3.x returned CSV. CSV is still available via Accept: text/csv (TIKA-4809).
+
+   * tika-server: the raw /tika family's 422 responses carry the extracted
+     content only; the exception is no longer appended to the body -- use
+     /rmeta for the structured exception (TIKA-4809).
+
+   * tika-server: /async validates fetcher/emitter ids at POST time (400),
+     rejects a batch larger than the queue's total capacity with 400 instead
+     of throttling it, and one bad tuple no longer stops the async workers
+     (TIKA-4809).
+
+   * tika-server: /pipes returns 400 with the reason for a malformed request
+     body and rejects emit strategies other than EMIT_ALL, whose passed-back
+     data the /pipes response cannot carry (TIKA-4809).
+
+   * tika-server: the 'endpoints' allowlist now also gates SPI-provided
+     resources; a discovered resource binds only when its root endpoint is
+     enabled (TIKA-4809).
+
+   * FetchEmitTuple JSON now names the per-tuple parse context "parse-context"
+     (was "parseContext") and rejects unknown tuple fields with an error naming
+     the field (TIKA-4809).
+
+   * The pipes config keys staleFetcherTimeoutSeconds and
+     staleFetcherDelaySeconds have been removed; a config still carrying them
+     fails startup (TIKA-4809).
+
+   * ExceptionUtils.trimMessage has been removed from tika-core; it moved into
+     tika-eval-core (TIKA-4809).
+
+   * BasicContentHandlerFactory.parseHandlerType now throws
+     IllegalArgumentException for an unrecognized handler name instead of
+     silently returning the supplied default (TIKA-4809).
+
+   * HttpClientFactory's inert redirect-host allowlist accessors
+     (get/setAllowedHostsForRedirect) have been removed (TIKA-4809).
+
+   * TimeoutLimits: progressTimeoutMillis of 0 combined with a positive
+     totalTaskTimeoutMillis is now rejected at config load; it would kill
+     every task immediately (TIKA-4809).
 
    * The http-fetcher now verifies TLS certificates and hostnames by default;
      set verifySsl:false to opt out (TIKA-4809).

docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc

@@ -118,7 +118,8 @@ curl -s -X PUT -H "Accept: application/json" -T testPDF.pdf http://localhost:999
 curl -s -X PUT -T testPDF.pdf http://localhost:9998/meta/Content-Type
 ----
 
-*Expected:* `Content-Type,application/pdf`
+*Expected:* JSON containing only the requested field: `{"Content-Type":"application/pdf"}`
+(the header-less default is JSON; add `-H "Accept: text/csv"` for `Content-Type,application/pdf`)
 
 === Test 9: PUT /rmeta
 
@@ -155,7 +156,8 @@ curl -s -X PUT -T test_recursive_embedded.docx http://localhost:9998/unpack/all
 unzip -l /tmp/unpack.zip
 ----
 
-*Expected:* ZIP file containing extracted embedded files plus `__TEXT__` and `__METADATA__` files.
+*Expected:* ZIP file containing the extracted embedded files, a `*.metadata.json` entry
+per file, and the original container document.
 
 === Test 13: GET /parsers
 

docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc

@@ -110,7 +110,14 @@ The `/stream` and `/string` suffixes never described anything a caller could cho
 
 Collapsing them also removes the `/detect/stream` vs `/detectors` near-collision. `/detectors` is unchanged -- it lists the server's configured detectors and does not detect anything.
 
-**Migration:** drop the suffix. `PUT /detect/stream` becomes `PUT /detect`; `PUT /language/stream` and `PUT /language/string` both become `PUT /language`. Request bodies, headers, and responses are unchanged.
+**Migration:** drop the suffix. `PUT /detect/stream` becomes `PUT /detect`; `PUT /language/stream` and `PUT /language/string` both become `PUT /language`. Request bodies and headers are unchanged, but behavior is not:
+
+* `/detect` now runs in the forked worker pool (detection opens containers over
+  caller-supplied bytes), so it can return `429` or `503` with `Retry-After`, and `413`,
+  like the parsing endpoints. A failure reading the body is now a `500`; 3.x returned
+  `200` with `application/octet-stream` as if detection had succeeded.
+* `/language` caps detection input at the first 100,000 characters and uses the default
+  `LanguageDetector` on the classpath; 3.x pinned it to Optimaize.
 
 === Handler-Type Changes on `/tika`
 
@@ -158,7 +165,12 @@ The HTTP status codes are also more precise:
   header, not `503` or `200`.
 * An unknown or reserved fetcher/emitter (`FETCHER_NOT_FOUND`, `EMITTER_NOT_FOUND`)
   returns `400`, not `500` — the request is permanently malformed, so retrying will not help.
-* A body over `maxRequestSizeBytes` (`PAYLOAD_LIMIT_EXCEEDED`) returns `413`.
+* Two distinct limits return `413`. A request body over `maxRequestSizeBytes` is
+  rejected by a request filter with a plain-text `413` -- both a declared
+  `Content-Length` over the limit and a chunked body that turns out to be too large.
+  Separately, a parse *result* too large for the pipes IPC channel
+  (`pipes.maxIpcPayloadBytes`) returns `413` with the JSON status body
+  `{"status":"PAYLOAD_LIMIT_EXCEEDED"}`.
 * `/pipes` and `/async` now signal the outcome through the HTTP status (the same
   `429`/`503`/`400`/`413` mappings, plus `Retry-After`) instead of always returning `200`
   with the failure only in the body.
@@ -183,9 +195,22 @@ the exception when there is one. It previously used a `/pipes`-only shape
 and a stringified `emitted`). A parse that threw is now reported by its status enum, not as
 `"ok"`.
 
+`/pipes` also rejects with `400`: a malformed request body (the reason is in the response),
+and any emit strategy other than `EMIT_ALL` -- the `/pipes` response carries only
+status and message, so a passback strategy would silently discard the parsed data. Use
+`/rmeta` if you want the data passed back.
+
 **`/async` request body.** `POST /async` now requires an object envelope,
 `{"tuples":[ ... ]}`, instead of a bare JSON array; the envelope leaves room for future
-batch-level fields. A bare array (or any body without a `tuples` array) is rejected with `400`.
+batch-level fields. A bare array (or any body without a `tuples` array) is rejected with `400`,
+as is a tuple naming a fetcher or emitter the server does not have -- validated at POST time,
+before anything is queued. A batch larger than the queue's *total* capacity is also a `400`
+telling the caller to split it: retrying cannot help. `429` with `Retry-After` is reserved
+for transient fullness, where the same batch can succeed later.
+
+**`FetchEmitTuple` wire format.** The per-tuple parse-context key is `parse-context`;
+3.x used `parseContext`. An unknown field anywhere in a tuple is rejected with a `400`
+naming the field and listing the known ones, instead of being silently ignored.
 
 === `/meta` Is Now Pipes-Backed
 
@@ -209,6 +234,10 @@ should check the new status codes above. Clients that inspected the response bod
 for error text should check `tk:exception:container-exception` (full-object
 endpoints) or the `422` body (`/meta/\{field}`).
 
+**The default representation is now JSON, not CSV.** A `/meta` request without an
+`Accept` header returned CSV in 3.x; it now returns JSON. CSV is still available with
+`Accept: text/csv`.
+
 Two changes to the returned metadata come with this, neither of which produces an
 error:
 
@@ -219,7 +248,8 @@ error:
   content handler, so there is no text for a language detector to work from.
 +
 **Migration:** configure a language-detection metadata filter
-(`charsoup-metadata-filter`, `optimaize`, or `opennlp`) and use `/rmeta` or
+(`charsoup-metadata-filter`, `optimaize-metadata-filter`, or
+`open-nlp-metadata-filter`) and use `/rmeta` or
 `/tika/json`, which capture content. The detected value arrives as
 `tk:detected-language`, with `tk:detected-language-confidence`. Note that these
 filters read `tk:content`, so they are no-ops on `/meta` and on any endpoint
@@ -269,9 +299,12 @@ The following `TikaServerConfig` options have been removed:
   digests there instead of in the `server` section.
 * `idBase` - Renamed to `id`.
 * `preventStopMethod`, `maxFiles`, `javaPath`, `maxRestarts`, `numRestarts`,
-  `forkedStatusFile`, `maxForkedStartupMillis`, `tmpFilePrefix` - Vestigial options from the
-  pre-4.0 spawn-child server model, which no longer exists. Delete them from your config;
-  leaving any in place now fails startup, because unrecognized config keys are rejected.
+  `forkedStatusFile`, `maxForkedStartupMillis`, `tempFilePrefix`, `noFork` - Vestigial
+  options from the pre-4.0 spawn-child server model, which no longer exists (the `-noFork`
+  CLI flag is gone too). Delete them from your config; leaving any in place now fails
+  startup, because unrecognized config keys are rejected.
+* `port` is now a single integer. The 3.x multi-instance port ranges and lists
+  (`-p 9995-9998`, `-p 9995,9997`) are no longer supported; run one server per port.
 
 === Configuration via HTTP Headers Removed
 
@@ -377,6 +410,12 @@ The capabilities are two default-`false` flags in the `server` section:
 
 `/status` is no longer gated: it exposes only aggregate counters, so it is enabled simply by listing `status` under `endpoints`.
 
+The `endpoints` allowlist now also gates SPI-provided resources: a discovered resource
+whose root path matches a named endpoint binds only when that endpoint is enabled (e.g.
+the `application/rdf+xml` XMP resource serves `/meta`, so omitting `meta` removes it too).
+An SPI resource with a custom root path still loads unconditionally -- installing the jar
+is the opt-in.
+
 **Migration:** if your config selects `pipes` or `async`, add `"allowPipes": true`; if you rely on per-request config, add `"allowPerRequestConfig": true`:
 
 [source,json]
@@ -422,6 +461,10 @@ and want `/pipes`/`/async` to fetch documents from a directory you control:
 }
 ----
 
+The 3.x `pipes` keys `staleFetcherTimeoutSeconds` and `staleFetcherDelaySeconds` are gone.
+A pipes config still carrying either fails startup: unknown pipes keys are rejected, not
+ignored.
+
 [IMPORTANT]
 ====
 Set `basePath` to a directory that contains only the documents you intend the

docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc

@@ -45,7 +45,7 @@ If you have build scripts or container images that drop in just the jar, update
 In 3.x the default content handler produced XHTML/XML. In 4.x the default is **Markdown** everywhere:
 
 * `tika-app` outputs Markdown by default (was XHTML). Pass `-x`/`--xml`, `-h`/`--html`, or `-t`/`--text` to choose another format.
-* `tika-server` — the `/tika` and `/rmeta` endpoints return Markdown content by default (was XHTML/XML). Use an explicit handler path (`/tika/xml`, `/rmeta/xml`, ...) to choose another format.
+* `tika-server` -- the `/tika` and `/rmeta` endpoints return Markdown content by default. In 3.x, `/rmeta` returned XML content, and a bare `/tika` PUT routed among plain text, HTML, and XHTML by `Accept` header -- nondeterministically for `*/*`. Use an explicit handler path (`/tika/xml`, `/rmeta/xml`, ...) to choose another format.
 * The async/pipes CLI emits Markdown by default (was plain text). Use `--handler x` (etc.) to choose another format.
 
 If you parse the extracted content programmatically and expect XHTML/XML, request it explicitly as shown above (TIKA-4663).

docs/modules/ROOT/pages/pipes/timeouts.adoc

@@ -271,7 +271,11 @@ the rest are safe by construction.
 *Rejected at config load or startup:*
 
 * Negative values for `totalTaskTimeoutMillis` or `progressTimeoutMillis` (zero is
-  accepted -- see below).
+  accepted in some combinations -- see below).
+* `progressTimeoutMillis: 0` with a *positive* `totalTaskTimeoutMillis` -- the stall
+  detector would fire at the first check, killing every task immediately despite the
+  remaining total budget. Rejected at config load, and again at task start for
+  programmatically-built limits.
 * `pipes.socketTimeoutMillis` less than or equal to `pipes.heartbeatIntervalMillis` -- the client
   would kill a healthy server between heartbeats.
 * Unknown or renamed configuration fields (including pre-4.0 `timeoutSeconds` names) --
@@ -300,11 +304,11 @@ the rest are safe by construction.
 
 *Accepted without warning (a coherent edge case, not a misconfiguration):*
 
-* Zero for `totalTaskTimeoutMillis` or `progressTimeoutMillis` -- legal, and useful
-  mainly in tests: it means the budget is already exhausted, so the task fails at the
-  next operation or embedded-document boundary. It does *not* disable the timeout. A
-  zero `progressTimeoutMillis` is in fact the *strictest* possible stall detector (any
-  silence at all trips it) -- the opposite of disabling it. To make stall detection
+* Zero for *both* timeouts -- legal, and useful mainly in tests: it means the budget is
+  already exhausted, so the task fails at the next operation or embedded-document
+  boundary. A zero `totalTaskTimeoutMillis` alone is likewise legal (the total budget is
+  spent). Zero does *not* disable a timeout -- and a zero `progressTimeoutMillis` next
+  to a positive total is rejected outright (see above). To make stall detection
   effectively inert, set `progressTimeoutMillis` at or above `totalTaskTimeoutMillis`
   (see above).
 * `Long.MAX_VALUE` (`9223372036854775807`) for `totalTaskTimeoutMillis` or
@@ -371,9 +375,8 @@ rename with the existing value is correct for all of them: `PipesConfig`'s
 flag (now `--timeoutMillis`), the DWG parser's `dwgReadTimeout` (now `timeoutMillis`,
 default unchanged at 300000), and `ExternalParser`'s `timeoutMs` (now `timeoutMillis`, a
 rename that landed in 4.0 itself so this doc previously lagged the code). Fields that are
-genuinely seconds-based and stayed that way (e.g. `staleFetcherTimeoutSeconds`, JDBC's
-`queryTimeoutSeconds`, which feeds `java.sql.Statement.setQueryTimeout(int seconds)`
-directly) were deliberately left alone.
+genuinely seconds-based and stayed that way (e.g. JDBC's `queryTimeoutSeconds`, which feeds
+`java.sql.Statement.setQueryTimeout(int seconds)` directly) were deliberately left alone.
 
 `ProcessUtils.checkCommand()` -- used by several detectors and parsers
 (`FileCommandDetector`, `GDALParser`, etc.) to probe whether an external binary is

docs/modules/ROOT/pages/using-tika/server/index.adoc

@@ -191,13 +191,12 @@ It is therefore outside the process isolation that protects `/tika`, `/rmeta`, `
 one worker.
 
 `/language` caps detection at the first 100,000 characters, since accuracy saturates well
-before that. That bounds the CPU per request, but *not* the memory: the request body is read
-into the server's heap before the cap applies. Bounding the body itself is what
-`maxRequestSizeBytes` is for; it defaults to 1 GiB, so one request cannot fill the temp
-directory or the heap with a small body, though enough concurrent large ones still can. Raise
-it for larger documents, or set a negative value to remove the limit. Treat the server as
-available only to trusted callers, the same as the rest of the server — see
-xref:security.adoc[the security model]. If you do not need `/language`, omit it from `endpoints`.
+before that. It reads at most that much from the body without materializing the rest, so the
+cap bounds both the CPU and the heap used per request. `maxRequestSizeBytes` (default 1 GiB)
+still bounds what a request may send at all; raise it for larger documents, or set a negative
+value to remove the limit. Treat the server as available only to trusted callers, the same as
+the rest of the server — see xref:security.adoc[the security model]. If you do not need
+`/language`, omit it from `endpoints`.
 
 NOTE: `/detect` runs in a forked pipes worker like the parsing endpoints. Detection opens
 containers — zip, OPC, POIFS — over caller-supplied bytes, so it gets the same isolation,
@@ -249,7 +248,10 @@ not help; fix the fetcher/emitter id.
 
 |`413 Payload Too Large`
 |`PAYLOAD_LIMIT_EXCEEDED`
-|The request body exceeded `maxRequestSizeBytes`.
+|The parse *result* was too large for the pipes IPC channel (`pipes.maxIpcPayloadBytes`).
+Distinct from a request *body* over `maxRequestSizeBytes`, which a request filter rejects
+with a plain-text `413` (declared `Content-Length` and chunked bodies alike) before any
+parsing starts.
 
 |`500 Internal Server Error`
 |`FAILED_TO_INITIALIZE`, `FETCH_EXCEPTION`, `EMIT_EXCEPTION`,
@@ -364,7 +366,7 @@ Server behavior beyond host/port is controlled by a JSON config file passed via
 
 |`maxQueuePauseMillis`
 |`60000`
-|How long a POST to `/async` blocks when the queue is full before it is rejected with `429` (`Retry-After`).
+|How long a POST to `/async` blocks when the queue is full before it is rejected with `429` (`Retry-After`). A batch larger than the queue's total capacity is rejected immediately with `400` -- retrying cannot help; split the batch.
 
 |`requestLogLevel`
 |_empty (off)_
@@ -448,10 +450,12 @@ mitigation (scope `endpoints` to what you actually use, or set
 
 === Config Endpoint Protection
 
-By default, the `/config` family of endpoints that expose server configuration are
-disabled. These endpoints can reveal sensitive information about your server,
-including parser settings and system properties (see
-https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3271[CVE-2015-3271]).
+By default, the `/config` family of endpoints is disabled. These endpoints *accept*
+per-request configuration -- they do not reveal the server's own config -- and a
+caller-supplied config can enable dangerous operations, such as pointing parsers at
+attacker-chosen resources (see
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3271[CVE-2015-3271] for the
+class of risk).
 
 Protected endpoints include:
 
@@ -476,9 +480,9 @@ your config file's `server` section:
 WARNING: Only enable `allowPerRequestConfig` if you have secured access to Tika
 Server through network controls (firewalls, private subnets), a reverse proxy
 (nginx, Apache httpd), or
-xref:using-tika/server/tls.adoc[2-way TLS authentication]. Exposing config endpoints
-to untrusted networks can help attackers identify vulnerabilities and craft
-targeted attacks.
+xref:using-tika/server/tls.adoc[2-way TLS authentication]. Per-request configuration
+lets callers change how documents are parsed, widening what anyone with access to the
+server can do.
 
 === Pipes and Async Endpoints
 

tika-server/README.md

@@ -85,9 +85,12 @@ Usage examples from command line with `curl` utility:
 * Get all document attachments as ZIP-file:  
 `curl -v -T Doc1_ole.doc http://localhost:9998/unpack > /var/tmp/x.zip`
 
-* Extract metadata to CSV format:  
+* Extract metadata as JSON (the default):  
 `curl -T price.xls http://localhost:9998/meta`
 
+* Extract metadata as CSV:  
+`curl -T price.xls -H "Accept: text/csv" http://localhost:9998/meta`
+
 * Detect media type from CSV format using file extension hint:  
 `curl -X PUT -H "Content-Disposition: attachment; filename=foo.csv" --upload-file foo.csv http://localhost:9998/detect`