Repository navigation
Conversation
…ed on main) Signed-off-by: Davide Polato <[email protected]>
Signed-off-by: Davide Polato <[email protected]>
…rpc.v2 lives in one module Signed-off-by: Davide Polato <[email protected]>
…stream PARTIAL_TIMEOUT is a success-category PipesResult status (the deadline hit, what was parsed is emitted); the mapper reported it as FAILED. fetchAndParseServerSideStreaming never called onCompleted(), so a v2 streaming client waited for its deadline after the reply. Signed-off-by: Davide Polato <[email protected]>
Signed-off-by: Davide Polato <[email protected]>
Signed-off-by: Davide Polato <[email protected]>
…k reports Signed-off-by: Davide Polato <[email protected]>
Signed-off-by: Davide Polato <[email protected]>
Signed-off-by: Davide Polato <[email protected]>
ParseBytes parses bytes sent in the request and returns the same Document as FetchAndParse. Content up to pipes.maxInlineBytes goes to the forked pipes worker with the request. Larger content goes through a temporary file, read by a file-system fetcher that the server registers under a reserved "__" id. GetFetcher, DeleteFetcher, ListFetchers and FetchAndParse treat "__" ids as unknown, and SaveFetcher refuses them. ParseBytes sends nothing to an emitter. On stop, the server closes the embedded Ignite node last, after the spool fetcher leaves the config store. New grpc settings: parseBytesMaxContentBytes and maxInboundMessageBytes. Signed-off-by: Davide Polato <[email protected]>
…tus, interrupted calls - MetadataTagger: a declared-boolean value that is not true/false, yes/no or 1/0 now falls the whole entry back to strings, like the other declared types, instead of arriving as a guessed false. - DocumentBuilder: PARSE_EXCEPTION_NO_EMIT maps to FAILED. Under the ON_PARSE_EXCEPTION.SKIP tika-grpc sends it is a parse that threw and carries no metadata, so there is nothing partial to return. - v1 and v2 FetchAndParse: when the fetch and parse is interrupted there is no outcome; close the call with UNAVAILABLE instead of completing a unary call without a reply. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Davide Polato <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
buf lint reported 17 errors, all in the new v2 protos (v1 and the pipes protos are untouched): - ParseStatus.Status values take the STATUS_ prefix, and the zero value is STATUS_UNSPECIFIED. - The service is TikaV2Service, so the generated class is TikaV2ServiceGrpc. - Each RPC has its own request and response message (FetchAndParseRequest/Response, FetchAndParseServerSideStreaming..., FetchAndParseBiDirectionalStreaming...). They carry the same fields under the same numbers as before, so the wire format does not change. TikaGrpcV2ServerImpl now builds every reply through one helper, and the tests, READMEs and CHANGES follow the new names. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Brings in the buf lint STANDARD rename from TIKA-4766 (and the upstream/main merge before it) and moves ParseBytes onto the new names: - ParseBytes is an RPC on TikaV2Service and returns ParseBytesResponse (was ParseBytesReply); the fields and their numbers are unchanged. - TikaGrpcV2ServerImpl keeps TIKA-4766's shared reply helper, which now calls runPublicFetchAndParse so the FetchAndParse RPCs still refuse the reserved "__" fetcher ids. - Tests use TikaV2ServiceGrpc and the STATUS_ enum values; README and CHANGES say TikaV2Service.ParseBytes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Keeps both sides of the TikaGrpcServerTest addition (the v2 terminal signal tests and the TIKA-5001 file store test) and files the TIKA-4766 CHANGES entry under SERVER, PIPES AND OPERATIONS. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3iXWuDQ3wg2AiEbD6t
v2 returns the Document and never emits, so the field was accepted and silently ignored. Field 4 and the name are reserved so they can return later. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BGnDTWcJiFwxgpvRQGSJ6L
- v2 FetchAndParse puts the pipes message (fetch failure, parse exception, timeout, crash) or the container stack trace of a partial parse into ParseStatus.errors instead of a generic "No metadata returned". - A client cancel interrupts the parse running for the call (through the call Context, which fires while the handler is blocked), so a cancelled call no longer holds a pipes worker; later requests on a cancelled call do not start. - The bidirectional stream drops requests after it closed the call with an error instead of calling onNext/onCompleted on a closed call. - v2 answers NOT_FOUND for an unknown fetcher and INTERNAL (cause logged, not returned) for unexpected failures. - Mapper: a multi-valued title/description/rights/created/modified keeps every value in the tagged tail instead of dropping all but the first. - Tests for a failing document, a blank document, EMPTY_OUTPUT, cancellation and bidi after an error. The ParseBytes half of this change goes on the TIKA-4795 branch. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3iXWuDQ3wg2AiEbD6t
Brings in upstream/main (TIKA-5001 step 1) and the TIKA-4766 v2 fixes, and resolves the conflicts with the ParseBytes half of the same change: - ParseBytes puts the pipes message or the container stack trace into ParseStatus.errors, with the spool directory scrubbed out. - ParseBytes answers INTERNAL (cause logged, not returned) instead of rethrowing unexpected failures as UNKNOWN, and shares the v2 cancellation. - FetchAndParse refuses fetcher ids in the reserved __ namespace, so the spool fetcher fails like an unknown id (NOT_FOUND on v2). - Tests for a failing and a blank ParseBytes document and spool scrubbing. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3iXWuDQ3wg2AiEbD6t
Since TIKA-5001 step 1, config-file components stay out of the config store. The ParseBytes spool fetcher comes from the effective config file, so it never reaches a shared store, and a server shutting down only looks it up. The shutdown-order tests verify the lookup instead of a remove, and the shared-store test checks that neither server's spool fetcher lands in the store. The test for a departing server's stale store copy is gone: there is no store copy to go stale. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3iXWuDQ3wg2AiEbD6t
.github/workflows/buf.yml runs buf lint on both proto modules and buf breaking against the PR base branch (npx @bufbuild/buf, no extra action). The breaking check compares <module>/src/main/proto so a base without a buf.yaml doesn't report every file as deleted. tika-grpc/buf.yaml covers the v1 tika.proto: STANDARD minus the six naming rules its wire API can't meet without breaking existing clients, each commented. v1 is frozen; behaviour changes go to v2. The v2 module (tika-grpc-api) passes STANDARD with no exceptions; its unused googleapis dep is dropped so lint needs no BSR access. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3h7V1SgZzdVcFGJK8q
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3h7V1SgZzdVcFGJK8q
…ated success is partial A failed parse returned no origin.sha256 and no content_type. ParseBytes holds the bytes, so it now computes the SHA-256 itself when the parse recorded none, and detects the content type from the magic bytes and resource name (tika-core MIME registry) when the parse recorded none. Values the parse did record win. A truncated PDF came back STATUS_SUCCESS with its title gone: PDFBox rebuilds a PDF whose trailer is cut off without any exception or warning, and the Info dictionary is lost. Tika has no signal for that, but the caller does: a success on content sent with truncated=true is now STATUS_PARTIAL with an error saying so. pipes_status keeps the raw value. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3h7V1SgZzdVcFGJK8q
… on cancel A busy pool, a pipes runtime or fetcher that failed to start, and a fetcher the worker cannot find now close a v2 call with RESOURCE_EXHAUSTED, UNAVAILABLE or NOT_FOUND, so retry policies keyed on status codes see them. A document the worker timed out, ran out of memory or crashed on stays a STATUS_FAILED Document. Parses take a permit per pipes client before asking the pipes parser for a client, so waiting for a worker and running on one are told apart. A cancelled or expired v2 call stops waiting at once; a parse already on a worker finishes and its result is dropped, instead of being interrupted, which in per-client mode restarted the worker JVM. grpc.cancelRunningParse restores the interrupt. v2 errors carry the first line of the pipes diagnostic, cut to 1024 characters, unless parse-context.exception-reporting is configured. v1 replies are unchanged: pool exhaustion is still an in-band CLIENT_UNAVAILABLE_WITHIN_MS. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
At most grpc.parseBytesMaxConcurrent ParseBytes requests (twice pipes.numClients by default) are in progress at once. One more is refused with RESOURCE_EXHAUSTED before its content is spooled, so a burst of large requests no longer holds its bytes in heap and java.io.tmpdir for up to maxWaitForClientMillis each. ParseBytes now takes the same path as v2 FetchAndParse: a busy pool or a pipes runtime that failed to start closes the call with a status, and a cancelled call lets a running parse finish instead of restarting the worker. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
When a ParseBytes parse failed before detecting a type, the server's fallback used magic bytes only, so a gzipped WARC came back as application/gzip, with or without a .warc.gz name. The fallback now runs the gzip specialization detector on gzip content (it reads at most 1 KiB of compressed bytes), so it reports application/warc+gz or application/arc+gz. It still skips container detectors, which open whole zip and OLE files, because it runs in the server JVM. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
- A fetcher that does not exist is NOT_FOUND; one that exists but cannot be built or read from its store is UNAVAILABLE (was NOT_FOUND for both). additional_fetch_config_json for a missing fetcher is NOT_FOUND, checked before the config is merged. - v1 error_message for fetch failures is scrubbed to the first line, as v2's errors are, unless parse-context.exception-reporting is set. - The server loads pipes config with PipesConfig.load, so the parent sees exception-reporting and timeout limits for the results it builds. - Cancellation.end always clears the thread's interrupt, so a shutdown interrupt does not leak into the next call on a pooled thread. - A FAILED or PARTIAL Document always carries at least one error. - SourceOrigin.byte_size is optional: unset is unknown, 0 is empty. - buf workflow: checkout pinned by SHA, buf binary checked by sha256, breaking check also on push, base buf.yaml used for the check. - tika.proto is byte-identical to main. - Tests for UNAVAILABLE fetchers, FETCHER_INITIALIZATION_EXCEPTION, status codes on server-streaming and bidi, CRLF diagnostics, interrupt clearing, the v1 scrub and old clients sending emitter_id. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
- The admission permit is an idempotent ParseBytesPermit, held by v2 until the reply is sent, so hashing and fallback detection are bounded by parseBytesMaxConcurrent too. - Payload.close is idempotent: a second close no longer decrements the spool's open-file count and deletes a directory another payload uses. - A spool write that fails (disk full, unwritable directory) is RESOURCE_EXHAUSTED, not INTERNAL; the cause stays in the server log. - At most the admitted size is read from the content stream. - GZipSpecializationDetector inflates at most 64 bytes of its 1 KiB prefix, instead of up to about 1 MiB one byte at a time. - Without maxInboundMessageBytes the inbound limit is the ParseBytes content limit plus 1 MiB (65 MiB by default) instead of gRPC's 4 MiB, so the default content limit is reachable and the startup warning is checked against the limit actually applied. - Truncated content gets the truncation note whatever the status; only SUCCESS becomes PARTIAL. - origin.sha256 is always lowercase hex SHA-256: a pipeline digest in another form is replaced by the server's own. - Tests for each, plus concurrent permit release and gzip edge cases. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
resolveFetcherId looks the fetcher up once and returns its configured id, so the fetch-and-parse paths (v1 and v2) share a single call site that will release a lease once getFetcher returns one (TIKA-5001 step 2). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
Brings in resolveFetcherId. The spool test reads the spool fetcher's basePath from the effective config file instead of through the fetcher manager, so it holds no fetcher. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
A web-crawl WARC of 100 MB or more is common, so the default parseBytesMaxContentBytes is now 256 MiB and the derived inbound limit 257 MiB. The startup checks now also warn when parseBytesMaxConcurrent requests of the full size could outgrow the heap or the spool directory's free space. dev-tika-config.json drops its 100 MiB inbound limit, which is now below the default content limit. The README lists the defaults and what each costs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
CHANGES.txt still gave 64 MiB for parseBytesMaxContentBytes and the ParseBytesRequest.content comment still gave gRPC's 4 MiB as the inbound limit; both now give the derived default. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
The example set a 100 MiB inbound limit, now below the 256 MiB content default; it now shows taking 1 GiB WARCs, inbound 1 MiB above content. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3mcUt3NdoxRq6LzNYB
tika-grpc/docs/v2-document-contract.md explains why v2 exists, how values get their types, which outcomes are a Document status and which a gRPC status, cancel behaviour, and how buf keeps v1 frozen. The README links it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3jQgutA3ZS6Ttgy8hR
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3jQgutA3ZS6Ttgy8hR
tika-grpc/docs/parse-bytes.md explains why ParseBytes exists, how the bytes reach the worker (inline or the reserved spool fetcher), how memory is bounded, the fallback digest and type, and the truncated flag. The README links it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3uD3jQgutA3ZS6Ttgy8hR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #2961. Adds TikaV2Service.ParseBytes: the caller sends the bytes it holds and gets the same Document back, with nothing fetched. It's for crawlers that want a parse of exactly what they captured. The proto change is additive.
Content is capped (256 MiB by default) and concurrency is capped, with RESOURCE_EXHAUSTED before anything is spooled. When a parse fails, ParseBytes still fills in the digest and content type, and a caller can mark content as truncated. There's a small change to tika-core's GZipSpecializationDetector.
Design and reasoning: tika-grpc/docs/parse-bytes.md. Start with ParseBytesRequest, admitParseBytes and ParseBytesSpool.
Tested with
mvn clean verify -pl tika-grpc-api,tika-grpc-mapper,tika-grpc(122 tika-grpc tests) and from StormCrawler with https://github.com/dpol1/tika-demo.Davide (dpol1) started this and wrote the first version.