[tballison]

Proof of concept for grpc v2. This is 100% 🤖 generated.

  Stacked on #3448 (TIKA-5010) -- review only the last commit (6480d6f) until that merges.

In short: /rmeta as protobuf -- a flat list of
  documents with embedded paths, content outside metadata, every metadata value as a
  string with type/cardinality hints from tika-metadata-schema. Unary, server-streaming
  and bidi; bidi reports per-request failures in-band. Package v2alpha1 until it freezes.

  Also in here:
  - FetchParseRuntime: v1 and v2 share the request gates and pipes path; v2 never touches v1's code
  - buf lint/breaking CI for the v1 and v2 protos (adapted from #2961)
  - MetadataTypeChangesTest: a key's type can change only if type-changes.json declares it
  - docs: gRPC message-size limits and the v2alpha1 service

Thanks to @krickert and @dpol1, whose #2961 this builds on.

Links: Design:TIKA-4776 and doc.

[tballison]

The main missing piece is typed values. @krickert @dpol1 wdyt of this, potentially?

Proposal: typed values without locking in types

  v2alpha1 sends every metadata value as a string, with type and cardinality hints from Tika's metadata registry. To spare
  clients from parsing, the server can also send typed copies:

  message Values {
    repeated string values = 1;            // always present
    ValueType type = 2;
    Cardinality cardinality = 3;
    // set only when the hint is typed and every value parses
    repeated int64 integers = 4;
    repeated double reals = 5;
    repeated bool booleans = 6;
    repeated google.protobuf.Timestamp instants = 7;  // zoned dates only
  }

  - Strings stay the source of truth. Typed copies are derived from them. If a key's type changes, its typed copy moves to
    another field or disappears, and the strings don't change. Type changes are gated in tika-metadata-schema
    (type-changes.json).
  - Nothing is guessed. If any value fails to parse, there's no typed copy. Dates with no zone, and date-only values, stay
    strings, because a Timestamp can't represent "floating" without inventing UTC.
  - No top-level Dublin Core fields for now. With typed copies, dcterms:created and similar keys already arrive typed in
    metadata. Promoting document properties is how the format taxonomy creeps back in.
  - Structural tk: values may become fields. These describe the parse itself, not the document, and Tika owns them.
    Candidates: per-document exceptions, the content handler type, and a "content truncated" flag. They'd stay in metadata
    too, so it remains a full /rmeta mirror.

[krickert]

First - much better than what I did.

Second - I think we can avoid the string conversion completely

Here's a short version that builds on what you have so far:

Protobuf already has a value type, google.protobuf.Value (plus ListValue and Struct). I'd use that instead of a custom Values. It's a first-class type that every client + proto-nerd already knows, and you get real numbers, booleans and lists on the wire instead of strings with hints. The struct is smaller too. However yours is superior in one way - all numbers in Value are doubles - so integers can lose precision.

Instead - how about this?

In java, every protobuf message extends Message. So the plug-in point can be one interface, found through SPI:

 public interface MessageParser<T extends Message> {
     Class<T> messageType();
     List<Property> properties();   // the keys it writes and their types
     List<Metadata> parse(T message) throws TikaException;
 }

So you'd have to just make a plugin jar ships its own generated stubs and one of these. The parser maps the message into Metadata the same way any Tika parser does, in plain Java. Tika never sees the .proto, and there's no mapping config or reflection. The service takes a google.protobuf.Any and picks the parser by its type URL. It returns the same document list as FetchAndParse, with unary, streaming and bidi calls just like yours. I have it working on top of your PR and can share the branch.

The problem with swapping strings are the weirdo things that happen - dates and times are annoying :) And converting to strings is really heavy on the CPU under heavy load.

[tballison]

Y, dates and times are a mess. We do have the date normalizing filter that slaps a user configured timezone on values that don't have them (some image metadata???) so that strict consumers don't complain.

Can you tell me how value would work?

1. Value: swap our Values for protobuf's built-in type in the metadata map:
  message ParsedDocument {
    string embedded_path = 1;
    string content = 2;
    map<string, google.protobuf.Value> metadata = 3;   // or google.protobuf.Struct
  }
  "xmpTPg:NPages":   { number_value: 12 }
  "dc:title":        { string_value: "Report" }          // or list_value, if multi-valued
  "tk:parsed-by":    { list_value: { values: [{string_value:"a"}, {string_value:"b"}] } }
  "dcterms:created": { string_value: "2019-06-01T12:00:00Z" }   // no timestamp type

[tballison]

Going back to #2961, would this work?

 A oneof of list wrappers (#2961's shape):
  message MetadataValue {
    oneof values {
      StringValues strings = 1;
      IntegerValues integers = 2;
      RealValues reals = 3;
      BooleanValues booleans = 4;
      TimestampValues timestamps = 5;
    }
  } 

Then:

  map<string, MetadataValue> metadata = 3;

[tballison]

And, sorry, what is the plugin point? Are you proposing a different metadata schema/parser per file type?

[krickert]

So the plugin -

Protobufs are already strongly typed. Just write a java mapper class to make the object go from one type of object to antoher.

Using SPI, it will bring the java protobuf class into the classpath. If it is an Any object, you can just proxy it too if you want.

You really should try to have everything strongly typed and avoid converting it from type->string->type. You can just do type->type with this solution.

public class WarcRecordParser implements MessageParser<WarcRecord> {
    static final Property STATUS = Property.externalInteger("warc:http-status");

    public Class<WarcRecord> messageType() { return WarcRecord.class; }

    public List<Property> properties() { return List.of(STATUS); }

    public List<Metadata> parse(WarcRecord record) {
        Metadata m = new Metadata();
        m.set(TikaCoreProperties.SOURCE_PATH, record.getTargetUri());
        m.set(STATUS, record.getHttpStatus());
        m.set(TikaCoreProperties.TIKA_CONTENT, record.getText());
        return List.of(m);
    }
}

and also to serialize we'd just call pack():

ParseMessageResponse r = stub.parseMessage(
        ParseMessageRequest.newBuilder().setMessage(Any.pack(record)).build());

The idea is that you don't convert it to a string. Converting some types from data->string->data can cause serialization headaches.

[krickert]

I'm gonna yell at bots to do it both ways so you'll see.

[tballison]

Separate proto parser for each Tika parser?

[tballison]

On #3451 (comment), I'm now getting on board with locking down value types. I had been worried that grpc would limit our flexibility to change value types in base Tika, but we really haven't done that very often.

So, should I update this PR to follow the pattern with oneof() in #2961?

[krickert]

@tballison so I made an update to the branch - it's a throwaway one because I implement both strategies. I'll push it in a sec

[krickert]

• Branch: https://github.com/ai-pipestream/tika/tree/TIKA-4771-message-parser
• Commit: ai-pipestream@f3907bd49
• Tim-diff: ai-pipestream/tika@6480d6f...TIKA-4771-message-parser

I added a write up here:
tika-grpc-message-parser.md

[krickert]

So let me know if this approach is clear. It's going to be a similar pattern I use for the grpc output we do for tika in OpenNLP - we'll have a document shape in there that'll get all the RAG folks excited including putting embeddings on the data (I'll build a plugin for you as well that decorates NLP data all over the place in Tika once we release 3.0).

That being said, I imagine we would make OpenNLP's enhancements a direct integration, or we can make it so it's either-or (if they want to scale the OpenNLP work, they can use the gRPC and if they want to just directly integrate it they can do that as well).

But the idea really falls around SPI and the ability to inject services into java now without reflection. Also, the fact that all protobufs carry the "Message" inheritance.

FastWARC's grpc server is a good example - I can see a lot of Tika people having WARC files. But my intent with that parser is just to have a useful example. I can pick another service if you want as a demonstration.

[dpol1]

So, should I update this PR to follow the pattern with oneof() in #2961?

Yes, if it keeps what your Values had: the declared type and the original strings. #2961 (I worked on it) drops the type when one value fails to convert, so extended-properties:TotalTime arrives as int64 from a .doc and as plain strings from an .odt (PT1M7S). It also turns zone-less dates into UTC. Keeping values and type next to the oneof would cover both, at the cost of sending typed keys twice. wdyt?

ParseBytes fits as the "Later" RPC in your design.

Two things here, from the code (not run):

tika/tika-grpc/src/main/java/org/apache/tika/pipes/grpc/FetchParseRuntime.java

Lines 138 to 139 in 6480d6f

	return pipesParser.parse(new FetchEmitTuple(fetchKey, new FetchKey(resolvedFetcherId, fetchKey),
	new EmitKey(), new Metadata(), parseContext, FetchEmitTuple.ON_PARSE_EXCEPTION.SKIP));

• No emit strategy, so with the default DYNAMIC a result of 100 kB or more goes to the empty EmitKey and fails. PASSBACK_ALL, as tika-server does, fixes v1 and v2alpha1.
• SKIP returns no documents when the container throws. /rmeta uses EMIT and returns what was parsed plus the stack trace. Should v2alpha1 match it? TIKA-4766: minimal typed Document parse contract for tika-grpc (stage 1) #2961 and TIKA-4795: ParseBytes, parse bytes the caller already has #3449 have the same SKIP.

[krickert]

Separate proto parser for each Tika parser?

It's not one per Tika parser. A MessageParser is keyed by the incoming protobuf message type, and it's only for users who already have a typed message (a WARC record, a Kafka record, a reply from their own service) and want it back as Tika documents. File parsing through FetchAndParse is unaffected.

The jar would ship its own generated stubs and decides how its message maps onto Tika's document shape. Tika never has to know the message's schema, and nothing goes through strings.