[F64116045]

What changes were proposed in this pull request?

The ceph/s3-tests test_object_set_get_unicode_metadata exposed that S3 Gateway does not handle non-ASCII user metadata correctly.

Investigation showed that S3 uses RFC 2047 for non-ASCII x-amz-meta-* values. Ozone currently stores encoded values as ordinary text and does not encode them when returning response headers.

This patch decodes RFC 2047 Q/B values before storing metadata and encodes non-ASCII values in GET and HEAD responses.

References:

• ceph/s3-tests failure case
• Amazon S3 metadata behavior

Note:
I kept this PR to the single encoded-word case so we can agree on the basic read&write behavior first.
Multiple words and long values need more work. If this direction looks good, I’ll take those on in a follow-up.

What is the link to the Apache JIRA?

https://issues.apache.org/jira/browse/HDDS-16672

How was this patch tested?

CI: https://github.com/F64116045/ozone/actions/runs/37118609154

Uploaded an object with this RFC 2047 encoded café metadata:

x-amz-meta-meta1: =?UTF-8?B?Y2Fmw6k=?=

The same request was tested against Ozone before and after this patch, and against S3.

B and Q are two RFC 2047 representations of the same café value.

	Ozone before	Ozone after	S3
Stored value	Encoded header text	café	Not observable
HEAD response	B-encoded café	Q-encoded café	Q-encoded café

[F64116045]

The original ceph/s3-tests case still fails because it sends raw non-ASCII metadata instead of RFC 2047. S3 does not round-trip that input either.

[rich7420]

=?UTF-8?Q?a?b?= passes this check, but QCodec.decode() returns a. I reproduced this in a PUT endpoint test: the stored metadata is silently truncated. Please also check that the first ? after the encoding token is the terminating ?=, preserving the original value otherwise, and add a regression test alongside the invalid-Base64 case.

[F64116045]

thanks for the catch! fixed.

[Russole]

Thanks @F64116045 for the patch. Left some comment.

[Russole]

For malformed RFC 2047 values, should we return the original value instead of decoding it? For example, =?UTF-8?Q??= currently becomes an empty string.

[F64116045]

Oops I missed that case, thanks for catching it.
Also I checked a few other malformed values while fixing it, found similar issues, and added tests for those too.

[adoroszlai]

Thanks for the patch. I would like to defer merging this after #11735, which will (hopefully) allow skipping most integration tests for S3-only changes.

[F64116045]

I would like to defer merging this after #11735, which will (hopefully) allow skipping most integration tests for S3-only changes.

Sure. Also thanks for all your recent work on improving CI!

[peterxcli]

should we use mature lib to do this? or do you have any reference for your decodeRfc2047MetadataValue implementation?

[F64116045]

@peterxcli Thanks for the question

should we use mature lib to do this or do you have any reference for your decodeRfc2047MetadataValue implementation?

Actually we’re already using Ozone’s existing Commons Codec here for decoding and encoding.
The extra check is there because decoding malformed values directly can silently change the metadata we store.
(For example, =?UTF-8?B?!!!?= gets decoded to an empty string without an error.)
Also FYI the check is based on the encoded-word syntax in RFC 2047, Sections 2 and 4

I also just looked into MimeUtility, and found it supports multiple encoded-words, which we were considering for a follow-up, so I think it’s worth using here.
That said, it can also partially decode malformed values, so we'd still need to check the input first, as we do now.
Wonder what do you think?