Skip to content

feat(io): refresh vended storage credentials per prefix #892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wgtmac merged 7 commits into from
Oct 17, 2026
Merged

feat(io): refresh vended storage credentials per prefix #892

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
098ba85
feat(io): refresh vended storage credentials before they expire
plusplusjiajia Oct 13, 2026
439397b
fix(io): make credential refresh safe for in-flight S3 requests
plusplusjiajia Oct 14, 2026
1ece5c9
test(rest): keep the HTTP client alive past its responses
plusplusjiajia Oct 14, 2026
6972755
fix(io): keep new S3 clients from taking the keys they replace
plusplusjiajia Oct 15, 2026
4e6578a
feat(rest,s3): simplify vended credential refresh
wgtmac Oct 17, 2026
51232b6
test(s3): match credentials to compatible URI schemes
wgtmac Oct 17, 2026
d3ea98f
test(s3): use a valid OSS credential prefix
wgtmac Oct 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions cmake_modules/IcebergThirdpartyToolchain.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -937,6 +937,14 @@ endif()

if(ICEBERG_BUILD_BUNDLE)
resolve_arrow_dependency()
# An installed Arrow does not export its bundled AWS SDK, which the S3 FileIO
# uses directly. Use the system SDK, which must match Arrow's.
if(ICEBERG_AWSSDK_BUNDLED AND NOT ARROW_VENDORED)
message(STATUS "Using the system AWS SDK, since Arrow is provided")
set(ICEBERG_AWSSDK_BUNDLED FALSE)
set(ICEBERG_AWSSDK_COMPONENTS core)
resolve_aws_sdk_dependency()
endif()
resolve_avro_dependency()
resolve_zstd_dependency()
endif()
Expand Down
1 change: 1 addition & 0 deletions mkdocs/docs/file-io.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ each file location's scheme.
| `s3.access-key-id` | `admin` | Static access key ID; must be set together with the secret key |
| `s3.secret-access-key` | `password` | Static secret access key |
| `s3.session-token` | `AQoDYXdzEJr...` | Session token, for temporary credentials. Ignored unless both static keys are set |
| `s3.session-token-expires-at-ms` | `1767225600000` | When the session token expires, in epoch milliseconds. Vended credentials are refreshed from the catalog five minutes before, including for files already open; once they have expired and cannot be refreshed, new operations under their prefix fail |
| `client.region` | `us-east-1` | Region to sign requests for |
| `s3.endpoint` | `https://127.0.0.1:9000` | Endpoint to use instead of the AWS one. When absent, the `AWS_ENDPOINT_URL_S3` / `AWS_ENDPOINT_URL` environment variables are consulted |
| `s3.path-style-access` | `true` | Address buckets as a path (`endpoint/bucket`) instead of a virtual host (`bucket.endpoint`). Only takes effect together with a custom endpoint |
Expand Down
10 changes: 10 additions & 0 deletions src/iceberg/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,16 @@ if(ICEBERG_BUILD_BUNDLE)
endforeach()
endif()

# The S3 FileIO signs with its own AWS credentials provider.
if(ICEBERG_S3)
list(APPEND ICEBERG_BUNDLE_STATIC_BUILD_INTERFACE_LIBS aws-cpp-sdk-core)
list(APPEND ICEBERG_BUNDLE_SHARED_BUILD_INTERFACE_LIBS aws-cpp-sdk-core)
if(NOT ICEBERG_AWSSDK_BUNDLED)
list(APPEND ICEBERG_BUNDLE_STATIC_INSTALL_INTERFACE_LIBS aws-cpp-sdk-core)
list(APPEND ICEBERG_BUNDLE_SHARED_INSTALL_INTERFACE_LIBS aws-cpp-sdk-core)
endif()
endif()

add_iceberg_lib(iceberg_bundle
SOURCES
${ICEBERG_BUNDLE_SOURCES}
Expand Down
Loading
Loading