Repository navigation
File Storage prototype (do not merge) - #6182
Draft
cloutiertyler wants to merge 33 commits into
Draft
cloutiertyler wants to merge 33 commits into
cloutiertyler wants to merge 33 commits into
Conversation
…store Adds FileId and FileRef to spacetimedb-lib, and a new spacetimedb-file-store crate that keeps file bytes under their BLAKE3 hash. LocalFileStore writes each object through a synced temporary file and a rename, so a crash never exposes a partial object, and reclaim_unreferenced deletes objects outside a set of roots once they are older than a grace period.
…napshot manifests st_file records each stored file and is the only strong reference to its bytes. st_file_staged holds the bytes of files created inside a reducer until the host persists them, so they reach the commitlog with the transaction that created them. Both are private and hidden from module code, which reaches files only through the file syscalls. Each snapshot manifest now lists the distinct file hashes that st_file references, appended to version 0 of the format. Older binaries ignore the trailing bytes, and manifests written before the field existed read as referencing no files. SnapshotRepository::retained_file_hashes collects the hashes every retained snapshot needs, for reclamation.
…les are persisted Adds DatabaseFiles, the File Storage state for one database: its file store, the staged files being persisted, and a cache of hashes known to be persisted. Persistence can now carry one, and the local provider keeps file bytes in the replica directory. When a database has files, its durability is wrapped in a FileGatedDurability. Transactions go to the inner durability immediately and in order, but the reported durable offset is capped just below the first transaction that still waits on a file. It learns each transaction's files by wrapping the lazily prepared payload, which the inner durability converts before it can report that transaction durable, and starts persisting staged bytes at the same moment. Every confirmed-read path waits on that offset, so none of them change. spawn_file_maintenance persists anything left in st_file_staged after a restart and deletes staged rows once their bytes are persisted.
POST /v1/database/:db/files asks the module's allow_upload reducer whether to accept the upload, before reading any bytes and with the FileId the file will have. It then spools and hashes the body, persists it, commits the st_file row, and responds once that row is durable. Without an allow_upload reducer, uploads are refused, and clients can't call allow_upload themselves. GET and HEAD /v1/database/:db/files/:file_ref serve a live file's bytes, including single byte ranges, with the serving headers from the proposal. A file a reducer just created is served from its staged bytes until it is persisted. GET .../metadata returns the live record. These routes sit beside the authenticated database routes, since a download is authorized by its FileRef. SQL DML on st_file and st_file_staged is refused, even for the owner, and the host starts file maintenance when it opens a database.
…oketest Modules reach files through five new syscalls in spacetime_10.7: file_create, file_copy, file_delete, file_metadata, and file_reference. The Rust bindings wrap them as ctx.files(), and FileId and FileRef can be table columns and index keys. A file created in a reducer is limited to 16 MiB, because its bytes travel through the commitlog until the host persists them. Reclamation keeps the files that live st_file rows name and that retained snapshots list, plus staged files still being persisted, and deletes other objects older than a grace period. File maintenance runs it hourly and holds the database weakly, so a closed database is dropped. Files inserted and deleted between two snapshots are a known gap: reverting to a point between them would need bytes that reclamation may have deleted. The standalone smoketest publishes a module and checks uploads, allow_upload, downloads and ranges, forged references, files created and copied in reducers, confirmed reads, deletion, and a restart.
Updates the FilterableValue diagnostic, and the bindings UI snapshots that list its implementors.
…y no longer needs it Deleting a file used to remove its st_file row, so a file inserted and deleted between two snapshots was in no manifest and no live row, and reclamation could delete bytes that reverting to a point between the snapshots would need. Deletion now marks the row with the offset of the deleting transaction. To modules and to the metadata route, a marked file is gone: metadata, reference, copy, and a second delete find nothing. But the row stays a reclamation root, and every snapshot lists it, for as long as history from before the deletion can be reverted to. Downloads keep working until then, as the proposal specifies, and making deletion revoke them is a one-line change in the download route. purge_deleted_files removes the records of files deleted before a horizon, the oldest offset a revert can reach. A standalone host keeps its whole commitlog and so never purges, which is how blob bytes already follow history. MutTxId::next_tx_offset gives a transaction its own offset.
…ile it runs Persisting an object that already exists now refreshes its age, and reclamation deletes through FileStore::delete_if_older_than, which checks the age again at the moment of deletion. A pass that chose an object before someone uploaded the same contents again therefore spares it. An S3 store would make this a conditional delete. Also formats the prototype's changes with rustfmt.
- After a restart, confirmed reads could see a transaction whose staged file was never persisted, because the gate started at the inner durable offset. The gate now reports nothing durable until file maintenance has persisted the files left staged when the database opened. - A content type with control characters made every download of the file, and of its copies, fail with 500. Content types must now be servable header values, both from reducers (INVALID_CONTENT_TYPE) and from uploads, and downloads fall back to application/octet-stream. Download names keep only visible ASCII. - close() could return None and lose the confirmed offset when the inner durability exited first. The gate now remembers the last offset it reported, and hands it out after the inner durability exits. - Reclamation could delete an object between its age check and a concurrent persist. Persists and reclamation's check-and-delete now exclude each other, and reclamation drops the cached answer first. - An upload that ended without persisting left an in_flight entry that waiters spun on. A guard removes it, and waiters fall back to the store. - Snapshot::from_bsatn is public, falls back to the old layout only when the manifest ends where file_hashes would begin, and rejects a malformed list. - Downloads retry the store when a staged row disappears mid-request, and malformed ranges are ignored, as RFC 9110 requires. - File maintenance's startup read runs off the async worker.
…loads complete Views get ctx.files() with metadata and reference, through the same syscalls as reducers, which record st_file reads so the view refreshes when files change. Uploads used to commit their st_file row directly, which skipped view re-evaluation, so a view that read a file's metadata stayed stale after its upload completed. They now commit through the module host, like environment updates: insert the row, re-evaluate subscribed views, then commit and broadcast. A smoketest subscribes to such a view and fails if the upload's commit bypasses it.
…file list Two ignored engine tests print timings. The gate's overhead on transactions without files is within run-to-run noise. Listing a million files' hashes for a snapshot takes about 285 ms under the snapshot lock, a small part of a snapshot that must also write those rows' pages.
An upload may carry an Idempotency-Key. When it completes, a private st_upload table remembers the key with the file it created, in the same transaction as the file's record, for a day. A retry with the same key returns the same file, with spacetime-idempotent-replay set, without asking allow_upload again or creating another file, even if the file has since been deleted. A retry while the first attempt is still in progress gets 409. Keys are scoped to their sender, so keyed uploads need an identity. GET /v1/database/:db/files/uploads/:key returns a completed upload's saved response, for a client that lost it. File maintenance expires records hourly. CORS now allows the Range, If-None-Match, and Idempotency-Key request headers, and exposes the file reference, replay, Content-Range, and ETag response headers.
… uploads on close - An upload may carry a Content-Digest (RFC 9530). Its SHA-256 is verified before the file is stored, and a mismatch is a 422. Other algorithms are ignored. - A database accepts at most 256 uploads at once, as the proposal specifies, and refuses more with a 413. - Closing a database stops its background uploads and waits. A stopped wait reports that the file wasn't persisted, so the gate never confirms a transaction because its database closed. Files left staged are persisted again when the database next opens.
- Uploads refreshed every subscribed view instance, and one failing view failed every upload. They now refresh only the views whose reads the upload touched, as reducer commits do, and they report a trapped view to the instance in both wasm and JS. The commit is broadcast as the uploader. - A replayed upload and the upload lookup returned a reference before its record was durable, so a crash could take it back. Both now wait until the state they read is durable. A file purged since its upload is a 410 rather than a fresh upload. - A retry could pass the completed-upload check, then lose to the original's commit and fail on st_upload's unique constraint. The key is now reserved before the check, and the commit itself reports a key another attempt already completed, which the route answers with that attempt's file. Expired records are ignored before they're deleted, replaced on reuse, and expired as soon as the database opens. - An upload in progress across a publish used the old module and failed. The route looks the module up again before committing. - The persist lock is striped by hash, so reclaiming one object doesn't stall unrelated persists. - If file maintenance can't read the staged rows at startup, it retries until shutdown instead of holding confirmed reads until a restart. - CORS allows Content-Digest, the parser ignores members it doesn't understand, and accepts unpadded base64. A retry of a completed upload no longer needs a free upload slot. - Restores Debug on UpdateEnvironmentResult.
…re they're persisted With the store refusing writes, a reducer's file can't be persisted, and the gate keeps its transaction from confirmed reads. After a restart with the store writable again, file maintenance persists the file from its staged bytes, which travelled in the commitlog, the gate confirms the transaction, and the staged row is cleaned up.
…mpatible server
S3FileStore keeps objects at {prefix}objects/{hash} in an S3-compatible
bucket, through the object_store crate. Uploads are spooled and hashed
locally, then stored in one request or, from 8 MiB, in parts. Reads are
ranged GETs.
S3 can't refresh an object's modification time, which reclamation relies on
to spare an object persisted again while it runs. So persisting an existing
object writes an empty {prefix}refreshed/{hash} marker, and an object's age
is the later of the two. S3 has no conditional delete, so
delete_if_older_than checks and then deletes; within one host the
per-hash persist lock covers the gap.
A standalone server keeps file bytes in a bucket when config.toml has a
[file-store.s3] section, under {prefix}{database identity}/. The smoketest
guard can write a server config from SPACETIMEDB_SMOKETEST_SERVER_CONFIG.
Tested against SeaweedFS: the store's own tests (including a 26 MiB multipart
upload and the refresh marker), and both end-to-end file smoketests with
every server keeping its files in the bucket. The S3 tests skip unless
SPACETIMEDB_TEST_S3_ENDPOINT is set.
A module now enables uploads with #[reducer(allow_upload)] on any reducer, rather than by naming one allow_upload. It's a new Lifecycle variant, so the host finds it the way it finds init, clients can't call it like any other lifecycle reducer, and v10 module definitions mark it private, which keeps it out of client bindings. Schema validation rejects an allow_upload reducer that doesn't take exactly one UploadRequest, comparing the shapes of the types so case conversion can't break the check. A wrong signature is now a publish error rather than a 500 on every upload. The generated C# and TypeScript Lifecycle enums gain the variant by hand, since regenerating them needs .NET; neither language can declare an allow_upload reducer yet. The smoketests use a reducer named check_upload to show that the name no longer matters.
Clients now upload and download files through the connection they already
have: conn.files() returns a handle with upload, upload_idempotent, download,
download_range, metadata and url. It builds its URLs from the connection's
URI and database, and authenticates as the connection does, including with
the token the server issues an anonymous connection. Codegen adds files() to
DbConnection and to every context type, as inherent methods, so bindings
generated before this change still compile.
FileRef becomes a special SATS type, the product { __file_ref__: U256 },
in the way Uuid is. Its BSATN is unchanged, but codegen can now recognize
it: Rust bindings give such columns and parameters the type
spacetimedb_sdk::FileRef, while C#, TypeScript and the C++ SDKs still see a
U256 until their SDKs grow a type for it.
A new module, sdk-test-files, and client, files-client, test the whole path
under the SDK test harness: an upload, a reducer storing the reference, the
subscribed row arriving with a FileRef, downloads by whole file and range,
metadata, idempotent retries, a forged reference finding nothing, deletion,
a file a reducer created, and an upload allow_upload refuses.
Browsers get the handle and url() but not the HTTP methods yet.
open_database_files opens the local or S3 store that a FileStoreConfig describes, so that the cloud's persistence provider can open stores the way standalone does.
A bucket without static keys now takes credentials from the environment, as the cloud's archival does: AWS_* variables, a web identity token, or the instance's role. Unsigned requests, for a local test server, now need unsigned = true.
… reducers A reclamation pass gathers its roots once, then deletes objects one by one, so a transaction that commits during the pass can name an object the roots don't include. For uploads, the persist under the per-hash lock already orders them. Files created in reducers commit before their bytes persist, and two things let one slip through. The gate confirmed a transaction without waiting when its cache already knew the staged file's hash, so a pass could delete the object after the cache check, and a confirmed state would name a missing object until the staged persist put it back. A staged file now always waits for its own persist, which can't start while a delete of that hash is under way. Reclamation also trusted its stale roots at the moment of deletion. Under the per-hash lock, it now forgets the hash, so later commits wait for a fresh persist, and checks the current st_file before deleting. Each new test fails without its fix.
…now says Inside a module, the module is the authority, so possessing a valid FileRef shouldn't decide whether a file may be copied or deleted. The module files API now takes FileIds: create and copy return them, metadata, copy and delete take them, and reference turns one into a FileRef when the module decides to disclose the file. A reference a client sent may carry any secret, so ctx.files().id(file_ref) checks it against the live record, and FileRef::id() becomes id_unchecked(), for references the module read from its own tables. Views get id too. FileMetadata has accessors in place of public fields, and FileUrl gives FileRef and FileMetadata a url(), a path on the host, since a module doesn't know the address clients reach it at. The unreleased spacetime_10.7 ABI passes FileIds for these calls. The Rust client SDK takes the proposal's shape too. upload(body, options) returns a FileRef, generates an idempotency key when none is given, and retries failed attempts with it. download(file_ref, range) returns a stream, or None for an unknown reference or reclaimed bytes, and metadata returns None once the file is deleted. url and url_with return Urls. FilesError and UploadError replace Error::FileRequest.
Four of the review's findings, each with a test that fails without its fix, except the first, which needs a crash to observe: - A local persist that found its object already in place reported success without syncing the object's directories. If another persist of the same bytes had just renamed it there, a crash could lose the object after its record committed. It now syncs them first. - Reclamation read snapshot manifests through all_snapshots, which skips locked snapshots, so a snapshot being compressed or written dropped out of the roots. A locked snapshot's manifest may be missing or partly written, so the pass now fails closed and deletes nothing until no snapshot is locked. - An upload that spanned a database reset persisted its bytes to the old database's store, then recorded them in the new database, whose gate then waited forever. The route now refuses with 503 unless the database it records into has the store that holds the bytes, and clients retry. - Between persisting an upload and committing its row, only the grace period protected the object, so a long stall could let reclamation delete it. persist now returns a PendingUpload guard that the route holds until the row is durable, and reclamation treats pending uploads as roots and checks them again under the per-hash lock.
Master's st_outbox (#6104) took table ID 26, so st_file, st_file_staged, and st_upload move to 27 through 29, with their constraints and indexes renumbered after st_outbox's.
A files submodule follows the proposal's patterns: allow_upload records each upload's owner, avatars may only be the caller's own uploads or a file the reducer draws, and gifts are public rows naming a FileId that the my_gifts view resolves for the recipient alone. The codegen snapshots gain the new tables, types, and reducers.
A FileRef's text form put its ID first, but its U256 puts the read secret in the high half, so a client calling a reducer over HTTP had to swap the halves of the reference the upload API returned. SQL printed it as a 77-digit decimal. Now its text form is its U256 in hex, as an Identity's is: SQL prints it in hex after 0x, URLs accept the 0x, and spacetime call accepts it bare. The smoketest reads references from SQL and passes one the upload API returned straight to a reducer.
Cargo.toml takes 2.13.0 for every crate, including spacetimedb-file-store. With #6154, a confirmed HTTP reducer call waits for its transaction to be durable, and the file gate holds durability until the transaction's files are persisted, so a call that creates files returns once they are stored, or with a 504 if that takes more than five seconds.
Uploads authorized only by an upload token ran allow_upload as Identity::ZERO, which is also the Cloud control database's identity, and allow_upload had no connection, so Rust modules saw it as internal and it never had a JWT. Now an upload without an identity token gets a new identity, as a client that connects without one does, and allow_upload runs inside an HTTP client connection, as a reducer called over HTTP does. Its sender is always the uploader, so UploadRequest.sender is gone. This matches what #5892 does for HTTP handlers. The upload token moves from ?token= to ?upload_token=. The auth layer reads ?token= on every route as an identity token, so a token-only upload got a 401, and an identity token sent that way reached allow_upload as the upload token.
Windows can't open a directory as a file, so syncing the object's directories after the rename failed, and every new upload got a 500 there. The commitlog's `sync_dir` is a no-op on Windows for the same reason.
Only Rust modules can store files so far, so the TypeScript and C# module-tests have no `avatar`, `file_owner` or `gift` tables. Like views, they're left out until those languages support File Storage.
The Rust codegen gives every connection and context a `files()` accessor, but only files-client's bindings were regenerated, so CI's check that `prepare_clients` leaves the tree unchanged failed. This is the diff CI produced.
CI's SDK test partitions run clients that `prepare_clients` built ahead of time, and files-client wasn't among them, so every file test failed to find its binary. Only native builds have the file tests, so only they prepare it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
This draft PR is a prototype of File Storage, built to test a design end to end, and is not meant to merge.
SpacetimeDB has no way to store files today. Applications keep them in a separate bucket that the database knows nothing about, so rollback, views, and access control don't cover them. This prototype adds a private
st_filesystem table that holds the only strong references to files. The bytes live in a content-addressed object store, local or S3-compatible. Snapshot manifests list the file hashes, so garbage collection can find every file a snapshot needs. A file must be in the object store before clients reading with confirmed reads can observe the transaction that references it.The prototype covers:
ctx.files()in reducers and views;#[reducer(allow_upload)]as a lifecycle reducer;FileRefas a special SATS type;conn.files();Files created in reducers carry their bytes into the commitlog through a private
st_file_stagedtable, and are uploaded after commit. None of this needed changes to the datastore's commit path, the commitlog format, or any confirmed-read path.API and ABI breaking changes
This doesn't break existing modules or clients, because it only adds a module ABI version, a lifecycle reducer kind, routes, and an SDK error variant. A module that uses File Storage would fail to publish on a host without it.
Rollback safety impact
This must not be deployed. It adds system tables and a field in snapshot manifests, so it would need its own rollback analysis before it could merge.
Expected complexity level and risk
Testing
allow_uploadsignature check.file_storage, against the local store and against an S3-compatible server (SeaweedFS).