Repository navigation
Conversation
Right now every player downloads each .dff and .txd as its own request. With a few hundred models that takes ages. This adds two routes to the models web server so it can also hand out all of them as a few compressed files, and a launcher can grab those before it starts the game. By the time the game asks for the models they're already in the cache, so there's nothing left for it to download. Old launchers and old servers just do the normal downloads like before.
GET /modelpack/v1/index.json
GET /modelpack/v1/pack/<n>
The index has the compressed size of each segment and, for every file, its cache name (0x<CRC32>.dff or .txd), size, CRC32 and the segment it's in. Segment n is a zlib stream of those files back to back. Range requests work.
The segments are written to a temp folder and served from there, so it doesn't matter how big the models are. Compressing a lot of models takes a while, so it happens on a background thread a couple of seconds after the gamemode has registered them. Until it's done both routes answer 503 and launchers just use the normal downloads. The temp folder is removed on shutdown, and one left behind by a crash is cleared on the next start.
Files go into segments in the order the models were registered, so models added at the end only change the last segment or two, and players who already have the rest only download that. A segment is a power of two in size (1 MiB up to 256 MiB), picked so there are about 20 of them.
The launcher side is a separate PR in openmultiplayer/launcher. Before it starts the game it asks the game ip and port for the index, works out which files are missing from the model cache, downloads only the segments that have them and checks each file's CRC32 before it goes in the cache. If there's no index (old server, different artwork.port) or anything goes wrong it does nothing and the game downloads the usual way.
At most 8 segment downloads run at once, the rest get a 503.
artwork.model_pack: false in the server config turns it off (default is true).
Adds zlib as a dependency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now every player downloads each .dff and .txd as its own request. With a few hundred models that takes ages. This adds two routes to the models web server so it can also hand out all of them as a few compressed files, and a launcher can grab those before it starts the game. By the time the game asks for the models they're already in the cache, so there's nothing left for it to download. Old launchers and old servers just do the normal downloads like before.
The index has the compressed size of each segment and, for every file, its cache name (0x.dff or .txd), size, CRC32 and the segment it's in. Segment n is a zlib stream of those files back to back. Range requests work.
The segments are written to a temp folder and served from there, so it doesn't matter how big the models are. Compressing a lot of models takes a while, so it happens on a background thread a couple of seconds after the gamemode has registered them. Until it's done both routes answer 503 and launchers just use the normal downloads. The temp folder is removed on shutdown, and one left behind by a crash is cleared on the next start.
Files go into segments in the order the models were registered, so models added at the end only change the last segment or two, and players who already have the rest only download that. A segment is a power of two in size (1 MiB up to 256 MiB), picked so there are about 20 of them.
The launcher side is a separate PR in openmultiplayer/launcher. Before it starts the game it asks the game ip and port for the index, works out which files are missing from the model cache, downloads only the segments that have them and checks each file's CRC32 before it goes in the cache. If there's no index (old server, different artwork.port) or anything goes wrong it does nothing and the game downloads the usual way.
At most 8 segment downloads run at once, the rest get a 503.
artwork.model_pack: false in the server config turns it off (default is true).
Adds zlib as a dependency.