Repository navigation
feat(models): Hy4-flash with sparse attention, engram table streaming, MTP and vision - #294
Merged
Merged
Conversation
… kernel, keys stored one row per token
…, a lazy node, so decode builds the next step while the gpu runs
… after it, same bits
…tops copying each layer's cache per token and stays flat with depth
…s, reading the pool in place, with pool keys stored in fp16
…olling cache, so cache memory past the window grows with the sparse layers only
…in one kernel each and reuses its constants, 400 fewer graph nodes a token with the same bits
…tches spread over the GPU instead of one threadgroup on one core, 3.7% faster hy4-flash decode
…d starts each read when the step is built, so a 16k prefill idles the gpu 3% for the table instead of 18% and decode is 1% faster
… and radix select, 1.4x faster prefill at 128k and 4% faster at 16k
…eys with one dense matmul over the dequantized stack, 4% faster prefill at 16k
…4 threads on the fp32 gate, 0.6% faster decode
…ds and the pooled indexer scores, a request no longer fails on the per-layer head list
…rom the drive during its own forward, 4% faster prefill at 16k
…ed at prefill, glm-5.3-flash drafts from the whole prompt again
…kernels, the read-ahead row fits the table lint
…le format, and the released file's tensor names against the model tree
…ns, and mtp on the vision container
…ags, registry rows and the vendor manifest follow the mtp and vision wiring
… kernels, 4 to 5% faster verify
…re torch is installed
…t-in draft ladder by first-draft confidence
…he unverified mtp and vision note
…s that table as weights when sizing the mlx cache
…x-kquant they need
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.