KronLSTM beats Gated DeltaNet-2 on 1.5B-scale benchmarks and 32k retrieval, but collapses on a repeated-noise needle test (11.6 vs Gated DeltaNet's 99.2 at 16k)
NX-AI's Kronecker Delta Rule lifts the six-task average from 58.12 to 59.72 and cuts long-context LAMBADA perplexity to 7.66 from 8.36. The code is MIT-licensed; the tables without seeds or intervals and one big failure case are the part to read before you switch.
This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling.
Comprehensive take
KronLSTM is the most credible new linear-attention update rule this month: the language-modeling and 32k-extension gains over Gated DeltaNet-2 are consistent, the proofs are public and the code is MIT-licensed. It is still a 1.5B preprint with tables that report no seeds or intervals and a severe failure on repeated-noise retrieval. If you train or fine-tune linear-attention models, run the repository on your own long-context data. If you serve hosted models, nothing changes for you yet.
Use cases
Researchers training long-context linear-attention models
The Triton kernels and PyTorch layer are flash-linear-attention compatible, so KronLSTM can be benchmarked beside GDN-2 and KDA in an existing training stack.
Teams that need constant-memory inference at 16k to 32k
The paper's context-extension run (25B tokens at 32k) is the closest published evidence for how the rule behaves when trained on long documents.
Anyone evaluating memory architectures for agents
The interference argument (writes only affect overlapping key-value pairs) is directly relevant to long-running agent contexts, but test with your own repeated or templated content given the S1 result.
Specs
| Model scale / data | 1.5B parameters, 100B FineWeb-Edu tokens, 8k training context source |
|---|---|
| Linear Avg-6 accuracy | 59.72 (KronLSTM) vs 58.12 (KDA, best rival) source |
| Hybrid (SWA) Avg-6 accuracy | 59.56 (KronLSTM) vs 58.79 (GDN-2) source |
| LAMBADA perplexity, linear / hybrid | 11.72 / 11.18 source |
| 32k extension, linear Avg-6 and LAMBADA ppl | 60.19 and 7.66 (GDN-2: 58.75 and 8.36) source |
| S1 needle accuracy at 16k, linear | 11.6 (KronLSTM) vs 99.2 (Gated DeltaNet) source |
| Training throughput, one H100-80GB | 36.8 to 44.5K tokens/s; 11 to 18% above GDN-2 source |
| Code and license | Triton kernels + PyTorch layer, MIT source |
Insights
- Axion reading: the gains over Gated DeltaNet-2 are largest where the paper's interference argument predicts them (long, natural-text retrieval) and smallest on short-context accuracy, which makes the story coherent.
- Axion caution: tables 1 to 4 report single values with no seeds or intervals, so differences of about a point on the six-task average should not be treated as settled.
- Axion reading: the paper discloses that Claude Fable 5 and 5.1 helped derive proofs and optimize kernels; the authors state they checked every proof and verified the kernels numerically.
What is actually new
Linear-attention models replace the growing key-value cache of a Transformer with a fixed-size matrix memory that is updated token by token. The open problem is interference: every write can disturb what is already stored, and the damage compounds as the context grows. The DeltaNet family (Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2) corrects that memory with a delta-rule update. The KronLSTM paper proposes a different update, the Kronecker Delta Rule, which revises a stored association along a single direction in matrix space, so a write touches only the content whose key and value both overlap with the new pair. The authors also prove that the memory dynamics converge to a steady state and that the expected memory norm stays bounded without extra normalization.
The architecture is an xLSTM-family design, and the code is public: the NX-AI repository ships Triton kernels and a PyTorch layer under the MIT license, compatible with flash-linear-attention, with models in a Hugging Face collection.
The numbers, and how big they are
All headline results are 1.5B-parameter models trained on 100B FineWeb-Edu tokens at 8k context, compared against Gated DeltaNet, Mamba-3, Kimi Delta Attention, Gated DeltaNet-2 and mLSTM, each as a pure linear model and as a hybrid with sliding-window attention (window 2048, 5:1 linear-to-window ratio).
On six language-modeling tasks, the pure linear KronLSTM averages 59.72 against 58.12 for the strongest rival, KDA, a 1.6-point gain, with LAMBADA perplexity of 11.72 against 12.54 for GDN-2. In the hybrid setting it averages 59.56 against 58.79 for GDN-2 and reaches LAMBADA perplexity 11.18 against 11.85. On the real-world recall suite at 2k context the margins are thin: 30.31 against 29.81 for linear models, and 37.40 against 36.96 for hybrids. Note that the hybrid average is not higher than the linear one in the language-modeling table (59.56 versus 59.72).
The long-context result is the one the paper is built around. After extending to 32k context with 25B additional tokens, KronLSTM scores 60.19 on the six-task average against 58.75 for GDN-2 and drops LAMBADA perplexity to 7.66 from 8.36. On RULER needle retrieval at 4k, the linear model reaches 73.4 on the S3 task against 54.8 for Gated DeltaNet. The authors report that the lead over GDN-2 persists from 16k onward on natural-text retrieval tasks, while stating that both models lose a substantial amount of accuracy at 32k.
Where it loses
The paper is candid about one failure. On S1, a synthetic task with frequently repeated noise sentences, the pure linear KronLSTM scores 66.2 at 4k and 11.6 at 16k, while Gated DeltaNet scores 100.0 and 99.2. The authors attribute this to residual accumulation for recurring keys and call the task unrealistic for real text, which is a reasonable argument but one practitioners with repetitive logs or templated documents should test for themselves. KronLSTM also trails at S2 at 16k (16.0 against 17.8 for Gated DeltaNet and 22.6 for mLSTM). Absolute retrieval accuracy at 16k is low for every model here, so none of these checkpoints is a replacement for full attention on long-document recall.
Training speed is competitive rather than leading: 36.8 to 44.5K tokens per second on one H100, 11 to 18 percent faster than GDN-2 and within 4 to 8 percent of KDA up to the 8K x 2 shape. Gated DeltaNet, with the cheapest kernels, remains the fastest.
One more detail worth reading
The paper's AI-use statement says Claude Fable 5 and Claude Fable 5.1 assisted in deriving the convergence and norm-bound properties and in optimizing the chunkwise-parallel and recurrent kernels, with the authors checking every proof and numerically verifying the kernels against naive reference implementations. They state that the update rule itself originated with the authors and that AI was not used to design experiments or interpret results.
Caveats
This is an arXiv preprint (v1). The tables we read report single values without confidence intervals or seed counts, the largest model is 1.5B parameters, and the authors list scaling up and training hybrids with full softmax attention as future work.