LatComm cuts a receiving LLM's input from 5,915 tokens to 103 and answers 3x faster, but on held-out data it trails plain text by 15 points
A frozen encoder compresses a document into a 384 KiB packet that a different frozen model reads per question. The speed and memory gains are measured; the accuracy gains are against other compressors, not against the original text.
The context is encoded once without the question, and the receiver selects what to read for each question.
Comprehensive take
Treat LatComm as a credible cost-cutting idea for repeated questions over one document, not as a drop-in replacement for passing text. The 3x speed-up is measured on one H100 with roughly 1B-parameter receivers; the 15-point held-out gap to raw text is the number to ask the authors about.
Use cases
Many questions over one document
Support, legal-review or QA pipelines that ask a dozen-plus questions about the same article are the case the paper measures (18.1 questions per article on QuALITY).
Heterogeneous model stacks
A larger encoder can feed a small receiver, or a different model family; the paper tests six pairs, including Granite-4.1-8B to Llama-3.2-1B.
Research on agent memory
The authors name packets as agent-memory entries as future work; no experiment in the paper tests it.
Specs
| QuALITY receiver input per question | 5,915 tokens (raw text) vs 103 (LatComm) source |
|---|---|
| Total time, 18.1 questions | 9.26 s vs 3.09 s, one H100, batch size 1 source |
| Receiver peak memory | 0.87 GiB vs 0.06 GiB source |
| Task Avg / Held-out Avg (Qwen3.5-4B encoder, 0.8B receiver) | 65.43 / 49.76 vs best compressor baseline 61.98 / 46.70; raw text 51.43 / 65.06 source |
| Default packet size | 384 KiB (24 layer pairs x 32 slots x 256 width, bfloat16) source |
Insights
- Axion reading: the efficiency table is the strongest result in the paper; the accuracy claims depend on which baseline you compare against.
- Axion caution: raw text beats LatComm by about 15 points on the held-out sets (65.06 vs 49.76), and the paper tests only English multiple choice with receivers around 1B parameters.
- Axion caution: the paper is an unreviewed preprint (marked 'Under Review') and all timing was run on one GPU by the authors.
What it does
When several models share a document, the usual handoff is plain text: the receiving model re-reads every token for every question. The LatComm authors put numbers on that waste. On the QuALITY reading-comprehension set an article comes with 18.1 questions on average, and the receiver reads 5,915 tokens per question, 98% of which is the article itself.
LatComm splits the job in two. A frozen "encoder" LLM reads the context once, without seeing any question, and a set of trained compressors squeezes its hidden states into a fixed-size packet. A different frozen "receiver" LLM then reads that packet through cross-attention, with gates computed from its own prompt deciding which slots matter for the question at hand. Neither LLM is fine-tuned; only the compressors, readers and gates are trained. The default packet is 24 layer pairs of 32 slots at width 256 in bfloat16, which the paper puts at 384 KiB, and its size does not grow with the document.
The numbers
On the QuALITY test set (115 articles, 2,086 questions), measured on one H100 at batch size 1 with a Qwen3.5-4B encoder and a Qwen3.5-0.8B receiver, the paper reports:
| Per-question cost | Raw text | LatComm |
|---|---|---|
| Receiver input tokens | 5,915 | 103 |
| Receiver time | 511 ms | 131 ms |
| Receiver peak memory | 0.87 GiB | 0.06 GiB |
| Total for 18.1 questions | 9.26 s | 3.09 s |
Encoding an article once costs 0.71 s, and that is already included in the 3.09 s total. That is where the abstract's "57x" input reduction and "3x" speed-up come from.
On accuracy, across 12 multiple-choice datasets with the Qwen3.5-0.8B receiver, LatComm with the Qwen3.5-4B encoder averages 65.43% against 61.98% for the best compressed-context baseline, and 49.76% against 46.70% on three held-out datasets. The "up to 10.2 points" in the abstract is the gap over single-vector baselines. Six encoder-receiver pairs are tested, including cross-family ones such as Granite-4.1-8B into Llama-3.2-1B.
Read the held-out row before you get excited
The table has an uncomfortable line. The untrained receiver given the full original text scores 65.06% on the held-out sets; LatComm scores 49.76%. On the 12 task datasets the picture flips (51.43% for raw text, 65.43% for LatComm), but that is partly because raw text is not trained on the task format while every compressed method is. In other words, LatComm beats other compressors, and wins on speed and memory, but on data it was not trained for it still gives up a lot of accuracy compared with just handing over the text. The no-context control matters too: a "constant packet" that carries no information about the document, with the same trained readers and gates, scores 60.25% on the task sets. LatComm is 5.2 points above that, which is the part of the score that genuinely comes from context.
Limits the authors state
The experiments are English, single-letter multiple-choice only. No free-form generation was tested, and neither were the multi-agent or agent-memory uses that motivate the paper. Receivers are about 1B parameters and encoders at most 9B, so behaviour with larger models is unknown. Only one configuration was run with three seeds (standard deviation 0.23 points on Task Avg); everything else is a single seed. Each new encoder-receiver pair needs its own compressors, readers and gates, and packets cannot be shared across pairs. The code is released at github.com/huawei-lin/LatComm.
Why practitioners should care anyway
Most multi-agent systems today pass text between models, and the cost shows up as repeated prefill. A reusable, fixed-size, cross-model packet is a plausible alternative to a KV cache, which grows with context length and only works for the model that made it. The speed and memory table is the most solid result; the accuracy story is promising on small models and unproven beyond them.