Big models,
run small.
Trapetum presses a model's weights to a quarter of their size and reads them straight in that compressed form, so a big model runs faster, on far less memory, using far less power. Same answers, token for token.
See it live: same model, compressed vs not, racing on two identical RTX 4090s.
In one sentence
When a model writes one word at a time, the machine spends almost all its effort just reading the model's weights out of memory, not doing math. Trapetum shrinks those weights to a quarter of their size and reads them compressed, so every word comes out faster, on far less memory, using far less power. The answers stay the same.
What it does
Four measured wins from one move: read fewer bytes.
One move causes all four: the fused kernel reads a quarter of the weight bytes straight out of the codebook.
End to end that is 2.0× (7B) to 2.45× (13B) faster per token. Weights drop from 13.48 GB to 3.66 GB on disk for a 7B model (4.7 GB peak VRAM at runtime); a 2-bit 70B fits a single 24 GB consumer card and stays sharper than an fp16 7B. The speculative-decoding pass on top is token-for-token identical to plain decoding: 1.32× wall-clock on Llama-2-7B with a 160M drafter in July (164 vs 125 tok/s, about 2% extra memory), and 2.51× end to end on Llama-3.1-8B-Instruct with a PARD-2 drafter in November 2026 (415.6 vs 165.5 tok/s, 520 of 520 outputs identical, four prompt sets, 4-bit target and drafter).
The bandwidth law
The kernel speedup follows GPU memory bandwidth: largest exactly where bandwidth is scarcest.
| GPU | Bandwidth class | Speedup vs cuBLAS fp16 |
|---|---|---|
| RTX 4090 | ~1.0 TB/s, consumer | 2.20× |
| A40 | ~0.7 TB/s, prosumer | 2.34× |
| H100 | ~3.3 TB/s, datacenter | 0.99× (parity) |
The 4-bit memory win is universal; the decode speedup is largest on bandwidth-limited consumer and prosumer cards, and ties on a bandwidth-rich H100 where fp16 is already near roofline. Full per-GPU table.
Runs real models, in pure Rust
No Python at inference. Eight architectures compressed to 4-bit and decoded coherently by the same runtime, including the hard modern ones.
Export is torch-free too: weights compress straight from safetensors, no PyTorch or transformers dependency. See which of these models fit your GPU.
What it unlocks
The point is not a faster benchmark. It is what becomes possible when a frontier model fits on hardware you already own.
Frontier models on your own hardware
70B on a single 24 GB consumer card, measured, not promised: Llama-2-70B in 2 bits decodes at 38.4 tok/s on one RTX 4090 in this runtime (November 2026, on the repository's main branch, not yet in the app; perplexity 4.49 on the full wikitext-2 test set, against 5.72 for the new exporter's 3.75 GB 7B 4-bit file); its PV-tuned 1x16 version scores 4.02 and decodes at 22.5 tok/s with exact speculative decoding, and Llama-3.1-70B-Instruct, a conversational 70B, reaches 21.3 tok/s on chat prompts with a 2-bit Llama-3.1-8B-Instruct drafter. DeepSeek-R1 671B serves the full model at 2.46 tok/s in this runtime today, on one commodity node, by keeping the experts in RAM and decoding them in place on the CPU, a x10.2 gain over disk offload.
Private by default
Everything runs locally and offline. No cloud, no account, no data leaving the machine.
Cheaper, greener serving
Roughly half the energy per token. Lower cost per request and a smaller footprint at scale.
New architectures, quickly
Mixture-of-Experts, Multi-head Latent Attention and logit softcaps are already ported and running. The kernel generalizes.
Convinced by the numbers?
Every figure above, except the November 2026 results, reproduces with one command. The next step takes one line.
Run it See the numbers Read the paper