🍋 Second Squeeze

The 2019 vs 2025 local-AI stack, squeezed on one laptop. A smaller 2025 model runs ~11× faster on the same CPU and actually answers. Measured carefully, and re-measured when the first number lied.

Same CPU, 2019 vs 2025
11×
faster token generation, smaller 2025 model over bigger 2019 model, identical processor
First token
24×
quicker to start answering
GPU vs CPU (2025 model)
3.8×
the lift from the graphics card
The number that lied. My first GPT-2 run measured 0.3 tok/s, which would make this a ~297× gap. But that run was contaminated: the GPU backend was still downloading and the 6.8 GB model was cold off disk while it timed itself. The reusable bench_all.py rerun gives a reproducible 8.5 tok/s, so the real gap is ~11×. The tool I built to make this rigorous caught my own headline. That's the whole point of the project, in one incident.

The squeeze: same processor, six years apart

Both models on the same CPU, so this measures the models, not the hardware. Tokens generated per second, higher is better.
GPT-2 XL · 2019 · 1.5B Qwen3-0.6B · 2025 · 0.6B

The hardware lift: same model, CPU vs GPU

The 2025 model on the same laptop, processor versus graphics card.
CPU GPU (NVIDIA)

Same prompt, different decade

Prompt sent to both: The capital of France is. GPT-2 is a base model with no instruction-following, so it just continues text. Qwen follows the instruction and answers.

The receipts

How this was measured, and what it isn't

  • All numbers from Lemonade's bench tool: . Same settings for every model, regenerated by bench_all.py so anyone can reproduce them.
  • The number lied twice, and I caught it twice. An early run compared GPT-2 on CPU to Qwen on GPU (a ~1000× gap that was mostly hardware); holding the backend fixed on CPU fixed that. And the very first GPT-2 CPU run read 0.3 tok/s under contaminated conditions; the reproducible rerun reads 8.5. Control your variables, and re-run, or the benchmark fools you.
  • This is the practical 2019 stack vs the practical 2025 stack, not an isolation of architecture. The ~11× reflects everything at once: fewer parameters, better quantization (Q8 vs Q4), instruction tuning, and runtime support. Every difference is named on purpose.
  • GPT-2 XL only ran on CPU; its 2019 architecture isn't supported on the GPU backend.
  • Accuracy benchmarks (LAMBADA, MMLU) aren't here: they're loglikelihood-based and need per-token logprobs for the prompt tokens, which my lm-eval run couldn't get from Lemonade's completions endpoint (a token_logprobs error). Lemonade documents a logprobs option for output tokens, so it's the prompt-token side that's missing. Filed as issue #2822; I measure speed and capability instead.