Results

HuggingFaceTB/SmolLM2-135M — 134.5M parameter decoder-only language model.

Benchmark config: seq_len=128, float32, input=[0,1,…,127].

Correctness: PyTorch vs ONNX Runtime: PASS (loss diff 3.2e-3). PyTorch vs JAX: PASS (loss diff 3.2e-3). PyTorch vs Meganeura: PASS (max error 1.7e-6, loss diff 5.3e-3). PyTorch vs llama.cpp: PASS (loss diff 4.5e-3). Candle, Luminal: CLOSE. Struck-through values are from frameworks running a different (simplified) model.

Caveats: - PyTorch and Meganeura load real model weights and run the full architecture — their outputs match.

• Candle runs the real LLaMA architecture with the same safetensors weights, but its forward() returns last-position logits only (private fields prevent getting all-position logits). Loss is computed on 1 position vs 128 for others — hence DIFFERENT MODEL in correctness check. Timing is valid. Backward not yet wired.
• Burn and Luminal use a simplified model (single-head attention, no RoPE/RMSNorm) with random weights.
• Luminal backward is estimated as a second forward pass.

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m SmolLM2-135M

lerobot/smolvla_base — SmolVLA action expert decoder for robotics.

Benchmark config: chunk_size=50, vlm_seq_len=16, float32, random weights, MSE loss.

Correctness: PyTorch vs Meganeura: CLOSE (loss diff 1e-5, max error 4.6e-3).

Caveats: - PyTorch and Meganeura implement the full action expert architecture and should produce matching outputs.

• Burn and Luminal do not implement this architecture yet (reported as ✗).
• Inputs are synthetic: random noisy actions, sinusoidal timestep, random VLM context.

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m SmolVLA

stable-diffusion-v1-5/stable-diffusion-v1-5 — Latent diffusion model for text-to-image generation.

Most frameworks (PyTorch, Meganeura, ONNX Runtime, JAX, MLX) run the simplified U-Net — Conv + GroupNorm + skip connections, no cross-attention or timestep embedding. Batch 2, 32×32×4 latent, base_channels=64, 3 levels, ~2M params. Shared architecture, but each framework uses its own random-init parameters, so losses don’t match across frameworks and several end up marked DIFFERENT MODEL even on identical structure.

Candle runs the full SD 1.5 U-Net (~860M params, 64×64×4 latent, cross-attention + timestep) — the real thing, marked DIFFERENT MODEL by design.

Run ./run.sh -m StableDiffusion to populate this table.

Caveats: - Only the UNet is benchmarked (not VAE encode/decode or text encoding).

• Input is deterministic synthetic data — no actual image generation.
• PyTorch and Meganeura use a simplified architecture for fair comparison.
• Candle runs the full SD 1.5 UNet but on CPU only (DIFFERENT MODEL vs others).

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m StableDiffusion

openai/whisper-tiny — Encoder-decoder transformer for speech recognition. ~39M parameters.

Uses a custom tiny configuration (4 encoder + 4 decoder layers) for fast benchmarking.

Benchmark config: 30s mel spectrogram (80x3000), 4-token decoder input, float32, random weights.

Correctness: PyTorch vs ONNX Runtime: PASS (loss diff 0.0).

Caveats: - Uses a custom tiny config (4+4 layers, d=384), not the full whisper-tiny from OpenAI

• Input is synthetic mel spectrogram, not real audio
• Decoder runs with a 4-token input (language/task tokens), not full transcription

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m Whisper-tiny