ArsalanZ/Nenya-2.5B-A0.5B-Base · Hugging Face

8 min read Original article ↗

Technical report Metal engine weights PyPI

License

Model overview

Nenya is a general purpose 0.5B active, 2.5B total parameter mixture-of-experts (MoE) base model. Nenya rivals Qwen3.5 2B on pretraining benchmarks with 500× lower training compute and 3.4× fewer active FLOPs/token.

Architecture Mixture-of-Experts (MoE)
Total Parameters 2.59B
Activated Parameters 553M
Routed-Expert Parameters 1.59B
Engram Table Parameters 528M
Number of Layers 32
Number of Dense Layers 2 (blocks 0–1)
Attention-Layer Composition 23 (GDN-2) + 9 (MLA)
Attention Hidden Dimension 960
Number of Attention Heads 12
Dense FFN Hidden Dimension 2560 (SwiGLU)
Latent MoE Dimension 384
MoE Hidden Dimension (per Expert) 384
Number of Experts 120
Selected Experts per Token 6
Number of Shared Experts 2
Expert Routing quantile-balanced
Residual Streams 4
Vocabulary Size 100,278
Padding Rows 74
Context Length 16,384
Attention Mechanism GDN-2 & MLA
Activation Function SwiGLU
Precision bf16
Modality Text

Quickstart

Any device with transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
repo = "ArsalanZ/Nenya-2.5B-A0.5B-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to(device)

inputs = tok("The mitochondrion is the organelle responsible for", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=40)     # samples: temperature 0.9, top-k 20 (the default)
print(tok.decode(out[0], skip_special_tokens=True))

# maths, code and short factual answers: greedy decoding
out = model.generate(**inputs, max_new_tokens=40, do_sample=False)

Any device with PyPI

Install it with pip install "nenya[cuda]" on NVIDIA GPUs to get the flash-linear-attention kernels and pip install "nenya" otherwise. With the pip package, trust_remote_code is not needed:

import torch
import nenya                                    # registers the model with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
repo = "ArsalanZ/Nenya-2.5B-A0.5B-Base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)

On a Mac with Apple silicon, use the Metal engine for maximum throughput:

pip install "nenya[mac]"
import nenya
print(nenya.generate("The mitochondrion is the organelle responsible for", max_new_tokens=40))
print(nenya.generate("Question: What is the capital of Australia?\nAnswer:", max_new_tokens=8, temperature=0))

It runs on the Mac's GPU, about 125 tok/s on an M4, and downloads its 3.5 GB weight file on first use. It samples with the model's defaults (temperature 0.9, top-k 20); temperature=0 is greedy decoding, for maths, code and short factual answers.

  • Kernels: on NVIDIA GPUs, install flash-linear-attention (the cuda extra, or pip install flash-linear-attention) so the GDN-2 layers run on its Triton kernels. Without it, on a CPU, or through transformers on a Mac, they run a pure-PyTorch recurrence: same results, much slower, especially on long prompts.
  • Precision: use bf16, as it was trained and evaluated in. fp32 also works.

Evaluation

Pretraining Benchmarks

Scored with lm-evaluation-harness 0.4.13 through transformers (--model hf), from log-likelihoods only: no task generates text. Every model runs with the same settings:

  • bf16 weights, fp32 log-softmax, 4,096-token windows
  • zero-shot except TriviaQA (5-shot, otherwise Qwen's think token invalidates results), lm-eval's default seeds
  • the reference models at their pinned commits

Our training data was decontaminated against these benchmarks' evaluation sets.

Four tasks are ours, and their files are in this repo under evaluation/lm_eval_tasks:

  • mmlu_cloze: lm-eval's 57 mmlu_*_continuation tasks, grouped so that the pooled acc_norm is reported. The stock group reports only acc.
  • csqa_cloze
  • triviaqa_5shot_greedy
  • uncheatable_2026_07

Accuracies are in %, ± one standard error.

lm-eval task Metric Our model (0.06T tokens) OLMo-2 1B (4T tokens) Qwen3.5 0.8B (>10T tokens) Qwen3.5 2B (>10T tokens) Gemma 4 E2B (>10T tokens) SmolLM3 3B (11.2T tokens)
hellaswag acc_norm 63.3 ± 0.5 68.3 ± 0.5 54.9 ± 0.5 66.0 ± 0.5 73.8 ± 0.4 75.2 ± 0.4
arc_easy acc 75.1 ± 0.9 72.9 ± 0.9 70.5 ± 0.9 76.2 ± 0.9 79.7 ± 0.8 78.5 ± 0.8
arc_challenge acc_norm 39.5 ± 1.4 41.6 ± 1.4 40.4 ± 1.4 45.8 ± 1.5 52.0 ± 1.5 52.1 ± 1.5
winogrande acc 61.2 ± 1.4 64.5 ± 1.3 60.5 ± 1.4 65.7 ± 1.3 65.8 ± 1.3 67.9 ± 1.3
piqa acc_norm 75.8 ± 1.0 75.8 ± 1.0 71.7 ± 1.1 74.9 ± 1.0 78.7 ± 1.0 78.6 ± 1.0
sciq acc_norm 95.5 ± 0.7 95.2 ± 0.7 91.9 ± 0.9 93.2 ± 0.8 96.0 ± 0.6 92.1 ± 0.9
openbookqa acc_norm 41.0 ± 2.2 39.8 ± 2.2 35.6 ± 2.1 37.6 ± 2.2 44.4 ± 2.2 45.0 ± 2.2
lambada_openai acc 59.2 ± 0.7 64.9 ± 0.7 50.8 ± 0.7 58.4 ± 0.7 67.2 ± 0.7 69.7 ± 0.6
mmlu_cloze acc_norm 39.3 ± 0.4 39.0 ± 0.4 35.3 ± 0.4 38.1 ± 0.4 44.2 ± 0.4 44.7 ± 0.4
csqa_cloze acc_norm 45.8 ± 1.4 55.8 ± 1.4 39.1 ± 1.4 39.5 ± 1.4 53.2 ± 1.4 49.2 ± 1.4
xstorycloze_en acc 70.5 ± 1.2 74.2 ± 1.1 64.7 ± 1.2 70.8 ± 1.2 75.0 ± 1.1 76.4 ± 1.1
triviaqa_5shot_greedy acc 33.3 ± 0.7 40.9 ± 0.8 17.8 ± 0.6 31.1 ± 0.7 47.2 ± 0.8 49.0 ± 0.8
avg mean of all 12 58.3 61.1 52.8 58.1 64.8 64.9
  • TriviaQA: 5-shot, on a fixed 4,000-question subset of the validation set. A question counts as right if greedy decoding reproduces any accepted alias, checked from log-likelihoods without generating. The examples show the short-answer format, so models that otherwise answer in sentences or <think> blocks (Qwen3.5 Base) are measured on knowledge.

UncheatableEval, July 2026 (bits per byte, lower is better)

The text was first published 1–25 July 2026 (Jellyfish042/UncheatableEval-2026-07), after every compared model's training data. Each category has 500 documents, each scored as a whole after the model's document-start token, in 4,096-token windows. Each group's average is the plain mean of its categories.

Group Our model (0.06T tokens) OLMo-2 1B (4T tokens) Qwen3.5 0.8B (>10T tokens) Qwen3.5 2B (>10T tokens) Gemma 4 E2B (>10T tokens) SmolLM3 3B (11.2T tokens)
English (3 categories) 0.815 0.788 0.911 0.833 0.745 0.741
Science: arXiv and bioRxiv (5 categories) 0.674 0.677 0.667 0.605 0.624 0.626
Code: GitHub (5 categories) 0.637 0.676 0.578 0.505 0.500 0.507

Code and maths (bits per byte of the reference solution, lower is better)

lm-eval scores GSM8K, MBPP and HumanEval only by generating answers. These rows come from the in-repo harness (pretraining/benchmarks.py, context 1,024): the reference models in fp32 with TF32 matmuls, this model in bf16.

Task Our model (0.06T tokens) OLMo-2 1B (4T tokens) Qwen3.5 0.8B (>10T tokens) Qwen3.5 2B (>10T tokens) Gemma 4 E2B (>10T tokens) SmolLM3 3B (11.2T tokens)
gsm8k_bpb 0.446 0.430 0.459 0.420 0.452 0.396
mbpp_bpb 0.295 0.431 0.315 0.279 0.308 0.238
humaneval_bpb 0.292 0.368 0.280 0.243 0.246 0.216

Verify our results

The custom task files ship in this repo. On a CUDA GPU:

# our runs: torch 2.11.0+cu128 on an RTX PRO 6000 Blackwell, datasets 5.0.1, huggingface_hub 1.33.0
pip install "lm-eval[hf]==0.4.13" "transformers==5.18.0" "torch==2.11.0" "flash-linear-attention==0.5.2" \
    "datasets==5.0.1"
hf download ArsalanZ/Nenya-2.5B-A0.5B-Base --include "evaluation/*" --local-dir nenya_eval
TASKS=nenya_eval/evaluation/lm_eval_tasks
ARGS=pretrained=ArsalanZ/Nenya-2.5B-A0.5B-Base,trust_remote_code=True,dtype=bfloat16,softmax_dtype=float32,max_length=4096
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
    --batch_size 32 --tasks hellaswag,arc_easy,arc_challenge,winogrande,piqa,sciq,openbookqa,lambada_openai,xstorycloze_en,csqa_cloze
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
    --batch_size 16 --tasks mmlu_cloze
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
    --batch_size 32 --tasks triviaqa_5shot_greedy
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
    --batch_size 2 --tasks uncheatable_2026_07

To score a reference model, set ARGS to its arguments below followed by ,dtype=bfloat16,softmax_dtype=float32,max_length=4096. Qwen3.5, Gemma 4 and SmolLM3 ran at half these batch sizes (their vocabularies are 128k–262k tokens). bf16 scores can move slightly with the GPU and the batch size.

Model --model_args
Our model pretrained=ArsalanZ/Nenya-2.5B-A0.5B-Base,trust_remote_code=True
OLMo-2 1B pretrained=allenai/OLMo-2-0425-1B,revision=a1847dff35000b4271fa70afc5db10fd29fedbdf
Qwen3.5 0.8B pretrained=Qwen/Qwen3.5-0.8B-Base,revision=dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68
Qwen3.5 2B pretrained=Qwen/Qwen3.5-2B-Base,revision=b1485b2fa6dfa1287294f269f5fb618e03d52d7c
Gemma 4 E2B pretrained=google/gemma-4-E2B,revision=d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f,add_bos_token=True
SmolLM3 3B pretrained=HuggingFaceTB/SmolLM3-3B-Base,revision=d78a42f79198603e614095753484a04c10c2b940,prefix_token_id=128000

Limitations

  • A base model. It continues text, it does not follow instructions, and it can produce false, biased or unsafe content.
  • Trained mostly on English web, educational, academic, code and maths data.
  • Quality past 16,384 tokens is untested. With no positional encoding there is no hard limit.

License

Apache 2.0: see LICENSE. NOTICE lists the third-party material: nenya_gdn2.py is derived from flash-linear-attention (MIT), and the tokenizer files come from allenai/Olmo-3.1-32B-Instruct (Apache 2.0).

Citation

Nenya builds on Engram, mHC, Gated DeltaNet-2 and Mixture-of-Experts:

% Engram: conditional memory via hashed n-gram lookup
@misc{cheng2026conditionalmemoryscalablelookup,
      title={Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models},
      author={Xin Cheng and Rui Tian and Wangding Zeng and Damai Dai and Qinyu Chen and Bingxuan Wang and Zhenda Xie and Kezhao Huang and Xingkai Yu and Chengqi Deng and Shangyan Zhou and Chenggang Zhao and Zhewen Hao and Yukun Li and Han Zhang and Zhengyan Zhang and Yixu Wei and M. Y Xu and Huishuai Zhang and Dongyan Zhao and Wenfeng Liang},
      year={2026},
      eprint={2601.07372},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.07372},
}

% mHC: manifold-constrained hyper-connections
@misc{xie2026mhcmanifoldconstrainedhyperconnections,
      title={mHC: Manifold-Constrained Hyper-Connections},
      author={Zhenda Xie and Yixuan Wei and Huanqi Cao and Chenggang Zhao and Chengqi Deng and Jiashi Li and Damai Dai and Huazuo Gao and Jiang Chang and Kuai Yu and Liang Zhao and Shangyan Zhou and Zhean Xu and Zhengyan Zhang and Wangding Zeng and Shengding Hu and Yuqing Wang and Jingyang Yuan and Lean Wang and Wenfeng Liang},
      year={2026},
      eprint={2512.24880},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.24880},
}

% Gated DeltaNet-2
@misc{hatamizadeh2026gateddeltanet2decouplingerase,
      title={Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention},
      author={Ali Hatamizadeh and Yejin Choi and Jan Kautz},
      year={2026},
      eprint={2605.22791},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.22791},
}

% Mixture-of-Experts: the sparsely-gated MoE layer
@misc{shazeer2017outrageouslylargeneuralnetworks,
      title={Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer},
      author={Noam Shazeer and Azalia Mirhoseini and Krzysztof Maziarz and Andy Davis and Quoc Le and Geoffrey Hinton and Jeff Dean},
      year={2017},
      eprint={1701.06538},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1701.06538},
}

% Mixture-of-Experts: fine-grained and shared experts (DeepSeekMoE)
@misc{dai2024deepseekmoeultimateexpertspecialization,
      title={DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
      author={Damai Dai and Chengqi Deng and Chenggang Zhao and R. X. Xu and Huazuo Gao and Deli Chen and Jiashi Li and Wangding Zeng and Xingkai Yu and Y. Wu and Zhenda Xie and Y. K. Li and Panpan Huang and Fuli Luo and Chong Ruan and Zhifang Sui and Wenfeng Liang},
      year={2024},
      eprint={2401.06066},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2401.06066},
}