Model overview
Nenya is a general purpose 0.5B active, 2.5B total parameter mixture-of-experts (MoE) base model. Nenya rivals Qwen3.5 2B on pretraining benchmarks with 500× lower training compute and 3.4× fewer active FLOPs/token.
| Architecture | Mixture-of-Experts (MoE) |
|---|---|
| Total Parameters | 2.59B |
| Activated Parameters | 553M |
| Routed-Expert Parameters | 1.59B |
| Engram Table Parameters | 528M |
| Number of Layers | 32 |
| Number of Dense Layers | 2 (blocks 0–1) |
| Attention-Layer Composition | 23 (GDN-2) + 9 (MLA) |
| Attention Hidden Dimension | 960 |
| Number of Attention Heads | 12 |
| Dense FFN Hidden Dimension | 2560 (SwiGLU) |
| Latent MoE Dimension | 384 |
| MoE Hidden Dimension (per Expert) | 384 |
| Number of Experts | 120 |
| Selected Experts per Token | 6 |
| Number of Shared Experts | 2 |
| Expert Routing | quantile-balanced |
| Residual Streams | 4 |
| Vocabulary Size | 100,278 |
| Padding Rows | 74 |
| Context Length | 16,384 |
| Attention Mechanism | GDN-2 & MLA |
| Activation Function | SwiGLU |
| Precision | bf16 |
| Modality | Text |
Quickstart
Any device with transformers:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
repo = "ArsalanZ/Nenya-2.5B-A0.5B-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to(device)
inputs = tok("The mitochondrion is the organelle responsible for", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=40) # samples: temperature 0.9, top-k 20 (the default)
print(tok.decode(out[0], skip_special_tokens=True))
# maths, code and short factual answers: greedy decoding
out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
Any device with PyPI
Install it with pip install "nenya[cuda]" on NVIDIA
GPUs to get the flash-linear-attention kernels and pip install "nenya" otherwise. With the pip package, trust_remote_code is not needed:
import torch
import nenya # registers the model with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
repo = "ArsalanZ/Nenya-2.5B-A0.5B-Base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)
On a Mac with Apple silicon, use the Metal engine for maximum throughput:
pip install "nenya[mac]"
import nenya
print(nenya.generate("The mitochondrion is the organelle responsible for", max_new_tokens=40))
print(nenya.generate("Question: What is the capital of Australia?\nAnswer:", max_new_tokens=8, temperature=0))
It runs on the Mac's GPU, about 125 tok/s on an M4, and downloads its 3.5 GB weight file on first use. It samples
with the model's defaults (temperature 0.9, top-k 20); temperature=0 is greedy decoding, for maths, code and short
factual answers.
- Kernels: on NVIDIA GPUs, install flash-linear-attention (the
cudaextra, orpip install flash-linear-attention) so the GDN-2 layers run on its Triton kernels. Without it, on a CPU, or through transformers on a Mac, they run a pure-PyTorch recurrence: same results, much slower, especially on long prompts. - Precision: use bf16, as it was trained and evaluated in. fp32 also works.
Evaluation
Pretraining Benchmarks
Scored with lm-evaluation-harness 0.4.13 through transformers
(--model hf), from log-likelihoods only: no task generates text. Every model runs with the same settings:
- bf16 weights, fp32 log-softmax, 4,096-token windows
- zero-shot except TriviaQA (5-shot, otherwise Qwen's think token invalidates results), lm-eval's default seeds
- the reference models at their pinned commits
Our training data was decontaminated against these benchmarks' evaluation sets.
Four tasks are ours, and their files are in this repo under evaluation/lm_eval_tasks:
mmlu_cloze: lm-eval's 57mmlu_*_continuationtasks, grouped so that the pooled acc_norm is reported. The stock group reports only acc.csqa_clozetriviaqa_5shot_greedyuncheatable_2026_07
Accuracies are in %, ± one standard error.
| lm-eval task | Metric | Our model (0.06T tokens) | OLMo-2 1B (4T tokens) | Qwen3.5 0.8B (>10T tokens) | Qwen3.5 2B (>10T tokens) | Gemma 4 E2B (>10T tokens) | SmolLM3 3B (11.2T tokens) |
|---|---|---|---|---|---|---|---|
hellaswag |
acc_norm | 63.3 ± 0.5 | 68.3 ± 0.5 | 54.9 ± 0.5 | 66.0 ± 0.5 | 73.8 ± 0.4 | 75.2 ± 0.4 |
arc_easy |
acc | 75.1 ± 0.9 | 72.9 ± 0.9 | 70.5 ± 0.9 | 76.2 ± 0.9 | 79.7 ± 0.8 | 78.5 ± 0.8 |
arc_challenge |
acc_norm | 39.5 ± 1.4 | 41.6 ± 1.4 | 40.4 ± 1.4 | 45.8 ± 1.5 | 52.0 ± 1.5 | 52.1 ± 1.5 |
winogrande |
acc | 61.2 ± 1.4 | 64.5 ± 1.3 | 60.5 ± 1.4 | 65.7 ± 1.3 | 65.8 ± 1.3 | 67.9 ± 1.3 |
piqa |
acc_norm | 75.8 ± 1.0 | 75.8 ± 1.0 | 71.7 ± 1.1 | 74.9 ± 1.0 | 78.7 ± 1.0 | 78.6 ± 1.0 |
sciq |
acc_norm | 95.5 ± 0.7 | 95.2 ± 0.7 | 91.9 ± 0.9 | 93.2 ± 0.8 | 96.0 ± 0.6 | 92.1 ± 0.9 |
openbookqa |
acc_norm | 41.0 ± 2.2 | 39.8 ± 2.2 | 35.6 ± 2.1 | 37.6 ± 2.2 | 44.4 ± 2.2 | 45.0 ± 2.2 |
lambada_openai |
acc | 59.2 ± 0.7 | 64.9 ± 0.7 | 50.8 ± 0.7 | 58.4 ± 0.7 | 67.2 ± 0.7 | 69.7 ± 0.6 |
mmlu_cloze |
acc_norm | 39.3 ± 0.4 | 39.0 ± 0.4 | 35.3 ± 0.4 | 38.1 ± 0.4 | 44.2 ± 0.4 | 44.7 ± 0.4 |
csqa_cloze |
acc_norm | 45.8 ± 1.4 | 55.8 ± 1.4 | 39.1 ± 1.4 | 39.5 ± 1.4 | 53.2 ± 1.4 | 49.2 ± 1.4 |
xstorycloze_en |
acc | 70.5 ± 1.2 | 74.2 ± 1.1 | 64.7 ± 1.2 | 70.8 ± 1.2 | 75.0 ± 1.1 | 76.4 ± 1.1 |
triviaqa_5shot_greedy |
acc | 33.3 ± 0.7 | 40.9 ± 0.8 | 17.8 ± 0.6 | 31.1 ± 0.7 | 47.2 ± 0.8 | 49.0 ± 0.8 |
| avg | mean of all 12 | 58.3 | 61.1 | 52.8 | 58.1 | 64.8 | 64.9 |
- TriviaQA: 5-shot, on a fixed 4,000-question subset of the validation set. A question counts as right if
greedy decoding reproduces any accepted alias, checked from log-likelihoods without generating. The examples
show the short-answer format, so models that otherwise answer in sentences or
<think>blocks (Qwen3.5 Base) are measured on knowledge.
UncheatableEval, July 2026 (bits per byte, lower is better)
The text was first published 1–25 July 2026 (Jellyfish042/UncheatableEval-2026-07), after every compared model's
training data. Each category has 500 documents, each scored as a whole after the model's document-start token, in
4,096-token windows. Each group's average is the plain mean of its categories.
| Group | Our model (0.06T tokens) | OLMo-2 1B (4T tokens) | Qwen3.5 0.8B (>10T tokens) | Qwen3.5 2B (>10T tokens) | Gemma 4 E2B (>10T tokens) | SmolLM3 3B (11.2T tokens) |
|---|---|---|---|---|---|---|
| English (3 categories) | 0.815 | 0.788 | 0.911 | 0.833 | 0.745 | 0.741 |
| Science: arXiv and bioRxiv (5 categories) | 0.674 | 0.677 | 0.667 | 0.605 | 0.624 | 0.626 |
| Code: GitHub (5 categories) | 0.637 | 0.676 | 0.578 | 0.505 | 0.500 | 0.507 |
Code and maths (bits per byte of the reference solution, lower is better)
lm-eval scores GSM8K, MBPP and HumanEval only by generating answers. These rows come from the in-repo harness
(pretraining/benchmarks.py, context 1,024): the reference models in fp32 with TF32 matmuls, this model in bf16.
| Task | Our model (0.06T tokens) | OLMo-2 1B (4T tokens) | Qwen3.5 0.8B (>10T tokens) | Qwen3.5 2B (>10T tokens) | Gemma 4 E2B (>10T tokens) | SmolLM3 3B (11.2T tokens) |
|---|---|---|---|---|---|---|
| gsm8k_bpb | 0.446 | 0.430 | 0.459 | 0.420 | 0.452 | 0.396 |
| mbpp_bpb | 0.295 | 0.431 | 0.315 | 0.279 | 0.308 | 0.238 |
| humaneval_bpb | 0.292 | 0.368 | 0.280 | 0.243 | 0.246 | 0.216 |
Verify our results
The custom task files ship in this repo. On a CUDA GPU:
# our runs: torch 2.11.0+cu128 on an RTX PRO 6000 Blackwell, datasets 5.0.1, huggingface_hub 1.33.0
pip install "lm-eval[hf]==0.4.13" "transformers==5.18.0" "torch==2.11.0" "flash-linear-attention==0.5.2" \
"datasets==5.0.1"
hf download ArsalanZ/Nenya-2.5B-A0.5B-Base --include "evaluation/*" --local-dir nenya_eval
TASKS=nenya_eval/evaluation/lm_eval_tasks
ARGS=pretrained=ArsalanZ/Nenya-2.5B-A0.5B-Base,trust_remote_code=True,dtype=bfloat16,softmax_dtype=float32,max_length=4096
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
--batch_size 32 --tasks hellaswag,arc_easy,arc_challenge,winogrande,piqa,sciq,openbookqa,lambada_openai,xstorycloze_en,csqa_cloze
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
--batch_size 16 --tasks mmlu_cloze
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
--batch_size 32 --tasks triviaqa_5shot_greedy
lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results \
--batch_size 2 --tasks uncheatable_2026_07
To score a reference model, set ARGS to its arguments below followed by ,dtype=bfloat16,softmax_dtype=float32,max_length=4096.
Qwen3.5, Gemma 4 and SmolLM3 ran at half these batch sizes (their vocabularies are 128k–262k tokens). bf16 scores
can move slightly with the GPU and the batch size.
| Model | --model_args |
|---|---|
| Our model | pretrained=ArsalanZ/Nenya-2.5B-A0.5B-Base,trust_remote_code=True |
| OLMo-2 1B | pretrained=allenai/OLMo-2-0425-1B,revision=a1847dff35000b4271fa70afc5db10fd29fedbdf |
| Qwen3.5 0.8B | pretrained=Qwen/Qwen3.5-0.8B-Base,revision=dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 |
| Qwen3.5 2B | pretrained=Qwen/Qwen3.5-2B-Base,revision=b1485b2fa6dfa1287294f269f5fb618e03d52d7c |
| Gemma 4 E2B | pretrained=google/gemma-4-E2B,revision=d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f,add_bos_token=True |
| SmolLM3 3B | pretrained=HuggingFaceTB/SmolLM3-3B-Base,revision=d78a42f79198603e614095753484a04c10c2b940,prefix_token_id=128000 |
Limitations
- A base model. It continues text, it does not follow instructions, and it can produce false, biased or unsafe content.
- Trained mostly on English web, educational, academic, code and maths data.
- Quality past 16,384 tokens is untested. With no positional encoding there is no hard limit.
License
Apache 2.0: see LICENSE. NOTICE lists the third-party material: nenya_gdn2.py is derived from flash-linear-attention (MIT), and the tokenizer files come from allenai/Olmo-3.1-32B-Instruct (Apache 2.0).
Citation
Nenya builds on Engram, mHC, Gated DeltaNet-2 and Mixture-of-Experts:
% Engram: conditional memory via hashed n-gram lookup
@misc{cheng2026conditionalmemoryscalablelookup,
title={Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models},
author={Xin Cheng and Rui Tian and Wangding Zeng and Damai Dai and Qinyu Chen and Bingxuan Wang and Zhenda Xie and Kezhao Huang and Xingkai Yu and Chengqi Deng and Shangyan Zhou and Chenggang Zhao and Zhewen Hao and Yukun Li and Han Zhang and Zhengyan Zhang and Yixu Wei and M. Y Xu and Huishuai Zhang and Dongyan Zhao and Wenfeng Liang},
year={2026},
eprint={2601.07372},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2601.07372},
}
% mHC: manifold-constrained hyper-connections
@misc{xie2026mhcmanifoldconstrainedhyperconnections,
title={mHC: Manifold-Constrained Hyper-Connections},
author={Zhenda Xie and Yixuan Wei and Huanqi Cao and Chenggang Zhao and Chengqi Deng and Jiashi Li and Damai Dai and Huazuo Gao and Jiang Chang and Kuai Yu and Liang Zhao and Shangyan Zhou and Zhean Xu and Zhengyan Zhang and Wangding Zeng and Shengding Hu and Yuqing Wang and Jingyang Yuan and Lean Wang and Wenfeng Liang},
year={2026},
eprint={2512.24880},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.24880},
}
% Gated DeltaNet-2
@misc{hatamizadeh2026gateddeltanet2decouplingerase,
title={Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention},
author={Ali Hatamizadeh and Yejin Choi and Jan Kautz},
year={2026},
eprint={2605.22791},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2605.22791},
}
% Mixture-of-Experts: the sparsely-gated MoE layer
@misc{shazeer2017outrageouslylargeneuralnetworks,
title={Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer},
author={Noam Shazeer and Azalia Mirhoseini and Krzysztof Maziarz and Andy Davis and Quoc Le and Geoffrey Hinton and Jeff Dean},
year={2017},
eprint={1701.06538},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/1701.06538},
}
% Mixture-of-Experts: fine-grained and shared experts (DeepSeekMoE)
@misc{dai2024deepseekmoeultimateexpertspecialization,
title={DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
author={Damai Dai and Chengqi Deng and Chenggang Zhao and R. X. Xu and Huazuo Gao and Deli Chen and Jiashi Li and Wangding Zeng and Xingkai Yu and Y. Wu and Zhenda Xie and Y. K. Li and Panpan Huang and Fuli Luo and Chong Ruan and Zhifang Sui and Wenfeng Liang},
year={2024},
eprint={2401.06066},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2401.06066},
}