GitHub - santosardr/riskernel: RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

This repository contains the official implementation of RIS-Kernel, a systems-level sparse attention inference engine that runs massive context windows (64k+ tokens) on commodity, unaccelerated CPU hardware.

📖 Abstract

Full self-attention in large language models scales as $O(N^2)$, limiting long-context document analysis to 65,536 tokens and requiring costly GPU clusters. The Reduced Interaction Sampling (RIS) inference engine addresses this constraint as a model-agnostic architecture. Without modifying weights, RIS reduces self-attention complexity to $O(N \log N)$ using sparse stochastic geometry that fits within commodity memory limits. We validate RIS on Qwen2-1.5B-Instruct across two regimes. In controlled evaluations at 32,768 tokens (where native dense attention serves as the upper bound), RIS-Stochastic at 1% density and 70 ensemble seeds achieves 75.00% accuracy, outperforming the native dense baseline (71.88%), while RIS-Stochastic at 5% density and 10 seeds matches it (71.88%). This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise, whereas higher density (5%) reintroduces distractor noise. Under the tightest budget, RIS-Structural reaches 68.75% accuracy at 1% density with just 10 seeds, recovering 75% of the contextual gap relative to the zero-context floor (59.38%). At 65,536 tokens, where dense attention triggers out-of-memory faults, RIS yields retrieval gains of up to 14.06 percentage points over the zero-context floor (51.56%). All evaluations run on commodity, unaccelerated CPU servers (16–128 GB of RAM), demonstrating that long-context LLM inference is feasible on standard academic hardware without GPU acceleration.

🔬 Scientific Context & PoC

RIS-Kernel acts as a model-agnostic layer that intercepts attention calls at runtime. By implementing Reduced Interaction Sampling (RIS), it bypasses the $O(N^2)$ memory and compute bottleneck of standard Transformers.

We utilize Qwen2-1.5B as a primary Proof of Concept (PoC), demonstrating that RIS maintains contextual coherence even under severe parameter constraints. Release V4 expands this architecture across model scales up to 32-billion parameter LLMs.

📰 News, Articles & Media Highlights

Explore technical analyses, benchmark comparisons, and reflections on RIS-Kernel:

• 🏎️ [Aug 2026] The Long-Context Inference Grand Prix: Why RIS-Kernel Outpaces Current LLM Optimizations
  A 4-way technical comparison (RIS-Kernel vs. FlashAttention, KV Quantization & Linear Attention).
  👉 Read Article

• 🤖 [Aug 2026] Evolution to 32B Models: Scaling Long-Context Inference on Disk
  5-step evolution scaling RIS-Kernel to 32B parameter models on consumer disk hardware.
  👉 Read Article

• 💭 [Aug 2026] "That's the Question": Reflections on Innovation Diffusion in Science
  An essay on Everett Rogers' Diffusion of Innovations and Mark Granovetter's Threshold Model.
  👉 Read Essay

• 🤗 [Jul 2026] Hugging Face Daily Papers Highlight
  Featured in the Hugging Face Daily Papers index.
  👉 View on Hugging Face | View on alphaXiv

• 📂 View All Articles & Media Index

⚡ Hardware Requirements & Performance Guidance

This implementation is optimized for CPU-only execution on commodity academic hardware (standard workstations or departmental servers) while remaining fully compatible with GPU environments (CUDA / PyTorch GPU):

• GPU Acceleration: RIS-Kernel drops peak VRAM at 40,000 tokens from 36.2 GB down to ~9.4 GB (-75% reduction), allowing 40k+ context windows to run on single free cloud GPUs (e.g., NVIDIA T4 15 GB on Google Colab or Kaggle) with generation speeds of multiple tokens per second.
• CPU Execution:
  • RAM Requirements: ~16–32 GB RAM for 1.5B models; ~64–128 GB RAM for 32B models.
  • NUMA Acceleration: On multi-socket Xeon servers, prefix commands with numactl --interleave=all.
  • CPU Multi-Threading: Set --threads 8 to avoid bus contention on multi-core servers.

🛠️ Components

• scripts/ris_attention_v4.py: Enhanced RIS sparse attention kernel supporting Qwen2.5 (1.5B–32B) with dynamic boundary safeguards (v4).
• scripts/inference_ris_v4.py: Production inference engine with multi-scale model support (1.5B to 32B), streaming prefill, schema-aware batch QA, and GPU/CPU auto-dispatch.
• scripts/ris_attention.py: Core implementation of Reduced Interaction Sampling sparse attention geometry (v3 legacy).
• scripts/inference_ris_v3.py: Legacy CPU-bound inference engine for 7B/14B models (v3 legacy).
• download_model.py: Model downloader for Qwen2, Qwen2.5, and TinyLlama checkpoints.
• extract_pdf.py: Text optimizer tailored for academic PDFs (strips hyphens, reconstructs paragraphs).
• scripts/benchmark/sample_qa_open_ended.json: Reference dataset with 10 open-ended scientific questions.
• scripts/benchmark/sample_qa_multiple_choice.json: Reference dataset with 10 multiple-choice questions (balanced A–E options).

🚀 Getting Started

🌐 1. Try Live Without Installing (Hugging Face, Colab & Kaggle)

If you want to experiment with RIS-Kernel immediately without setting up a local environment or installing any software, try these zero-install options:

1. 🤗 Hugging Face Space Live Web Demo: Try the interactive web interface directly in your browser: 👉 Hugging Face Space Demo

2. ⚡ Free Cloud Notebooks with 15 GB GPUs (Google Colab & Kaggle): Run RIS-Kernel on free cloud T4 GPUs (15 GB VRAM) with zero installation required:

💻 2. Local Installation (CPU-only)

python3 -m venv venv
source venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements-cpu.txt

(Note: For GPU execution, install standard CUDA PyTorch: pip install torch torchvision torchaudio)

3. Download Models

Run the downloader script to fetch model weights:

python download_model.py
# Select Qwen2 1.5B or Qwen2.5 32B

4. Prepare Context

Extract and reconstruct clean text from biochemical or biomedical PDFs (e.g. textbook chapters or articles):

python extract_pdf.py articles/your_paper.pdf data/processed_context.txt

5. Run Inference

A. Non-Interactive Prompt Mode (Single Query or File)

To run a single query against long documents non-interactively and exit immediately:

PYTHONPATH=code/scripts python code/scripts/inference_ris_v4.py \
  --model_class qwen2 \
  --window 31744 \
  --chat_buffer 1024 \
  --context_files code/scripts/data/genppi.txt \
  --prompt "What is the role of Random Forest in GenPPi?"

You can also read the prompt query directly from a text file:

PYTHONPATH=code/scripts python code/scripts/inference_ris_v4.py \
  --model_class qwen2 \
  --window 31744 \
  --chat_buffer 1024 \
  --context_files code/scripts/data/genppi.txt \
  --prompt_file data/prompt.txt

B. Interactive Chat Mode

To start a multi-turn chat session inside your terminal, ensure that the total capacity (window + chat_buffer) does not exceed the model's native ceiling (e.g., 32,768 tokens for Qwen2 1.5B):

Option 1 (Custom chat buffer, maximizing document window):

PYTHONPATH=code/scripts python code/scripts/inference_ris_v4.py \
  --model_class qwen2 \
  --window 30000 \
  --chat_buffer 2768 \
  --context_files code/scripts/data/genppi.txt

Option 2 (Default chat buffer of 8192, reducing window to compensate):

PYTHONPATH=code/scripts python code/scripts/inference_ris_v4.py \
  --model_class qwen2 \
  --window 24576 \
  --context_files code/scripts/data/genppi.txt

C. Standard Long-Context Execution

Launch the inference engine on single context documents using --dtype bfloat16 for optimal precision and memory efficiency:

python code/scripts/inference_ris_v4.py \
  --model_name Qwen/Qwen2-1.5B-Instruct \
  --window 32768 \
  --dtype bfloat16 \
  --density 0.01 \
  --n_seeds 70 \
  --context_files code/scripts/data/genppi.txt

Key Arguments:

• --window: Context window size in tokens.
• --density: Active attention density fraction (e.g., 0.01 for 1%, 0.05 for 5%).
• --n_seeds: Number of stochastically projected masks to ensemble.
• --save_graph: Exports the attention topology to a .dot file.
• --model_class / --model_name: Model family, alias, or HuggingFace ID (Qwen/Qwen2-1.5B-Instruct, qwen2.5-32b, tinyllama).
• --dtype: Precision selection (bfloat16 recommended for optimal RAM efficiency and numerical stability; float32 available for Haswell CPUs).
• --prompt_file: Path to a prompt file or batch QA JSON dataset.
• --output_file: Destination file for generated responses or CSV results.

🌟 Advanced Release V4 Execution Examples

Example A: Qwen2.5-32B Large Model Execution (NUMA Accelerated)

Run large-scale 32-billion parameter models over multi-document contexts with streaming chunked prefill:

numactl --interleave=all python code/scripts/inference_ris_v4.py \
  --model_class qwen2.5-32b \
  --window 87040 \
  --density 0.02 \
  --n_seeds 70 \
  --temp 0.0 \
  --repetition_penalty 1.0 \
  --context_files code/scripts/data/genppi.txt,code/scripts/data/aom.txt,code/scripts/data/ajinshanensis.txt,code/scripts/data/meta.txt \
  --prompt_file code/scripts/benchmark/sample_qa_open_ended.json \
  --output_file results/output_responses.txt

Example B: Qwen2 (1.5B) Fast Benchmark Sweep (Discriminative Multiple-Choice QA)

Run discriminative logit scoring across structured JSON datasets with isolated per-question KV-cache cropping using --dtype bfloat16:

python code/scripts/inference_ris_v4.py \
  --model_name Qwen/Qwen2-1.5B-Instruct \
  --window 32768 \
  --dtype bfloat16 \
  --density 0.01 \
  --n_seeds 70 \
  --temp 0.0 \
  --repetition_penalty 1.0 \
  --context_files code/scripts/data/genppi.txt,code/scripts/data/meta.txt,code/scripts/data/aom.txt,code/scripts/data/ajinshanensis.txt \
  --prompt_file code/scripts/benchmark/sample_qa_multiple_choice.json \
  --output_file results/benchmark_results_mc.csv

📈 Empirical Validation & Benchmarks

In real-world CPU-only evaluation on an unaccelerated academic server (16 GB RAM, 0 GPUs), Qwen2-1.5B-Instruct running under RIS-Kernel v4 (--dtype bfloat16, --density 0.01, --n_seeds 70, --window 32768) across 4 full-length academic papers (genppi.txt, meta.txt, aom.txt, ajinshanensis.txt) demonstrated state-of-the-art accuracy and memory efficiency:

📊 Visualization

You can export the sparse attention topology with the --save_graph flag. Open the resulting .dot file in Graphviz or Gephi to inspect the attention retrieval maps.

📢 Release Notes & Changelog

See RELEASE_NOTES_V4.md for full details on Release V4 capabilities, multi-scale model scaling, streaming prefill metrics, and memory optimization benchmarks.

📄 License & Citation

The code is available for scientific transparency and reproducibility under the MIT License. If you use this work, please cite both the theoretical paper and the repository implementation:

@article{santos2026ris,
  author    = {Santos, Anderson R.},
  title     = {Towards million-token context windows: a topology-preserving framework for adaptive transformer sparsification},
  journal   = {Scientific Reports},
  year      = {2026},
  doi       = {10.1038/s41598-026-59160-z},
  url       = {https://doi.org/10.1038/s41598-026-59160-z}
}

@misc{santos2026riskernelmodelagnosticarchitecturelongcontext,
  title     = {RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention}, 
  author    = {Santos, Anderson R.},
  year      = {2026},
  eprint    = {2607.21927},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url       = {https://arxiv.org/abs/2607.21927}, 
}