GitHub - RUC-GSAI/YuLan-SwarmIntell: 🐝 SwarmBench: Benchmarking LLMs' Swarm Intelligence

Figure 1: Natural Swarm Intelligence Inspiration and SwarmBench Tasks.

SwarmBench is a novel benchmark designed to systematically evaluate the swarm intelligence capabilities of Large Language Models (LLMs) acting as decentralized agents. It addresses the gap in existing benchmarks by focusing on scenarios with strict constraints, such as limited local perception and communication, characteristic of natural swarms.

Large Language Models (LLMs) show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems (MAS) under such constraints remains largely unexplored. SwarmBench aims to bridge this gap.

Figure 2: Conceptual Architecture of SwarmBench.

Figure 3: Overview of LLM Performance on SwarmBench Tasks

✨ Key Features

Novel Benchmark: Grounded in swarm intelligence principles, assessing emergent decentralized coordination.
Five Foundational MAS Tasks:
- 🎯 Pursuit: Agents collaboratively track and corner a faster-moving prey.
- ⏱️ Synchronization: Agents aim to synchronize an internal binary state across the swarm.
- 🧺 Foraging: Agents navigate to find a food source, transport it to a nest, and coordinate task allocation.
- 🐦 Flocking: Agents must move as a cohesive group, maintaining alignment and separation.
- 🚚 Transport: Multiple agents must cooperate to push a large object towards a designated goal.
Configurable 2D Grid Environment: Forces agents to rely on local sensory input (k × k view) and local communication.
Systematic Evaluation: Characterizes current LLM abilities and limitations in canonical swarm scenarios.
Analysis of Emergent Group Dynamics: Connects LLM swarm behavior to established collective intelligence concepts.
Open-Source Toolkit:
- Customizable and scalable physical system.
- Environments, standardized prompts, and evaluation scripts.
- Comprehensive experimental datasets.
- Designed for reproducible research into LLM-based MAS coordination.

🚀 Getting Started

🛠️ 1. Setup SwarmBench Environment

Ensure you have Conda installed. Then, create the environment using the provided file:

conda env create -f environment.yaml
conda activate swarmbench

🧪 2. Run SwarmBench Evaluation

Execute the eval.py script to run the benchmark evaluations:

from swarmbench import SwarmFramework

if __name__ == '__main__':
    name = 1
    for task in ('Transport', 'Pursuit'): #####  {'Transport', 'Pursuit', 'Synchronization', 'Foraging', 'Flocking'}
        for model in ('gpt-4o-mini', 'llama-3.1-70b'): ##### Models
            for seed in (27, 42):
                SwarmFramework.submit(
                    f'exp_{name}',
                    SwarmFramework.model_config(model, 'YOUR_API_KEY', 'YOUR_API_BASE'), ##### API
                    task,
                    log_dir='YOUR_LOG_DIR', ##### Logging
                    num_agents=10,
                    max_round=100,
                    width=10,
                    height=10,
                    seed=seed,
                    view_size=5
                )
                name += 1

    SwarmFramework.run_all(max_parallel=4)

This will run the LLM agents through the defined tasks and log the results.

🤗 3. Or Download SwarmBench Dataset Directly

🎬 4. Export Replay Videos

After running evaluations, you can generate replay videos of the simulations:

This script will process the log files from your experiments (by default in ./experiment_outputs/) and generate MP4 videos.

Example Replay: