GitHub - loadmill/test-mcp: Automated testing tool for MCP servers and agents

📦 test-mcp

What's test-mcp? • Installation • Getting Started • Configuration & Test Format • How to Run • CLI Flags • Test Discovery • Interactive Mode • Programmatic API • Roadmap • Contributing • License

test-mcp is a headless MCP client for automated testing of MCP servers and agents.

If you’re building an MCP server or agent, test-mcp lets you run natural-language test scripts and assertions end-to-end, so you can validate behavior in a fast and repeatable way.

💡 What's `test-mcp`?

test-mcp gives you three core components:

• Configuration – define your MCP servers and LLM provider in a single JSON file.
• Test Files – write flows of natural-language prompts and assertions in YAML.
• Runner – run tests from the CLI, get clear pass/fail results.

Together, these let you automate and validate MCP server behavior with simple, repeatable tests.

Supported Transports & Providers:

• MCP Servers: STDIO (local) and HTTP (remote)
• LLM Providers: Anthropic Claude and OpenAI GPT models
• MCP Features: Tools (done), Resources/Prompts/Sampling (planned)

🏗️ Installation

# using npm
npm install -g @loadmill/test-mcp

# or with pnpm
pnpm add -g @loadmill/test-mcp

When running from source:

git clone https://github.com/loadmill/test-mcp
cd test-mcp
npm install
# For OpenAI (default example)
echo "OPENAI_API_KEY=your_api_key_here" > .env
# Or for Anthropic
echo "ANTHROPIC_API_KEY=your_api_key_here" > .env
npm run build

🚀 Getting Started

To try test-mcp quickly with the included examples:

# from source
node build/index.js

This will run a demonstration that shows both local STDIO and remote HTTP MCP servers working together. The test rolls a local dice server and queries a remote MCP server registry.

📑 Configuration & Test Format

1) Example config (mcp.config.json)

{
  "mcpClient": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "api_key": "${env:OPENAI_API_KEY}"
  },
  "mcpServers": {
    "loadmill": {
      "type": "stdio",
      "command": "npx",
      "args": ["@loadmill/mcp"],
      "env": {
        "LOADMILL_API_TOKEN": "${env:LOADMILL_API_TOKEN}"
      }
    }
  }
}

OpenAI is also supported - see configuration variations in the examples/ folder.

2) Example test (tests/bank-transaction.test.yaml)

description: "Maker Checker Bank - Transaction Creation and Rejection Flow"

steps:
  - prompt: "Login with username alice and password alice123 and transfer $100 to Bob"
  - prompt: "Login with username bob and password bob456, reject transaction from Alice"
  - assert: "Validate the transaction was created and rejected successfully"

▶️ How to run

By default, test-mcp looks for mcp.config.json in the project root and runs tests in the tests/ folder.

Globally installed:

From source:

Point to a specific config or tests directory:

test-mcp --config mcp.config.json --tests-dir ./tests

💻 CLI Flags

Options:
  -c, --config <file>   Path to config file (default: mcp.config.json)
  -t, --tests-dir <dir> Directory containing test files (default: tests)
  -i, --interactive     Run in interactive chat mode
      --trace           Enable detailed tracing output
  -h, --help            Show help

🔎 Test Discovery

All files ending in .test.yaml under the tests/ directory are executed. Recursive discovery and full glob patterns are planned for later.

💬 Interactive Mode

Run the client without tests and chat with your MCP servers:

🔧 Programmatic API

You can use test-mcp programmatically in your Node.js code:

import { TestMCPClient } from '@loadmill/test-mcp';

const client = new TestMCPClient({
  llm: {
    provider: 'openai',
    model: 'gpt-4o-mini',
    apiKey: process.env.OPENAI_API_KEY
  },
  servers: {
    myServer: {
      type: 'stdio',
      command: 'node',
      args: ['./server.js']
    }
  }
});

await client.connect();
const response = await client.prompt('Your question');
const assertion = await client.assert('Expected behavior');
await client.disconnect();

See examples/api-example.js for a complete example.

🛣️ Roadmap

• Headless MCP client with Anthropic support
• Support for stdio transport
• Support for MCP tools
• Evaluator for natural-language assertions
• OpenAI support
• Support for http transport
• Test parameterization with ${}
• CI-friendly reports
• Support for MCP resources
• Support for MCP prompts
• Support for MCP sampling

🤝 Contributing

Contributions, ideas, and bug reports are welcome! See CONTRIBUTING.md.

📄 License

Apache License 2.0 © Loadmill