StressingLLMs - Complexity Benchmark - combined campaign results

Rounds completed87

Passed rounds35

Failed rounds52

Pass rate40.2%

Attempts per fixture1

Provider token work154.10Mcumulative processed tokens

About this benchmark

This project evaluates local language models running on a single NVIDIA DGX Spark. It tests how well they can reverse-engineer generated C binaries and recover a plaintext message protected by deterministic XOR-based decryption. Each fixture uses a configurable number of transformation rounds to increase the analysis challenge.

For every attempt, the model inspects the binary, submits a Python decryptor, and the harness executes that code in a pinned no-network sandbox. The attempt passes when an executed Python block prints the expected plaintext. “Results by round” shows the pass/fail outcome for each model at each tested complexity level.

Example: two-round fixture

This is the complete source from fixtures/src/fx_r0002_sl0016_sp0000.c. It includes the helper, generated round functions, state chaining, and final byte-wise XOR loop.

#include <stdint.h>
#include <stdio.h>
#include <stddef.h>

static uint32_t xorshift32(uint32_t x) {
    x ^= x << 13;
    x ^= x >> 17;
    x ^= x << 5;
    return x;
}

typedef struct TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens {
    uint64_t a;
    uint64_t b;
    uint64_t c;
} TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens;

__attribute__((used, noinline))
uint32_t TokenizerBench___R0(TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens *p) {
    uint32_t m1 = xorshift32(0x9336956du ^ 0x31bbf978u ^ (uint32_t)p->a);
    uint32_t m2 = xorshift32(0xcd6f55fcu ^ (uint32_t)p->b);
    p->a ^= ((uint64_t)m1 << 32) | (uint64_t)m2;
    p->b += (uint64_t)(0x9336956du ^ m2);
    p->b = (p->b << 10) | (p->b >> 54);
    p->c = (p->c + p->a) ^ (uint64_t)(0x31bbf978u ^ 0xcd6f55fcu);
    uint64_t r = p->a ^ p->b ^ p->c ^ (uint64_t)0x9336956du ^ (uint64_t)0x31bbf978u ^ (uint64_t)0xcd6f55fcu;
    return (uint32_t)(r ^ (r >> 32));
}

__attribute__((used, noinline))
uint32_t TokenizerBench___R1(TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens *p) {
    uint32_t m = xorshift32(0x366856bbu ^ (uint32_t)p->a);
    p->a ^= ((uint64_t)0x366856bbu << 32) | (uint64_t)m;
    p->b += p->a ^ (p->c + (uint64_t)0x72fcd409u);
    p->c = ((p->c ^ (uint64_t)0x3afd4cabu) << 24)
         | ((p->c ^ (uint64_t)0x3afd4cabu) >> 40);
    uint64_t r = p->a ^ p->b ^ p->c ^ (uint64_t)0x366856bbu ^ (uint64_t)0x72fcd409u ^ (uint64_t)0x3afd4cabu;
    return (uint32_t)(r ^ (r >> 32));
}

__attribute__((used, noinline))
uint32_t derive_state(uint32_t seed) {
    TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens x = {
        seed, seed ^ 0x12345678ULL, seed + 0x9ULL
    };
    uint32_t s = seed;
    s ^= TokenizerBench___R0(&x);
    s ^= TokenizerBench___R1(&x);
    s = xorshift32(s);
    return s;
}

int main(void) {
    uint8_t encrypted[] = { 0xbc, 0xbb, 0x47, 0xdd, 0xb7, 0xbc, 0x88, 0x9f,
        0xba, 0x15, 0xdd, 0x2e, 0x00 };
    uint32_t s = derive_state(0xdeadbeef);
    for (size_t i = 0; i < sizeof(encrypted) - 1; i++) {
        s = xorshift32(s + 0xA5A5A5A5u);
        encrypted[i] ^= (uint8_t)(s & 0xffu);
    }
    puts((const char *)encrypted);
    return 0;
}

How an attempt works

1. The benchmark generates and compiles a fixture, then imports the matching binary into Ghidra for analysis.
2. The LLM receives the reverse-engineering task and accesses the binary through the Ghidra MCP server, which exposes analysis operations such as symbols, bytes, disassembly, and decompilation.
3. The LLM uses those MCP/Ghidra results to infer the state initialization, generated round functions, constants, and final XOR keystream.
4. The LLM submits a Python decryptor. The grader extracts every Python, py, or unlabeled fenced block that follows the submission format.
5. Each extracted block runs in the pinned no-network Python sandbox. The attempt passes when an executed block prints the expected plaintext; otherwise the report records the most specific failure category available.

This test evaluates a model’s ability to:

• Navigate a compiled binary using Ghidra and MCP.
• Recover generated state transformations and constants.
• Reason across multiple rounds of obfuscated code.
• Produce an executable Python decryptor.
• Persist through increasing reverse-engineering complexity.
• Communicate a usable solution in the expected format.

The main limitations are:

• Fixtures are synthetic and generated from one family of transformations.
• Results are hardware- and timeout-dependent because everything ran on one DGX Spark.
• The sample size per model is uneven.
• Adaptive search means pass rates are not always directly comparable.
• The benchmark measures the full model-plus-tool workflow, not just model reasoning in isolation.

Results by round

Each cell is one model attempt on that exact round count. A pass means an executed Python block printed the expected plaintext.

Interactive: Hover over or select a cell to see its failure reason and runtime.

SearchFailureTrackResult

Controlled model track

These results compare local models under the benchmark’s controlled attempt settings, showing how far each model progressed across the tested challenge rounds.

Validation profile

This profile summarizes the checks applied to completed attempts and the proportion that produced a verified decryption result.

Latest completed rounds

This table lists the most recently completed attempts, including their track, provider, challenge level, result, and processing time.

Run summary

This campaign verified the expected plaintext in 35 of 87 completed attempts (40.2%).

deepseek-v4-flash-dspark showed the strongest progression, verifying 81.8% of its 11 tested rounds and reaching round 9.

Other consistent performers were qwen38-27b-nvfp4-dspark, ollama-gemma4-31b-it-qat, ollama-gemma4-26b, ollama-gemma4-31b, ollama-qwen3-5-27b, vllm-muse-glimmer, each verifying at least 60% of tested rounds. Together, these results show clear separation as the generated round structure becomes more demanding.