catid (@MrCatid) on X

X (formerly Twitter) ·

2 min read Original article ↗

Post

Post

catid on X: "## Can open-source LLMs detect bugs in C++ code? No: ``` LLaMa 65B (4-bit GPTQ) model: 1 false alarms in 15 good examples.  Detects 0 of 13 bugs. Baize 30B (8-bit) model: 0 false alarms in 15 good examples.  Detects 1 of 13 bugs. Galpaca 30B (8-bit) model: 0 false alarms in 15"

  • user avatar

    ## Can open-source LLMs detect bugs in C++ code? No: ``` LLaMa 65B (4-bit GPTQ) model: 1 false alarms in 15 good examples.  Detects 0 of 13 bugs. Baize 30B (8-bit) model: 0 false alarms in 15 good examples.  Detects 1 of 13 bugs. Galpaca 30B (8-bit) model: 0 false alarms in 15 good examples.  Detects 1 of 13 bugs. Koala 13B (8-bit) model: 0 false alarms in 15 good examples.  Detects 0 of 13 bugs. Vicuna 13B (8-bit) model: 2 false alarms in 15 good examples.  Detects 1 of 13 bugs. Vicuna 7B (FP16) model: 1 false alarms in 15 good examples.  Detects 0 of 13 bugs. GPT 3.5: 0 false alarms in 15 good examples.  Detects 7 of 13 bugs. GPT 4: 0 false alarms in 15 good examples.  Detects 13 of 13 bugs. ```

  • user avatar

    Tested with Cody, the AI assistant from

    @sourcegraph

    (based in Claude from

    @AnthropicAI

    ): 0 false alarms Detects 10 of 13 bugs. Failed examples: - count_vowels - remove_duplicates - largest_sum_subarray

  • user avatar

    How do you handle/overcome the very large context required? Are these bugs relatively simple and can be “figured” out on one page of code?