Settings

Theme

Cloudflare/Security-Audit-Skill

github.com

181 points by donk8r · 39 comments

Reader

9 threads
gbrindisi

Shameless plug: in case someone finds this requiring too many tokens, we shared the recipe on how we built our own in house audit skill so that it can easily be replicated and tuned to different environments https://www.synthesia.io/post/automating-code-security-revie...

prodigycorp

Hi Cloudflare people, if you are reading this. Please clean up your Cloudflare. Skills. There are way too many skills for the platform. You should consolidate all of your skills into a single skill and route everything thru that skill. The way it is right now pollutes our context window.

https://github.com/cloudflare/skills/tree/main/skills

drchaim

I threw 1M tokens for nothing in a medium codebase.

qsbuilder

Dumping 14 full schemas into the prompt is just lazy design. You burn tokens, spike latency for no reason

wslh

Tip for security professionals using LLMs: audit skills that explicitly frame the task as security research sometimes trigger refusals from the top OpenAI and Anthropic models because they guard against misuse. What works for me: separate skills for bug classes (and bugs in general) without the security framing, plus another skill that combines their findings to spot security bugs.

  • viraptor

    If you're a security professional, go through their validation. You won't get the security refusals anymore. Well... you'll still get the occasional downgrade from Fable, but not the "oh no, I can't do exploits for you" breaks.

    • xur17

      Except their validation doesn't seem to work. I've gone through both (both personally and for my company), and.. no response for weeks.

hyperionultra

Uf, how much tokens?

  • jesse_dot_id

    At least 150k on my relatively small FastAPI project, but hit my session limit. Continuing in a few hours.

    • chrisweekly

      Oof. YAGNI. 150k tokens is where you start hitting the "dumb zone" (model attention issues and inconsistent adherence to instructions).

acedTrex

Incredible, a post and repo dedicated to a markdown file, the downfall of this field has been swift.

  • decidu0us9034

    but they're very huge markdown files. look how much junk they're polluting the conext window with.

  • vntok

    Did you open the repo? There's a subdirectory with two dozens of files, around 300Kb of text.

    Storing/visualizing small text changes over time as revisions is exactly what Git is excellent at, how else would you keep track of updates to the prompts?

    • acedTrex

      > how else would you keep track of updates to the prompts

      I dont? because prompts are not a thing that are ever needed to be tracked lol.

tonymet

What’s the difference between a skill and a prompt? Separate files? Aren’t tokens, tokens?

9el

Any clues why "an OS-enforced sandbox" is in requirements?

  • donk8rOP

    Runs target builds, tests, fuzzers. No sandbox: workflow won't execute them. Lead stays needs_validation.

  • nicce

    Probably to save their skin if agent starts to do some unexpected things and bringing havoc. But I doubt that OpenAI models with normal subscription, for example, wont even work with this skill.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection