Respecting maintainer time should be in security policies

4 min read Original article ↗

Generative AI tools becoming more common means that vulnerability reports these days are loooong. If you're an open source maintainer, you unfortunately know what I'm talking about. Markdown-formatted, more than five headings, similar in length to a blog post, and characterized as a vulnerability worthy of its own domain name.

This makes triaging vulnerabilities by often under-resourced maintainer more difficult, time-consuming, and stressful. Whether a report is a genuine vulnerability or not, it now requires more time from maintainers to make a determination than is necessary. I've heard from multiple maintainers how specifically report length weighs negatively on maintainer time, whether these are “slop vulnerability reports” or just overly-thorough reporters.

David Lord, the maintainer of Flask and Pallets, captures this problem concisely, that the best security reports respect maintainer time:

Post by @davidism@mas.to

View on Mastodon

Here's my proposal: require security reports respect maintainer time in your security policy. This especially applies to “initial” security reports, where the reporter is most likely to send disproportionately more information than is needed to make an initial determination.

The best part about this framing is you don't have to mention the elephant in the room. Here's a few example security policy requirements to include:

  • Initial reports must be six or fewer sentences describing the issue and optionally a short proof-of-concept script. Follow-ups may be longer if the initial report is deemed a vulnerability.
  • Initial reports must not contain a severity (such as: "critical") or CVSS score. This calculation will be completed after the report is accepted by maintainers.
  • Reports must not make a determination whether a behavior of the software represents a vulnerability.

Notice I didn't even have to mention LLMs or generative AI, so there’s no ambiguity about whether a given report follows the policy or not. While you have reporters reading your security policy, you might also add suggestions that also benefit maintainers remediating vulnerabilities faster:

  • If applicable, submitting a patch or an expected behavior along with a proof-of-concept makes remediating a vulnerability easier.
  • We appreciate vulnerability reporters that are willing to review patches prior to publication.
  • If you would like to be credited as a reporter, please mention that in your report.

After this is added to a project security policy (preferably under its own linkable heading) then any security report that doesn't respect maintainer time can be punted back to the reporter with a canned response:

Your report doesn't meet our security policy: https://... Please amend your report to meet our policy so we may efficiently make a determination and remediation. Thank you!

Now your expectations have been made clear to reporters and valuable maintainer time is saved. From here the security report can evolve more like a dialogue which requires maintainer time and energy proportionate to the value that the report represents to the project.

I understand this runs counter to how many vulnerability teams work today. Many reporters opt to provide as much context and detail up-front in a single report, likely to reduce back-and-forth or push-back from projects. If teams reporting vulnerabilities to open source projects want to be the most effective, they should meet the pace and style that best suites the project they are reporting to.

Please note that many vulnerability reporters are acting in good faith and aren't trying to burden maintainers. If you believe your peer is acting in good faith, maybe give them a pass if the report doesn't strictly meet the requirements and save the canned response for the bigger offenders.

Have any thoughts about this topic? Have you seen this in any open source security policies already? Let me know! Thanks to OSTIF founder Derek Zimmer for reviewing this blog post prior to publication.

Wow, you made it to the end!