GitHub - islandbytesio/commit_comprehension_gate

Comprehension Gate

Developers are merging AI-generated code they've never read. Comprehension Gate blocks the merge until they can prove they understand what they're shipping.

A GitHub Action workflow that uses Claude to generate multiple-choice questions whenever a developer opens or updates a pull request. The PR author must answer all 3 questions correctly before the branch can merge.

No database, no external storage. Questions are embedded directly in the PR comment.

How it works

A PR is opened (or updated) targeting main
The workflow fetches the diff and sends it to Claude, which generates 3 multiple-choice questions about the actual logic and design decisions in the changes
The questions are posted as a PR comment and the commit status is set to pending
The PR author replies with their answers in the comment thread
The answer-checker workflow runs instantly (no API call) and sets the commit status to success or failure

The PR author can retry as many times as needed. Pushing new commits regenerates fresh questions.

Demo

[Video coming soon]

Setup

1. Copy the files

Copy these into your repository:

scripts/comprehension_gate.py
.github/workflows/comprehension-gate-pr.yml
.github/workflows/comprehension-check.yml

2. Add the API key secret

In your repository: Settings → Secrets and variables → Actions → New repository secret

Name	Value
`ANTHROPIC_API_KEY`	Your Anthropic API key

3. (Optional) Change which branch is gated

By default, PRs targeting main are gated. To change this, edit comprehension-gate-pr.yml:

on:
  pull_request:
    branches:
      - main   # ← change this

Blocking merges (required status check)

Without this step the gate is informational only — the PR can still merge while the status is pending or failed.

To enforce the gate:

Go to Settings → Branches
Click Add branch ruleset (or edit an existing protection rule for main)
Under Require status checks to pass, add:

The name must match exactly — it is case-sensitive.
Enable Require branches to be up to date before merging (recommended)
Save the ruleset

Once configured, GitHub will block the merge button until the commit status for Comprehension Gate is success.

Developer experience

When a PR is opened, the bot posts a comment like:

Q1. What does this change do when the input is empty?

A) Returns an empty list

B) Raises a ValueError

C) Returns None

D) Falls through to the default handler

The author replies with their answers:

The bot then immediately replies with a pass/fail breakdown and updates the commit status.

Skipping the gate

Draft PRs are skipped automatically. If you need to bypass the gate for a specific PR, a maintainer can manually set the commit status to success via the GitHub API or CLI:

gh api repos/{owner}/{repo}/statuses/{sha} \
  -f state=success \
  -f context="Comprehension Gate" \
  -f description="Manually approved"

API cost

Claude is called once per PR open or push to generate questions. Answer checking makes no API call.

The gate uses claude-opus-4-6. Diffs are capped at 12,000 characters before being sent.

Diff size	Approx. input tokens	Estimated cost
Small (< 2 KB)	~800	~$0.01
Medium (2–6 KB)	~2,000	~$0.04
Large (6–12 KB, truncated)	~3,500	~$0.07

Output (3 questions + choices) adds roughly ~~500 tokens (~~$0.04) regardless of diff size.

Typical cost: $0.05–$0.10 per PR. At current Anthropic pricing ($15/M input tokens, $75/M output tokens for Opus).

Requirements

Python 3.12+ (provided by the Actions runner)
anthropic Python package (installed automatically by the workflow)
An Anthropic API key

License

MIT License — see LICENSE file for details. Patent Pending — IslandBytes LLC

Contributing

Issues and PRs welcome. See CONTRIBUTING.md for guidelines.