Add TY25 Kimi K3 support via OpenRouter by michaelrbock · Pull Request #93 · column-tax/tax-calc-bench

Summary

• Adds TY25 support for moonshotai/kimi-k3 through OpenRouter using OPENROUTER_API_KEY.
• Includes Kimi K3 in default TY25 runs and restricts it to ultrathink, mapped to its required reasoning_effort="max".
• Sends the original PDFs as raw base64 file blocks for OpenRouter’s default PDF processing, without local text extraction or a parser override.
• Preserves slash-containing model IDs and calls LiteLLM with the full openrouter/moonshotai/kimi-k3 model name.
• Keeps Kimi out of TY25 web-search runs and rejects unsupported lower thinking levels.
• Adds saved model outputs and evaluation reports for all 50 TY25 cases.
• Updates the README leaderboard, detailed results, documentation, and regenerated TY25 charts.

Results

The saved artifacts contain one model output and one evaluation report for every TY25 case.

Implementation notes

• Uses reasoning_effort="max", max_tokens=131072, streaming, and the existing four-hour timeout and finish-reason checks.
• Omits sampling arguments, parser plugins, and web-search configuration.
• Retains LiteLLM 1.89.3. A regression test verifies that its generic OpenRouter adapter preserves the nested model slug, PDF file block, and explicitly allowed reasoning parameter without requiring static Kimi model metadata.
• Documents OpenRouter PDF processing, possible OCR charges, and the current upstream capacity warning.

Validation

• uv run pytest — 142 passed
• uv run ruff check tax_calc_bench tests
• uv run scripts/update_charts.py
• git diff --check
• Verified 50 model outputs and 50 evaluation reports
• Visually verified the regenerated TY25 overall and detailed charts