Cliphouse ResearchOctober 2026 / v1.1
Motion Graphics
Benchmark.
Find the right AI model for motion graphics
without testing every model yourself.
01 / Comparison
One brief. Every model.
14 models · 2 independent runs
12 seconds · 1080p · 30 fps · Silent
Product launch - full prompt
System instructions
You write a complete Remotion 4 composition. Return one JSON object with exactly one string property: "source". The source must be a complete TSX module exporting a named React component called BenchmarkVideo. Only import from "react", "remotion", and "./contract". Do not register a composition or change the project configuration. The wrapper supplies 1920 x 1080, 30 fps, and 360 frames. Use useCurrentFrame(), interpolate(), spring(), and Sequence for frame-based motion. Every frame must be deterministic. No Math.random(), dates, timers, network requests, audio, or external assets. The wrapper loads Archivo locally. Use the font family "Archivo" for all text. Import palette from "./contract": background #101114, foreground #F5F5F2, accent #B8F36B, secondary #78B9ED, muted #B1B4BC. Use these colors. Alpha variants are permitted. Keep text inside an 80-pixel safe margin on every side. No existing video examples are provided. Make all design decisions yourself. Do not include model names, provider names, explanations, markdown fences, or other files.
Brief
Create a silent, 12-second product-launch video for "Relay", a fictional team-planning app. Use Remotion, 1920 x 1080, 30 fps. From 0-3 seconds, introduce "Relay" and "Make room for focused work". From 3-9 seconds, show "Plan together", "Protect focus time", and "See progress" through an animated product interface. From 9-12 seconds, show "Start your next week with Relay". Keep the final message fully readable for at least two seconds. Use the supplied fonts and palette. Choose the layout, visual style, and transitions. Include no additional claims, external assets, or audio. Return a complete composition that follows the supplied component contract.
Product launch, run 1. 14 completed entries.
MiniMaxFree
MiniMax M3.1 Flash
New / Added October 19
- API cost
- $0 (free)
- Time to video
- ~324.5s
Run details
- Model ID
minimax/minimax-m3.1-flash:free- Reasoning effort
- high
- Model response time
- 249.7s across all attempts
- Render time
- 74.8s across all attempts
- OpenRouter generation
- 66.6s, successful attempt only
- Generation ID
gen-1792416332-aBumlEsYl8Pvrgw35Z0s- Status
- Passed after repair
- Provider
- minimax
- Generated
- 2026-10-19T13:22:22.278Z
DeepSeekOpen weights
DeepSeek V4.1 Flash 1017
New / Added October 17
Recovered / R1 · 1 additional attempt
- API cost
- $0.025
- Time to video
- ~297.3s
Run details
- Model ID
deepseek/deepseek-v4.1-flash-1017- Reasoning effort
- high
- Model response time
- 256.3s across all attempts
- Render time
- 41.0s across all attempts
- OpenRouter generation
- 22.9s, successful attempt only
- Generation ID
gen-1792260839-E2w5GrF2GYh7TCcsdbE9- Status
- Recovered in automated repair pass R1
- Original outcome
- Failed after 1 attempt. Invalid submission: Completion limit reached.
- Original API cost
- $0.019
- Additional repair cost
- $0.0060
- Repair note
- The original response reached the 32,000-token limit before the code was complete. The repair returned the complete module.
- Total attempts
- 2
- Provider
- deepseek
- Generated
- 2026-10-17T18:13:59.935Z
MetaOpen weights
Muse Spark Open
New / Added October 15
- API cost
- $0.081
- Time to video
- ~261.4s
Run details
- Model ID
meta/muse-spark-open- Reasoning effort
- high
- Model response time
- 177.9s across all attempts
- Render time
- 83.5s across all attempts
- OpenRouter generation
- 177.9s, successful attempt only
- Generation ID
gen-1792088119-3DyW0QnKb4ETjmHjrktq- Status
- Passed on first attempt
- Provider
- fireworks
- Generated
- 2026-10-15T18:15:19.879Z
GoogleMid-range
Gemini 4 Argon
New / Added October 13
- API cost
- $0.182
- Time to video
- ~175.4s
Run details
- Model ID
google/gemini-4-argon- Reasoning effort
- high
- Model response time
- 136.4s across all attempts
- Render time
- 39.1s across all attempts
- OpenRouter generation
- 136.3s, successful attempt only
- Generation ID
gen-1791924025-GhCRUz0mZHVKdsvIst4X- Status
- Passed on first attempt
- Provider
- google-vertex/global
- Generated
- 2026-10-13T20:40:25.820Z
AnthropicPremium
Claude Opus 5.5
- API cost
- $0.374
- Time to video
- ~258.6s
Run details
- Model ID
anthropic/claude-opus-5.5- Reasoning effort
- high
- Model response time
- 211.0s across all attempts
- Render time
- 47.6s across all attempts
- OpenRouter generation
- 211.0s, successful attempt only
- Generation ID
gen-1791473470-56iTa35pdSfCu38guL3O- Status
- Passed on first attempt
- Provider
- amazon-bedrock
- Generated
- 2026-10-08T15:31:10.508Z
AnthropicMid-range
Claude Sonnet 5.5
- API cost
- $0.135
- Time to video
- ~292.4s
Run details
- Model ID
anthropic/claude-sonnet-5.5- Reasoning effort
- high
- Model response time
- 80.3s across all attempts
- Render time
- 212.1s across all attempts
- OpenRouter generation
- 80.2s, successful attempt only
- Generation ID
gen-1791474384-b1YYEhNEJVtkCP49xr4o- Status
- Passed on first attempt
- Provider
- google-vertex/global
- Generated
- 2026-10-08T15:46:24.224Z
AnthropicBudget
Claude Haiku 5.5
- API cost
- $0.011
- Time to video
- ~125.0s
Run details
- Model ID
anthropic/claude-haiku-5.5- Reasoning effort
- high
- Model response time
- 90.3s across all attempts
- Render time
- 34.7s across all attempts
- OpenRouter generation
- 90.3s, successful attempt only
- Generation ID
gen-1791473327-4RQArKnAgnoFNuwUwbYh- Status
- Passed on first attempt
- Provider
- google-vertex/global
- Generated
- 2026-10-08T15:28:47.009Z
OpenAIMid-range
GPT-6.1 Sol
- API cost
- $0.116
- Time to video
- ~228.9s
Run details
- Model ID
openai/gpt-6.1-sol- Reasoning effort
- high
- Model response time
- 195.8s across all attempts
- Render time
- 33.1s across all attempts
- OpenRouter generation
- 195.8s, successful attempt only
- Generation ID
gen-1791475260-TVsJveMp49wbEOd6AnOc- Status
- Passed on first attempt
- Provider
- openai
- Generated
- 2026-10-08T16:01:00.939Z
OpenAIBudget
GPT-6 Luna
- API cost
- $0.0066
- Time to video
- ~164.3s
Run details
- Model ID
openai/gpt-6-luna- Reasoning effort
- high
- Model response time
- 130.0s across all attempts
- Render time
- 34.4s across all attempts
- OpenRouter generation
- 129.7s, successful attempt only
- Generation ID
gen-1791475475-yp2El122stCMuqOk5ZKr- Status
- Passed on first attempt
- Provider
- openai
- Generated
- 2026-10-08T16:04:35.984Z
GoogleBudget
Gemini 3.8 Flash
- API cost
- $0.087
- Time to video
- ~197.8s
Run details
- Model ID
google/gemini-3.8-flash- Reasoning effort
- high
- Model response time
- 164.0s across all attempts
- Render time
- 33.8s across all attempts
- OpenRouter generation
- 163.8s, successful attempt only
- Generation ID
gen-1791475707-lGBxIH9syDuIvJU8sTgx- Status
- Passed on first attempt
- Provider
- google-ai-studio
- Generated
- 2026-10-08T16:08:27.843Z
DeepSeekBudget
DeepSeek V4.1 Flash
Recovered / R1 · 1 additional attempt
- API cost
- $0.0068
- Time to video
- ~449.4s
Run details
- Model ID
deepseek/deepseek-v4.1-flash- Reasoning effort
- high
- Model response time
- 416.8s across all attempts
- Render time
- 32.7s across all attempts
- OpenRouter generation
- 51.9s, successful attempt only
- Generation ID
gen-1791484067-mR90dWIhvqONB97nu6X6- Status
- Recovered in automated repair pass R1
- Original outcome
- Failed after 1 attempt. Invalid submission: Completion limit reached.
- Original API cost
- $0.0057
- Additional repair cost
- $0.0010
- Total attempts
- 2
- Provider
- decart/fp4
- Generated
- 2026-10-08T18:27:47.516Z
Z.aiBudget
GLM 5.3 FlashX
Recovered / R1 · 1 additional attempt
- API cost
- $0.030
- Time to video
- ~165.3s
Run details
- Model ID
z-ai/glm-5.3-flashx- Reasoning effort
- high
- Model response time
- 132.6s across all attempts
- Render time
- 32.7s across all attempts
- OpenRouter generation
- 22.4s, successful attempt only
- Generation ID
gen-1791484229-6jyLrryIu1BtuiCKMLqq- Status
- Recovered in automated repair pass R1
- Original outcome
- Failed after 1 attempt. Invalid submission: Expected ',' or '}' after property value in JSON at position 23223 (line 2 column 23222)
- Original API cost
- $0.021
- Additional repair cost
- $0.0091
- Total attempts
- 2
- Provider
- z-ai/fp8
- Generated
- 2026-10-08T18:30:29.372Z
QwenOpen weights
Qwen3.8 2.4T A95B
- API cost
- $0.030
- Time to video
- ~133.6s
Run details
- Model ID
qwen/qwen3.8-2.4t-a95b- Reasoning effort
- medium
- Model response time
- 101.4s across all attempts
- Render time
- 32.2s across all attempts
- OpenRouter generation
- 101.3s, successful attempt only
- Generation ID
gen-1791476573-paPXDZ0Ahc1eIdUsTXWS- Status
- Passed on first attempt
- Provider
- novita
- Generated
- 2026-10-08T16:22:53.346Z
Moonshot AIOpen weights
Kimi K3
- API cost
- $0.356
- Time to video
- ~555.6s
Run details
- Model ID
moonshotai/kimi-k3- Reasoning effort
- high
- Model response time
- 519.2s across all attempts
- Render time
- 36.4s across all attempts
- OpenRouter generation
- 518.9s, successful attempt only
- Generation ID
gen-1791476771-hzcy498AYiscSSPjZhR7- Status
- Passed on first attempt
- Provider
- decart/mxfp4
- Generated
- 2026-10-08T16:26:11.447Z
A curated selection across price tiers, not a popularity ranking. Empty entries are untested, not failed.
Time to video is an estimate from prompt submission until the MP4 was ready. It adds every model response and render attempt, including repairs. It excludes waiting in the render queue and small file-handling overhead. OpenRouter timing records
Recovered entries show total API cost, including original attempts. The additional repair cost is itemized in run details. Recovery means the code compiled and rendered. It is not first-attempt success.
Late additions: Gemini 4 Argon (October 13), Muse Spark Open (October 15), DeepSeek V4.1 Flash 1017 (October 17), MiniMax M3.1 Flash (October 19) were added after the original ten models. They used the same protocol, briefs, token limit, renderer, and repair rule. Argon, Muse, and MiniMax M3.1 Flash rendered every run and needed no R1 recovery. MiniMax ran on OpenRouter's free endpoint, so its API cost is $0. DeepSeek V4.1 Flash 1017's five failed runs received the same R1 recovery pass as the original models, and are marked as recovered.
02 / Methodology
Same starting point.
No hand-picked winners.
Protocol v1.1
This tests language models writing motion-graphics code, not native text-to-video models. Every request goes through OpenRouter. The generated code is rendered with the same Remotion environment.
What stays the same
Three fixed briefs, two independent runs, one component contract, the same local fonts and palette, and a 32,000-token completion limit that includes reasoning. No browsing, external assets, or existing Cliphouse compositions.
What gets another attempt
Only a failed compile or render gets one repair request, with the error log. Successful videos receive no creative revision. Original attempts and repair costs are retained. Transport errors are recorded separately.
Automated recovery / R1
Only original failures receive up to three additional automated repair requests. Keep the same model, pinned provider, reasoning effort, brief, fonts, palette, and renderer. Return complete TSX directly; a single code fence or valid source JSON wrapper is accepted without editing the code. Supply the previous response and complete compiler/render diagnostics. Stop at the first successful render. No human code edits or creative feedback. Preserve original outcomes and report additional attempts and costs separately. Recovery uses a 64,000-token limit and direct TSX output instead of the original JSON-only format. These are additional repair results, not replacement first attempts.
Repair results · Repair protocol · Original results
What we disclose
The exact model ID, provider, request, reasoning setting, source code, estimated time to video, OpenRouter generation time, and API cost. Time to video adds local response and render times for all attempts. OpenRouter generation time is shown separately for the successful request. Providers are pinned and fallback is disabled. Up to four API requests overlap; renders remain sequential. Reasoning labels are not equivalent compute budgets across models.
14models
3fixed briefs
2runs per brief
Both runs are published, including failures. Two runs are exploratory evidence, not a statistically definitive ranking.
Protocol revision: A Haiku setup pilot used 13,309 reasoning tokens and was truncated at the original 16,000-token limit. Protocol 1.1 uses 32,000 tokens for all 60 comparable runs. The pilot cost $0.008 and remains in the spending record, outside the comparison.
Repair-feedback limitation: In this pilot, compile-failure repairs received a stack trace without the detailed TypeScript diagnostics. Treat repair outcomes as provisional. First-attempt outcomes are unaffected.
Addition A1: Late addition to the October 2026 edition. It uses the protocol 1.1 contract, briefs, palette, fonts, token limit, renderer image, single repair rule, and provider selection rule. The original 60 runs are unchanged. Automated recovery R1 does not apply to this addition. Addition results
Addition A2: Late addition to the October 2026 edition. It uses the protocol 1.1 contract, briefs, palette, fonts, token limit, renderer image, and single repair rule. The provider is pinned to Fireworks (unquantized listing) instead of the cheapest-price rule, because four providers list the same price. The original 60 runs and addition A1 are unchanged. Automated recovery R1 does not apply to this addition. Addition results
Addition A3: Late addition to the October 2026 edition. It uses the protocol 1.1 contract, briefs, palette, fonts, token limit, renderer image, and single repair rule. The provider is pinned to DeepSeek's first-party endpoint instead of the cheapest-price rule, so the snapshot is served by its publisher. This snapshot is separate from the original DeepSeek V4.1 Flash entry, which is unchanged, as are the original 60 runs and additions A1 and A2. Automated recovery R1 does not apply to this addition. Addition results
Addition A4: Late addition to the October 2026 edition. It uses the protocol 1.1 contract, briefs, palette, fonts, token limit, renderer image, and single repair rule. It runs on OpenRouter's free endpoint (minimax/minimax-m3.1-flash:free), pinned to MiniMax, the only provider of that endpoint, so every request is billed at $0. The original 60 runs and additions A1, A2, and A3 are unchanged. Automated recovery R1 does not apply to this addition. Addition results
03 / Editions
A record, not a moving target.
New model versions join the live comparison. Monthly reports preserve the tested versions, protocol, and results. Changes to the test receive a new protocol version.