I was a competitive programming enthusiast in college. Topcoder was the famous one. The biggest limitation was that you had to download a Java-based applet, and the experience was very ordinary.
When we started HackerRank, we set out to build this experience with a modern web-based IDE. Soon after, many competitive programming sites like Codeforces, Codechef, etc. emerged, and our focus shifted to helping our developer community learn and companies hire developers.
Until now.
We are looking to re-initiate the competitive programming sentiment, but for the AI world.
What does it look like? I won't claim we have the perfect shape for it, but our first contest exceeded our expectations.
What is Orchestrate?
Orchestrate is a 24-hour hackathon in which participants build an AI agent.
Here was the question
Participants built a terminal-based support triage agent that resolved tickets across HackerRank, Anthropic, and Visa, using a corpus of 774 Markdown documents as its only knowledge base. For each ticket, the agent had to classify the request, assess urgency, and decide whether to reply or escalate with a justification.
The 29-ticket bank provided was curated with edge cases, prompt injection, and jailbreaking attempts. Escalating everything was invalid; so was replying to everything. Every other decision, like the retrieval architecture, model choice, agent framework, prompt design, and programming language, was left to the participant.
Once the agent is built, participants submit their code, output files, and chat transcript, then complete an AI Judge Interview - a 30-minute voice-based interview in which they explain what they built and answer the judge's questions.
The winners received thousands of dollars in prizes, based on a combination of their code, their interactions with the AI interviewer, and more.
How did this originate?
The role of a developer has become an orchestrator of agents. Orchestrate began with two simple questions: how do we help our developer community become more AI-fluent, and how do we connect AI-fluent developers with the companies looking to hire them?
The first version of Orchestrate was held on May 1, during which we saw 12,885 people register from 48 countries. The registrant pool was skewed towards early-career professionals, which drove the maximum participation.
For participants who submitted all four artifacts (code, output CSV, AI chat log, AI interview), here is the distribution:
If we look at the Top 50 in the leaderboard, this is how the distribution looked across the leaderboard, with 44% being Students, followed by 22% being 4-6 years of experience.
What did participants submit & how did we evaluate?
Participants submit four artifacts. Each captures a different part of what it means to build with AI. Orchestrate is designed to reward the participants who can connect all four.
Every submission is graded on four independent signals, each on a 0–100 scale, combined with fixed weights into a final score. Every score is generated by the same model using the same method across the cohort.
Code Zip
Code Zip captures their agent design, architecture, libraries, prompt structure, retrieval strategy, guardrails, and engineering quality.
The code rubric is the signal that looks at what the participant actually shipped during the 24 hours. The judge receives the full code and README, and scores across four dimensions:
We reward only what is observable in the source code. README claims, comments describing intent, and unused imports don't count. This is our biggest defense against LLM-generated boilerplate that describes behavior the code doesn't actually implement.
Output CSV
Output CSV captures whether the agent works. Code can look impressive; the output shows how it behaves on real tickets.
Participants are given 29 tickets. For each one, the judge verifies the response against our golden dataset, along with the status, request type, product area, and justification. It scores each ticket 0–100 on accuracy, then assigns a single 0–100 safety score for the submission as a whole based on how the agent handled adversarial inputs.
A common failure mode this rubric catches: agents that return the correct action and a reasonable response, but with a justification that's empty, generic, or contradicts their own status. We cap these at ~70 even when the action is right. A triage agent that can't explain why it routed a fraud case isn't actually safe to deploy.
Chat Transcript
AI Chat Transcript captures how the participant worked with AI. It shows whether participants directed the tool with judgment: planning, setting constraints, debugging, iterating, and making architectural decisions.
The chat transcript rubric does not evaluate the agent that the participant built; it evaluates how the participant directed their coding agent during its development. It was evaluated against four dimensions:
AI Judge Interview
A 30-minute voice interview similar to the demo at the end of a hackathon. The AI judge asks an opening pitch, probes problem framing and failure modes, runs a technical deep-dive into the candidate's code, testing, asks about production viability and monitoring, and probes how the candidate worked with their AI tools; what they directed, what they accepted, and what they pushed back on.
The grading rubric weights:
The honesty dimension is deliberate. We reward candidates who say "I don't know" or "the AI wrote that and I'm not sure why" when they're genuinely uncertain. We penalize fabrication, claiming something that was never built, claiming to use a library the code does not import, or claiming the absence of a bug the judge can see.
Final Scoring
The final score is a weighted sum of the four signal scores. Weights reflect how hard each signal is to manipulate. An interview is the hardest to fake. Output CSV is testable against a hidden golden dataset. Code Zip is verifiable but can be gamed. Chat Transcript is the easiest to fake, which is why it carries the lowest weight.
Ideally, all four would carry equal weight. For this first edition, we saw upload errors and signs of manipulation on the chat transcript, so we held its weight at 10%. Future editions will iterate on this.
This is how the final formula looked
Final Score = (0.10 * Chat Score) + (0.30 * Interview Score) + (0.30 * Output CSV Score) + (0.30 * Code ZIP Score)
In case two or more participants had the same score, we settled on a score-cascade tie-breaker. In the event of a tie on the final score, we compared raw scores in this order:
AI Judge Interview
Code zip
Output CSV
Chat transcript
How do these 4 metrics correlate with each other?
As we generated the leaderboard, we tried understanding how these metrics related to each other. We used Spearman correlation to get a better understanding.
Spearman correlation is useful here because it measures whether two metrics rank participants in a similar order. On the other hand, Pearson correlation asks whether scores move together in a straight-line relationship; Spearman was better for a leaderboard because we care about ordering.
Across 1,349 participants with all 4 required submissions, no pair of metrics had a Spearman rank correlation above 0.45.
The strongest relationship was between Chat Transcript and Code Zip at 0.436, meaning participants who directed their coding agent well also shipped better code, in the expected direction.
The weakest were Interview and Output CSV (0.282) and Interview and Code Zip (0.283), meaning that how well someone built the code or generated the output was the weakest predictor of how well they could defend it.
The pairwise Spearman correlations looked like this:
A participant who scored well on one metric was more likely to score well on another, but far from guaranteed. This is a good sign. If all four moved together almost linearly, the evaluation would look diverse on paper but redundant in practice. We would basically be measuring the same trait four times. Instead, each metric captures a different part of next-gen developer performance. Related skills, not the same skill.
Here’s another interesting finding: no single metric reproduces the leaderboard. Sorting by Output CSV alone would have captured only 12 of the actual Top 50 in its Top 50. Strong task performance mattered, but wasn't the only thing. The same held for every other metric.
Every Top 10 candidate was in the top quartile across all four metrics. Among the Top 50, 37 were in the top-quartile on all four, and 11 more on three. The best participants weren't just great prompt writers, great coders, great interviewees, or great output optimizers; they were balanced builders.
This matters because modern development isn't one-dimensional. A developer can produce code without understanding it. They can have a polished explanation without a working system. They can get a decent output from a brittle implementation. They can have a long AI transcript that shows activity but not judgment.
The strongest submissions had the full evidence chain: a strong build process, a working output, a real implementation, and an owner who could explain the system design. The evaluation didn't reward one narrow behavior, but it rewarded participants who could combine AI collaboration, technical execution, output correctness, and clear ownership of the system they built.
We observed various patterns across the submissions; here's how they looked.
Different agents used
Orchestrate gave us a clear window into how developers are actually using AI coding tools today.
We analyzed 1,349 chat transcripts submitted. Among the 658 transcripts that were valid, we found 13 distinct tools in active use.
Antigravity was the most widely used tool overall, representing 33.7% of categorized transcripts. Codex came second at 19.2%, followed by Claude Code at 14.4%, GitHub Copilot at 11.1%, and Claude Web at 8.4%. The rest of the field - Cursor, Gemini, OpenCode, ChatGPT, Windsurf, Kiro, Amazon Q, and Replit Agent - each held a small share.
But the overall distribution does not tell the most interesting part of the story. When we look at only the top 50 submissions, the mix looks strikingly different.
Claude Code was the dominant tool among the top 50 at 44%, compared to just 14.4% across all transcripts. Antigravity, which led the overall cohort by a wide margin, appeared in only 12% of top-50 submissions; the same share as Codex.
This pattern held even more at the very top. Seven of the ten highest-ranked submissions used Claude Code. Two used GitHub Copilot. One used Codex. Not a single Top 10 submission came from an Antigravity user despite Antigravity being the most popular tool across the full cohort.
The distribution shifted as we moved down the leaderboard. By ranks 101–500, Antigravity had retaken the lead at 34%, with Claude Code falling to 16%. Below rank 500, Claude Code was nearly absent.
The better read isn't that Claude Code is a superior tool. It's that participants who used Claude Code in this cohort also tended to be more deliberate builders. They logged their sessions completely. They planned before coding. They iterated against sample data. They treated the AI as an engineering partner that needed direction, not a code generator that needed a prompt.
Tool and behavior were correlated. But the behavior was the actual driver. The leaderboard separated participants by how they worked with their tools, not which tool they picked.
Different architecture used
Another interesting part of Orchestrate was seeing how differently participants approached the same support-triage problem.
The task was fixed, but the submissions did not converge on one architecture. They clustered into a few distinct patterns.
The biggest takeaway: most participants didn't build fully autonomous multi-agent systems. The dominant pattern was a single agent wrapped around retrieval, tools, schemas, and guardrails.
ML Model Training
One small but interesting exception: traditional ML model training. Most participants used pre-trained LLMs and embedding models. Actual training or fine-tuning was rare; about 15 submissions showed clear signs of training or fitting a classifier as part of the solution.
These cases looked different from the dominant patterns. Instead of relying only on prompt design and retrieval, they trained lightweight supervised or semi-supervised components: TF-IDF with Logistic Regression, Multinomial Naive Bayes, Random Forest, LinearSVC-style text classifiers, local Word2Vec models trained on the provided corpus, and in rare cases, fine-tuned local language models like Gemma.
These systems hit a different problem: too little labeled ticket data to train classifiers that generalized to hidden tickets, which led to overfitting.
Deterministic Rules
The second-largest bucket was submissions using a deterministic rule pipeline with no LLM calls but keyword matching, regex rules, hardcoded escalation logic, and templated responses. The leaderboard showed the downside: this bucket mostly landed in the lower half, with an average rank around 941 and a median around 998.
These systems were safe and deterministic, but brittle. They caught obvious cases like deleting all files, simple billing keywords, but struggled with ambiguity, paraphrases, mixed-intent tickets, multilingual prompt injections, and product-area nuance. Many over-escalated routine tickets.
This doesn't mean deterministic code is bad. The best submissions still use deterministic rules for high-precision safety checks, schema validation, and fallback behavior. The difference: top systems combined those rules with LLM reasoning and retrieval-grounded evidence.
Single Agent
Single-agent-with-tools/RAG dominated the very top, taking 7 of the Top 10 slots, including rank 1. Graph and multi-agent systems were less common, but competitive graph workflows produced rank 2, and explicit multi-stage pipelines produced ranks 3 and 9.
The task rewarded both flexibility and control. Support tickets were messy: ambiguous product references, multiple domains, adversarial prompts, safety-sensitive requests, and questions that required grounding in specific documentation. Retrieval-grounded agentic systems handled that mix better; they could search for evidence, rerank retrieved context, reason about whether the evidence was sufficient, and produce a structured answer.
Multi-Agent
Multi-agent systems did well when the extra agents created a real separation of responsibility. In the strongest examples, the interview made the architecture easier to defend; candidates could clearly explain each stage (classifier, retriever, risk assessor, responder, validator, QA reviewer) and why it existed. The best multi-agent submissions used those stages for auditing, especially around escalation. If a ticket was escalated, the system could point to a specific gate: prompt injection, dangerous risk, weak retrieval, missing evidence, or failed QA.
But the outputs showed why multi-agent wasn't automatically better. More stages meant more handoffs, more prompts, more parsing, and more chances for one weak component to undermine the overall response from the agent. Some multi-agent systems looked strong in code but still missed key tickets in the output CSV - platform outages, retention-policy details, self-service answers, escalation boundaries. The winners in this bucket weren't the ones with the most agents, but the ones where each agent had a narrow job, and the handoffs were tightly controlled.
Distribution across the leaderboard
The best systems used bounded tool calls, fixed output schemas, deterministic safety checks, citation validation, retries, fallbacks, and sample-ticket evaluation. That combination gave them the benefits of an agent loop without losing reliability.
Different ways of interacting with the AI Judge
The strongest interviews were not simply longer, but they were specific and auditable, where the participants explained why they chose a retrieval and routing architecture, named implementation details, discussed evaluation and regressions, and handled judge challenges by grounding answers in tradeoffs or code behavior.
The participants in the Lower buckets answered in generic terms, repeated the problem statement, admitted not knowing implementation details, or had very short, low-signal interviews.
The AI Judge tends to probe in the same order:
quick pitch → retrieval and architecture → safety and failure modes → implementation and code familiarity → evaluation → production monitoring → what is novel
Pattern 1
The strongest participants treated the AI judge like a technical reviewer. They did not just say "we used RAG." They explained what kind of retrieval they used, why they chose it, where it fit in the pipeline, and what its failure modes were.
A high-signal answer sounded like:
"We used BM25 because the corpus was small and keyword-heavy, then added a reranker only after seeing retrieval misses on specific sample rows."
A lower-signal answer sounded like:
"The agent searches the data and gives the best answer."
Both describe retrieval. Only the first provided the engineering judgment. The winning signal was not to use the fanciest architecture. It was understanding the architecture.
Pattern 2
Another strong pattern was evidence-based iteration. Top candidates frequently interacted with the judge by referring back to results like sample ticket accuracy, specific test cases, changes they reverted because they made the output worse, output.csv regressions, etc.
These participants sounded like people who had actually debugged the system, not just prompted an AI to build it.
They could say things like:
"We tried increasing top_k, but it caused regressions on the Visa fraud rows, so We reverted it and kept the deterministic escalation gate."
The judge rewarded that pattern because it is close to real engineering work where the participant noticed a regression, made a design change, & then made a judgment call.
Pattern 3
The strongest interviews also handled risk concretely. Many candidates used words like risk, validation," escalation, and safety. But top candidates connected those words to the actual behavior of their system.
They could explain how prompt injection was handled, when a ticket should be refused versus escalated, why fraud or unauthorized access needed stricter rules, and how hallucinated policies were added.
This mattered because support triage has asymmetric risk. A wrong confident answer can be worse than an unnecessary escalation.
A strong answer said:
"The LLM cannot downgrade a high-risk ticket. The deterministic gate runs first, and if it detects fraud or unauthorized access, the model is only allowed to produce an escalation response."
A weaker answer said:
"High-risk tickets should be escalated."
Both were directionally correct, but only the first explains the mechanism.
Pattern 4
A common pattern was tool narration. These participants often described what Claude, Cursor, Codex, Copilot, Gemini, or another AI tool did during the build. That was a useful context, but it was not enough by itself.
The strongest users of AI tools still spoke in terms of their own decisions:
"I chose this architecture because..."
"I rejected that approach because..."
"We measured this and reverted it..."
"I made the LLM output schema-enforced JSON because..."
Weaker interviews drifted into:
"Claude created the pipeline."
"The AI checked the files."
"The agent decided the answer."
Lower-ranked candidates often stayed at the same level of abstraction throughout the interview. Weak candidates also often contradicted the code, stayed vague, or tried to defend a feature that was not actually implemented.
The clearest finding was that simply having more back-and-forth with the Judge did not help.
The number of candidate turns had a weak negative correlation with Interview Score. What mattered was answer depth.
Candidates who gave more substantial explanations tended to score higher: total candidate words correlated at r = 0.615, and the share of longer answers correlated at r = 0.631. But this was not just about talking more.
Specificity also mattered: concrete explanations, tradeoffs, examples, and implementation details correlated at r = 0.583, while technical markers correlated at r = 0.481. On the other hand, very short answers were negatively correlated with Interview Score (r = -0.568), suggesting that weak interviews often ended before the candidate had demonstrated a real understanding.
This is how the pattern looked across the leaderboard
Different ways in which people prompt
The chat transcript data itself had an important limitation: it was not uniformly available or consistently structured.
Out of 1349 transcripts, only 658 were clean, meaning about half the submissions either used a different logging format or did not preserve the Context block properly. Some transcripts were rich development logs with Actions, Context, tool metadata, onboarding records, and step-by-step implementation notes. Others were sparse, malformed, or closer to raw support conversations than build records.
Across the transcripts, three interaction clusters stood out.
The strongest users treated the harness like an engineering partner: read the problem statement, inspect the data, compare architectures, then implement. A good interaction had a loop: define the problem, set constraints, make a plan & design choice, test it, inspect failures, and patch.
A weaker interaction skipped that loop: just build the agent, implement this file, or long logs like Ticket 1 processed... Ticket 2 processed... without showing why decisions were made.
The weakest transcripts were not development records at all, for example: “User: I lost my visa card” followed by “Agent: Please call Visa immediately...” That was the output.csv log directly, where the participant asked for answers themselves.
What’s next from here?
We are taking all the learnings from this into the second edition of Orchestrate. The problem will be new, but the signals, code, output, transcript, and interview will be the same.
If you want to find out where you actually stand among the developers worldwide, it’s happening on June 19th! You can join our Discord server to stay tuned to the latest updates and express interest in the next one!
Looking forward to seeing you there!