salqadri/argon-asl-question-understanding-eval · Datasets at Hugging Face

Answering ASL questions from video: Gemini Argon leads a 21-question pilot

Syed Salman Qadri · Rayan Syed Qadri
October 2026 · Preliminary case study · 21 questions from one signer

American Sign Language (ASL) has grammar distinct from English. A model must follow handshape, movement, facial expression, body posture and timing; missing a small cue can change a question's meaning. Reliable ASL input could help Deaf signers ask questions, use tutoring tools and reach digital services in their own language. Signed STEM resources remain scarce. NIDCD on ASL · ASL STEM Wiki (EMNLP 2024)

We tested whether general video models could answer a signed question directly, without being asked to translate it. On the same 21 silent recordings and generic prompt, Gemini 4 Argon matched the full intended question in 69.4% of completed answers (59/85), versus 11.4% (12/105) for Gemini 3.1 Pro preview and 6.0% (6/100) for Gemini 3.8 Flash. All used HIGH thinking and five rounds per video. Argon completed 81.0% of requests (85/105), yielding an overall successful-answer rate of 56.2% (59/105). A later, separate one-round follow-up with Meta Muse Wonder 1.0 completed all 21 requests and yielded 0/21 matches under its provider's video pipeline.

Argon's lead on these recordings is striking, and several answers directly addressed complete science questions. The result is a single-signer case study, scored against contributor labels by OpenAI's GPT-6 Astra. Completed-answer matching describes performance conditional on producing an answer. Incomplete attempts may relate to question difficulty, and these recordings cannot establish success rates for ASL users generally.

Results

Inspect every attempt: Complete answers and judgments, organized by question, with downloadable records for the 336 original-prompt attempts and eight separately prompted follow-ups.

We emphasize intent matching among completed answers alongside completion rate. The former measures whether a returned answer addresses the intended question; it does not audit every factual claim. The latter measures how often a request produces a complete answer, whether right or wrong. The secondary all-attempt match rate counts incomplete answers and API errors as unsuccessful delivery. All three measures use the same saved judgments; no trials were rescored.

Argon's 20 unsuccessful deliveries included 4 HTTP errors, 13 token-limit stops and 3 incomplete responses without a recorded finish reason. Flash had 5 token-limit stops; Pro and Meta completed every request. These are not all API-stability failures. For example, four of Argon's five attempts on the airplane question reached the token limit, and its one completed answer was incorrect. Excluding incomplete attempts gives that question less weight, so completed-answer rates compare different subsets of attempts. Removing only the four recorded HTTP errors would give Argon 59/101 (58.4%).

Five Gemini rounds show substantial variation by question. Argon matched all five attempts on six questions, at least once on 15 of 21, and never on six. Those repetitions are not additional signers or independent language examples. The round counts below retain all 21 attempts per round, including unsuccessful deliveries.

For completed-answer matching, resampling the 21 question groups 20,000 times gave 95% percentile intervals of Flash 0.0–15.7%; Pro 3.8–21.0%; Argon 52.6–83.2%. Each draw keeps a selected question's five Gemini attempts together and divides total matches by total completed answers. The chart's whiskers show these intervals; its dots show each round's completed-answer rate. These intervals describe question selection under the bootstrap assumptions. They do not correct for selective noncompletion or represent signer diversity. The earlier all-attempt intervals remain in the summary. Meta has only one round, so no five-round interval is shown; 0/21 is not proof of zero underlying success.

Recordings and prompt

The 21 videos show one female ASL expert, as described by the authors, standing before a plain wall. The English intended questions came from contributor comments. No separate signer-confirmed transcript of every recording is documented.

Privacy-edited illustration derived from one frame. An AI image-editing tool applied strong blur; composition is approximate. It cannot be used to inspect signs, and the original image and videos remain private.

Exact prompt used for the 336 primary results:

The video shows a deaf person asking a question in American Sign Language (ASL). Answer the question directly in text. Do not translate, transcribe, or restate the signed question.

These are 1080p / 30fps source files submitted without cropping or video re-encoding. The 24 fps API field is a sampling request, not a claim that every source frame was seen. Google's documented VideoMetadata.fps range for this runner was at most 24. Google API reference

Scoring and the short-answer follow-up

OpenAI's GPT-6 Astra scored all 336 original-prompt responses against each contributor's intended question. It marked Yes when an answer preserved the essential objects, quantities and relationships, including when it corrected a false premise. Wrong relations, topic-only replies, translation without an answer and requests to repeat were No. Model identity was available during review. The released rows include judgment reasons and Astra's uncalibrated understanding-confidence estimates; these are not the tested models' self-reported confidence.

An answer could receive Yes when it substantively addressed the intended question, even if it also offered other interpretations. Extraneous interpretations could reduce confidence without automatically invalidating the match. Four accepted Argon answers to the 3,200-square-foot capacity question assumed a tent; one primarily discussed 320 square feet, then addressed 3,200 square feet as an alternative. A matched bubbles answer also discussed triangles and the Bermuda Triangle. These judgments credit coverage of the intended question without requiring an exclusively correct interpretation; the complete answers and reasons allow a stricter reading.

A bare number or Yes/No reply may answer a question without showing which question the model saw. The scoring rubric counted such compatible but nondiagnostic replies as No. Among completed answers, accepting every flagged short reply would raise Flash to 8/100 (8.0%) and Pro to 18/105 (17.1%); Argon remains 59/85 (69.4%).

We made eight selected reruns across three recordings: six Pro attempts and two Flash attempts, with one fresh generation for each flagged original response. The recordings asked about human capacity in a 3,200-square-foot square, whether fish can see water, and the time needed to walk around Earth. The silent files, models, HIGH settings and 24 fps request stayed the same. The prompt added this sentence after the original text:

Give a direct answer in at least 30 words, with enough context to make clear what you are answering.

Under that changed prompt, one of eight attempts matched full intent (1/6 Pro; 0/2 Flash). All seven returned answers met the length requirement; one request timed out. GPT-6 Sol reviewed these eight new answers using the same intent rubric. These were fresh generations, not elaborations of the original answers. The selected follow-up cannot establish whether the longer prompt improves performance generally or reveal what the models understood on their original attempts. It is excluded from all primary comparison statistics and charts.

Three Pro answers to “Can fish see water?” show why context matters. These are excerpts from separate reruns of the same recording:

All eight follow-up answers and their timings, original short replies and judgments remain available, including the timeout.

Generation time

Times cover the full non-streaming generation call, including thinking and network response, and exclude uploads and pacing. Gemini file activation occurred beforehand; Meta may process video within the timed call. The chart shows completed responses and marks failed attempts separately. Argon's completed-response median was 52.2 s, below Pro's 73.6 s, while its mean was 92.0 s versus Pro's 85.2 s and it had 20 unsuccessful deliveries. Provider processing, server load and collection order limit direct speed comparisons.

What the answers show

For “Why are bubbles round?”, Argon's first answer began:

Bubbles are round because of surface tension.

It went on to explain that a sphere minimizes surface area for the trapped air volume and that the balance of inward surface tension and outward air pressure produces the round shape. This directly addressed the signed question with a scientific explanation. Argon earned a match in all five rounds on this recording, although one answer also offered unrelated interpretations. Read all five answers.

Argon also matched every round on why people cannot see air, but missed every attempt on six other questions. When asked for an Xbox controller's response time, one Argon answer began, “To improve and increase the frames per second (FPS) on your Xbox,” then gave display-setting advice. It recognized the Xbox topic but missed the requested timing. Another answer to a question about travelling across the universe described squirrel lifespans.

Other failures wandered farther. A Flash answer to “Is an octopus a mammal?” described Isha prayer, despite the generic prompt containing no religious text. The signer wore a hijab; an appearance-related cue is one possible explanation, but clothing and appearance were not varied, so this study cannot establish a cause or any access to the authors' personal history.

Meta returned a completed answer for every clip, yet none matched full intent. For “Why do we dream when we sleep?”, it answered, “Nothing. As an AI, I never sleep, so I never dream.” The same files and prompt were supplied, but Meta's internal media processing could not be verified as equivalent to Gemini's. Its first round gave no indication that repeating the unchanged setup would be productive; its repeatability remains unmeasured.

Model scope and related work

Gemini's file-video path supported the raw private MP4 and requested 24 fps sampling. Meta offered a native uploaded-video route, so we added its one-round follow-up. The Gemini Live API documents at most one video frame per second; we did not use it for these continuous signing clips. No Live trials were run.

The reviewed OpenAI vision, Anthropic vision and Grok X Search interfaces did not offer our same native private-MP4-to-text route. Extracted frames or X-hosted video would require a different protocol.

Google's DeepMind recently introduced the SL2T translation model and launched it on Google Pixel 11 phones that seems really promising, however, it does not yet seem to have been made available for developers as an API or downloadable model.

Questions and consistency

Cells show full-intent matches out of all attempts from the unchanged original prompt, including incomplete answers and API errors in the denominator. Labels are contributor intent, not separately validated ASL transcripts.

Privacy and reuse

The signer does not consent to release her original videos or recognizable pictures. This repository shares results and one authorized, strongly blurred framing illustration. Source videos, original filenames, unblurred frames and album links are withheld. Others can inspect answers and judgments and recompute the charts, but cannot rerun inference from this public release. The results are for exploratory research, not validation of a deployed ASL interpreter.

There is one signer and a convenience sample of 21 questions. The Gemini runs were sequential across model blocks. Provider preprocessing and model versions may change. These limits matter when interpreting general use and response time.

The complete trial supplement presents all 344 attempts by question, including every returned answer, intent judgment, understanding-confidence estimate, generation time and unsuccessful attempt. Download the 336 original-prompt attempts—315 Gemini and 21 Meta—as CSV or JSON Lines, and the eight separately prompted reruns as CSV. Two answers, one original and one follow-up, have detailed appearance descriptions redacted and explicitly flagged; all other answer text is retained. Follow-up word counts refer to the original text before redaction.

Questions, original prompt, follow-up prompt, primary protocol, follow-up protocol, summary, analysis code and Meta input audit provide the supporting detail.

The text, results and charts are shared under CC BY 4.0. The blurred illustration is excluded from that license and provided for viewing here only; all rights are reserved for that image. Model answers are experimental outputs, not endorsed advice. The report, collection and scoring used AI assistance.

Citation

@misc{qadri2026argon_asl_pilot,
  author = {Qadri, Syed Salman and Qadri, Rayan Syed},
  title = {Answering ASL Questions from Silent Video: Argon Leads a 21-Question Pilot},
  year = {2026},
  howpublished = {Hugging Face evaluation results repository},
  url = {https://huggingface.co/datasets/salqadri/argon-asl-question-understanding-eval}
}