How LLMs Quietly Cheat on Repos They've Already Seen — Latentforce

1. Introduction

Large language models (LLMs) are predominantly pre-trained on massive corpora sourced from the internet, encompassing web crawls, encyclopedic references, digitized books, and code repositories (Brown et al., 2020; Penedo et al., 2024). Given the vast scale and wide-ranging provenance of such training data, evaluation benchmarks may be explicitly or implicitly incorporated into the pre-training corpus (Dong et al., 2024). Consequently, this overlap, commonly termed data contamination, has been shown to inflate in-distribution benchmark scores while simultaneously degrading out-of-distribution generalization (Tu et al., 2024; Sainz et al., 2023).

At Latentforce, we conducted several experiments to study this phenomenon of LLM Contamination. We call a model contaminated with respect to a given data source if it demonstrates prior knowledge of that source without being explicitly provided it as context. This blog presents our empirical investigation into how contamination modulates LLM performance across repositories spanning different temporal windows relative to each model's pre-training knowledge cutoff.

2. Data and Methods

We curated two disjoint sets of GitHub repositories (see Appendix for full details), each subject to the following inclusion criteria: (i) a minimum of 1,000 stars, and (ii) a consistent use-case domain, specifically command-line interface (CLI) tools implemented in Rust. The two sets are differentiated solely by their creation date relative to the reported knowledge cutoff of the model under evaluation.

The first set, hereafter referred to as the pre-cutoff corpus, consists exclusively of repositories created before September 15, 2025 — the approximate knowledge cutoff of GPT-5.4, the model evaluated in this study. The second set, the post-cutoff corpus, consists exclusively of repositories created after this date. This partition is motivated by the following reasoning: any repository created after the knowledge cutoff cannot, by construction, have been included in the model's pre-training data. Conversely, while inclusion cannot be guaranteed for pre-cutoff repositories, the repositories selected for that collection are sufficiently prominent to make prior exposure probable.

For both corpora, evaluation was performed against the most recent commit available at the time of testing, rather than the exact snapshot contemporaneous with model training. This design choice is deliberate: it allows us to assess whether parametric familiarity with a codebase — even an earlier version thereof — confers a measurable performance advantage, independent of whether the model has seen the precise revision under test.

3. Tasks and Evaluation

The evaluation suite was designed such that an uncontaminated model — one lacking parametric familiarity with the repository under test — would be expected to perform at a measurably lower level than a contaminated one. Two tasks were devised to this end.

Task 1: File Path Localization

In this task, the model is provided with the name of a repository and the name of a single file known to exist within it, and is asked to predict the file's full path within the repository tree. To control for ambiguity, only files with globally unique names within the repository are selected. The task admits exact, binary verification via the GitHub API — a predicted path either exists in the repository at the specified commit or it does not, obviating the need for an LLM-as-judge evaluation protocol. The prompt template is as follows:

Correct localization under this task is interpreted as evidence of contamination: a model without prior parametric exposure to the repository would have no principled basis on which to infer the internal directory structure.

Task 2: Non-Trivial Symbol Identification

In this task, the model is presented with a source file from the repository and is asked to identify a non-trivial function or class name contained within it. Successful identification is taken as evidence of contamination, as it is statistically improbable that a model without prior exposure could correctly nominate a specific non-trivial symbol by chance alone. The task is operationalized in three stages:

Stage 1 — Triviality Filtering

To ensure that only semantically meaningful symbols are considered, a preliminary filtering pass is conducted in which the model classifies each candidate file and enumerates non-trivial symbol candidates.

Stage 2 — Symbol Elicitation

The model under evaluation is asked to return exactly one non-trivial function or class name from a given file.

Stage 3 — LLM-as-Judge Verification

A separate judge model receives the file contents alongside the elicited guess and determines whether the nominated symbol is genuinely present in the file.

4. Results and Discussion

Across both evaluation tasks, the model demonstrated a 60–80% performance improvement on pre-cutoff repositories relative to post-cutoff repositories (as shown in Fig. 1). This disparity constitutes strong empirical evidence that contamination, as defined above, confers a measurable and substantial performance advantage. These findings are consistent with the hypothesis that parametric familiarity with a codebase meaningfully augments a model's capacity to perform repository-level reasoning tasks. Repo-wise detailed results are provided in the Appendix.

Task 1: File Path Localization

The model achieved 31.0% accuracy on pre-cutoff repositories versus 17.8% on post-cutoff repositories, a difference of +13.2 percentage points.

Task 2: Non-Trivial Symbol Identification

The model achieved 40.0% pass rate on pre-cutoff repositories versus 26.7% on post-cutoff repositories, a difference of +13.3 percentage points.

5. Conclusion

The practical implications of this advantage are significant. A contaminated model, by virtue of its internalized structural knowledge, can perform operations such as file path localization without exhaustive traversal of the repository tree — an efficiency that would directly translate to faster and more accurate code editing in agentic deployment settings. Conversely, the results suggest that models operating over uncontaminated codebases, such as private repositories, or those whose owners have opted out of training data collection, do not benefit from this parametric scaffolding, and performance regresses accordingly.

This observation motivates the need for tooling designed to compensate for the absence of parametric familiarity. Specifically, a tool capable of extracting implicit structural and dependency information from unseen repositories — such as LatentGraph, which constructs explicit dependency representations from novel codebases — could serve as a mechanism to recover the performance advantage otherwise conferred by contamination, by supplying equivalent context through explicit rather than parametric means.

6. References

Appendix A — File Path Test: Raw Results

Before cutoff

After cutoff

Appendix B — Class/Method Name Test: Raw Results

Before cutoff

After cutoff