Fix pathological performance in trait solver cycles with errors by erickt · Pull Request #155355 · rust-lang/rust

…uwer

Rollup of 6 pull requests

Successful merges:

 - #154465 (triagebot: Mention 'subtree' in clippy message)
 - #155355 (Fix pathological performance in trait solver cycles with errors)
 - #155716 (arm64e: set ptrauth ABI subtype on lib.rmeta Mach-O objects)
 - #155864 (Fix panic for doc attributes on where predicates)
 - #155865 (add test for accidentally fixed `binius_field` issue)
 - #155866 (Render `ConstContext` for diagnostics once)

Rollup merge of #155355 - erickt:trait-solver-hang, r=lcnr

Fix pathological performance in trait solver cycles with errors

Fuchsia's Starnix system has had a multi-year long bug where occasionally a typo could cause the rust compiler to take 10+ hours to report an error (see #136516 and #150907). This was particularly hard to trace down since Starnix's codebase is massive, over 384 thousand lines as of writing.

With the help of treereduce, cargo-minimize, and rustmerge, after about a month of running we reduced it down to a couple [lines of code], which only takes about 35 seconds to report an error on my machine. The bug also appears to happen with `-Z next-solver=no` and `-Z next-solver=coherence`, but does not occur with `-Z next-solver` or `-Z next-solver=globally`.

I used Gemini to help diagnose the problem and proposed solution (which is the one proposed in this patch):

1. The trait solver gets stuck in an exponential loop evaluating auto-trait bounds (like Send and Sync) on cyclic types that contain compilation errors (TyKind::Error).

2. Normally, if the solver detects a cycle, it prevents the result from being stored in the Global Cache because the result depends on the current evaluation stack. However, when an error is involved, the depth tracking gets pinned to a low value, forcing the solver to rely on the short-lived Provisional Cache. Since the provisional cache is cleared between high-level iterations of the fulfillment loop, the solver ends up re-discovering and re-evaluating the same large cycle thousands of times.

3. Allow global caching of results even if they appear stack-dependent, provided that the inference context is already "tainted by errors" (`self.infcx.tainted_by_errors().is_some()`). This violates the strict invariant that global cache entries shouldn't depend on the stack, but it is safe because the compilation is already guaranteed to fail due to the presence of errors. Prioritizing compiler responsiveness and termination over perfect correctness in error states is the correct trade-off here.

I added the reduction as the test case for this. However, I don't see an easy way to catch if this bug comes back. Should we add some way to timeout the test if it takes longer than 10 seconds to compile? That could be a source of flakes though.

I don't have any experience with the trait solver code, but I did try to review the code to the best of my ability. This approach seems a bit of a bandaid to the solution, but I don't see a better solution. We could try to teach the solver to not clear the provisional cache in this circumstance, but I suspect that'd be a pretty invasive change.

I'm guessing if this does cause problems, it might report an incorrect error, but I (and Gemini) were unable to come up with an example that reported a different error with and without this fix.

Resolves #150907

[lines of code]: https://gist.github.com/erickt/255bc4006292cac88de906bd6bd9220a