[release/10.0] Fix GC hole when method return is hijacked for GC suspension

[steveisok]

Fixes Issue #131267

main PR #129714

Description

Except on x86, no registers are scanned as part of the HijackFrame on top of the stack — HijackFrame::GcScanRoots_Impl is compiled only under #ifdef TARGET_X86. The hijacked return value is instead reported to the GC through the calling method's GC info. That works as long as the caller is managed code.

It breaks when the caller is CallDescrWorkerInternal, which is hand-written assembly with no GC info. An objectref left in the return register (x0/rax) at hijack time is then neither reported nor updated.

This declines to hijack when the return address is CallDescrWorkerInternal's, which is what the main fix does.

Customer Impact

Reported by Gearset, a commercial Salesforce DevOps product, as intermittent SIGSEGV test-host crashes in their CI on released 10.0.x. Their suite compiles LINQ Expressions and Moq mocks (LCG DynamicMethod emit) while a parallel test runner drives GC load. Crashes are rare per run but frequent enough across a day's builds to force manual re-runs and stall their merge queue, and there is no managed-code workaround.

Any Call_RetOBJECTREF call site is exposed — there are 33 in release/10.0 — as is reflection invoke via MethodBaseInvoker.InterpretedInvoke_Method -> RuntimeMethodHandle_InvokeMethod -> CallDescrWorkerInternal. Symptoms are silent heap corruption: SIGSEGV in gc_heap::mark_object_simple, or Object::ValidateInner / GetMethodTable asserts on checked builds.

Regression

Yes. This is a regression introduced in .NET 10 by #110799

Testing

Local A/B on a checked release/10.0 build (osx-arm64), swapping only libcoreclr.dylib, 8 runs x 30 s each against a reflection-invoke repro that returns a byte[] from a long straight-line (partially interruptible) region under concurrent GC.Collect:

Build	Failures
release/10.0 unpatched	6 / 8 — 4x Object::ValidateInner (pMT->Validate()), 2x GetMethodTable MARKED_BIT assert
this change	0 / 8 — ~79k iterations clean

The same repro also faults on shipped 10.0.9 (SIGSEGV in WKS::gc_heap::mark_object_simple) and on 11.0.0-preview.6, and passes on current main, which already carries #129714.

Separately, an instrumented build logging every hijack recorded one landing directly on System.Reflection.Emit.DynamicResolver::GetCodeInfo with the return address inside CallDescrWorkerInternal — i.e. the byte[] behind LCGMethodResolver::GetCodeInfo live and unreported in x0. Reaching that required widening the race window, since hijacking is rare in that workload.

./build.sh clr -rc checked is clean (0 errors, 0 warnings). All local runs are osx-arm64; CI covers the rest of the matrix.

Risk

Low. 15 lines in one function, guarded to non-x86.

The change only declines an optimization: hijacking is one of several ways to bring a thread to a safe point. A thread returning into CallDescrWorkerInternal is leaving managed code and will hit a preemptive-mode transition, so suspension still completes — it is not a hang risk.

The equivalent change has been in main since 2026-06-23 (#129714).

Note

This pull request description was drafted with the help of GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Backports the coreclr suspension/hijack safety check to release/10.0 to prevent a GC hole when a thread is hijacked while returning into CallDescrWorkerInternal (which has no GC info), avoiding untracked/unupdated objectrefs in the return register on non-x86 architectures.

Changes:

• Adds a non-x86 early-out in Thread::HijackThread when the return address is CallDescrWorkerInternal.
• Introduces a local forward declaration for IsCallDescrWorkerInternalReturnAddress to reuse the existing helper.

[jkotas]

Thanks!

[jkotas]

I have fixed inaccurate description. In particular, the original description said that this is not a regression that's not correct. This is fixing .NET 10 regression