Fix RZ1000 error for UTF-8 string literals in Razor files

[Copilot]

Summary of the changes

The Razor parser incorrectly flagged UTF-8 string literals ("text"u8) as unterminated, triggering RZ1000 errors.

@{
    ReadOnlySpan<byte> span = "hello"u8;  // RZ1000 before, compiles after
}

Changes:

• Added IsStringProperlyTerminated helper to RoslynCSharpTokenizer that validates strings ending with u8/U8 suffix
• Handles edge cases: empty strings (""u8), verbatim strings (@"path"u8), and raw strings ("""text"""u8)
• Added tokenizer unit tests (10 tests total: 6 for regular/verbatim UTF-8 strings + 4 for raw UTF-8 string literals)
• Added integration tests for .cshtml/.razor files covering @{ }, @( ), @code, and @functions blocks with UTF-8 string literals including raw strings

Original prompt

---

This section details on the original issue you should resolve

<issue_title>RZ1000 triggered when Utf8 strings literals are used</issue_title>
<issue_description>## Description

Using Utf8 string literals in a razor page trigger

RZ1000
Unterminated string literal.  Strings that start with a quotation mark (") must be terminated before the end of the line.  However, strings that start with @ and a quotation mark (@") can span multiple lines.

PR that introduced this (?): #11086

Repro

Create a ASP.NET Core Web App (Razor Pages) project and add the following code to Index.cshtml:

@page
@using System.Text
@model IndexModel

<div class="text-center">
    @{
        ReadOnlySpan<byte> span = "hello"u8;    // <- here's the error location
    }

    <p>@Encoding.UTF8.GetString(span)</p>
</div>

(yeah I know it's nonsense, but stripped down from a real project were Utf8 literals makes sense).

Expected behavior

Utf8 string literals are supported and don't trigger RZ1000.

Workaround

Update csproj with

<PropertyGroup>
    <NoWarn>$(NoWarn);RZ1000</NoWarn>
</PropertyGroup>
```</issue_description>

<agent_instructions>Please update the parser so that C# UTF-8 strings are supported. Create compiler integration tests, using the example repro in this issue, but also make sure to include coverage for .cshtml and .razor files, and for `u8` strings in `@code` and/or `@functions` blocks, `@{ .. }` blocks, and `@( ... )` blocks etc.</agent_instructions>

## Comments on the Issue (you are @copilot in this section)

<comments>
</comments>

• Fixes RZ1000 triggered when Utf8 strings literals are used #12132
• Fixes @functions block emitted inside ExecuteAsync (instead of as class members) when .cshtml contains UTF-8 string literals #12777

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[davidwengier]

@copilot it looks like the parser change will work for UTF-8 raw string literals, but there is no test coverage for them. Please add

[Copilot]

@copilot it looks like the parser change will work for UTF-8 raw string literals, but there is no test coverage for them. Please add

Added test coverage for UTF-8 raw string literals in commit d83c460. This includes:

• 4 new tokenizer tests for raw strings with u8/U8 suffix
• Raw string examples in .cshtml test file (@{ } blocks and @functions)
• Raw string example in component test (@code block)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• vb4vsblobprodcus33.vsblob.vsassets.io
  • Triggering command: /usr/share/dotnet/dotnet dotnet build src/Compiler/Microsoft.CodeAnalysis.Razor.Compiler/src/Microsoft.CodeAnalysis.Razor.Compiler.csproj (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[davidwengier]

LGTM

[ToddGrun]

[davidwengier]

@chsienki would appreciate your signoff on this too, in case there is a better spot you know of for a fix.