fix(rpc): stop classifying expected gRPC responses as trace errors

[sergerad]

Problem

On devnet we regularly observe a spike of errors in our traces, all originating from tower_http::trace::on_failure. TraceLayer::new_for_grpc() classifier treats every non-Ok gRPC status code as a failure, which emits ERROR-level trace events. The errors include:

• RESOURCE_EXHAUSTED (code 8) — rate limiting working as intended
• UNIMPLEMENTED (code 12) — security scanners probing non-existent RPC methods
• UNKNOWN (code 2) — scanners probing paths like .env, .git/config, .aws/credentials

None of these are actual server errors — they're either expected operational behavior (rate limiting) or client/scanner misbehavior.

Solution

Replace TraceLayer::new_for_grpc() with a customized GrpcErrorsAsFailures classifier that treats client-fault and expected-operational codes as successes:

Code	Name	Rationale
2	Unknown	Scanner probing garbage paths
3	InvalidArgument	Client validation errors
5	NotFound	Scanner probing non-existent resources
8	ResourceExhausted	Rate limiting (expected behavior)
12	Unimplemented	Scanner hitting non-existent RPC methods

These responses are still returned to clients with the same gRPC status codes — the only change is that they no longer trigger on_failure (ERROR-level) in traces, and instead flow through on_response (DEBUG-level).

Codes that remain classified as failures: Internal, Unavailable, DataLoss, DeadlineExceeded, Cancelled, PermissionDenied, Unauthenticated.

[Mirko-von-Leipzig]

Thank you!