Skip to content

Reader Error when trying to replicate to backup db #148

@tomfriedhof

Description

@tomfriedhof

Describe the bug
I tried to replicate a new database after an initial application boot. The application sets up several streams on boot to populate general configuration for the application. After everything is initialized, and after I have verified the application is working as expected, I run the replicator to mirror the db to another EventStore instance. The replicator gets to 95% complete and then produces a read error:

{
  "@t": "2025-07-07T06:29:50.3107288Z",
  "@m": "Reader error",
  "@i": "450d20cc",
  "@l": "Error",
  "@x": "System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.\n   at EventStore.Replicator.Esdb.Grpc.Internals.StreamMetadataJsonConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /app/src/EventStore.Replicator.Esdb.Grpc/Internals/StreamMetadataJsonConverter.cs:line 96\n   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\n   at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.MapMetadata(ResolvedEvent evt, Int32 sequence, Activity activity) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 142\n   at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.ReadEvents(Position fromPosition, Func`2 next, CancellationToken cancellationToken) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 93\n   at EventStore.Replicator.Read.ReaderPipe.<>c__DisplayClass1_0.<<-ctor>g__Reader|1>d.MoveNext() in /app/src/EventStore.Replicator/Read/ReaderPipe.cs:line 41",
  "SourceContext": "EventStore.Replicator.Read.ReaderPipe"
}

To Reproduce
Steps to reproduce the behavior:

  1. Boot up the application, and allow all setup streams to be appended to log. Verify application is working.
  2. Run the Replicator to copy db events to another instance
  3. Unable to finish replication due to a read error

Config/Logs/Screenshots
The logs from the replicator prior to the error look like this. It seems to be something with reading the stream metadata:

repl-replicator|{
  "@t": "2025-07-07T06:29:55.3112993Z",
  "@m": "gRPC: Read event with id \"f54f1c07-2d93-48fa-91be-5a24ded58389\" of type \"$ProjectionCheckpoint\" from \"$projections-tenantA--ContactAttachmentIndex-checkpoint\" at \"C:555279090/P:555279090\"",
  "@i": "4138b59c",
  "@l": "Debug",
  "@tr": "ec5c6baab66310cbd0ab33a62bf45e3c",
  "@sp": "8a756f934965c9c5",
  "Id": "f54f1c07-2d93-48fa-91be-5a24ded58389",
  "Type": "$ProjectionCheckpoint",
  "Stream": "$projections-tenantA--ContactAttachmentIndex-checkpoint",
  "Position": "C:555279090/P:555279090",
  "SourceContext": "EventStore.Replicator.Esdb.Grpc.GrpcEventReader"
}
repl-replicator|{
  "@t": "2025-07-07T06:29:55.3114730Z",
  "@m": "gRPC: Read event with id \"e545277b-6539-4bda-af97-611283c863d5\" of type \"$metadata\" from \"$$tenantB/$tenantB/UserNotification\" at \"C:555279316/P:555279316\"",
  "@i": "4138b59c",
  "@l": "Debug",
  "@tr": "22a6c7db1fc30648fd1ca2c45aadd87e",
  "@sp": "d99737bb3477b12c",
  "Id": "e545277b-6539-4bda-af97-611283c863d5",
  "Type": "$metadata",
  "Stream": "$$tenantB/$tenantB/UserNotification",
  "Position": "C:555279316/P:555279316",
  "SourceContext": "EventStore.Replicator.Esdb.Grpc.GrpcEventReader"
}

EventStore details

  • EventStore server version: eventstore:latest docker image

  • Operating system: Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions