Outdated Gitaly in 19.1+

[JKHSDTV]

I am uncertain how it is possible with how I understand the Docker image release pipeline works, but Gitaly in the 19.1.x images is stuck at 19.0.0:

$ docker run --rm -it docker.io/sameersbn/gitlab:19.1.1 grep -n "gitaly" /home/git/gitlab/Gemfile /home/git/gitlab/Gemfile.lock
Loading /etc/docker-gitlab/runtime/env-defaults
/home/git/gitlab/Gemfile:648:gem 'gitaly', '~> 19.0', feature_category: :gitaly
/home/git/gitlab/Gemfile.lock:777:    gitaly (19.0.0)
/home/git/gitlab/Gemfile.lock:2437:  gitaly (~> 19.0)
/home/git/gitlab/Gemfile.lock:2936:  gitaly (19.0.0) sha256=b908390020e4c3bcc101dd8651da65b998190912201a1821194283866a6c5d6a

This appears to be leading to all kinds of corrupted data in the database due to differences in the protobuf format.

[tollscoherent]

Hmmm for the 19.1.1 tag the gemfile in https://gitlab.com/gitlab-org/gitlab-foss/-/blob/v19.1.1/Gemfile?ref_type=tags also defines

# Gitaly GRPC protocol definitions
gem 'gitaly', '~> 19.0', feature_category: :gitaly

so looks like an upstream issue to me (if any)

[kkimurak]

• I think it's upstream issue
• I’m not sure if that will cause any issues, but I’m optimistic that it won’t be anything critical.

The update process of the dependency "gitaly" (in Gemfile) seems to be automated and update MR to v19.1.0 is already going on: gitlab-org/gitlab!239169
But seems there hasn't been a review for a month (created on 2026-06-04 and still open today 2026-06-30). The MR initially updated gitaly with v19.0.2 in mind, but the current target is v19.1.0 and pending for v19.1.1.

Additionally, the first stable GitLab release to include the previous MR gitlab-org/gitlab!229905 (the update of gitaly gem to 18.11.0) is v19.0.0-ee (or v19.0.0-rc42-ee if including RC releases).
This suggests that the GitLab version (in sameersbn/gitlab it is specified as GITLAB_VERSION in Dockerfile), Gitaly binary version (GITALY_SERVER_VERSION), and gitaly gem version (defined in Gemfile in gitlab codebase) do not necessarily have to match.

(The version of the gitaly gem should match the version specified in the GitLab codebase unless we intentionally modified it by applying a patch or something similar. To the best of my knowledge, we haven't done that at all over the past few years I'm contributing)

---

I believe that minor version discrepancies should not be an issue, provided there are no critical differences anyway..

Comparing Gitaly v19.0.0 and v19.1.1, there are changes to four files—specifically within the proto/ directory, which is likely to affect the gitaly gem. I think half of them (*.pb.go) are automatically generated from .proto so it looks like just a single field or enum was added to each of the two files (field exclude_patterns to 2 messages in ref.proto and enum value Phase::PREPARING_PHASE to message VoteTransactionRequest in transaction.proto).

I don't know how critical this is.

https://gitlab.com/gitlab-org/gitaly/-/compare/v19.0.0...v19.1.1?from_project_id=2009901

[JKHSDTV]

Ah, my bad - I saw the changelog claiming gitaly: upgrade to v19.1.1 so I assumed there was an error in the upgrade not taking somehow.

I have found various errors in regards to data, and after realizing they were unlikely due to the Gitaly version issue, since as you said those are present in upstream, I dug some more and found out the PostgreSQL 16->17 update via docker.io/kkimurak/sameersbn-postgresql:17 appears to have introduced a change similar to what is described in https://wiki.postgresql.org/wiki/Locale_data_changes though I don't think it is that exact problem.

@kkimurak I hope it is okay to tag you here, or we can open a new issue here or on that repo, or go to discussions. I am terribly unfamiliar with Postgres, so I am reluctant to make a proper issue out of my findings which I just very barely understand at all. It appears that the upgrade from 16 to 17 corrupted indexes, particular indexes on columns with textual values. I initially discovered this due to a bug that I narrowed down to there being a difference between SELECT count(*) FROM namespaces WHERE type LIKE 'Group' and SELECT count(*) FROM namespaces WHERE type = 'Group'. Rebuilding the indexes for the whole database fixed this issue and the bug, but I later started to discover more and more issues. My current working theory is that the 16-17 upgrade process is somehow subtly flawed; though it is possible that this somehow due to our specific data or setup. My theory continues that migrations running on the corrupted database then introduced further damage to our data, and while rebuilding the indexes helped mitigate some of the problems, others caused by migrations running with broken indexes led to actual data loss that can not be recovered. Let me know if and how you would like to further discuss my findings

[kkimurak]

@JKHSDTV Sorry but I won't be able to make time this week because I have a funeral to attend. I'll probably open a new issue in the kkimurak/docker-postgresql repository.

[JKHSDTV]

I am very sorry to hear that. We rolled back to 18.x, so for us, it isn't by any means urgent, take your time. We are planning on doing a controlled, slow upgrade attempt on a VM, which I can report back on - I'll open an issue on the postgres repo with more details.