Skip to content

Patroni and health checks proxy handling #1714

Description

@jpnorenam

Despite Patroni being healthy:

root@juju-78f33d-0:/var/lib/juju/agents/unit-postgresql-0/charm# curl -sk https://10.32.69.4:8008/cluster | jq .
{
  "members": [
    {
      "name": "postgresql-0",
      "role": "leader",
      "state": "running",
      "api_url": "https://10.32.69.4:8008/patroni",
      "host": "10.32.69.4",
      "port": 5432,
      "timeline": 1
    }
  ],
  "scope": "postgresql"
}


root@juju-78f33d-0:/var/lib/juju/agents/unit-postgresql-0/charm# curl -sk https://10.32.69.4:8008/switchover | jq .
{
  "state": "running",
  "postmaster_start_time": "2026-05-26 10:11:01.122042+00:00",
  "role": "master",
  "server_version": 160013,
  "xlog": {
    "location": 24840288
  },
  "timeline": 1,
  "dcs_last_seen": 1779791222,
  "database_system_identifier": "7644140955326675877",
  "patroni": {
    "version": "3.3.8",
    "scope": "postgresql",
    "name": "postgresql-0"
  }
}

It seems httpx.AsyncClient (v0.28+) respects HTTPS_PROXY environment variables by default and routes the Patroni API request through the HTTP proxy, while requests (used for the health check) also respect them but handle no_proxy CIDR notation differently. Resulting in the leader unit awaiting start of the primary:

May 26 10:10:55 juju-78f33d-0 systemd[1]: Started snap.charmed-postgresql.patroni.service - Service for snap application charmed-postgresql.patroni.
░░ Subject: A start job for unit snap.charmed-postgresql.patroni.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit snap.charmed-postgresql.patroni.service has finished successfully.
░░ 
░░ The job identifier is 3046.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The files belonging to this database system will be owned by user "_daemon_".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: This user must also own the server process.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The database cluster will be initialized with locale "en_US.UTF-8".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The default text search configuration will be set to "english".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: Data page checksums are enabled.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: fixing permissions on existing directory /var/snap/charmed-postgresql/common/var/lib/postgresql ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: fixing permissions on existing directory /var/snap/charmed-postgresql/common/data/logs ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: creating subdirectories ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting dynamic shared memory implementation ... posix
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default max_connections ... 100
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default shared_buffers ... 128MB
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default time zone ... Etc/UTC
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: creating configuration files ... ok
May 26 10:10:59 juju-78f33d-0 charmed-postgresql.patroni[7069]: running bootstrap script ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: performing post-bootstrap initialization ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: syncing data to disk ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: initdb: warning: enabling "trust" authentication for local connections
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: Success. You can now start the database server using:
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]:     /snap/charmed-postgresql/current/usr/lib/postgresql/16/bin/pg_ctl -D /var/snap/charmed-postgresql/common/var/lib/postgresql -l logfile start
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7087]: 10.32.69.4:5432 - no response
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:00 UTC [7086]: user=,db=,app=,client=,line=1 LOG:  00000: pgaudit extension initialized
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:00 UTC [7086]: user=,db=,app=,client=,line=2 LOCATION:  _PG_init, pgaudit.c:2263
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=3 LOG:  00000: redirecting log output to logging collector process
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=4 HINT:  Future log output will appear in directory "/var/snap/charmed-postgresql/common/var/log/postgresql".
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=5 LOCATION:  SysLogger_Start, syslogger.c:712
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7098]: 10.32.69.4:5432 - accepting connections
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7100]: 10.32.69.4:5432 - accepting connections
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: Exception in thread Thread-43 (process_request_thread):
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: Traceback (most recent call last):
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     self.run()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     self._target(*self._args, **self._kwargs)
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3/dist-packages/patroni/api.py", line 1685, in process_request_thread
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     request.do_handshake()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3.12/ssl.py", line 1320, in do_handshake
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     self._sslobj.do_handshake()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: ssl.SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:1000)

Steps to reproduce:

$ juju add model database
$ juju model-config http-proxy=http://squid.internal:3128
$ juju model-config https-proxy=http://squid.internal:3128
$ juju model-config no-proxy="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"

$ juju model-config cloudinit-userdata='''#cloud-config
write_files:
- path: /etc/environment
  permissions: '0644'
  owner: root:root
  content: |
    LANG=en_US.UTF-8
    LC_ALL=en_US.UTF-8
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
    http_proxy=http://10.22.2.1:3128
    https_proxy=http://10.22.2.1:3128
    no_proxy="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"
    HTTP_PROXY=http://10.22.2.1:3128
    HTTPS_PROXY=http://10.22.2.1:3128
    NO_PROXY="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"
'''
$ juju deploy postgresql -n 3

$ juju status
App         Version  Status  Scale  Charm       Channel     Rev  Exposed  Message
postgresql  16.13    active      3  postgresql  16/stable  1089  no       

Unit           Workload  Agent  Machine  Public address  Ports     Message
postgresql/0*  waiting   idle   0        10.32.69.4      5432/tcp  awaiting start of the primary
postgresql/1   waiting   idle   2        10.32.69.6      5432/tcp  awaiting for cluster to start
postgresql/2   waiting   idle   4        10.32.69.10     5432/tcp  awaiting for cluster to start

Machine  State    Address      Inst id        Base          AZ          Message
0        started  10.32.69.4   juju-78f33d-0  ubuntu@24.04  pc8b-infra  Running
2        started  10.32.69.6   juju-78f33d-2  ubuntu@24.04  pc8b-infra  Running
4        started  10.32.69.10  juju-78f33d-4  ubuntu@24.04  pc8b-infra  Running

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working as expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions