[BUG] Pillars are empty when running through custom execution module with salt-ssh

[helderco]

Description
I'm trying to access pillars inside a custom execution module using salt-ssh, but the __pillars__ dict is empty.

Using __salt__['pillar.get']() though.

Setup
I'm executing via a docker container in my local machine, targetting a Digital Ocean droplet.

# Dockerfile
ARG SALT_VERSION
FROM saltstack/salt:${SALT_VERSION}
ARG SALT_VERSION
RUN apk add --no-cache openssh-client
RUN USE_STATIC_REQUIREMENTS=1 pip3 install --no-cache-dir salt-ssh=="${SALT_VERSION}" pyopenssl
CMD ["salt-ssh"]

# composer.yaml
services:
  ssh:
    build:
      args:
        SALT_VERSION: 3004
    hostname: salt
    volumes:
      - ./etc/master:/etc/salt/master:ro
      - ./etc/roster:/etc/salt/roster:ro
      - ./roots:/srv/salt:ro
      - ./Saltfile:/root/.salt/Saltfile:ro
      - $HOME/.ssh/id_rsa:/etc/salt/pki/master/ssh/salt-ssh.rsa:ro

# /etc/salt/master

file_client: local

file_roots:
  base:
    - /srv/salt/base/salt
    - /srv/salt/formulas/mysql-formula
    - /srv/salt/formulas/users-formula
    - /srv/salt/formulas/redis-formula
    - /srv/salt/formulas/docker-formula

pillar_roots:
  base:
    - /srv/salt/base/pillar

# /etc/salt/roster
minion:
    host: my.example.net
    set_path: '/usr/local/opt/pyenv/versions/3.7.12/bin:$PATH'  # to make sure I have the same version on both sides

Here's /srv/salt/ structure:

.
├── base
│   ├── pillar
│   │   ├── ...
│   │   └── top.sls
│   └── salt
│       ├── _modules
│       │   └── helpers.py
│       ├── ...
│       └── top.sls
└── formulas
    ├── docker-formula
    ├── mysql-formula
    ├── redis-formula
    └── users-formula

# pillar/init.sls

user: myuser

# salt/_modules/helpers.py

def pillar_get(key, default=None, **options):
    return __salt__['pillar.get'](key.replace('/', ':'), default)

# to debug
def pillars():
    return ', '.join(__pillar__.keys())

Steps to Reproduce the behavior

$ docker compose run --rm ssh salt-ssh minion pillar.get user
minion:
    myuser
$ docker compose run --rm ssh salt-ssh minion helpers.pillar_get user
minion:
    None
$ docker compose run --rm ssh salt-ssh minion helpers.pillars
minion:

Expected behavior
I expect that __pillar__ inside _modules/helper.py is filled, and that __salt__['pillar.get']() works.

As you can see, it works when executing salt-ssh minion pillar.get directly. Also, salt-ssh minion pillar.items returns my pillars, so it's being matched correctly in pillar/top.sls.

Versions Report

salt-ssh --versions-report

Salt Version:
          Salt: 3004
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.17
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.6.1
        Python: 3.7.12 (default, Sep  8 2021, 01:55:52)
  python-gnupg: 0.4.4
        PyYAML: 5.4.1
         PyZMQ: 18.0.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: alpine 3.14.2 
        locale: UTF-8
       machine: x86_64
       release: 5.10.76-linuxkit
        system: Linux
       version: Alpine Linux 3.14.2

salt-ssh minion test.versions_report

minion:
    Salt Version:
              Salt: 3004
     
    Dependency Versions:
              cffi: Not Installed
          cherrypy: Not Installed
          dateutil: Not Installed
         docker-py: Not Installed
             gitdb: Not Installed
         gitpython: Not Installed
            Jinja2: 2.11.3
           libgit2: Not Installed
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         pycparser: Not Installed
          pycrypto: Not Installed
      pycryptodome: Not Installed
            pygit2: Not Installed
            Python: 3.7.12 (default, Feb 10 2022, 23:34:36)
      python-gnupg: Not Installed
            PyYAML: 5.4.1
             PyZMQ: Not Installed
             smmap: Not Installed
           timelib: Not Installed
           Tornado: 4.5.3
               ZMQ: Not Installed
     
    System Versions:
              dist: ubuntu 16.04 xenial
            locale: UTF-8
           machine: x86_64
           release: 4.4.0-148-generic
            system: Linux
           version: Ubuntu 16.04 xenial

Additional context
I've been using salt-master for a number of years, but I'm trying now to depend only on SSH. So my configs have been working for a long time, I'm just trying to make it all work with salt-ssh.

I've shutdown my master and the minion service to depend only on SSH.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[Ch3LL]

Most likely what is happening here is because its a custom module its not going through the ssh wrapper function logic. In the code it loads the wrapper functions like so: self.wfuncs = salt.loader.ssh_wrapper(opts, None, self.context)

As a workaround, you might be able to copy your custom module to the salt/client/ssh/wrapper/ directory and it might work. I think one solution here would be to add the ability to sync custom modules for salt-ssh to use as wrapper functions.