Use SSH key source secret w/ git+ssh in requirements.txt?

[jeremy303]

Using the 3.6 S2I container, pip needs to pull a dependency from a private git repo as specified in my requirements.txt, e.g:

git+ssh://git@github.organization.org:github-user/githib-repo.git@some_tag

The same SSH key source secret that the OpenShift build is presently using to pull the application source code also permits access to this dependency, however the git+ssh clone is failing:

Collecting git+ssh://git@github.organization.org:github-user/githib-repo.git@some_tag (from -r requirements.txt (line 9))
Cloning ssh://git@github.organization.org:github-user/githib-repo.git@some_tag (to some_tag) to /tmp/pip-7tf7baqk-build
ssh: Could not resolve hostname github.organization.org: Name or service not known
fatal: Could not read from remote repository.

Is there anything I can do that would allow pip to make use of the same SSH key source secret used to clone my app code when installing requirements.txt dependencies?

Thanks.

[GrahamDumpleton]

If understand what you want to do, you would need to use a build secret in the build configuration to have the appropriate SSH keys added into the build image. Are you using a build secret already?

[richardqa]

I have the same problem. I got failed when i deployed my app from Openshift template.
I think the secret resource only work for the GIT of the project, but not for some other GITs in the requirement file. How can I download other Repos GIT from a OpenShift template?

regards,

[jpramos123]

I'm facing a similar issue but using https.

I'm already using the source secret and build secret (both are the same secrets).
Inside the requirements.txt file the reference to the repo is the following:

git+https://gitlab.organization.com/group-user/repo.git

Collecting git+https://gitlab.organization.com/group-user/repo.git  (from -r
requirements.txt (line 1))
Cloning https://gitlab.organization.com/group-user/repo.git   to /tmp/pip-q4puhrby-build
fatal: could not read Username for 'https://gitlab.organization.com':  No such device or address
Command "git clone -q https://gitlab.organization.com/group-user/repo.git   /tmp/pip-q4puhrby-build" failed with error code 128 in None

Any idea on that?

[jpramos123]

Well, I tried what @GrahamDumpleton suggested and I still not able to clone the module from a private repository.

I created the ssh key with the following command: ssh-keygen -t rsa -C "jramos@organization.com" and updated the Private Key to OpenShift platform.

The build is able to clone my git project (which is in the same domain of the python module).

I created a ssh secret and assigned it to the Source Secret as well to the Build Secret.

My requirements.txt file is as follows:

git+ssh://git@gitlab.organization.com/group-user/repo.git
certifi==2018.1.18
chardet==3.0.4
click==6.7
 ...

I'm receiving the following error:

Collecting git+ssh://git@gitlab.organization.com/group-user/repo.git (from -r requirements.txt (line 1))
Cloning ssh://git@gitlab.organization.com/group-user/repo.git to /tmp/pip-rgq4xum2-build
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Waiting for the feedback!

Thanks a lot!

[richardqa]

Well, I tried what @GrahamDumpleton suggested and I still not able to clone the module from a private repository.

I created the ssh key with the following command: ssh-keygen -t rsa -C "jramos@organization.com" and updated the Private Key to OpenShift platform.

The build is able to clone my git project (which is in the same domain of the python module).

I created a ssh secret and assigned it to the Source Secret as well to the Build Secret.

My requirements.txt file is as follows:

git+ssh://git@gitlab.organization.com/group-user/repo.git
certifi==2018.1.18
chardet==3.0.4
click==6.7
 ...

I'm receiving the following error:

Collecting git+ssh://git@gitlab.organization.com/group-user/repo.git (from -r requirements.txt (line 1))
Cloning ssh://git@gitlab.organization.com/group-user/repo.git to /tmp/pip-rgq4xum2-build
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Waiting for the feedback!

Thanks a lot!

HI,
I didn't get it yet. The only solution that I got: ... in the moment to create your image with Dockerfile you attach your SSH public/private key, validate the keys with ssh-add. Thus, when you launch the App (oc new-app) the container already know these keys ssh and now It is able to install your requirements.txt.

• In your repo git+ssh://git@gitlab.organization.com/group-user/repo.git you should add your SSH keys.

On the other hand, I tried to get it using some resources of OpenShift but i hadn't sucessful

Regards,
Richard

[GrahamDumpleton]

Maybe the issue here is my use of the term "build secret" and what is meant by that. Strictly speaking, it is a build input secret. So you need to add the private SSH key and mount it/copy it into the .ssh directory with the file having appropriate permissions so it isn't rejected. If have multiple keys, would need to have a .ssh/config file to map which to use. If have to create/copy stuff as assemble, remember to remove when done else they will stay in the image.

Only article that comes close to explaining it is:

• https://medium.com/@aramalipoor/how-to-use-input-secrets-to-clone-multiple-repositories-in-openshifts-buildconfig-46cd4ced8b2c

What steps are you trying to use as is?

[GrahamDumpleton]

Worth highlighting is that the more complicated scenario of needing to prepare .ssh directory is only needed for accessing private repositories from requirements.txt. That is, when needing to do checkout as part of the build steps themselves. If you are only trying to use private Git repository as input for source/docker built itself, follow:

• https://cookbook.openshift.org/building-and-deploying-from-source/how-can-i-build-from-a-private-repository-on-github.html
• https://cookbook.openshift.org/building-and-deploying-from-source/how-can-i-build-from-a-private-repository-on-gitlab.html
  https://cookbook.openshift.org/building-and-deploying-from-source/how-can-i-build-from-a-private-repository-on-bitbucket.html

[frenzymadness]

Is this issue still relevant or did mentioned articles solved your problem?

[GrahamDumpleton]

Leave it open for now as a prompt to create a proper cookbook recipe about it as the solution is non obvious. After documenting it, we might find a need to add some helper code in the assemble script to make it easier.

[ihadgraft]

I got this working at one point when doing some initial testing and fortunately documented some things. Here's the solution I was able to work out:

MY_GIT_HOST=git-server.com

ssh-keyscan "$MY_GIT_HOST" > ~/git_known_hosts
echo 'ssh -i /path/to/private-key -o UserKnownHostsFile=/opt/app-root/src/git_known_hosts "$@"' > custom_ssh
chmod +x custom_ssh

export GIT_SSH=/opt/app-root/src/custom_ssh

After setting this up, git will use your custom command instead of ssh directly, i.e. any subsequent git calls will just work.

[frenzymadness]

Hi @ihadgraft . Would you mind to transfer your last comment to a piece of documentation?

[tumregels]

Try this. Should work.

$ export GIT_SSH_COMMAND="ssh -i ~/.ssh/private_key_name"
$ pip install -r requirements.txt