micropipenv as an alternative for pipenv for S2I images

[frenzymadness]

It seems that micropipenv might be a good alternative for pipenv in S2I images because it's a light piece of software and pipenv seems to be unmaintained for a while.

The main problem is that micropipenv does not support Python 2 which means that we can:

• either make it Python 2 compatible which does not seems to be an easy task (WIP: Python2 support thoth-station/micropipenv#18) and its maintainers obviously don't want to support EOL Python in the new fresh software
• or use it only for Python 3 images which would make our images different in another aspect and can possibly cause problems
• or install Python 3 on Python 2 images and create a virtual environment just for micropipenv which would make our images way bigger

We discussed this today on SCL meeting and decided that we'll keep this open. There are no issues related to older or buggy version of Pipenv in S2I containers. We can wait for a while and when any issue comes we can solve them by using micropipenv only for Python 3 or any other way we'll discover in the meantime.

[fridex]

We try to approach few issues with micropipenv we observed when using pipenv in AICoE:

Installing toolchain that is responsible for building the application during the actual build sounds like a not good approach. The s2i container images should come with all the packages that are needed to install an application.

A use case for that could be the usage of Python s2i inside a company that has restricted access to the outside world and allows to install only trusted software from internal Python indexes (in that case Python s2i fails).

Also, Pipenv version is not pinned - if a new buggy Pipenv gets released, we all get blocked as Python s2i always uses the latest release - I think we experienced this before #290

We will most probably continue using micropipenv as it fits our ongoing efforts - see #367, pypa/pipenv#4059, thoth-station/s2i-thoth#39, and #367 (comment) It would be great if we could combine efforts on this front and standardize Python s2i build process.

[frenzymadness]

Installing toolchain that is responsible for building the application during the actual build sounds like a not good approach. The s2i container images should come with all the packages that are needed to install an application.

Do you mean by this that pipenv should be installed in some specific version during the build od s2i-python container and not during the application deployment? We can implement something like this or some other way to pin the version of pipenv. Unfortunately, pipenv is not an RPM so we have limited possibilities here - mainly we cannot automatically rebuild the images when pipenv changes which we can do for RPMs.

A use case for that could be the usage of Python s2i inside a company that has restricted access to the outside world and allows to install only trusted software from internal Python indexes (in that case Python s2i fails).

You can configure pip inside a container to use your own index. Just add something like PIP_CONFIG_FILE=./pip.conf into .s2i/environment and then something like

[global]
index-url = https://example.org/PYPI

into pip.conf.

Also, Pipenv version is not pinned - if a new buggy Pipenv gets released, we all get blocked as Python s2i always uses the latest release - I think we experienced this before #290

As I said, it might be a good idea to pin the current version of pipenv and implement the same env variable we have for pip UPGRADE_PIP_TO_LATEST also for pipenv.

We will most probably continue using micropipenv as it fits our ongoing efforts - see #367, pypa/pipenv#4059, thoth-station/s2i-thoth#39, and #367 (comment) It would be great if we could combine efforts on this front and standardize Python s2i build process.

I'd like to do it and there are many more initiatives around python packaging ecosystem these days but I don't know which way to go right now.

[fridex]

Installing toolchain that is responsible for building the application during the actual build sounds like a not good approach. The s2i container images should come with all the packages that are needed to install an application.

Do you mean by this that pipenv should be installed in some specific version during the build od s2i-python container and not during the application deployment? We can implement something like this or some other way to pin the version of pipenv. Unfortunately, pipenv is not an RPM so we have limited possibilities here - mainly we cannot automatically rebuild the images when pipenv changes which we can do for RPMs.

Yes, it should be installed during s2i python container build in a specific (tested) version. The same reason why pip is not installed on the application build. It will resolve #278

A use case for that could be the usage of Python s2i inside a company that has restricted access to the outside world and allows to install only trusted software from internal Python indexes (in that case Python s2i fails).

You can configure pip inside a container to use your own index. Just add something like PIP_CONFIG_FILE=./pip.conf into .s2i/environment and then something like

[global]
index-url = https://example.org/PYPI

into pip.conf.

I think this will apply to pip - so installing application stack from requirements.txt. Pipenv has its own way of specifying Python indexes. The invocation of pip is different in the case of Pipenv, it will ignore this configuration file.

Also, Pipenv version is not pinned - if a new buggy Pipenv gets released, we all get blocked as Python s2i always uses the latest release - I think we experienced this before #290

As I said, it might be a good idea to pin the current version of pipenv and implement the same env variable we have for pip UPGRADE_PIP_TO_LATEST also for pipenv.

+1

We will most probably continue using micropipenv as it fits our ongoing efforts - see #367, pypa/pipenv#4059, thoth-station/s2i-thoth#39, and #367 (comment) It would be great if we could combine efforts on this front and standardize Python s2i build process.

I'd like to do it and there are many more initiatives around python packaging ecosystem these days but I don't know which way to go right now.

Indeed, it seems to be a little confusing in the whole Python community - is Pipenv alive pypa/pipenv#3369 ?, is Poetry the alternative to go further?

It looks like Pipenv is still the recommended tool by PyPA and Pipfile the requirements format they are willing to support. I understand the back-off for this and the impact of this change. As mentioned earlier, we will try to continue in AICoE with micropipenv efforts as it long-term supports our vision for Python software stacks. If you plan to make changes in the Python s2i, please let us know - we are happy to cooperate. We would be also happy for your input to micropipenv and thoth s2i Python as we are trying to create the same.

[frenzymadness]

Yes, it should be installed during s2i python container build in a specific (tested) version. The same reason why pip is not installed on the application build. It will resolve #278

You are right, but pip is distributed via RPM which means that when we release new pip in Centos or RHEL, the images are refreshed automatically. On the other hand, pipenv is not packaged and I don't think include it and all its dependencies in the image is a good idea.

Pinning of the latest stable release of pipenv and implementing a mechanism to update it to the latest version is a way to go now IMHO.

I think this will apply to pip - so installing application stack from requirements.txt. Pipenv has its own way of specifying Python indexes. The invocation of pip is different in the case of Pipenv, it will ignore this configuration file.

My point was that there is a way how to configure pip/pipenv if you need it. But I understand that the installation of pipenv during an application deploy might complicate things.

It looks like Pipenv is still the recommended tool by PyPA and Pipfile the requirements format they are willing to support. I understand the back-off for this and the impact of this change. As mentioned earlier, we will try to continue in AICoE with micropipenv efforts as it long-term supports our vision for Python software stacks. If you plan to make changes in the Python s2i, please let us know - we are happy to cooperate. We would be also happy for your input to micropipenv and thoth s2i Python as we are trying to create the same.

The recommendations might change soon: pypa/packaging.python.org#701
But yes, we should prepare for the incoming storm in pipenv development and micropipenv is IMO the best alternative for S2I images. We can schedule a meeting and discuss a long term plan for micropipenv if you want.

[fridex]

Yes, it should be installed during s2i python container build in a specific (tested) version. The same reason why pip is not installed on the application build. It will resolve #278

You are right, but pip is distributed via RPM which means that when we release new pip in Centos or RHEL, the images are refreshed automatically. On the other hand, pipenv is not packaged and I don't think include it and all its dependencies in the image is a good idea.

Pinning of the latest stable release of pipenv and implementing a mechanism to update it to the latest version is a way to go now IMHO.

This introduces the issue you mentioned above - including pipenv in the s2i container image looks like too much - especially considering its vendored dependencies - a lot of software required just to install packages well specified in a JSON file (no need to create virtual environment in s2i, no need to have full blown resolver, all the fancy progress bars, ...). That was one of the ideas that drove micropipenv development.

It looks like Pipenv is still the recommended tool by PyPA and Pipfile the requirements format they are willing to support. I understand the back-off for this and the impact of this change. As mentioned earlier, we will try to continue in AICoE with micropipenv efforts as it long-term supports our vision for Python software stacks. If you plan to make changes in the Python s2i, please let us know - we are happy to cooperate. We would be also happy for your input to micropipenv and thoth s2i Python as we are trying to create the same.

The recommendations might change soon: pypa/packaging.python.org#701
But yes, we should prepare for the incoming storm in pipenv development and micropipenv is IMO the best alternative for S2I images. We can schedule a meeting and discuss a long term plan for micropipenv if you want.

+1, that would be great. Let's also see how the PyPA and community will react to this storm.

[frenzymadness]

The current status is:

• micropipenv can be used in python-container with ENABLE_MICROPIPENV (from PyPI)
• pin a pipenv version will soon be possible [CRITICAL] Pipenv-related improvements #387

Our plan is to package micropipenv for RHEL so it will be always available as a stable option in all our containers in case something happen with pipenv itself. It also brings more attention to micropipenv so we might use it as a default in the future.

[frenzymadness]

Update: micropipenv will be available in RHEL 8.4.0. Then, we can make it available by default in all our containers and optionally replaced by pipenv per user request.