Python Packages and Virtual Environments
If you need a python package that isn't installed on Kodiak, contact HPRCS staff and just request it. Usually we can install what you need but if there are compatibility issues or if you need a different version, you may need to install it yourself using the instructions below.
Before trying to install a python package (or requesting that one be installed) make sure that you have loaded a python module and are not trying to run one of the system default copies of python. You can check this with
which python (or
which python3). If it returns "/usr/bin/python(3)" then you need to load either python/2.7.10, python/2.7.14, or python/3.7.2. Then run
pip list and see if the python package is already installed.
Note: If you require the "tensorflow" package, do not
pip install it, either as a "--user installation" or within a virtual environment as described below. We have found that the various prebuilt versions that can be downloaded often have compatibility issues with Kodiak. Instead, we have built tensorflow from source and installed that on Kodiak. To use it, load one of the tensorflow/1.13.1 modules (cpu or gpu). This installation will use its own internal copy of python 3.7.2 so if you need packages that aren't installed in the "tensorflow python" you may still need to create a virtual environment or otherwise install packages using the instructions below. (But request that it be installed first...)
You can create your own private "copy" of python, i.e., a virtual environment, in which you can install specific packages that aren't installed with the system version. You will need to use a version of python that includes venv or virtualenv and pip (python's package installer).
There are two ways to create a virtual environment. The current, officially blessed, method is to use "venv". This is supported in python 3.5 and later. The older method is "virtualenv" and is required for python 2.x. Although virtualenv still works with python 3.x, you should use venv.
Creating a Virtual Environment With "venv"
This requires python 3.5 or later. On Kodiak, there are modules for two versions of python 3.7.2. The module for the regular version is python/3.7.2 which has several installed packages. There is also a module, python/3.7.2-virtualenv, which is a minimal installation, i.e., no packages other than pip. The "3.7.2-virtualenv" module is the one you will want to use.
Note: It is possible, when creating your virtual environment, to include access to the system packages installed in the regular version of python if you want them. To do so, load the python/3.7.2 module instead of python/3.7.2-virtualenv, and add the
--system-site-packages venv option when creating your virtual environment. This document assumes you want to create a virtual environment without any of the default packages.
First, load the python/3.7.2-virtualenv module. You may want/need to
module purge first to make sure no other python modules are loaded, just in case.
module listNo Modulefiles Currently Loaded. $
module avail python------------------------ /usr/local/Modules/modulefiles ------------------------ python/2.7.10(default) python/3.6.6 python/3.7.2 python/2.7.14 python/3.7.0 python/3.7.2-virtualenv $
module load python/3.7.2-virtualenv
Note that python 2.7.10 is still the default version, so if you just
module load python you will get that one instead and venv will not work. Also, when using python 3.x, be sure to run python3 and not python. The latter may run the system default version of python which is not what you want.
You can test this by using the
which command that will show you which copy of python3 will get run:
which python3$ /usr/local/python/3.7.2-virtualenv/bin/python3
So to create a virtual environment, cd to the directory where you want your python directory to go. Here, we'll put it in /home/bobby/projects but you can put it wherever you want. You'll need to create the directory if it doesn't exist yet. Note that this is not the virtual environment directory itself, but is the parent directory where the virtual environment directory will be located.
venv to create the virtual environment. We'll create one and call it "my-python" but you can name it whatever you want. This will create a directory called "my-python" inside of the projects directory that will be the virtual environment copy of python.
python3 -m venv my-python$
ls my-pythonbin include lib lib64 pyvenv.cfg $
ls my-python/binactivate activate.fish easy_install-3.7 pip3 python activate.csh easy_install pip pip3.7 python3
To switch to this version of python, you'll run the
activate script within the my-python/bin directory. Note that you have to
source this script and not just run it so that it will modify your current session.
source /home/bobby/projects/my-python/bin/activate(my-python) $
You should see the current virtual environment (my-python) in your prompt. Now make sure that you really are using this virtual environment with the
which command again. You should see the copy of python3 within my-python/bin directory and not /usr/local/python.
At this point, you should be able to install individual python packages with
pip3). If you run
pip3 list you will see which packages were copied by the venv command. There should only be 2 right now. (You could also
pip3 list -v to show the paths to the packages if you want to confirm their location.)
pip3 listPackage Version ---------- ------- pip 18.1 setuptools 40.6.2 You are using pip version 18.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
If you get the message about upgrading pip, just run the command above.
pip3 install --upgrade pipCollecting pip Using cached .../pip-19.1.1-py2.py3-none-any.whl Installing collected packages: pip Found existing installation: pip 18.1 Uninstalling pip-18.1: Successfully uninstalled pip-18.1 Successfully installed pip-19.1.1
Now, to install a package, just
pip install [package-name].
pip3 install numpyCollecting numpy Downloading https://files.pythonhosted.org/.../numpy-1.16.4_x86_64.whl (17.3MB) |████████████████████████████████| 17.3MB 10.5MB/s Installing collected packages: numpy Successfully installed numpy-1.16.4
pip3 list again and hopefully see that numpy got installed.
pip3 listPackage Version ---------- ------- numpy 1.16.4 pip 19.1.1 setuptools 40.6.2
The virtual environment python copy will automagically get deactivated when you log out or your qsub-ed job exits. But if you want to stay logged in and do other, "non-my-python" related, python work, you can explicitly deactivate it with the
deactivate command. You can
which python3 to verify that it's no longer using your copy.
In your qsub-ed jobs that need to use the "my-python" version of python, be sure to add
source /home/bobby/projects/my-python/bin/activate to your script.
Creating a Virtual Environment With "virtualenv"
Coming real soon now...