In #682 we discovered that conda doesn't provide a fully isolated execution environment, and instead allows python dependencies installed in ~/.local to be picked up. Unfortunately these seems to be preferred to conda's own packages. Here's a simple test script:
import crontab from pprint import pprint as print import sys print(sys.path) print(crontab.__file__)
When a conda environment isn't active, run pip install --user crontab. Then conda activate openwpm. The environment variable PYTHONNOUSERSITE=True will control whether python will pick up the users local packages.
(openwpm) steven@apollo:~/research/OpenWPM/test$ python test.py
['/home/steven/research/OpenWPM/test',
'/home/steven/miniconda3/envs/openwpm/lib/python38.zip',
'/home/steven/miniconda3/envs/openwpm/lib/python3.8',
'/home/steven/miniconda3/envs/openwpm/lib/python3.8/lib-dynload',
'/home/steven/.local/lib/python3.8/site-packages',
'/home/steven/miniconda3/envs/openwpm/lib/python3.8/site-packages']
'/home/steven/.local/lib/python3.8/site-packages/crontab/__init__.py'
(openwpm) steven@apollo:~/research/OpenWPM/test$ PYTHONNOUSERSITE=True python test.py
['/home/steven/research/OpenWPM/test',
'/home/steven/miniconda3/envs/openwpm/lib/python38.zip',
'/home/steven/miniconda3/envs/openwpm/lib/python3.8',
'/home/steven/miniconda3/envs/openwpm/lib/python3.8/lib-dynload',
'/home/steven/miniconda3/envs/openwpm/lib/python3.8/site-packages']
'/home/steven/miniconda3/envs/openwpm/lib/python3.8/site-packages/crontab/__init__.py'
I don't know the exact internal logic python uses to order locations in sys.path. From PEP 370 it sounds like this is the intended order. This happens both with packages that are installed directly from conda's package repository and those installed via pip within conda.
We shouldn't assume that users don't have local packages installed. It's the default flow for pip when global packages aren't writable (i.e., most linux users?). While virtual environments are best practice, I'm sure many users will pip install a few packages to run a simple script every once in a while (or are human and don't always follow best practices). In fact, OpenWPM's previous install script would intentionally install packages into ~/.local (which in retrospect was a mistake) so this may impact a lot of our long-term users.
These issues aren't easy to debug either. As @vringar experienced when he hit the issue in: #614 (comment), conda list shows the conda packages so it's not immediately clear to the user.
When python dependencies are listed in a requirements.txt file, users can choose to run in a virtualenv or not, and if they choose not to they're taking things into their own hands. But that's not possible with environment.yaml, so it's on us to make sure things work.
This is less clear. This feels like something that should be fixed in conda, but reading through the discussions in conda-forge/python-feedstock#171, conda/conda#7173, and conda/conda#448 it seems like this is intended design despite the conda folks also billing conda as an alternate to pip and venv.
We could try to export PYTHONNOUSERSITE=True in a conda helper script, but as Sarah points out this may have some issues #682 (comment).
We could also try to use both virtualenv and conda, which is suggested by a conda developer here conda/conda#9788 (comment), but we'd have to refactor how we manage dependencies. Maybe this is the best long-term option.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3