Can't install extensions in JupyterLab due to pip error #210

New issue

Closed

opened 2023-10-10 16:45:41 +00:00 by a · 1 comment

a commented

2023-10-10 16:45:41 +00:00

Owner

Recently, Python's PEP 668 has started to take effect in a lot of distros, so pip install no longer works. The problem it's trying to solve is that pip install will use globally installed (using pacman) Python packages as dependencies, so when those globally installed packages are uninstalled or upgraded, you could and probably will get breakage.

Some possible solutions:

Do things the old way: There are several ways to do that listed here (you probably want to modify ~/.config/pip/pip.conf), but this can and probably will lead to breakage (such as #125, #198, #202). If your extension breaks, you'll just have to reinstall it.
Create a venv and install JupyterLab inside. This won't lead to breakage, but hub.exozy.me won't use the venv's JupyterLab but rather will use the globally installed JupyterLab. This means that if you start JupyterLab at hub.exozy.me, pip still won't work, and you would have to SSH in, source the venv, and start JupyterLab manually instead.
Use Conda. Wait, this is a joke. Conda will probably give you more pain.
Wait until JupyterLab upstream finds a way to solve this issue. The PEP 668 rollout has just started, so eventually people will start complaining and JupyterLab might find a better way to solve this.

TL;DR Python packaging sucks and PEP 668, which is trying to make it suck less, has managed to make it suck in other ways. Also, all hell breaks loose after a Python major version upgrade (Python 3.12 is sooner than you think). If you want JupyterLab extensions to work right now, then modify ~/.config/pip/pip.conf as suggested above and cross your fingers that nothing breaks over the next few months.

Recently, Python's [PEP 668](https://peps.python.org/pep-0668/) has started to take effect in a lot of distros, so `pip install` no longer works. The problem it's trying to solve is that `pip install` will use globally installed (using pacman) Python packages as dependencies, so when those globally installed packages are uninstalled or upgraded, you could and probably will get breakage. Some possible solutions: - Do things the old way: There are [several ways to do that listed here](https://stackoverflow.com/a/75722775) (you probably want to modify `~/.config/pip/pip.conf`), but this can and probably will lead to breakage (such as #125, #198, #202). If your extension breaks, you'll just have to reinstall it. - Create a venv and install JupyterLab inside. This won't lead to breakage, but hub.exozy.me won't use the venv's JupyterLab but rather will use the globally installed JupyterLab. This means that if you start JupyterLab at hub.exozy.me, pip still won't work, and you would have to SSH in, source the venv, and start JupyterLab manually instead. - Use Conda. Wait, this is a joke. Conda will probably give you more pain. - Wait until JupyterLab upstream finds a way to solve this issue. The PEP 668 rollout has just started, so eventually people will start complaining and JupyterLab might find a better way to solve this. TL;DR Python packaging sucks and PEP 668, which is trying to make it suck less, has managed to make it suck in other ways. Also, all hell breaks loose after a Python major version upgrade (Python 3.12 is sooner than you think). If you want JupyterLab extensions to work **right now**, then modify `~/.config/pip/pip.conf` as suggested above and cross your fingers that nothing breaks over the next few months.

a added the

bug

label 2023-10-10 16:45:41 +00:00

a commented

2023-10-11 15:56:01 +00:00

Author

Owner

The core of this problem is that JupyterLab assumes it's running in an environment where it has write access to Python packages, but this isn't the case for distro-installed JupyterLab.

Thus, I think the only sane solution for this is to not support installing extensions on hub.exozy.me. If you want to use JupyterLab extensions, you should make your own venv for that.

I also posted on Matrix:

Actually I have a new idea: if you make a script on your path called jupyterhub-singleuser it might be possible to get JupyterHub to run that instead of the default spawner.

This turns out to not work because in the JupyterHub code, it actually uses the root user's PATH when spawning the process and then setuids it to the correct user, so I would need to write a script called jupyterhub-singleuser and put it on the root user's PATH that gets the user's default shell and runs the jupyterhub-singleuser command using the user's PATH and falls back to /usr/bin/jupyterhub-singleuser. (Actually, this might be wrong and perhaps what actually happens is that JupyterHub spawns the process using Bash. Anyways it's complicated and IMHO not worth spending more time on. Just make a venv)

The core of this problem is that JupyterLab assumes it's running in an environment where it has write access to Python packages, but this isn't the case for distro-installed JupyterLab. Thus, I think the only sane solution for this is to **not support installing extensions on hub.exozy.me**. If you want to use JupyterLab extensions, **you should make your own venv** for that. I also posted on Matrix: > Actually I have a new idea: if you make a script on your path called `jupyterhub-singleuser` it might be possible to get JupyterHub to run that instead of the default spawner. This turns out to not work because in the [JupyterHub code](https://github.com/jupyterhub/jupyterhub/blob/70717dc9ab163b568c51f70369374d35b91ba56f/jupyterhub/spawner.py#L1711), it actually uses the root user's PATH when spawning the process and then setuids it to the correct user, so I would need to write a script called `jupyterhub-singleuser` and put it on the root user's PATH that gets the user's default shell and runs the `jupyterhub-singleuser` command using the user's PATH and falls back to `/usr/bin/jupyterhub-singleuser`. (Actually, this might be wrong and perhaps what actually happens is that JupyterHub spawns the process using Bash. Anyways it's complicated and IMHO not worth spending more time on. Just make a venv)

a closed this issue

2023-10-11 15:56:01 +00:00