Pip dependency resolution conflict with requirements.txt

[rossbar]

The first step of make html is pip install -q -r requirements.txt, but unfortunately it is not currently possible to install the dependencies with pip due to conflicts in pip's dependency resolver. The failure mode is particularly frustrating because it sends pip down a path where it tries to download very old versions of e.g. pytest and will only fail after these fail to build.

I've encountered this problem before with the executablebooks project (executablebooks/MyST-NB#333, executablebooks/MyST-NB#289) so I suspected jupyter-book/jupytext as the culprit. Indeed, if jupyterbook and jupytext are removed from the requirements file and installed separately, the failure is resolved.

[hmaarrfk]

I think this is a pretty big challenge.

It seems to me that sphinx 4.0 has been quite controversial.

• jupyter-book doesn't want to use it.
• furo needs it

[hmaarrfk]

I'm trying to run things with:

diff --git a/Makefile b/Makefile
index 30b70b8..701f261 100644
--- a/Makefile
+++ b/Makefile
@@ -6,7 +6,7 @@ LESSONS_DIR = lessons
 GENERATED_LESSONS_DIR = book/lessons
 
 _requirements.installed:
-       pip install -q -r requirements.txt
+       pip install -r requirements.txt
        touch _requirements.installed
 
 MARKDOWNS = $(wildcard $(LESSONS_DIR)/*.md)
diff --git a/requirements.txt b/requirements.txt
index ea8af47..e7b464c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,15 +1,12 @@
-scikit-image[data] >= 0.18
-numpy >= 1.12
-scipy >= 1.0
-matplotlib >= 2.1
-notebook >= 4.0
-scikit-learn >= 0.18
-jupyter-book >= 0.10.2
-napari[all]
+scikit-image[data] >= 0.18.3
+matplotlib >=3.4.3
+notebook >=6.4.3
+scikit-learn >=0.24.2
+jupyter-book >=0.11.3
+napari[all] >=0.4.11
 jupytext >=1.10.3
 sphinx_autodoc_typehints>=1.11.0
-ghp-import
-pytest
-pytest-qt
-pooch
-furo
\ No newline at end of file
+ghp-import >=2.0.1
+pytest >=6.0.1
+pytest-qt >=4.0.1
+pooch >=1.5.1

but i get an error called:

touch _requirements.installed
make[1]: *** book: No such file or directory.  Stop.
make: *** [Makefile:37: html] Error 2

[hmaarrfk]

I presume that this is the focus of
#59

[rossbar]

I presume that this is the focus of #59

To some degree - splitting up the "build" dependencies (e.g. the executablebook project deps) and the dependencies needed to actually run the notebooks can help IME, but it still might require some tweaking to get everything installed.

[hmaarrfk]

My guess is that something happened in pip to update their "solver" which is causing these headaches.

[rossbar]

Yeah there are multiple potential failure modes. Changes to the pip resolver have caused problems recently, see e.g. pypa/pip#10201, but there have also been dependency conflicts between e.g. jupytext and myst-nb (e.g. executablebooks/MyST-NB#333)

[hmaarrfk]

is there a solution that includes downgrading pip?

[jni]

@hmaarrfk

I think this is a pretty big challenge.

It seems to me that sphinx 4.0 has been quite controversial.

• jupyter-book doesn't want to use it.
• furo needs it

Sorry for the off-topic post, but this is such an ironically relevant story for the scikit-image 1.0 transition! 😂 I'm very glad the discussion settled on skimage2.

[hmaarrfk]

issue to follow

jupyter-book/jupyter-book#1438

[notatallshaw]

FYI I tested the requirements.txt with the changes I am proposing here: pypa/pip#10479

And it didn't get stuck backtracking but very quickly gives the error:

ERROR: Cannot install -r .\requirements.txt (line 10), -r .\requirements.txt (line 15) and -r .\requirements.txt (line 7) because these package versions have conflicting dependencies.

The conflict is caused by:
    jupyter-book 0.10.2 depends on sphinx<4 and >=2
    sphinx-autodoc-typehints 1.11.0 depends on Sphinx>=3.0
    furo 2021.9.8 depends on sphinx~=4.0
    jupyter-book 0.10.2 depends on sphinx<4 and >=2
    sphinx-autodoc-typehints 1.11.0 depends on Sphinx>=3.0
    furo 2021.8.31 depends on sphinx~=4.0