Switch build system to pixi and rattler-build#398
Open
Conversation
Contributor
Author
|
I decided to add a feature for all downstream OpenBioSim dependencies to avoid issues with dependency resolution in future. I've not tested this locally, so might need to make it less restrictive tomorrow if there are issues. (The standard BSS deps approach works.) |
cec1a23 to
77dee9a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements a major overhaul of the Sire build system by modernising the software stack to make use of
pixiandrattler-build. This resolves multiple issues with our CI, i.e. we once again have a working Windows build and environment resolution no longer hangs for Python 3.10 builds on Linux and macOS.Our build is made more complex as result of Sire being the foundational package for other packges within the OpenBioSim ecosystem. This leads to a complex dependency resolution when the Sire build is treated as the "host" environment for other tools and features. Previously, this was handled via a set of requirements files for specific features, which were resolved and injected into the conda recipe at build time. This approach doesn't work for
rattler, so we instead use thepixi.tomlas the "source-of-truth", where features and their required dependencies are stated explicitly. This can be used withpixito install an development environment for source install, or for arattlerbuild by first runningpython actions/generate_recipe.pyto generate the recipe from the information in thepixi.tomlfile.Following refactoring, I have discovered that the
SireGemmiinterface is currently broken. Going back through the CI logs it is apparent that this hasn't been tested for some time, sincegemmiwasn't correctly detected in the test environment, so the appropriate tests were skipped. The issue is due to an issue with the Boost::Python / pybind11 interface, which I have been unable to resolve locally. For now, I have commented out theSireGemmibuild in the appropriate CMake file. (gemmihas always been extremely brittle and both Boost:Python and pybind11 are unstable in the parts of the API required for the interface.)A local build on my workstation discovered that OpenMM requires a more recent version of
sysroot_linux-64in the host section of its recipe (2.34 rather than 2.17). This means that we need to match this when doing a source build, since we are effectively creating a "host" environment. This is handled via thepixi.toml. (The standardrattlerbuild is unaffected.)Note that there are a few wrinkles with using
rattler-build, which will be discussed further in my PRs for our other packages. The main issue is that it only does a shallow clone, so standard Python versioning tools, such asversioningitare unable to access the information needed at build time. This can be resolved by installing from a path rather than a git URL, but this also leads to other issues. (Everything is can be fixed, but it's painful that this is needed for standard pure-Python packages.) Usingrattleralso exposes some packaging issues with some of our BioSimSpace dependencies. This is because it reads dependency constraints from thepipmetadata as well as the conda recipe, so if a package specifies a constraint in the pip one but not the conda one, then the dependency resolution will fail. This is true fornglview, where it requiresnumpy<2.3, but this isn't added to the conda recipe. (They are currently building a new version with support for 2.4+, so this will be resolved soon.)Closes #268
Closes #339
Closes #392
Closes #396
develinto this branch before issuing this pull request (e.g. by runninggit pull origin devel): [y]