Add option to use 64-bit rapidhash for MacOS build by msricher · Pull Request #104 · theochem/PyCI

msricher · 2025-03-06T14:18:00Z

Uses 64-bit rapidhash implementation instead of 128-bit spookyhash. For playing around on a Mac laptop, this should be fine.

Sets up spookyhash or rapidhash based on whether the command uname -s outputs 'Darwin' (the MacOS kernel).

Uses 64-bit rapidhash implementation instead of 128-bit spookyhash. For playing around on a Mac laptop, this should be fine. Sets up spookyhash or rapidhash based on whether the command `uname -s` outputs 'Darwin' (the MacOS kernal).

FanwangM · 2025-03-06T16:54:31Z

Thanks for this improvement. I have tried to install the package in a MacOS which ends up with the following errors,

In [1]: import pyci
-------------------------------------------------------------------------
ModuleNotFoundError                     Traceback (most recent call last)
Cell In[1], line 1
----> 1 import pyci

File /xxxx/xxxxx/tmp_py311/lib/python3.11/site-packages/pyci/__init__.py:18
      1 # This file is part of PyCI.
      2 #
      3 # PyCI is free software: you can redistribute it and/or modify it under
   (...)
     13 # You should have received a copy of the GNU General Public License
     14 # along with PyCI. If not, see <http://www.gnu.org/licenses/>.
     16 r"""PyCI module."""
---> 18 from pyci._pyci import __version__, c_long, c_ulong, c_double
     19 from pyci._pyci import secondquant_op, wavefunction, one_spin_wfn, two_spin_wfn
     20 from pyci._pyci import doci_wfn, fullci_wfn, genci_wfn, sparse_op

ModuleNotFoundError: No module named 'pyci._pyci'

The package was installed with pip install . after cloning the package. @msricher

msricher · 2025-03-06T16:56:24Z

Ok, this seems to be a problem with file paths. It must need to be installed to different directory, or with a different file extension, for it to work. Although it seems to compile now, which is the hard part.

msricher · 2025-03-06T16:57:20Z

Can you try renaming the .so file to .dylib and trying again?

FanwangM · 2025-03-07T01:42:01Z

I don't fully understand what you mean by It must need to be installed in a different directory or with a different file extension for it to work. because I installed it with pip install ., which actually installed the pyci in the virtual environment.

Following your suggestions of renaming .so to .dylib, but I was not able to find any .so file from the source or the installed codes. Can you provide more explanations? Thanks.

@msricher

msricher · 2025-03-07T10:07:13Z

The instruction say to run make before pip install ., which compiles the C extension module, pyci.so (or similar name). Can you do that and then try renaming it from .so to .dylib?

FanwangM · 2025-03-07T18:18:34Z

yes, I just did that. Currently, I got errors when running make. I tried to install rapidhash manually, but didn't figure out a solution. Any suggestions on how to address the issue? Thanks.

c++ -std=c++14 -Wall -Wextra -pipe -O3 -fPIC -flto=auto -fvisibility=hidden -pthread -Ipyci/include -Ixxxx/tmp_py311/include/python3.11 -Ixxxx/tmp_py311/lib/python3.11/site-packages/numpy/_core/include -Ideps/eigen -Ideps/spectra/include -Ideps/parallel-hashmap -Ideps/pybind11/include -undefined dynamic_lookup -D_PYCI_VERSION='0.6.1' -D_GIT_BRANCH='rapidhash_macos' -D_BUILD_TIME='2025-03-07 18:15:43' -D_COMPILER_VERSION='Apple clang version 16.0.0 (clang-1600.0.26.6)' -D_USE_RAPIDHASH='1' -c pyci/src/SpookyV2.cpp -o pyci/src/SpookyV2.o
clang++: warning: argument unused during compilation: '-undefined dynamic_lookup' [-Wunused-command-line-argument]
c++ -std=c++14 -Wall -Wextra -pipe -O3 -fPIC -flto=auto -fvisibility=hidden -pthread -Ipyci/include -Ixxxx/tmp_py311/include/python3.11 -Ixxxx/tmp_py311/lib/python3.11/site-packages/numpy/_core/include -Ideps/eigen -Ideps/spectra/include -Ideps/parallel-hashmap -Ideps/pybind11/include -undefined dynamic_lookup -D_PYCI_VERSION='0.6.1' -D_GIT_BRANCH='rapidhash_macos' -D_BUILD_TIME='2025-03-07 18:15:43' -D_COMPILER_VERSION='Apple clang version 16.0.0 (clang-1600.0.26.6)' -D_USE_RAPIDHASH='1' -c pyci/src/ap1rog.cpp -o pyci/src/ap1rog.o
clang++: warning: argument unused during compilation: '-undefined dynamic_lookup' [-Wunused-command-line-argument]
In file included from pyci/src/ap1rog.cpp:16:
pyci/include/pyci.h:45:10: fatal error: 'rapidhash.h' file not found
   45 | #include <rapidhash.h>
      |          ^~~~~~~~~~~~~
1 error generated.
make: *** [pyci/src/ap1rog.o] Error 1

msricher · 2025-03-07T18:33:49Z

Yep, the rapid has directory needs to be specified in the Makefile with the -I flag. I must have missed it. I can fix it tonight.

…

On Fri, Mar 7, 2025, 13:18 Fanwang Meng ***@***.***> wrote: yes, I just did that. Currently, I got errors when running make. I tried to install rapidhash manually, but didn't figure out a solution. Any suggestions on how to address the issue? Thanks. c++ -std=c++14 -Wall -Wextra -pipe -O3 -fPIC -flto=auto -fvisibility=hidden -pthread -Ipyci/include -Ixxxx/tmp_py311/include/python3.11 -Ixxxx/tmp_py311/lib/python3.11/site-packages/numpy/_core/include -Ideps/eigen -Ideps/spectra/include -Ideps/parallel-hashmap -Ideps/pybind11/include -undefined dynamic_lookup -D_PYCI_VERSION='0.6.1' -D_GIT_BRANCH='rapidhash_macos' -D_BUILD_TIME='2025-03-07 18:15:43' -D_COMPILER_VERSION='Apple clang version 16.0.0 (clang-1600.0.26.6)' -D_USE_RAPIDHASH='1' -c pyci/src/SpookyV2.cpp -o pyci/src/SpookyV2.o clang++: warning: argument unused during compilation: '-undefined dynamic_lookup' [-Wunused-command-line-argument] c++ -std=c++14 -Wall -Wextra -pipe -O3 -fPIC -flto=auto -fvisibility=hidden -pthread -Ipyci/include -Ixxxx/tmp_py311/include/python3.11 -Ixxxx/tmp_py311/lib/python3.11/site-packages/numpy/_core/include -Ideps/eigen -Ideps/spectra/include -Ideps/parallel-hashmap -Ideps/pybind11/include -undefined dynamic_lookup -D_PYCI_VERSION='0.6.1' -D_GIT_BRANCH='rapidhash_macos' -D_BUILD_TIME='2025-03-07 18:15:43' -D_COMPILER_VERSION='Apple clang version 16.0.0 (clang-1600.0.26.6)' -D_USE_RAPIDHASH='1' -c pyci/src/ap1rog.cpp -o pyci/src/ap1rog.o clang++: warning: argument unused during compilation: '-undefined dynamic_lookup' [-Wunused-command-line-argument] In file included from pyci/src/ap1rog.cpp:16: pyci/include/pyci.h:45:10: fatal error: 'rapidhash.h' file not found 45 | #include <rapidhash.h> | ^~~~~~~~~~~~~ 1 error generated. make: *** [pyci/src/ap1rog.o] Error 1 — Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLODOS3KU3WGKVDZTYLOZL2THPJDAVCNFSM6AAAAABYOZ7MTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXGEYDKMJQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: FanwangM]*FanwangM* left a comment (theochem/PyCI#104) <#104 (comment)> yes, I just did that. Currently, I got errors when running make. I tried to install rapidhash manually, but didn't figure out a solution. Any suggestions on how to address the issue? Thanks. c++ -std=c++14 -Wall -Wextra -pipe -O3 -fPIC -flto=auto -fvisibility=hidden -pthread -Ipyci/include -Ixxxx/tmp_py311/include/python3.11 -Ixxxx/tmp_py311/lib/python3.11/site-packages/numpy/_core/include -Ideps/eigen -Ideps/spectra/include -Ideps/parallel-hashmap -Ideps/pybind11/include -undefined dynamic_lookup -D_PYCI_VERSION='0.6.1' -D_GIT_BRANCH='rapidhash_macos' -D_BUILD_TIME='2025-03-07 18:15:43' -D_COMPILER_VERSION='Apple clang version 16.0.0 (clang-1600.0.26.6)' -D_USE_RAPIDHASH='1' -c pyci/src/SpookyV2.cpp -o pyci/src/SpookyV2.o clang++: warning: argument unused during compilation: '-undefined dynamic_lookup' [-Wunused-command-line-argument] c++ -std=c++14 -Wall -Wextra -pipe -O3 -fPIC -flto=auto -fvisibility=hidden -pthread -Ipyci/include -Ixxxx/tmp_py311/include/python3.11 -Ixxxx/tmp_py311/lib/python3.11/site-packages/numpy/_core/include -Ideps/eigen -Ideps/spectra/include -Ideps/parallel-hashmap -Ideps/pybind11/include -undefined dynamic_lookup -D_PYCI_VERSION='0.6.1' -D_GIT_BRANCH='rapidhash_macos' -D_BUILD_TIME='2025-03-07 18:15:43' -D_COMPILER_VERSION='Apple clang version 16.0.0 (clang-1600.0.26.6)' -D_USE_RAPIDHASH='1' -c pyci/src/ap1rog.cpp -o pyci/src/ap1rog.o clang++: warning: argument unused during compilation: '-undefined dynamic_lookup' [-Wunused-command-line-argument] In file included from pyci/src/ap1rog.cpp:16: pyci/include/pyci.h:45:10: fatal error: 'rapidhash.h' file not found 45 | #include <rapidhash.h> | ^~~~~~~~~~~~~ 1 error generated. make: *** [pyci/src/ap1rog.o] Error 1 — Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLODOS3KU3WGKVDZTYLOZL2THPJDAVCNFSM6AAAAABYOZ7MTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXGEYDKMJQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

msricher · 2025-03-07T22:41:56Z

OK, it should build now. Can you try the latest commit, and then if not, by renaming the .so->.dylib?

FanwangM · 2025-03-08T01:51:26Z

Still got errors when run make. Here is part of the error messages,

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/sort.h:51:1: note: candidate template ignored: substitution failure [with _AlgPolicy = std::_ClassicAlgPolicy, _Compare = std::__less<void, void> &, _ForwardIterator = std::sort_with_arg::value_iterator_t<double, long>]
   51 | __sort3(_ForwardIterator __x, _ForwardIterator __y, _ForwardIterator __z, _Compare __c) {
      | ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/sort.h:92:7: error: no matching function for call to object of type 'std::__less<void, void>'
   92 |   if (__c(*__x4, *__x3)) {
      |       ^~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/comp.h:40:60: note: candidate template ignored: substitution failure [with _Tp = value_reference_t<double, long>, _Up = value_reference_t<double, long>]
   40 |   _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX14 bool operator()(const _Tp& __lhs, const _Up& __rhs) const {
      |                                                            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
1 warning and 20 errors generated.
make: *** [pyci/src/sparseop.o] Error 1

msricher · 2025-03-08T03:13:51Z

Okay, I can't debug it over messages, I really need access to a Mac or someone with a Mac and proficiency in C++. It might be an issue with the CFLAGS in the Makefile, but I'd have to play around with them. Thanks for trying, though.

msricher · 2025-03-11T14:45:53Z

I've made another commit that might work. @RichRick1 would you also mind trying this?

RichRick1 · 2025-03-14T19:05:19Z

So I basically have the same error that @FanwangM has
I tried to modify the makefile and it seems to be compiled correctly(?). What I mean by that is there were no errors after running make

Here is a make file

# Setup
# -----

# Set C compiler executable
CC ?= cc
export CC

# Set C++ compiler executable
CXX ?= c++
export CXX

# Set Python executable
PYTHON ?= python3

# Set C++ compile flags
CFLAGS := -std=c++14 -Wall -Wextra -pipe -O3
CFLAGS += -fPIC -flto=auto -fvisibility=hidden
CFLAGS += -pthread
CFLAGS += -Ipyci/include

ifneq ($(MAKE_NATIVE),)
CFLAGS += -mavx -mavx2 -msse4.2 -march=native -mtune=native
endif

# Set Python include directories
CFLAGS += $(shell $(PYTHON) tools/python_include_dirs.py)

# Set external projects and their include directories
DEPS := $(addprefix deps/,eigen spectra parallel-hashmap pybind11 rapidhash)
CFLAGS += $(addprefix -Ideps/,eigen spectra/include parallel-hashmap pybind11/include rapidhash)

# This C++ compile flag is needed in order for Macs to find system libraries
ifeq ($(shell uname -s),Darwin)
CFLAGS += -undefined dynamic_lookup
PYCI_EXTENSION := dylib
else
PYCI_EXTENSION := so
endif

# Set PyCI version number
VERSION_MAJOR := 0
VERSION_MINOR := 6
VERSION_PATCH := 1
PYCI_VERSION := $(VERSION_MAJOR).$(VERSION_MINOR).$(VERSION_PATCH)

# Set preprocessor directives
DEFS := -D_PYCI_VERSION='$(PYCI_VERSION)'

DEFS += -D_GIT_BRANCH='$(shell git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "no_git_repo")'
# DEFS += -D_GIT_BRANCH='$(shell git rev-parse --abbrev-ref HEAD)'
DEFS += -D_BUILD_TIME='$(shell date -u +%F\ %T)'
DEFS += -D_COMPILER_VERSION='$(shell $(CXX) --version | head -n 1)'
ifeq ($(shell uname -s),Darwin)
DEFS += -D_USE_RAPIDHASH='1'
endif

# Set objects
OBJECTS := $(patsubst %.cpp,%.o,$(wildcard pyci/src/*.cpp))


# Make commands
# -------------

.PHONY: all
all: pyci/_pyci.$(PYCI_EXTENSION).$(PYCI_VERSION) pyci/_pyci.$(PYCI_EXTENSION).$(VERSION_MAJOR) pyci/_pyci.$(PYCI_EXTENSION)

.PHONY: test
test:
	@set -e; $(PYTHON) -m pytest -sv ./pyci

.PHONY: clean
clean:
	rm -rf pyci/src/*.o pyci/_pyci.$(PYCI_EXTENSION)*

.PHONY: cleandeps
cleandeps:
	rm -rf deps

.PHONY: compile_flags.txt
compile_flags.txt:
	echo "$(CFLAGS)" | tr ' ' '\n' > $(@)


# Make targets
# ------------

pyci/src/%.o: pyci/src/%.cpp pyci/include/pyci.h $(DEPS)
	$(CXX) $(CFLAGS) $(DEFS) -c $(<) -o $(@)

pyci/_pyci.$(PYCI_EXTENSION).$(PYCI_VERSION): $(OBJECTS)
	$(CXX) $(CFLAGS) $(DEFS) -shared $(^) -o $(@)

pyci/_pyci.$(PYCI_EXTENSION).$(VERSION_MAJOR): pyci/_pyci.$(PYCI_EXTENSION).$(PYCI_VERSION)
	ln -sf $(notdir $(<)) $(@)

pyci/_pyci.$(PYCI_EXTENSION): pyci/_pyci.$(PYCI_EXTENSION).$(PYCI_VERSION)
	ln -sf $(notdir $(<)) $(@)

deps/eigen:
	[ -d $@ ] || git clone https://gitlab.com/libeigen/eigen.git $@

deps/spectra:
	[ -d $@ ] || git clone https://github.com/yixuan/spectra.git $@

deps/parallel-hashmap:
	[ -d $@ ] || git clone https://github.com/greg7mdp/parallel-hashmap.git $@

deps/pybind11:
	[ -d $@ ] || git clone https://github.com/pybind/pybind11.git $@

deps/rapidhash:
	[ -d $@ ] || git clone https://github.com/nicoshev/rapidhash.git $@

However now it doesn't see the the pyci library now

The error I get is >>> import pyci
Traceback (most recent call last):
File "", line 1, in
File "/Users/valeriichuiko/Desktop/projects/playground/PyCI/pyci/init.py", line 18, in
from pyci._pyci import version, c_long, c_ulong, c_double
ModuleNotFoundError: No module named 'pyci._pyci'

msricher · 2025-05-04T17:36:26Z

I think the extension for the C library should be different on different platforms. I made it so that Mac OSes build a .dylib file... maybe that doesn't work though. I'm not sure.

Add option to use 64-bit rapidhash for MacOS build

9a9c53b

Uses 64-bit rapidhash implementation instead of 128-bit spookyhash. For playing around on a Mac laptop, this should be fine. Sets up spookyhash or rapidhash based on whether the command `uname -s` outputs 'Darwin' (the MacOS kernal).

msricher requested review from FanwangM and marco-2023 March 6, 2025 14:18

Add -Ideps/rapidhash to CFLAGS

0c4d03d

Remove deprecated std::iterator use

9784aaa

Conversation

msricher commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FanwangM commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msricher commented Mar 6, 2025

Uh oh!

msricher commented Mar 6, 2025

Uh oh!

FanwangM commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msricher commented Mar 7, 2025

Uh oh!

FanwangM commented Mar 7, 2025

Uh oh!

msricher commented Mar 7, 2025 via email

Uh oh!

msricher commented Mar 7, 2025

Uh oh!

FanwangM commented Mar 8, 2025

Uh oh!

msricher commented Mar 8, 2025

Uh oh!

msricher commented Mar 11, 2025

Uh oh!

RichRick1 commented Mar 14, 2025

Uh oh!

msricher commented May 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

msricher commented Mar 6, 2025 •

edited

Loading

FanwangM commented Mar 6, 2025 •

edited

Loading

FanwangM commented Mar 7, 2025 •

edited

Loading