I am trying to run brian2 code on GPU through brian2cuda, but the tests are not yielding positive results.
I have:
brian2 2.4.2
brian2cuda 1.0a1
nvcc release 12.0, V12.0.140
build cuda 12.0.r12.0/compiler.32267302_0
Minimal code to reproduce the error:
from brian2 import *
import brian2cuda
set_device(“cuda_standalone”)
brian2cuda.example_run()
==================
Full traceback of error (if relevant)
ERROR Brian 2 encountered an unexpected error. If you think this is a bug in Brian 2, please report this issue either to the discourse forum at http://brian.discourse.group/, or to the issue tracker at https://github.com/brian-team/brian2/issues. Please include this file with debug information in your report: /tmp/brian_debug_pcwsspar.log You can also include a copy of the redirected std stream outputs, available at /tmp/brian_stdout_jf5pmdtl.log and /tmp/brian_stderr_hi63lyhj.log Thanks! [brian2]
Traceback (most recent call last):
File “”, line 1, in
File “/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/init.py”, line 55, in example_run
device.build(direct_call=False, directory=directory, **build_options)
File “/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/device.py”, line 1444, in build
self.generate_codeobj_source(self.writer)
File “/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/device.py”, line 724, in generate_codeobj_source
if hasattr(codeobj, ‘owner’) and hasattr(codeobj.owner, ‘_N’) and codeobj.owner._N != 0:
ReferenceError: weakly-referenced object no longer exists
PS as a new user I cannot upload the file " brian_debug_pcwsspar.log"
I can’t reproduce your error with the brian2 and brian2cuda versions you mentioned (the error doesn’t sound like it has anything to do with your CUDA version). Are you by any chance using a Jupyter notebook? If so, can you try if running the minimal example above in it’s own Python script works as expected?
Thanks for the answer.
I probably should’ve mentioned that I am working on Ubuntu 20.04.
I am not using Jupyter since I tried everything directly on the terminal, but I have tried now with Spyder (version 4.2.5) as IDE and I got a slightly different error:
INFO CUDA installation directory detected via location of `nvcc` binary: /home/lparrilla/anaconda3/envs/CUDA [brian2.devices.cuda_standalone]
INFO Compiling device code for GPU 0 (NVIDIA RTX A6000) [brian2.devices.cuda_standalone]
INFO Compiling device code for compute capability 8.6 (compiler flags: ['-arch=sm_86']) [brian2.devices.cuda_standalone]
INFO Using the following preferences for CUDA standalone: [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.SM_multiplier = 1 [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.parallel_blocks = 1 [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.launch_bounds = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.syn_launch_bounds = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.calc_occupancy = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.extra_threshold_kernel = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.random_number_generator_type = CURAND_RNG_PSEUDO_DEFAULT [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.random_number_generator_ordering = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.push_synapse_bundles = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.threads_per_synapse_bundle = {max} [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.bundle_threads_warp_multiple = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.no_pre_references = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.no_post_references = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.default_functions_integral_convertion = <class 'numpy.float64'> [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.use_atomics = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.profile_statemonitor_copy_to_host = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.gpu_heap_size = 128 [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.detect_gpus = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.gpu_id = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.extra_compile_args_nvcc = ['-w', '-use_fast_math'] [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.compute_capability = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.detect_cuda = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.cuda_path = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.cuda_runtime_version = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.device_query_path = None [brian2.devices.cuda_standalone]
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
make: *** [makefile:23: code_objects/neurongroup_thresholder_codeobject.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [makefile:23: code_objects/neurongroup_group_variable_set_conditional_codeobject.o] Error 1
make: *** [makefile:23: code_objects/neurongroup_stateupdater_codeobject.o] Error 1
make: *** [makefile:23: network.o] Error 1
make: *** [makefile:23: synapses_classes.o] Error 1
make: *** [makefile:23: main.o] Error 1
make: *** [makefile:23: rand.o] Error 1
make: *** [makefile:23: code_objects/neurongroup_resetter_codeobject.o] Error 1
make: *** [makefile:23: run.o] Error 1
make: *** [makefile:23: objects.o] Error 1
Traceback (most recent call last):
File "<ipython-input-4-af9ee0f286b0>", line 1, in <module>
brian2cuda.example_run()
File "/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/__init__.py", line 55, in example_run
device.build(direct_call=False, directory=directory, **build_options)
File "/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/device.py", line 1476, in build
self.compile_source(directory, cpp_compiler, debug, clean)
File "/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2/devices/cpp_standalone/device.py", line 990, in compile_source
raise RuntimeError(error_message)
RuntimeError: Project compilation failed (error code: 512). Consider running with "clean=True" to force a complete rebuild.
This looks like an issue with your CUDA installation. How did you install CUDA? Can you send me the output of the following commands
which nvcc
nvidia-smi
And can you send me a summary of your conda environment?
conda list
Your nvcc binary was detected inside your conda environment folder. That is typically not where CUDA is installed (except you installed it there manually?). It could be that your CUDA path is not correctly detected due to a conda package providing an nvcc wrapper.
If that is the case, you could manually set the CUDA path via the following preference:
from brian2 import *
...
prefs.devices.cuda_standalone.cuda_backend.cuda_path = /path/to/cuda/installation
where /path/to/cuda/installation should be the output of which nvcc without the trialing nvcc (e.g. if which nvcc returns /usr/local/cuda/nvcc, then the path should be /usr/local/cuda)
Ah, interesting. I wasn’t aware that NVIDIA makes CUDA available via conda now. Neat. For Brian2CUDA, you need the entire CUDA tool chain (e.g. cudatoolkit from conda-forge is not enough). Until recently, conda packages didn’t include the entire CUDA tool chain because of NVIDIA EULA requirements, which is why one Brian2CUDA requirement is a full CUDA toolkit installation (else compilation fails as you see in your case). Now that NVIDIA distributes their CUDA tool chain via conda, we could support a full Brian2CUDA conda package, that is great.
Anyways, back to your error. It appears you installed cuda-nvcc, which provides the nvcc binary, but not the CUDA toolchain I mentioned above. Installing the full cuda package fixed it for me:
conda install -c nvidia cuda
After installing this, I got an error about the wrong GLIBC version being installed. So I also had to install a newer GLIBC version (and you probably will too):
This command installs GLIBC 2.17, which is required by one of the CUDA libraries we use. This all should be more streamlined if we provide a conda package for Brian2CUDA. But for now, this might solve it for you. I also tried installing only cuda-runtime instead of cuda, but it didn’t provide cuda_runtime.h.
This should fix your compilation error.
For the weakly-referenced object error, I still can’t reproduce it. Is it still happening? Can you try to run your minimal example from above outside your IDE, that means just from your terminal with
python example.py
where example.py is
from brian2 import *
import brian2cuda
set_device("cuda_standalone")
brian2cuda.example_run()
Ok so I tried your suggestions but when I try to install sysroot_linux it appears that alot of packages have conflicts, so I created a new env from conda then run:
No frozen solve and anything, the installation went smooth, but when I try
python
from brian2 import *
import brian2cuda
set_device("cuda_standalone")
brian2cuda.example_run()
I have this brand new error:
INFO CUDA installation directory detected via location of `nvcc` binary: /home/lparrilla/anaconda3/envs/CUDACUDA [brian2.devices.cuda_standalone]
INFO Compiling device code for GPU 0 (NVIDIA RTX A6000) [brian2.devices.cuda_standalone]
INFO Compiling device code for compute capability 8.6 (compiler flags: ['-arch=sm_86']) [brian2.devices.cuda_standalone]
INFO Using the following preferences for CUDA standalone: [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.SM_multiplier = 1 [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.parallel_blocks = 1 [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.launch_bounds = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.syn_launch_bounds = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.calc_occupancy = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.extra_threshold_kernel = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.random_number_generator_type = CURAND_RNG_PSEUDO_DEFAULT [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.random_number_generator_ordering = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.push_synapse_bundles = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.threads_per_synapse_bundle = {max} [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.bundle_threads_warp_multiple = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.no_pre_references = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.no_post_references = False [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.default_functions_integral_convertion = <class 'numpy.float64'> [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.use_atomics = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.profile_statemonitor_copy_to_host = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.gpu_heap_size = 128 [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.detect_gpus = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.gpu_id = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.extra_compile_args_nvcc = ['-w', '-use_fast_math'] [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.compute_capability = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.detect_cuda = True [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.cuda_path = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.cuda_runtime_version = None [brian2.devices.cuda_standalone]
INFO devices.cuda_standalone.cuda_backend.device_query_path = None [brian2.devices.cuda_standalone]
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
/home/lparrilla/anaconda3/envs/CUDACUDA/bin/x86_64-conda-linux-gnu-c++: No such file or directory
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
nvcc fatal : Failed to preprocess host compiler properties.
make: *** [makefile:23: code_objects/neurongroup_resetter_codeobject.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [makefile:23: code_objects/neurongroup_stateupdater_codeobject.o] Error 1
make: *** [makefile:23: code_objects/neurongroup_group_variable_set_conditional_codeobject.o] Error 1
make: *** [makefile:23: network.o] Error 1
make: *** [makefile:23: main.o] Error 1
make: *** [makefile:23: synapses_classes.o] Error 1
make: *** [makefile:23: rand.o] Error 1
make: *** [makefile:23: run.o] Error 1
make: *** [makefile:23: objects.o] Error 1
make: *** [makefile:23: code_objects/neurongroup_thresholder_codeobject.o] Error 1
ERROR Brian 2 encountered an unexpected error. If you think this is a bug in Brian 2, please report this issue either to the discourse forum at <http://brian.discourse.group/>, or to the issue tracker at <https://github.com/brian-team/brian2/issues>. Please include this file with debug information in your report: /tmp/brian_debug__5_cdqsb.log You can also include a copy of the redirected std stream outputs, available at /tmp/brian_stdout_iyjr4an8.log and /tmp/brian_stderr_ir_uvcxc.log Thanks! [brian2]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/__init__.py", line 55, in example_run
device.build(direct_call=False, directory=directory, **build_options)
File "/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2cuda/device.py", line 1476, in build
self.compile_source(directory, cpp_compiler, debug, clean)
File "/home/lparrilla/anaconda3/envs/CUDA/lib/python3.6/site-packages/brian2/devices/cpp_standalone/device.py", line 990, in compile_source
raise RuntimeError(error_message)
RuntimeError: Project compilation failed (error code: 512). Consider running with "clean=True" to force a complete rebuild.
Same exact error returned if sysroot_lunux is not installed
This is now getting into conda environment specifics. Generally, I want to point out that the issues you are having are not related to Brian2CUDA, but to installing CUDA. The easiest solution would be to install the CUDA toolkit not via conda, or to figure out how to correctly install it via conda. This document should generally help you with that: CUDA Installation Guide for Linux
If you want to continue with your CUDA in conda installation, I can continue guessing solutions for you:
Did you install Brian2 via conda in that new environment? If not, please do so (the correct version 2.4.2, so conda install -c conda-forge brian2=2.4.2). The Brian2 conda package installs a C++ compiler via conda as dependency, which should solve the error you are seeing here (it looks like the conda cuda package expects a conda C++ package which you have not installed).
Alternatively, you could also just install C++ via conda: conda install -c conda-forge gxx_linux-64 gcc_linux-64 (I’m not sure which of the packages you actually need, but both will likely do).
I resolved the issue by creating a new user on the machine, first I installed anaconda, then I create a new environment WITH spyder, otherwise I will have:
The following specifications were found to be incompatible with your system:
- feature:/linux-64::__glibc==2.31=0
- feature:|@/linux-64::__glibc==2.31=0
if I try to install spyder after.
Then I proceed to install cuda toolkit (found several versions online with or without the -, this worked fine for me):
Note that this is 1) mostly due to the fact that Brian2CUDA is not yet compatible with the latest version of Brian (which already fixed this issue) and 2) only occurs with numpy ≥ 1.24 (earlier versions will only raise a warning).
Thanks for pinpointing that out!
Please note that I did not install anything (except the cudatoolkit) before installing brian2cuda, so I was assuming that the installation would proceed to find all the compatible versions of different libraries, anyway the important thing is that I have it up and running!
As you say, the important thing is that it is working now, but just to explain: brian2cuda still depends on brian2 2.4.2, released at the end of 2020 (!). Back then, numpy was at version 1.19. The aliases np.bool/np.float were deprecated and started to raise warnings only in numpy 1.20, and finally turned into errors with the recent numpy 1.24 release. Long story short: we need to make brian2cuda work with the most recent brian2 version (working on it…).
@lparr Sorry for the late response, I was occupied with a conference trip
I’m glad you got it working with cudatoolkit-dev. I was aware that it ships all the CUDA Toolkit stuff, but I never tried it myself!
For the Brian2CUDA and Brian2 versions: There is already a version of Brian2CUDA that is compatible with the latest Brian2 version (2.5.1), I just haven’t released it yet. If you want to, you can install it like this:
Uninstall brian2cuda if it is installed in the environment (pip uninstall brian2cuda)