Linker error after device reinitialization in cpp_standalone

Description of problem

Reinitializing the device/rebuilding under C++ standalone code generation with multiple runs seems to cause some issue with linked dependencies such as mkl. I get “symbol lookup error: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: floor” followed by a RuntimeError on the second execution (fine on the first)

Minimal code to reproduce problem

import os, sys, warnings
import numpy as np
import random as pyrandom
from brian2 import *
set_device("cpp_standalone", build_on_run=False)

def example_sim(tau):
    device = get_device()
    device.reinit()
    device.activate(build_on_run=False)

    num_inputs = 100
    input_rate = 10*Hz
    weight = 0.1
    tau *= ms
    # Construct the network each time
    P = PoissonGroup(num_inputs, rates=input_rate)
    eqs = '''
    dv/dt = -v/tau : 1
    '''
    G = NeuronGroup(1, eqs, threshold='v>1', reset='v=0', method='exact')
    S = Synapses(P, G, on_pre='v += weight')
    S.connect()
    M = SpikeMonitor(G)
    # Run it and store the output firing rate in the list
    run(1*second)
    device.build(directory='output', clean=True, compile=True, run=True, debug=False)

if __name__ == "__main__":
    print("Simulation 1")
    example_sim(5)
    print("Simulation 2")
    example_sim(50)

Expected behavior

Run both simulations

Actual behavior

Runs Simulation 1, prints “Simulation 2” but then

sh: symbol lookup error: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: floor
sh: symbol lookup error: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: floor
ERROR      Brian 2 encountered an unexpected error. If you think this is a bug in Brian 2, please report this issue either to the discourse forum at <http://brian.discourse.group/>, or to the issue tracker at <https://github.com/brian-team/brian2/issues>. Please include this file with debug information in your report: /tmp/brian_debug_mfx1kirh.log  Additionally, you can also include a copy of the script that was run, available at: /tmp/brian_script_0ydeexkq.py You can also include a copy of the redirected std stream outputs, available at /tmp/brian_stdout_b3wjq575.log and /tmp/brian_stderr_1s3ehowt.log Thanks! [brian2]
Traceback (most recent call last):
  File "bug_reproduce.py", line 33, in <module>
    example_sim(50)
  File "bug_reproduce.py", line 27, in example_sim
    device.build(directory='output', clean=True, compile=True, run=True, debug=False)
  File "/home/zhenrui/anaconda3/envs/lab3/lib/python3.7/site-packages/brian2/devices/cpp_standalone/device.py", line 1254, in build
    self.compile_source(directory, compiler, debug, clean)
  File "/home/zhenrui/anaconda3/envs/lab3/lib/python3.7/site-packages/brian2/devices/cpp_standalone/device.py", line 990, in compile_source
    raise RuntimeError(error_message)
RuntimeError: Project compilation failed (error code: 32512).

Hi @zspam . This is odd. I can run your example program (thanks for the minimal example!) on my machine without issues. Are you sure you are running the exact code you posted? I am also a bit confused, since Brian’s standalone mode does not link to any external library, including the mkl. Can you maybe switch on debug=True in the build option to get more output?

What happens if each of the runs compiles to a different directory (e.g. with directory=None)?

Oh, and what version of Brian are you using?

Hi @mstimberg . Thanks for the fast response! I discovered this issue with more complicated code but this is the minimum example that reproduces the issue on my machine. I have Brian 2.4.1 but just tested and was able to reproduce on Brian 2.5.1. With debug=True and directory=None I get

Simulation 1
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-cc -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/user/anaconda3/envs/brian/include -DNDEB
UG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/user/anaconda3/envs/brian/include  -c -o brianlib/randomkit/randomkit.o brianlib/randomkit/randomkit.c
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/after_run_neurongroup_spike_thresholder_codeobject.cpp -o code_objects/after_run_neurongroup_spike_thresholder_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/after_run_poissongroup_spike_thresholder_codeobject.cpp -o code_objects/after_run_poissongroup_spike_thresholder_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/before_run_synapses_pre_push_spikes.cpp -o code_objects/before_run_synapses_pre_push_spikes.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/neurongroup_spike_resetter_codeobject.cpp -o code_objects/neurongroup_spike_resetter_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/neurongroup_spike_thresholder_codeobject.cpp -o code_objects/neurongroup_spike_thresholder_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/neurongroup_stateupdater_codeobject.cpp -o code_objects/neurongroup_stateupdater_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/poissongroup_spike_thresholder_codeobject.cpp -o code_objects/poissongroup_spike_thresholder_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/spikemonitor_codeobject.cpp -o code_objects/spikemonitor_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/synapses_pre_codeobject.cpp -o code_objects/synapses_pre_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/synapses_pre_push_spikes.cpp -o code_objects/synapses_pre_push_spikes.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG code_objects/synapses_synapses_create_generator_codeobject.cpp -o code_objects/synapses_synapses_create_generator_codeobject.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG main.cpp -o main.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG network.cpp -o network.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG objects.cpp -o objects.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG run.cpp -o run.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG synapses_classes.cpp -o synapses_classes.o
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ -c -Wno-write-strings -Ibrianlib/randomkit -I/home/user/anaconda3/envs/brian/include -w -O3 -ffast-math -fno-finite-math-only -march=native -std=c++11 -I.  -g -DDEBUG -MM brianlib/randomkit/randomkit.c code_objects/after_run_neurongroup_spike_thresholder_codeobject.cpp code_objects/after_run_poissongroup_spike_thresholder_codeobject.cpp code_objects/before_run_synapses_pre_push_spikes.cpp code_objects/neurongroup_spike_resetter_codeobject.cpp code_objects/neurongroup_spike_thresholder_codeobject.cpp code_objects/neurongroup_stateupdater_codeobject.cpp code_objects/poissongroup_spike_thresholder_codeobject.cpp code_objects/spikemonitor_codeobject.cpp code_objects/synapses_pre_codeobject.cpp code_objects/synapses_pre_push_spikes.cpp code_objects/synapses_synapses_create_generator_codeobject.cpp main.cpp network.cpp objects.cpp run.cpp synapses_classes.cpp > make.deps
/home/user/anaconda3/envs/brian/bin/x86_64-conda-linux-gnu-c++ brianlib/randomkit/randomkit.o code_objects/after_run_neurongroup_spike_thresholder_codeobject.o code_objects/after_run_poissongroup_spike_thresholder_codeobject.o code_objects/before_run_synapses_pre_push_spikes.o code_objects/neurongroup_spike_resetter_codeobject.o code_objects/neurongroup_spike_thresholder_codeobject.o code_objects/neurongroup_stateupdater_codeobject.o code_objects/poissongroup_spike_thresholder_codeobject.o code_objects/spikemonitor_codeobject.o code_objects/synapses_pre_codeobject.o code_objects/synapses_pre_push_spikes.o code_objects/synapses_synapses_create_generator_codeobject.o main.o network.o objects.o run.o synapses_classes.o -o main -Lbrianlib/randomkit -L/home/user/anaconda3/envs/brian/lib -Wl,--disable-new-dtags,-R/home/user/anaconda3/envs/brian/lib -g
Number of spikes: 4
Number of synapses: 100
§imulation 2
sh: symbol lookup error: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: floor
sh: symbol lookup error: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: floor
ERROR      Brian 2 encountered an unexpected error. If you think this is a bug in Brian 2, please report this issue either to the discourse forum at <http://brian.discourse.group/>, or to the issue tracker at <https://github.com/brian-team/brian2/issues>. Please include this file with debug information in your report: /tmp/brian_debug_26w439pb.log  Additionally, you can also include a copy of the script that was run, available at: /tmp/brian_script_424z8s2s.py You can also include a copy of the redirected std stream outputs, available at '/tmp/brian_stdout_k2fcvhhi.log' and '/tmp/brian_stderr_h68_j50r.log'. Thanks! [brian2]
Traceback (most recent call last):
  File "bug_reproduce.py", line 33, in <module>
    example_sim(50)
  File "bug_reproduce.py", line 27, in example_sim
    device.build(directory=None, clean=True, compile=True, run=True, debug=True)
  File "/home/user/anaconda3/envs/brian/lib/python3.8/site-packages/brian2/devices/cpp_standalone/device.py", line 1290, in build
    self.compile_source(directory, compiler, debug, clean)
  File "/home/user/anaconda3/envs/brian/lib/python3.8/site-packages/brian2/devices/cpp_standalone/device.py", line 1031, in compile_source
    raise RuntimeError(error_message)
RuntimeError: Project compilation failed (error code: 32512).

From the debug output, it looks as if it did not get to the step of actually compiling… From the error, I am not sure this problem is actually related to Brian as such. There might be a conflict between libraries installed in your system (/opt/intel/mkl/...) and the ones installed in the conda environment. In the shell, does export | grep mkl bring anything up, e.g. a manually set LD_LIBRARY_PATH? If yes, can you try unsetting it? Still, no idea why this is only a problem on the second run…

Hm – strangely manually unsetting LD_LIBRARY_PATH doesn’t do anything but unsetting LD_PRELOAD (which I have set for other reasons) does fix it. Still not sure why it was only affecting the second run but fixed now – thanks!

Glad to hear that it worked, these problems are quite annoying to debug!