Strange redefinition error upon Brian upgrade to version 2.4+git: handling of multiple runs in standalone not consistent with previous Brian versions

Support
,
1

Description of problem

My code for some network model is not compiling any longer after Brian’s last upgrade – i.e., from version 2.2.x (or maybe 2.3.x) to v2.4+git. It is hard to figure out what has changed – but everything was working fine before.

I suppose that the way of handling multiple consecutive runs in standalone mode has changed with the last version. This issue at least seems related to this aspect.

Minimal code to reproduce the problem

The following MWE precisely reproduces the problem:

import os
from brian2 import *

code_dir = os.path.abspath('./code_gen')  # It is better to provide always the absolute path
set_device('cpp_standalone',directory=code_dir,build_on_run=False)
device.delete(force=True)  # Clean codegen directory for safety
defaultclock.dt = 5e-4*second

# Sample code
N_neurons = 10
# Jvals = np.r_[1.0] # Single run: This will just build fine the error
Jvals = np.arange(0.5,2.5,0.2) # Multiple runs: FAILS
G = NeuronGroup(N_neurons, 'dv/dt = -v / (10*ms) : 1',
                threshold='v > 1', reset='v = 0', name='neu')
S = Synapses(G, G, model='w:1', on_pre='v+=w', namespace={'J': 0.0},name='syn')
S.connect('i!=j')
prefs.devices.cpp_standalone.openmp_threads = 12
mynet = Network([G,S])

duration = 1.0*second

for j in Jvals:
    mynet['syn'].namespace['J'] = j
    mynet['syn'].w = 'J*rand()'
    mynet.run(duration=duration,report='text')

device.build(directory=code_dir)

What you have already tried.

It is clearly a problem with the handling of multiple runs in standalone with the upgrade. If you use only one run for a single parameter it will just build fine.

Expected output (if relevant)

N/A

Actual output (if relevant)

The full error message:

INFO       No numerical integration method specified for group 'neu', using method 'exact' (took 0.03s). [brian2.stateupdaters.base.method_choice]
WARNING    OpenMP code is not yet well tested, and may be inaccurate. [brian2.devices.cpp_standalone.device.openmp]
makefile:18: recipe for target 'main' failed
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
code_objects/before_run_syn_pre_push_spikes.o: In function `_before_run_syn_pre_push_spikes()':
before_run_syn_pre_push_spikes.cpp:(.text+0x0): multiple definition of `_before_run_syn_pre_push_spikes()'
code_objects/before_run_syn_pre_push_spikes.o:before_run_syn_pre_push_spikes.cpp:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status
make: *** [main] Error 1
ERROR      Brian 2 encountered an unexpected error. If you think this is a bug in Brian 2, please report this issue either to the mailing list at <http://groups.google.com/group/brian-development/>, or to the issue tracker at <https://github.com/brian-team/brian2/issues>. Please include this file with debug information in your report: /tmp/brian_debug_t7ggz793.log  Additionally, you can also include a copy of the script that was run, available at: /tmp/brian_script_1q3edmac.py You can also include a copy of the redirected std stream outputs, available at /tmp/brian_stdout_k4g2zt4y.log and /tmp/brian_stderr_n4t3ql69.log Thanks! [brian2]
Traceback (most recent call last):
  File "/home/mdepitta/Ongoing.Projects/DePitta.PNAS/Software/NGN.Spiking.Network/testing_brian_modules.py", line 28, in <module>
    device.build(directory=code_dir)
  File "/home/mdepitta/local/brian2/brian2/devices/cpp_standalone/device.py", line 1253, in build
    self.compile_source(directory, compiler, debug, clean)
  File "/home/mdepitta/local/brian2/brian2/devices/cpp_standalone/device.py", line 989, in compile_source
    raise RuntimeError(error_message)
RuntimeError: Project compilation failed (error code: 512). Consider running with "clean=True" to force a complete rebuild.

Note: specifying clean=True won’t solve the problem: the same error will be produced, regardless.

Full traceback of error (if relevant)

N/A

2

Have you tried deleting your .codegen directory? It might be that old files are interfering with new ones. If not, this looks like a regression that we should investigate.

1 Like
3

@dan I might owe you a night of sleep. I deleted it and now it seems to run fine. Let me check it though. I have multiple simulations. I will update you tomorrow, once I wake up and check the server where I am running the code.

I am a bit confused though: how come that my device.delete() instance in the second line of my sample code above, does not do the trick? Why do I have to manually delete the directory?

4

We don’t delete all the files in the directory in case you’ve stored something there you don’t want to delete. This is normally fine because we keep track of which files we created, but can cause problems across upgrades. Yan do this explicitly with .delete(force=True).

5

@dan the force=True option on delete was exactly what I needed/ Thanks.

6

It has most likely do with the initialization of the data structure storing synaptic events. Previously it was stored in its own file and it is now part of the file that also defines the propagation of synaptic events. But device.delete should have given you a warning that there are files that it won’t delete?

PS: It wouldn’t have changed anything with respect to this issue, but in general I would recommend to use a release (e.g. the latest 2.4.1) instead of using the development version directly from github.

7

Folks, sorry but I have bad news. Indeed the problem persists. Multiple simulations now do not run at all. The same problem. I accidentally thought I solved the issue. But I was indeed passing only 1-element parameter vector. Manually deleting, and using device.delete(force=True) won’t solve the issue. What should I do?

8

Could you reduce your example to something simple (e.g. two NeuronGroups with Synapses in between them) that reproduces the error? And just to make sure: with multiple simulations you mean multiple run statements like in your code above, not multiple calls to device.build?

9

Hi @mstimberg so let me try to work out a simple example. If the issue is on multiple runs as I suspect then we can proceed from there. I am coding it right now. I will post it shortly.

1 Like
10

@mstimberg @dan Ok, I am able to reproduce the problem. I am pasting the MWE above while amending my question accordingly since this looks indeed a problem with the new release. See the new code in the Minimal Working Example (MWE) above.

1 Like
11

ok, MWE posted for your insights.

12

Ok, great, that’s a perfect MWE (or is it a minimal not-working example?)! I can indeed reproduce the problem with your code. I’ll have a closer look this afternoon.

13

thanks @mstimberg. Looking forward your update.

14

So the issue seems to be that the same files were listed several times in the make file. The fix seems quite straightforward, I opened an issue here, and committed a fix to the fix_#1237 branch. Could you check whether this fixes the issue for you? If you are already using Brian directly from github, something like

$ git fetch
$ git checkout fix_#1237

in the Brian directory should do the trick.

15

Hi @mstimberg it seems to be back to functional with the fix. I am testing right now. If everything is going to be smooth it should end the simulations in a few hours. Thanks for now. If I won’t follow up it means that the issue has been solved and I will mark the reply accordingly.

On a side note: I recall that in the docs there was the mention that in standalone there was some hard limit on the number of multiple runs. Has this limit been lifted by the new upgrade? If not, is there a way to estimate how many consecutive runs I can include in a batch job, without the need for trial-and-error, when the job ends unexpectedly by the exhaustion of memory resources? (This might be as well a separate question/feature request).

16

The problem is the compilation stage. What standalone basically does to support multiple runs, is that it generates the whole network code over and over again for each run (one reason is that external constants are hardcoded into the code, so to take this into account we have to write the code again). For complex networks this means that there is a potentially huge number of code files that need to be compiled. This not only takes a long time, but depending on the compiler and OS it can also simply run into a “command line too long issue”. Finally, on linux we run make -j, which means that it uses independent parallel processes to compile all the source files. If there are many of them, this can take a lot of memory and make things crash. I would therefore consider changing the devices.cpp_standalone.extra_make_args_unix preference to something like ['-j', '8'] (for 8 processes, adapt this to the number of processors on your machine). I thought we had already changed this, but apparently not…

Finally, there’s a much faster technique where you only compile a single code and then change parameters “under the hood”, but this approach is not yet exposed by Brian. If you are interested in that approach you should have a look at this gist.

1 Like
17

@mstimberg I am going to love this pair of methods initialize_parameter and set_parameter_value: they will make things much easier. Looking forward to testing it once it is out (how can I get notifications?).

18

So right now I switched my brian2 to the fix_#1237 branch. When I will do a new git pull – say in a month or so from now – it won’t be an issue right? I mean, can I safely assume that the fixed branch will be automatically merged with the master branch?

19

You can use these functions right now as they are in the gist, but it is not guaranteed that it works forever since it accesses some internal Brian machinery. The final syntax will be probably somewhat different. We did not yet have a github issue specifically for this approach, so I opened one now: Multiple standalone runs with change in parameters · Issue #1239 · brian-team/brian2 · GitHub You can subscribe to it to stay informed.

20

If you just do a git pull it will give you an error, since by then the fix_#1237 branch will most likely no longer exist. You will have to git checkout master first to get back on the main branch.

1 Like