Brian2GeNN SIGSEGV Error

Description of problem

Hi Brian2 Team! I am building a model of cortical columns. Each column has 225 neurons, 12 subgroups and 68 different synapse definitions. I can run a 1.2 second simulation of a pair of columns in about 45 seconds using Brian2 and numpy on a regular CPU. But I am trying to speed this up so I can run 128 columns as fast as possible.

I’ve got it working using Brian2Genn on a TitanV GPU with 12GB onboard memory, running CUDA version 9.1.85. This takes about 120 seconds for a pair of columns, but I can only run a maximum of 64 columns before I get a RuntimeError (Command [’.main’,‘test’,‘1.2’] failed with error code -11) which says it “died with <Signals.SIGSEGV: 11>”.

Minimal code to reproduce problem

Code is quite long, but can be found at https://github.com/MunozatABI/CorticalModel. Specifically the column_b2genn.py file.

What you have already tried

I was able to run 64 columns on the CPU, it takes about 160 seconds. I can also run it on cython, but it takes about twice as long as numpy. Had a look at the debug files but couldn’t make sense of it.
The brian_debug_8w90b8q0.log, brian_script_1y0d2dcq.py, brian_stderr_ftm3sh_d.log, brian_stdout_zm27gccy.log can also be found here.

Sorry, I’m not much of coder! I’m a bit confused as to why cython and brian2genn is not making my code any faster. Any advice you have on this would be greatly appreciated.

Full traceback of error

For the error you are getting, I think the most likely cause is that you are running out of memory. You can check how much memory you are using for the example that runs, and then scale up to see the necessary memory.

About the performance: I did not have a detailed look at your code yet, but I am pretty sure that the main reason is the large number of objects you create. Presumably the slow time you see for Cython/Brian2GeNN is due to the compilation in the beginning. If you run with the report option (e.g. something like run(duration, report='text'), you’ll probably see that it takes a long time before it actually starts the simulation (or in the case of Cython, that the simulation is very slow in the very beginning since it compiles the code the first time it uses it).

So to improve things, you should try to put as much into the same object as possible. From a quick look it seems you are already doing this for the NeuronGroup (you only have two groups and then the individual layers are subgroups – you could even merge them into one, I think). Instead of having one SpikeMonitor per subgroup, you could have a single one for the full group. But most importantly, you do not need that many Synapses objects. Instead of having one Synapse group for each connection between subgroups, you can have a single Synapses object based on the full NeuronGroup, and then establish connections and set parameters where you need them. To make things manageable and to avoid having to deal with indices all the time, I’d recommend putting an attribute for the layer/type into your NeuronGroup. If you define something like this:

neurons = NeuronGroup(..., '''...
                           layer_type : integer (constant)''', ...)
L23E, L23I, L5E, L5I = 1, 2, 3, 4
neurons[:50] = L23E
neurons[50:100] = L23I
neurons[100:150] = L5E
neurons[150:200] = L5I

then you can use this to set for example connections and delays:

synapses.connect('layer_type_pre== L23E and layer_type_post == L5E', p='....')
synapses.delay[' layer_type_pre== L23E and layer_type_post == L5E'] = ...

Have a look at the first example from our paper where we use this technique (the variable is called label in that example).

1 Like

Thanks Marcel! This was really helpful, and defining a layer/type attribute with fewer synapse objects has sped up my code so it now runs in half the time.

However, using this method I am unable to implement a probability definition using a string input for p such as:

synapses.connect('layer_pre == {} and layer_post == {}'.format(src, tgt), p='{} '.format(pmax)

where src = L23E and tgt = L5E are neuron groups

Error Message : The identifier L23E could not be resolved

It works when I just use p = pmax, but I’m hoping to define the probability of connection using a stochastic spatial connectivity like p='{}*exp(-((X_pre-X_post)**2 + (Y_pre-Y_post)**2)/(2*(37.5*{})**2))'.format(Pmax,Radius)). How could I define this in the synapses.connect statement?

Also, because Brian2GeNN doesn’t support heterogeneous delays, I’m assuming I still would have to create multiple synapse objects if I want different delay values?

Really appreciate your time and advice.

Your connect statement should work, but I think you are defining the L23E constant elsewhere and it is not accessible in the function where you call synapses.connect. If this is the case, there are three options:

  1. You define the constants in that function as well
  2. You provide the constants as a dictionary to the namespace argument of the Synapses constructor, together with all other constants you are using.
  3. You do not use the name in the connect statement, but instead the value (e.g. directly layer_pre == 1 if you defined L23E as 1).

Oh wait, you are saying it works when you just use p=pmax? Now I am confused… Could you share some more complete code (or a link to it) where you have the problem?

Right, I actually forgot about that. GeNN (the underlying simulator that runs things on the GPU) actually now has support for heterogeneous delays, but we have not yet added support for it to Brian2GeNN (see this github issue). I wonder whether without this support and your model, Brian2GeNN will actually give you a speed up over standalone mode (with a single Synapses object, and using OpenMP to distribute the calculations over multiple processors).

Thanks for your response. By defining the constants, you mean L23E, L23I, L5E, L5I = 1, 2, 3, 4, etc.?
I did add this to the global space and the function, but it had no effect.

Yes, the connect statements work if I just assign p to a variable rather than a string.

A link to the code can be found here, I put it all on one script to make it easier to read (except the synaptic parameter values in the Esser_table1.csv file, which can be found in the repository here). I’ve defined it in the Synapses section and given an example of the string that doesn’t work as well as the p definition that does.

mstimberg:
Right, I actually forgot about that. GeNN (the underlying simulator that runs things on the GPU) actually now has support for heterogeneous delays, but we have not yet added support for it to Brian2GeNN (see this github issue ). I wonder whether without this support and your model, Brian2GeNN will actually give you a speed up over standalone mode (with a single Synapses object, and using OpenMP to distribute the calculations over multiple processors).

Sorry, would you mind expanding on this a little more? I’m under the impression that if I want to have heterogenous delays, I will need to define a synapse object for each group that has a different delay time. Did this thread talk about how to assign the delays in GeNN? I haven’t managed to speed it up with Brian2GeNN, but the normal CPU version has been sped up significantly by creating less synapse objects.

Thanks for your help!

It turns out that your issue is a bug in Brian! Fortunately this bug seems to have been fixed as a side effect of fixing another issue. Could you try updating Brian to the latest version (i.e. 2.4)? Your code runs works for me with that version.

Sorry if this wasn’t clear. What I meant: Currently, Brian2GeNN does not support heterogeneous delays, so you are right that you’ll have to create a Synapses object for each delay. We will add support for this feature in Brian2GeNN in the foreseeable future (since GeNN supports it now), and then you could use the same approach as on the CPU. I just wondered whether Brian2GeNN with multiple Synapses objects is slower than Brian2 in C++ standalone mode with a single (or very few) Synapses objects. Let me know if this still isn’t clear :slight_smile: