I want to reinitialize a set of monitors that are explicitly added to a Network object after a simulation time of length T/N. The reason is I’m recording a number of variables and things get quickly enormous. So I want to chop the whole simulation into N chunks, do light post-processing on them and save the important parameters, restart the monitors (clear all the recorded variables but keep the network state), and then continue the simulation for the next chunk.
I’ve realized from the previous conversations in the old google group that it often suffices to del those objects and redefine them again. However, I shamefully don’t know how to do it.
My code skeleton
I define my network as follows:
# imports and some configuration
...
b2.start_scope()
net = Network(collect())
# let's define 10 groups
G = []
for i in range(10):
Gi= NeuronGroup(...)
G.append(Gi)
net.add(G)
# let's connect them somehow
adj_mat = [[...]] # adjacency matrix
S = []
for i in len(adj_mat):
for j in len(adj_mat):
.... # some condition based on adj_mat
Sij = Synapse(...)
S.append(Sij)
net.add(S)
# lets define monitors
mons = []
for k in range(K):
mon_k = StateMonitor( ... , name=str(k)) # name for tracking monitors
mons.append(mon_k)
net.add(mon_k)
net.run(T/N)
# drink a coffee, come back and face your full RAM
# some post-processing
for k in range(K):
fancy_output_from_mon_k = ...
save_fancy()
# let's clean the monitors
>> HELP NEEDED HERE <<
# and add the originally define ones to the network again
net.add(mons)
# and finally redo the procedure N-1 other times
...
What I have tried
I tried going through items in mons (that contains all the monitors) and del them one by one. I also tried to remove the whole list mons. Neither freed up the space. It’s expected since mons or its contaminants are simply python objects. What I really have to delete are the objects that are added to net. I can see all the monitors in the net.objects. (Btw, I tried all this in Jupyter environment, although I don’t think it matters.)
So the question specifically would be how to modify network objects.
Any help or ideas that help me free up the space clogged by the monitors are appreciated.
Arash
import pickle
# Set up network
state_mon = StateMonitor(...)
run(...) # a long run
data = state_mon.get_states(...)
with open('first_part.data', 'w') as f:
pickle.dump(data, f)
del state_mon
del data
state_mon = StateMonitor(...)
run(...) # another long run
I updated my questions. In short, your suggestion would work perfectly if I had defined state_mon object like you did and had relied on the Brian’s magic to collect it upon run. But since I’m having a rather involved network, it makes more sense to define all groups, synapses, and monitors in a loop and add them to the network later. And because of this very addition, I don’t know anymore how to can I access and then remove/modify those monitors from the net object.
No. Strangely it didn’t. Nor it did free up the RAM.
After a simulation for some time net.remove(monitor_list) executes without any error/warning. However, when I try net.add(monitor_list) immediately after, I get a RuntimeError:
RuntimeError: syn_mon_0 has already been simulated, cannot add it to the network. If you were trying to remove and add an object to temporarily stop it from being run, set its active flag to False instead.
I have to correct my previous response. Indeed, net.remove(monitor_list) removes all the associated monitors from the network. What it doesn’t do is the memory release.
Turned out the missing step is redefining the list. By redefinition, python assigns a different reference to list object and which in turn can be added to the network without any problem. More importantly, it releases the memory.
@arashgmn could you please post here a minimalistic code example? That’s handy information, and it will be great to have a working example for reference.
@mstimberg I wish Brian would have a button to reset buffers (a function, of course ). Well, if it isn’t possible, maybe we can add some recipe “how to clean buffers” in official documentation.
Note: This structure writes on the memory. If you’re dealing with gigantic data or have a very old hard drive, this operation may be the bottleneck of your whole simulation!
from brian2 import *
import numpy as np
start_scope()
thr = -10
v_reset = -60
# lets' make some populations
nPops = 3
pops = []
for n in range(nPops):
pop = NeuronGroup(
N = 5,
model= """
dv/dt = (-v + I)/tau : 1
I: 1
tau: second
""",
threshold="v>thr",
reset="v=v_reset",
method="euler",
)
pop.tau[:] = abs(np.random.normal(10, 3))*ms
pop.v[:] = np.random.normal(-75, 3)
pop.I[:] = 1*(pop.i+1)
pops.append(pop)
# and connect them all together with synapses
syns = []
for l in range(nPops):
for m in range(nPops):
syn = Synapses(source = pops[l], target = pops[m],
model = """w : 1""",
on_pre = """v += w""",
method = 'euler',
)
syn.connect(condition = 'i!=j', p=0.1)
syn.w = np.random.normal(0.5,0.02, size=len(syn))
syns.append(syn)
net = Network(collect()) # collects nothing
net.add(pops)
net.add(syns)
# this function redefines the monitor objects conveniently
def monitor_maker(syns):
monitors = []
for syn in syns:
monitor = StateMonitor(syn, 'w', record=True)
monitors.append(monitor)
return monitors
# and this one just saves the recorded vars on the hard disk
def intra_sim_process(monitors, chunk_id):
for monitor in monitors:
np.save('w_'+str(chunk_id)+'.npy', monitor.w)
# let's a run simulation of total duration 1 second. We break it down
# to 10 subsimulation of duration 100 ms to avoid memory cloggage
for chunk in range(10):
print("*"*10 + " CHUNK ID : {} ".format(chunk)+ "*"*10 + "\n")
# add monitors
monitors = monitor_maker(syns)
net.add(monitors)
net.run(100*ms, report='text', report_period=20*ms)
intra_sim_process(monitors, chunk)
net.remove(monitors)
If you spotted anything wrong or saw potential improvements, please don’t hesitate to leave it as a reply below. I’d be thankful.
This problem is well known, at least in parallel computing. The solution is pretty simple, fill 1/2 of memory with data, transfer data to a thread that saves data in the background and continue to compute, filling up the other half of the memory. This “butterfly” switch of memory requires that obtaining new data is slower than saving on “the slow hard disk”.
Just to give some more context: as @arashgmn correctly noted, removing monitors from the network will not free their memory, it will only mean that they are no longer simulated (same as when you set their active attribute to False). To free their memory, Python has to garbage collect the object itself. This happens when there is nothing referring to the object anymore – in your case, you overwrite the list and therefore all references to the monitor. In the case of a single monitor, you’d typically do del monitor_name.
I don’t know how much of this is just for illustration purposes, but a more efficient/compact version of your example code would only use a single NeuronGroup, a single Synapses object, and a single StateMonitor. And also note that you can change the time step of monitors, e.g. maybe storing the values of the synaptic weights only every 1ms would be enough (StateMonitor(..., dt=1*ms))? This would immediately reduce the memory usage by a factor of 10
@mstimberg I have to reopen this topic because the @arashgmn’s solution above works only in cython or numpy modes.
It seems, in both standalone and standalone - OpenMP modes, brian returns an error after the net.run() function is called a second time:
File "aLIFcortex.py", line 860, in <module>
br2net.run((trenorm-tprev)*ms,report='text')
File "/home/rth/.local/lib/python3.8/site-packages/brian2/core/base.py", line 291, in device_override_decorated_function
return getattr(curdev, name)(*args, **kwds)
File "/home/rth/.local/lib/python3.8/site-packages/brian2/devices/cpp_standalone/device.py", line 1497, in network_run
raise RuntimeError("The network has already been built and run "
RuntimeError: The network has already been built and run before. Use set_device with build_on_run=False and an explicit device.build call to use multiple run statements with this device.
Is there any work around this issue? The message suggests to set_device(..., build_on_run=False) and then device.build(), but it isn’t clear that this will save all states of the model and restore them from the last simulation step at the previous run. Should it be something like this:
set_device('cpp_standalone', build_on_run=False)
prefs.devices.cpp_standalone.openmp_threads = os.cpu_count()
device.build()
....
br2obj = [ ... ] # all network objects
br2mon = [ ... ] # all monitors
net.add(br2obj)
net.add(br2mon)
for x in range(n_stops):
net.run(single_duration*ms)
net.remove(br2mon)
net.store('latest state')
#... save all monitors ...
del br2mon
net.restore('latest state')
#... create new monitors ...
net.add(br2mon)
Hi @rth. You can have multiple runs in standalone mode with the explicit device.build calls as described. But that won’t help you here, since standalone mode does not support store and restore. There is no convenient way of doing this kind of simulation currently in standalone mode. The only workaround would be to make a normal standalone run, save all the “interesting” state variables to disk, and then start a new run which initializes all values to the previously saved ones. Apart from being less convenient, this will not be completely equivalent to store/restore, though, since the latter will also store internal data, in particular spikes that are in the spike queue but not delivered yet.
Thank you, @mstimberg, for the clarifications.
Oh, that is pretty bad news. I’m facing a pretty big model, which should be run from 2 hours to 2 days of model time. It isn’t feasible to run such a simulation in a reasonable time on a single core. I’ve checked that even a 2-hours run crashes because it runs out of memory (of course, I record spikes, only three neurons out of a few hundred, and track synapses with dt=10s).
Any ideas on how to resolve this issue will be highly appreciated.
Hi @rth. In that case, I think the best solution would be to bypass the standard monitor system completely, and instead write things to disk directly. This should be quite straightforward in standalone mode, see my example here: Real Time Monitor - #2 by mstimberg
Thank you, @mstimberg! Let me ask a few more questions: In these simulations, I mostly care about the evolution of synaptic connections. Is there any approach to dump synaptic weights into a file? Say, for example, something like run_regularly for synapses, which saves weights. Maybe it is possible to get access to dynamical variables in memory and save all required data in one short?
You should be able to use the exact same approach. See below for example code that dumps all weights to a simple space-separated file (time in first column, weights in the remaining columns) every 100ms:
from brian2 import *
set_device('cpp_standalone')
@implementation('cpp','''
// Note that functions always need a return value at the moment
double store_weights(double w, double t) {
static std::ofstream weight_file("weights.txt"); // opens the file the first time
static double _prev_t = -1.;
// Store all values for the same time in a line
if (_prev_t != t) {
if (_prev_t != -1.)
weight_file << std::endl;
weight_file << t << " ";
_prev_t = t;
}
weight_file << w << " ";
return 0.; // unused
}
''')
@check_units(w=1, t=second, result=1)
def store_weights(w, t):
raise NotImplementedError('Use standalone mode')
G = NeuronGroup(2, '')
S = Synapses(G, G, 'dw/dt = 0.1/second : 1') # growing synapses
S.run_regularly('dummy_value = store_weights(w, t)', dt=100*ms) # Store weights every 100ms
S.connect()
S.w = 'rand()'
run(1000*ms)
import os
data = np.loadtxt(os.path.join(device.project_dir, 'weights.txt'))
plt.plot(data[:, 0]*second/ms, data[:, 1:], 'o-')
plt.show()
). Well, if it isn’t possible, maybe we can add some recipe “how to clean buffers” in official documentation.
