Non-standalone restoring states from saved states of standalone mode

Description of problem

One cannot use store() or restore() in the standalone modes. I want to restore the saved states from standalone mode in a cython initiation of network, for quick runs/ testing and not necessarily longer runs for which I used standalone initially. I just want to restore the network, and use the plotting codes I wrote for Cython implementation.

Minimal code to reproduce problem

#in cpp_standalone/ cuda_standalone

state = {}

for obj in self.network.sorted_objects:

    print(f"Saving state for {obj.name}")

    if hasattr(obj, 'get_states'):

        state[obj.name] = obj.get_states()


with open(filename, 'wb') as f:

    pickle.dump(state, f)

with open(filename.replace('.pkl', '_monitors.pkl'), 'wb') as f:

    pickle.dump(monitor_states, f)

#in non standalone (cython) implementation:
#assuming restoring the exact network structure (say, from a stored file having the initial states and weights)

with open(filename, 'rb') as f:
    state = pickle.load(f)
for obj in self.network.sorted_objects:
    if obj.name in state:
        print(f"Restoring state for {obj.name}")
    if hasattr(obj, '_restore_from_full_state'):
        new_state = {}
        for i in state[obj.name]:
            temp_entry = state[obj.name][i]
            temp_entry = (temp_entry, temp_entry.shape) #because this is how the input to _restore_from_full_state is needed. 
            new_state[i] = temp_entry
        obj._restore_from_full_state(new_state)

What you have aready tried

The above code.
I tried changing the groups.py _restore_from_full_state() function to:

def _restore_from_full_state(self, state):    
    for var_name, (values, size) in state.items():
        print(f"Restoring variable {var_name} with size {size}")
        var = self.variables[var_name]
    if isinstance(var, DynamicArrayVariable):
        var.resize(size)
    try:
        var.set_value(values)
    except Exception as e:
        print(f"Error restoring variable {var_name}: {e}")
        continue

To see what error arise when doing it manually.

Expected output (if relevant)

Actual output (if relevant)

Restoring variable N_incoming with size (3600,)
Error restoring state for synapses: '>' not supported between instances of 'tuple' and 'int'

Full traceback of error (if relevant)

Traceback when not catching exceptions:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 net.load_state(filename='./sim_data/final_network_state.pkl')

File ~/DHC/Simulator/NetworkInit2.py:1367, in NetworkInit.load_state(self, filename, monitor_filename)
   1365             temp_entry = (temp_entry, temp_entry.shape)
   1366             new_state[i] = temp_entry
-> 1367     obj._restore_from_full_state(new_state)
   1368 else:
   1369     for var, values in state[obj.name].items():

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/groups/group.py:653, in VariableOwner._restore_from_full_state(self, state)
    651 var = self.variables[var_name]
    652 if isinstance(var, DynamicArrayVariable):
--> 653     var.resize(size)
    654 var.set_value(values)

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/core/variables.py:660, in DynamicArrayVariable.resize(self, new_size)
    658     self.device.resize_along_first(self, new_size)
    659 else:
--> 660     self.device.resize(self, new_size)
    662 self.size = new_size

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/devices/device.py:535, in RuntimeDevice.resize(self, var, new_size)
    534 def resize(self, var, new_size):
--> 535     self.arrays[var].resize(new_size)

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/memory/dynamicarray.py:199, in DynamicArray1D.resize(self, newshape)
    197 def resize(self, newshape):
    198     (datashape,) = self._data.shape
--> 199     if newshape > datashape:
    200         (shape,) = self.shape  # we work with int shapes only
    201         newdatashape = max(newshape, int(shape * self.factor) + 1)

TypeError: '>' not supported between instances of 'tuple' and 'int'

Please don’t mind the methods called, before _restore_from_full_state()

Hi @Shreshthallbestno1. This is an interesting use case that we did not really consider. Luckily, I think in this direction (storing the state of a simulation in standalone mode and restoring it in runtime mode) there is actually a reasonably simple solution (the other direction is more complicated). The trick is that you should be able to store the state of a C++ standalone mode, if you switch to the runtime device after simulation. This state can then be restored in the usual way in runtime mode. Here’s a simple example that seems to work – but of course this does not necessarily mean that a more complex simulation will work as well.

Here’s the first simulation in standalone mode:

from brian2 import *

filename = "/tmp/store.pkl"
set_device("cpp_standalone")
tau = 10 * ms
group = NeuronGroup(
    100, "dv/dt = -v/tau : 1", threshold="v>1", reset="v=0", method="exact"
)
syn = Synapses(group, group, on_pre="v_post += 0.1")
syn.connect(p=0.1)
group.v = "rand()"
net = Network(collect())
net.run(50 * ms)

set_device("runtime")
net.store(filename=filename)

The last two lines store the state after switching back to runtime mode.
Then, in runtime mode you can restore the state as if it had been stored by another runtime simulation:

from brian2 import *

filename = "/tmp/store.pkl"

tau = 10 * ms
group = NeuronGroup(
    100, "dv/dt = -v/tau : 1", threshold="v>1", reset="v=0", method="exact"
)
syn = Synapses(group, group, on_pre="v_post += 0.1")
group.v = "rand()"
net = Network(collect())
net.restore(filename=filename)

net.run(50 * ms)

If this works for you, then this should be easier and more robust than storing and restoring the states manually. Please let us know if this is a feasible solution for you!

Hey,
Thanks for the quick reply,

I tried the solution above, and then I get the following error:

FileNotFoundError                         Traceback (most recent call last)
Cell In[3], line 13
10 run_network(net, tStep=config[‘tStep’], nStep=config[‘nStep’], STORAGE_DIR=STORAGE_DIR)
12 set_device(“runtime”)
—> 13 net.network.store(filename=‘./sim_data/final_network_state.net’)

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/core/base.py:335, in device_override..device_override_decorator..device_override_decorated_function(*args, **kwds)
333     return getattr(curdev, name)(*args, **kwds)
334 else:
 → 335     return func(*args, **kwds)

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/core/network.py:642, in Network.store(self, name, filename)
639 for clock in clocks:
640     clock._set_t_update_dt(target_t=self.t)
 → 642 state = self._full_state()
643 # Store the state of the random number generator
644 dev = get_device()

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/core/network.py:601, in Network._full_state(self)
599 for obj in all_objects:
600     if hasattr(obj, “_full_state”):
 → 601         state[obj.name] = obj._full_state()
602 clocks = {obj.clock for obj in all_objects}
603 for clock in clocks:

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/input/spikegeneratorgroup.py:194, in SpikeGeneratorGroup._full_state(self)
193 def _full_state(self):
 → 194     state = super()._full_state()
195     # Store the internal information we use to decide whether to rebuild
196     # the time bins
197     state[“_previous_dt”] = self._previous_dt

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/groups/group.py:644, in VariableOwner._full_state(self)
641     if var.owner is None or var.owner.name != self.name:
642         continue  # we only store the state of our own variables
 → 644     state[var.name] = (var.get_value().copy(), var.size)
646 return state

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/core/variables.py:517, in ArrayVariable.get_value(self)
516 def get_value(self):
 → 517     return self.device.get_value(self)

File ~/miniconda3/envs/myenv/lib/python3.9/site-packages/brian2/devices/cpp_standalone/device.py:554, in CPPStandaloneDevice.get_value(self, var, access_data)
552 dtype = var.dtype
553 fname = os.path.join(self.results_dir, self.get_array_filename(var))
 → 554 with open(fname, “rb”) as f:
555     data = np.fromfile(f, dtype=dtype)
556 # This is a bit of an heuristic, but our 2d dynamic arrays are
557 # only expanding in one dimension, we assume here that the
558 # other dimension has size 0 at the beginning

FileNotFoundError: [Errno 2] No such file or directory: ‘/home/shreshth/DHC/output/results/_array_spikegeneratorgroup__spikespace_37288933’

Should I remove the input objects?
I think your last solution may not hold for cuda_standalone.

Hi again. Ok, this seems indeed to be an issue with brian2cuda (but it would also be an issue with your manual approach). I think the problem is the following: when you run a simulation in standalone mode, it writes all the state at the end of the simulation to disk. When you ask for a state variable (including the use of get_states which is also used internally by store), it will read these values from the disk. Now, there are a few internal variables that users will never access directly, but which are necessary to reproduce the full state of a network. The _spikespace variable mentioned in the error is one of those. In C++ standalone mode, we simply write all variables to disk, including these internal variables. With brian2cuda, the _spikespace variables are handled a bit differently from other variables for technical reasons, and one of the side effects is that they are not written to disk… This isn’t intentional (I think), but it usually does not matter. I think it should be rather straightforward to fix, though. I will see whether I can do this in the coming days.
However, I noted from your error message that your environment seems to be on Python 3.9. Please be aware that the latest versions of Brian2/Brian2CUDA require at least Python 3.12 (see Scientific Python - SPEC 0 — Minimum Supported Dependencies).

1 Like

Thanks!
I updated to python 3.12. The issue remains. I guess I will have to wait for the new update.