Standalone object oriented simulator implementation not working with joblib

Support
,
1

Description of problem

I have a functioning standalone simulator but it does not work with joblib because my simulator is implemented in an object oriented way. I am trying to figure out how to make it work without completely changing the implementation. I am aware of this example and I basically need the object oriented version of it.

Minimal code to reproduce problem

I can’t provide minimal code but this pseudo code should illustrate the issue:
The simulator runs like this:

simulator = Simulator()

parameters = [1, 2, 3]

result = simulator.run(parameters)

Internally, Simulator() creates and compiles the network and simulator.run(parameters) parses the parameters and uses b2.device.run(run_args=run_args) to run the simulation. This works generally well but causes problems with joblib. As I understand it, the fundamental issue is that each job needs its own compiled version of the code. So Simulator() needs to create not one but n_job simulators. Deciding how many jobs there should be at simulator instantiation time is no problem. Simulator has a self.make_network() method and I will simply call it n_job times. But the above example uses os.getpid(), and I will only know the pid when I pass simulator.run() to joblib, so I cannot use it at instantiation time. Is there anything else I can use to make the jobs not interfere with each other? Any pointers/ideas would be appreciated.

2

Hi @danielmk. I just realized that the example you pointed to is a bit out of date, in the sense that one does not necessarily need one version of the compiled code per process anymore. Instead, it is enough to compile the code once, and specify separate directories for the results so that the actual run can be in parallel (using the run_args feature). This example demonstrates the approach: Example: standalone_multiple_processes — Brian 2 0.0.post128 documentation

Would that work for you?

1 Like
3

It worked with joblib! That’s my bad for not finding this example myself. I guess I missed it because I was specifically looking for joblib. I’ll still have to deploy to the cluster but so far speedup and everything looks good. Thanks.

1 Like
4

Good to hear! I marked this as solved for now, but let us know if you run into issues on the cluster, of course.