Reviving this thread since I’m doing a large scale exploration of parameters myself now and am running into the unfortunate fact that I can’t just pickle the whole Brian Network…
I’m experiencing some remorse at not having read this thread sooner, but the solution I’ve landed on fares comparably well with these, I think. I approached this problem from the “workflow engine” perspective. The reason is, these systems (like make) can cache intermediate results and rerun only what needs to be run. So no need to manually walk through part of a complex pipeline after changing parameter X.
My pipeline with Snakemake looks like this: I define the step (“rule”) of my workflow that produces a figure, which depends on a table of results tabulated across conditions. This table is defined in another rule of my workflow and is put together from the results of individual conditions. When I ask for a given figure, conditions that haven’t been run or need to be rerun (due to changes in parameters or input files) and the data I need is saved (spike monitor data, summary statistics, etc.). The post-processing rules put together the table and the figure. Each condition’s data is saved according to a programmatically generated filename, so I can go back and explore individual runs. Snakemake even has a “report” generation feature that lets you browse results.
For why I chose Snakemake (there are too many workflow engines out there), here are the 3 top candidates I found:
- redun - looks like a great design, automatically caches outputs of functions and tracks dependence on data, but not great adoption
- prefect - similar to redun, came earlier and has much bigger adoption. seems to be big in ML data engineering
- snakemake - built for bioinformaticists, pretty great for scientific computing projects with file intermediates, more tailored to academic research than data engineering
I ended up going with Snakemake since it had SLURM submission support, though I later found that through Dask you can accomplish the same with Prefect. I would try Prefect if I did it over again.