# Best practices for storing time series + metrics across parameter sweeps **URL:** \ **Category:** Science, Projects, and Showcases\ **Created:** [8 November 2021 18:13 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520 "2021-11-08T18:13:51Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [8 November 2021 18:13 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/1 "2021-11-08T18:13:51Z")
_(not sure whether Support is the right category since this is less about the features of Brian itself)_ I’ve been using [Pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/01_table_oriented.html#min-tut-01-tableoriented) lately, especially to store summary statistics from Brian runs. As I vary multiple parameters of the simulation I record those across columns of a data frame, and then also record output metrics as columns. This “flattens” high-dimensional parameter sweeps so each row is a single simulation run, and then I can map these back out to say a 2D slice of parameter space afterwards with panda’s functions like `groupby()`. The result looks something like this: ![table of parameters, outputs](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/87f02eb7a0a088f4e4b3483736fbc92cfced73c4.png) ## Challenge I’d like to tie the summary statistics to the primary time-series data that’s associated with it to more easily cross-reference results and verify the summary measures are quantifying what we’d expect. But this introduces time as a new index, which seems “clunky” to make work with dataframes. ## Options I’m considering 1. storing the time series as it’s own DataFrame, with time as the “instance index”/row and channels as the “feature index”/column. Then embed this time series as a new column of the dataframe a. either including the dataframe itself by nesting it b. or referencing an index / filename for where this data is stored so the higher-level dataframe doesn’t get cluttered 2. using [SKTime](https://Sktime) which is a machine learning library which seems to focus on time-series analysis and is built around extending pandas dataframes: 3. Creating a (more complicated) nested hierarchical data structure and saving it as [hdf5](https://docs.h5py.org/en/latest/quick.html) 4. What I used to do in MATLAB was to construct N-dimensional tensors and keep track of which dimension corresponded to which parameters / time index etc. ## Feedback - Are there any commonly used ways to do this within the Brian community? - Which of the above approaches seems like the best path forward? - Are there other options worth considering? ---
**Author:** ![rth](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/rth/32/16_2.png) [@rth](https://brian.discourse.group/u/rth)\ **Post date:** [10 November 2021 02:13 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/2 "2021-11-10T02:13:15Z")
@adam, a while ago, I stumbled on a similar problem with a bit of complication, that my simulations are pretty long (up to a few days in model time 😮 ). Of course, no one memory can hold even spike times for a whole run, say nothing about time series of voltages or whatever one needs from simulations. The simulator must be stopped every while, intermediate results should be saved, and all buffers must be cleared. So for this kind of data, we need a suitable file to hold chunks of data and keep track of each fragment. I’ve tried _Pandas_ and raw _hd5_ for this, but both weren’t up to the job. Specifically, _hd5_ is pretty fragile and doesn’t have enough redundancy to restore data structure when the simulation crashes with bugs and a hd-file doesn’t close properly. I lost 3 days run on this unfortunate ‘feature’ of _hd5_. I finally figured out that it isn’t a highly complex task and wrote my one python module to hold this kind of data. But this isn’t the end. Sorry for the long post. My models have lots of parameters, so I thought it may be handy to save parameters for each simulation along with very general statistics of the result, some graphs (in the form of pictures), and my comment on the run. This may be an excellent job for _Pandas_ frame or _hd5_, but after some tests, I landed to sqlight and general SQL queries. Again, I wrote a wrapper to have a tree-like structure to hold parameters. It got complicated here because I wanted to provide some help-line for each parameter and change parameters through CLI arguments (you see, I’m lazy)! So what I’m using now looks like that: #### 1. Default configuration. ```python stkdefault = """ /{ Some general words about the model The model was coded by me } /network/{ /geom/{ /shape = 'H' ; H-hexagonal or S-square /x = 7 ; number of columns /y = 16 ; number of raws } /gj/{ /geom/{ /p = 0.3 ; gap junction probability /unit = 'n' ; if n - probability of connection per neuron. it 'c' - probability of connection per pear of neurons within maxd distance /maxd = 1.5 ; gap junction max distance /dir = None ; gap junction direction (random if None) } /r = 700. ; gap junction resistance } /syn/{ /gsyn = None \; synaptic conductance: \; if positive - set for everyone, \; if negative - set the portion of minimal synaptic conductance for each neuron, ; if None set to 1/3 of the minimal synaptic conductance for each neuron. /rand = False ; If ture - gsyn randomly init between 0 and /syn/gsyn value /prenorm = True \; If true the total synaptic conductance will be set to /syn/gsyn value for each neuron in the **beginning**! ; It is active only in model building. For dynamical normalization see /network/btdp /geom/{ /pmax = None \; peak of synaptic probability for exp or exp2 distribution. ; If None closest LGN cells are connected to each rGC /sig = None ; sigma for exponential distribution. Works only if /network/syn/geom/pmax is not None /sig2 = None ; sigma for gaussian distribution. Works only if /network/syn/geom/pmax is not None and /sig is None /o2o = False ; one-to-one creates only one connection for each rGC to a closest LGN cell. /a2a = False ; all-to-all connections: each rGC connects all dLGN cells. It is the best configuration for btdp } /NMDA2AMPA = 2.25 ; ratio between peak of NMDA current to peak of AMPA current /AMPA/{ /tau1 = 1. ; AMPA rising time constant /tau2 = 2.2 ; AMPA falling time contant From Chen & Regehr 2000 Table 1 P10-P14 } /NMDA/{ /tau1 = 1. ; NMDA rising time constant /tau2 = 150. ; NMDA falling time contant From Chen & Regehr 2000 Table 1 P10-P14 } /ppr_u0 = 0.3 \; sets presynaptic single spike depression ; it set to obtain paired-pulse ratio 0.73 From Chen & Regehr 2000 Table 1 P10-P14 /toneExc = None ; Conductance for tonic excitation (None to block) /toneInh = None ; Conductance for GABA_B receptors (None to block) } ...... } """ #building model parameter tree and process a CLI arguments mth = methods(stkdefault, 'mth', locals(), argvs = args ) #and using them syn_equ=""" da1/dt = -a1/{/AMPA/tua1}/ms :1 da2/dt = -a2/{/AMPA/tua2}/ms :1 dm1/dt = -m1/{/NMDA/tua1}/ms :1 dm2/dt = -m2/{/NMDA/tua2}/ms :1 """.format(**mth["/network/syn"]) ``` #### 2. Help generation. ```bash python mymodel.py --help Some general words about the model The model was coded by me Usage: mymodel.py [options] [parameters] Any model parameter can be altered by /parameter_name=parameter_value in command line Parameters: /network/geom/shape = 'H' : H-hexagonal or S-square /network/geom/x = 7 : number of columns /network/geom/y = 16 : number of raws /network/gj/geom/p = 0.3 : gap junction probability /network/gj/geom/unit = 'n' : if n - probability of connection per neuron. it 'c' - probability of connection per : pear of neurons within maxd distance /network/gj/geom/maxd = 1.5 : gap junction max distance /network/gj/geom/dir = None : gap junction direction (random if None) /network/gj/r = 700. : gap junction resistance /network/syn/gsyn = None : synaptic conductance: if positive - set for everyone, if negative - set the : portion of minimal synaptic conductance for each neuron, if None set to 1/3 of the : minimal synaptic conductance for each neuron. /network/syn/rand = False : If ture - gsyn randomly init between 0 and /syn/gsyn value /network/syn/prenorm = True : If true the total synaptic conductance will be set to /syn/gsyn value for each neuron : in the **beginning**! It is active only in model building. For dynamical : normalization see /network/btdp /network/syn/geom/pmax = None : peak of synaptic probability for exp or exp2 distribution. If None closest LGN cells : are connected to each rGC /network/syn/geom/sig = None : sigma for exponential distribution. Works only if /network/syn/geom/pmax is not None /network/syn/geom/sig2 = None : sigma for gaussian distribution. Works only if /network/syn/geom/pmax is not None and : /sig is None /network/syn/geom/o2o = False : one-to-one creates only one connection for each rGC to a closest LGN cell. /network/syn/geom/a2a = False : all-to-all connections: each rGC connects all dLGN cells. It is the best configuration : for btdp /network/syn/NMDA2AMPA = 2.25 : ratio between peak of NMDA current to peak of AMPA current /network/syn/AMPA/tau1 = 1. : AMPA rising time constant /network/syn/AMPA/tau2 = 2.2 : AMPA falling time contant From Chen & Regehr 2000 Table 1 P10-P14 /network/syn/NMDA/tau1 = 1. : NMDA rising time constant /network/syn/NMDA/tau2 = 150. : NMDA falling time contant From Chen & Regehr 2000 Table 1 P10-P14 /network/syn/ppr_u0 = 0.3 : sets presynaptic single spike depression it set to obtain paired-pulse ratio 0.73 : From Chen & Regehr 2000 Table 1 P10-P14 /network/syn/toneExc = None : Conductance for tonic excitation (None to block) /network/syn/toneInh = None : Conductance for GABA_B receptors (None to block) ``` #### 3. Changes in parameters. Try model with different AMPA time constant as simple as ```bash python mymodel.py /network/syn/AMPA/tau2=30.5 ``` ### 3. All parameters are saved in a single database ![Screenshot_20211109_195657](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/dc773c40a96265a63eda7c9affcf74963a9d96fc.png) even with some preview ![Screenshot_20211109_195706](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/e1a8d5c007288b3e892c2a65b7ced31d3c7cc58e.png) and diff between model parameters ![Screenshot_20211109_201956](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/df924abb301610139b10d9c458ba5d77302c6439.png) **Of course, one can put lightweight statistics into a “parameter file” and analyze it later.** I usually compute mean firing rates or R2 synchronization index at the end of the run, so I can also see the diff in results with the diff in parameters. Moreover, I can collect parameters/data through the database and export it to Pandas/NumPy for analyzes. #### 4. BUT Both codes for parameter manipulation and for storing big chunks of data are combined in one _simulation toolkit_ `simtoolkit`. Unfortunately, all of this isn’t in great shape. It needs lots of development and helps with bug fixing. However, whenever I’m trying to finish documentation and create a GitHub page for this project, I’m sinking into words and postponing the GitHub repository. At this point, it works only in my hands, and I’m not sure if it will be helpful to anyone else. **I must spend a couple of days cleaning code and documenting this project at some point, but if you find it interesting to consider as an alternative, I’ll commit to doing this work asap.** RTH ---
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [10 November 2021 14:59 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/3 "2021-11-10T14:59:34Z")
Thanks for the detailed response! It sounds like your experiments are at a much larger scale than mine, but a lot of needs for data management overlap. Some of the hierarchical parameter description is quite similar to the custom classes I ended up putting together to describe my simulations. I usually divide things into specifications for: - the time-span / structure of the run - the network - the neurons - any interventions / stimulation and then try to have similarly structured specifications for any analysis / post-processing. I’ve found this to be quite effective at the small-to-medium scale of simulation both for reminding me the details of the simulation, and acting as a container for often-used post-processing steps. This also allows for easily encoding default parameter values and passing in only the modifications you want to sweep across. It seems like this functionality is also handled well with your command line options to the script! Your custom database solution sounds like it has a number of useful features, and I would be interested in trying it out! But for now it looks like a pandas dataframe for summary features + a reference to a filename containing the full timeseries (saved currently using [numpy’s `savez` format](https://numpy.org/doc/stable/reference/generated/numpy.savez.html) ) get’s the job done for me. So I don’t think it’s worth dramatically reprioritizing your time, but please keep me updated on progress! ---
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [10 November 2021 15:40 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/4 "2021-11-10T15:40:25Z")
Some further scattered thoughts as I’ve tried to cobble together a solution: ## “Wide” vs “Long”/“Tidy” data formats I’ve been working less with imperative graphing libraries (like matplotlib) where I have low-level control over the pipeline from file/database → extracting arrays → plotting arrays to declarative visualization frameworks based around “grammar of graphics” (altair / vega-lite / ggplot) where the inputs are usually a dataframe and a column name, and the library then handles extracting the data. This means there are stricter requirements on what structure the data takes by the time it gets visualized. Specifically altair / vega-lite seem to have a strong preference for a “tidy” format, i.e. one row per observation, rather than grid-like layouts. I’ve been wrestling with [pandas `melt()` function](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.melt.html) to handle this conversion but I’m currently finding it a little unintuitive. @rth if there was a simple way to ensure a “tidy-format” export / rearrangement from your database that would be great. ## Misc. considerations a good storage system - plain-text “human-readable” formats v.s. binary - this has an impact on the ability to visually inspect the integrity of the data which is very nice - but also seems to impact git’s ability to diff files. Binary files seem very slow to add to git because of this. The standard solution to this seems to be simple to avoid version-controlling binary data. - ability to load partial chunks of data - this seems to be one of the primary selling points of hdf5 - easy of integration with summary visualization tools - also integration with data-analysis tools like numpy / pandas - tight connection between metadata / summary data and primary timeseries ## Additional considerations (which are good to have, but don’t strongly impact my user experience at the moment) - memory efficiency of storage - speed of saving/loading to/from files - portability across platforms (i.e. python versions) ---
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [10 November 2021 18:01 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/5 "2021-11-10T18:01:06Z")
RTH, a colleague who works on machine learning suggested I look into [weights and biases](https://docs.wandb.ai/) which has - [incremental logging](https://docs.wandb.ai/guides/track/log) of diverse data-types - could fit with your use-case of having to stop mid-simulation to offload data - wrapper functions for managing [hyperparameter sweeps](https://docs.wandb.ai/guides/sweeps) - tools for [visualizing results in progress](https://docs.wandb.ai/guides/data-vis) I think the biggest downside is it’s optimized for machine-learning applications, but it seems flexible enough to be able to record biophysical network sims for instance. This might be worth looking into as a related solution to compare features to. ---
**Author:** ![rth](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/rth/32/16_2.png) [@rth](https://brian.discourse.group/u/rth)\ **Post date:** [10 November 2021 19:33 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/6 "2021-11-10T19:33:40Z")
> [@adam](#): > > I’ve found this to be quite effective at the small-to-medium scale of simulation both for reminding me the details of the simulation, and acting as a container for often-used post-processing steps. That was precisely my starting point. I used git as an ‘experimental’ journal for many years to keep model parameters and some messages (like old school experimentalists, who had a barn book-keeping journal to write every preparation and manipulation). BTW, the git tag system is very handy to mark some parameters that produce a particular figure, so you can return and reproduce it (or put instruction on ModelDB). However, I got more sophisticated and diverse models after a while, so git failed to keep all I wanted. Two of the critical demands: that parameters should be annotated somehow and that parameters quite often depend on each other, drove me to develop this database. My practice (maybe quite unhealthy) is to develop analytic code and visualization tools for each project I’m working on. This is almost because it isn’t possible to predict what kind of results you will have and the direction the project will go. So from this perspective, `simtoolkit` is universal without any specificity, even for computational neuroscience. > [@adam](#): > > @rth if there was a simple way to ensure a “tidy-format” export / rearrangement from your database that would be great. That is a good point, @adam. Because in DB, all parameter names are virtually in the tree structure, there are **no** reasons that extraction of tables of parameters with parameter names as the names of columns should be a problem. It’s SQL DB, and therefore, this must be only one query internally and should be quite fast (I guess). However, it isn’t implemented (yet) almost because I never use `simtoolkit` in this way. I should be more explicit here. There are two file types that `stktoolkit` will offer: `stkdb` for parameters and lightweight statistics (SQL like DB internally) and `stkdata` for chunked bulky data. `stkdata` won’t offer parallel writing but parallel reading for sure. Variables in `stkdata` also virtually in the tree structure. So yes, the data/parameters labeling and representation are consistent through the whole module. @adam, you are asking good questions. Please continue. ---
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [11 November 2021 15:36 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/7 "2021-11-11T15:36:53Z")
wonderful! sounds like there’s a lot of overlap between how we’re thinking about these issues. Your discussion of git tags reminds me of an idea I had a while ago, and partially implemented: automatically stamping a snapshot of the script name and commit id that generates a figure into the background as a “watermark.” Obviously you’d take this out of production figures. But the idea of having a default behavior of linking all figures back to their generating code (especially with minimal need for manual intervention) seems very useful **Another thought:** Are there any ways such a system could be well-integrated with brian? i.e. an automated network specification export and import. Some version of this may already exist that I’m not aware of, but if it doesn’t it seems like it would be useful, but potentially a lot of work ---
**Author:** ![rth](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/rth/32/16_2.png) [@rth](https://brian.discourse.group/u/rth)\ **Post date:** [11 November 2021 21:37 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/8 "2021-11-11T21:37:11Z")
Now I feel shame that I haven’t githubed the project yet 😮 ! > [@adam](#): > > Are there any ways such a system could be well-integrated with brian? i.e. an automated network specification export and import. Some version of this may already exist that I’m not aware of, but if it doesn’t it seems like it would be useful, but potentially a lot of work I don’t think so! Note, that stkmethods convert everything in parameters into python objects. So … ```auto stkdefaults=""" /a = 1 ; integer /b = 1.2 ; float /c = [1,2,4] ; list /d =(1,2,3) ;tuple /e = "Hello STK" ;string """ mth = methods(stkdefaults, 'mth', locals(), argvs = args ) mth.generate() # Now for n in mth: print (n, type(mth[n])) ``` and the result is ```auto /a /b /c /d /e ``` So one can put all Brian’s equations directly into the model parameter, although it would look cumbersome. Moreover, one can address a whole subtree, which can be seen as a dictionary. This also works very well, and I use it a lot! ```auto syn_equ=""" da1/dt = -a1/{/AMPA/tua1}/ms :1 da2/dt = -a2/{/AMPA/tua2}/ms :1 dm1/dt = -m1/{/NMDA/tua1}/ms :1 dm2/dt = -m2/{/NMDA/tua2}/ms :1 """.format(**mth["/network/syn"]) ``` Finally, and that is pretty handy, methods can do calculations on the fly and refer to each other. ```auto >>> stkdefaults=""" ... /a = 3 ; just a number ... /b=@/a@*12; just 12 of them ... """ >>> mth = methods(stkdefaults, 'mth', locals(), argvs = None ) >>> mth.generate() False >>> for n in mth: print(n, "=", mth[n]) ... /a = 3 /b = 36 ``` For example, I compute synaptic amplitude normalization in this way and can switch between peak and space-under-the-curve normalizations in one CLI argument `/network/syn/AMPA/norm=@/network/syn/AMPA/peack@` This normalization will be recalculated if I add `/network/syn/AMPA/tau2=1000.` I think a more in-depth integration with `brian` is possible but will make simtoolkit a more niche module. Maybe simtoolkit may have submodules that will tune it to a specific simulator more tightly: for example, import unites into defaults 😃 . P.S. I’m going to travel next few days, but I may have a bit of time to work on `simtoolkit` and push it towards github a little bit ---
**Author:** ![mstimberg](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/mstimberg/32/11_2.png) [@mstimberg](https://brian.discourse.group/u/mstimberg)\ **Post date:** [15 November 2021 15:38 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/9 "2021-11-15T15:38:44Z")
This is an amazing thread, thanks for sharing @adam and @rth ! I’ll move it to the #Science, Projects, and Showcases category for now, simply because we’d like to reserve the #Support section for things that can be answered (which makes it easier for us to keep track of open support section). But maybe we should have something like a “best practice” section for this kind of discussion? It’s been awhile since I did large-scale explorations/simulations, so in my case I can usually “get away” with a relatively straightforward directory structure storing all the raw data in a directory per simulation, and having a single csv file with all summary statistics, etc. Regarding the initial question, I’d probably simply have a unique name for each simulation stored in the data frame and have that translate to a directory and/or file name. I’ve never actually tried to put something like raw `StateMonitor` data into a pandas data frame. Some more random remarks: - @rth: Your system seems to have some similarities with `sumatra` ([code](https://github.com/open-research/sumatra), [paper](https://doi.ieeecomputersociety.org/10.1109/MCSE.2012.41)). I don’t think it has seen much development lately, though… - Regarding Brian integration: Please have a look at the export facilities in the `brian2tools` package: [Base exporter — brian2tools documentation](https://brian2tools.readthedocs.io/en/stable/user/baseexporter.html) The idea behind this was to make things available in a format that is easy to extend for more specific export facilities. The is currently a “proof-of-concept” exporter that uses this facility to export to a human-readable description in Markdown: [Markdown exporter — brian2tools documentation](https://brian2tools.readthedocs.io/en/stable/user/mdexporter.html) (shoutout to @Vigneswaran who authored all this during his Google Summer of Code). - Regarding HDF5: opinions differ on this, a lot also depends on the size of files. Here are blog posts arguing for the different sides: “Against HDF5”: [Cyrille Rossant - Moving away from HDF5](https://cyrille.rossant.net/moving-away-hdf5/) and [Cyrille Rossant - Should you use HDF5?](https://cyrille.rossant.net/should-you-use-hdf5/) , “In favour of HDF5”: [http://blog.khinsen.net/posts/2016/01/07/on-hdf5-and-the-future-of-data-management/](http://blog.khinsen.net/posts/2016/01/07/on-hdf5-and-the-future-of-data-management/) I have to say I’m rather in the “anti-HDF5” camp, I quite like the directory-based approach, e.g. the one detailed here: [https://www.frontiersin.org/articles/10.3389/fninf.2018.00016/full](https://www.frontiersin.org/articles/10.3389/fninf.2018.00016/full) - Potentially interesting to some of you: The `pypet` package helps to organize parameter explorations and has some built-in support for Brian: [Welcome to pypet’s documentation — pypet 0.5.0 documentation](https://pypet.readthedocs.io/en/latest/?badge=latest) - Regarding versioning big binary data: This is typically handled via [git lfs](https://git-lfs.github.com/) or [git annex](https://git-annex.branchable.com/) (basically it stores the data elsewhere, and just puts a text file with the link to the data into the git repo). There are also some more sophisticated solutions built on top of this, e.g. [datalad](https://www.datalad.org/). > [@adam](#): > > Your discussion of git tags reminds me of an idea I had a while ago, and partially implemented: > automatically stamping a snapshot of the script name and commit id that generates a figure into the background as a “watermark.” > Obviously you’d take this out of production figures. But the idea of having a default behavior of linking all figures back to their generating code (especially with minimal need for manual intervention) seems very useful Hmm, interesting thought. I wonder why it is not more common to use a figure’s metadata for it! E.g. with matplotlib’s `savefig` you can do something like `plt.savefig('myfig.png', metadata={'script_name': os.path.abspath( __file__ )`) – there are some standard meta data entries (creator, etc.), but you can add arbitrary keys. ---
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [15 November 2021 18:52 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/10 "2021-11-15T18:52:57Z")
Thanks, there’s lots of great advice / resources here. It seems like the simplest, robust-enough solution currently is indeed to have summary stats in plaintext / csv / yaml then reference the time series as filename to say a numpy binary. In putting together the notebook for the gaussian delay issue I finally bit the bullet on figuring out how [pandas hierarchical indexing works](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html#advanced-indexing-with-hierarchical-index). Here’s a table where the highest index is the neuron group name, and the lower index is the neuron id within that group: ![Screen Shot 2021-11-15 at 1.37.29 PM](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/77d1a05c0950de63448004ea4124478f7b8b856c.png) this is particularly convenient for plotting libraries that like DataFrames (plotly, altair etc.) where you can create all your subplots with: ```python fig = px.line(dfm, x='time [ms]' ,y='voltage', color='population, facet_row='total_neuron_idx') ``` (with the caveat that this requires “melting” the dataframe in a slightly unintuitive way) ![hierarchical_voltage](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/7815fbe62f8db5c0365157db9978d0a712977eba.png) same data plotted, faceting along different dimensions, only changing a single line of code: ```python fig = px.line(dfm, x='time [ms]', y='voltage', facet_row='population', color='neuron') ``` ![hierarchical_voltage2](https://global.discourse-cdn.com/free1/uploads/brian/original/1X/22d76ab0c7ed175cbb5f5fa61c9e1607e2fac808.png) * * * This snippets for how I achieved this will be in the gaussian delay notebook, but I’m happy to turn them into a pull request if they end up being useful. I saw that there’s been some prior work in exporting Brian traces to pandas dataframes: - [storing current state to dataframe](https://brian2.readthedocs.io/en/stable/user/models.html#storing-state-variables) using `get_states()` - related github issue [Exporting/importing Brian data from/to other formats](https://github.com/brian-team/brian2/issues/306#)#306 - [PandasImportExport class docs](https://brian2.readthedocs.io/en/2.4.2/reference/brian2.importexport.dictlike.PandasImportExport.html) but as far as I can tell this can only store 1D variables to dataframes, is this correct? EDIT: now under version control at [GitHub - awillats/brian\_delayed\_gaussian](https://github.com/awillats/brian_delayed_gaussian) ---
**Author:** ![mstimberg](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/mstimberg/32/11_2.png) [@mstimberg](https://brian.discourse.group/u/mstimberg)\ **Post date:** [16 November 2021 20:49 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/11 "2021-11-16T20:49:52Z")
> [@adam](#): > > I saw that there’s been some prior work in exporting Brian traces to pandas dataframes: > > - [storing current state to dataframe](https://brian2.readthedocs.io/en/stable/user/models.html#storing-state-variables) using `get_states()` > - related github issue [Exporting/importing Brian data from/to other formats](https://github.com/brian-team/brian2/issues/306#)#306 > - [PandasImportExport class docs](https://brian2.readthedocs.io/en/2.4.2/reference/brian2.importexport.dictlike.PandasImportExport.html) > > but as far as I can tell this can only store 1D variables to dataframes, is this correct? This is correct, the pandas export only handles 1D arrays. In fact, it even fails with an ugly error message for 2D arrays 🙁 It would be great to support this somehow… The general architecture of the exporter is to deal with each variable individually (and you can ask `get_states` to only return a single variable). This approach does not quite work for recorded 2D array, since you need at least the `t` variable to interpret things. But maybe it is worth changing the export so that it handles variables like `i` and `t` in a special way? ---
**Author:** ![adam](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/adam/32/197_2.png) [@adam](https://brian.discourse.group/u/adam)\ **Post date:** [16 November 2021 20:58 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/12 "2021-11-16T20:58:59Z")
yeah, for my example above, my wrapper function assigns the channels of the monitor output to the corresponding column of the dataframe, then appends the `time [ms]` column afterwards. It makes sense to me for export methods to always add `t` or at least `i` as mandatory indexing variables. One minimal way to do this with pandas would be to assign the simulation index to the row index with the [set\_index() method](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.set_index.html). This would mean not having to add an additional column just to do this job. You could then have an optional `append_timestamp` flag to include `t` as a new column EDIT: I have my work-in-progress implementation for expanding voltage monitors to hierarchical dataframes here: > ---
**Author:** ![kjohnsen](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/kjohnsen/32/200_2.png) [@kjohnsen](https://brian.discourse.group/u/kjohnsen)\ **Post date:** [20 January 2025 20:10 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/13 "2025-01-20T20:10:33Z")
Reviving this thread since I’m doing a large scale exploration of parameters myself now and am running into the unfortunate fact that I can’t just pickle the whole Brian `Network`… I’m experiencing some remorse at not having read this thread sooner, but the solution I’ve landed on fares comparably well with these, I think. I approached this problem from the “workflow engine” perspective. The reason is, these systems (like make) can cache intermediate results and rerun only what needs to be run. So no need to manually walk through part of a complex pipeline after changing parameter X. My pipeline with Snakemake looks like this: I define the step (“rule”) of my workflow that produces a figure, which depends on a table of results tabulated across conditions. This table is defined in another rule of my workflow and is put together from the results of individual conditions. When I ask for a given figure, conditions that haven’t been run or need to be rerun (due to changes in parameters or input files) and the data I need is saved (spike monitor data, summary statistics, etc.). The post-processing rules put together the table and the figure. Each condition’s data is saved according to a programmatically generated filename, so I can go back and explore individual runs. Snakemake even has a “report” generation feature that lets you browse results. For why I chose Snakemake (there are too many workflow engines out there), here are the 3 top candidates I found: - [redun](https://github.com/insitro/redun) - looks like a great design, automatically caches outputs of functions and tracks dependence on data, but not great adoption - [prefect](https://prefect.io) - similar to redun, came earlier and has much bigger adoption. seems to be big in ML data engineering - [snakemake](https://snakemake.github.io/) - built for bioinformaticists, pretty great for scientific computing projects with file intermediates, more tailored to academic research than data engineering I ended up going with Snakemake since it had SLURM submission support, though I later found that through Dask you can accomplish the same with Prefect. I would try Prefect if I did it over again. ---
**Author:** ![mstimberg](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/mstimberg/32/11_2.png) [@mstimberg](https://brian.discourse.group/u/mstimberg)\ **Post date:** [22 January 2025 15:52 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/14 "2025-01-22T15:52:52Z")
Hi @kjohnsen. Thanks for sharing your experience. I’ve never used a full workflow engine in this context, I’ve only used functions decorated with [joblib’s Memory](https://joblib.readthedocs.io/en/stable/memory.html) class in order to not re-calculate values for known parameter values. But this is obviously less powerful and only works for relatively simple setups. I’ve used `snakemake` a bit and also heard good things about it in general. I am curious: what do you think is missing from it that leads you to say “I would try Prefect if I did it over again”? As an aside: a long time ago, Andrew Davison created a quite nice tool called `sumatra` which was specifically tailored to the use case of running simulations with different parameters. It does not seem to be developed any more, though: [GitHub - open-research/sumatra](https://github.com/open-research/sumatra) ---
**Author:** ![kjohnsen](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/kjohnsen/32/200_2.png) [@kjohnsen](https://brian.discourse.group/u/kjohnsen)\ **Post date:** [22 January 2025 16:34 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/15 "2025-01-22T16:34:33Z")
I did see sumatra from the thread above, but that it wasn’t developed anymore. Like I said, there are way, way, way too many workflow engines out there that can do similar things, so I tried to narrow down choices to the biggest/best ones. Prefect has two main advantages over Snakemake from what I can tell: wider adoption (~17k vs. ~3k GitHub stars) and caching task results automatically (under the hood serializing and storing in a DB). This is much easier for a pure Python workflow. By contrast, Snakemake works with file intermediates between steps, so if you’re using Python (like I am with Brian), you need to go through the extra trouble of serializing and unserializing between steps. This makes more sense for bioinformatics where you’re using a bunch of (other people’s) command line tools that use files for input and output. I chose Snakemake mainly because it had support for SLURM execution, but I since found that this is possible (if not as fully featured) on Prefect through its Dask executor. In general, Snakemake is more tailored to research while Prefect is more tailored to industry ML/data engineering. ---
**Author:** ![mstimberg](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/mstimberg/32/11_2.png) [@mstimberg](https://brian.discourse.group/u/mstimberg)\ **Post date:** [22 January 2025 16:44 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/16 "2025-01-22T16:44:10Z")
Many thanks for the clarification, I definitely need to look into Prefect more. On a slightly unrelated note : Do you know about the OCNS/INCF Software Working group (disclosure: @sanjayankur31 and I are co-chairing it 😉 ) [Home | INCF/OCNS Software WG](https://ocns.github.io/SoftwareWG/) ? This kind of topic would be interesting to present/discuss in that context, I think. ---
**Author:** ![kjohnsen](https://yyz2.discourse-cdn.com/free1/user_avatar/brian.discourse.group/kjohnsen/32/200_2.png) [@kjohnsen](https://brian.discourse.group/u/kjohnsen)\ **Post date:** [22 January 2025 19:52 UTC](https://brian.discourse.group/t/best-practices-for-storing-time-series-metrics-across-parameter-sweeps/520/17 "2025-01-22T19:52:33Z")
I would be happy to contribute to a WG discussion on this, though I will probably be more useful in a few months once I have this Snakemake project running end-to-end—this is the first time I’ve used it.