qcodes dataset cannot be stored easily

[eendebakpt]

The qcodes dataset can only be stored to disk using a qcodes formatter. To save results totgether with other data it would be good to be able to serialize a dataset to a string or something like pickle, json or xml. So someting like:

s=dataset.tostring()
dataset2 = qcodes.load_dataset_string(s)

or

mystring=pickle.dumps(dataset)
dataset2=pickle.loads(mystring)

The method used for converting the dataset to a string could be taken from the formatter.

[giulioungaretti]

yes that is a problem. The entire data_set needs more love.
I am thinking and thinkering with sqlite. This should allow us to have a better life 🥇

[giulioungaretti]

@eendebakpt @AdriaanRol @damazter @MerlinSmiles

now the big question is:
- do we want binary or not ? (pickle, sqlite, hdf5) vs (json, younameit)

[MerlinSmiles]

@giulioungaretti i'd like it to be independent(no funny software to install to run a database just to get a single trace, and fast as in saving and loading a dataset. And i want to be able to load that data in 15 years from now.

[giulioungaretti]

@MerlinSmiles so no HDF5, or any thing for you just text.

[MerlinSmiles]

Im not saying just text?! I dont see the issue with HDF5, pretty standard thing? But i am very unsure about sql stuff for instance, I dont know much about it. And there are requirements for data archiving and stuff that we might not know about

[jenshnielsen]

outside the science world the use of SQL like databases is probably 1000 of times larger than HDF5 and sqlite even has build in support in python https://docs.python.org/3.5/library/sqlite3.html

[MerlinSmiles]

As I said I dont know much about it. Would just be a pitty to use something and in half a year your profs come and tell you no way.

[jenshnielsen]

Ok, makes sense I understood it as you were primarily concerned with the format going out of use in general and being to obscure

[peendebak]

@Unga @MerlinSmiles @jenshnielsen Both binary and text are fine with me. Sqlite is pretty standard, so quite future proof. However, i am not sure sqlite is really suitable for storing mixed type data (e.g. dicts containing strings, floats, numpy arrays, datasets and others dicts)

Both json and pickle do a good job of storing such structures, except for the dataset.