Save HDF5 into custom folder without `artiq_run -o`

ngkiaboon

I understand that ARTIQ saves

artiq_version
datasets (from set_dataset(...))
expid (which contains log_level, file ran, class used, and arguments from setattr_argument(...))
rid
run_time
start_time
into an HDF5 file stored under ~/artiq-master/results/<date>/<hour> each time a FILE is submitted and ran.

How do I get it to save under a custom folder? I understand that I can use artiq_run -o HDF5 FILE to save it to HDF5, but can I also do it from the dashboard or specify it from within the FILE itself?

This is so that I can do some folder management on my side to group HDF5 files according to the type of the experiment instead of the time of the experiment. I would be happy to learn of other ways to do this (other than what I mentioned above); it is just that the experimental contexts are not immediately obvious from the HDF5 filenames and directories in which they are stored before we open them and see what is inside.

sb10q

This is not currently implemented.

Nefelibata

Is the functionality of artiq_run -o possible when submitting the experiment through artiq_client rather than using artiq_run?

sb10q

The master will save a HDF5 file already.

Nefelibata

sb10q Right, but is it possible to specify the name of the file to which it should save? Or the file location?

sb10q

You would need to modify the ARTIQ source code for this.

choelzl

The master worker of an experiment saves the hdf5 file in the current working directory, so the trick we do isto just change that in the prepare phase and delete the originally created folders. It is what we are using to get rid of the hour folders. It is probably not the most nice method but it didn't cause any problems in the last two years. Just make sure that you run it in every experiment so you don't get inconsistent folders.

       try:
            hourdir = os.path.join("..", os.getcwd().split("/")[-1])
            dirname = os.path.join("..","{:09}-{}".format(self.scheduler.rid, self.__class__.__name__))
            os.makedirs(dirname, exist_ok=True)
            os.chdir(dirname)
            self.saveDir = os.getcwd()
            try:
                os.removedirs(hourdir)
            except (FileNotFoundError, OSError):
                pass
        except AttributeError as e:
            #is run as DummyDevice (for example when generating the dashboard)
            pass