I have multiple experiments that will be submitted where I would like them to be able to pass data back and forth. For example, I have a background experiment that calls set_dataset("key", value0, broadcast=True). Then another experiment with a higher priority is submitted, which updates the same dataset via set_dataset("key", value1, broadcast=True). Then when the higher priority experiment terminates I want the background experiment to be able to access the new value value1 that the higher priority experiment wrote to the dataset via get_dataset("key").
However, from the manual and confirmed by my experimenting:
The local storage is searched first, followed by the master storage (which contains the broadcasted datasets from all experiments) if the key was not found initially.
Currently the first experiment always returns value0 even though the most recent dataset value is value1 presumably because it only searches the local storage (both experiments are running on the same crate in the same pipeline though). Is there no way to force the experiment to search the master storage?