artiq_influx_generic
https://gitlab.com/charlesbaynham/artiq_influx_generic
Abstract
An ARTIQ controller to interface to a local InfluxDB database. Supports transparent batching and async. queueing of writes to maximize throughput without slowing down experiment code.
Detail
InfluxDB is a time-series database optimized for storing data that naturally is associated with a timestamp (see https://www.influxdata.com/time-series-database/ for more information). ARTIQ already comes with built-in drivers that support:
- Recording Experiments submitted to the scheduler, including git hash and parameters (
artiq_influxdb_schedule)
- Storing the contents of datasets (
artiq_influxdb)
This driver instead aims to allow generic writing of data to an InfluxDB database, with the tags/values written under the control of the user. This is useful e.g. for experiments that do not fit the Experiment / Dataset model, for example, an optical clock that takes continuous data over many days/weeks, even though that data was actually acquired using several different EnvExperiment runs. Another difference vs. artiq_influxdb is that artiq_influx_generic writes data immediately, instead of at the end of an EnvExperiment.
The driver is designed to write to a single table, although support for multiple tables would be an easy extension. Writing supports any combination of tags and fields. This driver is implemented as a network controller so that it can queue data to be written asynchronously. The package includes unit tests that will fail if write performance drops below 500 writes/s - on my PC the driver can support >3500 writes/s with buffering and batching handled transparently by the driver. This allows it to achieve much higher performance than the official python API client provided by influxdb, without slowing down running experiments.
The driver also supports reading from the database, however queries are very basic, supporting only one tag filter. For more advanced querying, I'd recommend the official client instead.
Installation
For installation and sample usage, see the readme at https://gitlab.com/charlesbaynham/artiq_influx_generic.
This format
Thanks @sb10q for setting up this "contributions" channel. There's some small discussion of this format at https://github.com/m-labs/artiq/issues/1668 . The above is a example/suggestion for how contributions in this forum could be organised, i.e. with an abstract, small intro and URL. The idea is to enable people to submit their code in whatever state they want, whether that's a single-purpose driver like this, or a complete experiment repository "warts-and-all". Let me know what you think!