Judging from the survey results, it seems a number of people have trouble with RTIOUnderflow and want a graphical tool to display slack.

Such functionality already exists through artiq_coreanalyzer and GtkWave (development of the few patches required to make it work on the GtkWave side was funded by Joe a long time ago).

Take the following experiment:

from artiq.experiment import *


class BlinkForever(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("led")

    @kernel
    def run(self):
        self.core.reset()
        while True:
            self.led.pulse(.1*us)
            delay(.1*us)

which underflows:

> artiq_run repository/blink_forever.py 
Core Device Traceback (most recent call last):
  File "repository/blink_forever.py", line 13, in artiq_run_blink_forever.BlinkForever.run(..., ...) (RA=+0x3e4)
    self.led.pulse(.1*us)
  File "<artiq>/coredevice/ttl.py", line 79, in ... artiq.coredevice.ttl.TTLOut.pulse<artiq.coredevice.ttl.TTLOut>(...) (inlined)
    self.on()
  File "<artiq>/coredevice/ttl.py", line 53, in ... artiq.coredevice.ttl.TTLOut.on<artiq.coredevice.ttl.TTLOut>(...) (inlined)
    self.set_o(True)
  File "<artiq>/coredevice/ttl.py", line 45, in ... artiq.coredevice.ttl.TTLOut.set_o<artiq.coredevice.ttl.TTLOut>(...) (inlined)
    rtio_output(self.target_o, 1 if o else 0)
  File "ksupport/rtio.rs", line 67, column 14, in (Rust function)
    <unknown>
                 ^
artiq.coredevice.exceptions.RTIOUnderflow(0): RTIO underflow at 269814732751296 mu, channel 19, slack -280 mu
Traceback (most recent call last):
  File "/nix/store/28b9qy0b45zwdk9rdszf82n5m7pdc001-python3.8-artiq-7.7613.eb38b664.beta/bin/.artiq_run-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/frontend/artiq_run.py", line 225, in main
    return run(with_file=True)
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/frontend/artiq_run.py", line 211, in run
    raise exn
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/frontend/artiq_run.py", line 204, in run
    exp_inst.run()
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/language/core.py", line 54, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/coredevice/core.py", line 137, in run
    self.comm.serve(embedding_map, symbolizer, demangler)
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/coredevice/comm_kernel.py", line 644, in serve
    self._serve_exception(embedding_map, symbolizer, demangler)
  File "/nix/store/3ids0mmd1wbdsm7ahrjr1r8bv5ak3x1s-python3-3.8.8-env/lib/python3.8/site-packages/artiq/coredevice/comm_kernel.py", line 636, in _serve_exception
    raise python_exn
artiq.coredevice.exceptions.RTIOUnderflow: RTIO underflow at 269814732751296 mu, channel 19, slack -280 mu

You can then get a VCD trace of the event using:

artiq_coreanalyzer -w underflow.vcd

The VCD file can then be opened in GtkWave. In the signal list, select rtio_slack[63:0] and ttl/led, then click Append. Then right click rtio_slack[63:0] in the graph legend, and select Data Format > BitsToReal followed by Data Format>Analog>Step (or "Interpolated" if you want a smoother display).

The RTIO slack is then plotted alongside the TTL state:

Under the hood, the value of the RTIO timestamp counter (TSC) is stored when an event is submitted. Then, at each event, the difference between the TSC and the time of the event is plotted as rtio_slack.

Let's look at a more interesting example:

    @kernel
    def run(self):
        self.core.reset()
        for i in range(10):
            for i in range(30):
                self.led.pulse(10*ns)
                delay(10*ns)
            delay(50*us)
            for i in range(30):
                self.led.pulse(10*ns)
                delay(10*ns)

On the "interpolated" display, the slack can clearly be seen to be consumed by the short pulses, and recovering during the 50-microsecond delay:

It is also possible to look at the actual value of the slack (in seconds) for any given output event, by disabling the "analog" plotting and/or by looking at the value on the left panel:

I tried the example, but I get an error. My system is running on ARTIQ 6.7480.a97b4633.beta, OS Windows 10, Anaconda package manager.
I can execute the experiment with no issue and get the RTIO Underflow. Then I execute the command artiq_coreanalyzer -w underflow.vcd and immediately get the error WARNING:artiq.coredevice.comm_analyzer:unable to determine DDS sysclk. Nothing happens afterwards and the shell just prompts again.
I could not find any such error in the documentation, but it does appear in two issues: https://github.com/m-labs/artiq/issues/1035 and https://github.com/m-labs/artiq/issues/1376
The first one mentions that SAWG is not compatible with the analyzer, but I do not use this module (either in this experiment or in general). The second issue is connected to the DDS boards and some sysclock entry, but the device_db entry looks very different from mine.


device_db["urukul0_cpld"] = {
    "type": "local",
    "module": "artiq.coredevice.urukul",
    "class": "CPLD",
    "arguments": {
        "spi_device": "spi_urukul0",
        "sync_device": None,
        "io_update_device": "ttl_urukul0_io_update",
        "refclk": 100000000.0,
        "rf_sw": 0b0000,
        "clk_sel": 1
    }
}

device_db["urukul0_ch0"] = {
    "type": "local",
    "module": "artiq.coredevice.ad9912",
    "class": "AD9912",
    "arguments": {
        "pll_n": 10,
        "chip_select": 4,
        "cpld_device": "urukul0_cpld",
        "sw_device": "ttl_urukul0_sw0"
    }
}

I also tried to specify the device_db explicitly as shown in this issue, but that caused no change.
What is the error here? Do I need to specify the DDS sysclock specifically as well? I see, that the underflow file is still generated.

    steine01 immediately get the error WARNING:artiq.coredevice.comm_analyzer:unable to determine DDS sysclk.

    Ignore it.

    Nothing happens afterwards and the shell just prompts again.

    Wasn't the VCD file created?

      Sorry, I didn't write that very clearly. The VCD file is created and I can open it with GtkWave. The data also is included as far as I can tell. I am mainly worried about that warning.

      Comment from student in my group...

      I’d say its moderately helpful, although in experiments where the slack varies fairly significantly (e.g. large slack after 3 ms of cooling, but little slack between many rapid pulses with computations, changing urukul settings, detection) it becomes hard to resolve the details of what is going on.

      Since the point of plotting slack in GtkWave is quickly identifying when slack is approaching zero it would be helpful to be able to plot log10(rtio_slack).

      From my mental notes supporting users with RTIo issues, some problem with using gtkwave and vcd to debug RTIO issues, certainly for beginners are:

      (a) The gigantic dynamic range in time scales ob both horizontal and vertical scale. Need to point out the --vcd-uniform-interval. When used: no indication of timescale.
      (b) Split brain with tooling: looping through writing an experiment, running it, downloading the trace, and displaying it. Maybe some parsed/processed version of the trace can automatically be emitted if we exit and traceback with RTIOUnderflow.
      (c) General user interface difficulty with gtkwave archaisms. Could be improved somewhat with generating .gtkw as well.
      (d) Lack and/or semantic breaks/inaccuracies of RTIO data parsers for most coredevice drivers.
      (e) The trace can't be understood without understanding RTIO. It doesn't explain or give any hint to why slack is what it is in many cases (which is the question users ask themselves). One could write a user guide that explains both in parallel though.

      I agree; the purpose of my post was to point out to people who already have some familiarity with RTIO that such a tool exist (even if it's not perfect) since it's not obvious.
      There were some discussions about integrating the analyzer and a waveform plotter with the dashboard. Maybe there could be an IDE or IDE plugins for ARTIQ as well.