RTIO event submission if lane FIFO is full

clos

Based on the code in https://github.com/m-labs/artiq/blob/master/artiq/gateware/rtio/core.py and https://github.com/m-labs/artiq/blob/master/artiq/gateware/rtio/sed/lane_distributor.py, I would have expected the SED to switch lanes when a FIFO gets full. However, when I run the following MWE, it seems that the SED blocks when the fifo becomes full and only continues once events start to get taken out:
`
from artiq.experiment import delay_mu, EnvExperiment, kernel, ms

class RtioFifoFull(EnvExperiment):
def build(self):
self.setattr_device("core")
self.setattr_device("aux_trigger")

@kernel
def run(self):
    n_pulses = 100

    self.core.reset()
    delay(100 * ms)
    for _ in range(n_pulses):
        self.aux_trigger.on()
        delay_mu(40)
        self.aux_trigger.off()
        delay_mu(40)

`
This experiment underflows. Artiq version used comes from master branch of a private repo forked from m-labs/master. As far as I can tell, there are no changes in that repo compared to m-labs/master that would cause a difference in behaviour here. Looking at core analyser, it seems that all the slack gets eaten up between events 129 and 130 (one-indexed). I am surprised that it's not between events 128 and 129 as this should be when the fifo of the first lane becomes full.
Relevant bits of code:

Is this expected behaviour or a bug?

clos

@sb10q @rjo Have you been able to reproduce this behaviour on your systems?

rjo

clos You don't provide any information about your system. Likely not enough to reproduce anything interesting.
From the channel number one could speculate that this is a DRTIO channel on a satellite. Is the behavior the same if you don't core.reset() (why do you do that anyway?)? And is it the same if you use a master-attached channel?

clos

rjo More information on configuration: the channel used in the MWE is a ttl output channel on the kasli that is used as the drtio master in our setup. The channel number is misleading because for historical reasons our drtio master does not start with base address 0 in the routing table. So the underflow seems to occur even on the drtio master. I only use core.reset() because I wanted to make sure there weren't accidentally any rtio events programmed far in the future by some other experiment (there should not be, but I thought using core.reset() is more appropriate to start from a clean state than core.break_realtime()). The experiment still underflows even if core.break_realtime() is used instead.

sb10q Independently of this, is there a reason not to use enable_spread=True on a satellite? I initially encountered the blocking behaviour when updating amplitudes for a phaser oscillator on a satellite kasli. Only then did I check on the master kasli with MWE provided above. If the satellite blocks whenever a fifo is full, this would be problematic if several phaser oscillators simultaneously need to have their amplitudes updated for pulse shaping (or in fact whenever a relatively large number of rtio events with increasing timestamps need to be submitted in rapid succession without DMA). I cannot use DMA in this particular case because the phases for the oscillators need to be computed based on the timestamp at which events are submitted, which is only known at runtime.

sb10q

Satellites use enable_spread=False.
https://github.com/m-labs/artiq/blob/bdaaf3c1d7fe362b9c741f4e8c13bca1e52d8874/artiq/gateware/drtio/core.py#L65

sb10q

DRTIO takes a pessimistic estimate of the available space in a satellite which is used by the master to know when it can push more data without having to wait for an acknowledgement from the satellite. This is done for latency reduction reasons. The estimate reported by the satellites is the space available in the fullest FIFO (see what report_buffer_space=True does). So enable_spread=True would have very limited benefits there, since the previous FIFO would remain full.
You could try to implement a variant of enable_spread that switches on a FIFO high watermark instead and let us know how that goes. Note that you only need to implement it for synchronous FIFOs because (1) this is what DRTIO satellites use (2) async RTIO FIFOs will be deprecated and removed in ARTIQ-8; so it should be fairly straightforward.