Unxpected LED blinking observation

melanieAQ

In this initial experiment, we did not observe the expected behavior from the LED lights on the Sinara box. When we examined the signals using gtkwave, the wave forms did look correct. But when we counted the visually observable number of blinks from LED0 and LED1, some of the blinks were missing, and the blinks were not synchronized as expected.

Here is the experiment:
Assume TTL4 (output) is connected to TTL0 (input)
Generate 10 pulses on TTL4 output
In parallel, count input pulses on TTL0
See if they match

from artiq.experiment import *

class LED(EnvExperiment):

def build(self):
    self.setattr_device("core")
    self.setattr_device("led0")   # Sync with output for debugging
    self.setattr_device("led1")
    self.setattr_device("ttl4")   # Output
    self.setattr_device("ttl0")   # Input

@kernel
def run(self):
    self.core.reset()
    self.initial_state()
    with parallel:
        self.blink_output()
        self.blink_other_led()

@kernel
def initial_state(self):
    self.ttl4.off()
    self.led0.off()
    self.led1.off()

@kernel
def blink_output(self):
    for _ in range(5):
        self.ttl4.on()
        self.led0.on()
        delay(1*s)
        self.ttl4.off()
        self.led0.off()
        delay(1*s)

@kernel
def blink_other_led(self):
    for _ in range(10):
        self.led1.on()
        delay(500*ms)
        self.led1.off()
        delay(500*ms)

melanieAQ

This is what we observed visually:

led0 (labeled LED1 on the panel): blinks 5 times at 1Hz (correct)
led1 (labeled LED2 on the panel): only starts blinking at the 3rd blink of led0
and only blinks 6 times (instead of 10) at 2Hz
If we reverse blink_output and blink_other_led:
led1 blinks 10 times at 2 Hz (correct)
led0 blinks 3 times at 1 Hz, starting at the first led1 blink, but last 2 blinks are missing

sb10q

Most likely running out of SED lanes (a new lane is used whenever timestamps aren't strictly increasing, e.g. self.ttl4.on(); self.led0.on() would use a new one). You should see sequence errors reported in the log and you can fix it by adding small delays (8ns is enough) to avoid having so many lane changes (or by reordering events manually).

melanieAQ

sb10q Thanks for your speedy reply!

bruno

Also from my side, thank you for the quick response.

Indeed,

(a) We did see "ERROR:firmware.runtime.rtio_mgt:RTIO sequence error involving channel 4" error message in the log

(b) Adding 8 ns delays did fix the issue.

Is there any reason why this is not a fatal error? The fact that the output signal is simply not generated could be very problematic. Unless one notices the log message, it would very difficult to debug (considering that the Gtkwave output reports that the output signal WAS generated)

PS: As far as I can tell, the worst-case number of output events that are scheduled at the same time (with zero delay) happens at the beginning: 3 events in the initialization, 2 events in blink_output, and 1 event in blink_other_led. That is a total of 6 events, which should not exhaust the 8 available lanes.

sb10q

bruno Is there any reason why this is not a fatal error?

Yes, performance of RTIO event submission. Those errors are reported asynchronously.

sb10q

bruno PS: As far as I can tell, the worst-case number of output events that are scheduled at the same time (with zero delay) happens at the beginning: 3 events in the initialization, 2 events in blink_output, and 1 event in blink_other_led. That is a total of 6 events, which should not exhaust the 8 available lanes.

You need to count the number of timestamp reversals when the events are submitted to the RTIO core, not simply the number of simultaneous events. And note that with parallel is just executing its branches one by one with the time cursor reset in between.