Urukul underflow at dds init function

LTGiardino

Hello,

We've been experimenting some strange (strange to us, at least) Underflow Errors during some experiments, which used to run properly a week or so ago.
They arise at the dds (ad9910) channel initialization step.

Below is a minimal example with the corresponding traceback call.
We are using a custom wrapper class around Urukul channels for simplicity, we included a minimal version in this example code (ofc feedback and improvements are welcomed!) .

The error at least appears consistent between runs (i.e. it usually raises with slack ~ -5000mu).
We've tried adding delays before each step on our code to increase slack, but to no success.

Sorry if this is a stupid question but could it be that we somehow need to "clear" or "free" the channel after being used in a experiment before initiating it again in a new one?

Thanks!!

Artiq version: ARTIQ v5.7158.beb49d4d (planning on updating to 6 soon)

Working example code:

from artiq.experiment import *

class UrukulCh(HasEnvironment):

    def build(self, ch=0, freq=100.0, amp=1.0):
        self.ch = ch
        self.amp = amp
        self.freq = freq
        
        self.channel = self.get_device(f"urukul0_ch{self.ch}")
        
        self.frequency = self.get_argument(f"freq_ch{self.ch}",
                            NumberValue(self.freq*MHz, unit='MHz', scale=MHz, min=1*MHz, max=400*MHz),
                            f"channel_{self.ch}")

        self.amplitude = self.get_argument(f"amp_ch{self.ch}",
                            NumberValue(self.amp, min=0., max=1.),
                            f"channel_{self.ch}")

    @kernel
    def initialize_channel(self):
        # initialises CPLD the selected channel
        self.channel.cpld.init()
        self.channel.init()

    @kernel
    def set_channel(self):
        self.channel.set(self.frequency, amplitude=self.amplitude)

    @kernel
    def set_o(self, state):
        self.channel.sw.set_o(state)

class TestUnderflow(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.laserIR = UrukulCh(self, ch=1, freq=208.0, amp=0.35)
        
    @kernel
    def run(self):
        self.core.reset()
        self.laserIR.initialize_channel()
        self.core.break_realtime()
        self.laserIR.set_channel()
        
        delay(10*us)

        # Just try to pulse the laser
        self.laserIR.set_o(True)
        delay(1*ms)
        self.laserIR.set_o(False)

Traceback:

root:Terminating with exception (RTIOUnderflow: RTIO underflow at 3929960246184 mu, channel 8, slack -5384 mu)
Core Device Traceback (most recent call last):
  File "---/minimal_example_underflow.py", line 44, in artiq_worker_minimal_example_underflow.TestUnderflow.run(..., ...) (RA=+0xa8c)
      self.laserIR.initialize_channel()
  File "---/minimal_example_underflow.py", line 25, in ... artiq_worker_minimal_example_underflow.UrukulCh.initialize_channel<artiq_worker_minimal_example_weird_underflow.UrukulCh>(...) (inlined)
      self.channel.init()
  File "<artiq>/coredevice/ad9910.py", line 422, in ... artiq.coredevice.ad9910.AD9910.init<artiq.coredevice.ad9910.AD9910>(...) (RA=+0x1c14)
      sta = self.cpld.sta_read()
  File "<artiq>/coredevice/urukul.py", line 226, in ... artiq.coredevice.urukul.CPLD.sta_read<artiq.coredevice.urukul.CPLD>(...) (inlined)
      self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END | spi.SPI_INPUT, 24,
  File "<artiq>/coredevice/spi2.py", line 169, in ... artiq.coredevice.spi2.SPIMaster.set_config_mu<artiq.coredevice.spi2.SPIMaster>(...) (inlined)
      rtio_output((self.channel << 8) | SPI_CONFIG_ADDR, flags |
  File "ksupport/rtio.rs", line 67, column 14, in (Rust function)
      <unknown>
                       ^
 artiq.coredevice.exceptions.RTIOUnderflow(0): RTIO underflow at 3929960246184 mu, channel 8, slack -5384 mu

sb10q

Strange. Are you sure the code as posted underflows? Do you actually need the rest of the code to reproduce the problem, or is this sufficient:

    @kernel
    def run(self):
        self.core.reset()
        self.laserIR.channel.cpld.init()
        self.laserIR.cpld.init()

LTGiardino

Tried your proposal and there was also the same Underflow Error.

We did the sensible thing and rebooted both artiq and local computer.
It seems now everything's up and running ok.

This error puzzles me still, I suspect it will arise again in the future.
Also, doing a power cycle every time an error occurs doesn't seem like a proper fix.
Do you have any insight as to why random underflows like these may occur? Because we've had some other underflows randomly appearing before, luckily they were gone after a re-submit or so.

Could it be due to not doing a good job at scheduling/terminating experiments?
Sorry for the wild guesses, I'm just trying to wrap my head around the whole artiq system.

sb10q

The simple code I posted isn't supposed to underflow regardless.

jsch

We see the same issue on artiq 7.8123.3038639. Most of the time the code below will run without issue, but occasionally the system gets in a state where it underflows at the self.urukul0_ch0.init() line repeatedly. Not even a break_realtime and 5s delay will prevent the underflow. Sometimes the issue disappears after repeated attempts to run the code, sometimes we resort to restarting the master (this sometimes seems to work, but perhaps not always) or power cycling the kasli. I cannot correlate this behavior with anything else about the system state or other lines of code. It is possible that the issue pops up more when there are many Scannables set to arguments as in the commented out line. It is also possible a complicated prepare method can make the issue worse, but I can't say either of these things with confidence. Would appreciate any insight on where the issue might be coming from. Happy to run some test to try to sort it out as it is a fairly frequent nuisance in our lab.

from artiq.experiment import *
import numpy as np

class UrukulUnderflow(EnvExperiment):
	def build(self):
		self.setattr_device('core')
		self.setattr_device('ttl0')
		self.setattr_device('urukul0_ch0')

		## self.setattr_argument('test_arg', Scannable(global_max=40, global_min=-40, ndecimals=0, default=[RangeScan(-20, 20, 40)])) # maybe makes worse?

	@kernel
	def run(self):
		self.core.reset()
		self.urukul0_ch0.init()

sb10q

Maybe a delay inside the driver needs to be increased, check the underflow trace carefully and look into the driver.