Ignored delays and infinite wait after installing satellite

steine01

I installed a satellite and can initialize the devices on both the satellite and the master with the startup kernel successfully (e. g. I see that the red leds of the Urukul units turn off once I initialize them). Afterwards I tried to check that I can get an output on the TTLs and Urukul channels. For the TTL I used the following program and check with a voltmeter:

class Idle(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl4")
@kernel
    def run(self):
        self.core.break_realtime()
        logger.info("A")
        self.ttl4.on()
        logger.info("B")
        delay(5*s)
        logger.info("C")
        self.ttl4.off()
        logger.info("D")
        delay(5*s)
        logger.info("E")
        self.ttl4.on()
        logger.info("F")
        delay(5*s)
        logger.info("G")
        self.ttl4.off()
        logger.info("H")
        self.core.wait_until_mu(now_mu())

Instead of a different voltage every 5 seconds and the corresponding slow log messages, I get no voltage change and all messages (A-H) appear instantly on the dashboard console. Then the program never exits and I have to delete it. If I try the reverse and read a TTL instead, it also hangs at the line where some response is expected:

class Idle(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl0")
@kernel
    def run(self):
        self.core.break_realtime()
        logger.info("A")
        self.ttl0.gate_rising(80*ns)
        logger.info("B")
        try:
            logger.info("B1")
            self.ttl0.count(now_mu())
            logger.info("B2")
        except RTIOOverflow:
            logger.info("B3")
            pass
        logger.info("C")

So I see A, B and B1 immediately in the console and then it hangs indefinitely. I see similiar behaviour for the Urukuls. What is the problem here? All of these devices are on the master, so I am confused why the satellite should change something. I also checked the device_db entries of these devices before the switch and now and did not find any difference. This is the current device_db entry for the core and TTL devices:


device_db = {
    "core": {
        "type": "local",
        "module": "artiq.coredevice.core",
        "class": "Core",
        "arguments": {"host": core_addr, "ref_period": 1e-09},
    },
    "core_log": {
        "type": "controller",
        "host": "::1",
        "port": 1068,
        "command": "aqctl_corelog -p {port} --bind {bind} " + core_addr
    },
    "core_cache": {
        "type": "local",
        "module": "artiq.coredevice.cache",
        "class": "CoreCache"
    },
    "core_dma": {
        "type": "local",
        "module": "artiq.coredevice.dma",
        "class": "CoreDMA"
    },

    "i2c_switch0": {
        "type": "local",
        "module": "artiq.coredevice.i2c",
        "class": "PCA9548",
        "arguments": {"address": 0xe0}
    },
    "i2c_switch1": {
        "type": "local",
        "module": "artiq.coredevice.i2c",
        "class": "PCA9548",
        "arguments": {"address": 0xe2}
    },
}

# master peripherals

device_db["ttl0"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLInOut",
    "arguments": {"channel": 0x000000},
}


device_db["ttl1"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLInOut",
    "arguments": {"channel": 0x000001},
}


device_db["ttl2"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLInOut",
    "arguments": {"channel": 0x000002},
}


device_db["ttl3"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLInOut",
    "arguments": {"channel": 0x000003},
}


device_db["ttl4"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLOut",
    "arguments": {"channel": 0x000004},
}


device_db["ttl5"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLOut",
    "arguments": {"channel": 0x000005},
}


device_db["ttl6"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLOut",
    "arguments": {"channel": 0x000006},
}


device_db["ttl7"] = {
    "type": "local",
    "module": "artiq.coredevice.ttl",
    "class": "TTLOut",
    "arguments": {"channel": 0x000007},
}

sb10q

steine01 I get no voltage change

Is your TTL channel configured as output?

and all messages (A-H) appear instantly on the dashboard console.

This is expected - those delays only affect timestamps of scheduled RTIO operations, not program execution.

steine01

@sb10q Yes I set the hardware switch to output (the green light on the panel is on) and the TTL itself is only configured as a TTLOut in the device_db. In the past, I also used this TTL successfully.

Just so I am not going crazy: this should still mean, that I get a change of the voltage every 5 seconds, no? The parts where the TTL is high should be equivalent to self.ttl.pulse(5*s), right?

sb10q

Yes, TTL on()/off() are RTIO operations and follow the delay() calls.

steine01

After some experimenting, it looks like self.core.break_realtime() is the issue. If I replace it with self.core.reset() it works. If I remove that line it also does not work.

To be clear, this experiment hangs indefinitely and produces no output on the TTL.

class Idle(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl4")
    @kernel
    def run(self):
        self.core.break_realtime()
        self.ttl4.on()
        delay(5*s)
        self.ttl4.off()
        delay(5*s)
        self.ttl4.on()
        delay(5*s)
        self.ttl4.off()
        self.core.wait_until_mu(now_mu())

This experiment finishes and I see the expected voltage changes on the TTL:

class Idle(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl4")
    @kernel
    def run(self):
        self.core.reset()
        self.ttl4.on()
        delay(5*s)
        self.ttl4.off()
        delay(5*s)
        self.ttl4.on()
        delay(5*s)
        self.ttl4.off()
        self.core.wait_until_mu(now_mu())

I assume this is not what is intended. What issue could the satellite cause here, that self.core.reset() solves?

sb10q

Previous experiment setting the time cursor far into the future and seamless handover picking up that position. And you need at least one core.reset() done by one experiment to initialize the position correctly.

steine01

The previous experiment was startup, where I do a core.reset() after waiting for the uplinks and then initialize all devices:

class Startup(EnvExperiment):
    """Startup kernel.

    Resets and initializes devices and establishes default parameters.

    This should be used as the persistent startup kernel and can also be used
    to re-initialize devices.
    """
    def build(self):
        self.setattr_device("core")

        self.setattr_device("sampler0")
        
        self.setattr_device("zotino0")
        self.setattr_device("zotino1")
        
        self.setattr_device("ttl0")
        self.setattr_device("ttl1")
        self.setattr_device("ttl2")
        self.setattr_device("ttl3")
        
        self.setattr_device("ttl24")
        self.setattr_device("ttl25")
        self.setattr_device("ttl26")
        self.setattr_device("ttl27")

        self.setattr_device("urukul0_cpld")
        self.setattr_device("urukul0_ch0")
        self.setattr_device("urukul0_ch1")
        self.setattr_device("urukul0_ch2")
        self.setattr_device("urukul0_ch3")
        
        self.setattr_device("urukul1_cpld")
        self.setattr_device("urukul1_ch0")
        self.setattr_device("urukul1_ch1")
        self.setattr_device("urukul1_ch2")
        self.setattr_device("urukul1_ch3")
        
        self.setattr_device("urukul2_cpld")
        self.setattr_device("urukul2_ch0")
        self.setattr_device("urukul2_ch1")
        self.setattr_device("urukul2_ch2")
        self.setattr_device("urukul2_ch3")

        self.setattr_device("urukul3_cpld")
        self.setattr_device("urukul3_ch0")
        self.setattr_device("urukul3_ch1")
        self.setattr_device("urukul3_ch2")
        self.setattr_device("urukul3_ch3")
        
        self.setattr_device("urukul4_cpld")
        self.setattr_device("urukul4_ch0")
        self.setattr_device("urukul4_ch1")
        self.setattr_device("urukul4_ch2")
        self.setattr_device("urukul4_ch3")

        self.setattr_device("urukul5_cpld")
        self.setattr_device("urukul5_ch0")
        self.setattr_device("urukul5_ch1")
        self.setattr_device("urukul5_ch2")
        self.setattr_device("urukul5_ch3")
        
        self.setattr_device("urukul6_cpld")
        self.setattr_device("urukul6_ch0")
        self.setattr_device("urukul6_ch1")
        self.setattr_device("urukul6_ch2")
        self.setattr_device("urukul6_ch3")

        self.setattr_device("urukul7_cpld")
        self.setattr_device("urukul7_ch0")
        self.setattr_device("urukul7_ch1")
        self.setattr_device("urukul7_ch2")
        self.setattr_device("urukul7_ch3")
        
        self.setattr_device("urukul8_cpld")
        self.setattr_device("urukul8_ch0")
        self.setattr_device("urukul8_ch1")
        self.setattr_device("urukul8_ch2")
        self.setattr_device("urukul8_ch3")

        

    @kernel
    def run(self):
        self.core.reset()
        
        while not self.core.get_rtio_destination_status(0):
            pass
        
        while not self.core.get_rtio_destination_status(1):
            pass

        self.core.break_realtime()
        self.ttl0.input()
        self.ttl1.input()
        self.ttl2.input()
        self.ttl3.input()
        
        self.ttl24.input()
        self.ttl25.input()
        self.ttl26.input()
        self.ttl27.input()

        # DDS
        self.urukul0_cpld.init()
        self.urukul0_ch0.init()
        self.urukul0_ch1.init()
        self.urukul0_ch2.init()
        self.urukul0_ch3.init()

        self.urukul1_cpld.init()
        self.urukul1_ch0.init()
        self.urukul1_ch1.init()
        self.urukul1_ch2.init()
        self.urukul1_ch3.init()
        
        self.urukul2_cpld.init()
        self.urukul2_ch0.init()
        self.urukul2_ch1.init()
        self.urukul2_ch2.init()
        self.urukul2_ch3.init()
        
        self.urukul3_cpld.init()
        self.urukul3_ch0.init()
        self.urukul3_ch1.init()
        self.urukul3_ch2.init()
        self.urukul3_ch3.init()

        self.urukul4_cpld.init()
        self.urukul4_ch0.init()
        self.urukul4_ch1.init()
        self.urukul4_ch2.init()
        self.urukul4_ch3.init()

        self.urukul5_cpld.init()
        self.urukul5_ch0.init()
        self.urukul5_ch1.init()
        self.urukul5_ch2.init()
        self.urukul5_ch3.init()
        
        self.urukul6_cpld.init()
        self.urukul6_ch0.init()
        self.urukul6_ch1.init()
        self.urukul6_ch2.init()
        self.urukul6_ch3.init()

        self.urukul7_cpld.init()
        self.urukul7_ch0.init()
        self.urukul7_ch1.init()
        self.urukul7_ch2.init()
        self.urukul7_ch3.init()
        
        self.urukul8_cpld.init()
        self.urukul8_ch0.init()
        self.urukul8_ch1.init()
        self.urukul8_ch2.init()
        self.urukul8_ch3.init()

        self.zotino0.init()
        self.zotino1.init()
        
        self.sampler0.init()
        for i in range(8):
            self.sampler0.set_gain_mu(i, 0)
            delay(10*us)
            
        #self.core.wait_until_mu(now_mu())

Uncommenting the last line changes nothing, startup terminates and idle does not do anything without core.reset(). I also checked with an experiment that just does core.reset() and then execute idle, but this produces the same result. By the way, if I remove the last line in idle (self.core.wait_until_mu(now_mu())), the experiment immediately terminates in the dashboard and I see no change on the TTL. If I use core.reset() instead of core.break_realtime(), it also immediately terminates, but I see the voltage changes on the TTL.

sb10q

steine01 By the way, if I remove the last line in idle (self.core.wait_until_mu(now_mu())), the experiment immediately terminates in the dashboard

Please read up and understand the RTIO section of the documentation. The immediate termination is just the same thing as those logger calls not being affected by the delays.

and I see no change on the TTL.

Did you also flash it as idle kernel? In that case that would be because it starts again and resets the core device.

steine01

I understand, that the termination is expected, it was only to demonstrate that there is a difference between the two non-working versions of the program. And I did read this section of the manual multiple times, sometimes struggling to understand how the timeline interacts with devices other than a TTL. What I was confused by (and still somewhat am), is what is actually affected by the RTIO system. For a long time I assumed, that all calls in a @kernel function have to be scheduled by the RTIO system, but that does not seem to be the case. See this section:

The ARTIQ code that describes a given experiment is composed of two types of “programs”: regular Python code that is executed on the host and ARTIQ kernels that are executed on a core device. The CPU that executes the ARTIQ kernels has direct access to specialized programmable I/O timing logic (part of the gateware). The two types of code can invoke each other and transitions between them are seamless.

Also you focus on the TTL change, instead of also addressing the second part of the question: if I do add core.wait_until_mu(now_mu()) as the last line, the experiment never finishes. Considering I showed you the startup kernel, where no timeline change into the far future is executed, this should not be the case, should it? Why is this happening?

But it looks like you were correct. I exchanged the idle kernel from what it was before (not this experiment, but maybe something similiar with a reset in the beginning) to just

class Idle(EnvExperiment):
    def build(self):
        self.setattr_device("core")
    @kernel
    def run(self):
        pass

And checked the TTL with a seperate experiment again. Now even with core.break_realtime() I see the change on the TTL and the experiment finishes. My point still stands though: why would a core.reset() in the idle kernel following execution of an experiment cause ARTIQ to hang indefinitely?

sb10q

steine01 Considering I showed you the startup kernel, where no timeline change into the far future is executed, this should not be the case, should it? Why is this happening?

Print the values of the RTIO counter (core.get_rtio_counter_mu()) and the time cursor (now_mu()) to find out.