Subkernel with DDMA runs once, then crate must be power cycled

debry

I am looking into running DDMA from in a subkernel, which the manual says is supported. I specifically need it to run in a subkernel on my satellite crate to free up the master Kasli for other tasks (instead of just DDMA from the master). I need DMA for high rate operations. Both crates are Kasli-SOC with ARTIQ 9, and the only peripheral used is a TTL.

The following minimal example implements the same sequence of TTL pulses on my satellite crate, first as a normal subkernel, and second using DDMA. The first, normal subkernel code is mostly there to show that I can successfully use DRTIO at all.

What I'm seeing is that this runs once, printing out both statements that the standard and DDMA blocks finished, and I see the expected output on a scope. However, the second time I submit the same experiment, only the first (non-DDMA) section finishes, and the DDMA section times out. Any subsequent runs receive a "subkernel failed to load" error, which can only be fixed by power cycling my crates.

Have I implemented this incorrectly in some way? Or is this a bug?

from artiq.experiment import *
import numpy as np

class DDMASatDemo(EnvExperiment):
    "DDMA Satellite Demo"
    def build(self):
        self.setattr_device("core")
        self.setattr_device("core_dma")
        self.setattr_device("trigger")
        self.setattr_device("ttl15")

        self.setattr_argument("num_pulses", NumberValue(default=100, min=1, step=0, precision=0))

    def prepare(self):
        self.delay = 500*us

    
    @subkernel(destination=1)
    def satellite_subkernel_standard(self):
        self.core.break_realtime()
        for _ in range(self.num_pulses):
            self.ttl15.pulse(self.delay)
            delay(self.delay)

    @subkernel(destination=1)
    def satellite_subkernel_ddma(self):
        self.core.break_realtime()
        
        with self.core_dma.record("satellite_subkernel_ddma"):
            for _ in range(self.num_pulses):
                self.ttl15.pulse(self.delay)
                delay(self.delay)
        
        self.core.break_realtime()

        self.core_dma.playback("satellite_subkernel_ddma")

    @kernel
    def run(self):
        self.core.reset()
        self.core.break_realtime()
        
        subkernel_preload(self.satellite_subkernel_standard)
        self.core.break_realtime()

        self.satellite_subkernel_standard()
        self.core.break_realtime()
        self.trigger.pulse(1*us)

        subkernel_await(self.satellite_subkernel_standard, 5000)
        
        self.core.break_realtime()

        print("standard finished")

        ###########################################################
        self.core.reset()
        self.core.break_realtime()

        subkernel_preload(self.satellite_subkernel_ddma)
        self.core.break_realtime()

        self.satellite_subkernel_ddma()
        self.core.break_realtime()
        self.trigger.pulse(1*us)

        subkernel_await(self.satellite_subkernel_ddma, 5000)
        
        self.core.break_realtime()
        
        print("DDMA finished")

First run:

Second run:

root:Terminating with exception (SubkernelError: Subkernel timed out)
Core Device Traceback:
Traceback (most recent call first):
  File "libksupport/src/kernel/subkernel.rs", line 44, column 63, in (Rust function)
    <unknown>
                                                                  ^
  File "/home/artiq/artiq-master/repository/ddma_satellite_demo.py", line 151, in artiq_worker_ddma_satellite_demo.DDMASatDemo.run(..., ...) (RA=+0x294)
    subkernel_await(self.satellite_subkernel_ddma, 5000)
artiq.coredevice.exceptions.SubkernelError(0): Subkernel timed out

End of Core Device Traceback
Traceback (most recent call last):
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/master/worker_impl.py", line 378, in main
    exp_inst.run()
    ~~~~~~~~~~~~^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/language/core.py", line 54, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/core.py", line 177, in run
    self._run_compiled(kernel_library, embedding_map, symbolizer, demangler)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/core.py", line 166, in _run_compiled
    self.comm.serve(embedding_map, symbolizer, demangler)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/comm_kernel.py", line 744, in serve
    self._serve_exception(embedding_map, symbolizer, demangler)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/comm_kernel.py", line 726, in _serve_exception
    raise python_exn
artiq.coredevice.exceptions.SubkernelError: Subkernel timed out

LoadError on third and later runs:

root:Terminating with exception (LoadError: subkernel failed to load)
Traceback (most recent call last):
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/master/worker_impl.py", line 378, in main
    exp_inst.run()
    ~~~~~~~~~~~~^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/language/core.py", line 54, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/core.py", line 176, in run
    self.compile_and_upload_subkernels(embedding_map, args, subkernel_arg_types)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/core.py", line 211, in compile_and_upload_subkernels
    self.comm.upload_subkernel(kernel_library, sid, destination)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jyahbc2h91iph3pfsiv46qvb975q2yqs-python3-3.13.12-env/lib/python3.13/site-packages/artiq/coredevice/comm_kernel.py", line 398, in upload_subkernel
    raise LoadError(self._read_string())
artiq.coredevice.comm_kernel.LoadError: subkernel failed to load

spaqin

Seems the satellite has crashed at some point, probably a bug as on first glance code looks fine; can you please get the device logs from UART through the USB port?

debry

Here are the logs (with only the DDMA section of the code active)
Satellite:

[    76.298811s]  INFO(satman::drtiosat_aux): resetting RTIO
[    76.302965s]  INFO(ksupport::kernel::core1): kernel starting
[    76.304403s] ERROR(satman::subkernel): exception in kernel
[    76.309584s] ERROR(satman::subkernel): Exception 2 from (Rust function) in libksupport/src/kernel/dma.rs:199:17, message: RTIO destination unreachable, output, at {1} mu, channel {rtio_channel_info:0}
[    76.327378s] ERROR(satman::subkernel): stack pointers: [StackPointerBacktrace { stack_pointer: 288109776, initial_backtrace_size: 0, current_backtrace_size: 1 }]
[    76.341786s] ERROR(satman::subkernel): backtrace: [(508, 288109776)]
Core 0 panic at libboard_artiq/src/lib.rs:130:10: cannot get device map before it is set up

Master:

[    60.163270s]  INFO(ksupport::kernel::core1): kernel starting
[    60.418889s] ERROR(runtime::rtio_mgt::drtio): [DEST#1] communication failed (timed out)
[    60.620886s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
[    60.821884s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
… (more aux packet errors)
[    64.841878s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
[    65.042875s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
[    65.168877s] ERROR(runtime::subkernel): Remote subkernel finish await timed out
[    65.202688s]  INFO(runtime::comms): peer closed connection
[    65.202704s]  INFO(runtime::comms): no idle kernel found
[    65.243875s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
[    65.444872s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
[    65.645876s] ERROR(runtime::moninj::remote_moninj): aux packet error (timed out)
…
[    74.288860s] ERROR(runtime::rtio_mgt::drtio): [DEST#1] communication failed (timed out)

When I run the non-DDMA (still using @subkernel, but no DMA) version, I also get these errors on the satellite UART logs (no errors on the master logs):

[    32.105793s]  INFO(satman::drtiosat_aux): resetting RTIO
[    32.109879s]  INFO(ksupport::kernel::core1): kernel starting
[    32.111198s] ERROR(satman): write overflow
[    32.115228s] ERROR(satman): write overflow
[    32.119726s] ERROR(satman): write overflow
[    32.123725s] ERROR(satman): write overflow
[    32.127727s] ERROR(satman): write overflow
[    32.131726s] ERROR(satman): write overflow
[    32.135726s] ERROR(satman): write overflow
[    32.139724s] ERROR(satman): write overflow
[    32.144225s] ERROR(satman): write overflow
[    32.146723s]  INFO(ksupport::kernel::core1): kernel finished
[    32.153459s]  INFO(satman::subkernel): subkernel 3 finished, with exception: false

spaqin

Cheers, thanks; looks like an oversight on my part with the device map not being set up. I'll get it fixed ASAP.

The write overflow log - do you see if it's affecting the output? I believe it's because we switch from DRTIO to local kernel, the part that handles DRTIO errors sees writes that it's not expecting and thus freaks out - that should've been suppressed to avoid confusion.

debry

spaqin I'm not sure I understand exactly what you're asking, but if you are asking whether the non-DDMA satellite outputd the correct pulse sequence, then I believe it does. I see 100 pulses when I ask for 100 pulses, but I have not carefully checked the timing or other things that could be affected. If you have something specific I should be looking out for I'm happy to check.

Thanks for taking a look at this!