We have a system where a kasli SoC v1.1 (master) connects directly to 3 kasli v2.0.2 (satellites). After updating the gateware to the latest release-8 branch (on all kaslis) we have problems with the startup kernel below:
from artiq.experiment import *
class StartupKernel(EnvExperiment):
def build(self):
print("Generating startup kernel...")
self.core = self.get_device("core")
@kernel
def run(self):
self.core.reset()
print("Starting startup-kernel...")
self._wait_for_drtio_destinations()
print("Startup-kernel finished.")
self.core.wait_until_mu(now_mu())
@kernel
def _wait_for_drtio_destinations(self):
for d in range(4):
while not self.core.get_rtio_destination_status(d):
delay(100 * ms)
self.core.wait_until_mu(now_mu())
The master cannot establish the link to the satellites and it prints on the UART:
[ 2.853581s] INFO(runtime::comms): Loading startup kernel...
[ 2.859356s] INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up
[ 2.866080s] INFO(runtime::comms): Starting startup kernel...
[ 2.871910s] INFO(ksupport::kernel::core1): kernel starting
[ 2.877566s] INFO(ksupport::kernel::api): kernel: Starting startup-kernel...
[ 5.091775s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging
[ 25.098963s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed
[ 25.105131s] INFO(runtime::rtio_mgt::drtio): [LINK#2] link RX became up, pinging
[ 45.111915s] ERROR(runtime::rtio_mgt::drtio): [LINK#2] ping failed
[ 45.318912s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging
[ 65.325877s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed
[ 65.332044s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging
[ 85.338840s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed
[ 85.345000s] INFO(runtime::rtio_mgt::drtio): [LINK#2] link RX became up, pinging
[ 105.351800s] ERROR(runtime::rtio_mgt::drtio): [LINK#2] ping failed
[ 105.558792s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging
[ 125.565761s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed
[ 125.571922s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging
[ 145.578716s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed
[ 145.584877s] INFO(runtime::rtio_mgt::drtio): [LINK#2] link RX became up, pinging
[ 165.591676s] ERROR(runtime::rtio_mgt::drtio): [LINK#2] ping failed
[ 165.798672s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging
[ 185.805640s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed
[ 185.811800s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging
[ 205.818598s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed
[ 205.824764s] INFO(runtime::rtio_mgt::drtio): [LINK#2] link RX became up, pinging
[ 225.831560s] ERROR(runtime::rtio_mgt::drtio): [LINK#2] ping failed
[ 226.038552s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging
[ 246.045516s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed
[ 246.051677s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging
[ 246.259513s] INFO(runtime::rtio_mgt::drtio): [LINK#1] remote replied after 1 packets
[ 246.470514s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] failed to load routing table (timed out)
[ 246.678521s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] failed to set rank (timed out)
[ 246.686333s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link initialization completed
[ 246.694058s] INFO(runtime::rtio_mgt::drtio): [LINK#2] link RX became up, pinging
[ 266.700477s] ERROR(runtime::rtio_mgt::drtio): [LINK#2] ping failed
[ 266.906480s] ERROR(runtime::rtio_mgt::drtio): [DEST#2] communication failed (timed out)
While on the satellites the UART prints [ 4.589290s] WARN(satman): received unexpected aux packet multiple times.
On the other hand, if I rewrite the startup kernel such that it does not wait for self.core.get_rtio_destination_status(d) and does nothing the link can be established without a problem.
I think what is happening is that self.core.get_rtio_destination_status(d) always returns False. And somehow, while the startup kernel is running, communication to the satellites fails.
How can we fix this?