Ok, I must be doing something wrong:
- We used to run on 3 Sinara crates, each with a Kasli v2.0.1.
- I put the original Kaslis back into their respective crates. I have not flashed them or done anything else to them. They are in exactly the state that used to work without problem. This is confirmed by the gateware identifications in the UART logs below.
- Now, everything works on the master but the two satellites are not seen, i.e. I always get
artiq.coredevice.exceptions.RTIODestinationUnreachable: RTIO destination unreachable, output, at 150084880248 mu, channel 65536 and artiq.coredevice.exceptions.RTIODestinationUnreachable: RTIO destination unreachable, output, at 10621998936 mu, channel 131072.
- Even worse, the UART logs of all 3 Kaslis do not mention any attempt at establishing a DRTIO link among them. Simply nothing. See the UART logs at the very bottom.
device_db.py and code
Photos of Kaslis
Master:
Satellite 1:
Satellite 2:
Error messages
Trying to run smth on satellite 1:
$ artiq_run do_stuff.py [20:28:03]
INFO: DIO init() complete
WARNING:artiq.coredevice.comm_kernel:sequence error(s) reported during kernel execution
Core Device Traceback:
Traceback (most recent call first):
File "ksupport/rtio.rs", line 74, column 14, in (Rust function)
<unknown>
^
File "do_stuff.py", line 119, in artiq_run_do_stuff.TestSatellites.run(..., ...) (RA=+0x1dc)
self.init()
File "<artiq>/coredevice/fastino.py", line 94, in ... artiq.coredevice.fastino.Fastino.write<artiq.coredevice.fastino.Fastino>(...) (RA=+0x798)
rtio_output(self.channel | addr, data)
File "<artiq>/coredevice/fastino.py", line 208, in ... artiq.coredevice.fastino.Fastino.set_cfg<artiq.coredevice.fastino.Fastino>(...) (inlined)
self.write(0x22, (reset << 0) | (afe_power_down << 1) |
File "<artiq>/coredevice/fastino.py", line 72, in ... artiq.coredevice.fastino.Fastino.init<artiq.coredevice.fastino.Fastino>(...) (inlined)
self.set_cfg(reset=0, afe_power_down=0, dac_clr=0, clr_err=1)
File "do_stuff.py", line 67, in ... artiq_run_do_stuff.TestSatellites.init<artiq_run_do_stuff.TestSatellites>(...) (inlined)
fastino.init()
artiq.coredevice.exceptions.RTIODestinationUnreachable(0): RTIO destination unreachable, output, at 150084880248 mu, channel 65536
End of Core Device Traceback
Traceback (most recent call last):
File "/home/m15/.pyenv/versions/3.12.9/bin/artiq_run", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/m15/workspace/artiq/artiq/frontend/artiq_run.py", line 224, in main
return run(with_file=True)
^^^^^^^^^^^^^^^^^^^
File "/home/m15/workspace/artiq/artiq/frontend/artiq_run.py", line 210, in run
raise exn
File "/home/m15/workspace/artiq/artiq/frontend/artiq_run.py", line 203, in run
exp_inst.run()
File "/home/m15/workspace/artiq/artiq/language/core.py", line 54, in run_on_core
return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/m15/workspace/artiq/artiq/coredevice/core.py", line 143, in run
self._run_compiled(kernel_library, embedding_map, symbolizer, demangler)
File "/home/m15/workspace/artiq/artiq/coredevice/core.py", line 133, in _run_compiled
self.comm.serve(embedding_map, symbolizer, demangler)
File "/home/m15/workspace/artiq/artiq/coredevice/comm_kernel.py", line 718, in serve
self._serve_exception(embedding_map, symbolizer, demangler)
File "/home/m15/workspace/artiq/artiq/coredevice/comm_kernel.py", line 700, in _serve_exception
raise python_exn
artiq.coredevice.exceptions.RTIODestinationUnreachable: RTIO destination unreachable, output, at 150084880248 mu, channel 65536
Trying to run smth on satellite 2:
$ artiq_run do_stuff.py [20:32:20]
INFO: DIO init() complete
INFO: Fastino init() complete
WARNING:artiq.coredevice.comm_kernel:sequence error(s) reported during kernel execution
Core Device Traceback:
Traceback (most recent call first):
File "ksupport/rtio.rs", line 74, column 14, in (Rust function)
<unknown>
^
File "do_stuff.py", line 119, in artiq_run_do_stuff.TestSatellites.run(..., ...) (RA=+0x1dc)
self.init()
File "do_stuff.py", line 77, in ... artiq_run_do_stuff.TestSatellites.init<artiq_run_do_stuff.TestSatellites>(...) (RA=+0x824)
urukul_cpld.init()
File "<artiq>/coredevice/spi2.py", line 169, in ... artiq.coredevice.spi2.SPIMaster.set_config_mu<artiq.coredevice.spi2.SPIMaster>(...) (RA=+0xc68)
rtio_output((self.channel << 8) | SPI_CONFIG_ADDR, flags |
File "<artiq>/coredevice/urukul.py", line 229, in ... artiq.coredevice.urukul.CPLD.sta_read<artiq.coredevice.urukul.CPLD>(...) (inlined)
self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END | spi.SPI_INPUT, 24,
File "<artiq>/coredevice/urukul.py", line 250, in ... artiq.coredevice.urukul.CPLD.init<artiq.coredevice.urukul.CPLD>(...) (inlined)
proto_rev = urukul_sta_proto_rev(self.sta_read())
artiq.coredevice.exceptions.RTIODestinationUnreachable(0): RTIO destination unreachable, output, at 10621998936 mu, channel 131072
End of Core Device Traceback
Traceback (most recent call last):
File "/home/m15/.pyenv/versions/3.12.9/bin/artiq_run", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/m15/workspace/artiq/artiq/frontend/artiq_run.py", line 224, in main
return run(with_file=True)
^^^^^^^^^^^^^^^^^^^
File "/home/m15/workspace/artiq/artiq/frontend/artiq_run.py", line 210, in run
raise exn
File "/home/m15/workspace/artiq/artiq/frontend/artiq_run.py", line 203, in run
exp_inst.run()
File "/home/m15/workspace/artiq/artiq/language/core.py", line 54, in run_on_core
return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/m15/workspace/artiq/artiq/coredevice/core.py", line 143, in run
self._run_compiled(kernel_library, embedding_map, symbolizer, demangler)
File "/home/m15/workspace/artiq/artiq/coredevice/core.py", line 133, in _run_compiled
self.comm.serve(embedding_map, symbolizer, demangler)
File "/home/m15/workspace/artiq/artiq/coredevice/comm_kernel.py", line 718, in serve
self._serve_exception(embedding_map, symbolizer, demangler)
File "/home/m15/workspace/artiq/artiq/coredevice/comm_kernel.py", line 700, in _serve_exception
raise python_exn
artiq.coredevice.exceptions.RTIODestinationUnreachable: RTIO destination unreachable, output, at 10621998936 mu, channel 131072
Complete UART logs
Master log, including one execution of the experiment:
MiSoC Bootloader
Copyright (c) 2017-2022 M-Labs Limited
Bootloader CRC passed
Gateware ident 7.8179.928ca50;mp01-dio-master
Initializing SDRAM...
Read leveling scan:
Module 1:
00000000000111111111000000000000
Module 0:
00000000001111111111000000000000
Read leveling: 15+-4 14+-5 done
SDRAM initialized
Memory test passed
Booting from flash...
Starting firmware.
[ 0.000015s] INFO(runtime): ARTIQ runtime starting...
[ 0.003939s] INFO(runtime): software ident 7.8179.928ca50;mp01-dio-master
[ 0.010825s] INFO(runtime): gateware ident 7.8179.928ca50;mp01-dio-master
[ 0.017880s] INFO(runtime): log level set to INFO by default
[ 0.023502s] INFO(runtime): UART log level set to INFO by default
[ 0.161111s] INFO(runtime::rtio_clocking): using 10MHz reference to make 125MHz RTIO clock with PLL
[ 0.424490s] INFO(board_artiq::si5324): waiting for Si5324 lock...
[ 2.577604s] INFO(board_artiq::si5324): ...locked
[ 2.608508s] INFO(runtime): network addresses: MAC=e8-eb-1b-45-dd-76 IPv4=192.168.1.30 IPv6-LL=fe80::eaeb:1bff:fe45:dd76 IPv6=no configured address
[ 2.622358s] INFO(board_artiq::drtio_routing): could not read routing table from configuration, using default
[ 2.631074s] INFO(board_artiq::drtio_routing): routing table: RoutingTable { 0: 0; 1: 1 0; 2: 2 0; 3: 3 0; }
[ 2.644694s] INFO(runtime::mgmt): management interface active
[ 2.656751s] INFO(runtime::session): accepting network sessions
[ 2.669824s] INFO(runtime::session): running startup kernel
[ 2.674440s] INFO(runtime::session): no startup kernel found
[ 2.680107s] INFO(runtime::session): no connection, starting idle kernel
[ 2.687070s] INFO(runtime::session): no idle kernel found
[ 2.692353s] INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up
[ 8.970754s] INFO(runtime::mgmt): new connection from 192.168.1.1:43030
[ 22.601428s] INFO(runtime::session): new connection from 192.168.1.1:48484
[ 22.667662s] INFO(runtime::kern_hwreq): resetting RTIO
[ 22.839679s] ERROR(runtime::rtio_mgt): RTIO sequence error involving channel 9
[ 22.882397s] INFO(runtime::session): no connection, starting idle kernel
[ 22.888436s] INFO(runtime::session): no idle kernel found
Satellite 1, including one execution of the experiment (which does not appear in the logs!):
MiSoC Bootloader
Copyright (c) 2017-2022 M-Labs Limited
Bootloader CRC passed
Gateware ident 7.8188.02dfb3e;mp01-analog-satellite
Initializing SDRAM...
Read leveling scan:
Module 1:
00000000011111111110000000000000
Module 0:
00000000011111111111000000000000
Read leveling: 13+-5 14+-5 done
SDRAM initialized
Memory test passed
Booting from flash...
Starting firmware.
[ 0.000003s] INFO(satman): ARTIQ satellite manager starting...
[ 0.005892s] INFO(satman): software ident 7.8188.02dfb3e;mp01-analog-satellite
[ 0.013213s] INFO(satman): gateware ident 7.8188.02dfb3e;mp01-analog-satellite
[ 0.421183s] INFO(board_artiq::si5324): waiting for Si5324 lock...
[ 3.087710s] INFO(board_artiq::si5324): ...locked