We also ran into the same problem. Unlike @evilmav we use a Kasli-SoC as master and a Kasli v2 as satellite. The DRTIO link is also setup with transceivers recommended in the Sinara wiki on github. We use the most recent version from artiq-zynq ARTIQ v8.8954+4235309 (for building the gateware for both crates and the software on the host).

We used the artiq_sinara_tester and it seems like the error messages (see evilmav) only appear when a TTL device is involved. When the DIO cards or the Urukuls (because of the RF switches, which appear as TTL devices in the device_db) test routines are executed we see these errors. But if we only execute the test routine for the Fastinos there are no errors.

We also switched the roles of the crates, so the Kasli v2 is now the master and the Kasli SoC is the satellite. Then we don't see the communication errors at all when we run the sinara tester.

      aonkes

      Does my example cause errors for you too? I'm under impression it is not as much matter of TTL or fastino as it is of the IDLE times...

        5 days later

        We have the same problem with ARTIQ 8.8973+80ae6f5.
        We also use the recommended fiber links and the same setup worked yesterday before we replaced the master kasli 2.0 by the kasli soc with which it now fails.

        On the satellite I get this:

        [   967.304396s]  WARN(satman): aux packet error (protocol error: unknown packet 0x9a)
[  1204.295618s]  WARN(satman): aux packet error (packet CRC failed)
[  1297.778924s]  WARN(satman): received unexpected aux packet
[  1375.372610s]  WARN(satman): received unexpected aux packet
[  1581.811100s]  WARN(satman): received unexpected aux packet
[  1699.203397s]  WARN(satman): aux packet error (packet CRC failed)
[  1885.340720s]  WARN(satman): received unexpected aux packet
[  1991.075698s]  WARN(satman): aux packet error (protocol error: unknown packet 0xe2)

        Did any of you got this fixed?
        Any input on how to get it fixed/debug it?

        After a while, I get the following error on the satellite kasli 2.0:

        panic at /nix/store/1n2jcd6aqajnh7236w6cvqfk1chhhvw5-python3.11-artiq-8.8973+80ae6f5/lib/python3.11/site-packages/artiq/firmware/libproto_artiq/drtioaux_proto.rs:291:40: range end index 65535 out of range for slice of length 1010

        and the error LED turns on. The master kasli then starts to spam

        ERROR(runtime::rtio_mgt::drtio): [DEST#1] communication failed (timed out)

        every 409ms. Is there any monitoring or something polling on this interval?
        Because the DRTIO seems to work, at least we could not find a signal yet which is not replayed by the satellite in this state.

        Fixing this sort of bug isn't easy for anyone.

        I fully agree
        I see there is a pull request on the git, I will try it out as soon as I have the time.

          choelzl With the patch applied, I was unable to reproduce the error with the code from the original post. That being said, ping us or reach out to the helpdesk if you still run into problems.

              All in all the errors are less, but we still get some. Also the recurring (every 409ms) from the moninj driver are still there. I start to suspect that it is a hardware fault with the soc. We had it send to quartiq because we had similar issues before and got it back now with a note that it is fixed. Probably it is not...

              srenblad

              we've updated gateware to the current artiq-zynq release-8 state (incorporating "drtio: add InjectionRequest to expects_response" commit) and the issue is still present with the example code...

              (If I could make a Christmas wish: any chance the coremgmt satellite support could be backported to release-8? This would immensely help testing this stuff =) )

                20 days later
                22 days later