"[DEST#N] communication failed (timed out)" with ARTIQ8-Zynq?

sb10q

Do you have the same firmware version on all boards?

evilmav

All gatewares have been built from 367061aab8a939cae0b7683b80f43f1d3d87c0f7. Unfortunately, UART output does not seem to contain the version string on boot to verify what it thinks it is though...

EDIT: Reflashed all, definitely same gateware. No joy, same issue. Switched the copper SFP-SFP cable we use to an optical link to see if this is the cause, no joy either.

aonkes

We also ran into the same problem. Unlike @evilmav we use a Kasli-SoC as master and a Kasli v2 as satellite. The DRTIO link is also setup with transceivers recommended in the Sinara wiki on github. We use the most recent version from artiq-zynq ARTIQ v8.8954+4235309 (for building the gateware for both crates and the software on the host).

We used the artiq_sinara_tester and it seems like the error messages (see evilmav) only appear when a TTL device is involved. When the DIO cards or the Urukuls (because of the RF switches, which appear as TTL devices in the device_db) test routines are executed we see these errors. But if we only execute the test routine for the Fastinos there are no errors.

We also switched the roles of the crates, so the Kasli v2 is now the master and the Kasli SoC is the satellite. Then we don't see the communication errors at all when we run the sinara tester.

evilmav

aonkes

Does my example cause errors for you too? I'm under impression it is not as much matter of TTL or fastino as it is of the IDLE times...

choelzl

We have the same problem with ARTIQ 8.8973+80ae6f5.
We also use the recommended fiber links and the same setup worked yesterday before we replaced the master kasli 2.0 by the kasli soc with which it now fails.

On the satellite I get this:

[   967.304396s]  WARN(satman): aux packet error (protocol error: unknown packet 0x9a)
[  1204.295618s]  WARN(satman): aux packet error (packet CRC failed)
[  1297.778924s]  WARN(satman): received unexpected aux packet
[  1375.372610s]  WARN(satman): received unexpected aux packet
[  1581.811100s]  WARN(satman): received unexpected aux packet
[  1699.203397s]  WARN(satman): aux packet error (packet CRC failed)
[  1885.340720s]  WARN(satman): received unexpected aux packet
[  1991.075698s]  WARN(satman): aux packet error (protocol error: unknown packet 0xe2)

choelzl

Did any of you got this fixed?
Any input on how to get it fixed/debug it?

choelzl

After a while, I get the following error on the satellite kasli 2.0:

panic at /nix/store/1n2jcd6aqajnh7236w6cvqfk1chhhvw5-python3.11-artiq-8.8973+80ae6f5/lib/python3.11/site-packages/artiq/firmware/libproto_artiq/drtioaux_proto.rs:291:40: range end index 65535 out of range for slice of length 1010

and the error LED turns on. The master kasli then starts to spam

ERROR(runtime::rtio_mgt::drtio): [DEST#1] communication failed (timed out)

every 409ms. Is there any monitoring or something polling on this interval?
Because the DRTIO seems to work, at least we could not find a signal yet which is not replayed by the satellite in this state.

evilmav

Someone has crossposted the issue to m-labs gitea, but there is no activity for 3 months: https://git.m-labs.hk/M-Labs/artiq-zynq/issues/322 .

sb10q

Fixing this sort of bug isn't easy for anyone.

choelzl

I fully agree
I see there is a pull request on the git, I will try it out as soon as I have the time.

srenblad

choelzl With the patch applied, I was unable to reproduce the error with the code from the original post. That being said, ping us or reach out to the helpdesk if you still run into problems.

evilmav

srenblad

we've updated gateware to the current artiq-zynq release-8 state (incorporating "drtio: add InjectionRequest to expects_response" commit) and the issue is still present with the example code...

(If I could make a Christmas wish: any chance the coremgmt satellite support could be backported to release-8? This would immensely help testing this stuff =) )

choelzl

All in all the errors are less, but we still get some. Also the recurring (every 409ms) from the moninj driver are still there. I start to suspect that it is a hardware fault with the soc. We had it send to quartiq because we had similar issues before and got it back now with a note that it is fixed. Probably it is not...

evilmav

The issue is solved after https://git.m-labs.hk/M-Labs/artiq-zynq/pulls/345