We are operating an ARTIQ system with one Kasli-SoC crate as the master and two Kasli 2.0 crates as the satellites. This setup has worked normally for the past two years, but recently it has generated the following core_log of "aux packet error (timed out)" very frequently.
This error indicates exceptions of the connection between the Kasli-SoC and the computer, and can occur regardless of whether there is an experiment running. The computer can eventually reconnect with the Kasli-SoC; however, this connection issue keeps showing up.
In addition, we frequently encounter the "RTIO Destination Unreachable" error, indicating an issue with communication between the master and satellites. Regarding these two issues, we tried rebooting the computer, power-cycling the master and satellites, and cleaning the fiber connector, but none of these methods solved the problem. What should we do?
For your reference, we are currently using ARTIQ 7.8185.cc81464 on a Linux computer. Our ARTIQ packages were installed using Nix package management system.