Frequently DRTIO connection failure

evilgenius

Hello,
We meet trouble when we try to connect our two satellites to master. Our hardware and software setup refer to Quote QR0307R3. Here are the details:
When we turn on the master and satellites, we can setup DRTIO link at a very low possibility (<5%). At most time, master just keeps pinging:
[ 9.481982s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 29.488945s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed
However, once the connection is set after a successful try, it seems the link of each satellite to master will never fail no matter how many times we power on and off the satellites. ONLY IF we reboot the master, the trouble comes again.
Above all, we seek for a method can stably set up connection without a lot of trying. Perhaps the problems come from firmware, referring to https://github.com/m-labs/artiq/issues/2028.
Looking forward to your reply, thank you in advance!

evilgenius

Today we powered on and off master and two satellites for about 30 to 40 times, in order to connect satellites to master, but all failed. Here is the serial output:
[ 0.019989s] INFO(szl): Simple Zynq Loader starting... [ 0.025196s] DEBUG(libboard_zynq::clocks::source): Set ARM_PLL to 2000000000 Hz [ 0.007039s] DEBUG(libboard_zynq::clocks::source): Set IO_PLL to 1000000000 Hz [ 0.016258s] DEBUG(libboard_zynq::clocks::source): Set DDR_PLL to 1066666666 Hz [ 0.023608s] DEBUG(libboard_zynq::ddr): DDR 3x/2x clocks: 533333328/355555552 [ 0.030777s] DEBUG(libboard_zynq::ddr): DDR DCI clock: 10062892 Hz (divisors=2*53) [ 0.042000s] DEBUG(libboard_zynq::sdio): Reset SDIO! [ 0.046948s] DEBUG(libboard_zynq::sdio): Changing clock frequency to 400000 [ 0.053895s] INFO(szl): Card inserted. Mounting file system. [ 0.073929s] DEBUG(libboard_zynq::sdio): Changing clock frequency to 25000000 [ 0.081060s] DEBUG(libboard_zynq::sdio::sd_card): Getting bus width [ 0.087869s] DEBUG(libboard_zynq::sdio::sd_card): 4 bit support [ 0.093767s] DEBUG(libboard_zynq::sdio::sd_card): Changing bus width [ 0.102001s] DEBUG(libboard_zynq::sdio): Set block size to 512 [ 0.108957s] DEBUG(libconfig::sd_reader): Partition ID: C [ 0.115534s] INFO(szl): Loading gateware [ 0.119737s] DEBUG(libconfig::bootgen): Partition header pointer = C80 [ 0.126990s] DEBUG(libconfig::bootgen): Unencrypted length = C5B04 [ 0.133148s] DEBUG(libconfig::bootgen): Partition start address: B5D0 [ 0.543657s] DEBUG(libboard_zynq::devc): Invalidate DCache for bitstream buffer [ 0.557421s] DEBUG(libboard_zynq::devc): Init preload FPGA [ 0.562886s] DEBUG(libboard_zynq::devc): Toggling PROG_B [ 0.593650s] DEBUG(libboard_zynq::devc): Waiting for done [ 0.599031s] DEBUG(libboard_zynq::devc): Init postload FPGA [ 0.604585s] INFO(szl): Loading runtime [ 0.609234s] DEBUG(libconfig::bootgen): Partition header pointer = C80 [ 0.616479s] DEBUG(libconfig::bootgen): Unencrypted length = B008 [ 0.622554s] DEBUG(libconfig::bootgen): Unencrypted length = 2F138 [ 0.628715s] DEBUG(libconfig::bootgen): Partition start address: D10E0 [ 0.734200s] INFO(szl): Preparing for runtime execution [ 0.739754s] INFO(szl): executing payload [ 0.000067s] INFO(runtime): NAR3/Zynq7000 starting... [ 0.005246s] INFO(runtime): gateware ident: rb2master [ 0.015559s] INFO(libboard_zynq::i2c): PCA9548 detected [ 0.238699s] INFO(runtime::rtio_clocking): using 125MHz reference to make 125MHz RTIO clock with PLL [ 0.611862s] INFO(libboard_artiq::si5324): waiting for Si5324 lock... [ 2.542338s] INFO(libboard_artiq::si5324): ...locked [ 2.551996s] INFO(runtime::rtio_clocking): RTIO PLL locked [ 2.562753s] INFO(libboard_zynq::i2c): PCA9548 detected [ 2.600779s] INFO(runtime::comms): network addresses: MAC=fc-0f-e7-07-c3-75 IPv4=192.168.31.202 IPv6-LL=fe80::fe0f:e7ff:fe07:c375 IPv6: no configured address [ 2.617486s] INFO(libboard_artiq::drtio_routing): could not read routing table from configuration, using default [ 2.627732s] INFO(libboard_artiq::drtio_routing): routing table: RoutingTable { 0: 0; 1: 1 0; 2: 2 0; 3: 3 0; 4: 4 0; } [ 2.641212s] INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up [ 6.641089s] INFO(libboard_zynq::eth): eth: got Link { speed: S1000, duplex: Full } [ 10.686981s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 30.693940s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed [ 30.700024s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging [ 50.706901s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed [ 50.913900s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 70.920859s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed [ 70.927024s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging [ 90.933822s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed [ 91.140820s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 111.147781s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed [ 111.153946s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging [ 131.160842s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed [ 131.367740s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 151.374701s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed [ 151.380869s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging [ 171.387661s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed [ 171.594660s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 191.601622s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed [ 191.607782s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging [ 211.614582s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed [ 211.821581s] INFO(runtime::rtio_mgt::drtio): [LINK#0] link RX became up, pinging [ 231.828541s] ERROR(runtime::rtio_mgt::drtio): [LINK#0] ping failed

esavkin

Just to update, the investigation ended up in totally working Kasli-SoC, but problematic SFP adapters from unknown sources (using such may void the warranty actually).

evilgenius

esavkin
I would like to know the model of DAC SFPs you are using, or if possible, provide me with a purchase link. Additionally, I noticed that when purchasing SFPs, it is necessary to select the compatible system. Could you please provide me with this information as well?

sb10q

esavkin using such may void the warranty actually

It is very unlikely that a low-quality SFP would cause permanent hardware damage.

evilgenius Additionally, I noticed that when purchasing SFPs, it is necessary to select the compatible system. Could you please provide me with this information as well?

It doesn't matter, you can select any, including the generic "FS.com" option. The "compatible system" selection is only for FS.com to program the SFP EEPROM with data that makes branded network equipment recognize the SFP as legit. ARTIQ systems ignore the contents of the EEPROM and always attempt to use whatever SFP you installed.

esavkin

sb10q It is very unlikely that a low-quality SFP would cause permanent hardware damage.

But what if it will overheat, cause fire or just get stuck?