Master-Slave setup problems

multi131

Hello,
I'm new here and unexperienced with ARTIQ.

In my lab we're working on setting up the Berkeley5master ARTIQ boxes. We had some success in talking to the master Kasli-SoC and it's direct peripherals. Unfortunately all of our attempts in talking to the slave Box were unsuccessful so far.
We get the artiq.coredevice.exceptions.RTIODestinationUnreachable(0): RTIO destination unreachable error whenever addressing any peripheral on the slave box.

We've tried the get_rtio_destination_status() function which returns True for the destination 0 and False for the destination 1 (which we think is the slave).

We connect via the Ethernet port on the Kasli-SoC to the master. SFP1 on the master is connected to SFP0 on the slave with the optical fiber provided by M-labs. All other SFP ports are empty.

This forum article describes a similar situation involving flashing the firmware on the box. I was expecting our firmware to be ready to establish the master-slave connection, but maybe that's the issue.

Any help/suggestions would be highly appreciated!

Thanks,
Malte & E9 experiment

sb10q

What's in the device logs (master and satellite)?

multi131

artiq_coremgmt log gives this output

[ 0.000067s] INFO(runtime): NAR3/Zynq7000 starting... [ 0.005160s] INFO(runtime): detected gateware: GenericMaster [ 0.016154s] INFO(libboard_zynq::i2c): PCA9548 detected [ 0.049482s] WARN(runtime::rtio_clocking): error reading configuration. Falling back to default. [ 0.058339s] WARN(runtime::rtio_clocking): Using default configuration - internal 125MHz RTIO clock. [ 0.067540s] INFO(runtime::rtio_clocking): using internal 125MHz RTIO clock [ 0.458302s] INFO(libboard_artiq::si5324): waiting for Si5324 lock... [ 7.745634s] INFO(libboard_artiq::si5324): ...locked [ 7.754986s] INFO(runtime::rtio_clocking): RTIO PLL locked [ 7.765739s] INFO(libboard_zynq::i2c): PCA9548 detected [ 7.797773s] INFO(runtime::comms): network addresses: MAC=e8-eb-1b-13-4c-0b IPv4=192.168.1.56 IPv6-LL=fe80::eaeb:1bff:fe13:4c0b IPv6: no configured address [ 7.812911s] INFO(libboard_artiq::drtio_routing): could not read routing table from configuration, using default [ 7.823086s] INFO(libboard_artiq::drtio_routing): routing table: RoutingTable { 0: 0; 1: 1 0; 2: 2 0; 3: 3 0; 4: 4 0; } [ 7.834390s] INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up [ 11.834080s] INFO(libboard_zynq::eth): eth: got Link { speed: S1000, duplex: Full } [ 142.741561s] INFO(runtime::kernel::core1): kernel starting [ 142.867323s] INFO(runtime::comms): peer closed connection [ 164.079362s] INFO(runtime::mgmt): received connection

I assume this is the Master log - I don't know how to get the slave log. I'll try to find it out and post it if I'm successful.

multi131

I noticed that these lines repeatedly appear in the log file as well (I tried everything again).
[ 48.114907s] ERROR(runtime::rtio_mgt::drtio): [LINK#1] ping failed [ 48.321907s] INFO(runtime::rtio_mgt::drtio): [LINK#1] link RX became up, pinging

Edit: These lines don't seem to appear deterministically. With similar hardware setup they sometimes appear after startup and sometimes don't.

rowanq

Hi, an update on this issue: In our log file, we now see "RTIO PLL failed to lock" and no LINK#1 ping. In addition, we no longer see a response from the Zotinos in the master box -- for example, when using self.zotino1.set_leds() to turn on the LEDs on the Zotino, the lights do not turn on. Previously this did work, as well as the set_dac() command.

(artiq) C:\Users\E5\Artiq>artiq_coremgmt log
[     0.000067s]  INFO(runtime): NAR3/Zynq7000 starting...
[     0.005238s]  INFO(runtime): detected gateware: GenericMaster
[     0.016157s]  INFO(libboard_zynq::i2c): PCA9548 detected
[     0.049482s]  WARN(runtime::rtio_clocking): error reading configuration. Falling back to default.
[     0.058340s]  WARN(runtime::rtio_clocking): Using default configuration - internal 125MHz RTIO clock.
[     0.067462s]  INFO(runtime::rtio_clocking): using internal 125MHz RTIO clock
[     0.458264s]  INFO(libboard_artiq::si5324): waiting for Si5324 lock...
[     7.811704s]  INFO(libboard_artiq::si5324):   ...locked
[     7.820985s] ERROR(runtime::rtio_clocking): RTIO PLL failed to lock
[     7.832437s]  INFO(libboard_zynq::i2c): PCA9548 detected
[     7.864398s]  INFO(runtime::comms): network addresses: MAC=e8-eb-1b-13-4c-0b IPv4=192.168.1.56 IPv6-LL=fe80::eaeb:1bff:fe13:4c0b IPv6: no configured address
[     7.879540s]  INFO(libboard_artiq::drtio_routing): could not read routing table from configuration, using default
[     7.889792s]  INFO(libboard_artiq::drtio_routing): routing table: RoutingTable { 0: 0; 1: 1 0; 2: 2 0; 3: 3 0; 4: 4 0; }
[     7.901026s]  INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up
[    11.536204s]  INFO(runtime::mgmt): received connection
[    11.900079s]  INFO(libboard_zynq::eth): eth: got Link { speed: S1000, duplex: Full }
[    53.426548s]  INFO(runtime::mgmt): received connection
[   510.217888s]  INFO(runtime::mgmt): received connection
[   787.199284s]  INFO(runtime::kernel::core1): kernel starting
[   787.204867s]  INFO(runtime::kernel::core1): kernel finished
[   787.210521s]  INFO(runtime::comms): peer closed connection

sb10q

Do you have an electronics engineer on your team who can plot the Kasli-SoC MGT power supply voltage on an oscilloscope, without damaging the hardware?

rowanq

Here are images of the 12V power supply we're using for the Kasli SoC on an oscilloscope (and the FFT). I'm not sure if this is what you meant.

Thanks!

sb10q

No - the internal MGT supply. This requires probing the PCB, while taking appropriate precautions against ESD and short circuits to prevent causing damage.

Also update to the latest Kasli-SoC firmware via AFWS and artiq_coremgmt if you haven't already. We also recently fixed DRTIO gateware bugs that caused such symptoms (intermittently). Email helpdesk@ for any issues or questions about AFWS.

sb10q

Related Issue: https://github.com/sinara-hw/Kasli-SOC/issues/68
As far as I know we shipped you a fixed board but let's double-check.

rowanq

Hi Sébastien, thanks for the suggestions and link. I sent an email re. the firmware update.
We took a look at our Kasli-SoC PCB but could not find C125 to check if a parallel capacitor had been added. The PCB is labeled "Kasli-SoC v1.0." We don't feel confident to probe the PCB.

rowanq

An update -- we pulled out the PCB further and found C125. Are you able to tell if the board is a fixed one looking at this image? We didn't manage to measure the capacitance of C125.

Updating the firmware had no effect on our issue.