Master-Satellite-Setup

nm101

Hello,
after our satellite arrived i tried to setup it up and followed all steps provided by the manual.
I also had to flash the master and satellite which worked perfectly fine (no errors where thrown and in logs the right soft- and gateware version are shown).
After all of that i tried to test the setup with artiq_sinara_tester. The master "hardware" worked perfectly fine
but as the urukul DDS on our satellite should be tested it began to stuck and didnt react anymore:
*** Testing Urukul DDSes. urukul0_cpld: initializing CPLD... urukul0_cpld: testing attenuator digital control... urukul0_cpld: done urukul1_cpld: initializing CPLD...

After that I powercycled both the master and satellite to get the serial output .
1) Master boots without satellite on power:

                 __________   __
                / ___/__  /  / /
                \__ \  / /  / /
               ___/ / / /__/ /___
              /____/ /____/_____/

             (C) 2020-2022 M-Labs

[ 0.019988s] INFO(szl): Simple Zynq Loader starting...
[ 0.025194s] DEBUG(libboard_zynq::clocks::source): Set ARM_PLL to 2000000000 Hz
[ 0.007039s] DEBUG(libboard_zynq::clocks::source): Set IO_PLL to 1000000000 Hz
[ 0.016259s] DEBUG(libboard_zynq::clocks::source): Set DDR_PLL to 1066666666 Hz
[ 0.023608s] DEBUG(libboard_zynq::ddr): DDR 3x/2x clocks: 533333328/355555552
[ 0.030771s] DEBUG(libboard_zynq::ddr): DDR DCI clock: 10062892 Hz (divisors=2*53)
[ 0.042000s] DEBUG(libboard_zynq::sdio): Reset SDIO!
[ 0.046869s] DEBUG(libboard_zynq::sdio): Changing clock frequency to 400000
[ 0.053894s] INFO(szl): Card inserted. Mounting file system.
[ 0.073946s] DEBUG(libboard_zynq::sdio): Changing clock frequency to 25000000
[ 0.081076s] DEBUG(libboard_zynq::sdio::sd_card): Getting bus width
[ 0.087705s] DEBUG(libboard_zynq::sdio::sd_card): 4 bit support
[ 0.093601s] DEBUG(libboard_zynq::sdio::sd_card): Changing bus width
[ 0.101000s] DEBUG(libboard_zynq::sdio): Set block size to 512
[ 0.109084s] DEBUG(libconfig::sd_reader): Partition ID: B
[ 0.117926s] INFO(szl): Loading gateware
[ 0.123127s] DEBUG(libconfig::bootgen): Partition header pointer = C80
[ 0.130843s] DEBUG(libconfig::bootgen): Unencrypted length = B999A
[ 0.137001s] DEBUG(libconfig::bootgen): Partition start address: B620
[ 0.731802s] DEBUG(libboard_zynq::devc): Invalidate DCache for bitstream buffer
[ 0.745171s] DEBUG(libboard_zynq::devc): Init preload FPGA
[ 0.750634s] DEBUG(libboard_zynq::devc): Toggling PROG_B
[ 0.779897s] DEBUG(libboard_zynq::devc): Waiting for done
[ 0.785277s] DEBUG(libboard_zynq::devc): Init postload FPGA
[ 0.790831s] INFO(szl): Loading runtime
[ 0.795948s] DEBUG(libconfig::bootgen): Partition header pointer = C80
[ 0.803663s] DEBUG(libconfig::bootgen): Unencrypted length = B058
[ 0.809738s] DEBUG(libconfig::bootgen): Unencrypted length = 3C0E2
[ 0.815899s] DEBUG(libconfig::bootgen): Partition start address: C4FC0
[ 1.019633s] INFO(szl): Preparing for runtime execution
[ 1.025264s] INFO(szl): executing payload
[ 0.000067s] INFO(runtime): NAR3/Zynq7000 starting...
[ 0.005237s] INFO(runtime): gateware ident: dumaster
[ 0.015459s] INFO(libboard_zynq::i2c): PCA9548 detected
[ 0.173210s] INFO(runtime::rtio_clocking): using 10MHz reference to make 125MHz RTIO clock with PLL
[ 0.546289s] INFO(libboard_artiq::si5324): waiting for Si5324 lock...
[ 2.630901s] INFO(libboard_artiq::si5324): ...locked
[ 2.686995s] INFO(runtime::rtio_clocking): SYS CLK switched successfully
[ 2.698963s] INFO(libboard_zynq::i2c): PCA9548 detected
[ 2.744622s] INFO(runtime::comms): network addresses: MAC=fc-0f-e7-0e-ef-bd IPv4=10.34.16.100 IPv6-LL=fe80::fe0f:e7ff:fe0e:efbd IPv6: no configured address
[ 2.782439s] INFO(libboard_artiq::drtio_routing): routing table: RoutingTable { 0: 0; 1: 0; }
[ 2.795724s] INFO(runtime::rtio_mgt): SED spreading disabled by default
[ 2.817596s] INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up
[ 2.824283s] INFO(runtime::rtio_mgt::drtio): [DEST#1] destination is up
[ 6.817089s] INFO(libboard_zynq::eth): eth: got Link { speed: S1000, duplex: Full }

2) Satellite boots

| \/ (_) ___| ___ / |
| |\/| | _ \ / _ | |
| | | | |) | () | |
|| |||/ _/ ___|

MiSoC Bootloader
Copyright (c) 2017-2024 M-Labs Limited

Bootloader CRC passed
Gateware ident 8.8892+86a8d2d;dusatellite
Initializing SDRAM...
Read leveling scan:
Module 1:
00000000111111111000000000000000
Module 0:
00000001111111111000000000000000
Read leveling: 11+-4 11+-5 done
SDRAM initialized
Memory test passed

Booting from flash...
Starting firmware.
[ 0.000017s] INFO(satman): ARTIQ satellite manager starting...
[ 0.004571s] INFO(satman): software ident 8.8892+86a8d2d;dusatellite
[ 0.011008s] INFO(satman): gateware ident 8.8892+86a8d2d;dusatellite
[ 0.027998s] INFO(board_misoc::io_expander): MCP23017 io expander 0 not found. Checking for PCA9539.
[ 0.056521s] INFO(board_misoc::io_expander): MCP23017 io expander 1 not found. Checking for PCA9539.
[ 0.454456s] INFO(board_artiq::si5324): waiting for Si5324 lock...
[ 1.405013s] INFO(board_artiq::si5324): ...locked
[ 1.408588s] INFO(satman): Switching sys clock, rebooting...

| \/ (_) ___| ___ / |
| |\/| | _ \ / _ | |
| | | | |) | () | |
|| |||/ _/ ___|

MiSoC Bootloader
Copyright (c) 2017-2024 M-Labs Limited

Bootloader CRC passed
Gateware ident 8.8892+86a8d2d;dusatellite
Initializing SDRAM...
Read leveling scan:
Module 1:
00000001111111111000000000000000
Module 0:
00000001111111111000000000000000
Read leveling: 11+-5 11+-5 done
SDRAM initialized
Memory test passed

Booting from flash...
Starting firmware.
[ 0.000017s] INFO(satman): ARTIQ satellite manager starting...
[ 0.004571s] INFO(satman): software ident 8.8892+86a8d2d;dusatellite
[ 0.011009s] INFO(satman): gateware ident 8.8892+86a8d2d;dusatellite
[ 0.028000s] INFO(board_misoc::io_expander): MCP23017 io expander 0 not found. Checking for PCA9539.
[ 0.056523s] INFO(board_misoc::io_expander): MCP23017 io expander 1 not found. Checking for PCA9539.
[ 0.199068s] INFO(satman): Clocking has already been set up.
[ 0.203440s] INFO(satman): SED spreading disabled by default
[ 0.210272s] INFO(satman): uplink is up, switching to recovered clock
[ 0.243480s] INFO(board_artiq::si5324): waiting for Si5324 lock...
[ 0.264391s] INFO(board_artiq::si5324): ...locked
[ 2.924057s] INFO(board_artiq::si5324::siphaser): calibration successful, lead: 80, width: 430 (345deg)
[ 3.248955s] INFO(satman): TSC loaded from uplink
[ 3.354048s] INFO(satman): rank: 1
[ 3.356148s] INFO(satman): routing table: RoutingTable { 0: 0; 1: 0; }

3) after that i tried to set a frequency to urukul1_ch0 via the Dashboards Moninj widgets. Nothing happens and in
the coremgmt log only
INFO(runtime::mgmt): received connection
appears

Right know I am at the end of my knowledge how to troubleshoot this problem by myself.
The master and satellite are connected via an optical link which was delivered with the satellite and are connected
how it was proposed in our bitstream readme.
Special in our setup is that we use an externel clock reference of 10MHz connected to the master clock in.
I thank you guys in advance for ur help!

rjo

Make sure master gateware, satellite dateware, device db template version, and software version all match and that the device db is generated from the right JSONs.

nm101

rjo i checked artiq_ddb_template --version and compared it to the soft- and gateware versions (at least the satellite shows both as the same in the logs) and all where the same.

architeuthis yes that was one mistake and i changed it to 1: 1 0 .
now artiq_sinara_tester stucks at urukul2_cpld: initializing CPLD... .

architeuthis

While you're double-checking you also might want to check your routing table, though I'm not sure if it was a routing problem if you wouldn't get RTIODestinationUnreachable first --

[ 2.782439s] INFO(libboard_artiq::drtio_routing): routing table: RoutingTable { 0: 0; 1: 0; }

reads to me like it's routing both destination 0 and destination 1 to the local RTIO core 0, which is correct for destination 0, but not for destination 1, which being a satellite surely requires at least one downstream hop first.

mb842

Can you post your device_db and the code you're trying to run? Also did you generate your ddb using both the master and the satellite json files? This forum post might help

bodokaiser

Can you try to change eeprom_urukulN to

device_db["eeprom_urukulN0"] = {
    "type": "local",
    "module": "artiq.coredevice.kasli_i2c",
    "class": "KasliEEPROM",
    "arguments": {"port": "LOC0", "address": 0xAE},
}

in your device_db.py?