Urukul v1.5.4: LD5, LD6, LD8 and LD2 remain dark, `AD9910.init()` crashes

dtsevas

Our upper-most crate contains:

1x Kasli SoC v1.1.1 powered by a 200 W cassette-mounted power supply and flashed as a DRTIO satellite.
6x Urukul_9910 v1.5.4 with their EEM0 and EEM1 connected to the Kasli SoC's EEM{n} and EEM{n+1}, respectively, with n = 0, 2, 4, 6, 8, 10 from left-most Urukul (n = 0) to right-most Urukul (n = 10) in the photos below.
1x Clocker v1.1rc2 powered by a separate 80 W power supply with its IN_CLK0_N and IN_CLK0_P (socket J2) connected to the Kasli SoC's EEM_CLK_OUT0 (socket J1) and with its EEM_CLK_OUT4 through EEM_CLK_OUT9 (sockets J7, J8, J9, J11, J13, J14) connected to the Urukuls' INT_CLK_IN (socket J7).

When we power on our entire Sinara system and before running any code, LD7A labelled PWR GOOD lights up green on all Urukuls and LD5B, LD6B, LD8B and LD2B light up red on all Urukuls except the right-most in the first photo below:

${dark_urukul_card_in_bay_6}$

We can call without error:

artiq.coredevice.urukul.CPLD.init() for self.get_device("urukul0_cpld") through self.get_device("urukul6_cpld").
artiq.coredevice.ad9910.AD9910.init() for self.get_device("urukul0_ch0") through self.get_device("urukul5_ch3"), which darkens their LD5B, LD6B, LD8B and LD2B.

But artiq.coredevice.ad9910.AD9910.init() for self.get_device("urukul6_ch0") through self.get_device("urukul6_ch3") has no effect on any of the right-most Urukul board's LEDs and fails with:

Core Device Traceback:
Traceback (most recent call first):
  File "<artiq>/coredevice/ad9910.py", line 480, column 17, in ... artiq.coredevice.ad9910.AD9910.init<artiq.coredevice.ad9910.AD9910>(...)
    raise ValueError("Urukul AD9910 AUX_DAC mismatch")
    ^
  File "test_satellites.py", line 109, in artiq_run_test_satellites.TestSatellites.run(..., ...) (RA=+0x128)
    self.init()
  File "test_satellites.py", line 73, in ... artiq_run_test_satellites.TestSatellites.init<artiq_run_test_satellites.TestSatellites>(...) (RA=+0x604)
    dds.init()
  File "<artiq>/coredevice/ad9910.py", line 0, in ... artiq.coredevice.ad9910.AD9910.init<artiq.coredevice.ad9910.AD9910>(...) (RA=+0xf20)
    <unknown>
ValueError(0): Urukul AD9910 AUX_DAC mismatch

End of Core Device Traceback

Traceback (most recent call last):
  File "/home/dimitry/.pyenv/versions/artiq-py311/bin/artiq_run", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/frontend/artiq_run.py", line 269, in main
    return run(with_file=True)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/frontend/artiq_run.py", line 253, in run
    raise exn
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/frontend/artiq_run.py", line 245, in run
    exp_inst.run()
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/language/core.py", line 54, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/coredevice/core.py", line 177, in run
    self._run_compiled(kernel_library, embedding_map, symbolizer, demangler)
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/coredevice/core.py", line 166, in _run_compiled
    self.comm.serve(embedding_map, symbolizer, demangler)
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/coredevice/comm_kernel.py", line 739, in serve
    self._serve_exception(embedding_map, symbolizer, demangler)
  File "/home/dimitry/.pyenv/versions/artiq-py311/lib/python3.11/site-packages/artiq/coredevice/comm_kernel.py", line 721, in _serve_exception
    raise python_exn
ValueError: Urukul AD9910 AUX_DAC mismatch

We moved the problematic Urukul board from the Kasli SoC's EEM10 and EEM11 to EEM8 and EEM9 and we also physically moved it one crate bay to the left. The problematic Urukul board still didn't work while all other boards worked, so we conclude that the Kasli SoC has no problem:

${urukul_cards_in_bays_5_and_6_swapped_now_dark_urukul_card_in_bay_5}$

We thought that the problematic Urukul board's ribbon cables might be damaged, so we left all boards in their bays and only swapped out the ribbon cables between the problematic Urukul board and another Urukul board. The problematic Urukul board still didn't work while all other boards worked, so we conclude that none of the ribbon cables have a problem:

${as_last_one_but_ribbon_cables_swapped_between_bays_5_and_6_still_dark_urukul_card_in_bay_5}$

We do not think that the firmware on the Kasli SoC could cause this problem because we compiled it with artiq-zynq commit a0281e498927d5c8467dd18f28aa497cb75150fd (latest commit on the master branch at the time of writing in January 2025) and with this system description file:

{
    "target"            : "kasli_soc",
    "hw_rev"            : "v1.1",
    "base"              : "use_drtio_role",
    "drtio_role"        : "satellite",
    "sed_lanes"         : 32,
    "enable_wrpll"      : true,
    "min_artiq_version" : "8.0",
    "variant"           : "kasli_soc_artiq_master_20250109_drtio_satellite_wrpll_true_sed_lanes_32_peripherals_Urukul_6",
    "peripherals"       : [
    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [0, 1],
        "synchronization": true,
        "clk_sel": 2
    },
    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [2, 3],
        "synchronization": true,
        "clk_sel": 2
    },
    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [4, 5],
        "synchronization": true,
        "clk_sel": 2
    },
    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [6, 7],
        "synchronization": true,
        "clk_sel": 2
    },
    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [8, 9],
        "synchronization": true,
        "clk_sel": 2
    },
    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [10, 11],
        "synchronization": true,
        "clk_sel": 2
    }
    ]
}

Finally, we confirmed that the problematic Urukul board's IFC_MODE_SEL0 through IFC_MODE_SEL3 (switch SW1) are in the same position as for all other Urukul boards.

At this point, we are certain that our problematic Urukul board has some issue.

Is there smth easy for us to check on the hardware? Or should the whole Urukul board be sent for repairs / thrown away?

occheung

dtsevas
In your JSON file, the problematic Urukul is this entry:

    {
        "type": "urukul",
        "hw_rev": "v1.5",
        "ports": [10, 11],
        "synchronization": true,
        "clk_sel": 2
    }

But it looks like the urukul is an AD9912 instead of AD9910. You should add "dds": "ad9912" to this JSON entry.
See our assembly guide on Urukul.

Edit: Synchronization feature is not available to AD9912.

dtsevas

Oh shit, thank you for spotting the AD9912 label! Another team did me dirty. One of our Sinara systems arrived earlier than theirs, so they took ours and we later received theirs, which apparently contains one Urukul with AD9912 for testing. I'll recompile the firmware.