CLK LED red, UART unavailable

Bruce

Hello M-Labs team,

I am currently upgrading an ARTIQ 9 system based on Kasli-SoC with two Phaser MTDDS boards connected through DRTIO-over-EEM.

After rebuilding and flashing the satellite firmware, the Kasli-SoC satellite cannot boot normally.

The current situation is:

All LEDs on the Kasli-SoC satellite are blinking continuously.
The CLK LED near the clock section is red.
UART output from the satellite is unavailable.
artiq_coremgmt cannot communicate with the satellite correctly.

I would like to ask whether this is related to:

incorrect satellite firmware generation,
clock initialization,
DRTIO-over-EEM calibration state,
or SD card configuration.
System information

ARTIQ:

ARTIQ 9

Hardware:

+-------------+
| Phaser MTDDS|
| 12 tones |
| 500 MHz |
+-------------+

Both Phaser boards are connected to the Kasli-SoC satellite.

Master JSON

The master configuration is:

{
"target": "kasli_soc",
"min_artiq_version": "7.0",
"variant": "unitaryquantum7master",
"hw_rev": "v1.2",
"base": "master",
"core_addr": "192.168.1.75",
"peripherals": [
{
"type": "urukul",
"dds": "ad9910",
"ports": [0, 1],
"synchronization": true,
"clk_sel": 2,
"proto_rev": 9
},
{
"type": "urukul",
"dds": "ad9910",
"ports": [2, 3],
"synchronization": true,
"clk_sel": 2,
"proto_rev": 9
},
{
"type": "urukul",
"dds": "ad9910",
"ports": [4, 5],
"synchronization": true,
"clk_sel": 2,
"proto_rev": 9
},
{
"type": "urukul",
"dds": "ad9910",
"ports": [6, 7],
"synchronization": true,
"clk_sel": 2,
"proto_rev": 9
},
{
"type": "urukul",
"dds": "ad9910",
"ports": [8, 9],
"synchronization": true,
"clk_sel": 2,
"proto_rev": 9
},
{
"type": "urukul",
"dds": "ad9910",
"ports": [10, 11],
"synchronization": true,
"clk_sel": 2,
"proto_rev": 9
}
]
}
Satellite JSON

The satellite configuration is:

{
"target": "kasli_soc",
"min_artiq_version": "7.0",
"variant": "unitaryquantum7satellite",
"hw_rev": "v1.2",
"base": "satellite",

"peripherals": [
    {
        "type": "fastino",
        "ports": [0]
    },
    {
        "type": "fastino",
        "ports": [1]
    },
    {
        "type": "fastino",
        "ports": [2]
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [3],
        "bank_direction_low": "input",
        "bank_direction_high": "output"
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [4],
        "bank_direction_low": "output",
        "bank_direction_high": "output"
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [5],
        "bank_direction_low": "output",
        "bank_direction_high": "output"
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [6],
        "bank_direction_low": "output",
        "bank_direction_high": "output"
    },
    {
        "type": "phaser_drtio",
        "gateware_variant": "mtdds",
        "hardware_variant": "baseband",
        "tones": 12,
        "dds_bandwidth": 500e6,
        "ports": [7]
    },
    {
        "type": "phaser_drtio",
        "gateware_variant": "mtdds",
        "hardware_variant": "baseband",
        "tones": 12,
        "dds_bandwidth": 500e6,
        "ports": [10]
    }
]

}
DRTIO routing table

The routing table currently used:

0: 0
1: 1 0
5: 1 4 0
6: 1 5 0

My understanding:

DEST#1:
Kasli-SoC satellite

DEST#5:
First Phaser MTDDS connected through EEM

DEST#6:
Second Phaser MTDDS connected through EEM
Previous working situation

I previously had a similar ARTIQ 9 system working.

For the previous system, the master log showed:

[DEST#1] destination is up
[DEST#5] destination is up
[DEST#6] destination is up

and the DRTIO-over-EEM calibration completed successfully.

Current problem

After flashing the new satellite firmware:

The satellite does not provide normal UART output.

The board status:

All LEDs blinking
CLK LED red

Sometimes before this issue, when the firmware was running, the master showed:

DEST#1 communication failed (timed out)
DEST#5 communication failed (timed out)
DEST#6 communication failed (timed out)

and the satellite showed:

setup/hold timing calibration failed, retry in 1s...
Things already tried

I have tried:

Rebuilding the satellite firmware.
Rebuilding the master firmware.
Updating the routing table:
0: 0
1: 1 0
5: 1 4 0
6: 1 5 0
Removing old calibration values:
eem_drtio_delay0
eem_drtio_delay1

from the satellite configuration.

Removing old files from SD card and keeping only:
boot.bin
Testing without Phaser boards connected.

The issue still appears.

Questions
Does the current symptom (all LEDs blinking + CLK LED red + no UART output) indicate that the generated boot.bin is incorrect?
Can an incorrect Kasli-SoC satellite JSON configuration cause the board to fail before UART initialization?
Is there a recommended recovery procedure when Kasli-SoC cannot output UART after flashing?
For Phaser MTDDS connected through DRTIO-over-EEM:
Should eem_drtio_delay be erased on both satellite and Phaser sides?
Is there any additional clock connection requirement?
Is the routing table:
0: 0
1: 1 0
5: 1 4 0
6: 1 5 0

correct for this topology?

Is there any recommended way to verify that the generated boot.bin is actually the satellite firmware (satman) rather than a normal runtime image?

I can provide:

generated build files,
full JSON files,
UART photos,
SD card contents,
and FPGA build logs.

Thank you.