DRTIO link cannot initialize

Guest

Hello,

I am upgrading an existing ARTIQ 9 Kasli-SoC system to add two Phaser Baseband boards using MTDDS over DRTIO-over-EEM.

The intended topology is:

Both Phaser boards have already been flashed separately with MTDDS gateware configured as:

hardware_variant = baseband
gateware_variant = mtdds
tones = 12
dds_bandwidth = 500 MHz

The Phaser flashing completed successfully.

Master JSON

The Master is a Kasli-SoC v1.2 at 192.168.1.75:

{
"target": "kasli_soc",
"min_artiq_version": "7.0",
"variant": "unitaryquantum6master",
"hw_rev": "v1.2",
"base": "master",
"core_addr": "192.168.1.75",

"peripherals": [
    {
        "type": "urukul",
        "dds": "ad9910",
        "ports": [0, 1],
        "synchronization": true,
        "clk_sel": 2,
        "proto_rev": 9
    },
    {
        "type": "urukul",
        "dds": "ad9910",
        "ports": [2, 3],
        "synchronization": true,
        "clk_sel": 2,
        "proto_rev": 9
    },
    {
        "type": "urukul",
        "dds": "ad9910",
        "ports": [4, 5],
        "synchronization": true,
        "clk_sel": 2,
        "proto_rev": 9
    },
    {
        "type": "urukul",
        "dds": "ad9910",
        "ports": [6, 7],
        "synchronization": true,
        "clk_sel": 2,
        "proto_rev": 9
    },
    {
        "type": "urukul",
        "dds": "ad9910",
        "ports": [8, 9],
        "synchronization": true,
        "clk_sel": 2,
        "proto_rev": 9
    },
    {
        "type": "urukul",
        "dds": "ad9910",
        "ports": [10, 11],
        "synchronization": true,
        "clk_sel": 2,
        "proto_rev": 9
    }
]

}
Satellite JSON

The Satellite is also Kasli-SoC v1.2.

Its current JSON is:

{
"target": "kasli_soc",
"min_artiq_version": "7.0",
"variant": "unitaryquantum6satellite",
"hw_rev": "v1.2",
"base": "satellite",

"peripherals": [
    {
        "type": "fastino",
        "ports": [0]
    },
    {
        "type": "fastino",
        "ports": [1]
    },
    {
        "type": "fastino",
        "ports": [2]
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [3],
        "bank_direction_low": "input",
        "bank_direction_high": "output"
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [4],
        "bank_direction_low": "input",
        "bank_direction_high": "output"
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [5],
        "bank_direction_low": "output",
        "bank_direction_high": "output"
    },
    {
        "type": "dio",
        "board": "DIO_SMA",
        "ports": [6],
        "bank_direction_low": "output",
        "bank_direction_high": "output"
    },
    {
        "type": "phaser_drtio",
        "gateware_variant": "mtdds",
        "hardware_variant": "baseband",
        "tones": 12,
        "dds_bandwidth": 500e6,
        "ports": [7]
    },
    {
        "type": "phaser_drtio",
        "gateware_variant": "mtdds",
        "hardware_variant": "baseband",
        "tones": 12,
        "dds_bandwidth": 500e6,
        "ports": [10]
    }
]

}

The Satellite build completes successfully.

The generated gateware reports:

DrtioOverEEM (EEM7) starting at DRTIO channel 0x050000
DrtioOverEEM (EEM10) starting at DRTIO channel 0x060000

The timing constraints are also met.

Routing table

I currently use:

0: 0
1: 1 0
5: 1 5 0
6: 1 6 0

The Master confirms that it loads:

RoutingTable { 0: 0; 1: 1 0; 5: 1 5 0; 6: 1 6 0; }
Working baseline

An important A/B test is that the same Master and the same physical Master-Satellite DRTIO connection work correctly when I use a Satellite build without any phaser_drtio peripherals.

With the baseline Satellite firmware I get:

[LINK#0] remote replied
[LINK#0] link initialization completed
[DEST#1] destination is up
[DEST#1] buffer space is 128

The Satellite then reports:

ARTIQ satellite manager starting...
uplink is up, switching to recovered clock
...
siphaser calibration successful
TSC loaded from uplink
rank: 1

So the following appear to be working:

Master firmware
Satellite hardware
Master-Satellite optical DRTIO link
SFPs / fiber
DEST#1 routing
Failure after enabling Phaser DRTIO-over-EEM

As soon as I build the Satellite firmware with the two phaser_drtio peripherals, the Master sees:

[LINK#0] link RX became up, pinging
...
[LINK#0] ping failed
[LINK#0] link RX became up, pinging
...
[LINK#0] ping failed

and this repeats continuously.

Because DEST#1 never becomes reachable,

artiq_coremgmt -D 192.168.1.75 -s 1 log

fails with:

OSError: Incorrect reply from device:
Reply.Error (expected Reply.LogContent)
Satellite UART error

Direct UART output from the Satellite shows:

ERROR(libboard_artiq::drtio_eem):
setup/hold timing calibration failed, retry in 1s...

This repeats indefinitely:

setup/hold timing calibration failed, retry in 1s...
setup/hold timing calibration failed, retry in 1s...
setup/hold timing calibration failed, retry in 1s...
...
EEM port correction already tested

Initially I mistakenly configured the second Phaser on EEM8.

After checking the Kasli-SoC PCB silk screen carefully, I found that the actual physical connections are:

Phaser #1 -> EEM7
Phaser #2 -> EEM10

I rebuilt the Satellite firmware with EEM7 and EEM10.

The generated gateware now correctly reports:

DrtioOverEEM (EEM7) starting at DRTIO channel 0x050000
DrtioOverEEM (EEM10) starting at DRTIO channel 0x060000

However, the same setup/hold calibration failure remains.

Questions

Could you please help clarify the following?

Is this topology officially supported?
Kasli-SoC Master
-> Kasli-SoC Satellite
-> Phaser Baseband MTDDS via DRTIO-over-EEM
-> Phaser Baseband MTDDS via DRTIO-over-EEM
Is the following Phaser configuration correct?
{
"type": "phaser_drtio",
"gateware_variant": "mtdds",
"hardware_variant": "baseband",
"tones": 12,
"dds_bandwidth": 500e6,
"ports": [7]
}

and similarly for EEM10?

Is the routing table below correct?
0: 0
1: 1 0
5: 1 5 0
6: 1 6 0
Does:
DrtioOverEEM (EEM7) starting at DRTIO channel 0x050000

mean that this Phaser is DRTIO destination #5, while the second one at 0x060000 is destination #6?

What exactly causes:
setup/hold timing calibration failed

for DRTIO-over-EEM?

Could this indicate:

incorrect EEM cabling
missing clock connection
stale eem_drtio_delay
Phaser gateware/storage state
DRTIO-over-EEM timing
or another hardware/configuration issue
Since both Phaser boards and the Kasli-SoC Satellite were reflashed, should eem_drtio_delay be explicitly erased on both the Satellite and Phaser sides?
For a Zynq/Kasli-SoC Satellite, what is the recommended way to reset DRTIO-over-EEM calibration state?
Is there a way to enable additional debug output to identify whether EEM7 or EEM10 is the link that fails setup/hold calibration?
Would you recommend first testing with only one Phaser configured, for example only:
EEM7 -> Phaser #1

before enabling the second DRTIO-over-EEM peripheral?

Are there any known issues in ARTIQ 9 with Phaser MTDDS DRTIO-over-EEM when the carrier itself is a Kasli-SoC Satellite?

The key A/B result is:

Same Master
Same Satellite hardware
Same SFPs/fiber

Satellite without phaser_drtio:
PASS
DEST#1 up

Satellite with phaser_drtio:
FAIL
drtio_eem setup/hold calibration loops indefinitely
Master sees RX up but DRTIO ping fails

Any advice on the correct Phaser DRTIO-over-EEM initialization/calibration procedure would be appreciated.

Thanks.

morgan

Hello, welcome to the forum.

Is this topology officially supported?

Yes, DRTIO-over-EEM peripheral (i.e. phaser_mtdds) connected to satellite is supported.

Is the following Phaser configuration correct?

The json entry is correct.

Is the routing table below correct?

It's incorrect as SoC satellite only has 3 SFPs available after connecting SFP0 to master. So it should be
0: 0
1: 1 0
5: 1 4 0 (1-3 are SFP on the satellite, and after that are DRTIO-over-EEM, so we need to choose 4)
6: 1 5 0

mean that this Phaser is DRTIO destination #5, while the second one at 0x060000 is destination #6?

You are correct. The first 16 bits of the rtio address are the drtio_destination. While [code that prints DrtioOverEEM (EEM7) starting at DRTIO channel 0x050000] doesn't show this relationship, we can reference artiq_ddb_template.py to verify this.

What exactly causes: setup/hold timing calibration failed

There are lots of reasons that caused this issue as shown in DRTIO-over-EEM#link-revalibration

But in your case, the satellite spamming the setup/hold timing calibration failed, retry in 1s... usually means the phaser_mtdds is not sending any symbol for the satellite to calibrate the DRTIO-over-EEM link.

Can you check whether the phaser_mtdds mmcx 125 MHz clock is connected to the satellite? If not, can you provide a simple block diagram of how the clock is connected with their respecitvely frequency. As the clocking requirement is different from the phaser_classic.

Since both Phaser boards and the Kasli-SoC Satellite were reflashed, should eem_drtio_delay be explicitly erased on both the Satellite and Phaser sides?

Yes as stated in the DRTIO-over-EEM#link-revalibration

For a Zynq/Kasli-SoC Satellite, what is the recommended way to reset DRTIO-over-EEM calibration state?

I posted a detail step by step guide in another post here. A simple version can be found on DRTIO-over-EEM#link-revalibration

Is there a way to enable additional debug output to identify whether EEM7 or EEM10 is the link that fails setup/hold calibration?

The calibration orders should follow the json order as explained in another post.

Would you recommend first testing with only one Phaser configured, for example only:
EEM7 -> Phaser #1 before enabling the second DRTIO-over-EEM peripheral?

It's not recommended. 1) satellite can calibrate multiple DRTIO-over-EEM 2) there is a chance satellite used the wrong eem_drtio_delay{n} for the extra phaser and caused other DRTIO errors.

Are there any known issues in ARTIQ 9 with Phaser MTDDS DRTIO-over-EEM when the carrier itself is a Kasli-SoC Satellite?

Other than the calibration process of DRTIO-over-EEM and some small tone value on phaser_mtdds will fail timing. There are none that I know of.

Btw the forum support markdown syntax, so you can use backticks (`) to create a code block and improve readability https://www.markdownguide.org/extended-syntax/#fenced-code-blocks

jose

The topology is supported.

The JSON phaser configuration is correct for both EEM7 and EEM10. You can refer to the official phaser documentation here. And also make sure that the binaries where built with the correct commands.

The routing table depends on your configuration, please refer here.

Bruce

morgan thanks a lot, I fix this problem

Bruce

jose thanks