DRTIO link between Kasli and EFC

fevi_at_jila

Hi,

I have a Kasli and EFC1v1. My goal is to drive a FMC card on the EFC with an interface similar to the PDQ units.
Initially I had some issues with the clocking and I am happy to report that I got the boards up with 100 MHz core/RTIO frequencies. Thanks for the help to troubleshoot that!

When I boot up the boards, I see the the comma alignment step is passed but then the Kasli sends a DRTIO ping to which it doesn't get a response. At the same time the EFC generated a lot of error messages, basically indicating that received packets are invalid. For this I am forwarding the clock via the MMCX connector, removing the cable prevents the EFC from booting, and when the cable is present I get valid UART output for the expected 100 MHz frequency - so I think the clocking here looks solid.

I figured this may be an issue related to my 100 MHz clock setup, so I tried to go back to the standard 125 MHz frequency. When building the shuttlerdemo, the EFC errors out when some hardware ID doesn't match. So I changed the EFC gateware target so that no shuttler-specific gateware modules are build when the variant input does not match "shuttler" while making "shuttler" the default for consistency.

When building gateware with the "--variant blank" AND for 125 MHz (no --drtio100mhz option) I expect to get gateware that establishes the DRTIO link, pings successfully and idles. That is not what I observe, instead I am getting the same DRTIO packet errors that I was seeing with my 100 MHz version. This happens both for a forwarded 125 MHz clock as well as when both boards run on their internal 125 MHz oscillators.

Could you please review what I did here and give me some feedback regarding what I might be missing here? I think it'd be very useful to have a standard EFC gateware build that does not require any specific FMC module (like the shuttler one) and comes up indicating a good DRTIO link. Perhaps my troubleshooting patch (see attached) can serve as the basis for such build. I'll be happy to continue working on this to turn it into a PR if you can agree that a basic test build (without any FMC module attached) is a good idea.

Thank you for the help!

Felix

PS: Uploading patches isn't allowed, so here is a link instead:
Patch to build blank EFC

occheung

Yes I was building on the latest master.

fevi_at_jila That is not what I observe, instead I am getting the same DRTIO packet errors that I was seeing with my 100 MHz version.

If you have ~~aligned the comma~~ performed setup/hold time alignment with different hardware configuration (e.g. different clock), you will need to erase the computed delay.
See this.
https://github.com/m-labs/artiq/blob/de8f8af3dd5f9b858343b38df8088b4ccbe98c81/artiq/firmware/libboard_artiq/drtio_eem.rs#L214
Both master and satellite use this key.

occheung

Haven't built anything working yet, but have you adjusted MiSoC clock frequencies as well? I got the MMCM 5x clock report as such after synthesis:

Clock                                        Waveform(ns)         Period(ns)      Frequency(MHz)
-----                                        ------------         ----------      --------------
...
    main_satellite_crg_mmcm_sys5x            {0.000 0.800}        1.600           625.000

The correct value should be 500 MHz.

I think the frequency is hard coded by the MMCM in MiSoC, it should be adjusted accordingly. Though it is just a constraint number.

occheung

I got a link.

[     4.721509s]  INFO(runtime::rtio_mgt::drtio): [LINK#3] link RX became up, pinging
[     4.930068s]  INFO(runtime::rtio_mgt::drtio): [LINK#3] remote replied after 1 packets
[     5.303093s]  INFO(runtime::rtio_mgt::drtio): [LINK#3] link initialization completed
[     5.309546s]  INFO(runtime::rtio_mgt::drtio): [DEST#0] destination is up
[     5.317966s]  INFO(runtime::rtio_mgt::drtio): [DEST#4] destination is up
[     5.323371s]  INFO(runtime::rtio_mgt::drtio): [DEST#4] buffer space is 128

Patch for MiSoC. https://pastebin.com/iV1pv8zv
I haven't verified if the DRTIO connection actually works.

fevi_at_jila

Thank you, I missed the MiSoC piece for the 100 MHz version!

The problem I described above occurs at 125 MHz, though, where the MiSoC hard coded version should not cause any trouble.

In particular, I have the Kasli and EFC connected via EEM 1 on the Kasli (remember EEM0 could not be shuttler, which caused the Kasli to not come up) to EEM0 on the EFC. Then I have a clock connection from the Kasli J1 to INT CLK IN on the EFC. The Kasli is running off of the internal 125 MHz clock.

In this configuration I do not get a link and I see the problem described in my initial post above. Would you mind describing your setup and how you built gateware/firmware in a little more detail so I can follow more closely?

Thank you for the help!

fevi_at_jila

Hang on, I see there are a bunch of changes in your upstream that are not yet reflected in my repository.
Let me pull that in, run the basic test at 125 MHz and if that works I'll apply the MiSoC patch to hopefully get this up at 100 MHz as well.

Thanks!

occheung

The full setup is the following:

Kasli 2.0, running 100 MHz DRTIO clock by specifying "rtio_frequency": 100e6 and int_100 clocking via artiq_coremgmt
EFC 1.1 (with a patch that forwards drtio100mhz) Patch: https://pastebin.com/LP7NMepJ
Connected through EEM 0 of both Kasli and EFC.
I have also disabled DAC init intended for Shuttler, so it doesn't panic immediately.

I can blink the EFC LEDs from the artiq_sinara_tester script.
The EFC log can be captured through coremgmt-over-DRTIO.

$ artiq_coremgmt -D 192.168.1.200 -s 4 log
[     0.008512s]  INFO(satman): ARTIQ satellite manager starting...
[     0.013229s]  INFO(satman): software ident 9.0+unknown.beta;satellite
[     0.019667s]  INFO(satman): gateware ident 9.0+unknown.beta;satellite
[     0.026108s]  INFO(satman): log level set to INFO by default
[     0.031741s]  INFO(satman): UART log level set to INFO by default
[     0.105602s]  INFO(satman): SED spreading disabled by default
[     0.110276s]  INFO(board_artiq::drtio_eem): loading calibrated timing values from flash
[     0.118300s]  INFO(satman): uplink is up, switching to recovered clock
[     0.711121s]  INFO(satman): TSC loaded from uplink
[    92.082152s]  INFO(satman): resetting RTIO

fevi_at_jila

Thank you! There were three things that were off here:

When working with Dip in October, we swapped the peripheral order in kasli_shuttler.json because the board would not generate any serial output at all with shuttler on port 0. Turns out shuttler is needed to be at port 0 for timing, is that correct?
I needed to erase the computed delay from flash. I had values for 125 MHz saved and didn't realize that.
Pulling from the repository made the workaround for 1. no longer needed so that I could move the shuttler peripheral back to port 0.

With these changes, I now have a working 100 MHz setup in front of me.

PS: Turns out I had the MiSoC clock issue addressed the same way you did, either one of those versions works.

Is there anything you want me to send a PR for or do you already have everything that got changed?

Thanks again for the help!

Felix

occheung

fevi_at_jila When working with Dip in October, we swapped the peripheral order in kasli_shuttler.json because the board would not generate any serial output at all with shuttler on port 0. Turns out shuttler is needed to be at port 0 for timing, is that correct?

We also came up with some other workarounds, like declaring the shuttler entry before the DIO entry in the JSON file, while keeping the EEM port the same; or manually interchange the DIO IO and shuttler IO on the generated verilog. In general, I found that any change of gateware may workaround the issue.

fevi_at_jila Is there anything you want me to send a PR for or do you already have everything that got changed?

Please send us PRs.