Urukul PR9 set_profile non-deterministic intermediate hamming path activation

tomhepz

Hi, I have been using the ProtoRev9 Urukul to perform a complicated pulse sequence by changing the profile pins on each of the AD9910's in an Urukul card (new ProtoRev9 Feature). While programming this sequence, I kept experiencing unexpected tones appearing on my scope after changing profile pins while on a satellite urukul.

I believe I have narrowed this down to the effects of intermediate profiles on the Hamming path activating, although I am not certain the mechanism beyond that as I am not certain how the SPI transfer and subsequent activation of the profile pins occurs at the CPLD level. I would expect that the CPLD would simultaneously change the state of all the profile pins at least simultaneously? Although I don't know how the 'clock domains' if that's the right word cross between the SPI bus, CPLD, and the urukul.

It appears to be non-deterministic on two levels

When changing with from profile x to profile y with set_profile(y) , sometimes the hamming path intermediate between x and y latches on and stays on, with some given probability (see below). E.g. if changing from profile 6 = 110_b to profile 0=000_b, I sometimes end up playing profile 2=010_b or profile 4=100_b.
When turning off and on the crate with the urukul in, the amount of randomness appears to be random. Some power on cycles I see no errors, others I see lots.

This to me, an uninitiated fellow, appears to be some random clock synchronisation issue that sometimes lines up with the chip reading the profile pins on an unstable edge. However locking to the coarse RTIO clock with at_mu(now_mu() & ~7) doesn't work and this issue still appears. Perhaps the SPI core is aligned to the coarse RTIO clock (125MHz), but the urukul SYNC_CLK = SYS_CLK/4 (Also 125MHz, derived from the core clock through the MMCX) has some random phase that sometimes makes the SPI writing to the register while it's being read?

I have seen this behaviour on urukuls in the master crate, and in the satellite crate, although (purely based on an empirical data) that I have noticed it be more problematic on the satellite. This could be confirmation bias from using the satellite and power cycling it more to test.

Does anyone more acquainted with the Urukul control have any ideas? Further things I could test?
Thanks for your help in advance 🙂 .

Here is a fairly minimal 'working' example:
(set scope to 1ms/div)
It should look like (smallamp-off-bigamp-off)⁵⁰ but sometimes randomly looks broken.
See below screenshots

from artiq.language.units import MHz, dB, s, ms, us, V, ns
from artiq.language.core import delay, kernel, rpc, delay_mu
from artiq.language.environment import EnvExperiment
from artiq.coredevice.ad9910 import AD9910
from artiq.coredevice.core import Core
from artiq.coredevice.urukul import CPLD
from artiq.coredevice.ttl import TTLOut

class UrukulWeirdExample(EnvExperiment):
    
    def build(self):
        self.setattr_device("core")
        self.core: Core
        self.setattr_device("ttl0")
        self.ttl0: TTLOut

        self.dds_cpld: CPLD = self.get_device("urukul6_cpld")
        self.dds_ch: AD9910 = self.get_device("urukul6_ch1")
        
        kernel_invariants = getattr(self, "kernel_invariants", set())
        self.kernel_invariants = kernel_invariants | {"dds_cpld", "dds_ch"}
    
    @kernel
    def init_dds(self, dds):
        dds.init()
        dds.set_att(6.*dB)
        dds.cfg_sw(False)

    @kernel
    def configure_RB1AB_single_tone_mode(self, dds):
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.75, profile=1)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.85, profile=2)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.5, profile=3)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.8, profile=4)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.37, profile=5)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.37, profile=6)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.37, profile=7)

        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.0, profile=0) # Off
        delay(10*us) #Give some time for io update after
        dds.set_profile(0) # Set profile pins to default of 0. 
        # This should act like an IO_update, but given issue, also...
        # ... pulse IO_Update (this will pulse on all 4x AD9910 channels unfortunately) 
        dds.io_update.pulse_mu(8)
        delay(20*us)
        
    
    @kernel
    def run(self):
        self.core.reset()
        self.core.break_realtime()
        # Initialise and setup urukul cpld, channels, and attenuators
        self.dds_cpld.init()
        self.init_dds(self.dds_ch)

        # Setup single tone mode
        self.configure_RB1AB_single_tone_mode(self.dds_ch)

        self.core.break_realtime()
        # Profile playback
        # Note that set_profile actually advances the timeline
        # (by 0.86us when I measured it) so the delays for direct profile switches
        # are not necessarily the delays you expect. Therefore be careful when trying
        # to do a preciscely timed square pulse with the RAM_MODE_DIRECTSWITCH mode.
        # This is because the configuration register of the CPLD is written over SPI
        # Which *then* changes the profile pins internally.

        self.dds_ch.cfg_sw(True) # Enable RF switch

        # Lock cursor to a 4ns (coarse RTIO) timestamps (rounded down)
        at_mu(now_mu() & ~7)
        #delay(2*ns) # Phase shift?
        # Now everything must be done on a 4ns grid.
        # SPI set_profile is 860ns/4 = 215 so that's fine.
        # You just have to make it so that the delays are also multiples of 4ns

        self.ttl0.pulse(1*us) # scope trigger

        for i in range(50):
            # Axial n-4
            self.dds_ch.set_profile(6)  # 6 = 110_b
            delay(40*us)
            self.dds_ch.set_profile(0)  # 0 = 000_b
            delay(15*us)

            # I expect here we could accidently be in P2=010_b or P4=100_b
                        
            # Radial x
            self.dds_ch.set_profile(3)  # 3 = 011_b,
            delay(40*us)
            self.dds_ch.set_profile(0)
            delay(15*us)

            # I expect here we could accidently be in P2=010_b or P1=001_b

        self.dds_ch.cfg_sw(False)

Intended behavior (power cycled until it looked good):

After power cycling the satellite crate a few times until the condition occurs (took approx 6 power cycles):

Using artiq-9 master, and the latest ProtoRev9 CPLD firmware.

claytonho

this is an issue that's come up before, but it is a bit niche:
basically, yes, the "trajectory" (or "hamming path") of the profile pins as they change profiles is non-deterministic - see the following issues:

the root cause is that profile pins are set by the CPLD, which doesn't set them synchronously wrt the RTIO clock.

with regards to your "profile latching" problem, try adding a cpld.io_update.pulse_mu(8) after every profile update to ensure that the DDS latches the profile correctly - that's what we do, and it seems to work for us.

fyi you should be careful about profile-switching when you want phase coherence/tracking, as incorrect profiles can be temporarily loaded onto the AD9910 during switching, causing unknown phase shifts in the phase accumulator.
we personally get around that issue (when we care about phase coherence) by setting all the profiles to have the same frequency, or simply not switching profiles lol

tomhepz

claytonho Thanks for your help and knowledge,

I had tried adding cpld.io_update.pulse_mu(8) after the profile update however for this pulse train, some of the AD9910s are in RAM mode, and some are in single tone mode, and the pulses are not necessarily starting/ending at the same time. So pulsing the shared io_update pin will restart the RAM profile playback. I feel this undoes the help the new ProtoRev9 per-chip .set_profile() gives. I guess the equivalent here would be to just split the hamming path into the full path and set the profile pins one after the other and use the RF switch to block out any transients (at the expense of the phase accumulation you mention, although this won't be a problem for my specific use case).

There has been lots of work done I believe for phase coherent SU servo and also the synchronisation of the AD9910's across the chips that if there is a regime where this works nicely, I feel like we could get around this by synchronising the clocks. My initial thought is to brute force reset the PLL of the urukul in a loop until the 'bad' clock regime is removed, provided we can sense this through the register values.

claytonho

tomhepz
here’s an idea wrt using global io_update but having it only address certain channels: try using the mask_nu bits in urukul - these disable w given channel from listening to global io_update

tomhepz

I have done some more digging today and found that manually making a call to self.dds_ch.tune_sync_delay(13) after dds initialisation fixes this issue (in that I have not been able to reproduce the failure condition).

It appears because my EEPROM was not populated with a value for the sync seed delay this part was being skipped in device initialisation, despite me having the sync machinery enabled in my hardware description and device_db.

Here the prints are the values in the sync register, and initially after a power cycle these are not set, but after calling the tune sync delay, it is populated, and this register value persists across runs (assuming the EEPROM empty). See code below.

I can also confirm that the proposed 'use DMA to set profiles' does not work to fix this problem at least in my case as was suggested in https://github.com/m-labs/artiq/issues/1994 .

Some more questions to those in the know:

Is this the expected way to go about this?
The tune_sync_delay accepts dds_channel_idx which is weird (it should know the channel index right?) Is this for some SUservo reason?
Is the SPI clock derived from the same clock the urukul is sync'd to?
How is this EEPROM expected to have been written, as it is empty in my crate I see the SyncDataEeprom class in urukul.py. I note that test_urukuls in artiq_sinara_tester.py appears to write to this EEprom after a calibration. This is quite an odd way I feel to expect this to happen? I.e. your crate will only function if you have at some point run this command... I feel we can document this better

Below is the example that should 'always work'

from artiq.language.units import MHz, dB, s, ms, us, V, ns
from artiq.language.core import delay, kernel, rpc, delay_mu
from artiq.language.environment import EnvExperiment
from artiq.coredevice.ad9910 import AD9910
from artiq.coredevice.core import Core
from artiq.coredevice.urukul import CPLD
from artiq.coredevice.ttl import TTLOut

from artiq.coredevice.ad9910 import _AD9910_REG_SYNC
from numpy import int32


"""
      110-------111
     / |       / |
    /  |      /  |
  010-------011  |

   |   |     |   |
   |  100----|--101
   | /       | /
   |/        |/
  000-------001
"""
class UrukulWeirdExample(EnvExperiment):
    
    def build(self):
        self.setattr_device("core")
        self.core: Core
        self.setattr_device("core_dma")
        self.setattr_device("ttl0")
        self.ttl0: TTLOut

        self.dds_cpld: CPLD = self.get_device("urukul6_cpld")
        self.dds_ch: AD9910 = self.get_device("urukul6_ch1")
        
        kernel_invariants = getattr(self, "kernel_invariants", set())
        self.kernel_invariants = kernel_invariants | {"dds_cpld", "dds_ch"}
    
    @kernel
    def init_dds(self, dds):
        dds.init()
        dds.set_att(6.*dB)
        dds.cfg_sw(False)

    @kernel
    def configure_RB1AB_single_tone_mode(self, dds):
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.75, profile=1)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.85, profile=2)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.5, profile=3)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.8, profile=4)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.37, profile=5)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.37, profile=6)
        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.37, profile=7)

        dds.set(frequency=2*MHz, phase=0.0, amplitude=0.0, profile=0) # Off
        delay(10*us) #Give some time for io update after
        dds.set_profile(0) # Set profile pins to default of 0. 
        # This should act like an IO_update, but given issue, also...
        # ... pulse IO_Update (this will pulse on all 4x AD9910 channels unfortunately) 
        dds.io_update.pulse_mu(8)
        delay(20*us)


    @kernel
    def check_print_sync_reg(self):
        reg0a = int32(self.dds_ch.read32(_AD9910_REG_SYNC))
        self.core.break_realtime()
        # Decode bits according to datasheet/ARTIQ set_sync():
        window   = (reg0a >> 28) & 0xF      # [31:28]
        rx_en    = (reg0a >> 27) & 0x1      # [27]
        gen_en   = (reg0a >> 26) & 0x1      # [26]
        gen_pol  = (reg0a >> 25) & 0x1      # [25]
        preset   = (reg0a >> 18) & 0x3F     # [23:18]
        in_delay = (reg0a >> 3)  & 0x1F     # [7:3]

        # Read CPLD status register and extract per-channel bits
        sta = self.dds_ch.cpld.sta_read()
        smp_err_all = (sta >> 4) & 0xF      # STA_SMP_ERR base bit offset = 4
        pll_lock_all = (sta >> 8) & 0xF     # STA_PLL_LOCK base bit offset = 8

        ch = self.dds_ch.chip_select - 4            # chip_select 4..7 -> channel 0..3
        smp_err = (smp_err_all >> ch) & 0x1
        pll_lock = (pll_lock_all >> ch) & 0x1

        print(reg0a, rx_en, gen_en, gen_pol, window, preset, in_delay, smp_err, pll_lock)

    @kernel
    def run(self):
        self.core.reset()
        self.core.break_realtime()
        # Initialise and setup urukul cpld, channels, and attenuators
        self.dds_cpld.init()
        self.init_dds(self.dds_ch)
        
        # If you haven't set the EEPROM at any point, then after a power cycle this will read
        # 0 0 0 0 0 0 0 0 1
        # You can run `artiq_sinara_tester --only urukuls` to write values to the EEPROM
        self.check_print_sync_reg()
        self.core.break_realtime()

        in_delay_func, window_func = self.dds_ch.tune_sync_delay(13) # This call also has dds_ch_index for some reason
        print(in_delay_func, window_func)
        self.core.break_realtime()

        self.check_print_sync_reg()
        self.core.break_realtime()

        # Setup single tone mode
        self.configure_RB1AB_single_tone_mode(self.dds_ch)

        self.core.break_realtime()
        # Profile playback
        # Note that set_profile actually advances the timeline
        # (by 0.86us when I measured it) so the delays for direct profile switches
        # are not necessarily the delays you expect. Therefore be careful when trying
        # to do a preciscely timed square pulse with the RAM_MODE_DIRECTSWITCH mode.
        # This is because the configuration register of the CPLD is written over SPI
        # Which *then* changes the profile pins internally.

        self.dds_ch.cfg_sw(True) # Enable RF switch

        # Lock cursor to a 4ns (coarse RTIO) timestamps (rounded down)
        at_mu(now_mu() & ~7)
        delay_mu(2) # Phase shift?
        # Now everything must be done on a 4ns grid.
        # SPI set_profile is 860ns/4 = 215 so that's fine.
        # You just have to make it so that the delays are also multiples of 4ns

        self.ttl0.pulse(1*us) # scope trigger

        for i in range(50):
            # Axial n-4
            self.dds_ch.set_profile(6)  # 6 = 110_b
            delay(40*us)
            self.dds_ch.set_profile(0)  # 0 = 000_b
            delay(15*us)

            # I expect here we could accidently be in P2=010_b or P4=100_b
                        
            # Radial x
            self.dds_ch.set_profile(3)  # 3 = 011_b,
            delay(40*us)
            self.dds_ch.set_profile(0)
            delay(15*us)

            # I expect here we could accidently be in P2=010_b or P1=001_b

        self.dds_ch.cfg_sw(False)

dpn

fyi you should be careful about profile-switching when you want phase coherence/tracking, as incorrect profiles can be temporarily loaded onto the AD9910 during switching, causing unknown phase shifts in the phase accumulator.

From our (Oxford) work on the phase coherence/multi-chip sync implementation back then, I recall never quite being sure what exactly the setup/hold requirements are on the PROFILE pins, except that they are probably sampled on SYNC_CLK. The latter suggests that aligning any profile transitions to it (much in the same way as done for IO_UPDATE) would potentially give deterministic results. You'd probably want to customise your CPLD gateware to gate PROFILE updates behind a latch clocked from a separate LVDS pair. IIRC in non-SUServo mode, there ought to be a diff pair spare still. This way, you could easily tune the timing.

As for doing this via the timing of the SPI writes to the CPLD directly, what you would be missing is the fine alignment within the 8 ns period to make sure the signals are set up at the SYS_CLK edges (spaced by 4 ns). It's possible that using SERDES outputs for the SPI signals would allow you to do the same (I haven't checked how the timing in the PHY works), but the jitter in that could probably use verification using a fast scope and careful probing. You could of course get lucky and find that in your particular setup, 0 ns offset w.r.t. the coarse RTIO timeline (SPI core just running at 125 MHz) just happens to meet the setup requirements for whatever SYNC_CLK ends up being in relation to that.

If you get this to work reliably, I'd be interested to know!

urukul SYNC_CLK = SYS_CLK/4 (Also 125MHz, derived from the core clock through the MMCX)

SYNC_CLK is usually 1 GHz / 4 = 250 MHz. Of course, aligning to an 8 ns grid, i.e. twice the period, still does the trick, as long as you know the alignment. I don't use the EEPROM storage for the sync delay seeds/… (instead passing them manually in the constructor as our original implementation in Oxford didn't have EEPROM/auto-tuning support), but yes, without it, the SYNC_CLK alignment will not be constant w.r.t. the clock used by Kasli (125 MHz -> 1 GHz -> 250 MHz leads to a random phase offset in the latter). And yes, all the relevant clocks on Kasli (e.g. the SPI core) should be the same as the MMCX outputs.