Loop with dma playback not working

max

Running on a Kasli board via artiq, the following loop is meant to play back a DMA sequence (created before):

                for k in range(shots):
                    self.core.break_realtime()
                    self.core_dma.playback_handle(sequence_handle)

But it doesn't: For a value of shots=50, the sequence is not played back actually 50 times, but less, seemingly randomly between about 35 and 45 times.
Artiq raises no exception or any other kind of error I'm aware of.

Does someone have an idea, how a single execution instance of a DMA sequence can be fine, but the loop above can fail? (My understanding was, that DMA sequences are just static sequences of RTIO output events, and it should make no difference, how many times they are actually executed).

Could this be associated with the issue?
https://github.com/m-labs/artiq/issues/946

It doesn't sound like it though, as I'm creating fewer pulses (<100), which all are longer (> 100 us).

In case, that would be the reason for my issue: Would the issue persist, when I don't use DMA, but do everything in realtime instead (assuming I can manage the timing, and not create any underflows)?

Any other tips how to start debugging this are appreciated!

Max

PS: I'm using artiq 5 on the host PC and Kasli, and execute the
code using artiq_run in a nix environment on a Debian system.

max

I was since able to investigate a bit further, and it seems the function set_phase_mode() is the culprit here.

Some example code triggering the behavior is this:

    @kernel
    def run(self):
        SHOTS = 50
        is_standby = True
        STANDBY_CHUNK_DURATION = 1

        while True:
            self.core.break_realtime()
            if is_standby:
                self.krn_update_standby()
                delay(STANDBY_CHUNK_DURATION * s)
                is_standby = False
            else:
                self.krn_create_sequence_dma()
                sequence_handle = self.core_dma.get_handle("sequence")

                for k in range(SHOTS):
                    self.core.break_realtime()
                    self.core_dma.playback_handle(sequence_handle)
                is_standby = True

This loop just alternates between a "standby mode" and a "sequence mode", where the latter creates a DMA sequence and plays back 50 times.

The DMA sequence is created like that ( I have 12 Urukul channels available in dds_devs[]):

 @kernel
    def krn_create_sequence_dma(self):
        with self.core_dma.record("sequence"):
            with parallel:
                self.krn_set_dds_state()
                self.krn_set_digital_out_state()

    @kernel
    def krn_set_dds_state(self):
        FRQUENCY_MHZ = 150
        PHASE = 0.0
        AMPLITUDE = 0.3
        DDS_CHANNELS = 12
        for i in range(DDS_CHANNELS):
            delay(10 * us)

            self.dds_devs[i].set(FRQUENCY_MHZ * MHz,
                                 phase=PHASE,
                                 amplitude=AMPLITUDE)

        for i in range(DDS_CHANNELS):
            delay(10 * us)
            self.dds_devs[i].sw.on()
            delay(10 * us)
            self.dds_devs[i].sw.off()

    @kernel
    def krn_set_digital_out_state(self):
        self.digital_out_devs[0].pulse(5 * ms)
        delay(5 * ms)

Now for the "standby mode" (this is, where I'm setting the phase mode):

    @kernel
    def krn_update_standby(self):
        self.core.break_realtime()
        self.krn_update_digital_out_standby()
        self.krn_update_dds_standby()
        return

    @kernel
    def krn_update_digital_out_standby(self):
        self.core.break_realtime()
        DIGITAL_OUT_CHANNELS = 19
        for i in range(DIGITAL_OUT_CHANNELS):
            self.core.break_realtime()
            self.digital_out_devs[i+1].pulse(100 *us)
            delay(100 * us)
        return

    @kernel
    def krn_update_dds_standby(self):
        self.core.break_realtime()
        FREQUENCY_MHZ = 150
        PHASE = 0.0
        AMPLITUDE = 0.5
        STANDBY_CHANNELS = 4  # <- critical parameter
        for i in range(STANDBY_CHANNELS):
            self.core.break_realtime()
            self.dds_devs[i].set_phase_mode(3)  # <- critical function
            delay(1 * ms)
            self.dds_devs[i].set(FREQUENCY_MHZ * MHz, phase=PHASE, amplitude=AMPLITUDE)

        for i in range(STANDBY_CHANNELS):
            self.dds_devs[i].sw.on()
            delay(10 * ms)
            self.dds_devs[i].sw.off()
        return

One would expect, that the DMA sequence is executed 50 times, and there are therefore 50 digital pulses output on DIO channel 0.

But this isn't what's happening:
Seemingly randomly, there are only between 35 and 50 digital pulses created.
If I remove the call to set_phase_mode() in the non-DMA part, the DMA sequence works as intended.
Those pulses are silently omitted, I only found when actually checking the output.

I couldn't find any errors, exceptions or any other indication, that something was wrong.

Moreover, with set_phase_mode() in place, the amount of "surviving" digital pulses in dependent on STANDBY_CHANNELS: If this number is decreased, less pulses are omitted, and if it is increased above 6, no pulses are left at all.

I'm aware of the comment in the documentation, that set_phase_mode() can be problematic when used in a DMA sequence, but that is not the case here.

To me, this looks like a bug.

But before filing it, I wanted to check with the reads of this forum, if I'm getting something wrong.

Thanks for reading, and sorry for the large amount of example code!

Max

max

Another addition: The problem persists, if I use the phase_mode argument from the set() method instead.

It is less pronounced though, i.e. less lost pulses while using more STANDBY_CHANNELS. But it's still there, and it disappears, if I don't set the argument.

How do I safely set the phase mode on an Urukul channel?

Max

rjo

phase_mode = 3 is undefined.
What's the intention?

max

rjo
Oh, my bad! But the the problem persists, if a valid value for the phase mode is set (e.g. phase_mode(1)).
The intention is to set the phase mode of the Urukul in a safe fashion.
I'm sorry, if this doesn't come across in the code example above, but I tried to keep it somewhat minimal.

rjo

Ok. There are two potential routes of investigation. One (maybe somewhat less likely) is https://github.com/m-labs/artiq/issues/1521 which has similar symptoms but different conditions (wide RTIO and current ARTIQ master).
The other is what the phase mode means for DMA. DMA only records the RTIO events as they would be emitted and not any CPU-side state (e.g. not any change to the the self.phase_mode property). That makes setting/using phase_mode very tricky with DMA. Reproducing it with current ARTIQ master would be my first step. I'll try to set this up but it might take a few days.

max

Concerning the issue with self.phase_mode and DMA: Are you referring to this warning from the documentation:

This setting may become inconsistent when used as part of a DMA recording. When using DMA, it is recommended to specify the phase mode explicitly when calling set() or set_mu().

If so, please note, that the issue persist, if i use set()/set_mu(), as suggested.
Also note, that in the example above, 'self.phase_mode` ist not set in DMA mode, but outside of it.

rjo The other is what the phase mode means for DMA. DMA only records the RTIO events as they would be emitted and not any CPU-side state (e.g. not any change to the the self.phase_mode property). That makes setting/using phase_mode very tricky with DMA.

I'm afraid, I can't quite follow you here:
If I understand the code correctly, set_mu() effectively writes to profile0 of the AD9910 viaself.write64(). That means, the DDS' asf, pow and ftw are all written there.
The chosen phase mode just determines the specific value of the pow in that self.write64.
Depending on the chosen phase mode, additionally set_cfr1() (effectively a self.write32()) needs to be called, to set the autoclear phase accumulator bit.
My understanding was, that those self.write32/64() calls are comprised in the end of RTIO events, and those are recorded/played back, when using DMA.
I don't understand, what would be only a change to the CPU state here.

So, at least using set() to set the phase mode instead of set_phase_mode() should work in DMA, the same way setting the frequency and amplitude works, right?

Well, obviously not, so it would be great, if you could point me to, where I'm getting things wrong.

Also, if I can do something more on my side to investigate this more, I'll be happy to help.

max

Some update on this:

It seems, the issue wasn't really related to set() or set_phase_mode() especially, but more to the fact that the entirety of my kernel code (of which the examples above are only excerpts) was causing Artiq sequence errors (as in https://m-labs.hk/artiq/manual/rtio.html). Getting rid of them also solved the issue described above.

rjo

Thanks for following up!
There was some discussion and proposals about raising the severity of the asynchronously detected and reported errors (like SequenceError) so that they would not be overlooked. But I can't find it right now.

airwoodix

@rjo: do you mean #1644?

rjo

That's certainly one. Thanks