Thanks for the reply! We managed to mitigate the problem somehow. We are not exactly sure why the mitigation works, but at least it works, for now.
Here's some additional information in case there are people who are interested: the
execute method above belongs to a "DAC helper" that we wrote. The DAC helper has methods that takes in values for voltages of waveforms that we wish to generate that correspond to various parts of the experiment, e.g.
<DAC_helper>.translate_ions(N=1, t=1*ms) would generate a table of discretized values that modulate the voltages of all the relevant channels in our DACs in a sine for N=1 cycle in a time t=1*ms. We also have a "TTL helper" that takes in all the TTL related sequences and converts them into a table of when each TTL channel turns on or off.
It turns out that if a "TTL helper" that runs before a DAC helper (i.e,
<TTL_helper>.execute() is >=1 line above
<DAC_helper>.execute()) and the "TTL helper" has a sequence that occurs after some of the DAC sequences, some of the remaining DAC sequences will not execute. This also happens even if the exact time at which the TTL event happens is different from when the DACs are supposed to update their values. We used to believe that as long as we do not have more than 8 events happening at the exact same time (for an 8 laned SED) this would not happen, and we certainly do not have 8
zotino.set_dac(...) happening at the exact same
at_mu(), but it seems like we do not fully understand how the system works.
We mitigate this problem by splitting the "TTL helper" into multiple "TTL helpers", such that the time at which they are active do not overlap with that of the "DAC helper".
For a more concrete example, here's one where we had to split
# Start TTL sequence
# Translation DAC sequence
# Start post-translation TTL sequence
# Kick out DAC sequence
self.dac_kick is an "instantaneous" DAC set for all the channels that happens just once when called, but
self.dac_translate* are DAC sets that propagate for some time to generate the sine waveform to translate our ions.
We tend to write our code (and pseudo code) in a way that is highly influenced by our predecessor experiment that uses LabVIEW and Matlab. If people have suggestions on how we should be using ARTIQ, we would gladly welcome them!