Accessing a list inside a kernel takes >10x longer than writing to a list

tommtommbom

Hello ARTIQ community,
I am wondering what the fastest way is to access elements of a list in a kernel, ideally reaching <5us. I believe this should be possible because writing to a list takes <500ns. I was able to get this information by looking at the rtio_counter using artiq_coreanalyzer with some test code to investigate how long it takes to write/read from a list between ttl pulses:

class TTLdelaytest(EnvExperiment): 
  kernel_invariants = {"delay_time"}
 
def build(self):
    self.setattr_device("core")
    self.setattr_device("ttl6") `

def prepare(self):
    self.delay_time = 1 * us
    self.temp_data = [0] * 2
    self.list = np.array([5,6])*us

@kernel
def run(self):
    self.core.reset()
    self.ttl6.pulse(50*ns)
    #delay(self.list[0])
    self.temp_data[0] = 1
    self.ttl6.pulse(50*ns)`

The line self.temp_data[0] = 1 takes <500ns on average while delay(self.list[0]) takes >25us. This seems quite odd to me since I would assume a "get()" operation in machine code would be a lot faster than a "put()". To me, this likely points to something going on with the ARTIQ compiler not optimizing correctly with lists. Overall, I have a pre-allocated list I would like to sweep over the elements which is why I am using a list in the first place. If I can modify ARTIQ's source code to be able to do this, I would be more than happy to make it work, but I am wondering if this is possible to do.

If there is a faster way to access elements in a list, I would greatly appreciate any advice! I've played around with:

making the list in the kernel itself and then accessing it
trying different types of lists like np.int32 or float
setting the list to a variable in another kernel and then accessing it
setting the wanted element in the list to a variable and then using it
but they have all shown the 25us delay.

dpn

You may want to try accessing self.core.get_rtio_counter_mu() programmatically for more accurate timing.

There is no way accessing a list element takes 25 µs in general. delay() will have some overhead due to the floating point manipulations involved, but it will still be on the order of some relatively small number of clock cycles, not 25 µs.

500 ns for a store also seems way too long; you might want to examine the generated machine code to check what you are actually benchmarking. You can do this e.g. by using ARTIQ_DUMP_LLVM=foo artiq_run foo.py (or …UNOPT_LLVM, ASM, etc.).

tommtommbom

Thank you for the help! I didn't realize it could dump the machine code like that which is very helpful. Looking at the machine code, I found that it was the conversion that was taking a long time so switching to delay_mu() and converting the list to mu units apriori instead of delay() made it reach the expected _200ns.

You're right, I mis-interpreted the information the coreanalyzer was giving me. the 500ns store is also adding in the time it takes for the ttl pulse. In actuality it is a lot faster.

Thank you again for the help!

asc

To add a little more detail, calling delay() takes a very long time (at least on a Kasli 2.0) because the soft CPU has no built in method to convert a float into an int64, and instead it has to call a function to do this. In my benchmarking, float -> int64 conversion takes around 1500 ns.

However, if you go via a 32 bit int, then it's actually pretty fast because the conversion to int32 takes place in gateware. My tests found that np.int64(np.int32(n)) only takes ~ 100 ns. The caveat with this hack is that if the delay is longer than a 32 bit signed int can hold (around 2 s) then overflows will bite you. But if you're delaying for 2 s you probably don't care how long it takes.

Precalculating or working exclusively in mu is of course better, but if you need to use a floating point delay in a tight loop this can be a nice trick .

tommtommbom

Thank you, converting everything into mu units apriori solved this issue