RPC Error: type mismatch: cannot serialize None as <type>

alessandro

In order to get some moninj functionality on an AD9910 board I have written some code using the scheme Dan Slichter suggests here: https://github.com/m-labs/artiq/issues/1679#issuecomment-845690994

I wrote a GUI which updates a dataset and a polling kernel which examines that dataset for changes every 1/10th of a second. This works most of the time and gives manual control over our AD9910 boards, TTLs and DAC channels.

However, sometimes, seemingly randomly, when I try to change the dataset with the GUI I get this error:

INFO:worker(335,manual_control.py):print:artiq.coredevice.comm_kernel.RPCReturnValueError: type mismatch: cannot serialize None as bool (<function Scheduler.check_pause at 0x7f27f0966488> has returned None)

The check_pause() function is in the condition of the while which loops the polling kernel

while not self.scheduler.check_pause(): <rest of polling kernel code>

I have also seen this error (type mismatch: cannot serialize None as <type>) occur with other RPC calls (the ones that read the datasets into the polling kernel), but after some modifications to the core device code they mysteriously disappeared.

alessandro

I've reduced the severity of this error by just retrying the RPC whenever it fails but that's pretty dumb.

sb10q

Can you find out under which conditions self.scheduler.check_pause() returns None?
Does it ever do so when called outside a kernel?

alessandro

It never happens when called outside of the kernel. It seems like this part of the problem has gone away, but If I see it again I'll let you know what was going on.

Now I get this error randomly when writing to a dataset.

INFO:worker(492,manual_control.py):print:Traceback (most recent call last):
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/artiq/hci-artiq/repository/manual_control.py", line 296, in write
INFO:worker(492,manual_control.py):print:    self.set_dataset("dds_conf", dds_conf, broadcast=True, archive=False, persist=True)
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/language/environment.py", line 328, in set_dataset
INFO:worker(492,manual_control.py):print:    self.__dataset_mgr.set(key, value, broadcast, persist, archive)
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker_db.py", line 128, in set
INFO:worker(492,manual_control.py):print:    self._broadcaster[key] = persist, value
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/sipyco/sync_struct.py", line 245, in __setitem__
INFO:worker(492,manual_control.py):print:    "value": value})
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 51, in parent_action
INFO:worker(492,manual_control.py):print:    reply = get_object()
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 39, in get_object
INFO:worker(492,manual_control.py):print:    return pyon.decode(line)
INFO:worker(492,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/sipyco/pyon.py", line 210, in decode
INFO:worker(492,manual_control.py):print:    return eval(s, _eval_dict, {})
INFO:worker(492,manual_control.py):print:  File "<string>", line 1
INFO:worker(492,manual_control.py):print:    .0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "status": "ok"}
INFO:worker(492,manual_control.py):print:                                                                            ^
INFO:worker(492,manual_control.py):print:SyntaxError: invalid syntax
ERROR:master:artiq.master.scheduler:got worker exception in run stage, deleting RID 492
ERROR:master:artiq.master.worker:worker exception details
Traceback (most recent call last):
  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/scheduler.py", line 273, in _do
    completed = await run.run()
  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/scheduler.py", line 34, in worker_method
    return await m(*args, **kwargs)
  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker.py", line 278, in run
    completed = await self._worker_action({"action": "run"})
  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker.py", line 252, in _worker_action
    completed = await self._handle_worker_requests()
  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker.py", line 208, in _handle_worker_requests
    obj = await self._recv(self.watchdog_time())
  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/master/worker.py", line 195, in _recv
    format(self.rid))
artiq.master.worker.WorkerError: Worker ended while attempting to receive data (RID 492)

The dataset is just 32 0.0s.

alessandro

The code around line 492

               for i in range(len(self.dac_keys)):
492>>>             dac = self.dacs[i]
                   for j in range(32):
                       self.core.break_realtime()
                       # If any changes have been made to the dac_conf
                       # load the new conf and break the loop to go to the next dac
                       if dac_conf[i * 32 + j] != dac_conf_old[i * 32 + j]:
                           # at_mu(now)
                           dac.load()  # Switch to using dac.set_dac?
                           # TODO: confirm that dac.load() causes dacs to change simultaneously with
                           #  TTL and dds changes. Use Oscope triggering on a ttl line?
                           #  if not see if set_dac does.
                           break  # Move on to next dac card.

Where self.dacs is a list build from get_device(device_key) on all the dacs in our crate.

alessandro

The condition that appears to cause the Read Failure in check pause is having the host write to a data_set, but because we have this polling kernel that's periodically looking at the dataset maybe that's causing an issue.

Here's our experiment's source code:
https://pastebin.com/ZP0aZS2p

I can give you the QT interface stuff too if somebody actually wants to run this.

sb10q

alessandro I can give you the QT interface stuff too if somebody actually wants to run this.

It would be much better if you could come up with minimal example code that reproduces the problems.

alessandro

I realize this but I have been unable to do that so far. The error occurs so infrequently (but enough to be annoying) that any testing is impractical outside of production use.

sb10q

Do you want SSH access to a testing system at M-Labs?

alessandro

That sounds great. How do I apply for that?

sb10q

alessandro That sounds great. How do I apply for that?

Send your SSH public key to helpdesk@

alessandro

Well I tried my previous strategy of catching the SyntaxError exception and retrying the set_dataset over again. Now I get a truly inscrutable error message:

INFO:worker(580,manual_control.py):print:Exception in thread Thread-1:
INFO:worker(580,manual_control.py):print:Traceback (most recent call last):
INFO:worker(580,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/threading.py", line 914, in _bootstrap_inner
INFO:worker(580,manual_control.py):print:    self.run()
INFO:worker(580,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/threading.py", line 862, in run
INFO:worker(580,manual_control.py):print:    self._target(*self._args, **self._kwargs)
INFO:worker(580,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/language/core.py", line 54, in run_on_core
INFO:worker(580,manual_control.py):print:    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
INFO:worker(580,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/coredevice/core.py", line 137, in run
INFO:worker(580,manual_control.py):print:    self.comm.serve(embedding_map, symbolizer, demangler)
INFO:worker(580,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/coredevice/comm_kernel.py", line 507, in serve
INFO:worker(580,manual_control.py):print:    self._serve_exception(embedding_map, symbolizer, demangler)
INFO:worker(580,manual_control.py):print:  File "/home/ioncontroller/anaconda3/envs/artiq/lib/python3.5/site-packages/artiq/coredevice/comm_kernel.py", line 498, in _serve_exception
INFO:worker(580,manual_control.py):print:    raise python_exn
INFO:worker(580,manual_control.py):print:  File "<string>", line None
INFO:worker(580,manual_control.py):print:SyntaxError: invalid syntax (<string>, line 1)
INFO:worker(580,manual_control.py):print:

Note that the none of our code is even in this traceback...

The line SyntaxError: invalid syntax (<string>, line 1) can also be caused in python by defining a function like def func("abc") as can be seen here https://stackoverflow.com/questions/34970049/python-invalid-syntax-on-a-string . But how can this be related?

sb10q

alessandro INFO:worker(580,manual_control.py):print:SyntaxError: invalid syntax (<string>, line 1)

Could be a problem with PYON. Try modifying sipyco so it raises a different exception when the input fails to deserialize.

alessandro

I discovered how to handle this exception.

I added print(traceback) to the end of CommKernel._serve_exception. This showed that the SyntaxError exception was being raised by res = self.scheduler.check_pause(). Wrapping this in a try except block inside a for loop lets me retry this RPC until it succeeds.