Hi,
I have a problem using the dashboard override panel. Whenever I use it it causes the connection to the ARTIQ box to stop and I have to restart and re-test the sinara box. This is the error code I get when I press the override button on a random TTL channel (takes around 5s to show up):
WARNING:controller(core_log):firmware.runtime.mgmt:connection terminated: NetworkError(Truncated)
WARNING:controller(core_log):firmware.runtime.mgmt:connection terminated: NetworkError(Truncated)
C:\Users\E9-ARTIQ\AppData\Local\m-labs\artiq\7\artiq_dashboard_..1_3250.pyon
INFO:dashboard:root:ARTIQ dashboard version: 7.8123.3038639
INFO:dashboard:root:ARTIQ dashboard connected to moninj_proxy (::1)
INFO:dashboard:artiq.dashboard.moninj:ARTIQ dashboard connected to moninj (::1)
INFO:applet(untitled 3):print:External WM_DESTROY received for QWidgetWindow(0x1e424432220, name="ProgressWidgetClassWindow") , parent: QWindow(0x0) , transient parent: QWindow(0x0)
ERROR:controller(core_log):artiq.frontend.aqctl_corelog:Logging connection terminating with exception
Traceback (most recent call last):
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\artiq\frontend\aqctl_corelog.py", line 63, in get_logs
length, = struct.unpack(endian + "l", await reader.readexactly(4))
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 708, in readexactly
await self._wait_for_data('readexactly')
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 502, in _wait_for_data
await self._waiter
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\proactor_events.py", line 282, in _loop_reading
length = fut.result()
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 817, in _poll
value = callback(transferred, key, ov)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 485, in finish_recv
return ov.getresult()
OSError: [WinError 121] The semaphore timeout period has expired
ERROR:controller(core_log):artiq.frontend.aqctl_corelog:Logging connection terminating with exception
Traceback (most recent call last):
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\artiq\frontend\aqctl_corelog.py", line 63, in get_logs
length, = struct.unpack(endian + "l", await reader.readexactly(4))
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 708, in readexactly
await self._wait_for_data('readexactly')
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 502, in _wait_for_data
await self._waiter
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\proactor_events.py", line 282, in _loop_reading
length = fut.result()
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 817, in _poll
value = callback(transferred, key, ov)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 485, in finish_recv
return ov.getresult()
OSError: [WinError 121] The semaphore timeout period has expired
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Controller core_log exited
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Restarting in 5.0 seconds
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Controller core_log exited
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Restarting in 5.0 seconds
ERROR:controller(core_moninj):artiq.coredevice.comm_moninj:Moninj connection terminating with exception
Traceback (most recent call last):
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\artiq\coredevice\comm_moninj.py", line 84, in _receive_cr
ty = await self._reader.read(1)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 669, in read
await self._wait_for_data('read')
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 502, in _wait_for_data
await self._waiter
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\proactor_events.py", line 282, in _loop_reading
length = fut.result()
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 817, in _poll
value = callback(transferred, key, ov)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 485, in finish_recv
return ov.getresult()
OSError: [WinError 121] The semaphore timeout period has expired
ERROR:controller(core_moninj):artiq.coredevice.comm_moninj:Moninj connection terminating with exception
Traceback (most recent call last):
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\artiq\coredevice\comm_moninj.py", line 84, in _receive_cr
ty = await self._reader.read(1)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 669, in read
await self._wait_for_data('read')
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 502, in _wait_for_data
await self._waiter
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\proactor_events.py", line 282, in _loop_reading
length = fut.result()
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 817, in _poll
value = callback(transferred, key, ov)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 485, in finish_recv
return ov.getresult()
OSError: [WinError 121] The semaphore timeout period has expired
ERROR:dashboard:artiq.dashboard.moninj:lost connection to moninj
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Controller core_moninj exited
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Restarting in 5.0 seconds
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Controller core_moninj exited
WARNING:ctlmgr(DESKTOP-M78QCSB):artiq_comtools.ctlmgr:Restarting in 5.0 seconds
ERROR:dashboard:artiq.dashboard.moninj:failed to connect to moninj. Is aqctl_moninj_proxy running?
Traceback (most recent call last):
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\artiq\dashboard\moninj.py", line 693, in mi_connector
await new_mi_connection.connect(self.mi_addr, self.mi_port)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\artiq\coredevice\comm_moninj.py", line 32, in connect
self._reader, self._writer = await async_open_connection(
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\sipyco\keepalive.py", line 80, in async_open_connection
reader, writer = await asyncio.open_connection(host, port, *args, **kwargs)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\streams.py", line 47, in open_connection
transport, _ = await loop.create_connection(
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\base_events.py", line 1064, in create_connection
raise exceptions[0]
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\base_events.py", line 1049, in create_connection
sock = await self._connect_sock(
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\base_events.py", line 960, in _connect_sock
await self.sock_connect(sock, address)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\proactor_events.py", line 705, in sock_connect
return await self._proactor.connect(sock, address)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\site-packages\qasync\_windows.py", line 43, in _process_events
value = callback(transferred, key, ov)
File "C:\Users\E9-ARTIQ\anaconda3\envs\artiq\lib\asyncio\windows_events.py", line 604, in finish_connect
ov.getresult()
ConnectionRefusedError: [WinError 1225] The remote computer refused the network connection

I'm using the artiq_client to open the dashboard, so I think I can't access the artiq_log file. When I try it only shows entries from after disconnecting the artiq_client. Not sure if there's a way around that and/or if the log file would be helpful.

Please help with any advice on how to debug/fix this problem. This feature is essential for effective debugging of my experiment.
Thanks,
Malte

7 days later

Would appreciate any help on this. Unclear to me how to debug further

    10 days later

    multi131
    Hey Malte,
    Is the artiq controller manager being run?
    How is aqctl_moninj_proxy being run?

        claytonho
        Hey Clayton,
        I'm running the dashboard via the artiq_session command which should start the artiq_ctlmgr (which I think starts the moninj?). The moninj is still a moderate mystery to me, so if there are any troubleshooting steps I can take to try to keep it running when modifying the override panel then please let me know.

            4 days later

            multi131

            The aqctl_moninj_proxy is basically the controller for the devices available on the ARTIQ dashboard (i.e. your TTLs in question), and has to be run separately from artiq_session.

            You can run it by entering your ARTIQ environment and running "aqctl_moninj_proxy <core ip address>."

                10 days later

                claytonho has to be run separately from artiq_session

                With a correct device_db, artiq_session will run it through ctlmgr.