artiq_coremgmt log hangs if we set the log level to lower than INFO. For example,

artiq_coremgmt log clear
artiq_coremgmt log set_level DEBUG
artiq_coremgmt log  # log works at this point.
# run a simple experiment
artiq_coremgmt log  # the terminal hangs forever

The simple experiment we used to test the above code is

from artiq.experiment import *


class Test(EnvExperiment):
    def build(self):
        self.setattr_device("core")

    @kernel
    def run(self):
        self.core.reset()

If we set the log level to INFO, both artiq_coremgmt log commands finish as intended. The second command produces the following outputs:

[   614.377945s]  INFO(runtime::mgmt): new connection from 192.168.1.83:53149
[   614.385615s]  INFO(runtime::mgmt): changing log level to INFO
[   618.761149s]  INFO(runtime::mgmt): new connection from 192.168.1.83:53150
[   626.513949s]  INFO(runtime::session): new connection from 192.168.1.83:53154
[   626.559975s]  INFO(runtime::kern_hwreq): resetting RTIO
[   626.595632s]  INFO(runtime::session): no connection, starting idle kernel
[   626.601490s]  INFO(runtime::session): no idle kernel found
[   629.808044s]  INFO(runtime::mgmt): new connection from 192.168.1.83:53155

If we don't run the above experiment, both artiq_coremgmt log commands finish as well.

This is tested on the ucsb2 variant (Kasli 2.0), with Windows 10, artiq latest master (3f812c4c), and the same version for the firmware and gateware.

@sb10q do you have insights on this problem? Any suggestion on how to debug the problem would be helpful for us.

Check the serial console output when the crash occurs.

@sb10q thank you for the suggestion! We tried that and the uart log works properly.

The issue was that the maximum transmission unit (MTU) was set incorrectly on the network interface. We are using an usb to ethernet adapter that could be the problem. After setting the MTU to 1500, we can read the log with no problem.

What was the MTU before?

I don't remember the exact value and it was somewhere 8000/9000.

Ah, jumbo frames. They should be relatively straightforward to support in MiSoC though.

    sb10q They should be relatively straightforward to support in MiSoC though.

    That is good to know! I think supporting MTU of 1500 should work for most of the users. This adapter might had a driver / factory config issue. Other same-model adapters on other computers give MTU of 1500.