artiq_coremgmt produces no output (log empty, config write hangs)

ip-10

Hello,
I am having an issue with artiq_coremgmt and I would appreciate some guidance. I can successfully communicate with the core device from my host PC:

I can access logs via the micro-USB serial console, so the core itself is alive
ping <IP> works reliably
artiq_session and previously written ARTIQ experiments run and communicate with the core without issues
I have previously used ARTIQ successfully with another core device, and I am now testing a second ARTIQ system.
The core firmware appears to be running normally

However, when I try to use artiq_coremgmt, I do not get any output at all. For example, when I run artiq_coremgmt -D <IP> log, it returns immediately with no output (just a newline).
I understand that the log may be empty if there are no errors, but the same behavior occurs with other commands: when I try to change the IP address: artiq_coremgmt -D <OLD_IP> config write -s ip <NEW_IP> the terminal appears to hang silently (no output, pressing Enter just produces a newline). I have also tried running these commands from a directory containing device_db.py, but the behavior is the same.

My main concern is not just viewing the log, but being able to reliably use artiq_coremgmt (especially for configuration tasks like changing the IP). Is this behavior expected when changing the IP (connection drop during config write), or could there be an issue with my setup or ARTIQ installation?
Any insight would be appreciated.
Thank you.

dly

Hi, the described behavior is not usual, and some issues like Artiq version mismatch between host and device could cause silent hang.
To debug, you could try these observation:

From micro-USB log, what is the firmware and gateware version (from the powering on message INFO(runtime): software/gateware ident <revision number>;<variant>)?
What is your host Artiq software version, and did you install it using nix profile install, or clone the full repository and flake, or other method?
From micro-USB log, when running artiq_coremgmt, is there any new output, like INFO(runtime::mgmt): new connection from <ip>, ERROR(runtime::mgmt): aborted: unknown packet 0x0, ...? If so could you share the log please.

Thank you for sharing the result!

ip-10

dly Thanks for the observations. At the moment I do not have physical access to the hardware, so the information I shared is from a previous saved micro-USB log:

The micro-USB log returns INFO(runtime): gateware ident: 9+009c322c;bremen2
The host has ARTIQ v8.9024+a1bf0e1 and it came pre-installed with the purchase
The last time I checked the micro-USB log, INFO(runtime::mgmt) did not even appeared, perhaps I closed the log too soon but I did not get the INFO(runtime::mgmt): received connection. However I do get INFO(runtime::comms): no idle kernel found, which is the last log I saved. But perhaps this is because the PC was not connected to the same network as the ARTIQ when I opened the log (but I can't remember). As soon as I have access to the hardware, I will run the test to rule out that error.

Is there anything else I can already verify on the host side (e.g. expected version alignment, known issues with silent hangs in coremgmt, or specific management-port/network assumptions) before I regain access to the hardware?
Once I have access again, I plan to re-run artiq_coremgmt with the micro-USB log open and check for runtime::mgmt messages as suggested.

dly

ip-10 Thank you for sharing the details.

There would be incompatibility when device firmware is v9.0+ and host installed ARTIQ is v8.0+. When I reproduced locally, I do found in this case artiq_coremgmt silently hangs (however, didn't find the immediate return phenomenon) .

For testing, you could get to [(https://git.m-labs.hk/M-Labs/artiq)], the master branch is the currently developing v9.0. You could try cloning the repository and entering the environment from that flake.nix. Alternatively, you can replace you local flake.nix with the remote one, and copy fix-rust-overlay-unpack.diff and artiq/ folder into the same directory, and reenter the environment.

If artiq_coremgmt command works with these method, you could consider using the cloned master branch for further experiments until the official release of ARTIQ v9.0. If you prefer to continue with v8.0+, you may contact AWFS to obtain v8.0+ firmware. Note that Urukuls with hardware rev1.6+ is not supported in ARTIQ 8.

If the issue still exist, please share the micro-USB log (with v9.0 host), thank you!

ip-10

dly actually, the Urukul of this second ARTIQ is v1.5.6 and I was not able to initialized it while running tests: now I understand why. I'll look into whether obtaining the v8.0 firmware or ARTIQ v9.0.

Thanks again for the help.