Running two instances of artiq_ctlmgr on a single computer

jroth

I am trying to get our third party instruments to work nicely with our ARTIQ installation. I will start by describing our current set up, then I'll describe the issue, and then my proposed workaround (which I have been unable to figure out so far).

Our artiq_master is running on our main computer (an Ubuntu machine with ARTIQ installed via a nix flake). This computer also runs an instance of artiq_ctlmgr that is responsible for core_moninj, core_analyzer, etc.. Our third party network device support packages (NDSPs) are running on a Windows computer running an instance of artiq_ctlmgr installed via

pip install git+https://github.com/m-labs/artiq-comtools.git
pip install git+https://github.com/m-labs/sipyco.git

This is not ideal as some of our third party devices record a lot of data, which can take a long time to transfer over to our main computer. It is also inconvenient. Because of this I would like to move several of our NDSPs over to our main computer.

The first solution I attempted was to run the NDSPs in the instance of artiq_ctlmgr already running on the main computer. However the NDSPs rely on several python packages (pyserial, pyvisa, nidaqmx, and vxi11, to name a few), and I have been having trouble getting these packages to work in our nix environment. Particularly pyvisa and nidaqmx depend on external libraries that nix cannot locate/import properly.

To get around this issue, I had the idea to run a second instance of artiq_ctlmgr on the main computer. The idea was to install artiq_ctlmgr via pip in a virtual environment alongside all of the needed packages. However this runs into an issue as you need to choose a host for each NDSP in the device_db.py file, and you can't have both instances of artiq_ctlmgr sharing the same host as there is no way to differentiate between the two.

Has anyone found a workaround for this kind of issue? I would be happy to try and get the needed python packages working with nix, but that seems like a less flexible solution.

sb10q

jroth Particularly pyvisa and nidaqmx depend on external libraries that nix cannot locate/import properly.

That's most likely solvable, what did you try and what was the error message?

jroth

Thanks @sb10q, here are the exact errors I'm seeing.

`pyvisa`

In a python virtual environment I am able to run pyvisa and connect to remote instruments, for example

>>> import pyvisa
>>> rm = pyvisa.ResourceManager()                
>>> rm.visalib
<IVIVisaLibrary('/usr/lib/x86_64-linux-gnu/libvisa.so.24.8.0')>             
>>> rm.list_resources()
('ASRL1::INSTR', 'ASRL2::INSTR', 'ASRL3::INSTR', 'ASRL4::INSTR', 'ASRL5::INSTR', 'TCPIP0::A-N8733A-3941R.local::inst0::INSTR', 'TCPIP0::A-33511B-01703.local::inst0::INSTR', 'TCPIP0::K-33511B-01728.local::inst0::INSTR', 'TCPIP0::K-33511B-02229.local::inst0::INSTR', 'TCPIP0::K-33511B-02356.local::inst0::INSTR')
>>> inst=rm.open_resource('TCPIP0::A-N8733A-3941R.local::inst0::INSTR')
>>> print(inst.query('*IDN?'))
Agilent Technologies,N8733A,US20L3941R,C.00.10.328,REV3.3K:B

In the nix flake attempting to initialize an instance of ResourceManager produces the following error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/highlevel.py", line 2992, in __new__
    visa_library = open_visa_library(visa_library)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/highlevel.py", line 2904, in open_visa_library
    return cls(argument)
           ^^^^^^^^^^^^^
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/highlevel.py", line 175, in __new__
    raise OSError("Could not open VISA library:\n" + "\n".join(errs))
OSError: Could not open VISA library:
Error while accessing /usr/lib/x86_64-linux-gnu/libvisa.so.24.8.0: libstdc++.so.6: cannot open shared object file: No such file or directory

If I pass the same IVIVisaLibrary path the error persists

Traceback (most recent call last):
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/ctwrapper/highlevel.py", line 162, in _init
    lib = Library(self.library_path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h723hb9m43lybmvfxkk6n7j4v664qy7b-python3-3.11.9/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libstdc++.so.6: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/highlevel.py", line 2992, in __new__
    visa_library = open_visa_library(visa_library)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/highlevel.py", line 2904, in open_visa_library
    return cls(argument)
           ^^^^^^^^^^^^^
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/highlevel.py", line 191, in __new__
    obj._init()
  File "/nix/store/q9g09n6ijiz25vgq6037awy9dn3sk6fy-python3-3.11.9-env/lib/python3.11/site-packages/pyvisa/ctwrapper/highlevel.py", line 164, in _init
    raise errors.LibraryError.from_exception(exc, self.library_path)
pyvisa.errors.LibraryError: Error while accessing /usr/lib/x86_64-linux-gnu/libvisa.so.24.8.0: libstdc++.so.6: cannot open shared object file: No such file or directory

Do I need to include the .so library in my nix flake?

`nidaqmx`

When I include nidaqmx in the nix flake I get the following error when running nix shell

error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:34:12:
           33|
           34|   strict = derivationStrict drvAttrs;
             |            ^
           35|

       … while evaluating derivation 'artiq-env'
         whose name attribute is located at /nix/store/5cypwijbjdxmr4i7ckpq1in7f09547fc-source/pkgs/stdenv/generic/make-derivation.nix:333:7

       … while evaluating attribute 'passAsFile' of derivation 'artiq-env'
         at /nix/store/5cypwijbjdxmr4i7ckpq1in7f09547fc-source/pkgs/build-support/trivial-builders/default.nix:69:9:
           68|         inherit buildCommand name;
           69|         passAsFile = [ "buildCommand" ]
             |         ^
           70|           ++ (derivationArgs.passAsFile or [ ]);

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: nidaqmx-0.5.7 not supported for interpreter python3.11

Surely this is just an error with how the package is defined in the nix package repository?

My `flake.nix` file

{
  inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git?ref=release-8";
  outputs = { self, extrapkg }:
    let
      pkgs = extrapkg.pkgs;
      artiq = extrapkg.packages.x86_64-linux;
    in {
      defaultPackage.x86_64-linux = pkgs.buildEnv {
        name = "artiq-env";
        paths = [
          # ========================================
          # EDIT BELOW
          # ========================================
          (pkgs.python3.withPackages(ps: [
            # List desired Python packages here.
            artiq.artiq
	    #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
            #artiq.flake8-artiq
            #artiq.dax
            #artiq.dax-applets

            # The NixOS package collection contains many other packages that you may find
            # interesting. Here are some examples:
            #ps.pandas
            #ps.numba
            #ps.matplotlib
            ps.pyqt6
	    ps.pyvisa
	    ps.pyserial
	    ps.vxi11
	    ps.nidaqmx	    
            # or if you need Qt (will recompile):
            #(ps.matplotlib.override { enableQt = true; })
            #ps.bokeh
            #ps.cirq
            #ps.qiskit
            # Note that NixOS also provides packages ps.numpy and ps.scipy, but it is
            # not necessary to explicitly add these, since they are dependencies of
            # ARTIQ and available with an ARTIQ install anyway.
          ]))
          artiq.openocd-bscanspi
	  #artiq.korad_ka3005p
          #artiq.novatech409b
          # List desired non-Python packages here
          # Other potentially interesting non-Python packages from the NixOS package collection:
          #pkgs.gtkwave
          #pkgs.spyder
          #pkgs.R
          #pkgs.julia
          # ========================================
          # EDIT ABOVE
          # ========================================
        ];
      };
    };
  nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
    extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
    extra-substituters = "https://nixbld.m-labs.hk";
  };
}

sb10q

jroth Do I need to include the .so library in my nix flake?

Yes, try to get Nix to install libstdc in the environment.

jroth error: nidaqmx-0.5.7 not supported for interpreter python3.11

Does nidaqmx actually work with Python 3.11?
If not you can use e.g. python39.withPackages(...) to install it.

Note that with or without Nix, you can also run the controllers through wrapper shell scripts that set up the environment (venv activate, ...) before exec'ing the actual controller.

jroth

The virtual environment approach seems like the cleanest option. However I am having trouble getting the controller manager to call a script that activates the virtual environment and starts the controller/driver.

I've set up a minimal example by doing the following:

Virtual environment creation

python3 -m venv driver_env
 source driver_env/bin/activate
 pip install git+https://github.com/m-labs/sipyco.git
 pip install git+https://github.com/m-labs/artiq-comtools.git
 pip install pyvisa
python
>>> import pyvisa
>>> rm = pyvisa.ResourceManager()

The resource manager is initialized successfully.

aqctl_visa_test.py

import argparse
import logging
from sipyco.pc_rpc import simple_server_loop
from sipyco import common_args
import pyvisa

logger = logging.getLogger("controller_visa_device")

class VISADevice:

    def ping(self):
        try:
            self.conn.query('*IDN?')
            return True
        except TimeoutError:
            return False

    def __init__(self, address):
        self.rm = pyvisa.ResourceManager()
        self.address = address
        self.conn = self.rm.open_resource(self.address)

    def close(self):
        self.conn.close()


def get_argparser():
    parser = argparse.ArgumentParser(description="NI VISA device")
    common_args.simple_network_args(parser, 3249)
    parser.add_argument("-d", "--device", default=None, help="NI VISA address of the device.")
    common_args.verbosity_args(parser)
    return parser

def main():
    args = get_argparser().parse_args()
    common_args.init_logger_from_args(args)
    logger.debug(f"Connecting to {args.device}")
    dev = VISADevice(args.device)
    try:
        logger.debug("Starting server")
        simple_server_loop({"dev": dev}, common_args.bind_address_from_args(args), args.port)
    finally:
        dev.close()

if __name__ == "__main__":
    main()

device_db.py entry

device_db['visadev'] = {
    "type": "controller",
    "host": '::1',
    "port": 40100,
    "command": "source run_driver.sh"
}

run_driver.sh

source driver_env/bin/activate
python aqctl_visa_test.py -p 40100 -d TCPIP0::K-33511B-02356.local::inst0::INSTR -v

I then start up our ARTIQ environment via

nix shell artiq8 --command artiq_session -c=-v

(I've registered the artiq 8 flake.nix file specified in the ARTIQ installation instructions with nix, no additional packages are included.) This produces the following error

ARTIQ master is now ready.
INFO:ctlmgr(control-system):artiq_comtools.ctlmgr:Starting controller visadev with command: source test.sh
WARNING:ctlmgr(control-system):artiq_comtools.ctlmgr:Controller visadev failed to start
WARNING:ctlmgr(control-system):artiq_comtools.ctlmgr:Restarting in 5.0 seconds
INFO:ctlmgr(control-system):artiq_comtools.ctlmgr:Starting controller visadev with command: source test.sh
WARNING:ctlmgr(control-system):artiq_comtools.ctlmgr:Controller visadev failed to start
WARNING:ctlmgr(control-system):artiq_comtools.ctlmgr:Restarting in 5.0 seconds

I expect I've just done something silly and am not calling the test.sh script properly? Unfortunately nothing I've tried seems to fix the issue.

sb10q

jroth "command": "source run_driver.sh"

That command is incorrect, don't use source here. Your script also appears to lack a shebang line.

jroth

Ah yes, thanks for pointing that out.

I've gotten everything working now. Copying my updated files in case others are running into the same issue.

Virtual environment creation

python3 -m venv driver_env
 source driver_env/bin/activate
 pip install git+https://github.com/m-labs/sipyco.git
 pip install git+https://github.com/m-labs/artiq-comtools.git
 pip install pyvisa
python
>>> import pyvisa
>>> rm = pyvisa.ResourceManager()

aqctl_visa_test.py

import argparse
import logging
from sipyco.pc_rpc import simple_server_loop
from sipyco import common_args
import pyvisa

logger = logging.getLogger("controller_visa_device")

class VISADevice:

    def ping(self):
        try:
            self.conn.query('*IDN?')
            return True
        except TimeoutError:
            return False

    def __init__(self, address):
        self.rm = pyvisa.ResourceManager()
        self.address = address
        self.conn = self.rm.open_resource(self.address)

    def close(self):
        self.conn.close()


def get_argparser():
    parser = argparse.ArgumentParser(description="NI VISA device")
    common_args.simple_network_args(parser, 3249)
    parser.add_argument("-d", "--device", default=None, help="NI VISA address of the device.")
    common_args.verbosity_args(parser)
    return parser

def main():
    args = get_argparser().parse_args()
    common_args.init_logger_from_args(args)
    logger.debug(f"Connecting to {args.device}")
    dev = VISADevice(args.device)
    try:
        logger.debug("Starting server")
        simple_server_loop({"dev": dev}, common_args.bind_address_from_args(args), args.port)
    finally:
        dev.close()

if __name__ == "__main__":
    main()

device_db.py entry

device_db['visadev'] = {
    "type": "controller",
    "host": '::1',
    "port": 40100,
    "command": "./run_driver"
}

run_driver

#!/usr/bin/env bash

source driver_env/bin/activate
python aqctl_visa_test.py -p 40100 -d TCPIP0::K-33511B-02356.local::inst0::INSTR -v
exit

Don't forget to make run_driver executable by running chmod +x run_driver

Running ARTIQ

nix shell artiq8 --command artiq_session -c=-v