• General
  • Help on Nix Manager - package installs

Hi,

Thanks for the help and headsups. To be clear on the expected behavior. After running nix develop I should have access to spyder and miniconf? I appear to only have access after entering a shell.

`
grpartiq@3085:~/ARTIQ/Artiq_Workplace$ nix develop
I have no name!@3085:~/ARTIQ/Artiq_Workplace$ spyder
bash: spyder: command not found
I have no name!@3085:~/ARTIQ/Artiq_Workplace$ nix shell
bash: shopt: progcomp: invalid shell option name
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: shopt: hostcomplete: invalid shell option name
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
[][]I have no name!@3085[]:[]~/ARTIQ/Artiq_Workplace[]$ spyder
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
Gtk-Message: 18:08:27.297: Failed to load module "canberra-gtk-module"
Gtk-Message: 18:08:27.297: Failed to load module "canberra-gtk-module"
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled

`

    pyquest Yes, nix develop should give you access to both spyder and miniconf.

    Did you change these lines as follows?

    -      packages.x86_64-linux.default = pkgs.buildEnv {
    +      devShells.x86_64-linux.default = pkgs.mkShell {
             name = "artiq-env";
    -        paths = [
    +        packages = [

    Could you share the output of nix flake show?

      fsagbuya

      Hi fsagbuya,

      Thank you for the help. I missed that I needed to change path to packages. As a general question, as I am new to NIX.

      1.) What is the practical difference between mkShell and mkEnv
      2.) Where do most folks create their nix flakes? I find myself jumping around alot and its making me wonder if this is the workflow intended for artiq. So for instance I'd do something like

      `
      ~ cd/ARTIQ/Nix_Working_Dir Nix Develop --------- startup the artiq environment/shell
      gnome-terminal --tab --------- add new tab (2) while staying in the shell
      ~ cd/ARTIQ/artiq-master ---------go to new tab to dir for artiq master
      gnome-terminal --tab ---------add new tab (3) while staying in the shell
      gnome-terminal --tab ---------add new tab (4) while staying in the shell
      artiq_master ---------startup artiq master in tab 2
      artiq_ctlmgr --------- startup artiq_ctlmgr in tab 3
      artiq_dashboard ---------startup artiq_dashboard in tab 4

      gnome-terminal --tab --------- add new tab (5) while staying in the shell
      ~ cd/ARTIQ/stabilizer --------- go do stabilizer stuff

      `

        1.
        buildEnv creates a directory with symlinks to the packages you include in the paths attribute. When you use nix shell with it, you're basically entering that directory with everything set up.

        mkShell creates a shell script that sets up the environment, which runs when you use nix develop.

        You won’t notice much difference between nix shell and nix develop while using them. One feature of buildEnv is that you can install it more permanently with nix profile install from the flake root.

        2.
        Preferred way is to create Nix flakes in the root directory where you want to run your ARTIQ environment. You can also create a flake per experiment if you want different packages available for each one.

        As for the workflow, just remember to always run nix develop or nix shell on your ARTIQ flake before executing any ARTIQ commands.

          fsagbuya

          2.) I am doing that but presumably its not recommended to do that in artiq-master but we still run artiq master and the repository etc is still in the artiq-master install directory? If artiq-master is run from a terminal that is not in the nix develop/shell how is it exposed to the packages?

          For example:

          running artiq_ctlmgr in the ubuntu terminal

          vs running artiq_ctlmgr in the nix-develop shell/terminal

          3.) With respect to the installs. I am still a bit lost. While you helped me get it working (thank you). I am not sure that my understanding has increased. Here is another attempt (pypylon for basler cameras) at installing a github repository that is not in nix pkgs.

          `

          {
          inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git";
          inputs.miniconf.url = "github:quartiq/miniconf";
          inputs.pypylon = {url = github.com/basler/pypylon ;flake = false; };
          outputs = { self, extrapkg, miniconf,pypylon }:
          
          let
            pkgs = extrapkg.pkgs;
            artiq = extrapkg.packages.x86_64-linux;
            pylon = pkgs.python3Packages.buildPythonPackage rec {
          	name = "pypylon";
          	src = pypylon.url;
          	format = "pyproject";
          	nativeBuildInputs = [ pkgs.python3Packages.setuptools pkgs.python3Packages.swig pkgs.python3Packages.wheel pkgs.python3Packages.lief pkgs.python3Packages.darwin];
          	};
          
          in {
             	
            devShells.x86_64-linux.default = pkgs.mkShell {
              name = "artiq-env";
              packages = [
                # ========================================
                # EDIT BELOW
                # ========================================
                (pkgs.python3.withPackages(ps: [
                  # List desired Python packages here.
                  artiq.artiq
                  #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
                  artiq.flake8-artiq
                  artiq.dax
                  artiq.dax-applets
          
                  # The NixOS package collection contains many other packages that you may find
                  # interesting. Here are some examples:
                  ps.pandas
                  ps.numpy
                  ps.scipy
                  ps.spyder
                  pypylon
          
                  #ps.numba
                  ps.matplotlib
                  ps.jupyterlab
                  # or if you need Qt (will recompile):
                  #(ps.matplotlib.override { enableQt = true; })
                  ps.bokeh
                  ps.cirq
                  #ps.qiskit
                ]))
                
                miniconf.packages.x86_64-linux.miniconf-mqtt-py
                       
          
                
                #artiq.korad_ka3005p
                #artiq.novatech409b
                # List desired non-Python packages here
                artiq.openocd-bscanspi  # needed if and only if flashing boards
                # Other potentially interesting non-Python packages from the NixOS package collection:
                #pkgs.gtkwave
                #pkgs.spyder
          
                #pkgs.R
                #pkgs.julia
                # ========================================
                # EDIT ABOVE
                # ========================================
              ];
            };
          };
           nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
          extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
          extra-substituters = "https://nixbld.m-labs.hk";
          };
          }

          `

            pyquest

            Still not working so posting another flake file in the hope that someone can help me figure this out. I am following the discussion here.
            https://discourse.nixos.org/t/fetch-from-github-dependencies-conky/62245/2

            `

            {
             inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git";
             inputs.miniconf.url = "github:quartiq/miniconf";
             inputs.pypylon = {url = github.com/basler/pypylon ;flake = false; };
              
              
              
            
            outputs = { self, extrapkg, miniconf,pypylon }:
            
            let
              pkgs = extrapkg.pkgs;
              artiq = extrapkg.packages.x86_64-linux;
              pypylon = pkgs.python3Packages.buildPythonPackage rec {
            	pname = "pypylon";
            	format = "pyproject";
            	version = "4.1.0";
            	doCheck = false; # check requires pip
            	src =  pkgs.fetchFromGitHub {owner = "basler";
            	    	 	 repo = "pypylon";
            	    	 	 rev = "v${version}";};
            	};	
            
            
            in {
               	
              devShells.x86_64-linux.default = pkgs.mkShell {
                name = "artiq-env";
                packages = [
                  # ========================================
                  # EDIT BELOW
                  # ========================================
                  (pkgs.python3.withPackages(ps: [
                    # List desired Python packages here.
                    artiq.artiq
                    #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
                    artiq.flake8-artiq
                    artiq.dax
                    artiq.dax-applets
            
                    ps.pandas
                    ps.numpy
                    ps.scipy
                    ps.spyder
                    pypylon
            
                    #ps.numba
                    ps.matplotlib
                    ps.jupyterlab
                    # or if you need Qt (will recompile):
                    #(ps.matplotlib.override { enableQt = true; })
                    ps.bokeh
                    ps.cirq
                    #ps.qiskit
                  ]))
                  
                  miniconf.packages.x86_64-linux.miniconf-mqtt-py
            
            
                  
                  #artiq.korad_ka3005p
                  #artiq.novatech409b
                  # List desired non-Python packages here
                  artiq.openocd-bscanspi  # needed if and only if flashing boards
                  # Other potentially interesting non-Python packages from the NixOS package collection:
                  #pkgs.gtkwave
                  #pkgs.spyder
            
                  #pkgs.R
                  #pkgs.julia
                  # ========================================
                  # EDIT ABOVE
                  # ========================================
                ];
                };
                };
               nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
              extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
              extra-substituters = "https://nixbld.m-labs.hk";
               };
                }

            `

            This spits out:

            `
            grpartiq@hrlit3085:~/ARTIQ/Artiq_Workplace$ nix develop
            warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
            error: builder for '/nix/store/fllgkd5xbafpq1zpiq6sjybcrx7r2z1q-source.drv' failed with exit code 1;
            last 25 log lines:
            > establish a secure connection to it. To learn more about this situation and
            > how to fix it, please visit the webpage mentioned above.
            >
            > trying https://github.com/basler/pypylon/archive/v4.1.0.tar.gz
            > % Total % Received % Xferd Average Speed Time Time Time Current
            > Dload Upload Total Spent Left Speed
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > curl: (22) The requested URL returned error: 404
            > Warning: Problem (retrying all errors). Will retry in 1 seconds. 3 retries
            > Warning: left.
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > curl: (22) The requested URL returned error: 404
            > Warning: Problem (retrying all errors). Will retry in 2 seconds. 2 retries
            > Warning: left.
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > curl: (22) The requested URL returned error: 404
            > Warning: Problem (retrying all errors). Will retry in 4 seconds. 1 retries
            > Warning: left.
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
            > curl: (22) The requested URL returned error: 404
            > error: cannot download source from any mirror
            For full logs, run:
            nix log /nix/store/fllgkd5xbafpq1zpiq6sjybcrx7r2z1q-source.drv
            error: 1 dependencies of derivation '/nix/store/l0f6z2mrkprynjx7ffd3yljvs9pmk551-python3.12-pypylon-4.1.0.drv' failed to build
            error: 1 dependencies of derivation '/nix/store/f2a03rd612vgzvc5rbd5dfif67lszsd2-python3-3.12.9-env.drv' failed to build
            error: 1 dependencies of derivation '/nix/store/n91jwggh2wcrq6265p8qyqchc6n8np6i-artiq-env-env.drv' failed to build

            `

              pyquest

              You can remove the pypylon input and instead define the package directly like this:

                    pypylon = pkgs.python3Packages.buildPythonPackage rec {
                      pname = "pypylon";
                      version = "4.1.0";
                      src = pkgs.fetchFromGitHub {
                        owner = "basler";
                        repo = "pypylon";
                        rev = "${version}";
                        hash = "sha256-qi74kq9l8OvmPZQ202+evnTChm2hY4PJkgzJAmv+/vY=";
                      };
                      nativeBuildInputs = [
                        pkgs.swig
                      ];
                      doCheck = false;
                    };

              However, this package seems to require pylon SDK to be installed and expects it at /opt/pylon, as shown by the error:
              FileNotFoundError: [Errno 2] No such file or directory: '/opt/pylon/bin/pylon-config'

                fsagbuya

                So that is the weird thing. I have pylon installed at /opt/pylon/bin and the directory contains a pylon-config.

                However, I still can't get the nix package manager to handle/find the file.

                `
                warning: updating lock file '"/home/grpartiq/ARTIQ/Artiq_Workplace/flake.lock"':
                • Removed input 'pypylon'
                error: builder for '/nix/store/f2bhkdwfn4vsdh0dzpm8c4zlx7z7p1li-python3.12-pypylon-4.1.0.drv' failed with exit code 1;
                last 25 log lines:
                > File "/build/source/nix_run_setup", line 8, in <module>
                > exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))
                > File "setup.py", line 1093, in <module>
                > bs = BuildSupport.make()
                > ^^^^^^^^^^^^^^^^^^^
                > File "setup.py", line 372, in make
                > return BuildSupportLinux()
                > ^^^^^^^^^^^^^^^^^^^
                > File "setup.py", line 820, in init
                > config_cflags = self.call_pylon_config("--cflags")
                > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                > File "setup.py", line 908, in call_pylon_config
                > res = subprocess.check_output(params, universal_newlines=True)
                > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                > File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 468, in check_output
                > return run(popenargs, stdout=PIPE, timeout=timeout, check=True,
                > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                > File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 550, in run
                > with Popen(
                popenargs, **kwargs) as process:
                > ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                > File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 1028, in init
                > self._execute_child(args, executable, preexec_fn, close_fds,
                > File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 1963, in _execute_child
                > raise child_exception_type(errno_num, err_msg, err_filename)
                > FileNotFoundError: [Errno 2] No such file or directory: '/opt/pylon/bin/pylon-config'
                For full logs, run:
                nix log /nix/store/f2bhkdwfn4vsdh0dzpm8c4zlx7z7p1li-python3.12-pypylon-4.1.0.drv
                error: 1 dependencies of derivation '/nix/store/9z7z9g20sfnlkw3n5520rj4qkbgg7b1d-python3-3.12.9-env.drv' failed to build
                error: 1 dependencies of derivation '/nix/store/dcrfmm96pdhvw2x2v67cmpx47jjspp6m-artiq-env-env.drv' failed to build

                `

                  pyquest
                  Hi, I managed to build the pypylon package with the following changes:

                  • packaged pylonSdk in nix and used it as buildInputs for pypylon
                  • updated the pypylon version due to swig issues
                  • added /opt to sandbox paths so it can access /opt/pypylon/bin
                  --- a/flake.nix
                  +++ b/flake.nix
                  @@ -7,15 +7,29 @@
                         pkgs = extrapkg.pkgs;
                         artiq = extrapkg.packages.x86_64-linux;
                   
                  +      pylon-sdk = pkgs.stdenv.mkDerivation {
                  +        name = "pylon-sdk";
                  +        version = "8.1.0";
                  +        dontUnpack = true;
                  +        installPhase = ''
                  +          mkdir -p $out/include $out/lib
                  +          cp -r /opt/pylon/include/* $out/include/
                  +          cp -r /opt/pylon/lib/* $out/lib/
                  +        '';
                  +      };
                  +
                         pypylon = pkgs.python3Packages.buildPythonPackage rec {
                           pname = "pypylon";
                  -        version = "4.1.0";
                  +        version = "4.2.0rc2";
                           src = pkgs.fetchFromGitHub {
                             owner = "basler";
                             repo = "pypylon";
                             rev = "${version}";
                  -          hash = "sha256-qi74kq9l8OvmPZQ202+evnTChm2hY4PJkgzJAmv+/vY=";
                  +          hash = "sha256-juzQ3Mluf+YskKM+qha8TH2ALgSjVMEKVB0BdVDKVew=";
                           };
                  +        buildInputs = [
                  +          pylon-sdk
                  +        ];
                           nativeBuildInputs = [
                             pkgs.swig
                           ];
                  @@ -75,5 +89,6 @@
                     nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
                       extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
                       extra-substituters = "https://nixbld.m-labs.hk";
                  +    extra-sandbox-paths = "/opt";
                     };
                   }
                    9 days later

                    pyquest

                    Following up on this question. I really think documentation needs to be written for correct usage of nix environment with artiq.

                    I am finding that I am getting RTIO overflow errors but am unable to use event spreading from within the nix environment.

                    However if I run the command from not-within the nix develop environment I can.

                    I have no way of knowing if this command is going to the microcontroller properly. I run the artiq master from a nix develop shell so is the microcontroller getting this command? All artiq master , com , dashboard were closed for running these commands.

                    You're probably mixing up ARTIQ versions, hence the error. You need to use the same major version for the firmware and the software, regardless how you install it.
                    artiq_coremgmt works independently of the other software components.

                      sb10q

                      How do I fix / install/ setup the nix manager appropriately? I have followed the instructions to the best of my know know. I am currently overwhelmed with the number of directories artiq/nix requires.

                      Currently running like this

                      `
                      ~ cd/ARTIQ/Nix_Working_Dir Nix Develop --------- startup the artiq environment/shell
                      gnome-terminal --tab --------- add new tab (2) while staying in the shell
                      ~ cd/ARTIQ/artiq-master ---------go to new tab to dir for artiq master
                      gnome-terminal --tab ---------add new tab (3) while staying in the shell
                      gnome-terminal --tab ---------add new tab (4) while staying in the shell
                      artiq_master ---------startup artiq master in tab 2
                      artiq_ctlmgr --------- startup artiq_ctlmgr in tab 3
                      artiq_dashboard ---------startup artiq_dashboard in tab 4

                      gnome-terminal --tab --------- add new tab (5) while staying in the shell
                      ~ cd/ARTIQ/stabilizer --------- go do stabilizer stuff

                      `

                      Other issues include - not sure where the dax files are being installed or why the artiq master is not seeing them? I think they should be installed per the flake.

                      Similiarly, modifying the flake to include py.pyvisa is very confusing. For example:

                      `
                      import pyvisa

                      a = pyvisa.ResourceManager()

                      ValueError Traceback (most recent call last)
                      Cell In[11], line 1
                      ----> 1 a = pyvisa.ResourceManager()

                      File /nix/store/g2nglci3s4yyxvfyw08al5ivwzdy2mb8-python3-3.12.9-env/lib/python3.12/site-packages/pyvisa/highlevel.py:2992, in ResourceManager.new(cls, visa_library)
                      2982 """Create a new resource manager tied to the specified VISA library.
                      2983
                      2984 Parameters
                      (...)
                      2989
                      2990 """
                      2991 if not isinstance(visa_library, VisaLibraryBase):
                      -> 2992 visa_library = open_visa_library(visa_library)
                      2994 if visa_library.resource_manager is not None:
                      2995 obj = visa_library.resource_manager

                      File /nix/store/g2nglci3s4yyxvfyw08al5ivwzdy2mb8-python3-3.12.9-env/lib/python3.12/site-packages/pyvisa/highlevel.py:2899, in open_visa_library(specification)
                      2897 wrapper = "ivi"
                      2898 else:
                      -> 2899 wrapper = _get_default_wrapper()
                      2901 cls = get_wrapper_class(wrapper)
                      2903 try:

                      File /nix/store/g2nglci3s4yyxvfyw08al5ivwzdy2mb8-python3-3.12.9-env/lib/python3.12/site-packages/pyvisa/highlevel.py:2858, in _get_default_wrapper()
                      2856 except ValueError:
                      2857 logger.debug("Did not find pyvisa-py package")
                      -> 2858 raise ValueError(
                      2859 "Could not locate a VISA implementation. Install either the IVI binary or pyvisa-py."
                      2860 )

                      ValueError: Could not locate a VISA implementation. Install either the IVI binary or pyvisa-py.

                      `
                      The error is not on the import it is on calling the resource manager.

                      fsagbuya

                      This compiled for me with nix develop. However it doesn't work in a way that I understand.

                        pyquest Is Jupyter running inside the same Nix environment? If not, you should run it from there by adding ps.jupyter to the flake, running nix develop, and then executing the command jupyter notebook.

                          fsagbuya

                          It was running from within the nix develop environment. The notebook was launched with jupyter lab from a terminal after nix develop with our flake and with ps.jupyterlab in the flake.