Help on Nix Manager - package installs

pyquest

Hi,

Thanks for the help and headsups. To be clear on the expected behavior. After running nix develop I should have access to spyder and miniconf? I appear to only have access after entering a shell.

`
grpartiq@3085:~/ARTIQ/Artiq_Workplace$ nix develop
I have no name!@3085:~/ARTIQ/Artiq_Workplace$ spyder
bash: spyder: command not found
I have no name!@3085:~/ARTIQ/Artiq_Workplace$ nix shell
bash: shopt: progcomp: invalid shell option name
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: shopt: hostcomplete: invalid shell option name
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
bash: complete: command not found
[][]I have no name!@3085[]:[]~/ARTIQ/Artiq_Workplace[]$ spyder
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
Gtk-Message: 18:08:27.297: Failed to load module "canberra-gtk-module"
Gtk-Message: 18:08:27.297: Failed to load module "canberra-gtk-module"
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled

fsagbuya

pyquest Yes, nix develop should give you access to both spyder and miniconf.

Did you change these lines as follows?

-      packages.x86_64-linux.default = pkgs.buildEnv {
+      devShells.x86_64-linux.default = pkgs.mkShell {
         name = "artiq-env";
-        paths = [
+        packages = [

Could you share the output of nix flake show?

pyquest

fsagbuya

Hi fsagbuya,

Thank you for the help. I missed that I needed to change path to packages. As a general question, as I am new to NIX.

1.) What is the practical difference between mkShell and mkEnv
2.) Where do most folks create their nix flakes? I find myself jumping around alot and its making me wonder if this is the workflow intended for artiq. So for instance I'd do something like

`
~ cd/ARTIQ/Nix_Working_Dir Nix Develop --------- startup the artiq environment/shell
gnome-terminal --tab --------- add new tab (2) while staying in the shell
~ cd/ARTIQ/artiq-master ---------go to new tab to dir for artiq master
gnome-terminal --tab ---------add new tab (3) while staying in the shell
gnome-terminal --tab ---------add new tab (4) while staying in the shell
artiq_master ---------startup artiq master in tab 2
artiq_ctlmgr --------- startup artiq_ctlmgr in tab 3
artiq_dashboard ---------startup artiq_dashboard in tab 4

gnome-terminal --tab --------- add new tab (5) while staying in the shell
~ cd/ARTIQ/stabilizer --------- go do stabilizer stuff

pyquest

Following up on this question. I really think documentation needs to be written for correct usage of nix environment with artiq.

I am finding that I am getting RTIO overflow errors but am unable to use event spreading from within the nix environment.

However if I run the command from not-within the nix develop environment I can.

I have no way of knowing if this command is going to the microcontroller properly. I run the artiq master from a nix develop shell so is the microcontroller getting this command? All artiq master , com , dashboard were closed for running these commands.

fsagbuya

1.
buildEnv creates a directory with symlinks to the packages you include in the paths attribute. When you use nix shell with it, you're basically entering that directory with everything set up.

mkShell creates a shell script that sets up the environment, which runs when you use nix develop.

You won’t notice much difference between nix shell and nix develop while using them. One feature of buildEnv is that you can install it more permanently with nix profile install from the flake root.

2.
Preferred way is to create Nix flakes in the root directory where you want to run your ARTIQ environment. You can also create a flake per experiment if you want different packages available for each one.

As for the workflow, just remember to always run nix develop or nix shell on your ARTIQ flake before executing any ARTIQ commands.

pyquest

fsagbuya

2.) I am doing that but presumably its not recommended to do that in artiq-master but we still run artiq master and the repository etc is still in the artiq-master install directory? If artiq-master is run from a terminal that is not in the nix develop/shell how is it exposed to the packages?

For example:

running artiq_ctlmgr in the ubuntu terminal

vs running artiq_ctlmgr in the nix-develop shell/terminal

3.) With respect to the installs. I am still a bit lost. While you helped me get it working (thank you). I am not sure that my understanding has increased. Here is another attempt (pypylon for basler cameras) at installing a github repository that is not in nix pkgs.

{
inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git";
inputs.miniconf.url = "github:quartiq/miniconf";
inputs.pypylon = {url = github.com/basler/pypylon ;flake = false; };
outputs = { self, extrapkg, miniconf,pypylon }:

let
  pkgs = extrapkg.pkgs;
  artiq = extrapkg.packages.x86_64-linux;
  pylon = pkgs.python3Packages.buildPythonPackage rec {
	name = "pypylon";
	src = pypylon.url;
	format = "pyproject";
	nativeBuildInputs = [ pkgs.python3Packages.setuptools pkgs.python3Packages.swig pkgs.python3Packages.wheel pkgs.python3Packages.lief pkgs.python3Packages.darwin];
	};

in {
   	
  devShells.x86_64-linux.default = pkgs.mkShell {
    name = "artiq-env";
    packages = [
      # ========================================
      # EDIT BELOW
      # ========================================
      (pkgs.python3.withPackages(ps: [
        # List desired Python packages here.
        artiq.artiq
        #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
        artiq.flake8-artiq
        artiq.dax
        artiq.dax-applets

        # The NixOS package collection contains many other packages that you may find
        # interesting. Here are some examples:
        ps.pandas
        ps.numpy
        ps.scipy
        ps.spyder
        pypylon

        #ps.numba
        ps.matplotlib
        ps.jupyterlab
        # or if you need Qt (will recompile):
        #(ps.matplotlib.override { enableQt = true; })
        ps.bokeh
        ps.cirq
        #ps.qiskit
      ]))
      
      miniconf.packages.x86_64-linux.miniconf-mqtt-py
             

      
      #artiq.korad_ka3005p
      #artiq.novatech409b
      # List desired non-Python packages here
      artiq.openocd-bscanspi  # needed if and only if flashing boards
      # Other potentially interesting non-Python packages from the NixOS package collection:
      #pkgs.gtkwave
      #pkgs.spyder

      #pkgs.R
      #pkgs.julia
      # ========================================
      # EDIT ABOVE
      # ========================================
    ];
  };
};
 nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
extra-substituters = "https://nixbld.m-labs.hk";
};
}

pyquest

Still not working so posting another flake file in the hope that someone can help me figure this out. I am following the discussion here.
https://discourse.nixos.org/t/fetch-from-github-dependencies-conky/62245/2

{
 inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git";
 inputs.miniconf.url = "github:quartiq/miniconf";
 inputs.pypylon = {url = github.com/basler/pypylon ;flake = false; };
  
  
  

outputs = { self, extrapkg, miniconf,pypylon }:

let
  pkgs = extrapkg.pkgs;
  artiq = extrapkg.packages.x86_64-linux;
  pypylon = pkgs.python3Packages.buildPythonPackage rec {
	pname = "pypylon";
	format = "pyproject";
	version = "4.1.0";
	doCheck = false; # check requires pip
	src =  pkgs.fetchFromGitHub {owner = "basler";
	    	 	 repo = "pypylon";
	    	 	 rev = "v${version}";};
	};	


in {
   	
  devShells.x86_64-linux.default = pkgs.mkShell {
    name = "artiq-env";
    packages = [
      # ========================================
      # EDIT BELOW
      # ========================================
      (pkgs.python3.withPackages(ps: [
        # List desired Python packages here.
        artiq.artiq
        #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
        artiq.flake8-artiq
        artiq.dax
        artiq.dax-applets

        ps.pandas
        ps.numpy
        ps.scipy
        ps.spyder
        pypylon

        #ps.numba
        ps.matplotlib
        ps.jupyterlab
        # or if you need Qt (will recompile):
        #(ps.matplotlib.override { enableQt = true; })
        ps.bokeh
        ps.cirq
        #ps.qiskit
      ]))
      
      miniconf.packages.x86_64-linux.miniconf-mqtt-py


      
      #artiq.korad_ka3005p
      #artiq.novatech409b
      # List desired non-Python packages here
      artiq.openocd-bscanspi  # needed if and only if flashing boards
      # Other potentially interesting non-Python packages from the NixOS package collection:
      #pkgs.gtkwave
      #pkgs.spyder

      #pkgs.R
      #pkgs.julia
      # ========================================
      # EDIT ABOVE
      # ========================================
    ];
    };
    };
   nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
  extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
  extra-substituters = "https://nixbld.m-labs.hk";
   };
    }

This spits out:

`
grpartiq@hrlit3085:~/ARTIQ/Artiq_Workplace$ nix develop
warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
error: builder for '/nix/store/fllgkd5xbafpq1zpiq6sjybcrx7r2z1q-source.drv' failed with exit code 1;
last 25 log lines:
> establish a secure connection to it. To learn more about this situation and
> how to fix it, please visit the webpage mentioned above.
>
> trying https://github.com/basler/pypylon/archive/v4.1.0.tar.gz
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> curl: (22) The requested URL returned error: 404
> Warning: Problem (retrying all errors). Will retry in 1 seconds. 3 retries
> Warning: left.
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> curl: (22) The requested URL returned error: 404
> Warning: Problem (retrying all errors). Will retry in 2 seconds. 2 retries
> Warning: left.
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> curl: (22) The requested URL returned error: 404
> Warning: Problem (retrying all errors). Will retry in 4 seconds. 1 retries
> Warning: left.
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
> curl: (22) The requested URL returned error: 404
> error: cannot download source from any mirror
For full logs, run:
nix log /nix/store/fllgkd5xbafpq1zpiq6sjybcrx7r2z1q-source.drv
error: 1 dependencies of derivation '/nix/store/l0f6z2mrkprynjx7ffd3yljvs9pmk551-python3.12-pypylon-4.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/f2a03rd612vgzvc5rbd5dfif67lszsd2-python3-3.12.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/n91jwggh2wcrq6265p8qyqchc6n8np6i-artiq-env-env.drv' failed to build

fsagbuya

pyquest

You can remove the pypylon input and instead define the package directly like this:

      pypylon = pkgs.python3Packages.buildPythonPackage rec {
        pname = "pypylon";
        version = "4.1.0";
        src = pkgs.fetchFromGitHub {
          owner = "basler";
          repo = "pypylon";
          rev = "${version}";
          hash = "sha256-qi74kq9l8OvmPZQ202+evnTChm2hY4PJkgzJAmv+/vY=";
        };
        nativeBuildInputs = [
          pkgs.swig
        ];
        doCheck = false;
      };

However, this package seems to require pylon SDK to be installed and expects it at /opt/pylon, as shown by the error:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pylon/bin/pylon-config'

pyquest

fsagbuya

So that is the weird thing. I have pylon installed at /opt/pylon/bin and the directory contains a pylon-config.

However, I still can't get the nix package manager to handle/find the file.

`
warning: updating lock file '"/home/grpartiq/ARTIQ/Artiq_Workplace/flake.lock"':
• Removed input 'pypylon'
error: builder for '/nix/store/f2bhkdwfn4vsdh0dzpm8c4zlx7z7p1li-python3.12-pypylon-4.1.0.drv' failed with exit code 1;
last 25 log lines:
> File "/build/source/nix_run_setup", line 8, in <module>
> exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))
> File "setup.py", line 1093, in <module>
> bs = BuildSupport.make()
> ^^^^^^^^^^^^^^^^^^^
> File "setup.py", line 372, in make
> return BuildSupportLinux()
> ^^^^^^^^^^^^^^^^^^^
> File "setup.py", line 820, in init
> config_cflags = self.call_pylon_config("--cflags")
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "setup.py", line 908, in call_pylon_config
> res = subprocess.check_output(params, universal_newlines=True)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 468, in check_output
> return run(popenargs, stdout=PIPE, timeout=timeout, check=True,
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 550, in run
> with Popen(popenargs, **kwargs) as process:
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 1028, in init
> self._execute_child(args, executable, preexec_fn, close_fds,
> File "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/subprocess.py", line 1963, in _execute_child
> raise child_exception_type(errno_num, err_msg, err_filename)
> FileNotFoundError: [Errno 2] No such file or directory: '/opt/pylon/bin/pylon-config'
For full logs, run:
nix log /nix/store/f2bhkdwfn4vsdh0dzpm8c4zlx7z7p1li-python3.12-pypylon-4.1.0.drv
error: 1 dependencies of derivation '/nix/store/9z7z9g20sfnlkw3n5520rj4qkbgg7b1d-python3-3.12.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/dcrfmm96pdhvw2x2v67cmpx47jjspp6m-artiq-env-env.drv' failed to build

fsagbuya

pyquest
Hi, I managed to build the pypylon package with the following changes:

packaged pylonSdk in nix and used it as buildInputs for pypylon
updated the pypylon version due to swig issues
added /opt to sandbox paths so it can access /opt/pypylon/bin

--- a/flake.nix
+++ b/flake.nix
@@ -7,15 +7,29 @@
       pkgs = extrapkg.pkgs;
       artiq = extrapkg.packages.x86_64-linux;
 
+      pylon-sdk = pkgs.stdenv.mkDerivation {
+        name = "pylon-sdk";
+        version = "8.1.0";
+        dontUnpack = true;
+        installPhase = ''
+          mkdir -p $out/include $out/lib
+          cp -r /opt/pylon/include/* $out/include/
+          cp -r /opt/pylon/lib/* $out/lib/
+        '';
+      };
+
       pypylon = pkgs.python3Packages.buildPythonPackage rec {
         pname = "pypylon";
-        version = "4.1.0";
+        version = "4.2.0rc2";
         src = pkgs.fetchFromGitHub {
           owner = "basler";
           repo = "pypylon";
           rev = "${version}";
-          hash = "sha256-qi74kq9l8OvmPZQ202+evnTChm2hY4PJkgzJAmv+/vY=";
+          hash = "sha256-juzQ3Mluf+YskKM+qha8TH2ALgSjVMEKVB0BdVDKVew=";
         };
+        buildInputs = [
+          pylon-sdk
+        ];
         nativeBuildInputs = [
           pkgs.swig
         ];
@@ -75,5 +89,6 @@
   nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
     extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
     extra-substituters = "https://nixbld.m-labs.hk";
+    extra-sandbox-paths = "/opt";
   };
 }

pyquest

fsagbuya

This compiled for me with nix develop. However it doesn't work in a way that I understand.

sb10q

You're probably mixing up ARTIQ versions, hence the error. You need to use the same major version for the firmware and the software, regardless how you install it.
artiq_coremgmt works independently of the other software components.

pyquest

sb10q

How do I fix / install/ setup the nix manager appropriately? I have followed the instructions to the best of my know know. I am currently overwhelmed with the number of directories artiq/nix requires.

Currently running like this

gnome-terminal --tab --------- add new tab (5) while staying in the shell
~ cd/ARTIQ/stabilizer --------- go do stabilizer stuff

Other issues include - not sure where the dax files are being installed or why the artiq master is not seeing them? I think they should be installed per the flake.

Similiarly, modifying the flake to include py.pyvisa is very confusing. For example:

`
import pyvisa

a = pyvisa.ResourceManager()

ValueError Traceback (most recent call last)
Cell In[11], line 1
----> 1 a = pyvisa.ResourceManager()

File /nix/store/g2nglci3s4yyxvfyw08al5ivwzdy2mb8-python3-3.12.9-env/lib/python3.12/site-packages/pyvisa/highlevel.py:2992, in ResourceManager.new(cls, visa_library)
2982 """Create a new resource manager tied to the specified VISA library.
2983
2984 Parameters
(...)
2989
2990 """
2991 if not isinstance(visa_library, VisaLibraryBase):
-> 2992 visa_library = open_visa_library(visa_library)
2994 if visa_library.resource_manager is not None:
2995 obj = visa_library.resource_manager

File /nix/store/g2nglci3s4yyxvfyw08al5ivwzdy2mb8-python3-3.12.9-env/lib/python3.12/site-packages/pyvisa/highlevel.py:2899, in open_visa_library(specification)
2897 wrapper = "ivi"
2898 else:
-> 2899 wrapper = _get_default_wrapper()
2901 cls = get_wrapper_class(wrapper)
2903 try:

File /nix/store/g2nglci3s4yyxvfyw08al5ivwzdy2mb8-python3-3.12.9-env/lib/python3.12/site-packages/pyvisa/highlevel.py:2858, in _get_default_wrapper()
2856 except ValueError:
2857 logger.debug("Did not find pyvisa-py package")
-> 2858 raise ValueError(
2859 "Could not locate a VISA implementation. Install either the IVI binary or pyvisa-py."
2860 )

ValueError: Could not locate a VISA implementation. Install either the IVI binary or pyvisa-py.

`
The error is not on the import it is on calling the resource manager.

fsagbuya

pyquest Is Jupyter running inside the same Nix environment? If not, you should run it from there by adding ps.jupyter to the flake, running nix develop, and then executing the command jupyter notebook.

pyquest

fsagbuya

It was running from within the nix develop environment. The notebook was launched with jupyter lab from a terminal after nix develop with our flake and with ps.jupyterlab in the flake.