flake.nix for local library (C++ with its python binder)

5ik

Hello, I also posted same question here- https://discourse.nixos.org/t/flake-nix-for-local-library-with-binder-spinnaker-sdk-pyspin/66489, but not sure nix forum is the best place to artiq flake.nix problem so I'm asking a same question here too.

I'm new to Nix and trying to do following:
1) We're using nix (flake.nix) for Artiq system. It's a quantum computer control system.
You can find more information about Artiq with Nix from here:

https://m-labs.hk/artiq/manual/installing.html

2) Based on pre-built flake.nix file, what I want to do is add libraries (Spinnaker SDK and PySpin).
Spinnaker SDK is C++ camera library and PySpin is python binding for this.
You can find more information about this from here:
(You might need to log-in to access the second link).

https://www.teledynevisionsolutions.com/products/spinnaker-sdk/?model=Spinnaker%20SDK&vertical=machine%20vision&segment=iis

https://www.teledynevisionsolutions.com/support/support-center/software-firmware-downloads/iis/spinnaker-sdk-download/spinnaker-sdk--download-files/?pn=Spinnaker+SDK&vn=Spinnaker+SDK

Since Spinnaker SDK and PySpin isn't on nixpackage, I've created the derivation and trying to install on the Nix.
The file directories are following:

aqpcontrol (top directory)
|-flake.nix
|-spinnaker-deb.nix
|-spinnaker_python.nix
|-spinnaker-debs
| |-(contains .deb files)
|-spinnaker-whl
| |-spinnaker_python.whl

The way to install this in the "local" computer (not through flake.nix) is run install.sh file which basically install every .deb files and install .whl file. Then we can import PySpin in the .py.

I didn't include every .deb files on git, but added a .whl file.
Two derivation and flake.nix I've created is following:
flake.nix

{
  inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git?ref=release-8";
  inputs.spinnaker-debs = {
    type = "path";
    path = "./spinnaker-debs";
    flake = false;
  };
  outputs = { self, extrapkg, spinnaker-debs}:
    let
      pkgs = extrapkg.pkgs;
      artiq = extrapkg.packages.x86_64-linux;
      # Originally it was spinnaker_python-4.2.0.46-cp310-cp310-linux_x86_64.whl. I changed the name.
      spinnaker_whl = ./spinnaker-whl/spinnaker_python-4.2.0.46-cp310-cp310-linux_x86_64.whl;


    # Spinnaker SDK derivation import
      spinnaker = import ./spinnaker-sdk.nix {
        inherit (pkgs) stdenv dpkg lib;
        spinnaker_debs = spinnaker-debs;
      };
      # Spinnaker_python wheel derivation import
      spinnaker_python = import ./spinnaker_python.nix {
        inherit pkgs;
        inherit (pkgs) lib stdenv;
        buildPythonPackage = pkgs.python310Packages.buildPythonPackage;
        inherit spinnaker spinnaker_whl;
      };
      
      myPython = pkgs.python310.withPackages(ps: [
        spinnaker_python
        #test integrattion of artiq with spinnaker
        artiq.artiq
        ps.pandasps.matplotlib
        ps.pyvisa
        ps.pyvisa-py
      ]);

    in {
      devShells.x86_64-linux = builtins.mapAttrs makePythonShell pythonVersions;
      devShells.default = pkgs.mkShell {
        buildInputs = [
          spinnaker
          myPython
        ];

        shellHook = ''
          export SPINNAKER_ROOT=${spinnaker}
          export LD_LIBRARY_PATH=${spinnaker}/lib:$LD_LIBRARY_PATH
          export CPLUS_INCLUDE_PATH=${spinnaker}/include
          echo "Spinnaker dev environment activated!"
        '';
      };
      packages.x86_64-linux.spinnaker_python = spinnaker_python;
      # This section defines the new environment
      packages.x86_64-linux.default = pkgs.buildEnv {
        name = "artiq-env";
        paths = [
          # ========================================
          # ADD PACKAGES BELOW
          # ========================================
          (pkgs.python3.withPackages(ps : [
            # List desired Python packages here.
            #artiq.artiq
            myPython
            #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
            #artiq.flake8-artiq
            #artiq.dax
            #artiq.dax-applets

            # The NixOS package collection contains many other packages that you may find
            # interesting. Here are some examples:
            #ps.pandas
            #ps.numba
            #ps.matplotlib
            #ps.pyvisa
            #ps.pyvisa-py
            # or if you need Qt (will recompile):
            #(ps.matplotlib.override { enableQt = true; })
            #ps.bokeh
            #ps.cirq
            #ps.qiskit
            # Note that NixOS also provides packages ps.numpy and ps.scipy, but it is
            # not necessary to explicitly add these, since they are dependencies of
            # ARTIQ and incorporated with an ARTIQ install anyway.
          ]))
          # List desired non-Python packages here
          # Additional NDSPs can be included:
          #artiq.korad_ka3005p
          #artiq.novatech409b
          # Other potentially interesting non-Python packages from the NixOS package collection:
          #pkgs.gtkwave
          #pkgs.spyder
          #pkgs.R
          #pkgs.julia
          # ========================================
          # ADD PACKAGES ABOVE
          # ========================================
        ];
      };
    };
  # This section configures additional settings to be able to use M-Labs binary caches
  nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
    extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
    extra-substituters = "https://nixbld.m-labs.hk";
  };
}

spinnaker_python.nix

{ pkgs, lib, stdenv, buildPythonPackage, spinnaker, spinnaker_whl }:

let
  fixedWheel = pkgs.runCommand "fixed-wheel" {
    nativeBuildInputs = [ pkgs.coreutils ];
  } ''
    mkdir -p $out
    cp ${spinnaker_whl} $out/spinnaker_python-4.2.0.46-cp310-cp310-linux_x86_64.whl
  '';
in

buildPythonPackage rec {
  pname = "spinnaker_python";
  version = "4.2.0.46";
  format = "wheel";

  src = "${fixedWheel}/spinnaker_python-4.2.0.46-cp310-cp310-linux_x86_64.whl";

  propagatedBuildInputs = [ spinnaker ];

   postInstall = ''
    echo "Patching _PySpin shared object..."
    sofile=$(find $out -name '_PySpin*.so' | head -n1 || true)
    if [ -n "$sofile" ]; then
      patchelf --set-rpath ${spinnaker}/lib:$(patchelf --print-rpath "$sofile") "$sofile"
    else
      echo "Warning: _PySpin*.so not found!"
    fi
  '';

  pythonImportsCheck = [ "PySpin" ];

  shellHook = ''
    export LD_LIBRARY_PATH=${spinnaker}/lib:$LD_LIBRARY_PATH
  '';

  preCheck = ''
    export LD_LIBRARY_PATH=${spinnaker}/lib:$LD_LIBRARY_PATH
  '';

  meta = with lib; {
    description = "FLIR Spinnaker Python bindings";
    platforms = platforms.linux;
  };
}

Finally,
spinnaker-sdk.nix

{ stdenv, dpkg, lib, spinnaker_debs }:

stdenv.mkDerivation rec {
  pname = "spinnaker-sdk";
  version = "4.2.0.46";

  src = spinnaker_debs;

  nativeBuildInputs = [ dpkg ];

  unpackPhase = ''
    for deb in $src/*.deb; do
      echo "Unpacking $deb..."
      dpkg-deb -x "$deb" .
    done
  '';

  installPhase = ''
    mkdir -p $out
    cp -r * $out/
  '';

  meta = with lib; {
    description = "Spinnaker SDK from FLIR (.deb unpacked)";
    platforms = platforms.linux;
  };
}

The problem is, when I tried to import PySpin after develop it, it doesn't recognize PySpin library. Also, from the

nix build .#spinnaker_python --show-trace

I realized that .whil flie doesn't recognize .so files.

error: builder for '/nix/store/pg2sbl4wx9pf3l38fq581fzc8izhhg4w-python3.10-spinnaker_python-4.2.0.46.drv' failed with exit code 1;
       last 25 log lines:
       > checking for references to /build/ in /nix/store/0fwhkzjsqvzvzrzcp1cxag2b6d0n0jql-python3.10-spinnaker_python-4.2.0.46-dist...
       > patching script interpreter paths in /nix/store/0fwhkzjsqvzvzrzcp1cxag2b6d0n0jql-python3.10-spinnaker_python-4.2.0.46-dist
       > Executing pythonRemoveTestsDir
       > Finished executing pythonRemoveTestsDir
       > Running phase: installCheckPhase
       > no Makefile or custom installCheckPhase, doing nothing
       > Running phase: pythonCatchConflictsPhase
       > Running phase: pythonRemoveBinBytecodePhase
       > Running phase: pythonImportsCheckPhase
       > Executing pythonImportsCheckPhase
       > Check whether the following modules can be imported: PySpin
       > Traceback (most recent call last):
       >   File "<string>", line 1, in <module>
       >   File "<string>", line 1, in <lambda>
       >   File "/nix/store/wm85wvkj8w9lk3s53k6s3zkgi5p4xf2j-python3-3.10.17/lib/python3.10/importlib/__init__.py", line 126, in import_module
       >     return _bootstrap._gcd_import(name[level:], package, level)
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
       >   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
       >   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
       >   File "/nix/store/m87nsd4r5a0ipmskqmhl44k2kzgs023b-python3.10-spinnaker_python-4.2.0.46/lib/python3.10/site-packages/PySpin.py", line 15, in <module>
       >     import _PySpin
       > ImportError: libSpinnaker.so.4: cannot open shared object file: No such file or directory
       For full logs, run:

While I'm writing down this question, I realized that even I added

      pythonVersions = {
        python38 = pkgs.python38;
        python39 = pkgs.python39;
        python310 = pkgs.python310;
        default = pkgs.python310;
      };

      # A function to make a shell with a python version
      makePythonShell = shellName: pythonPackage: pkgs.mkShell {
        # You could add extra packages you need here too
        packages = [ pythonPackage ]; 
        # You can also add commands that run on shell startup with shellHook
        shellHook = ''
          echo "Now entering ${shellName} environment."
        '';
      };
``` 


and set environment as python 3.10, it seems like nix environment still use python 3.12 only. Does this the main reason? since Spinnaker SDK only supports 3.10. 
Also does anyone tried to add C++ libraries with python bindings on Nix? If so, could you give me advise to fix this error?
Sorry for the unorganized questions and any advises would be very appreciated.
Thanks
-Ohik

fsagbuya

Hi,

I was able to build it using a minimal setup. I’ve only tested it with the latest version 4.2.0.88, but you should be able to just change the version string if you have a different one. Basic import works, though I haven’t tested functionality beyond that.

Here’s the relevant flake.nix diff:

diff --git a/flake.nix b/flake.nix
index ad4b828..30f02b4 100644
--- a/flake.nix
+++ b/flake.nix
@@ -4,6 +4,47 @@
     let
       pkgs = extrapkg.pkgs;
       artiq = extrapkg.packages.x86_64-linux;
+
+      spinnaker-sdk = pkgs.stdenv.mkDerivation rec {
+        pname = "spinnaker-sdk";
+        version = "4.2.0.88";
+        src = ./spinnaker-debs;
+        nativeBuildInputs = [
+          pkgs.dpkg
+          pkgs.autoPatchelfHook
+        ];
+        buildInputs = [
+          pkgs.stdenv.cc.cc.lib
+          pkgs.libusb1
+          pkgs.zlib
+          pkgs.ffmpeg_4
+        ];
+        unpackPhase = ''
+          for deb in $src/*.deb; do
+            dpkg-deb -x "$deb" .
+          done
+        '';
+        installPhase = ''
+          mkdir -p $out/lib
+          cp -r opt/spinnaker/lib/* $out/lib/
+        '';
+      };
+
+      spinnaker = pkgs.python310Packages.buildPythonPackage rec {
+        pname = "spinnaker";
+        version = "4.2.0.88";
+        format = "wheel";
+        src = ./spinnaker_python-${version}-cp310-cp310-linux_x86_64.whl;
+        nativeBuildInputs = [ pkgs.autoPatchelfHook ];
+        buildInputs = [
+          pkgs.stdenv.cc.cc.lib
+          spinnaker-sdk
+        ];
+        propagatedBuildInputs = [
+          pkgs.python310Packages.numpy
+        ];
+        pythonImportsCheck = [ "PySpin" ];
+      };
     in {
       # This section defines the new environment
       packages.x86_64-linux.default = pkgs.buildEnv {
@@ -12,9 +53,10 @@
           # ========================================
           # ADD PACKAGES BELOW
           # ========================================
-          (pkgs.python3.withPackages(ps: [
+          (pkgs.python310.withPackages(ps: [
             # List desired Python packages here.
             artiq.artiq
+            spinnaker
             #ps.paramiko  # needed if and only if flashing boards remotely (artiq_flash -H)
             #artiq.flake8-artiq
             #artiq.dax

Directory structure:

.
├── flake.lock
├── flake.nix
├── spinnaker-debs
│   ├── libspinnaker_4.2.0.88_amd64.deb
│   ├── ...     # other deb files
└── spinnaker_python-4.2.0.88-cp310-cp310-linux_x86_64.whl

Tested with:

Python 3.10.17 (main, Apr  8 2025, 12:10:59) [GCC 14.2.1 20250322] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import PySpin
>>>

5ik

Thank you for your response fsagbuya .
I was also able to success before you responded my question and now I can import PySpin.
However, what I'm facing now is this nix shell can not recognize "artiq_run" or any artiq scripts.
I also changed flake.nix file to your method and it still does not work.

aqp-linux@aqp-linux-inspiron:~/artiq_test/aqpcontrol$ artiq_run
bash: artiq_run: command not found```

One thing I'm very suspicious is in our computer we already have used artiq with python 3.12.
So it seems like it conflicts each other whenever we call them. (or artiq doesn't support python 3.10?)
Does this problem also occurs in your setup @fsagbuya?
Thanks

5ik

Okay now it works, but I'm not sure whether it's good way to fix it.
I just simply took out artiq.aritq from pkgs.python310.withPackages. So mine looks like

(pkgs.python310.withPackages(ps:[
#python packages
]))
artiq.artiq

I think it should be fine since artiq isn't python package, but in that case why we have artiq.artiq in the python.withPackages block at the first time?

fsagbuya

I think it should be fine since artiq isn't python package, but in that case why we have artiq.artiq in the python.withPackages block at the first time?

Actually, ARTIQ is a python package. When it's included inside python.withPackages, it makes sure you can import artiq in python scripts. If you leave it out, you'll still be able to run the command-line tools like artiq_run and artiq_master, but python code that tries to import artiq might not work depending on your environment.

I tried using ARTIQ with Python 3.10 to match it with spinnaker-python, but ran into compatibility issues. The latest spinnaker wheel expects numpy 2, which only supports python 3.11+ according to numpy’s deprecation policy. That creates a conflict to the installed artiq.

One possible solution is to use an older version of the spinnaker SDK and API that’s still based on numpy 1.x.

5ik

fsagbuya Thank you for the explanation.
I've look up the spinnaker-python requirements and indeed they need numpy>=2.0.
And by looking at python version and numpy version of m-lab's artiq flake.nix, I realized that it uses python 3.10 and numpy 2.25 as default.
However, I think python 3.10.17 (python version pkgs.python310) also supports numpy>=2.0 according to https://numpy.org/devdocs/release/2.2.5-notes.html.

This concludes, I guess, that when we call artiq.artiq inside of pkgs.python310.withPackages, it will not install artiq since it uses python 3.12 by default.
Do you think it's possible to use python 3.10 for artiq 8? If it's possible, how we can change python 3.12 to python 3.10?
Thanks

fsagbuya

Do you think it's possible to use python 3.10 for artiq 8? If it's possible, how we can change python 3.12 to python 3.10?

It’s technically possible, but it requires modifying the ARTIQ flake itself. You’ll need to use an older commit of the artiq repo that still used Python 3.10, and then patch numpy to version 2.0 manually, since artiq 8 is built against numpy 1.x. After that, you’ll have to recompile artiq and potentially resolve any resulting compatibility issues.