Including a local python package in an ARTIQ Nix shell

Jjroth · Jun 30, 2024

Hi,

I've been trying to figure out how to install a local python package into my ARTIQ Nix environment on Ubuntu 22.04.4. The package will need to import ARTIQ as it implements several kernel functions. I've been running into trouble getting Nix to include the package for me.

As a minimum example I've tried creating the following files:

artiq-8-custompkg/
├── example_pkg
│   ├── example_pkg
│   │   ├── example.py
│   │   └── __init__.py
│   ├── pyproject.toml
│   └── setup.py
└── flake.nix

The contents of each file are:

example.py:

from artiq.experiment import kernel

def example():
  print("Testing 1")

@kernel
def kernel_example():
  print("Testing 2")

pyproject.toml

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

setup.py

#!/usr/bin/env python

from setuptools import setup, find_packages

setup(name='example_pkg',
      version='1.0',
      # Modules to import from other scripts:
      packages=find_packages(),
     )

flake.nix

{
  inputs.extrapkg.url = "git+https://git.m-labs.hk/M-Labs/artiq-extrapkg.git?ref=release-8";
  outputs = { self, extrapkg }:
    let
      pkgs = extrapkg.pkgs;
      artiq = extrapkg.packages.x86_64-linux;
      test = extrapkg.pkgs.python3Packages.buildPythonPackage rec {
        pname = "example_pkg";
        version="1.0";
        src = "example_pkg";
      };
    in {
      defaultPackage.x86_64-linux = pkgs.buildEnv {
        name = "artiq-env";
        paths = [
          # ========================================
          # EDIT BELOW
          # ========================================
          (pkgs.python3.withPackages(ps: [
            # List desired Python packages here.
            artiq.artiq
            ps.flask
            test
          ]))
        ];
      };
    };
  nixConfig = {  # work around https://github.com/NixOS/nix/issues/6771
    extra-trusted-public-keys = "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=";
    extra-substituters = "https://nixbld.m-labs.hk";
  };
}

When I run nix shell from theartiq-8-custompkg directory I get the following error:

error: builder for '/nix/store/ngpgf2zp3wl4xaqsvvzzwfrx296i9bsv-python3.11-example_pkg-1.0.drv' failed with exit code 1;
       last 16 log lines:
       > Sourcing python-remove-tests-dir-hook
       > Sourcing python-catch-conflicts-hook.sh
       > Sourcing python-remove-bin-bytecode-hook.sh
       > Sourcing setuptools-build-hook
       > Using setuptoolsBuildPhase
       > Sourcing pypa-install-hook
       > Using pypaInstallPhase
       > Sourcing python-imports-check-hook.sh
       > Using pythonImportsCheckPhase
       > Sourcing python-namespaces-hook
       > Sourcing python-catch-conflicts-hook.sh
       > Sourcing setuptools-check-hook
       > Using setuptoolsCheckPhase
       > Running phase: unpackPhase
       > unpacking source archive example_pkg
       > do not know how to unpack source archive example_pkg
       For full logs, run 'nix log /nix/store/ngpgf2zp3wl4xaqsvvzzwfrx296i9bsv-python3.11-example_pkg-1.0.drv'.
error: 1 dependencies of derivation '/nix/store/ayk4qf9wwzda1i91sbkl6f63zwab7h6q-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/nffx49gvg9x7vlff8b8sjb5dypyisqlr-artiq-env.drv' failed to build

Do I have the right approach here? Or is there a better approach for installing a local python package?

Jjroth · Jun 30, 2024

I've just realized I can include pip in the Nix environment by including ps.pip. Then I can install the local package via pip install -e example_pkg. Is there any reason to avoid this approach?

fsagbuya · Jul 1, 2024

Hi jroth,

The main error you were having before was likely due to the src string not being correctly interpreted as a path. You can fix this by changing it as shown in this diff:

--- a/example_pkg/setup.py
+++ b/example_pkg/setup.py
@@ -6,4 +6,5 @@ setup(name='example_pkg',
       version='1.0',
       # Modules to import from other scripts:
       packages=find_packages(),
+      install_requires=['artiq'],
      )

--- a/flake.nix
+++ b/flake.nix
@@ -4,10 +4,17 @@
     let
       pkgs = extrapkg.pkgs;
       artiq = extrapkg.packages.x86_64-linux;
-      test = extrapkg.pkgs.python3Packages.buildPythonPackage rec {
+      test = pkgs.python3Packages.buildPythonPackage rec {
         pname = "example_pkg";
+        pyproject = true;
         version="1.0";
-        src = "example_pkg";
+        src = ./example_pkg;
+        nativeBuildInputs = [
+          pkgs.python3Packages.setuptools
+          pkgs.qt5.wrapQtAppsHook
+        ];
+        propagatedBuildInputs = [ artiq.artiq ];
+        dontWrapQtApps = true;
       };
     in {
       defaultPackage.x86_64-linux = pkgs.buildEnv {

Also added other requirements to ensure the package builds correctly. These should address potential issues with ARTIQ dependencies and Qt handling.

I've only tested basic imports, so please let me know if you encounter any issues with more complex usage.

fsagbuya · Jul 1, 2024

jroth Using pip in Nix can reduce reproducibility and add complexity, as it's not a pure Nix approach.

However, if you want to try it for development, this nixpkgs manual section might help:
https://nixos.org/manual/nixpkgs/stable/#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems

Jjroth · Jul 1, 2024

Hi @fsagbuya,
Thanks a bunch! This solved my problem. I appreciate it

Mmcstron · Nov 20, 2024

Hello, I am running into a similar issue installing a local python package. As a sanity check, I replicated this simple example (using @fsagbuya's edits) and am unable to make it work. I am also on Ubuntu 22.04, but with ArtiQ 7 using flakes.

I have copied all the exact files from this post and replicated the same file structure, with my artiq directory at the top and the example_pkg contained within. When I attempt to run nix develop I get the following error:

… while evaluating attribute 'src' of derivation 'python3.9-example_pkg-1.0'
         at /nix/store/dqz1gp2bs05hy22r9rl97wzravvfp4mv-source/flake.nix:16:10:
           15|          version="1.0";
           16|          src = ./example_pkg;
             |          ^
           17|          nativeBuildInputs = [

       error: path '/nix/store/dqz1gp2bs05hy22r9rl97wzravvfp4mv-source/example_pkg' does not exist

Here is the flake.nix that I am using for this test.

{
    inputs.daxpkgs.url = tarball+http://build-mblab.duckdns.org/channel/custom/dax-7/fast/dax-fast/nixexprs.tar.xz;
    inputs.nixpkgs.follows = "daxpkgs/nixpkgs";
    inputs.artiqpkgs.follows = "daxpkgs/artiqpkgs";

  outputs = { self, daxpkgs, nixpkgs, artiqpkgs }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; };
      dax-full = daxpkgs.packages.x86_64-linux;
      artiq-full = artiqpkgs.packages.x86_64-linux;

      test = pkgs.python3Packages.buildPythonPackage rec {
         pname = "example_pkg";
         pyproject = true;
         version="1.0";
         src = ./example_pkg;
         nativeBuildInputs = [
           pkgs.python3Packages.setuptools
           pkgs.qt5.wrapQtAppsHook
         ];
         dontWrapQtApps = true;
        };

      in
      {
      devShells.x86_64-linux.default = pkgs.mkShell {
        name = "artiq-env";
        buildInputs = [
          (pkgs.python3.withPackages(ps: [
            # List desired Python packages here.
            dax-full.dax # includes ARTIQ package
            test
          ]))

        ];
      };
    };

  nixConfig = {
    extra-trusted-public-keys = [
      "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc="
    ];
    extra-substituters = [ "https://nixbld.m-labs.hk" ];
    bash-prompt-suffix = "\[nix\]";
  };
}

Jjroth · Nov 20, 2024

mcstron what directory is your flake.nix file located in? In the above example it is in the same directory as example_pkg. Is this also true in your setup?

Mmcstron · Nov 20, 2024

Yes, this is also true in my setup! They are both in a directory that is equivalent to your "artiq-8-custompkg" directory.

fsagbuya · Nov 21, 2024

mcstron

I tested your flake but couldn't reproduce the issue. Could you provide more details about your file structure? A tree-like view, as shown in @jroth's example, would be helpful.