Nix build setup - SSL certificate failure

nx-

Hi!

I have now tried for a while to set up the ARTIQ nix build system on my Ubuntu 20.04.6 LTS machine (I want to modify ARTIQ so I do need to build everything). I installed nix 2.13.3 and I am using custom SSL certificates, should that be important.

I am following the steps given in the "Develop ARTIQ" section of the documentation. The problem I did not manage to sort out myself is when I try to activate the development shell which uses the flake.nix file in the ARTIQ repository (cloned branch release-7 into a local repo, cd'd there and invoking nix develop). I removed download retries that yielded the same result.

$ nix develop -v
Using saved setting for 'extra-sandbox-paths = /opt' from ~/.local/share/nix/trusted-settings.json.
Using saved setting for 'extra-substituters = https://nixbld.m-labs.hk' from ~/.local/share/nix/trusted-settings.json.
Using saved setting for 'extra-trusted-public-keys = nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=' from ~/.local/share/nix/trusted-settings.json.
warning: error: unable to download 'https://nixbld.m-labs.hk/gpd2hhhn8b4qh534i0vxm3gkd91x4a9g.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 277 ms
[...]
error: unable to download 'https://nixbld.m-labs.hk/gpd2hhhn8b4qh534i0vxm3gkd91x4a9g.narinfo': SSL peer certificate or SSH remote key was not OK (60)
(use '--show-trace' to show detailed location information)

The problem seems to occur when querying channel-rust-nightly.toml from https://nixbld.m-labs.hk. I do have the public keys set up for https://nixbld.m-labs.hk, I am not sure why this fails. It must be something connected to the certificates though, as nix-channel --update also fails.

nix-channel --update
warning: error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full': SSL peer certificate or SSH remote key was not OK (60); retrying in 347 ms
[...]

I do have the trusted key and substituters set up I think. Different tutorials and GitHub issues suggest different things, I ended up trying all locations I know, nothing worked. The keys to M-labs should also be set up in flake.nix anyway..

$ cat /etc/nix/nix.conf 
substituters = https://cache.nixos.org https://nixbld.m-labs.hk
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=
$ cat ~/.config/nix/nix.conf 
experimental-features = nix-command flakes
substituters = https://cache.nixos.org https://nixbld.m-labs.hk
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=

Any ideas? Sorry should this be blatantly obvious - I am not familiar with nix and find its setup (at least behind a corporate firewall) very confusing 😃

Thanks!

sb10q

nx- The keys to M-labs should also be set up in flake.nix anyway

Those keys have nothing to do with SSL.

The SSL certificate on nixbld.m-labs.hk is from Letsencrypt. Is it included in your custom certificates? Why do you need custom certificates anyway? Is it because your employer intercepts your SSL traffic? In such case you need to set up Nix to use the certificate of the MITM proxy.

nx-

Some more info just in case. I set the environment variable NIX_SSL_CERT_FILE before the nix (multi-user, daemon) installation as shown here, this is the only option that did not complain during installation. I also set this and SSL_CERT_FILE in my ~/.bashrc. I added nix-channels artiq-full and nixos-22.11 (this was the version I could see in artiq/flake.nix).

$ nix-channel --list
artiq-full https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full
nixpkgs https://nixos.org/channels/nixos-22.11
$ nix --version
nix (Nix) 2.13.3

nx-

Sorry for the confusion with those keys, you are right that was irrelevant for SSL. I should probably remove those tripe-stated keys and have them at just one location. Anyway, yes, my company does intercept SSL traffic. I have (tried to) set up NIX with those certificates, but I think either the nix develop shell does not see them as it creates its own environment or maybe the custom certificate does not include the letsencrypt ones?

Thank you for your quick reply though - I see now this might not be an issue related to ARTIQ at all.

sb10q

In theory, Nix uses the system SSL certificates and does not set up its own.
This forum is also using Letsencrypt, so if you are not seeing issues here and assuming you are posting from the same machine/network, it would seems the MITM proxy does support the Letsencrypt certificates.
Do you also have issues installing the main Nix packages from cache.nixos.org? Remove nixbld.m-labs.hk if Nix still tries to access it then.

sb10q

Do you know what triggers the SSL interception? If it's the port number (and they don't block other ports), it seems I could perhaps open a second port on the server e.g. 8892 and then you could add e.g. :8892 to all *.m-labs.hk addresses to stop it. Note that nixbld.m-labs.hk can serve many cache.nixos.org packages as well.

sb10q

Are you able to access https://nixbld.m-labs.hk:8892 (without manual proxy setup, and without custom certificates)?

nx-

Sorry for the long delay - I forgot to enable notifications for answers to my discussion, done now. Nix does use system SSL certificates, the first mistake I made was to save it as mycert.cer instead of mycert.crt (in /usr/local/share/ca-certificates which was then ignored by update-ca-certificates. I did correct this, not sure if I made it worse or just uncovered the next hurdle, but I have worse problems now running nix-channel --update. I do think, as last time, that these issues are not ARTIQ related at all - so I don't expect you to waste your time with this. FYI, I cannot access https://nixbld.m-labs.hk:8892, have tried on several machines.

Basically what I get now is HTTP error 500 from the server, so not even the server knows whats going on.

$ nix-channel --update
warning: error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full/nixexprs.tar.xz': HTTP error 500; retrying in 319 ms
...
warning: error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full/nixexprs.tar.bz2': HTTP error 500; retrying in 269 ms
...
error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full/nixexprs.tar.bz2': HTTP error 500

Probably my own fault again, although I don't really know where to start. I had other issues ($USER containing domain and backslash, problems in creating nix per-user lock files, but fixed that by creating a folder there manually) - but the HTTP error 500 persists.

I will update this or create a new discussion should I find a solution 🙂

nx-

Weirdly the HTTP error does only occur with the ARTIQ channel, not the NIXOS channel... the warning for removing artiq-full I do not understand as it clearly works, but thats for another time.

$ nix-channel --list
artiq-full https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full
nixpkgs https://nixos.org/channels/nixos-22.11
$ nix-channel --remove artiq-full
warning: selector 'artiq-full' matched no installed derivations
$ nix-channel --list
nixpkgs https://nixos.org/channels/nixos-22.11
$ nix-channel --update
unpacking channels...
$ nix-channel --add https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full
$ nix-channel --update
warning: error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full/nixexprs.tar.xz': HTTP error 500; retrying in 319 ms
...
warning: error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full/nixexprs.tar.bz2': HTTP error 500; retrying in 269 ms
...
error: unable to download 'https://nixbld.m-labs.hk/channel/custom/artiq/full/artiq-full/nixexprs.tar.bz2': HTTP error 500

sb10q

The error 500 problem is the same as https://forum.m-labs.hk/d/567-nix-channel-down

nx-

Thats the issue when you use old instructions.. thanks!! I am able to run nix-develop in the ARTIQ source directory now, using the flake.nix file. It cannot download from either cache.nixos.org or nixbld.m-labs.hk (the error is warning: unable to download 'https://<>/nix-cache-info': Problem with the SSL CA cert (path? access rights?) (77) in both cases) and is now building stuff like gcc from source but oh well - it will take a while but work at some point 😃 Think this is a curl error of curl in the nix-develop-shell, not sure why it cannot find the certificates. Anyway, I'll figure it out - thank you for your help @sb10q!