Convert the server from Arch to NixOS

a commented

2022-05-20 17:41:06 +00:00

Owner

This is a very big change, so I propose to break this down into smaller parts:

#138 Package everything installed on the current server that's not in nixpkgs.
Create a NixOS container to duplicate our current Arch Linux setup.
Migrate data from Arch to the NixOS container.
Boot into a live USB and move the container filesystem to the root filesystem. This requires about 30 minutes of downtime.

All four of these tasks are pretty difficult so let's work on this incrementally.

This is a very big change, so I propose to break this down into smaller parts: 1. #138 Package everything installed on the current server that's not in nixpkgs. 2. Create a NixOS container to duplicate our current Arch Linux setup. 3. Migrate data from Arch to the NixOS container. 4. Boot into a live USB and move the container filesystem to the root filesystem. This requires about 30 minutes of downtime. All four of these tasks are pretty difficult so let's work on this incrementally.

a added the

question

enhancement

help wanted

labels 2022-05-20 17:41:06 +00:00

codedotjs was assigned by a

2022-05-20 17:41:06 +00:00

a added this to the (deleted) project 2022-05-20 17:41:06 +00:00

a added a new dependency 2022-05-20 17:41:24 +00:00

#138 Nix packaging

a added this to the v9.0 milestone 2022-05-24 19:33:01 +00:00

a commented

2022-06-26 02:00:44 +00:00

Author

Owner

I finished creating a NixOS container #160, now time to make our configuration.nix! @codedotjs maybe let's start by setting up nginx, PostgreSQL, and Gitea in the container? The container uses the host's networking so we'll have to run them on different ports.

I finished creating a NixOS container #160, now time to make our [configuration.nix](https://git.exozy.me/exozyme/nixos-config)! @codedotjs maybe let's start by setting up nginx, PostgreSQL, and Gitea in the container? The container uses the host's networking so we'll have to run them on different ports.

a commented

2022-07-19 15:51:55 +00:00

Author

Owner

Another issue we have to worry about is that we need some way to clean out the Nix store periodically, since it usually grows to be very big after a few months.

Also, one of the main benefits of switching to NixOS is that users will be able to choose their WM or DE without having them all installed globally. However, it looks like you can only easily install a DE in the global configuration.nix with something like services.xserver.desktopManager.gnome.enable = true;, but there's no equivalent of this in Home Manager. I guess a workaround could be to install the GNOME packages manually. (I'm not 100% sure about this, but at least according to https://search.nixos.org, Nix has "package sets", so maybe you could install the GNOME package set)

Another issue we have to worry about is that we need some way to clean out the Nix store periodically, since it usually grows to be very big after a few months. Also, one of the main benefits of switching to NixOS is that users will be able to choose their WM or DE without having them all installed globally. However, it looks like you can only easily install a DE in the global `configuration.nix` with something like `services.xserver.desktopManager.gnome.enable = true;`, but there's no equivalent of this in Home Manager. I guess a workaround could be to install the GNOME packages manually. (I'm not 100% sure about this, but at least according to https://search.nixos.org, Nix has "package sets", so maybe you could install the GNOME package set)

thehedgehog commented

2022-07-27 11:42:00 +00:00

Another issue we have to worry about is that we need some way to clean out the Nix store periodically, since it usually grows to be very big after a few months.

This can be solved with the nix.gc options(see here

Also, one of the main benefits of switching to NixOS is that users will be able to choose their WM or DE without having them all installed globally. However, it looks like you can only easily install a DE in the global configuration.nix with something like services.xserver.desktopManager.gnome.enable = true;, but there's no equivalent of this in Home Manager. I guess a workaround could be to install the GNOME packages manually. (I'm not 100% sure about this, but at least according to https://search.nixos.org, Nix has "package sets", so maybe you could install the GNOME package set)

I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space.

> Another issue we have to worry about is that we need some way to clean out the Nix store periodically, since it usually grows to be very big after a few months. This can be solved with the `nix.gc` options(see [here](https://search.nixos.org/options?query=nix.gc) > Also, one of the main benefits of switching to NixOS is that users will be able to choose their WM or DE without having them all installed globally. However, it looks like you can only easily install a DE in the global `configuration.nix` with something like `services.xserver.desktopManager.gnome.enable = true;`, but there's no equivalent of this in Home Manager. I guess a workaround could be to install the GNOME packages manually. (I'm not 100% sure about this, but at least according to https://search.nixos.org, Nix has "package sets", so maybe you could install the GNOME package set) I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space.

👍 1

a commented

2022-07-27 15:11:12 +00:00

Author

Owner

This can be solved with the nix.gc options(see here

Sounds good, that'll work.

I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space.

If all the DEs are installed globally, will all their packages and apps be available to users? I actually don't want this, since then you'll have 5 different file managers and terminals installed.

> This can be solved with the `nix.gc` options(see [here](https://search.nixos.org/options?query=nix.gc) Sounds good, that'll work. > I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space. If all the DEs are installed globally, will all their packages and apps be available to users? I actually don't want this, since then you'll have 5 different file managers and terminals installed.

thehedgehog commented

2022-07-27 16:16:13 +00:00

If all the DEs are installed globally, will all their packages and apps be available to users? I actually don't want this, since then you'll have 5 different file managers and terminals installed.

While that is the case by default, for all the major DEs(Plasma, GNOME, Mate, LxQT, Pantheon and Cinnamon), they have an excludePackages option[1] that you can use to disable default applications in that DE. It is annoying, but it can be done. Also, currently, the excludePackages option for Plasma is only in the unstable branch of nixpkgs.

1: https://search.nixos.org/options?query=excludePackages

> If all the DEs are installed globally, will all their packages and apps be available to users? I actually don't want this, since then you'll have 5 different file managers and terminals installed. While that is the case by default, for all the major DEs(Plasma, GNOME, Mate, LxQT, Pantheon and Cinnamon), they have an excludePackages option[1] that you can use to disable default applications in that DE. It is annoying, but it can be done. Also, currently, the excludePackages option for Plasma is only in the unstable branch of nixpkgs. 1: https://search.nixos.org/options?query=excludePackages

thehedgehog commented

2022-07-27 16:22:55 +00:00

Obviously, you can't remove required packages, but many apps such as okular, konsole, and the like can be removed. There isn't a central location for all of this information, so you will have to dig through the code to see whether a package is required or not. For instance, for Plasma, you can only remove packages defined in this array or this array. However, all of these optionalPackages are things like terminal emulators and file managers, so this should be what you're looking for.

Obviously, you can't remove required packages, but many apps such as okular, konsole, and the like can be removed. There isn't a central location for all of this information, so you will have to dig through the code to see whether a package is required or not. For instance, for Plasma, you can only remove packages defined in [this array](https://github.com/NixOS/nixpkgs/blob/8f73de28e63988da02426ebb17209e3ae07f103b/nixos/modules/services/x11/desktop-managers/plasma5.nix#L353) or [this array](https://github.com/NixOS/nixpkgs/blob/8f73de28e63988da02426ebb17209e3ae07f103b/nixos/modules/services/x11/desktop-managers/plasma5.nix#L489). However, all of these optionalPackages are things like terminal emulators and file managers, so this should be what you're looking for.

thehedgehog commented

2022-07-27 16:24:08 +00:00

Also, there are still many KDE apps missing from nixpkgs(mainly educational and rarely-used ones, along with many games). If you depend on one and it's unchecked on this list, let me know and I'll package it. However, all core packages and things like okular and konsole should be packaged.

Also, there are still many KDE apps missing from nixpkgs(mainly educational and rarely-used ones, along with many games). If you depend on one and it's unchecked on [this list](https://github.com/NixOS/nixpkgs/issues/62168), let me know and I'll package it. However, all core packages and things like okular and konsole should be packaged.

a commented

2022-07-27 19:53:01 +00:00

Author

Owner

While that is the case by default, for all the major DEs(Plasma, GNOME, Mate, LxQT, Pantheon and Cinnamon), they have an excludePackages option[1] that you can use to disable default applications in that DE. It is annoying, but it can be done. Also, currently, the excludePackages option for Plasma is only in the unstable branch of nixpkgs.

Cool, that should work then.

I just thought of another interesting idea for users that would like to install a different DE: we can set up a GPU-accelerated container for each user with systemd-nspawn or podman, and then configure xrdp to start the X server inside the container instead of on the host. This doesn't even require NixOS so we could even implement this idea now.

Also, there are still many KDE apps missing from nixpkgs(mainly educational and rarely-used ones, along with many games). If you depend on one and it's unchecked on this list, let me know and I'll package it. However, all core packages and things like okular and konsole should be packaged.

We have a list of packages that we need to package here. I'm currently too busy to work on it, but you could get in touch with @codedotjs to finish packaging the rest.

> While that is the case by default, for all the major DEs(Plasma, GNOME, Mate, LxQT, Pantheon and Cinnamon), they have an excludePackages option[1] that you can use to disable default applications in that DE. It is annoying, but it can be done. Also, currently, the excludePackages option for Plasma is only in the unstable branch of nixpkgs. Cool, that should work then. I just thought of another interesting idea for users that would like to install a different DE: we can set up a GPU-accelerated container for each user with systemd-nspawn or podman, and then configure xrdp to start the X server *inside the container* instead of on the host. This doesn't even require NixOS so we could even implement this idea now. > Also, there are still many KDE apps missing from nixpkgs(mainly educational and rarely-used ones, along with many games). If you depend on one and it's unchecked on [this list](https://github.com/NixOS/nixpkgs/issues/62168), let me know and I'll package it. However, all core packages and things like okular and konsole should be packaged. We have a list of packages that we need to package [here](https://git.exozy.me/exozyme/packages/issues). I'm currently too busy to work on it, but you could get in touch with @codedotjs to finish packaging the rest.

a commented

2022-07-27 20:42:20 +00:00

Author

Owner

I just thought of another interesting idea for users that would like to install a different DE: we can set up a GPU-accelerated container for each user with systemd-nspawn or podman, and then configure xrdp to start the X server inside the container instead of on the host. This doesn't even require NixOS so we could even implement this idea now.

A bit offtopic, but one very easy way to do this (without any system configuration) is to create a new Toolbox container, install the WM/DE inside it, and then start it in a remote desktop session using ~/.xinitrc.

> I just thought of another interesting idea for users that would like to install a different DE: we can set up a GPU-accelerated container for each user with systemd-nspawn or podman, and then configure xrdp to start the X server *inside the container* instead of on the host. This doesn't even require NixOS so we could even implement this idea now. A bit offtopic, but one very easy way to do this (without any system configuration) is to create a new [Toolbox container](https://git.exozy.me/exozyme/exozyme/wiki/Toolbox), install the WM/DE inside it, and then start it in a remote desktop session using `~/.xinitrc`.

thehedgehog commented

2022-07-28 21:12:25 +00:00

I just thought of another interesting idea for users that would like to install a different DE: we can set up a GPU-accelerated container for each user with systemd-nspawn or podman, and then configure xrdp to start the X server inside the container instead of on the host. This doesn't even require NixOS so we could even implement this idea now.

A bit offtopic, but one very easy way to do this (without any system configuration) is to create a new Toolbox container, install the WM/DE inside it, and then start it in a remote desktop session using ~/.xinitrc.

Good idea!

Slight note: Toolbox is not currently packaged in Nix, however, an alternative, distrobox is packaged. However, they should be mostly interoperable. Also, it requires your default shell be set to bash, however we can obviously set the default shell to fish per-user.

> > I just thought of another interesting idea for users that would like to install a different DE: we can set up a GPU-accelerated container for each user with systemd-nspawn or podman, and then configure xrdp to start the X server *inside the container* instead of on the host. This doesn't even require NixOS so we could even implement this idea now. > > A bit offtopic, but one very easy way to do this (without any system configuration) is to create a new [Toolbox container](https://git.exozy.me/exozyme/exozyme/wiki/Toolbox), install the WM/DE inside it, and then start it in a remote desktop session using `~/.xinitrc`. Good idea! Slight note: Toolbox is not currently packaged in Nix, however, an alternative, [distrobox](https://github.com/89luca89/distrobox) is packaged. However, they should be mostly interoperable. Also, it requires your default shell be set to bash, however we can obviously set the default shell to fish per-user.

👍 1

a commented

2022-07-28 22:30:38 +00:00

Author

Owner

Slight note: Toolbox is not currently packaged in Nix, however, an alternative, distrobox is packaged. However, they should be mostly interoperable. Also, it requires your default shell be set to bash, however we can obviously set the default shell to fish per-user.

Distrobox looks like a much better alternative to Toolbox. Maybe we should uninstall Toolbox on the current server and replace it with Distrobox. In particular, the feature to "export" apps to the host sounds really nice. (Disclaimer: I did contribute a PR to Toolbox, but I don't care if we uninstall it)

I found a few more alternatives listed here, which makes me wonder why there are so many ways to do this... (but I guess fragementation is just everywhere in the free software world)

Edit: I'm not sure if Distrobox fixes the Authorization required, but no authorization protocol specified bug with running X11 apps (for Toolbox you have to run xauth add $DISPLAY . $(xauth list | head -n 1 | cut -d' ' -f5) first to fix it), so I'll have to investigate this further. It seems like Distrobox also requires running that command first, unfortunately. Why can't X11 authorization just work?

Edit 2: The xauth bug was fixed in #168.

> Slight note: Toolbox is not currently packaged in Nix, however, an alternative, [distrobox](https://github.com/89luca89/distrobox) is packaged. However, they should be mostly interoperable. Also, it requires your default shell be set to bash, however we can obviously set the default shell to fish per-user. Distrobox looks like a much better alternative to Toolbox. Maybe we should uninstall Toolbox on the current server and replace it with Distrobox. In particular, the feature to "export" apps to the host sounds really nice. (Disclaimer: I did contribute a PR to Toolbox, but I don't care if we uninstall it) I found a few more alternatives listed [here](https://github.com/containers/toolbox/issues/348#issuecomment-1002785754), which makes me wonder why there are so many ways to do this... (but I guess fragementation is just everywhere in the free software world) Edit: I'm not sure if Distrobox fixes the `Authorization required, but no authorization protocol specified` bug with running X11 apps (for Toolbox you have to run `xauth add $DISPLAY . $(xauth list | head -n 1 | cut -d' ' -f5)` first to fix it), so I'll have to investigate this further. It seems like Distrobox also requires running that command first, unfortunately. Why can't X11 authorization just work? Edit 2: The xauth bug was fixed in #168.

a referenced this issue

2022-07-29 01:36:10 +00:00

Toolbox or Distrobox? #169

a commented

2022-07-29 21:53:01 +00:00

Author

Owner

I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space.

Correct me if I'm wrong, but I think it should be possible for a user to install a DE using something like nix-env -iA nixos.xfce. I'm not sure how well this would work though, since xfce.nix seems to do a lot more than just install some packages.

> I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space. Correct me if I'm wrong, but I think it should be possible for a user to install a DE using something like `nix-env -iA nixos.xfce`. I'm not sure how well this would work though, since [xfce.nix](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/x11/desktop-managers/xfce.nix) seems to do a lot more than just install some packages.

thehedgehog commented

2022-07-30 11:20:59 +00:00

I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space.

Correct me if I'm wrong, but I think it should be possible for a user to install a DE using something like nix-env -iA nixos.xfce. I'm not sure how well this would work though, since xfce.nix seems to do a lot more than just install some packages.

So, you'd want to look at the default.nix for the xfce package set, as nix-env installs packages and does not use the modules system. Running nix-env -iA nixos.xfce is something that you could do, but it doesn't do things such as enabling the screensaver, adding packages that integrate with your current environment, enabling a default config, and more. That's why I'd recommend using the Nix modules for DEs and WMs. Also, nix-env is generally considered bad to use, as installing packages from there is not 100% reproducible, unless your channels are pinned to the same commit. Because of the way nix-env works, different users can have different versions of the same channel, and therefore your store will become much bigger as you have many more versions of different packages.

TL;DR: I'd suggest using the Nix modules and not using nix-env.

> > I don't believe that there currently is a way to install DEs or WMs per-user, however, installing them globally may be a better option, so that we can guarantee that there is only one version at a time in the nix store, to save space. > > Correct me if I'm wrong, but I think it should be possible for a user to install a DE using something like `nix-env -iA nixos.xfce`. I'm not sure how well this would work though, since [xfce.nix](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/x11/desktop-managers/xfce.nix) seems to do a lot more than just install some packages. So, you'd want to look at the [default.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/desktops/xfce/default.nix) for the `xfce` package set, as nix-env installs packages and does not use the modules system. Running `nix-env -iA nixos.xfce` is something that you could do, but it doesn't do things such as enabling the screensaver, adding packages that integrate with your current environment, enabling a default config, and more. That's why I'd recommend using the Nix modules for DEs and WMs. Also, `nix-env` is generally considered bad to use, as installing packages from there is not 100% reproducible, unless your channels are pinned to the same commit. Because of the way `nix-env` works, different users can have different versions of the same channel, and therefore your store will become much bigger as you have many more versions of different packages. TL;DR: I'd suggest using the Nix modules and not using `nix-env`.

a commented

2022-07-30 15:31:13 +00:00

Author

Owner

TL;DR: I'd suggest using the Nix modules and not using nix-env.

What about using Home Manager for this?

> TL;DR: I'd suggest using the Nix modules and not using `nix-env`. What about using Home Manager for this?

a commented

2023-01-07 20:48:43 +00:00

Author

Owner

I propose that we close this issue, mainly because there isn't anyone actively working on this anymore, and we've found workarounds like Distrobox that provide alternatives to the advantages of NixOS. Here's a full list of reasons for my proposal.

No one has the time to put in the tremendous amount of effort needed to make this possible. This will require writing hundreds of lines of Nix and hours of troubleshooting and migrating data, with the possibility of data loss. In addition, I no longer have physical access to the server, so I'll need to contact someone that does in order to prepare the NixOS USB and stuff.
Migrations suck. I helped migrate the MIT Mastodon instance from a standard install on Ubuntu 18.04 to a brand new declarative installation using Ansible, and it was at least 15 hours of headaches. Setting up the new server took over 8 hours, and that was just for one service. We host a lot of stuff here at exozyme and getting it all to work will take way longer.
From a user's perspective, exozyme with Arch right now isn't much different than NixOS. You can use nix-env to install packages and set some environment variables (see the wiki) to get desktop integration. You can even use Distrobox to get your own custom container with root, and you could use NixOS for that container.
Arch Linux maintenance isn't that much work. Sure, things break sometimes, but not often, and I'm very familiar with Arch, the makepkg system, and how to troubleshoot and fix things. NixOS is very different (for instance, not following the FHS), so I and potentially other sysadmins would have to relearn a lot.

In summary, migrating to NixOS sounded like a very appealing idea a few months ago, but now, it just seems like an unnecessary headache I'd like to avoid. If anyone wants to take charge with the migration, I'll keep this issue open. Otherwise, we'll just have to stay on Arch.

I propose that we close this issue, mainly because there isn't anyone actively working on this anymore, and we've found workarounds like Distrobox that provide alternatives to the advantages of NixOS. Here's a full list of reasons for my proposal. 1. No one has the time to put in the tremendous amount of effort needed to make this possible. This will require writing hundreds of lines of Nix and hours of troubleshooting and migrating data, with the possibility of data loss. In addition, I no longer have physical access to the server, so I'll need to contact someone that does in order to prepare the NixOS USB and stuff. 2. Migrations suck. I helped migrate the [MIT Mastodon instance](https://mastodon.mit.edu) from a standard install on Ubuntu 18.04 to a brand new declarative installation using Ansible, and it was at least 15 hours of headaches. Setting up the new server took over 8 hours, and that was just for one service. We host a lot of stuff here at exozyme and getting it all to work will take way longer. 3. From a user's perspective, exozyme with Arch right now isn't much different than NixOS. You can use `nix-env` to install packages and set some environment variables (see the wiki) to get desktop integration. You can even use Distrobox to get your own custom container with root, and you could use NixOS for that container. 4. Arch Linux maintenance isn't that much work. Sure, things break sometimes, but not often, and I'm very familiar with Arch, the `makepkg` system, and how to troubleshoot and fix things. NixOS is very different (for instance, not following the FHS), so I and potentially other sysadmins would have to relearn a lot. In summary, migrating to NixOS sounded like a very appealing idea a few months ago, but now, it just seems like an unnecessary headache I'd like to avoid. If anyone wants to take charge with the migration, I'll keep this issue open. Otherwise, we'll just have to stay on Arch.

a closed this issue

2023-01-07 22:31:10 +00:00

Convert the server from Arch to NixOS #142