I recently got around to fixing an old Ubuntu-based NAS that I set up years ago. It was initially set up before I was using configuration management tools like Ansible and had a number of issues that made it nearly refuse to boot. This called for a full reinstall and I thought I’d try to build it all with NixOS given my recent interest in it. I thought I’d use this opportunity to actually set up everything properly (with the caveat of being compatible with my old arrays).
Installing NixOS
ZFS All the Way Down
Since my old system was ZFS-based, I decided to go with ZFS for all my filesystems. I roughly followed the instructions in the NixOS wiki:
# Set this to your root partition
export ROOT_PART=/dev/disk/by-id/...
sudo cryptsetup luksFormat $ROOT_PART
sudo cryptsetup open --type luks $ROOT_PART crypt-root
sudo zpool create -O mountpoint=none nixos-rpool /dev/mapper/crypt-root
# Make a reservation since ZFS is copy on write and will explode
# if we try to delete files after running out of space
sudo zfs create -o refreservation=1G -o mountpoint=none nixos-rpool/reserve
sudo zfs create \
-o xattr=sa \
-o acltype=posixacl \
-o compression=zstd \
-o relatime=on \
nixos-rpool/nixos
sudo zfs create nixos-rpool/nixos/nix
sudo zfs create -o mountpoint=legacy nixos-rpool/nixos/root
Note that if you make a mistake in setting mountpoint=legacy, the ZFS dataset will automatically get mounted to /, shadow the installer’s nix store, and break the install, causing you to start over from the beginning!
Compression and Deduplication
I’m no expert in ZFS and was not sure if I should enable compression or deduplication for the root datasets. After a number of weeks with this setup (and using it as my main host for morph deployments), the compression on the root volume seems to be quite useful:
$ zfs get compressratio
NAME PROPERTY VALUE SOURCE
nixos-rpool compressratio 1.19x -
nixos-rpool/nixos compressratio 1.19x -
nixos-rpool/nixos/nix compressratio 2.12x -
nixos-rpool/nixos/root compressratio 1.14x -
nixos-rpool/reserve compressratio 1.00x -
Additionally, I have been using deduplication on the root volume. This has also stretched my root nVME drive a bit further:
$ zpool list -v nixos-rpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nixos-rpool 472G 160G 312G - - 17% 33% 1.24x ONLINE -
$ zpool status -D nixos-rpool
pool: nixos-rpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
nixos-rpool ONLINE 0 0 0
crypt-root ONLINE 0 0 0
errors: No known data errors
dedup: DDT entries 1877158, size 324B on disk, 181B in core
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 1.46M 154G 128G 128G 1.46M 154G 128G 128G
2 326K 30.0G 26.9G 26.9G 749K 70.7G 64.0G 64.0G
4 12.4K 494M 185M 185M 56.4K 2.32G 871M 871M
8 1.51K 53.6M 29.9M 29.9M 17.2K 572M 290M 290M
16 549 3.55M 2.37M 2.37M 9.19K 64.7M 44.6M 44.6M
32 43 1.07M 806K 806K 1.79K 37.1M 27.2M 27.2M
64 15 10.5K 8.50K 8.50K 1.28K 944K 760K 760K
128 2 1K 1K 1K 334 167K 167K 167K
256 4 5.50K 3.50K 3.50K 1.78K 2.44M 1.55M 1.55M
1K 1 512B 512B 512B 1.21K 621K 621K 621K
Total 1.79M 184G 155G 155G 2.28M 228G 193G 193G
Encryption
As seen in ZFS All the Way Down, I initially created LUKS volumes before creating the ZFS datasets. This is to prevent reading of the drive if anyone were to physically obtain my server’s drive.
I went for LUKS encryption over ZFS-native encryption because my old NAS was built using LUKS over ZFS (ZFS-native encryption didn’t exist back then).
Root
NixOS allows pretty easy setup for regular LUKS devices via boot.initrd.luks.devices, providing you want to walk over to your keyboard to unlock on boot:
boot.loader.grub.enableCryptodisk = true;
boot.loader.grub.zfsSupport = true;
boot.initrd.luks.forceLuksSupportInInitrd = true;
boot.initrd.luks.devices = {
crypt-root = {
device = "/dev/disk/by-id/...";
preLVM = true;
};
};
However, I didn’t want to go over to my server to unlock on every boot and wanted to be able to reboot while not physically at my apartment.
While I think a solution like pikvm would be a better solution here (lack of cryptographic holes and ability to boot into previous NixOS generations if something goes wrong), I followed a couple guides12 to set up an SSH server during the init phase where I can unlock the root volume.
After a lot of trial and error (and being saved by NixOS generations), I came up with this:
# Enable SSH in initrd so we can SSH in and unlock the volume
# Adapted from https://mth.st/blog/nixos-initrd-ssh/
boot.initrd.network = {
enable = true;
ssh = {
enable = true;
# Use a different port so we won't always have host key conflicts
port = 2222;
authorizedKeys = config.users.users.daniel.openssh.authorizedKeys.keys;
# Note that these will probably be unencrypted in our setup, but it's mostly fine
hostKeys = [
"/etc/secrets/initrd/ssh_host_rsa_key"
"/etc/secrets/initrd/ssh_host_ed25519_key"
];
};
# Set the shell profile to meet SSH connections with a decryption
# prompt that writes to /tmp/continue if successful.
postCommands =
let
disk = "/dev/disk/by-id/...";
in
''
echo 'cryptsetup open ${disk} crypt-root --type luks && echo > /tmp/continue && exit' >> /root/.profile
echo 'starting sshd...'
'';
};
# Even though the device is marked as neededForBoot, it doesn't seem to mount in time to decrypt
# the storage/media disks
# Adapted from https://nixos.wiki/wiki/Full_Disk_Encryption#Option_2:_Copy_Key_as_file_onto_a_vfat_usb_stick
boot.initrd.postDeviceCommands = pkgs.lib.mkBefore ''
echo 'waiting for root device to be opened...'
mkfifo /tmp/continue
cat /tmp/continue
mkdir -m 0755 ${keyMount} # This will go away after stage 1
mkdir -m 0755 /key-mount
sleep 2
echo 'mounting key volume'
mount -n -t ext4 -o ro `findfs UUID=953ef32c-e9cb-4049-bf33-56f9e2c84b55` /key-mount
cp /key-mount/etc/apollo.kf /key
cp /key-mount/etc/keyfile /key
umount /key-mount # This will make fsck work
rmdir /key-mount
'';
Basically, we enable an SSH server in the init process (thankfully, NixOS makes this easy with boot.initrd.network.ssh) and we create a FIFO at /tmp/continue that will pause boot until it is written to. By default, when an SSH session starts, the script
cryptsetup open ${disk} crypt-root --type luks && echo > /tmp/continue && exit runs and cryptsetup waits for passphrase input. Note that you can Ctrl-C out of this to get a shell, though it is extremely limited at this point.
Note that there is one cryptographic hole in this setup: initrd needs to be readable for the boot process and contains the host keys for the initrd SSH server (the keys under /etc/secrets/initrd). If someone is able to access your hardware, they could pretend to be your server and intercept your input of the encryption passphrase. You could move this somewhere else with a KVM, but an attacker could do a similar attack on a KVM if you don’t secure access to it.
Storage Arrays
My old arrays were encrypted at the disk level with LUKS and then added to a ZFS array. To support them, I needed a way to open LUKS devices and mount the corresponding ZFS arrays. I’m not sure if this is the most efficient way to do this, but I put together a few functions to handle this:
uuidToDevice = uuid: "/dev/disk/by-uuid/${uuid}";
# Note that the keys are located where they're seen during the init process,
# not where the filesystem is after boot has completed
# See boot.initrd.postDeviceCommands for how the key is handled
keyMount = "/key";
storageKey = "${keyMount}/storage";
storageUUIDs = [
# Main disks
...
];
storageVolumes = (map
(uuid: {
name = uuid;
value = {
device = uuidToDevice uuid;
preLVM = false;
keyFile = storageKey;
};
})
storageUUIDs);
Once those functions are defined, you can set
boot.initrd.luks.devices = builtins.listToAttrs storageVolumes;
and generalize the functions to other arrays to append to storageVolumes.
Sharing
For the whole “network attached” part of NAS, we’ll need some sort of sharing. In my old setup, this was a nightmarish manual management of /etc/samba/smb.conf that would often break from any change.
Luckily, NixOS gives easy access to the shares I needed via services.samba and services.nfs.
Your shares really depend on your use-case, but I use Samba for most sharing and NFS for my Proxmox servers. It’s outside the scope of this post, but it may be worth noting that NFS is insecure by default and the only real security option is Kerberos, where you effectively delegate security to the clients.
ZFS Maintenance
Keeping your pools in working order is pretty important if you want them to be reliable. Unfortunately, this is something that I haven’t spent a ton of time on. NixOS does offer services.zfs.autoScrub.* to automatically scrub pools, but you ideally want to be notified when something goes wrong.
The only way to handle this in NixOS 22.05 seems to be services.zfs.zed.*, which would work if I had an email stack set up, but I don’t. I currently have a horrible script to send status to Discord, but I’m looking for something better when I get a chance. I have ZFS stats exported via Prometheus with
services.prometheus = {
exporters = {
node = {
enable = true;
openFirewall = true;
enabledCollectors = [
"zfs"
];
};
};
};
which I thought would work when combined with Grafana’s alerting, but only low-level stats seem to be exported rather than high-level information like faulted disks.
mdlayher/zedhook also looks like a possible option, but I haven’t dug into it yet.
Backups
If you care about your data, you should have backups. Personally I use restic via services.restic.backups and handle secrets with agenix, but NixOS also supports other common tools such as borg and rsnapshot.
A robust backup strategy could take an entire post of its own, but you should be sure to test that your backups work and limit the ability for a given host to destroy its own backups in the case of malware.
Running Applications
A bunch of storage isn’t much if you don’t do something with that storage! Aside from using network shares on my desktops and servers, I also use it for storage and management of media, such as family photos and my music collection.
My family photos have always been a bit of a mess, but I wanted to get them into order with my NAS rework (in fact, I restored my NAS to access those photos). After looking around at a number of options, it seemed that PhotoPrism was a solid place to start; it would import all my photos into a database and organize the directory structure. It has a number of interesting features, but it also organizes my photos into a structure that I can migrate to some other application (or build my own!) if I decide to.
NixOS doesn’t have a module for PhotoPrism. I considered writing a module for it, but I decided to go a simpler route and just run a container (in the past, most of my NAS applications have been container based). I wrote my own module for this:
{ config, lib, ... }:
let
cfg = config.services.photoprism;
in
{
options.services.photoprism = {
uid = lib.mkOption {
type = lib.types.int;
default = 990;
description = "UID of protoprism user";
};
gid = lib.mkOption {
type = lib.types.int;
default = 990;
description = "GID of photoprism primary group";
};
extraGroups = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
description = "Additional groups to add to photoprism user";
};
};
config = {
users.groups.photoprism = {
gid = cfg.gid;
};
users.users.photoprism = {
isSystemUser = true;
group = "photoprism";
extraGroups = cfg.extraGroups;
uid = cfg.uid;
};
virtualisation.oci-containers = {
backend = "podman";
containers = {
photoprism = {
image = "photoprism/photoprism:latest";
ports = [ "127.0.0.1:2342:2342" ];
user = "990:990";
environment = {
PHOTOPRISM_ADMIN_PASSWORD = "supersecure";
PHOTOPRISM_HTTP_COMPRESSION = "gzip";
PHOTOPRISM_PUBLIC = "false";
PHOTOPRISM_READONLY = "false";
PHOTOPRISM_DISABLE_CHOWN = "true";
PHOTOPRISM_DISABLE_WEBDAV = "true";
PHOTOPRISM_DATABASE_DRIVER = "sqlite";
PHOTOPRISM_FFMPEG_ENCODER = "intel";
};
volumes = [
"/dev/dri:/dev/dri"
"/mnt/media/photos-original:/photoprism/originals"
"/mnt/media/photos-import:/photoprism/import"
"/data/photoprism:/photoprism/storage"
];
};
};
};
};
}
and used it in configuration.nix with
imports = [
./photoprism.nix
];
services.photoprism.extraGroups = [ "your-groups-here" ];
End Thoughts
This was quite a journey and there are still some open questions on properly maintaining it in the future.
I think it would be neat to build up a reusable NAS configuration and make it public in a way that it can just be imported and enabled with something like
services.nas = {
enable = true;
# Shares to configure with sensible defaults for Samba/NFS
shares = {
photos = {
path = "/data/photos";
nfs.enable = false;
};
isos = {
path = "/data/isos";
# By default, NFS is enables and the path is
# /export/${share}
nfs.readonly = true;
};
}
};
but that would require a fair bit of refactoring from my system-specific configuration. I may also just open source my specific configurations if I have time to extract out any information that I don’t want public.