diff --git a/src/content/blogs/nixos-the-perfect-distro-for-a-production-server.mdx b/src/content/blogs/nixos-the-perfect-distro-for-a-production-server.mdx new file mode 100644 index 0000000..cd5c491 --- /dev/null +++ b/src/content/blogs/nixos-the-perfect-distro-for-a-production-server.mdx @@ -0,0 +1,287 @@ +--- +title: "NixOS: The perfect distro for a production server" +description: My experience using NixOS to manage my production VPS +date: 2025-01-01 +--- + +NixOS is a linux distribution praised for it's functional-style declarative +configuration that allows you to manage the entire system through the Nix +programming language, making your sytem reproducible and more reliable. + +This is in contrast to other linux distros that use a more imperative approach +of managing the system by running commands to install packages, create +users, etc. + +## NixOS as a daily-driver operating system + +About half a year ago I started using Windows subsystem for linux which enabled +me to run a linux distro in the terminal, 6 weeks ago I finally decided to make +the jump and switch from windows to a full linux desktop experience. + +I wanted a linux distro with a large set of latest-version packages but still +stable without risk of getting my system broken (sorry Arch). + +I was considering going with Tumbleweed, a rolling release distro +by openSUSE which I was already familiar with because I was using it in WSL. + +But then I remembered Nix, this weird thing that is an operating system, +programming language, and a package manager. + +Nixpkgs is the the package repository with the most fresh packages (even more than AUR), +and it has a really cool benefit of being able to install all my packages +through a config file, similar to NVIM with lazy.nvim or rust with cargo. + +Initially I was considering using something like Debian with nixpkgs as I heard +that was possible, but some people warned me that it's a bad idea to not use +the native package manager, so I ended up **going with NixOS as my first linux +distro on desktop**. + +I followed a [great video about installing NixOS](https://www.youtube.com/watch?v=a67Sv4Mbxmc) by Vimjoyer +and was able to get it up and running. +Although I experienced some issues in the first few days, that was expected +and I was able to solve all of them. + +One trap to be aware of is the nix community is sometimes too enthusiastic and +will try to convince you to do things "the nix way" like rewriting your +perfectly fine existing configs in nix or moving from stow to home manager. + +In my opinion, I don't want to waste so much time configuring everything, +I am OK with only configuring system settings and packages while having things +like other dotfiles or scripts be handled the "normal linux way". + +## NixOS as a production server distro + +A few days ago I got my first VPS, a RS 1000 from netcup (use code +`36nc17355743780` for a $5 discount), +my goal with it was to host my website along with some other self-hosted apps +including my own server for a terminal-based social media I am working on +(which I will make a blog post about in the future). + +Infrastructure as code (IaC) is the practice of managing servers through +configuration files rather than manual instructions (such as running commands), +this has several benefits such as being able to use version control and ensuring +a more reproducible server that can be easily deployed and updated using these +files. + +This is where NixOS comes in and really shines, all these "nix ways" of +configuring stuff as a desktop OS seems overkill and time consuming but for a +server that's exactly the point, to be able to reproduce the entire system +from only a bunch of nix files. + +## Configuring a server with NixOS + +After struggling to install NixOS, I discovered [NixOS Anywhere](https://github.com/nix-community/nixos-anywhere) +which made it very easy to install remotely through SSH. + +First thing was setting up proper SSH authentication and security. + +Nix makes it really simple to do, just add these 2 lines to enable ssh +but disable the insecure password authentication + +```nix +services.openssh.enable = true; +services.openssh.settings.PasswordAuthentication = false; +``` + +And to add a public key to connect with to the root user + +```nix +users.users.root.openssh.authorizedKeys.keys = [ + "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO7P9K9D5RkBk+JCRRS6AtHuTAc6cRpXfRfRMg/Kyren" +]; +``` + +If you would like to know how I generated a key that ends with `/Kyren` read +my first blog post - [The search for the perfect ssh key](http://localhost:4321/blogs/the-search-for-the-perfect-ssh-key) + +### Managing secrets with sops-nix + +Next, before we can get into configuring a webserver, we need a good way +to store secrets for things like SSL certifications, [sops-nix](https://github.com/Mic92/sops-nix) is a nix module +that integrates the [sops CLI tool](https://github.com/getsops/sops) into nix. + +Rather than having to set up 20 different secrets for each thing, it allows +us only setup 1 secret that is stored on the server and on the admin's +system, then, any additional secret can be defined in a file that gets +encrypted automatically by sops which can safely be committed into version control. + +At runtime sops will write all these files to a directory which can then be +accessed by anything that needs access to that secret, sops also makes it easy +to let specific users or groups access to a specific secret. + +### Setting up automatic updates + +We have added sops-nix, but how can we get it into the server? + +The simple approach would be to ssh as root into the server, git clone the +repository with the config and then run + +```sh +nixos-rebuild switch --flake .#default +``` + +_assuming you are inside the git directory where your `flake.nix` is and your +`nixosConfigurations` is called `default`_ + +While that certainly works, a better approach would be to listen for changes +to the git repository and automatically pull and rebuild the system. + +```nix +system.autoUpgrade = { + enable = true; + flake = "github:kyren223/server#default"; + dates = "minutely"; # Poll interval + flags = [ + "--no-write-lock-file" # Prevent flake.lock from upgrading + "--option" "tarball-ttl" "0" # Required for polling below 1h + ]; +}; +``` + +This will automatically run every minute to fetch the github repository +and rebuild the system. + +But wait, what if the repository is private? that's where sops-nix comes in! + +```nix +sops.secrets.github-access-token = { }; +nix.extraOptions = "!include /run/secrets/github-access-token"; +``` + +The first line defines a new `github-access-token` secret, and the second line +makes sure nix is aware of it so it can use it when trying to fetch the repo. + +Of course, make sure to define the secret in your `secrets.yml` file, +The key would be `gituhb-access-token` and the value would be a github PAT +token. +It's recommended to only give the token read-only access to the repo's contents. + +## Serving a static website + +Now that we have a way to manage our secrets, and continious deployment of our +nix configuration, we can finally start working on a website. + +The plan is to host multiple websites in the future so I'll be using [Nginx](https://nginx.org/en/) +as a reverse proxy. + +```nix +services.nginx.enable = true; +services.nginx.virtualHosts."kyren.codes" = { + useACMEHost = "kyren.codes"; + forceSSL = true; + locations."/" = { + index = "index.html"; + root = "/srv/website"; + }; +}; +``` + +This enables nginx, which installs it on our system then defines a new virtual +host with the domain `kyren.codes`, later if we were to host other sites, the +virtualHost can be something like `git.kyren.codes`. + +The `locations."/"` sets the base url of our website to `/srv/website` which is +where I have my website's static files, and nginx will serve `index.html` as +the default page when you go to that domain, later I will explain how to setup +continious deployment to automatically update this directory on push to master. + +The `useACMEHost` and `forceSSL` makes sure the server uses https, you'd need +to get a SSL certificate from [Let's Encrypt](https://letsencrypt.org/), once +again, Nix makes it really simple to do + +```nix +# Define the token using nix, make sure the "acme" user can access it +sops.secrets.cloudflare-dns-api-token = { mode = "0440"; owner = "acme"; }; + +security.acme.acceptTerms = true; +security.acme.defaults.email = "kyren223@proton.me"; # Your email + +security.acme.certs."kyren.codes" = { + domain = "kyren.codes"; + extraDomainNames = [ "*.kyren.codes" ]; + dnsProvider = "cloudflare"; + environmentFile = "${pkgs.writeText "cf-creds" '' + CF_DNS_API_TOKEN_FILE=${config.sops.secrets.cloudflare-dns-api-token.path} + ''}"; + webroot = null; +}; + +# Allow nginx to access acme certs +users.users.nginx.extraGroups = [ "acme" ]; +``` + +This defines a `kyren.codes` entry (which is what `useACMEHost` references) +I am using cloudflare as my DNS provider, but you can get a full list of +supported DNS providers at [this link](https://go-acme.github.io/lego/dns/), and +of course make sure to change the environment variables to match your provider. + +Now if we commit and push these changes and wait about a minute for the server +to pull the changes, we should have a working static site at `kyren.codes`. + +### Setting up continious deployment for the website + +The way we are going to do this on the server is create a new linux user and +add to it a public ssh key that will be used for deployments. + +It's recommended that the user is restricted as much as possible so in the case +of a hacker getting access to the key, they won't be able to get into other +services that are running on the server. + +We can create a user declaratively like this + +```nix +users.users.website = { + isNormalUser = true; + group = "users"; + openssh.authorizedKeys.keys = [ + # Allow an admin to manually SSH + "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO7P9K9D5RkBk+JCRRS6AtHuTAc6cRpXfRfRMg/Kyren" + + # The deployment public key + "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ1B/i/AQLYt6mrz0P/oUJItpvWXp7z0xHNzmcPdtwWd" + ]; +}; + +# Make sure the /srv/website always exists +# Give read and write permissions to the website user +# Give read permissions to the nginx group (so nginx can serve it) +systemd.tmpfiles.rules = [ + "d /srv/website 0750 website nginx" +]; +``` + +Now all you need to do is have a github action (or similar) that builds your +website then copies the build output into the server by SSH-ing into it. +You can take a look at [my github action](https://github.com/Kyren223/website/blob/master/.github/workflows/deploy.yml) +for my website to see an example of how you can do it. + +## Conclusions + +NixOS helps system admins and developers achieve infrastructure as code by +allowing them to declare the entire machine's configuration using the powerful +Nix programming language. + +This makes the state of production machines more consistent and reliable while +making it easy to reproduce across hundreds of machines. + +Compared with the traditional approach of a debian or ubuntu server, where +system admins would have to painfully manage the state of all machines ensuring +all the dependencies, firewalls, OS settings, etc are configured properly. + +Because of the difficulty, OCI containers such as docker became popular, but +they come with a cost, mainly performane due to virtualization. NixOS makes it +easy to configure programs to run on bare-metal making it a better, more +performant alternative to docker. + +**Thanks for reading! wishing you a year full of code, growth and new oppurtunities!** + +If you'd like to learn more about NixOS, here are some resources I found useful: + +- [NixOS in production](https://github.com/Gabriella439/nixos-in-production) +- [Vimjoyer's Youtube Channel](https://www.youtube.com/@vimjoyer/videos) + +Have questions or just want to chat? feel free to contact me on discord at +Kyren223 or email me at Kyren223@proton.me. + +Also shoutout to Aubrey for inspiration for the website design, you can visit +her site at [aubrey.rs](https://aubrey.rs/)