I've had a fun time following the buzz about Stable Diffusion (SD), but I've been
itching to get my hands dirty with it.
This weekend, I put in a lot of hours to get a flake.nix working for SD. It
was rough, but I'm happy now that I'm on the other side of it. I didn't manage
to get it 100% declarative, but almost. I've probably also committed 100 sins
in terms of what you shouldn't do when packaging stuff with Nix, but hey - it
works!
And that's it! But there's a big BUT: this will take a couple of hours (depends
on your HW) since you have to compile some packages, like OpenCV and PyTorch,
with CUDA support. It's probably possible to reduce some of the compilation
time, but I don't care much at this point since I've already got the compiled
result... Sorry!
Once the installation completes, you should download the model weights from
here.
It's absolutely bonkers how far we've come with image generation. In my
Master's thesis, 2019, we used different variations of CycleGAN to create
synthetic training data (didn't succeed). It could generate 256x256px images
of blurry zebras (with horses as input). Today we can render 4K images of
anything. Can't wait to follow the progress on video for the next couple of
years.
Thank you Nixpkgs maintainers. I really appreciate the work you do. (Whenever
I do a deep dive like this I am humbled by the amount of work to get
dependencies installed correctly)
I've now been using NixOS for one year. I use it to manage both my private PC and my laptop at work. Overall, it has been a great experience. I really appreciate the scale and control it brings to my digital life.
In May 2022, I stepped it up a notch by adding a third device running NixOS, namely a custom-built router! After a month of interspersed work I had successfully replaced my home network router.
In this blog post I'll go through why I embarked on this journey, how I did it, and lessons learned.
Note: I try to keep this blog post up-to-date with my current configuration. If you notice something that seems off, feel free to reach out.
I've had sporadic issues with my previous router (Asus N56U). It worked fine most of the time, but when it didn't it was bad:
network speeds dropping off a cliff for a few minutes
connectivity dying completely for a few seconds
For most activities this wasn't a deal breaker, but it was a huge pain not being able to take important meetings at home and be confident that I'd have a good connection. Routers should just work™ and this one didn't.
Earlier in the year, a friend was asking for shopping advice for a new router. We quickly agreed that regular consumer tests can't be relied on because they don't test network stability over time, but instead focus on measuring speed and latency. I couldn't give a recommendation, aside from referring to hardware that could be flashed with open source router software.
A few weeks or months after our conversation, I found out about PC Engines APU2. An x86 router with open source hardware that can be flashed with practically any Linux distribution. It seemed like a good solution to stability concerns: if you have complete control over your router, you can "just" fix it if something's not working reliably.
With NixOS in mind and a will to learn more about home networking, I was convinced that this would be a good choice. I ordered a unit from TekLager (not affiliated) with the following configuration:
Low power consumption at ~6W (actually turned out to be ~12W)
Lots of recommendations on Reddit and, above all, some people successfully running NixOS on APU2.
Hardware was assembled by the folks at TekLager, neatly packaged, and delivered within a few days. Then followed an intense, interspersed battle of configuration until I finally got it working at the end of May. As I'm writing this blog post, my home network is fully supported by this little beast!
There are a number of distributions that are designed to be run on routers. Why would I use NixOS instead of pfSense, DD-WRT, or OpenWRT?
If you are experienced with Nix(OS), it should come as no surprise to you that I want all my devices to use it. But if you're new to NixOS or haven't been sold on its benefits, I'd say that it boils down to three things:
Since I'm already using NixOS to configure my work & private PCs, the additional overhead of a new device is pretty small. This is mainly because I'm using a single Git repository to maintain all devices, so reusability is very high.
Package updates are easy, requiring only one or two commands. And updating to new releases is also relatively painless, just requiring a one-line change to bump the nixpkgs release. Furthermore, I will likely get these updates for free, since I'm already declaring them for my other devices.
The router can be configured exactly as I want it to be. It will be running a subset of the same software with the same configurations as I'm used to having on my PCs. It will have Neovim and tmux, a range of networking tools, and dotfiles, all of them configured like I'm used to - with practically zero overhead.
I'm still a Nix newbie, so this is a good project to grow that skillset. And in particular, by forcing myself to declaritively design my home network I'll have a good opportunity to learn more about networking!
In real time it took about a month to get everything working, but in total I think I've spent less than 25 hours. I did most work before I had set up ActivityWatch, so I don't have any reliable data to go on. Most things were very easy to set up, but WiFi took a long time to get right.
The most time consuming task in the project though has definitely been writing this blog post (not included in the 25h)!
Getting an initial wired connection was fairly straight forward, although I didn't add any features except being able to forward traffic and to accept SSH. With that of the way, I turned my attention to WiFi which was significantly harder to configure.
After a lot of struggling, I managed to get hostapd launching without errors, but to my surprise I was only getting ~10Mbit/s. Much lower than the benchmark done by TekLager at 200Mbit/s.
Since TekLager had achieved 200Mbit/s on OpenWRT, I thought it would be a good test to see if I could too. I flashed the board with OpenWRT instead and after a few minutes of basic setup in their web UI, I confirmed the ~200Mbit/s number. Evidently, the low speeds I had on NixOS was not hardware related.
K900 on the NixOS Matrix channel informed me that OpenWRT has a lot of patches on top of hostapd, as well as the wireless driver, which could be the differing factor.
Determined to fix this, I set out to find out how I could use the same patches that were used in OpenWRT while still using NixOS.
May 11, 14: Learned how to override Nix package attributes#
This took quite a while to wrap my head around. As they say, Nix is super easy 95% of the time, but in the remaining 5% it's terribly difficult to control.
Having learnt how overlays work, I successfully managed to apply OpenWRT's patches on hostapd. Some of them didn't apply cleanly, such as the ones bringing in support for dbus which is an IPC software used in OpenWRT, so I ignored those.
With the patched version of hostapd running, I was then able to take the configuration that OpenWRT set up and reduce it until there were no superfluous or erroneous options left. And taking new measurements with iperf, I was able to get the same speeds as on OpenWRT! However, as explained in the entry for June 12 below, the patches were not what made the network speeds go up.
More details on how I added the patches can be found in this section.
May 14: WiFi network up and running, but DHCP wasn't working.#
Looking at systemd logs on both the router and my PC, I could see that the WiFi protocol was working as intended. Logs looked something like this:
workstation kernel: wlp39s0: authenticate with 04:f0:21:ac:39:faworkstation kernel: wlp39s0: send auth to 04:f0:21:ac:39:fa (try 1/3)workstation kernel: wlp39s0: authenticatedworkstation kernel: wlp39s0: associate with 04:f0:21:ac:39:fa (try 1/3)workstation kernel: wlp39s0: RX AssocResp from 04:f0:21:ac:39:fa (capab=0x411 status=0 aid=3)workstation kernel: wlp39s0: associated
But after being authenticated, clients would not receive an IP address. I didn't find out why and other things in life took precedent, so the project got benched for a while.
Two weeks later, I took the project up again and found the issue immediately. The firewall was blocking clients from negotiating an IP over DHCP. Fixing that was easy, and from this point on it was mostly smooth sailing.
After a lot of system rebuilds and experimentation, I finally got all vital features working. This was the day when I replaced my old router. Since then, the APU2 has been powering my home network.
Since I have a few devices that I want to be able to SSH to or mount files from/to, it makes things much easier if I can guarantee that they'll have the same IP. This was a simple one-line change per device to DnsMasq's config.
I've been using Grafana at work a lot for monitoring & alerts, and it seemed like a great fit to monitor my home network health. Xe has an excellent blog post from 2020 on how to set it up on NixOS, and getting the bare minimum up and running is ridiculously simple:
Enables Grafana to run as a service and spins up its Web UI at 0.0.0.0:8888 (accessible to any network interface, i.e. any connected client as long as it's allowed by the firewall).
Enables Prometheus to collect Systemd metrics.
Makes Prometheus available as a data source running on 127.0.0.1:9991 (only accessible on the loopback interface, i.e. only by the router itself).
Note that this opens up Grafana to the public as well, since traffic from enp1s0 is also able to reach 0.0.0.0. However, it's not a problem because traffic to port 8888 from enp1s0 will be blocked by the firewall.
June 5: Revert from 5GHz WiFi to 2.4Ghz, since our robot vacuum cleaner didn't support 5GHz.#
When trying to clean our apartment, we found out that our RoboRock S5 Max didn't support 5GHz WiFi. Since my NIC can only support 2.4 or 5GHz at any given time, I therefore had to regress to 2.4Ghz.
This actually turned out to be a good thing though, since I didn't need the extra speed that 5GHz offers and the range with 2.4GHz is much better.
June 12: Publish this blog post & verify hostapd patches#
To really find out what made WiFi speeds jump from 10Mbit/s to 200, I removed the OpenWRT patches and rebuilt the system to see if that was the differing factor. Using iperf, I found no significant difference between patched and non-patched builds. This means that it was not the patches from OpenWRT that made the WiFi go faster, but rather config changes I made to Hostapd at the same time. Exactly what those changes were, I don't know - and I don't have the interest to pursue that rabbit hole.
I had heard about WireGuard from a colleague a few years back, but I had no real conceptions of how it worked besides that it was some kind of improvement over OpenVPN.
I watched this talk by the creator of WireGuard and was convinced. Basically it's a VPN based on asymmetric cryptography (public/private keys). Compared to other VPNs it's ridiculously easy to set up. With it I can access my photos and home media from anywhere, even on my phone.
As a prerequisite step I configured my domain vpn.skogsbrus.xyz and the router to use dynamic DNS. This way vpn.skogsbrus.xyz will always resolve to my router, regardless of its IP address.
With this commit there were a couple of interesting changes:
Add WireGuard, running on all my managed machines.
My servers are given static IPs and hostnames with DnsMasq
DnsMasq now resolves the foobar.home domain to the local IP of the host foobar
With this all of my devices use WireGuard to connect to my home network. They use my router as their primary DNS server and are thus able to resolve the .home domains configured by DnsMasq.
In my first configuration I had a rather simplistic firewall: trusted interfaces and untrusted interfaces. I did not distinguish between outbound and inbound connections for the untrusted interfaces, meaning that the ports I opened for the guest network were also opened for inbound traffic from "the outside".
To fix this, I had to dig through a few Github issues to find the correct syntax. The solution was to specify allowed TCP/UDP ports per interface instead of globally. See more in the Firewall section below.
This was fixed in the same commit as the one introducing Wireguard. Commit here.
This was relatively unrelated to the router configuration itself, but was a refactoring of my whole NixOS repository. Instead of using things like devices' host names to configure conditional logic, every service or app is now a standalone module that can be enabled by devices.
For any given machine foobar, it is now defined as follows:
It is "declared" in flake.nix
Its system configuration is imported from hosts/foobar/system.nix
Its home configuration is imported from hosts/foobar/home.nix
Its hardware configuration is imported from hosts/foobar/hardware.nix
This is how it looks in real life. I've got two permanent units connected over wire: a NAS and a Raspberry Pi (the third wire connected to the switch is temporary). The rest of all my devices are connected over WiFi.
I've had a vague idea about most of these topics, but I haven't fully understood any of them. I still don't understand the details of many of these things, mainly because NixOS makes it so easy to configure :) But still, I have learned a lot!
IP forwarding should be enabled when you want the system to act as a router, that is transfer IP packets from one network to another.
This will allow packets to flow between different network interfaces. Unless enabled, these packets would be dropped by the kernel. Having this enabled is what allows us to communicate with the outside world.
NAT is an abbreviation of Network address translation. The router is the only device that is public facing. Any packets going to clients connected to the router will be addressed to the router. It is then the router's responsibility to forward the packet to correct client.
This means that any traffic coming from the enp1s0 interface to br0 or wguest will need to translate the destination IP with NAT. Likewise, for any traffic from br0 or wguest the source IP will be replaced with the router's external IP.
For more information, see Wikipedia. The image in their article about NAT is great to get an intuitive understanding of what's happening.
A (virtual) network interface (as shown in e.g. ip link) is an abstraction that can correspond directly to a network interface controller (NIC), but it doesn't have to. An example of a real use case of these abstracted interfaces, which we will see later on, is to separate a NIC into multiple virtual interfaces and then apply different traffic rules on those abstractions. So they use the same hardware, but their traffic is treated differently.
In order to enable communication between wired and wireless clients, we need a network bridge. A network bridge will allow traffic from two interfaces to act as if come from the same interface.
Note that we specify a subnet IP block for the bridge and not individually for the network interfaces as devices on those will be assigned IPs from the bridge's IP block.
DHCP is a client-server protocol, where clients are given an IP address by a server. We'll need to use the router as a DHCP server in order for our clients to receive an IP address. Clients need to have an IP address in order to receive data (how can the mailman deliver your mail if he doesn't know your address?).
This protocol is also used between the router and ISP, in order for the router to receive an external IP address.
A fun detail I discovered when I hadn't gotten DHCP to work yet, was that there is a reserved block of IP addresses used when no IP address is specified (by static assignment or DHCP): 169.254.0.0/16.
Which interfaces should the router act as a server on?#
In NixOS, the config option networking.useDHCP and networking.interfaces.x.useDHCP configure whether the device should request an IP as a DHCP client. This makes sense as most devices aren't routers, but for a router we want to make some special configurations here.
For the physical network interfaces, we configure DHCP like so:
networking.useDHCP = false; # Disable it by default networking.interfaces.enp1s0.useDHCP = true; # Request an IP from the ISP networking.interfaces.enp2s0.useDHCP = true; # Request an IP on the local wired interface networking.interfaces.wlp3s0.useDHCP = true; # Request an IP on the wireless interface
Now, enabling it for enp1s0 makes sense because we need to be assigned an external IP by the internet service provider. But why are we acting as a DHCP client on the local interfaces enp2s0 and wlp3s0?
If you want to configure the network based on the physical interfaces directly, you wouldn't. But since I've configured my network with a bridge and a virtual interface for the guest network, we enable DHCP on the physical interfaces so that they are assigned IPs from the virtual interfaces. [note: I'm speculating here, take it with a grain of salt and please LMK if it is wrong!]
A service responsible for DNS and DHCP. A very powerful little tool!
With this small config, we've got everything we need.
domain-needed makes sure that we ignore DNS requests that don't have a dot or domain specified. I'm not sure what threat this prevents, but why not!
The list of interface are matched in order with the list of dhcp-range, so the network bridge br0 is assigned the IP block 192.168.1.10-192.168.1.254 and wguest is assigned 192.168.2.10-192.168.2.254.
The 24h specified at the end of dhcp-range is the lease time in which to cache a client's IP address. So if a client reconnects within that time range, it will be assigned the same IP.
The dhcp-host entries specify static IP addresses for a couple of devices on my network
In addition to adding the patches, I made sure to use Hostapd 2.10 instead of 2.9 and edited the unpackPhase to include files added by OpenWRT that were referenced but not included in their patches.
However, as explained previously, I later discovered that the patches made no noticable difference in terms of network speed. So I removed them. I still use an overlay to use version 2.10 instead though.
NixOS has a firewall module with which you can neatly specify a lot of rules. I thought it would make sense to be strict about what clients on the guest network should be able to do, but not restrict traffic on my private network.
A simple configuration could look something like this:
Tha allowedTCPPorts and allowedUDPPorts attributes specify allowed ports for all interfaces. If we open up 2222 for TCP, this also also opens up the same port for the outwards facing interface.
To fix this, we shold specify allowed ports per interface instead of globally:
With the above firewall rules, clients on an untrusted interfaces will still be able to SSH to the router (if they have my key), even if port 22 isn't explicitly opened. This is because the ssh service in NixOS will open the firewall unless we explicitly tell it not to:
# Prevent sshd from opening port 22 (circumventing the firewall) services.openssh.openFirewall = false;
I don't have good enough monitoring in place yet to give a factual report about stability. When I do, I'll come back with data and my experience on how things have gone!
The switch is a cheap thing I picked up a few years ago, capped at 100Mbit/s. Until I replace it with a gigabit switch, this is the maximum speed with which my wired clients can communicate.
Theoretically, 2.4GHz should be able to support faster speeds than this. There might be a few improvements that can be made here with hostapd, but I will most likely not experiment with it since I don't really need a faster connection than this.
I think so! It's been a very fun and rewarding project. Most side projects I work on tend to get discontinued after a while, but since this has a huge impact on my daily life it has been very motivating to work on!
For me, yes. There are surely tons of cheaper and better routers out there. But with a router completely under my control, with a distribution that I can work efficiently with, I think I've got a much greater chance to guarantee network stability than what I would have with an off-the-shelf router.
It's also an investment in my skills, with a very good potential ROI as I'm still very early in my career.
I can't count how many times the serial connection saved me. Without it I'd have bricked the router by making it inaccessible.
I think it's pretty insane how relatively easy this was to set up. There's just so much power that comes included with NixOS.
Don't forget to delete old builds when you don't need them anymore! With
300(!) system generations, I was barely able to fit anything else on the
system. That's pretty cool though.
My journey with NixOS is now 6 months old. In this post, I'd like to think back on this time and reflect what my experience has been like, what I have learned, and what I still haven't learned.
For the curious, this is my current config: https://github.com/skogsbrus/os. Probably not an inspiration on how to structure your configs (yet!), but it might serve as a reference if you find anything there interesting.
The story began by installing NixOS on an old & unused laptop, to get a feel for it (Converting to NixOS). I think the installation and my first commit took a few hours to get done, which in my mind was not a big deal.
After a few weekends of tinkering, I was sold on the promises of the OS. My vacation was coming up, and so I made the switch on my work PC as well.
From there I've taken things slowly, adding things incrementally.
In July 2021, I switched to using flakes (an upcoming Nix feature) for my OS config. I was persuaded by community discussions, specifically regarding the pros about pinned dependencies vs channels, and a desperate desire for installing nightly neovim. It took some work getting my config flake:ified, but there hasn't been any mental overhead of using flakes since - so I do think it was worth it!
In hindsight it seems like a rather small change, but I remember struggling a lot with flake:ifying my system because I really didn't know what I was doing. I relied completely on documentation and other people's published configs, understanding the general gist of what I was doing but not the syntax.
This was a giddy experience, almost at the same level as my first fascination of the OS. Until just last week, I was using NixOS on only one system (my work PC).
With practically no effort (except for getting dual-boot to work, which meant switching my Windows installation from legacy boot to UEFI), I installed the same system on my home workstation as on my work PC. It really felt incredible to do in under an hour what has previously taken my weeks of "Oh right, I've forgotten to install X".
Not only are my dotfiles synced, but applications, system settings, and gnome settings (keyboard shortcuts, extensions, themes) are too. It's such a weight off my shoulders to only have to care about keeping one configuration up to date.
In my relatively simple workflows, NixOS has yet to stop me from getting work done. If anything, making dotfiles management simpler has actually helped me!
I largely think this is a great thing - the ecosystem is mature enough that I, as a user, barely have to know the language or internal details in order to configure my systems.
However, this "ease of use" (ironically the opposite of what NixOS is famous for) does hurt me whenever I want to do more advanced things like writing my own package or setting up a complicated dev environment. In these moments, using NixOS becomes relatively slow as I am still inexperienced with Nix (even after 6 months with NixOS as my main OS!) and have to constantly look at documentation, other people's code, or support from the community.
With that said - the added friction of having separate dev environments is a welcome change from my previous workflow, where my global system was polluted by various project dependencies.
I still haven't grokked the Nix language, but even so it's gone okay. I'm not at a level where I can write fluently, but the language is intuitive enough that I can "build legos" with all the snippets I find online.
I find that this still holds true today. I am far from an expert in the Nix ecosystem and still barely know the language, but even so I manage to be a productive user in a NixOS system.
Yes. It's a paradigm shift that I've enjoyed taking part in. I don't particularly love the Nix language or the general structure of how derivations are defined, but it's an investment. If I learn these things I get a powerful system that I can take with me for life - regardless of which machine I'm using.
And if the past 6 months are any indication, I don't even have to become an expert in order to yield the benefits!
Most of my configuration relies on other people's work of porting packages to nixpkgs; and of course all of the development that goes into the language, package manager, and the OS! Without all their efforts, I couldn't have set up my system as easily as I have. So a big thanks to the whole Nix(OS) community for that! I hope that I can contribute in the future.
In the beginning of June 2021, I started tinkering with NixOS on an old laptop (my beloved Samsung Ativbook 9+ from 2011). Many users on Hacker News had expressed that it was a paradigm shift and that the OS had completely rewired their understanding and wants of a modern OS. It had peaked my interest.
With a couple of afternoons' worth of tinkering, I managed to produce a configuration that had:
terminal utilities
dotfiles for vim, tmux, zsh
gtk theme
steam
custom keyboard bindings (US layout with an AltGr layer: ; - ö, ' - ä, [ - å)
It wasn't much, but having one place to configure my whole system felt really powerful, like dotfiles on steroids. Along with idea of direnv + nix packages for developer environments, it was enough to get me hooked.
Fast-forward a month, and I've just now wiped my work pc and installed NixOS on it. I've officially converted and will be spending my private and work time on a NixOS-based device.
I still haven't grokked the Nix language, but even so it's gone okay. I'm not at a level where I can write fluently, but the language is intuitive enough that I can "build legos" with all the snippets I find online.
My career in software is still in its infancy, but with the last year of remote work in a large team I have learned a big lesson: I suck at communicating effectively. I can communicate well, but I am prone to use too many words.
The primary goal of this website will therefore be to train myself in communicating effectively. Meanwhile, I hope to learn by teaching, spread knowledge, and provide guidance to those who follow in my footsteps.
Monk with rice hat begging on the street, neon light from shops, cyborgs walking past, cinematic
Russian tank on fire, simon stålenhag, animated, dystopia
Robot playing chess, simon stålenhag, animated, cinematic, sci-fi future
Breath of the wild, Link on a surfboard, simon stålenhag
Gandalf the gray, epic scenery, miyasaki 4k
Gandalf the gray, breath of the wild
Turning torso, cinematic, malmö harbor 4k
M.C. Escher mixed with Salvador Dali, impressionist
Renaissance painting of a security guard with a security camera on his head
Cyborg monk, solarpunk, sci-fi, triadic colors, neon light, cinematic, 4k, farm