Posts on GA

Introducing Box

Sun, 04 Jan 2026 00:00:00 +0000

Package managers are my favorite part of Linux. Having a consistent, curated way to install and upgrade your system makes everything easier and more secure. That’s not to say there can’t still be security issues, but if you can’t generally trust the system packages you probably have bigger issues, and it beats downloading random binaries off the internet.

Unfortunately (and ironically) this system starts to fall apart when it comes to software development, especially at work where you may have less control over which tools and dependencies are used. While most Linux distributions include a wide array of programming tools, there are often weird requirements for specific versions or third-party tooling which may not be packaged. And although we can assume that the language tooling itself is secure if it comes from the package manager, the software we build and run with it may have vulnerable or even malicious dependencies that could wreak havoc on our system.

A common way to ameliorate these issues is to use containers for development. Although they aren’t true sandboxes, containers can add additional layers of protection by preventing binaries from accessing parts of your system they shouldn’t (i.e. most of your home directory), and since they don’t share a root filesystem with the host, they can run arbitrary versions of tools without interference. Programs like docker compose are often used to automate running a set of services for development (e.g. PostgreSQL), but using containers for running CLI tools (e.g. go build) has significantly worse user experience than installing and running the tools directly.

The generally accepted solution to this appears to be devcontainers, which installs and executes all the required tools for a project inside a semi-persistent development environment. This is a nice idea in theory, as new team members or contributors can just spin up a container with everything preconfigured, but since the development environments need to be codified, it works best when everyone uses the same tooling and workflows (even code editors - devcontainers are rather tightly coupled with Visual Studio Code). What I want is the opposite: rely on my system’s package manager as much as possible, and only run specific tools in containers based on security or versioning concerns.

Enter `box`

My solution is box: a CLI for transparently running your development tools in a container, with the goal of combining the security and versioning benefits of containers with the ease of running tools directly on the host. box (ab)uses your PATH to replace your development tools with symlinks to itself, then executes the corresponding command in a container. As a result, you can continue using the tool as if it were natively installed, including in scripts or in your editor (e.g. Neovim’s language server integration).

box is essentially a wrapper around a container runtime (currently Docker and Podman are supported). It is configured via environment variables, which map loosely to the runtime’s directives, e.g. BOX_ENVS="LOG_LEVEL=debug HTTP_PORT=8080". This configuration method was chosen primarily because it allows for nested configs and overrides out-of-the-box, since environment variables can be composed from other environment variables (e.g. export BOX_ENVS="$BOX_ENVS HTTP_HOST=0.0.0.0"). As an added benefit, it also integrates very well with tools that manage environment variables, such as direnv.

A Case `box` Study

At work I am writing a Rust-based firmware for an ESP32-S3 microcontroller. This device has an Xtensa microprocessor, which is not an officially supported Rust target, and as such requires a forked toolchain maintained by Espressif (the makers of the ESP platform). In this case, we can use box to run everything safely in a container.

First we’ll need to set our runtime; since this will probably be the same for all projects, I like to set this in my .profile:

export BOX_RUNTIME=podman

Next we’ll need an image from which to create a container. Espressif publishes an image with the toolchain already included, but we’ll need to add some extra tools:

FROM espressif/idf-rust:esp32s3_1.88.0.0

RUN rm ~/.cargo/bin/rust-analyzer && \
	curl -sL https://github.com/rust-lang/rust-analyzer/releases/download/2025-08-25/rust-analyzer-x86_64-unknown-linux-gnu.gz | gunzip -c - > ~/.cargo/bin/rust-analyzer && \
	chmod +x ~/.cargo/bin/rust-analyzer

RUN rm ~/.cargo/bin/cargo-espflash ~/.cargo/bin/espflash && \
	curl -sL https://github.com/esp-rs/espflash/releases/download/v4.3.0/cargo-espflash-x86_64-unknown-linux-gnu.zip | gunzip -c - >~/.cargo/bin/cargo-espflash && \
	curl -sL https://github.com/esp-rs/espflash/releases/download/v4.3.0/espflash-x86_64-unknown-linux-gnu.zip | gunzip -c - >~/.cargo/bin/espflash && \
	chmod +x ~/.cargo/bin/cargo-espflash ~/.cargo/bin/espflash

COPY ./entrypoint.sh /usr/local/bin/entrypoint.sh

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

Here we install rust-analyzer (Rust’s language server) and a newer version of espflash (for flashing and monitoring the microcontrollers). We need to replace the rust-analyzer that is already in the image since it is installed for the default toolchain and won’t work with the ESP toolchain.

The entrypoint mentioned sources some necessary setup scripts:

#!/bin/bash -eu

source "$HOME/.cargo/env"
source "$HOME/export-esp.sh"

exec "$@"

By default the container uses /bin/bash as the CMD, but we need these scripts to be sourced for the toolchain to work.

With our image ready, we can configure box. I use direnv to automatically set the necessary environment variables when I enter the directory, but they could also be manually set or sourced from a script. Here’s my .envrc:

export BOX_IMAGE="esp"

dotenv_if_exists
envs_from_dotenv="$(envs_from_dotenv)"
export BOX_ENVS="CARGO_HOME=$HOME/.cargo $envs_from_dotenv"

export BOX_VOLUMES="$HOME/.cargo/git:$HOME/.cargo/git $HOME/.cargo/registry:$HOME/.cargo/registry $PWD:$PWD"
export BOX_WORKDIR="$PWD"
export BOX_EXTRA_OPTS="--userns=keep-id --cap-drop=ALL --read-only"

serial_extra_opts='--group-add=keep-groups --device=/dev/ttyACM0'
export BOX_cargo_run_EXTRA_OPTS="$BOX_EXTRA_OPTS $serial_extra_opts"
export BOX_espflash_monitor_EXTRA_OPTS="$BOX_EXTRA_OPTS $serial_extra_opts"

export BOX_esptool_MATCH='esptool.py'
export BOX_esptool_IMAGE=espressif/idf:release-v5.3
export BOX_esptool_EXTRA_OPTS="$BOX_EXTRA_OPTS $serial_extra_opts"

There’s a lot going on here so we’ll break it down by section:

export BOX_IMAGE="esp"

First we set the image we’ll use to run our commands (in this case, the image I described above, built with the tag esp).

dotenv_if_exists
envs_from_dotenv="$(envs_from_dotenv)"
export BOX_ENVS="CARGO_HOME=$HOME/.cargo $envs_from_dotenv"

This section automatically loads a .env file if it exists, and specifies some environment variables that should be set in the container. The envs_from_dotenv is a custom direnv directive that will automatically export variables loaded from the .env into the container; this is a great way to include secrets without having them be part of the .envrc directly.

export BOX_VOLUMES="$HOME/.cargo/git:$HOME/.cargo/git $HOME/.cargo/registry:$HOME/.cargo/registry $PWD:$PWD"
export BOX_WORKDIR="$PWD"

Next we set some bind-mounts so the container has access to certain paths on the host. The first two entries mount our cargo cache so we don’t need to redownload dependencies every time we run a command (this is also where the CARGO_HOME environment variable comes into play). We also mount the current working directory to the same path in the container so cargo can access our project, and set the container’s working directory correspondingly. Mounting the current directory to the same path in the container also means we can run rust-analyzer inside the container and have jump-to-definition work correctly.

export BOX_EXTRA_OPTS="--userns=keep-id --cap-drop=ALL --read-only"

The BOX_EXTRA_OPTS directive is an escape hatch for specifying any arbitrary Docker/Podman args that aren’t represented by another directive. In my case I want to make sure that the user in the container has the same uid as my host user, so the file permissions match up. The capability drop and read-only flags are an extra bit of security hardening.

serial_extra_opts='--group-add=keep-groups --device=/dev/ttyACM0'
export BOX_cargo_run_EXTRA_OPTS="$BOX_EXTRA_OPTS $serial_extra_opts"
export BOX_espflash_monitor_EXTRA_OPTS="$BOX_EXTRA_OPTS $serial_extra_opts"

It’s very likely that not all tools will require the same resources; case in point, when we want to flash the firmware to an actual ESP, we’ll need access to the serial port from the host. While we could set these flags globally, they’re really only necessary for two commands: cargo run (which builds and flashes the code) and espflash monitor (which monitors the logs of the device over the serial port). box allows us to specify overrides for specific commands so we can tailor-make the container with exactly the resources it needs. By using BOX_cargo_run_EXTRA_OPTS, we will only set these additional parameters for the cargo run command, but not for other commands (e.g. cargo check).

export BOX_esptool_MATCH='esptool.py'
export BOX_esptool_IMAGE=espressif/idf:release-v5.3
export BOX_esptool_EXTRA_OPTS="$BOX_EXTRA_OPTS $serial_extra_opts"

The last block adds overrides for esptool, which is a Python-based tool that has some additional functionality currently missing from espflash. What’s neat here is that we can even override the image on a per-tool basis, so you don’t necessarily need to create one giant image with all the tooling. The BOX_esptool_MATCH directive also shows off how to create overrides that are not space-delimited: it maps the command esptool.py to the esptool override; any directives with the same override will be applied.

Finally, we need to create symlinks for the tools we want to run with box. We can use the box alias add <alias>... subcommand to automatically create them:

$ box alias add cargo espflash esptool rust-analyzer

This will create symlinks to box in the XDG_DATA_HOME/box directory (typically ~/.local/share/box on most Linux systems), but this can be customized using the BOX_PATH directive. This path also needs to be added to your PATH, e.g. in your .profile:

export PATH="$HOME/.local/share/box:$PATH"

With all that in place, we can now run cargo build normally in our terminal, but have it magically executed in a container, safely separated from our host system!

Try It Out!

You can find box’s code on Codeberg, along with additional documentation on all the available directives. If you use direnv you’ll also want to check out the included library script to make it even easier to integrate. Currently the easiest way to install box is with cargo install --git https://codeberg.org/galen/box-cli (or by cloning and cargo building manually).

Thanks for reading, and I look forward to hearing about your experience with box!

Using KubeVirt with Cilium's kube-proxy Replacement Mode

Sun, 30 Mar 2025 00:00:00 +0000

A few months back I set up a new Kubernetes cluster for self-hosting, and decided on Cilium for the CNI implementation. Partly to learn more about Cilium (it’s a project with a rather large scope), but also because it has some neat features, including cluster-wide firewalling and a UI called Hubble for visualizing network connections within the cluster.

So far it’s worked well, but I recently ran into an issue while experimenting with KubeVirt. Cilium is installed as a replacement to kube-proxy, which depends on a feature called “socket-LB” designed to transparently bypass Kubernetes service-level NAT. Unfortunately, this feature breaks the cluster-level networking of KubeVirt virtual machines, so my VMs were able to access external IPs but not services within the cluster (including internal DNS or the public interface of exposed services). After some searching I was able to fix it with the following steps:

Enable the hostNamespaceOnly option in Cilium; if you’re using the Helm chart, this can be enabled by setting the value:
```
socketLB:
  hostNamespaceOnly: true
```
If you’re installing by another method (e.g. Kustomize), it seems this flag sets the option bpf-lb-sock-hostns-only: "true" in the cilium-config ConfigMap.

This is rather hidden in their documentation about the kube-proxy replacement mode: Socket LoadBalancer Bypass in Pod Namespace. I originally found the option mentioned in a GitHub issue about networking not working with KubeVirt.
Restart the Cilium pods! I spent quite some time wondering why the networking was still broken after applying the fix; it turns out the config isn’t automatically reloaded until the Cilium pods are restarted. My assumption was that updating the Helm release would apply the change, but it seems to just update the ConfigMap. After Cilium is restarted you can verify that the config is updated by running kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose; under the KubeProxyReplacement Details header you’ll see Socket LB Coverage:
```
[...]
KubeProxyReplacement Details:
  Status:                 True
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Hostns-only
[...]
```

Since this feature is designed to bypass some additional networking steps, it’s likely that enabling the hostNamespaceOnly option comes at some efficiency cost; however my cluster and use-case are small enough that I don’t expect it to have a noticeable impact.

An Open Music Setup

Thu, 15 Aug 2019 00:00:00 +0000

A Bit of Background

I have been belligerently reducing the number of online services I use, opting instead for open-source software that I can self-host or run locally. One service I wanted to replace was Spotify, which meant finding a proper alternative for managing my music.

I have had issues with Spotify for a while¹, including unfair artist compensation and locking content behind DRM, which forces users to keep paying for a subscription or risk losing their music library. You are also restricted to using Spotify’s client or a client backed by their API, rather than the huge number of music players that can play normal audio files. And since you can’t download music from Spotify in a non-encrypted format, you can even lose access to music completely if it is removed from the service.

Switching to self-hosted / local software was the perfect opportunity to leave Spotify and start building an actual music library. Unfortunately, emulating the benefits of their service, such as cross-device synchronization and automatic library management, ended up being more trouble than I anticipated. In this post, I’ll detail some of my solutions to these problems as well as my current music setup, in case you too are looking for an alternative to online streaming services.

Content Management

The first difficulty with managing your own music library is ensuring that the audio files are properly labeled, sorted, and categorized. Fortunately, there are a number of tools available that make it easy to process your music.

Beets

My library management tool of choice is an excellent command-line app called beets, which lets you easily tag and organize music using the MusicBrainz database, an “open music encyclopedia” with metadata for millions of songs. Beets requires some manual input to ensure it correctly identifies your music, but the payoff is a well-structured library that can be searched, sorted, and updated by just about any reasonable media player.

Beets also supports extensions, which can add features such as automatically downloading and embedding art and lyrics. Check out the documentation for the complete list, as well as instructions on how to get started.

MusicBrainz Picard

In case a command-line app isn’t your cup of tea, there is also a tool published by MusicBrainz themselves called Picard. It features a GUI and, like beets, integrates directly with MusicBrainz to tag your music. I personally prefer beets’ interface and customizability, but Picard also works well.

Cross-device Synchronization

The main reason online services are popular is because they are convenient, so I needed find a setup that could offer similar features. Specifically, I wanted my music to be available on all of my devices, as well as automatically synchronize new music and playlists. Here are a couple of tools that fit the bill.

Funkwhale

My initial solution to this problem was to set up my own “streaming service” using Funkwhale: a federated music sharing software built on top of the decentralized social networking protocol ActivityPub (the same protocol powering Mastodon). In addition to a web interface, Funkwhale exposes a Subsonic API endpoint², which allows integration with a number of existing clients. I switched away from Funkwhale since it offered a lot more functionality and complexity than I needed, but I still think it’s a cool project with a welcoming community.

Syncthing

One of my favorite recently-discovered pieces of software is Syncthing: a peer-to-peer, decentralized file synchronization system. It allows you to register devices with one another, then share files and directories between them without any intermediary server. It sometimes takes a while to sync, but overall is a useful tool. I was already using it to sync phone backups to my server, and decided to try using it for music as well.

The setup was very straightforward; I “paired” my phone and laptop, then added my music directory as a shared folder between the two. Aside from a small hiccup where my backup software tried to cyclically back up the music directory and gobbled all the space on my phone, the entire process went smoothly and I was left with a synchronized music library on both my devices.

One nifty benefit of syncing the files directly between devices is that playlist management becomes very straightforward. Many music players support the m3u playlist format, which is basically just a list of audio file paths. One gotcha with this method is that the paths need to be the same on all devices, so using relative paths typically works better than absolute. I have my playlist files in the root of my music directory, and reference audio files with paths relative to that.

rsync

I feel I should at least give rsync an honorable mention, since I use it on an almost daily basis for transferring files between servers. I considered setting up a recurring script to periodically rsync music between my devices, but I wanted a “smarter” solution that only synchronized when there were new items or changes.

Additionally, it would be difficult to automatically rsync between a phone and laptop, since they would need to be on the same network. This could be solved by having a central music server that all devices sync against, but I prefer the reduced complexity of a decentralized setup.

Media Players

There are a lot of media players out there. In fact, it wouldn’t surprise me if there are more media players than any other category of software. The great thing about having such a variety of players is that you can choose the one that best fits your use-cases. Whether you prefer a command-line tool for running in a terminal, a web-based player that’s accessible from anywhere, or a native desktop application, there are countless great options.

In my opinion, client lock-in is one of the worst problems with using online streaming services, since it greatly restricts how you can interact with your music. You’ll typically get a single web interface, desktop application, phone app, and, if you’re lucky, a handful of third-party clients hacked together to work with the service’s API. The lock-in becomes especially egregious when the company behind the client randomly decides to introduce unasked-for changes (looking at you, Spotify), forcing you to choose to continue using an increasingly worse client, or leave the service and lose your music. Thankfully, we suffer none of these issues when using open file formats and clients.

On the Computer

These days, most of my computer use is in a terminal session, so having a good command-line music player is a must. To that end, I am very happy with ncmpcpp backed by MPD. The setup is super simple; just point MPD to your media files and it’s off to the races. You can even do neat stuff like stream from MPD to remote devices³. MPD supports a lot of different configurations, so check out their documentation to get a feel for how it can best work for you.

For my Phone

Unfortunately, phones aren’t as well suited to command-line interfaces, so I needed to find a usable Android app for listening to music on the go. Almost any music player available for Android can play standard-format audio files and read m3u playlists, so I set out testing a bunch of players to see which one suited me best.

A problem I quickly ran into is how Android handles media and playlists. In an attempt to prevent lots of repeated filesystem calls from locking up your phone, the Android system will automatically scan and index local audio, video, image and playlist files, and expose them to apps through an API. While the pros for the system being built in this way probably outweigh the cons⁴, it makes rolling your own media management difficult due to two large issues:

The only way for apps to access your media is through this API, so until new media is indexed, they don’t know it exists. There is also no reasonable way to force the system to index files, short of clearing the current index, restarting the phone, and waiting for the system to rebuild the index sometime thereafter⁵.
Apps don’t handle playlist changes very well. Either as part of the their own internal storage system, or through the interaction with the media API, playlist changes that are updated from an external source (i.e. Syncthing) don’t get picked up quickly (or ever) by the app, and changes to the playlists within the app need to be manually exported back to the filesystem, which often breaks the path references for other devices.

I spent a good deal of time researching workarounds for these issues when I had a thought: perhaps I could just mimic my desktop setup, and use MPD? As it turns out, running MPD as a local media server on my phone works great, especially since there are a number of quality open-source apps for streaming music from MPD. My current favorite is M.A.L.P., which offers a nice UI and seamless MPD integration (plus the bonus of being able to connect to and control remote instances, like your computer).

For the MPD daemon, I originally used Termux (an app that provides a terminal emulator and faux Linux environment) with the MPD package. This setup is usable, although I have since switched to MPD’s official Android implementation which I find to be more convenient.

Closing Thoughts

This post ended up being a lot longer than I anticipated, but hopefully it’s given you an idea of how you can set up your own media management system without relying on an online service. I set out to meet fairly specific goals which added some complexity, but parts of it could certainly be pared down or adapted to fulfill a different use-case.

There are a few kinks still left to sort out, including:

Having to store my entire library on all devices is inefficient, especially if I am only listening to a certain playlist at a time. For now, my library is small enough that it’s not an issue, but in the future I might need to start filtering which songs get synchronized.
The local-MPD phone instance is a bit of a hack, and M.A.L.P. has some quirks when connecting to a local server, including one related to volume control. I have an issue open to try and resolve this, so hopefully this won’t be a problem in the future!

Overall, I am super pleased with this setup, and enjoy the amount of control and customizability it has given me over my music. If you are also becoming disenchanted with Spotify or other online streaming services, then I encourage you to take it out for a spin! And if you come up with your own system or find a way to improve mine, I would love to hear about it.

These criticisms apply to most, if not all, online streaming platforms. ↩︎
Subsonic is another music server software that ended up going closed-source, although many successive projects borrow its API design. ↩︎
I did consider streaming music from a centralized MPD instance as an enabler for cross-device synchronization. Unfortunately, this isn’t an ideal option for listening to music on a phone, since MPD doesn’t directly support caching, and having to stream music live every time will quickly eat mobile data. ↩︎
It’s worth noting that mobile OS’s with proper file systems (like Sailfish OS) don’t suffer either of these issues, and basically Just Work™ as you’d expect them to. ↩︎
Technically, using Android’s MTP (Media Transfer Protocol) functionality to transfer media should cause an immediate reindex, but it only works over USB. ↩︎

Random Certificate Serials in FreeIPA

Tue, 23 Oct 2018 00:00:00 +0000

Intro

I recently broke my FreeIPA setup, and needed to reinstall from scratch. Since FreeIPA by default creates certificates with serial numbers starting at 1, I ended up with a Certificate Authority with the same serial number and fields as my previous installation, but with a different certificate. Unfortunately, this caused a sec_error_reused_issuer_and_serial error in Firefox, which I couldn’t bypass. I tried a number of different solutions to try and fix this issue, including clearing out the certificate database as well as resetting my Firefox profile, but none of them worked.

Solution

In the end, I was able to work around the Firefox error by enabling random certificate serial numbers in FreeIPA and reinstalling. If random serials are enabled, FreeIPA will use a random, nonsequential number for the certificate serial number rather than starting at 1. If this is enabled before FreeIPA is installed, the Certificate Authority will also be generated with a random serial, which prevents Firefox from throwing an error.

To enable random certificate serial numbers, we need to modify the Dogtag PKI configuration. The Dogtag system is used by FreeIPA to create the integrated Certificate Authority and provision certificates. Before installing FreeIPA, edit the /etc/pki/default.cfg file on your IPA server and find the following line:

pki_random_serial_numbers_enabled=False

Change it to:

pki_random_serial_numbers_enabled=True

Save and close the file. You can now complete the FreeIPA installation as per usual; after it finishes installing, we can check the list of certificates in the WebUI under Authentication -> Certificates, and we will see that the certificate serial numbers are all random and nonsequential. Firefox should also now load pages signed by the FreeIPA Certificate Authority without complaining about reused serial numbers.