23 KiB
RhemaWave Kiosk Infrastructure — Complete Maintainer Guide
Purpose
This document describes the complete infrastructure for building, distributing, and deploying the RhemaWave kiosk application on NixOS. It is written so that a new developer can reconstruct the entire system from scratch on a fresh machine.
Table of Contents
- Architecture Overview
- Repositories
- Key Files and Secrets — What Lives Where
- Development Machine Setup (From Scratch)
- The RhemaWave Flake (Application)
- The NixConfig Flake (Kiosk System)
- Binary Cache Setup
- Caddy Server Configuration
- Signing Keys
- Sops-Nix Secrets Management
- GitHub Access
- Building and Publishing a New Release
- Deploying a New Kiosk (nixos-anywhere)
- Auto-Update System
- Testing and Debugging
- Common Errors and Solutions
1. Architecture Overview
┌─────────────────────────────────┐
│ Developer Machine (P15) │
│ │
│ RhemaWave repo NixConfig repo│
│ (Go application) (NixOS cfg) │
│ │ │ │
│ nix build nix flake │
│ nix store sign update │
│ nix copy git push │
│ │ │ │
└─────────┼───────────────┼───────┘
│ │
▼ ▼
┌─────────────────┐ ┌──────────┐
│ Web Server │ │ GitHub │
│ (Caddy) │ │ (private)│
│ │ │ │
│ Binary cache: │ │ NixConfig│
│ repository. │ │ flake │
│ rhemawavetech │ │ │
│ .com │ │ RhemaWave│
│ │ │ flake │
│ Static files: │ │ │
│ .narinfo + .nar │ └────┬─────┘
└────────┬────────┘ │
│ │
▼ ▼
┌─────────────────────────────────┐
│ Kiosk Machines │
│ │
│ systemd timer (every 15min): │
│ 1. Fetch NixConfig from │
│ GitHub │
│ 2. Evaluate flake │
│ 3. Download RhemaWave binary │
│ from binary cache │
│ 4. nixos-rebuild switch │
│ │
│ Cage compositor → RhemaWave │
│ (fullscreen, locked, no TTY │
│ switching, auto-restart) │
└─────────────────────────────────┘
2. Repositories
| Repo | URL | Branch | Contents |
|---|---|---|---|
| RhemaWave | github:Rhema-Wave-Technologies/RhemaWave |
dev |
Go application source, flake.nix (package + NixOS module), nixos/module.nix |
| NixConfig | github:Rhema-Wave-Technologies/NixConfig |
main |
NixOS kiosk configuration, disk layout, encrypted secrets, sops config |
Both repositories are private. Access requires a GitHub fine-grained personal access token with read-only Contents permission on both repos.
3. Key Files and Secrets — What Lives Where
On the Developer Machine
| File | Path | Purpose | Backup? |
|---|---|---|---|
| Binary cache private key | ~/Keys/nix/cache/rhemawave/cache-priv-key.pem |
Signs packages for the binary cache | CRITICAL — without this, you cannot publish new builds. Back up to encrypted offline storage. |
| Binary cache public key | ~/Keys/nix/cache/rhemawave/cache-pub-key.pem |
Distributed to all client machines for verification | Can be regenerated from private key |
| Sops age private key | ~/Keys/nix/sops/age-key.txt |
Encrypts/decrypts secrets in NixConfig | CRITICAL — without this, you cannot edit secrets. Back up to encrypted offline storage. |
| GitHub access token | ~/.config/nix/nix.conf |
Nix uses this to access private GitHub repos | Can be regenerated from GitHub |
| Sops age key env var | ~/.bashrc |
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt |
Convention only |
On the Web Server
| File | Path | Purpose |
|---|---|---|
| Binary cache files | /var/lib/nix-cache/ |
Static files served by Caddy |
nix-cache-info |
/var/lib/nix-cache/nix-cache-info |
Metadata file for Nix clients |
| Caddy config | Caddyfile (location varies) | Serves the cache directory |
The server does not need any signing keys. It is a dumb static file host.
On Each Kiosk
| File | Path | Purpose |
|---|---|---|
| SSH host key (private) | /etc/ssh/ssh_host_ed25519_key |
Used by sops-nix to decrypt secrets |
| SSH host key (public) | /etc/ssh/ssh_host_ed25519_key.pub |
Converted to age key for sops .sops.yaml |
| Decrypted secrets | /run/secrets/* |
Runtime-only, never persisted to disk |
What to Back Up (Priority Order)
~/Keys/nix/cache/rhemawave/cache-priv-key.pem— If lost, you must generate a new key pair and update the public key in every kiosk configuration and redeploy.~/Keys/nix/sops/age-key.txt— If lost, you must re-encrypt all secrets with a new key.- GitHub token — Easily regenerated, low priority.
Store backups on an encrypted USB drive or a password manager's secure notes. Never store these in any Git repository, cloud drive, or unencrypted medium.
4. Development Machine Setup (From Scratch)
Starting from a fresh NixOS installation on a development machine.
4.1 NixOS Configuration Prerequisites
In your system configuration.nix:
nix.settings = {
experimental-features = [ "nix-command" "flakes" ];
trusted-users = [ "root" "yourusername" ];
substituters = [
"https://cache.nixos.org"
"https://repository.rhemawavetech.com"
];
trusted-public-keys = [
"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
"repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40="
];
};
nixpkgs.config.allowUnfree = true;
Rebuild: sudo nixos-rebuild switch
4.2 Add the RhemaWave Flake to Your System Config
In your system flake.nix, add the input and package:
inputs.rhemawave = {
url = "github:Rhema-Wave-Technologies/RhemaWave/dev";
inputs.nixpkgs.follows = "nixpkgs";
};
Add rhemawave to the outputs function arguments, then add to a module:
{
environment.systemPackages = [
rhemawave.packages.x86_64-linux.default
];
}
4.3 GitHub Access Token
Generate a fine-grained personal access token at github.com/settings/tokens:
- Resource owner:
Rhema-Wave-Technologies - Repository access: Select
NixConfigandRhemaWave - Permissions: Contents → Read-only
- Expiration: Set as appropriate
Configure Nix to use it:
mkdir -p ~/.config/nix
echo "access-tokens = github.com=YOUR_TOKEN_HERE" > ~/.config/nix/nix.conf
Also copy to root (needed for sudo operations):
sudo mkdir -p /root/.config/nix
sudo cp ~/.config/nix/nix.conf /root/.config/nix/nix.conf
4.4 Signing Keys
If you're taking over from an existing maintainer, obtain:
cache-priv-key.pem→ save to~/Keys/nix/cache/rhemawave/cache-priv-key.pemage-key.txt→ save to~/Keys/nix/sops/age-key.txt
If starting fresh (new cache):
mkdir -p ~/Keys/nix/cache/rhemawave
nix key generate-secret --key-name repository.rhemawavetech.com-1 > ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem > ~/Keys/nix/cache/rhemawave/cache-pub-key.pem
If starting fresh (new sops key):
mkdir -p ~/Keys/nix/sops
nix shell nixpkgs#age -c age-keygen -o ~/Keys/nix/sops/age-key.txt
Note the public key (starts with age1...) — it goes in .sops.yaml.
4.5 Sops Environment Variable
Add to ~/.bashrc:
echo 'export SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt' >> ~/.bashrc
source ~/.bashrc
4.6 Clone the Repositories
mkdir -p ~/Work/RhemaWave
cd ~/Work/RhemaWave
git clone git@github.com:Rhema-Wave-Technologies/RhemaWave.git
git clone git@github.com:Rhema-Wave-Technologies/NixConfig.git
4.7 Verify Everything Works
# Test GitHub access
cd ~/Work/RhemaWave/NixConfig
nix flake show github:Rhema-Wave-Technologies/NixConfig
# Test RhemaWave build
cd ~/Work/RhemaWave/RhemaWave
nix build .#default
# Test binary runs
./result/bin/rhemawave
# Test secrets decryption
cd ~/Work/RhemaWave/NixConfig
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml
# Should open in editor with plaintext values
# Close without saving
5. The RhemaWave Flake (Application)
Located at ~/Work/RhemaWave/RhemaWave/flake.nix.
Key Points
- Uses
buildGoModulewithvendorHash = null(vendor directory committed to repo) - CGo is enabled (
env.CGO_ENABLED = "1") - SDL2 mixer/ttf/image flags are set in
preBuildviapkg-config - The
share/directory is copied to$out/shareinpostInstall - License is set to
unfree— consumers must setallowUnfreePredicate - Exports
nixosModules.default(the kiosk module) alongside per-system packages and dev shells
Flake Outputs
| Output | Description |
|---|---|
packages.<system>.default |
The RhemaWave binary |
devShells.<system>.default |
Development shell with Go + SDL2 deps |
nixosModules.default |
NixOS module for kiosk configuration (greetd, cage, PipeWire, polkit) |
The NixOS Module (nixos/module.nix)
Imported via import ./nixos/module.nix self — the self argument gives the module access to the flake's packages. The module exposes services.rhemawave.* options.
6. The NixConfig Flake (Kiosk System)
Located at ~/Work/RhemaWave/NixConfig/flake.nix.
Flake Inputs
nixpkgs → github:NixOS/nixpkgs/nixpkgs-unstable
disko → github:nix-community/disko
nixos-facter → github:numtide/nixos-facter-modules
rhemawave → github:Rhema-Wave-Technologies/RhemaWave/dev
sops-nix → github:Mic92/sops-nix
All inputs follow nixpkgs to ensure a single nixpkgs version.
System Configuration Modules
The NixOS system (rhemawavekiosk) loads these modules:
disko.nixosModules.disko— disk partitioningrhemawave.nixosModules.default— RhemaWave kiosk modulesops-nix.nixosModules.sops— secrets management./configuration.nix— main system config- Hardware facter report
- Inline module adding
rhemawave.packages.x86_64-linux.defaulttoenvironment.systemPackages
7. Binary Cache Setup
What It Is
A Nix binary cache is a static HTTP file server with a specific directory structure. When a NixOS machine needs a package, it checks each configured cache by making HTTP GET requests for .narinfo files. If found and the signature is valid, it downloads the pre-built .nar.xz archive instead of building from source.
Directory Structure
/var/lib/nix-cache/
├── nix-cache-info # Cache metadata
├── fv5iinvsldz9l2s8ikm0mw8ykhriad1g.narinfo # Package metadata + signature
├── nar/
│ └── 1j7d9cif4xjq7i5rw5dmivxyninh7z1cm7zvdknsp8vjd9rjmzkc.nar.xz # Compressed package
└── ... (more .narinfo and .nar.xz files)
nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
This file must exist at the root. Priority: 40 means this cache is checked after the official NixOS cache (priority 30).
Creating the Cache (One-Time)
If setting up a new cache server:
# On the server
sudo mkdir -p /var/lib/nix-cache
cat > /var/lib/nix-cache/nix-cache-info <<EOF
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
EOF
Then configure Caddy (see section 8).
8. Caddy Server Configuration
The web server at repository.rhemawavetech.com runs Caddy with a file_server directive pointing at the cache directory.
Caddyfile
repository.rhemawavetech.com {
root * /var/lib/nix-cache
file_server browse
}
The browse directive is optional (useful for debugging). Caddy handles TLS automatically via Let's Encrypt.
Verification
curl https://repository.rhemawavetech.com/nix-cache-info
Should return the three-line metadata file.
9. Signing Keys
How Signing Works
- The developer builds a package with
nix build - The store path is signed with the private key:
nix store sign --key-file ... - The signed package is exported to cache format:
nix copy --to file:// ... - The
.narinfofile contains aSig:line with the key name and signature - Client machines verify the signature against the public key in
trusted-public-keys
Key Name
The key name repository.rhemawavetech.com-1 is embedded in both the key files and the .narinfo signatures. If you generate new keys, use the same naming convention with an incremented number (e.g., -2).
Generating New Keys
nix key generate-secret --key-name repository.rhemawavetech.com-1 > cache-priv-key.pem
nix key convert-secret-to-public < cache-priv-key.pem > cache-pub-key.pem
Deriving Public Key from Private Key
nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
This outputs the public key string. It should match what's in every client's trusted-public-keys.
Current Public Key
repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40=
10. Sops-Nix Secrets Management
How It Works
- Secrets are encrypted in
secrets.yamlusing age public keys .sops.yamldefines which keys can encrypt/decrypt- Each kiosk's SSH host key is converted to an age key for decryption
- On activation, sops-nix decrypts secrets and places them at
/run/secrets/<name>
.sops.yaml
keys:
- &admin age1ptpn9zee8x7vjaaktx9dv65vysdjfmmgtj96wc3l49vd8g8huy6syk8l26
- &kiosk-dev age1nmn0r66hl24wrz73q5htt0r90vfse9dvyjwae2evqwrealfgycfsy3apmy
creation_rules:
- path_regex: secrets\.yaml$
key_groups:
- age:
- *admin
- *kiosk-dev
Adding a New Kiosk
-
Get the kiosk's SSH host public key:
ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub" -
Convert to age key:
nix shell nixpkgs#ssh-to-age -c ssh-to-age < <(ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub") -
Add the age key to
.sops.yamlunderkeys:with a descriptive anchor (e.g.,&kiosk-prod-001) -
Add the anchor reference to the creation rule's age list
-
Re-encrypt secrets with the new key:
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops updatekeys secrets.yaml -
Commit and push
.sops.yamlandsecrets.yaml
Editing Secrets
cd ~/Work/RhemaWave/NixConfig
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml
The file opens in your editor with plaintext values. Edit, save, close — sops re-encrypts automatically.
Secret Format for github-token
The github-token secret must contain the full nix.conf line (because it's included via !include):
github-token: "access-tokens = github.com=ghp_yourtokenhere"
11. GitHub Access
Developer Machine
The token is stored in ~/.config/nix/nix.conf:
access-tokens = github.com=YOUR_TOKEN_HERE
Also needs to be in /root/.config/nix/nix.conf for sudo operations.
Flake URLs use the github: scheme (not git+https://), which reads the access-tokens setting.
Kiosk Machines
The token is managed via sops-nix and included in nix.conf via !include /run/secrets/github-token.
Token Scope
Fine-grained personal access token:
- Resource owner:
Rhema-Wave-Technologies - Repositories:
NixConfig,RhemaWave - Permissions: Contents → Read-only
12. Building and Publishing a New Release
Complete procedure, every command, in order. All commands run on your development machine.
Step 1: Commit Application Changes
cd ~/Work/RhemaWave/RhemaWave
# If go.mod/go.sum changed:
nix shell nixpkgs#go -c go mod vendor
git add vendor/
git add -A
git commit -m "your changes"
git push origin dev
Step 2: Build
nix build .#default
Step 3: Sign
nix store sign --key-file ~/Keys/nix/cache/rhemawave/cache-priv-key.pem $(nix path-info ./result)
Step 4: Export to Cache Format
rm -rf /tmp/nix-cache
nix copy --to file:///tmp/nix-cache ./result --secret-key-files ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
Step 5: Verify Signature
cat /tmp/nix-cache/*.narinfo | grep Sig
Must show Sig: repository.rhemawavetech.com-1:...
Step 6: Upload to Server
rsync -av /tmp/nix-cache/ YOUR_USER@your-server:/var/lib/nix-cache/
Step 7: Bump NixConfig
cd ~/Work/RhemaWave/NixConfig
nix flake update rhemawave
git add flake.lock
git commit -m "bump rhemawave"
git push
Step 8: Wait
Kiosks pick up the update within 15 minutes via the auto-update timer.
13. Deploying a New Kiosk (nixos-anywhere)
Prerequisites
- Target machine booted from NixOS installer USB
- SSH access to the target as root
- Target's IP address known
Procedure
From your development machine:
cd ~/Work/RhemaWave/NixConfig
# Deploy
nix run github:nix-community/nixos-anywhere -- --flake .#rhemawavekiosk root@TARGET_IP
This:
- kexec's into the NixOS installer on the target
- Partitions the disk according to
disk-config.nix - Installs NixOS with the full kiosk configuration
- Reboots
Note: Uses the local flake (.#) so it can access local files and secrets. The --flake path must be the NixConfig directory.
After Deployment
- Get the kiosk's age key (see section 10, "Adding a New Kiosk")
- Add it to
.sops.yamland re-encrypt secrets - Push changes
- The kiosk will pick up the updated secrets on its next auto-update cycle
If nixos-rebuild --target-host Fails with Bootloader Errors
Use the manual three-step deploy instead:
nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel
nix copy --to ssh://root@TARGET_IP ./result
ssh root@TARGET_IP "$(readlink -f ./result)/bin/switch-to-configuration switch"
14. Auto-Update System
How It Works
Each kiosk runs a systemd timer (kiosk-auto-update.timer) that fires every 15 minutes. The associated service runs:
nixos-rebuild switch --flake github:Rhema-Wave-Technologies/NixConfig#rhemawavekiosk
This:
- Fetches the latest NixConfig flake from GitHub (using the encrypted GitHub token)
- Evaluates the configuration (including the pinned RhemaWave version from
flake.lock) - Checks the binary cache for pre-built packages
- Downloads any new packages
- Switches to the new configuration
What Triggers an Update
An update is triggered when either:
flake.lockchanges (new RhemaWave version pinned)configuration.nixor other NixConfig files change- Any flake input is updated
If nothing has changed, nixos-rebuild is a no-op.
Monitoring
# Check timer status
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer"
# Check last update result
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.service"
# Full update logs
ssh root@KIOSK_IP "journalctl -u kiosk-auto-update -n 100"
Force Immediate Update
ssh root@KIOSK_IP "systemctl start kiosk-auto-update"
15. Testing and Debugging
SSH into a Running Kiosk
Even with the kiosk locked down (cage compositor, single TTY), SSH always works:
ssh root@KIOSK_IP
Check What's Running
ssh root@KIOSK_IP "systemctl status greetd"
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer"
ssh root@KIOSK_IP "ls -la /run/secrets/"
ssh root@KIOSK_IP "cat /run/secrets/wifi-psk" # verify decryption
Check Binary Cache Connectivity
ssh root@KIOSK_IP "curl -s https://repository.rhemawavetech.com/nix-cache-info"
Check GitHub Connectivity
ssh root@KIOSK_IP "cat /etc/nix/nix.conf | grep access-tokens"
Restart the Kiosk Application
If the application crashes or you want to restart it without a full reboot:
ssh root@KIOSK_IP "systemctl restart greetd"
greetd will restart, auto-login the kiosk user, and re-launch rhemawave in cage.
View Application Logs
ssh root@KIOSK_IP "journalctl -u greetd -n 100"
Test a Configuration Change Without Pushing
Build and deploy directly from your development machine:
cd ~/Work/RhemaWave/NixConfig
nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel
nix copy --to ssh://root@KIOSK_IP ./result
ssh root@KIOSK_IP "$(readlink -f ./result)/bin/switch-to-configuration switch"
16. Common Errors and Solutions
env attribute set cannot contain any attributes passed to derivation
Cause: CGO_ENABLED set as both a top-level attribute and in env.
Fix: Use only env.CGO_ENABLED = "1"; in the buildGoModule block.
vendor/modules.txt inconsistency
Cause: go.mod changed but vendor/ wasn't updated.
Fix: Run nix shell nixpkgs#go -c go mod vendor, then git add vendor/ and commit.
unfree license, refusing to evaluate
Cause: Missing allowUnfreePredicate in the nixpkgs config.
Fix: Add config.allowUnfreePredicate = pkg: (nixpkgs.lib.getName pkg) == "rhemawave"; to the pkgs import.
warning: ignoring the client-specified setting 'secret-key-files'
Cause: Your user isn't in trusted-users.
Fix: Add your username to nix.settings.trusted-users and rebuild your NixOS system.
invalid character in Base64 string
Cause: The trusted-public-keys value is malformed — likely doubled key name or invisible characters.
Fix: Regenerate the public key: nix key convert-secret-to-public < cache-priv-key.pem and paste it fresh into the config.
attribute 'nixosModules' missing
Cause: The flake.lock points to an old RhemaWave commit that doesn't export nixosModules.
Fix: Run nix flake update rhemawave in the NixConfig directory.
path '/home/...' is forbidden in pure evaluation mode
Cause: Using absolute paths (like builtins.readFile /home/...) in a flake. Flakes evaluate in pure mode.
Fix: Use relative paths (./file.txt) or manage secrets with sops-nix.
access to absolute path '/home' is forbidden
Cause: Same as above — readFile with an absolute path.
Fix: Use sops-nix for secrets or deploy with the local flake (.#) instead of github:.
Bootloader errors during remote nixos-rebuild --target-host
Cause: GRUB can't detect the disk layout properly over SSH.
Fix: Use the three-step manual deploy: nix build → nix copy --to ssh:// → switch-to-configuration switch.
SSH host key changed warning
Cause: After reinstalling a kiosk, its SSH host key changes.
Fix: ssh-keygen -R KIOSK_IP to remove the old key, then reconnect.
Auto-update not working on kiosk
Check:
- Timer is active:
systemctl status kiosk-auto-update.timer - Service logs:
journalctl -u kiosk-auto-update -n 50 - GitHub token is decrypted:
cat /etc/nix/nix.conf | grep access-tokens - Network connectivity:
curl https://github.com - Cache connectivity:
curl https://repository.rhemawavetech.com/nix-cache-info