Files
radiantwave-legacy/maintainer-guide.md
T

23 KiB

RhemaWave Kiosk Infrastructure — Complete Maintainer Guide

Purpose

This document describes the complete infrastructure for building, distributing, and deploying the RhemaWave kiosk application on NixOS. It is written so that a new developer can reconstruct the entire system from scratch on a fresh machine.


Table of Contents

  1. Architecture Overview
  2. Repositories
  3. Key Files and Secrets — What Lives Where
  4. Development Machine Setup (From Scratch)
  5. The RhemaWave Flake (Application)
  6. The NixConfig Flake (Kiosk System)
  7. Binary Cache Setup
  8. Caddy Server Configuration
  9. Signing Keys
  10. Sops-Nix Secrets Management
  11. GitHub Access
  12. Building and Publishing a New Release
  13. Deploying a New Kiosk (nixos-anywhere)
  14. Auto-Update System
  15. Testing and Debugging
  16. Common Errors and Solutions

1. Architecture Overview

┌─────────────────────────────────┐
│      Developer Machine (P15)    │
│                                 │
│  RhemaWave repo   NixConfig repo│
│  (Go application)  (NixOS cfg)  │
│         │               │       │
│    nix build        nix flake   │
│    nix store sign   update      │
│    nix copy         git push    │
│         │               │       │
└─────────┼───────────────┼───────┘
          │               │
          ▼               ▼
┌─────────────────┐  ┌──────────┐
│   Web Server    │  │  GitHub  │
│   (Caddy)       │  │ (private)│
│                 │  │          │
│ Binary cache:   │  │ NixConfig│
│ repository.     │  │ flake    │
│ rhemawavetech   │  │          │
│ .com            │  │ RhemaWave│
│                 │  │ flake    │
│ Static files:   │  │          │
│ .narinfo + .nar │  └────┬─────┘
└────────┬────────┘       │
         │                │
         ▼                ▼
┌─────────────────────────────────┐
│         Kiosk Machines          │
│                                 │
│  systemd timer (every 15min):   │
│    1. Fetch NixConfig from      │
│       GitHub                    │
│    2. Evaluate flake            │
│    3. Download RhemaWave binary │
│       from binary cache         │
│    4. nixos-rebuild switch      │
│                                 │
│  Cage compositor → RhemaWave    │
│  (fullscreen, locked, no TTY    │
│   switching, auto-restart)      │
└─────────────────────────────────┘

2. Repositories

Repo URL Branch Contents
RhemaWave github:Rhema-Wave-Technologies/RhemaWave dev Go application source, flake.nix (package + NixOS module), nixos/module.nix
NixConfig github:Rhema-Wave-Technologies/NixConfig main NixOS kiosk configuration, disk layout, encrypted secrets, sops config

Both repositories are private. Access requires a GitHub fine-grained personal access token with read-only Contents permission on both repos.


3. Key Files and Secrets — What Lives Where

On the Developer Machine

File Path Purpose Backup?
Binary cache private key ~/Keys/nix/cache/rhemawave/cache-priv-key.pem Signs packages for the binary cache CRITICAL — without this, you cannot publish new builds. Back up to encrypted offline storage.
Binary cache public key ~/Keys/nix/cache/rhemawave/cache-pub-key.pem Distributed to all client machines for verification Can be regenerated from private key
Sops age private key ~/Keys/nix/sops/age-key.txt Encrypts/decrypts secrets in NixConfig CRITICAL — without this, you cannot edit secrets. Back up to encrypted offline storage.
GitHub access token ~/.config/nix/nix.conf Nix uses this to access private GitHub repos Can be regenerated from GitHub
Sops age key env var ~/.bashrc SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt Convention only

On the Web Server

File Path Purpose
Binary cache files /var/lib/nix-cache/ Static files served by Caddy
nix-cache-info /var/lib/nix-cache/nix-cache-info Metadata file for Nix clients
Caddy config Caddyfile (location varies) Serves the cache directory

The server does not need any signing keys. It is a dumb static file host.

On Each Kiosk

File Path Purpose
SSH host key (private) /etc/ssh/ssh_host_ed25519_key Used by sops-nix to decrypt secrets
SSH host key (public) /etc/ssh/ssh_host_ed25519_key.pub Converted to age key for sops .sops.yaml
Decrypted secrets /run/secrets/* Runtime-only, never persisted to disk

What to Back Up (Priority Order)

  1. ~/Keys/nix/cache/rhemawave/cache-priv-key.pem — If lost, you must generate a new key pair and update the public key in every kiosk configuration and redeploy.
  2. ~/Keys/nix/sops/age-key.txt — If lost, you must re-encrypt all secrets with a new key.
  3. GitHub token — Easily regenerated, low priority.

Store backups on an encrypted USB drive or a password manager's secure notes. Never store these in any Git repository, cloud drive, or unencrypted medium.


4. Development Machine Setup (From Scratch)

Starting from a fresh NixOS installation on a development machine.

4.1 NixOS Configuration Prerequisites

In your system configuration.nix:

nix.settings = {
  experimental-features = [ "nix-command" "flakes" ];
  trusted-users = [ "root" "yourusername" ];
  substituters = [
    "https://cache.nixos.org"
    "https://repository.rhemawavetech.com"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40="
  ];
};
nixpkgs.config.allowUnfree = true;

Rebuild: sudo nixos-rebuild switch

4.2 Add the RhemaWave Flake to Your System Config

In your system flake.nix, add the input and package:

inputs.rhemawave = {
  url = "github:Rhema-Wave-Technologies/RhemaWave/dev";
  inputs.nixpkgs.follows = "nixpkgs";
};

Add rhemawave to the outputs function arguments, then add to a module:

{
  environment.systemPackages = [
    rhemawave.packages.x86_64-linux.default
  ];
}

4.3 GitHub Access Token

Generate a fine-grained personal access token at github.com/settings/tokens:

  • Resource owner: Rhema-Wave-Technologies
  • Repository access: Select NixConfig and RhemaWave
  • Permissions: Contents → Read-only
  • Expiration: Set as appropriate

Configure Nix to use it:

mkdir -p ~/.config/nix
echo "access-tokens = github.com=YOUR_TOKEN_HERE" > ~/.config/nix/nix.conf

Also copy to root (needed for sudo operations):

sudo mkdir -p /root/.config/nix
sudo cp ~/.config/nix/nix.conf /root/.config/nix/nix.conf

4.4 Signing Keys

If you're taking over from an existing maintainer, obtain:

  • cache-priv-key.pem → save to ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
  • age-key.txt → save to ~/Keys/nix/sops/age-key.txt

If starting fresh (new cache):

mkdir -p ~/Keys/nix/cache/rhemawave
nix key generate-secret --key-name repository.rhemawavetech.com-1 > ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem > ~/Keys/nix/cache/rhemawave/cache-pub-key.pem

If starting fresh (new sops key):

mkdir -p ~/Keys/nix/sops
nix shell nixpkgs#age -c age-keygen -o ~/Keys/nix/sops/age-key.txt

Note the public key (starts with age1...) — it goes in .sops.yaml.

4.5 Sops Environment Variable

Add to ~/.bashrc:

echo 'export SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt' >> ~/.bashrc
source ~/.bashrc

4.6 Clone the Repositories

mkdir -p ~/Work/RhemaWave
cd ~/Work/RhemaWave
git clone git@github.com:Rhema-Wave-Technologies/RhemaWave.git
git clone git@github.com:Rhema-Wave-Technologies/NixConfig.git

4.7 Verify Everything Works

# Test GitHub access
cd ~/Work/RhemaWave/NixConfig
nix flake show github:Rhema-Wave-Technologies/NixConfig

# Test RhemaWave build
cd ~/Work/RhemaWave/RhemaWave
nix build .#default

# Test binary runs
./result/bin/rhemawave

# Test secrets decryption
cd ~/Work/RhemaWave/NixConfig
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml
# Should open in editor with plaintext values
# Close without saving

5. The RhemaWave Flake (Application)

Located at ~/Work/RhemaWave/RhemaWave/flake.nix.

Key Points

  • Uses buildGoModule with vendorHash = null (vendor directory committed to repo)
  • CGo is enabled (env.CGO_ENABLED = "1")
  • SDL2 mixer/ttf/image flags are set in preBuild via pkg-config
  • The share/ directory is copied to $out/share in postInstall
  • License is set to unfree — consumers must set allowUnfreePredicate
  • Exports nixosModules.default (the kiosk module) alongside per-system packages and dev shells

Flake Outputs

Output Description
packages.<system>.default The RhemaWave binary
devShells.<system>.default Development shell with Go + SDL2 deps
nixosModules.default NixOS module for kiosk configuration (greetd, cage, PipeWire, polkit)

The NixOS Module (nixos/module.nix)

Imported via import ./nixos/module.nix self — the self argument gives the module access to the flake's packages. The module exposes services.rhemawave.* options.


6. The NixConfig Flake (Kiosk System)

Located at ~/Work/RhemaWave/NixConfig/flake.nix.

Flake Inputs

nixpkgs          → github:NixOS/nixpkgs/nixpkgs-unstable
disko            → github:nix-community/disko
nixos-facter     → github:numtide/nixos-facter-modules
rhemawave        → github:Rhema-Wave-Technologies/RhemaWave/dev
sops-nix         → github:Mic92/sops-nix

All inputs follow nixpkgs to ensure a single nixpkgs version.

System Configuration Modules

The NixOS system (rhemawavekiosk) loads these modules:

  1. disko.nixosModules.disko — disk partitioning
  2. rhemawave.nixosModules.default — RhemaWave kiosk module
  3. sops-nix.nixosModules.sops — secrets management
  4. ./configuration.nix — main system config
  5. Hardware facter report
  6. Inline module adding rhemawave.packages.x86_64-linux.default to environment.systemPackages

7. Binary Cache Setup

What It Is

A Nix binary cache is a static HTTP file server with a specific directory structure. When a NixOS machine needs a package, it checks each configured cache by making HTTP GET requests for .narinfo files. If found and the signature is valid, it downloads the pre-built .nar.xz archive instead of building from source.

Directory Structure

/var/lib/nix-cache/
├── nix-cache-info                              # Cache metadata
├── fv5iinvsldz9l2s8ikm0mw8ykhriad1g.narinfo   # Package metadata + signature
├── nar/
│   └── 1j7d9cif4xjq7i5rw5dmivxyninh7z1cm7zvdknsp8vjd9rjmzkc.nar.xz  # Compressed package
└── ... (more .narinfo and .nar.xz files)

nix-cache-info

StoreDir: /nix/store
WantMassQuery: 1
Priority: 40

This file must exist at the root. Priority: 40 means this cache is checked after the official NixOS cache (priority 30).

Creating the Cache (One-Time)

If setting up a new cache server:

# On the server
sudo mkdir -p /var/lib/nix-cache
cat > /var/lib/nix-cache/nix-cache-info <<EOF
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
EOF

Then configure Caddy (see section 8).


8. Caddy Server Configuration

The web server at repository.rhemawavetech.com runs Caddy with a file_server directive pointing at the cache directory.

Caddyfile

repository.rhemawavetech.com {
    root * /var/lib/nix-cache
    file_server browse
}

The browse directive is optional (useful for debugging). Caddy handles TLS automatically via Let's Encrypt.

Verification

curl https://repository.rhemawavetech.com/nix-cache-info

Should return the three-line metadata file.


9. Signing Keys

How Signing Works

  1. The developer builds a package with nix build
  2. The store path is signed with the private key: nix store sign --key-file ...
  3. The signed package is exported to cache format: nix copy --to file:// ...
  4. The .narinfo file contains a Sig: line with the key name and signature
  5. Client machines verify the signature against the public key in trusted-public-keys

Key Name

The key name repository.rhemawavetech.com-1 is embedded in both the key files and the .narinfo signatures. If you generate new keys, use the same naming convention with an incremented number (e.g., -2).

Generating New Keys

nix key generate-secret --key-name repository.rhemawavetech.com-1 > cache-priv-key.pem
nix key convert-secret-to-public < cache-priv-key.pem > cache-pub-key.pem

Deriving Public Key from Private Key

nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem

This outputs the public key string. It should match what's in every client's trusted-public-keys.

Current Public Key

repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40=

10. Sops-Nix Secrets Management

How It Works

  1. Secrets are encrypted in secrets.yaml using age public keys
  2. .sops.yaml defines which keys can encrypt/decrypt
  3. Each kiosk's SSH host key is converted to an age key for decryption
  4. On activation, sops-nix decrypts secrets and places them at /run/secrets/<name>

.sops.yaml

keys:
  - &admin age1ptpn9zee8x7vjaaktx9dv65vysdjfmmgtj96wc3l49vd8g8huy6syk8l26
  - &kiosk-dev age1nmn0r66hl24wrz73q5htt0r90vfse9dvyjwae2evqwrealfgycfsy3apmy
creation_rules:
  - path_regex: secrets\.yaml$
    key_groups:
      - age:
          - *admin
          - *kiosk-dev

Adding a New Kiosk

  1. Get the kiosk's SSH host public key:

    ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub"
    
  2. Convert to age key:

    nix shell nixpkgs#ssh-to-age -c ssh-to-age < <(ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub")
    
  3. Add the age key to .sops.yaml under keys: with a descriptive anchor (e.g., &kiosk-prod-001)

  4. Add the anchor reference to the creation rule's age list

  5. Re-encrypt secrets with the new key:

    SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops updatekeys secrets.yaml
    
  6. Commit and push .sops.yaml and secrets.yaml

Editing Secrets

cd ~/Work/RhemaWave/NixConfig
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml

The file opens in your editor with plaintext values. Edit, save, close — sops re-encrypts automatically.

Secret Format for github-token

The github-token secret must contain the full nix.conf line (because it's included via !include):

github-token: "access-tokens = github.com=ghp_yourtokenhere"

11. GitHub Access

Developer Machine

The token is stored in ~/.config/nix/nix.conf:

access-tokens = github.com=YOUR_TOKEN_HERE

Also needs to be in /root/.config/nix/nix.conf for sudo operations.

Flake URLs use the github: scheme (not git+https://), which reads the access-tokens setting.

Kiosk Machines

The token is managed via sops-nix and included in nix.conf via !include /run/secrets/github-token.

Token Scope

Fine-grained personal access token:

  • Resource owner: Rhema-Wave-Technologies
  • Repositories: NixConfig, RhemaWave
  • Permissions: Contents → Read-only

12. Building and Publishing a New Release

Complete procedure, every command, in order. All commands run on your development machine.

Step 1: Commit Application Changes

cd ~/Work/RhemaWave/RhemaWave

# If go.mod/go.sum changed:
nix shell nixpkgs#go -c go mod vendor
git add vendor/

git add -A
git commit -m "your changes"
git push origin dev

Step 2: Build

nix build .#default

Step 3: Sign

nix store sign --key-file ~/Keys/nix/cache/rhemawave/cache-priv-key.pem $(nix path-info ./result)

Step 4: Export to Cache Format

rm -rf /tmp/nix-cache
nix copy --to file:///tmp/nix-cache ./result --secret-key-files ~/Keys/nix/cache/rhemawave/cache-priv-key.pem

Step 5: Verify Signature

cat /tmp/nix-cache/*.narinfo | grep Sig

Must show Sig: repository.rhemawavetech.com-1:...

Step 6: Upload to Server

rsync -av /tmp/nix-cache/ YOUR_USER@your-server:/var/lib/nix-cache/

Step 7: Bump NixConfig

cd ~/Work/RhemaWave/NixConfig
nix flake update rhemawave
git add flake.lock
git commit -m "bump rhemawave"
git push

Step 8: Wait

Kiosks pick up the update within 15 minutes via the auto-update timer.


13. Deploying a New Kiosk (nixos-anywhere)

Prerequisites

  • Target machine booted from NixOS installer USB
  • SSH access to the target as root
  • Target's IP address known

Procedure

From your development machine:

cd ~/Work/RhemaWave/NixConfig

# Deploy
nix run github:nix-community/nixos-anywhere -- --flake .#rhemawavekiosk root@TARGET_IP

This:

  1. kexec's into the NixOS installer on the target
  2. Partitions the disk according to disk-config.nix
  3. Installs NixOS with the full kiosk configuration
  4. Reboots

Note: Uses the local flake (.#) so it can access local files and secrets. The --flake path must be the NixConfig directory.

After Deployment

  1. Get the kiosk's age key (see section 10, "Adding a New Kiosk")
  2. Add it to .sops.yaml and re-encrypt secrets
  3. Push changes
  4. The kiosk will pick up the updated secrets on its next auto-update cycle

If nixos-rebuild --target-host Fails with Bootloader Errors

Use the manual three-step deploy instead:

nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel
nix copy --to ssh://root@TARGET_IP ./result
ssh root@TARGET_IP "$(readlink -f ./result)/bin/switch-to-configuration switch"

14. Auto-Update System

How It Works

Each kiosk runs a systemd timer (kiosk-auto-update.timer) that fires every 15 minutes. The associated service runs:

nixos-rebuild switch --flake github:Rhema-Wave-Technologies/NixConfig#rhemawavekiosk

This:

  1. Fetches the latest NixConfig flake from GitHub (using the encrypted GitHub token)
  2. Evaluates the configuration (including the pinned RhemaWave version from flake.lock)
  3. Checks the binary cache for pre-built packages
  4. Downloads any new packages
  5. Switches to the new configuration

What Triggers an Update

An update is triggered when either:

  • flake.lock changes (new RhemaWave version pinned)
  • configuration.nix or other NixConfig files change
  • Any flake input is updated

If nothing has changed, nixos-rebuild is a no-op.

Monitoring

# Check timer status
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer"

# Check last update result
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.service"

# Full update logs
ssh root@KIOSK_IP "journalctl -u kiosk-auto-update -n 100"

Force Immediate Update

ssh root@KIOSK_IP "systemctl start kiosk-auto-update"

15. Testing and Debugging

SSH into a Running Kiosk

Even with the kiosk locked down (cage compositor, single TTY), SSH always works:

ssh root@KIOSK_IP

Check What's Running

ssh root@KIOSK_IP "systemctl status greetd"
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer"
ssh root@KIOSK_IP "ls -la /run/secrets/"
ssh root@KIOSK_IP "cat /run/secrets/wifi-psk"  # verify decryption

Check Binary Cache Connectivity

ssh root@KIOSK_IP "curl -s https://repository.rhemawavetech.com/nix-cache-info"

Check GitHub Connectivity

ssh root@KIOSK_IP "cat /etc/nix/nix.conf | grep access-tokens"

Restart the Kiosk Application

If the application crashes or you want to restart it without a full reboot:

ssh root@KIOSK_IP "systemctl restart greetd"

greetd will restart, auto-login the kiosk user, and re-launch rhemawave in cage.

View Application Logs

ssh root@KIOSK_IP "journalctl -u greetd -n 100"

Test a Configuration Change Without Pushing

Build and deploy directly from your development machine:

cd ~/Work/RhemaWave/NixConfig
nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel
nix copy --to ssh://root@KIOSK_IP ./result
ssh root@KIOSK_IP "$(readlink -f ./result)/bin/switch-to-configuration switch"

16. Common Errors and Solutions

env attribute set cannot contain any attributes passed to derivation

Cause: CGO_ENABLED set as both a top-level attribute and in env. Fix: Use only env.CGO_ENABLED = "1"; in the buildGoModule block.

vendor/modules.txt inconsistency

Cause: go.mod changed but vendor/ wasn't updated. Fix: Run nix shell nixpkgs#go -c go mod vendor, then git add vendor/ and commit.

unfree license, refusing to evaluate

Cause: Missing allowUnfreePredicate in the nixpkgs config. Fix: Add config.allowUnfreePredicate = pkg: (nixpkgs.lib.getName pkg) == "rhemawave"; to the pkgs import.

warning: ignoring the client-specified setting 'secret-key-files'

Cause: Your user isn't in trusted-users. Fix: Add your username to nix.settings.trusted-users and rebuild your NixOS system.

invalid character in Base64 string

Cause: The trusted-public-keys value is malformed — likely doubled key name or invisible characters. Fix: Regenerate the public key: nix key convert-secret-to-public < cache-priv-key.pem and paste it fresh into the config.

attribute 'nixosModules' missing

Cause: The flake.lock points to an old RhemaWave commit that doesn't export nixosModules. Fix: Run nix flake update rhemawave in the NixConfig directory.

path '/home/...' is forbidden in pure evaluation mode

Cause: Using absolute paths (like builtins.readFile /home/...) in a flake. Flakes evaluate in pure mode. Fix: Use relative paths (./file.txt) or manage secrets with sops-nix.

access to absolute path '/home' is forbidden

Cause: Same as above — readFile with an absolute path. Fix: Use sops-nix for secrets or deploy with the local flake (.#) instead of github:.

Bootloader errors during remote nixos-rebuild --target-host

Cause: GRUB can't detect the disk layout properly over SSH. Fix: Use the three-step manual deploy: nix build → nix copy --to ssh:// → switch-to-configuration switch.

SSH host key changed warning

Cause: After reinstalling a kiosk, its SSH host key changes. Fix: ssh-keygen -R KIOSK_IP to remove the old key, then reconnect.

Auto-update not working on kiosk

Check:

  1. Timer is active: systemctl status kiosk-auto-update.timer
  2. Service logs: journalctl -u kiosk-auto-update -n 50
  3. GitHub token is decrypted: cat /etc/nix/nix.conf | grep access-tokens
  4. Network connectivity: curl https://github.com
  5. Cache connectivity: curl https://repository.rhemawavetech.com/nix-cache-info