# RhemaWave Kiosk Infrastructure — Complete Maintainer Guide ## Purpose This document describes the **complete** infrastructure for building, distributing, and deploying the RhemaWave kiosk application on NixOS. It is written so that a new developer can reconstruct the entire system from scratch on a fresh machine. --- ## Table of Contents 1. Architecture Overview 2. Repositories 3. Key Files and Secrets — What Lives Where 4. Development Machine Setup (From Scratch) 5. The RhemaWave Flake (Application) 6. The NixConfig Flake (Kiosk System) 7. Binary Cache Setup 8. Caddy Server Configuration 9. Signing Keys 10. Sops-Nix Secrets Management 11. GitHub Access 12. Building and Publishing a New Release 13. Deploying a New Kiosk (nixos-anywhere) 14. Auto-Update System 15. Testing and Debugging 16. Common Errors and Solutions --- ## 1. Architecture Overview ``` ┌─────────────────────────────────┐ │ Developer Machine (P15) │ │ │ │ RhemaWave repo NixConfig repo│ │ (Go application) (NixOS cfg) │ │ │ │ │ │ nix build nix flake │ │ nix store sign update │ │ nix copy git push │ │ │ │ │ └─────────┼───────────────┼───────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌──────────┐ │ Web Server │ │ GitHub │ │ (Caddy) │ │ (private)│ │ │ │ │ │ Binary cache: │ │ NixConfig│ │ repository. │ │ flake │ │ rhemawavetech │ │ │ │ .com │ │ RhemaWave│ │ │ │ flake │ │ Static files: │ │ │ │ .narinfo + .nar │ └────┬─────┘ └────────┬────────┘ │ │ │ ▼ ▼ ┌─────────────────────────────────┐ │ Kiosk Machines │ │ │ │ systemd timer (every 15min): │ │ 1. Fetch NixConfig from │ │ GitHub │ │ 2. Evaluate flake │ │ 3. Download RhemaWave binary │ │ from binary cache │ │ 4. nixos-rebuild switch │ │ │ │ Cage compositor → RhemaWave │ │ (fullscreen, locked, no TTY │ │ switching, auto-restart) │ └─────────────────────────────────┘ ``` --- ## 2. Repositories | Repo | URL | Branch | Contents | |------|-----|--------|----------| | RhemaWave | `github:Rhema-Wave-Technologies/RhemaWave` | `dev` | Go application source, `flake.nix` (package + NixOS module), `nixos/module.nix` | | NixConfig | `github:Rhema-Wave-Technologies/NixConfig` | `main` | NixOS kiosk configuration, disk layout, encrypted secrets, sops config | Both repositories are **private**. Access requires a GitHub fine-grained personal access token with read-only Contents permission on both repos. --- ## 3. Key Files and Secrets — What Lives Where ### On the Developer Machine | File | Path | Purpose | Backup? | |------|------|---------|---------| | Binary cache private key | `~/Keys/nix/cache/rhemawave/cache-priv-key.pem` | Signs packages for the binary cache | **CRITICAL** — without this, you cannot publish new builds. Back up to encrypted offline storage. | | Binary cache public key | `~/Keys/nix/cache/rhemawave/cache-pub-key.pem` | Distributed to all client machines for verification | Can be regenerated from private key | | Sops age private key | `~/Keys/nix/sops/age-key.txt` | Encrypts/decrypts secrets in NixConfig | **CRITICAL** — without this, you cannot edit secrets. Back up to encrypted offline storage. | | GitHub access token | `~/.config/nix/nix.conf` | Nix uses this to access private GitHub repos | Can be regenerated from GitHub | | Sops age key env var | `~/.bashrc` | `SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt` | Convention only | ### On the Web Server | File | Path | Purpose | |------|------|---------| | Binary cache files | `/var/lib/nix-cache/` | Static files served by Caddy | | `nix-cache-info` | `/var/lib/nix-cache/nix-cache-info` | Metadata file for Nix clients | | Caddy config | Caddyfile (location varies) | Serves the cache directory | The server does **not** need any signing keys. It is a dumb static file host. ### On Each Kiosk | File | Path | Purpose | |------|------|---------| | SSH host key (private) | `/etc/ssh/ssh_host_ed25519_key` | Used by sops-nix to decrypt secrets | | SSH host key (public) | `/etc/ssh/ssh_host_ed25519_key.pub` | Converted to age key for sops `.sops.yaml` | | Decrypted secrets | `/run/secrets/*` | Runtime-only, never persisted to disk | ### What to Back Up (Priority Order) 1. **`~/Keys/nix/cache/rhemawave/cache-priv-key.pem`** — If lost, you must generate a new key pair and update the public key in every kiosk configuration and redeploy. 2. **`~/Keys/nix/sops/age-key.txt`** — If lost, you must re-encrypt all secrets with a new key. 3. **GitHub token** — Easily regenerated, low priority. Store backups on an encrypted USB drive or a password manager's secure notes. Never store these in any Git repository, cloud drive, or unencrypted medium. --- ## 4. Development Machine Setup (From Scratch) Starting from a fresh NixOS installation on a development machine. ### 4.1 NixOS Configuration Prerequisites In your system `configuration.nix`: ```nix nix.settings = { experimental-features = [ "nix-command" "flakes" ]; trusted-users = [ "root" "yourusername" ]; substituters = [ "https://cache.nixos.org" "https://repository.rhemawavetech.com" ]; trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" "repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40=" ]; }; nixpkgs.config.allowUnfree = true; ``` Rebuild: `sudo nixos-rebuild switch` ### 4.2 Add the RhemaWave Flake to Your System Config In your system `flake.nix`, add the input and package: ```nix inputs.rhemawave = { url = "github:Rhema-Wave-Technologies/RhemaWave/dev"; inputs.nixpkgs.follows = "nixpkgs"; }; ``` Add `rhemawave` to the `outputs` function arguments, then add to a module: ```nix { environment.systemPackages = [ rhemawave.packages.x86_64-linux.default ]; } ``` ### 4.3 GitHub Access Token Generate a fine-grained personal access token at `github.com/settings/tokens`: - Resource owner: `Rhema-Wave-Technologies` - Repository access: Select `NixConfig` and `RhemaWave` - Permissions: Contents → Read-only - Expiration: Set as appropriate Configure Nix to use it: ```bash mkdir -p ~/.config/nix echo "access-tokens = github.com=YOUR_TOKEN_HERE" > ~/.config/nix/nix.conf ``` Also copy to root (needed for `sudo` operations): ```bash sudo mkdir -p /root/.config/nix sudo cp ~/.config/nix/nix.conf /root/.config/nix/nix.conf ``` ### 4.4 Signing Keys If you're taking over from an existing maintainer, obtain: - `cache-priv-key.pem` → save to `~/Keys/nix/cache/rhemawave/cache-priv-key.pem` - `age-key.txt` → save to `~/Keys/nix/sops/age-key.txt` If starting fresh (new cache): ```bash mkdir -p ~/Keys/nix/cache/rhemawave nix key generate-secret --key-name repository.rhemawavetech.com-1 > ~/Keys/nix/cache/rhemawave/cache-priv-key.pem nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem > ~/Keys/nix/cache/rhemawave/cache-pub-key.pem ``` If starting fresh (new sops key): ```bash mkdir -p ~/Keys/nix/sops nix shell nixpkgs#age -c age-keygen -o ~/Keys/nix/sops/age-key.txt ``` Note the public key (starts with `age1...`) — it goes in `.sops.yaml`. ### 4.5 Sops Environment Variable Add to `~/.bashrc`: ```bash echo 'export SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt' >> ~/.bashrc source ~/.bashrc ``` ### 4.6 Clone the Repositories ```bash mkdir -p ~/Work/RhemaWave cd ~/Work/RhemaWave git clone git@github.com:Rhema-Wave-Technologies/RhemaWave.git git clone git@github.com:Rhema-Wave-Technologies/NixConfig.git ``` ### 4.7 Verify Everything Works ```bash # Test GitHub access cd ~/Work/RhemaWave/NixConfig nix flake show github:Rhema-Wave-Technologies/NixConfig # Test RhemaWave build cd ~/Work/RhemaWave/RhemaWave nix build .#default # Test binary runs ./result/bin/rhemawave # Test secrets decryption cd ~/Work/RhemaWave/NixConfig SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml # Should open in editor with plaintext values # Close without saving ``` --- ## 5. The RhemaWave Flake (Application) Located at `~/Work/RhemaWave/RhemaWave/flake.nix`. ### Key Points - Uses `buildGoModule` with `vendorHash = null` (vendor directory committed to repo) - CGo is enabled (`env.CGO_ENABLED = "1"`) - SDL2 mixer/ttf/image flags are set in `preBuild` via `pkg-config` - The `share/` directory is copied to `$out/share` in `postInstall` - License is set to `unfree` — consumers must set `allowUnfreePredicate` - Exports `nixosModules.default` (the kiosk module) alongside per-system packages and dev shells ### Flake Outputs | Output | Description | |--------|-------------| | `packages..default` | The RhemaWave binary | | `devShells..default` | Development shell with Go + SDL2 deps | | `nixosModules.default` | NixOS module for kiosk configuration (greetd, cage, PipeWire, polkit) | ### The NixOS Module (`nixos/module.nix`) Imported via `import ./nixos/module.nix self` — the `self` argument gives the module access to the flake's packages. The module exposes `services.rhemawave.*` options. --- ## 6. The NixConfig Flake (Kiosk System) Located at `~/Work/RhemaWave/NixConfig/flake.nix`. ### Flake Inputs ```nix nixpkgs → github:NixOS/nixpkgs/nixpkgs-unstable disko → github:nix-community/disko nixos-facter → github:numtide/nixos-facter-modules rhemawave → github:Rhema-Wave-Technologies/RhemaWave/dev sops-nix → github:Mic92/sops-nix ``` All inputs follow `nixpkgs` to ensure a single nixpkgs version. ### System Configuration Modules The NixOS system (`rhemawavekiosk`) loads these modules: 1. `disko.nixosModules.disko` — disk partitioning 2. `rhemawave.nixosModules.default` — RhemaWave kiosk module 3. `sops-nix.nixosModules.sops` — secrets management 4. `./configuration.nix` — main system config 5. Hardware facter report 6. Inline module adding `rhemawave.packages.x86_64-linux.default` to `environment.systemPackages` --- ## 7. Binary Cache Setup ### What It Is A Nix binary cache is a static HTTP file server with a specific directory structure. When a NixOS machine needs a package, it checks each configured cache by making HTTP GET requests for `.narinfo` files. If found and the signature is valid, it downloads the pre-built `.nar.xz` archive instead of building from source. ### Directory Structure ``` /var/lib/nix-cache/ ├── nix-cache-info # Cache metadata ├── fv5iinvsldz9l2s8ikm0mw8ykhriad1g.narinfo # Package metadata + signature ├── nar/ │ └── 1j7d9cif4xjq7i5rw5dmivxyninh7z1cm7zvdknsp8vjd9rjmzkc.nar.xz # Compressed package └── ... (more .narinfo and .nar.xz files) ``` ### nix-cache-info ``` StoreDir: /nix/store WantMassQuery: 1 Priority: 40 ``` This file must exist at the root. `Priority: 40` means this cache is checked after the official NixOS cache (priority 30). ### Creating the Cache (One-Time) If setting up a new cache server: ```bash # On the server sudo mkdir -p /var/lib/nix-cache cat > /var/lib/nix-cache/nix-cache-info < cache-priv-key.pem nix key convert-secret-to-public < cache-priv-key.pem > cache-pub-key.pem ``` ### Deriving Public Key from Private Key ```bash nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem ``` This outputs the public key string. It should match what's in every client's `trusted-public-keys`. ### Current Public Key ``` repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40= ``` --- ## 10. Sops-Nix Secrets Management ### How It Works 1. Secrets are encrypted in `secrets.yaml` using age public keys 2. `.sops.yaml` defines which keys can encrypt/decrypt 3. Each kiosk's SSH host key is converted to an age key for decryption 4. On activation, sops-nix decrypts secrets and places them at `/run/secrets/` ### .sops.yaml ```yaml keys: - &admin age1ptpn9zee8x7vjaaktx9dv65vysdjfmmgtj96wc3l49vd8g8huy6syk8l26 - &kiosk-dev age1nmn0r66hl24wrz73q5htt0r90vfse9dvyjwae2evqwrealfgycfsy3apmy creation_rules: - path_regex: secrets\.yaml$ key_groups: - age: - *admin - *kiosk-dev ``` ### Adding a New Kiosk 1. Get the kiosk's SSH host public key: ```bash ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub" ``` 2. Convert to age key: ```bash nix shell nixpkgs#ssh-to-age -c ssh-to-age < <(ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub") ``` 3. Add the age key to `.sops.yaml` under `keys:` with a descriptive anchor (e.g., `&kiosk-prod-001`) 4. Add the anchor reference to the creation rule's age list 5. Re-encrypt secrets with the new key: ```bash SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops updatekeys secrets.yaml ``` 6. Commit and push `.sops.yaml` and `secrets.yaml` ### Editing Secrets ```bash cd ~/Work/RhemaWave/NixConfig SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml ``` The file opens in your editor with plaintext values. Edit, save, close — sops re-encrypts automatically. ### Secret Format for github-token The `github-token` secret must contain the full nix.conf line (because it's included via `!include`): ```yaml github-token: "access-tokens = github.com=ghp_yourtokenhere" ``` --- ## 11. GitHub Access ### Developer Machine The token is stored in `~/.config/nix/nix.conf`: ``` access-tokens = github.com=YOUR_TOKEN_HERE ``` Also needs to be in `/root/.config/nix/nix.conf` for `sudo` operations. Flake URLs use the `github:` scheme (not `git+https://`), which reads the `access-tokens` setting. ### Kiosk Machines The token is managed via sops-nix and included in `nix.conf` via `!include /run/secrets/github-token`. ### Token Scope Fine-grained personal access token: - Resource owner: `Rhema-Wave-Technologies` - Repositories: `NixConfig`, `RhemaWave` - Permissions: Contents → Read-only --- ## 12. Building and Publishing a New Release Complete procedure, every command, in order. All commands run on your development machine. ### Step 1: Commit Application Changes ```bash cd ~/Work/RhemaWave/RhemaWave # If go.mod/go.sum changed: nix shell nixpkgs#go -c go mod vendor git add vendor/ git add -A git commit -m "your changes" git push origin dev ``` ### Step 2: Build ```bash nix build .#default ``` ### Step 3: Sign ```bash nix store sign --key-file ~/Keys/nix/cache/rhemawave/cache-priv-key.pem $(nix path-info ./result) ``` ### Step 4: Export to Cache Format ```bash rm -rf /tmp/nix-cache nix copy --to file:///tmp/nix-cache ./result --secret-key-files ~/Keys/nix/cache/rhemawave/cache-priv-key.pem ``` ### Step 5: Verify Signature ```bash cat /tmp/nix-cache/*.narinfo | grep Sig ``` Must show `Sig: repository.rhemawavetech.com-1:...` ### Step 6: Upload to Server ```bash rsync -av /tmp/nix-cache/ YOUR_USER@your-server:/var/lib/nix-cache/ ``` ### Step 7: Bump NixConfig ```bash cd ~/Work/RhemaWave/NixConfig nix flake update rhemawave git add flake.lock git commit -m "bump rhemawave" git push ``` ### Step 8: Wait Kiosks pick up the update within 15 minutes via the auto-update timer. --- ## 13. Deploying a New Kiosk (nixos-anywhere) ### Prerequisites - Target machine booted from NixOS installer USB - SSH access to the target as root - Target's IP address known ### Procedure From your development machine: ```bash cd ~/Work/RhemaWave/NixConfig # Deploy nix run github:nix-community/nixos-anywhere -- --flake .#rhemawavekiosk root@TARGET_IP ``` This: 1. kexec's into the NixOS installer on the target 2. Partitions the disk according to `disk-config.nix` 3. Installs NixOS with the full kiosk configuration 4. Reboots **Note:** Uses the local flake (`.#`) so it can access local files and secrets. The `--flake` path must be the NixConfig directory. ### After Deployment 1. Get the kiosk's age key (see section 10, "Adding a New Kiosk") 2. Add it to `.sops.yaml` and re-encrypt secrets 3. Push changes 4. The kiosk will pick up the updated secrets on its next auto-update cycle ### If nixos-rebuild --target-host Fails with Bootloader Errors Use the manual three-step deploy instead: ```bash nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel nix copy --to ssh://root@TARGET_IP ./result ssh root@TARGET_IP "$(readlink -f ./result)/bin/switch-to-configuration switch" ``` --- ## 14. Auto-Update System ### How It Works Each kiosk runs a systemd timer (`kiosk-auto-update.timer`) that fires every 15 minutes. The associated service runs: ```bash nixos-rebuild switch --flake github:Rhema-Wave-Technologies/NixConfig#rhemawavekiosk ``` This: 1. Fetches the latest NixConfig flake from GitHub (using the encrypted GitHub token) 2. Evaluates the configuration (including the pinned RhemaWave version from `flake.lock`) 3. Checks the binary cache for pre-built packages 4. Downloads any new packages 5. Switches to the new configuration ### What Triggers an Update An update is triggered when either: - `flake.lock` changes (new RhemaWave version pinned) - `configuration.nix` or other NixConfig files change - Any flake input is updated If nothing has changed, `nixos-rebuild` is a no-op. ### Monitoring ```bash # Check timer status ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer" # Check last update result ssh root@KIOSK_IP "systemctl status kiosk-auto-update.service" # Full update logs ssh root@KIOSK_IP "journalctl -u kiosk-auto-update -n 100" ``` ### Force Immediate Update ```bash ssh root@KIOSK_IP "systemctl start kiosk-auto-update" ``` --- ## 15. Testing and Debugging ### SSH into a Running Kiosk Even with the kiosk locked down (cage compositor, single TTY), SSH always works: ```bash ssh root@KIOSK_IP ``` ### Check What's Running ```bash ssh root@KIOSK_IP "systemctl status greetd" ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer" ssh root@KIOSK_IP "ls -la /run/secrets/" ssh root@KIOSK_IP "cat /run/secrets/wifi-psk" # verify decryption ``` ### Check Binary Cache Connectivity ```bash ssh root@KIOSK_IP "curl -s https://repository.rhemawavetech.com/nix-cache-info" ``` ### Check GitHub Connectivity ```bash ssh root@KIOSK_IP "cat /etc/nix/nix.conf | grep access-tokens" ``` ### Restart the Kiosk Application If the application crashes or you want to restart it without a full reboot: ```bash ssh root@KIOSK_IP "systemctl restart greetd" ``` greetd will restart, auto-login the kiosk user, and re-launch rhemawave in cage. ### View Application Logs ```bash ssh root@KIOSK_IP "journalctl -u greetd -n 100" ``` ### Test a Configuration Change Without Pushing Build and deploy directly from your development machine: ```bash cd ~/Work/RhemaWave/NixConfig nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel nix copy --to ssh://root@KIOSK_IP ./result ssh root@KIOSK_IP "$(readlink -f ./result)/bin/switch-to-configuration switch" ``` --- ## 16. Common Errors and Solutions ### `env attribute set cannot contain any attributes passed to derivation` **Cause:** `CGO_ENABLED` set as both a top-level attribute and in `env`. **Fix:** Use only `env.CGO_ENABLED = "1";` in the `buildGoModule` block. ### `vendor/modules.txt` inconsistency **Cause:** `go.mod` changed but `vendor/` wasn't updated. **Fix:** Run `nix shell nixpkgs#go -c go mod vendor`, then `git add vendor/` and commit. ### `unfree license, refusing to evaluate` **Cause:** Missing `allowUnfreePredicate` in the nixpkgs config. **Fix:** Add `config.allowUnfreePredicate = pkg: (nixpkgs.lib.getName pkg) == "rhemawave";` to the `pkgs` import. ### `warning: ignoring the client-specified setting 'secret-key-files'` **Cause:** Your user isn't in `trusted-users`. **Fix:** Add your username to `nix.settings.trusted-users` and rebuild your NixOS system. ### `invalid character in Base64 string` **Cause:** The `trusted-public-keys` value is malformed — likely doubled key name or invisible characters. **Fix:** Regenerate the public key: `nix key convert-secret-to-public < cache-priv-key.pem` and paste it fresh into the config. ### `attribute 'nixosModules' missing` **Cause:** The `flake.lock` points to an old RhemaWave commit that doesn't export `nixosModules`. **Fix:** Run `nix flake update rhemawave` in the NixConfig directory. ### `path '/home/...' is forbidden in pure evaluation mode` **Cause:** Using absolute paths (like `builtins.readFile /home/...`) in a flake. Flakes evaluate in pure mode. **Fix:** Use relative paths (`./file.txt`) or manage secrets with sops-nix. ### `access to absolute path '/home' is forbidden` **Cause:** Same as above — `readFile` with an absolute path. **Fix:** Use sops-nix for secrets or deploy with the local flake (`.#`) instead of `github:`. ### Bootloader errors during remote `nixos-rebuild --target-host` **Cause:** GRUB can't detect the disk layout properly over SSH. **Fix:** Use the three-step manual deploy: `nix build` → `nix copy --to ssh://` → `switch-to-configuration switch`. ### SSH host key changed warning **Cause:** After reinstalling a kiosk, its SSH host key changes. **Fix:** `ssh-keygen -R KIOSK_IP` to remove the old key, then reconnect. ### Auto-update not working on kiosk **Check:** 1. Timer is active: `systemctl status kiosk-auto-update.timer` 2. Service logs: `journalctl -u kiosk-auto-update -n 50` 3. GitHub token is decrypted: `cat /etc/nix/nix.conf | grep access-tokens` 4. Network connectivity: `curl https://github.com` 5. Cache connectivity: `curl https://repository.rhemawavetech.com/nix-cache-info`