radiantwave-legacy/maintainer-guide.md

# RhemaWave Kiosk Infrastructure — Complete Maintainer Guide

## Purpose

This document describes the **complete** infrastructure for building, distributing, and deploying the RhemaWave kiosk application on NixOS. It is written so that a new developer can reconstruct the entire system from scratch on a fresh machine.

---

## Table of Contents

1. Architecture Overview
2. Repositories
3. Key Files and Secrets — What Lives Where
4. Development Machine Setup (From Scratch)
5. The RhemaWave Flake (Application)
6. The NixConfig Flake (Kiosk System)
7. Binary Cache Setup
8. Caddy Server Configuration
9. Signing Keys
10. Sops-Nix Secrets Management
11. GitHub Access
12. Building and Publishing a New Release
13. Deploying a New Kiosk (nixos-anywhere)
14. Auto-Update System
15. Testing and Debugging
16. Common Errors and Solutions

---

## 1. Architecture Overview

```
┌─────────────────────────────────┐
│      Developer Machine (P15)    │
│                                 │
│  RhemaWave repo   NixConfig repo│
│  (Go application)  (NixOS cfg)  │
│         │               │       │
│    nix build        nix flake   │
│    nix store sign   update      │
│    nix copy         git push    │
│         │               │       │
└─────────┼───────────────┼───────┘
          │               │
          ▼               ▼
┌─────────────────┐  ┌──────────┐
│   Web Server    │  │  GitHub  │
│   (Caddy)       │  │ (private)│
│                 │  │          │
│ Binary cache:   │  │ NixConfig│
│ repository.     │  │ flake    │
│ rhemawavetech   │  │          │
│ .com            │  │ RhemaWave│
│                 │  │ flake    │
│ Static files:   │  │          │
│ .narinfo + .nar │  └────┬─────┘
└────────┬────────┘       │
         │                │
         ▼                ▼
┌─────────────────────────────────┐
│         Kiosk Machines          │
│                                 │
│  systemd timer (every 15min):   │
│    1. Fetch NixConfig from      │
│       GitHub                    │
│    2. Evaluate flake            │
│    3. Download RhemaWave binary │
│       from binary cache         │
│    4. nixos-rebuild switch      │
│                                 │
│  Cage compositor → RhemaWave    │
│  (fullscreen, locked, no TTY    │
│   switching, auto-restart)      │
└─────────────────────────────────┘
```

---

## 2. Repositories

| Repo | URL | Branch | Contents |
|------|-----|--------|----------|
| RhemaWave | `github:Rhema-Wave-Technologies/RhemaWave` | `dev` | Go application source, `flake.nix` (package + NixOS module), `nixos/module.nix` |
| NixConfig | `github:Rhema-Wave-Technologies/NixConfig` | `main` | NixOS kiosk configuration, disk layout, encrypted secrets, sops config |

Both repositories are **private**. Access requires a GitHub fine-grained personal access token with read-only Contents permission on both repos.

---

## 3. Key Files and Secrets — What Lives Where

### On the Developer Machine

| File | Path | Purpose | Backup? |
|------|------|---------|---------|
| Binary cache private key | `~/Keys/nix/cache/rhemawave/cache-priv-key.pem` | Signs packages for the binary cache | **CRITICAL** — without this, you cannot publish new builds. Back up to encrypted offline storage. |
| Binary cache public key | `~/Keys/nix/cache/rhemawave/cache-pub-key.pem` | Distributed to all client machines for verification | Can be regenerated from private key |
| Sops age private key | `~/Keys/nix/sops/age-key.txt` | Encrypts/decrypts secrets in NixConfig | **CRITICAL** — without this, you cannot edit secrets. Back up to encrypted offline storage. |
| GitHub access token | `~/.config/nix/nix.conf` | Nix uses this to access private GitHub repos | Can be regenerated from GitHub |
| Sops age key env var | `~/.bashrc` | `SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt` | Convention only |

### On the Web Server

| File | Path | Purpose |
|------|------|---------|
| Binary cache files | `/var/lib/nix-cache/` | Static files served by Caddy |
| `nix-cache-info` | `/var/lib/nix-cache/nix-cache-info` | Metadata file for Nix clients |
| Caddy config | Caddyfile (location varies) | Serves the cache directory |

The server does **not** need any signing keys. It is a dumb static file host.

### On Each Kiosk

| File | Path | Purpose |
|------|------|---------|
| SSH host key (private) | `/etc/ssh/ssh_host_ed25519_key` | Used by sops-nix to decrypt secrets |
| SSH host key (public) | `/etc/ssh/ssh_host_ed25519_key.pub` | Converted to age key for sops `.sops.yaml` |
| Decrypted secrets | `/run/secrets/*` | Runtime-only, never persisted to disk |

### What to Back Up (Priority Order)

1. **`~/Keys/nix/cache/rhemawave/cache-priv-key.pem`** — If lost, you must generate a new key pair and update the public key in every kiosk configuration and redeploy.
2. **`~/Keys/nix/sops/age-key.txt`** — If lost, you must re-encrypt all secrets with a new key.
3. **GitHub token** — Easily regenerated, low priority.

Store backups on an encrypted USB drive or a password manager's secure notes. Never store these in any Git repository, cloud drive, or unencrypted medium.

---

## 4. Development Machine Setup (From Scratch)

Starting from a fresh NixOS installation on a development machine.

### 4.1 NixOS Configuration Prerequisites

In your system `configuration.nix`:

```nix
nix.settings = {
  experimental-features = [ "nix-command" "flakes" ];
  trusted-users = [ "root" "yourusername" ];
  substituters = [
    "https://cache.nixos.org"
    "https://repository.rhemawavetech.com"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40="
  ];
};
nixpkgs.config.allowUnfree = true;
```

Rebuild: `sudo nixos-rebuild switch`

### 4.2 Add the RhemaWave Flake to Your System Config

In your system `flake.nix`, add the input and package:

```nix
inputs.rhemawave = {
  url = "github:Rhema-Wave-Technologies/RhemaWave/dev";
  inputs.nixpkgs.follows = "nixpkgs";
};
```

Add `rhemawave` to the `outputs` function arguments, then add to a module:

```nix
{
  environment.systemPackages = [
    rhemawave.packages.x86_64-linux.default
  ];
}
```

### 4.3 GitHub Access Token

Generate a fine-grained personal access token at `github.com/settings/tokens`:
- Resource owner: `Rhema-Wave-Technologies`
- Repository access: Select `NixConfig` and `RhemaWave`
- Permissions: Contents → Read-only
- Expiration: Set as appropriate

Configure Nix to use it:

```bash
mkdir -p ~/.config/nix
echo "access-tokens = github.com=YOUR_TOKEN_HERE" > ~/.config/nix/nix.conf
```

Also copy to root (needed for `sudo` operations):

```bash
sudo mkdir -p /root/.config/nix
sudo cp ~/.config/nix/nix.conf /root/.config/nix/nix.conf
```

### 4.4 Signing Keys

If you're taking over from an existing maintainer, obtain:
- `cache-priv-key.pem` → save to `~/Keys/nix/cache/rhemawave/cache-priv-key.pem`
- `age-key.txt` → save to `~/Keys/nix/sops/age-key.txt`

If starting fresh (new cache):

```bash
mkdir -p ~/Keys/nix/cache/rhemawave
nix key generate-secret --key-name repository.rhemawavetech.com-1 > ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem > ~/Keys/nix/cache/rhemawave/cache-pub-key.pem
```

If starting fresh (new sops key):

```bash
mkdir -p ~/Keys/nix/sops
nix shell nixpkgs#age -c age-keygen -o ~/Keys/nix/sops/age-key.txt
```

Note the public key (starts with `age1...`) — it goes in `.sops.yaml`.

### 4.5 Sops Environment Variable

Add to `~/.bashrc`:

```bash
echo 'export SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt' >> ~/.bashrc
source ~/.bashrc
```

### 4.6 Clone the Repositories

```bash
mkdir -p ~/Work/RhemaWave
cd ~/Work/RhemaWave
git clone git@github.com:Rhema-Wave-Technologies/RhemaWave.git
git clone git@github.com:Rhema-Wave-Technologies/NixConfig.git
```

### 4.7 Verify Everything Works

```bash
# Test GitHub access
cd ~/Work/RhemaWave/NixConfig
nix flake show github:Rhema-Wave-Technologies/NixConfig

# Test RhemaWave build
cd ~/Work/RhemaWave/RhemaWave
nix build .#default

# Test binary runs
./result/bin/rhemawave

# Test secrets decryption
cd ~/Work/RhemaWave/NixConfig
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml
# Should open in editor with plaintext values
# Close without saving
```

---

## 5. The RhemaWave Flake (Application)

Located at `~/Work/RhemaWave/RhemaWave/flake.nix`.

### Key Points

- Uses `buildGoModule` with `vendorHash = null` (vendor directory committed to repo)
- CGo is enabled (`env.CGO_ENABLED = "1"`)
- SDL2 mixer/ttf/image flags are set in `preBuild` via `pkg-config`
- The `share/` directory is copied to `$out/share` in `postInstall`
- License is set to `unfree` — consumers must set `allowUnfreePredicate`
- Exports `nixosModules.default` (the kiosk module) alongside per-system packages and dev shells

### Flake Outputs

| Output | Description |
|--------|-------------|
| `packages.<system>.default` | The RhemaWave binary |
| `devShells.<system>.default` | Development shell with Go + SDL2 deps |
| `nixosModules.default` | NixOS module for kiosk configuration (greetd, cage, PipeWire, polkit) |

### The NixOS Module (`nixos/module.nix`)

Imported via `import ./nixos/module.nix self` — the `self` argument gives the module access to the flake's packages. The module exposes `services.rhemawave.*` options.

---

## 6. The NixConfig Flake (Kiosk System)

Located at `~/Work/RhemaWave/NixConfig/flake.nix`.

### Flake Inputs

```nix
nixpkgs          → github:NixOS/nixpkgs/nixpkgs-unstable
disko            → github:nix-community/disko
nixos-facter     → github:numtide/nixos-facter-modules
rhemawave        → github:Rhema-Wave-Technologies/RhemaWave/dev
sops-nix         → github:Mic92/sops-nix
```

All inputs follow `nixpkgs` to ensure a single nixpkgs version.

### System Configuration Modules

The NixOS system (`rhemawavekiosk`) loads these modules:
1. `disko.nixosModules.disko` — disk partitioning
2. `rhemawave.nixosModules.default` — RhemaWave kiosk module
3. `sops-nix.nixosModules.sops` — secrets management
4. `./configuration.nix` — main system config
5. Hardware facter report
6. Inline module adding `rhemawave.packages.x86_64-linux.default` to `environment.systemPackages`

---

## 7. Binary Cache Setup

### What It Is

A Nix binary cache is a static HTTP file server with a specific directory structure. When a NixOS machine needs a package, it checks each configured cache by making HTTP GET requests for `.narinfo` files. If found and the signature is valid, it downloads the pre-built `.nar.xz` archive instead of building from source.

### Directory Structure

```
/var/lib/nix-cache/
├── nix-cache-info                              # Cache metadata
├── fv5iinvsldz9l2s8ikm0mw8ykhriad1g.narinfo   # Package metadata + signature
├── nar/
│   └── 1j7d9cif4xjq7i5rw5dmivxyninh7z1cm7zvdknsp8vjd9rjmzkc.nar.xz  # Compressed package
└── ... (more .narinfo and .nar.xz files)
```

### nix-cache-info

```
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
```

This file must exist at the root. `Priority: 40` means this cache is checked after the official NixOS cache (priority 30).

### Creating the Cache (One-Time)

If setting up a new cache server:

```bash
# On the server
sudo mkdir -p /var/lib/nix-cache
cat > /var/lib/nix-cache/nix-cache-info <<EOF
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
EOF
```

Then configure Caddy (see section 8).

---

## 8. Caddy Server Configuration

The web server at `repository.rhemawavetech.com` runs Caddy with a `file_server` directive pointing at the cache directory.

### Caddyfile

```caddyfile
repository.rhemawavetech.com {
    root * /var/lib/nix-cache
    file_server browse
}
```

The `browse` directive is optional (useful for debugging). Caddy handles TLS automatically via Let's Encrypt.

### Verification

```bash
curl https://repository.rhemawavetech.com/nix-cache-info
```

Should return the three-line metadata file.

---

## 9. Signing Keys

### How Signing Works

1. The developer builds a package with `nix build`
2. The store path is signed with the private key: `nix store sign --key-file ...`
3. The signed package is exported to cache format: `nix copy --to file:// ...`
4. The `.narinfo` file contains a `Sig:` line with the key name and signature
5. Client machines verify the signature against the public key in `trusted-public-keys`

### Key Name

The key name `repository.rhemawavetech.com-1` is embedded in both the key files and the `.narinfo` signatures. If you generate new keys, use the same naming convention with an incremented number (e.g., `-2`).

### Generating New Keys

```bash
nix key generate-secret --key-name repository.rhemawavetech.com-1 > cache-priv-key.pem
nix key convert-secret-to-public < cache-priv-key.pem > cache-pub-key.pem
```

### Deriving Public Key from Private Key

```bash
nix key convert-secret-to-public < ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
```

This outputs the public key string. It should match what's in every client's `trusted-public-keys`.

### Current Public Key

```
repository.rhemawavetech.com-1:hbaEpYz5jE9SeNgkF2jAgjMtjXzruj7lEk6Lvx9ll40=
```

---

## 10. Sops-Nix Secrets Management

### How It Works

1. Secrets are encrypted in `secrets.yaml` using age public keys
2. `.sops.yaml` defines which keys can encrypt/decrypt
3. Each kiosk's SSH host key is converted to an age key for decryption
4. On activation, sops-nix decrypts secrets and places them at `/run/secrets/<name>`

### .sops.yaml

```yaml
keys:
  - &admin age1ptpn9zee8x7vjaaktx9dv65vysdjfmmgtj96wc3l49vd8g8huy6syk8l26
  - &kiosk-dev age1nmn0r66hl24wrz73q5htt0r90vfse9dvyjwae2evqwrealfgycfsy3apmy
creation_rules:
  - path_regex: secrets\.yaml$
    key_groups:
      - age:
          - *admin
          - *kiosk-dev
```

### Adding a New Kiosk

1. Get the kiosk's SSH host public key:
   ```bash
   ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub"
   ```

2. Convert to age key:
   ```bash
   nix shell nixpkgs#ssh-to-age -c ssh-to-age < <(ssh root@KIOSK_IP "cat /etc/ssh/ssh_host_ed25519_key.pub")
   ```

3. Add the age key to `.sops.yaml` under `keys:` with a descriptive anchor (e.g., `&kiosk-prod-001`)

4. Add the anchor reference to the creation rule's age list

5. Re-encrypt secrets with the new key:
   ```bash
   SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops updatekeys secrets.yaml
   ```

6. Commit and push `.sops.yaml` and `secrets.yaml`

### Editing Secrets

```bash
cd ~/Work/RhemaWave/NixConfig
SOPS_AGE_KEY_FILE=~/Keys/nix/sops/age-key.txt nix shell nixpkgs#sops -c sops secrets.yaml
```

The file opens in your editor with plaintext values. Edit, save, close — sops re-encrypts automatically.

### Secret Format for github-token

The `github-token` secret must contain the full nix.conf line (because it's included via `!include`):

```yaml
github-token: "access-tokens = github.com=ghp_yourtokenhere"
```

---

## 11. GitHub Access

### Developer Machine

The token is stored in `~/.config/nix/nix.conf`:

```
access-tokens = github.com=YOUR_TOKEN_HERE
```

Also needs to be in `/root/.config/nix/nix.conf` for `sudo` operations.

Flake URLs use the `github:` scheme (not `git+https://`), which reads the `access-tokens` setting.

### Kiosk Machines

The token is managed via sops-nix and included in `nix.conf` via `!include /run/secrets/github-token`.

### Token Scope

Fine-grained personal access token:
- Resource owner: `Rhema-Wave-Technologies`
- Repositories: `NixConfig`, `RhemaWave`
- Permissions: Contents → Read-only

---

## 12. Building and Publishing a New Release

Complete procedure, every command, in order. All commands run on your development machine.

### Step 1: Commit Application Changes

```bash
cd ~/Work/RhemaWave/RhemaWave

# If go.mod/go.sum changed:
nix shell nixpkgs#go -c go mod vendor
git add vendor/

git add -A
git commit -m "your changes"
git push origin dev
```

### Step 2: Build

```bash
nix build .#default
```

### Step 3: Sign

```bash
nix store sign --key-file ~/Keys/nix/cache/rhemawave/cache-priv-key.pem $(nix path-info ./result)
```

### Step 4: Export to Cache Format

```bash
rm -rf /tmp/nix-cache
nix copy --to file:///tmp/nix-cache ./result --secret-key-files ~/Keys/nix/cache/rhemawave/cache-priv-key.pem
```

### Step 5: Verify Signature

```bash
cat /tmp/nix-cache/*.narinfo | grep Sig
```

Must show `Sig: repository.rhemawavetech.com-1:...`

### Step 6: Upload to Server

```bash
rsync -av /tmp/nix-cache/ YOUR_USER@your-server:/var/lib/nix-cache/
```

### Step 7: Bump NixConfig

```bash
cd ~/Work/RhemaWave/NixConfig
nix flake update rhemawave
git add flake.lock
git commit -m "bump rhemawave"
git push
```

### Step 8: Wait

Kiosks pick up the update within 15 minutes via the auto-update timer.

---

## 13. Deploying a New Kiosk (nixos-anywhere)

### Prerequisites

- Target machine booted from NixOS installer USB
- SSH access to the target as root
- Target's IP address known

### Procedure

From your development machine:

```bash
cd ~/Work/RhemaWave/NixConfig

# Deploy
nix run github:nix-community/nixos-anywhere -- --flake .#rhemawavekiosk root@TARGET_IP
```

This:
1. kexec's into the NixOS installer on the target
2. Partitions the disk according to `disk-config.nix`
3. Installs NixOS with the full kiosk configuration
4. Reboots

**Note:** Uses the local flake (`.#`) so it can access local files and secrets. The `--flake` path must be the NixConfig directory.

### After Deployment

1. Get the kiosk's age key (see section 10, "Adding a New Kiosk")
2. Add it to `.sops.yaml` and re-encrypt secrets
3. Push changes
4. The kiosk will pick up the updated secrets on its next auto-update cycle

### If nixos-rebuild --target-host Fails with Bootloader Errors

Use the manual three-step deploy instead:

```bash
nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel
nix copy --to ssh://root@TARGET_IP ./result
ssh root@TARGET_IP "$(readlink -f ./result)/bin/switch-to-configuration switch"
```

---

## 14. Auto-Update System

### How It Works

Each kiosk runs a systemd timer (`kiosk-auto-update.timer`) that fires every 15 minutes. The associated service runs:

```bash
nixos-rebuild switch --flake github:Rhema-Wave-Technologies/NixConfig#rhemawavekiosk
```

This:
1. Fetches the latest NixConfig flake from GitHub (using the encrypted GitHub token)
2. Evaluates the configuration (including the pinned RhemaWave version from `flake.lock`)
3. Checks the binary cache for pre-built packages
4. Downloads any new packages
5. Switches to the new configuration

### What Triggers an Update

An update is triggered when either:
- `flake.lock` changes (new RhemaWave version pinned)
- `configuration.nix` or other NixConfig files change
- Any flake input is updated

If nothing has changed, `nixos-rebuild` is a no-op.

### Monitoring

```bash
# Check timer status
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer"

# Check last update result
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.service"

# Full update logs
ssh root@KIOSK_IP "journalctl -u kiosk-auto-update -n 100"
```

### Force Immediate Update

```bash
ssh root@KIOSK_IP "systemctl start kiosk-auto-update"
```

---

## 15. Testing and Debugging

### SSH into a Running Kiosk

Even with the kiosk locked down (cage compositor, single TTY), SSH always works:

```bash
ssh root@KIOSK_IP
```

### Check What's Running

```bash
ssh root@KIOSK_IP "systemctl status greetd"
ssh root@KIOSK_IP "systemctl status kiosk-auto-update.timer"
ssh root@KIOSK_IP "ls -la /run/secrets/"
ssh root@KIOSK_IP "cat /run/secrets/wifi-psk"  # verify decryption
```

### Check Binary Cache Connectivity

```bash
ssh root@KIOSK_IP "curl -s https://repository.rhemawavetech.com/nix-cache-info"
```

### Check GitHub Connectivity

```bash
ssh root@KIOSK_IP "cat /etc/nix/nix.conf | grep access-tokens"
```

### Restart the Kiosk Application

If the application crashes or you want to restart it without a full reboot:

```bash
ssh root@KIOSK_IP "systemctl restart greetd"
```

greetd will restart, auto-login the kiosk user, and re-launch rhemawave in cage.

### View Application Logs

```bash
ssh root@KIOSK_IP "journalctl -u greetd -n 100"
```

### Test a Configuration Change Without Pushing

Build and deploy directly from your development machine:

```bash
cd ~/Work/RhemaWave/NixConfig
nix build .#nixosConfigurations.rhemawavekiosk.config.system.build.toplevel
nix copy --to ssh://root@KIOSK_IP ./result
ssh root@KIOSK_IP "$(readlink -f ./result)/bin/switch-to-configuration switch"
```

---

## 16. Common Errors and Solutions

### `env attribute set cannot contain any attributes passed to derivation`

**Cause:** `CGO_ENABLED` set as both a top-level attribute and in `env`.
**Fix:** Use only `env.CGO_ENABLED = "1";` in the `buildGoModule` block.

### `vendor/modules.txt` inconsistency

**Cause:** `go.mod` changed but `vendor/` wasn't updated.
**Fix:** Run `nix shell nixpkgs#go -c go mod vendor`, then `git add vendor/` and commit.

### `unfree license, refusing to evaluate`

**Cause:** Missing `allowUnfreePredicate` in the nixpkgs config.
**Fix:** Add `config.allowUnfreePredicate = pkg: (nixpkgs.lib.getName pkg) == "rhemawave";` to the `pkgs` import.

### `warning: ignoring the client-specified setting 'secret-key-files'`

**Cause:** Your user isn't in `trusted-users`.
**Fix:** Add your username to `nix.settings.trusted-users` and rebuild your NixOS system.

### `invalid character in Base64 string`

**Cause:** The `trusted-public-keys` value is malformed — likely doubled key name or invisible characters.
**Fix:** Regenerate the public key: `nix key convert-secret-to-public < cache-priv-key.pem` and paste it fresh into the config.

### `attribute 'nixosModules' missing`

**Cause:** The `flake.lock` points to an old RhemaWave commit that doesn't export `nixosModules`.
**Fix:** Run `nix flake update rhemawave` in the NixConfig directory.

### `path '/home/...' is forbidden in pure evaluation mode`

**Cause:** Using absolute paths (like `builtins.readFile /home/...`) in a flake. Flakes evaluate in pure mode.
**Fix:** Use relative paths (`./file.txt`) or manage secrets with sops-nix.

### `access to absolute path '/home' is forbidden`

**Cause:** Same as above — `readFile` with an absolute path.
**Fix:** Use sops-nix for secrets or deploy with the local flake (`.#`) instead of `github:`.

### Bootloader errors during remote `nixos-rebuild --target-host`

**Cause:** GRUB can't detect the disk layout properly over SSH.
**Fix:** Use the three-step manual deploy: `nix build` → `nix copy --to ssh://` → `switch-to-configuration switch`.

### SSH host key changed warning

**Cause:** After reinstalling a kiosk, its SSH host key changes.
**Fix:** `ssh-keygen -R KIOSK_IP` to remove the old key, then reconnect.

### Auto-update not working on kiosk

**Check:**
1. Timer is active: `systemctl status kiosk-auto-update.timer`
2. Service logs: `journalctl -u kiosk-auto-update -n 50`
3. GitHub token is decrypted: `cat /etc/nix/nix.conf | grep access-tokens`
4. Network connectivity: `curl https://github.com`
5. Cache connectivity: `curl https://repository.rhemawavetech.com/nix-cache-info`