# RadiantWave Kiosk Update System Plan

**Goal**: Bulletproof automatic update system for RadiantWave kiosk  
**Deadline**: Production in 2 days  
**Architecture**: Manifest-based app updates + Flake lock system updates + systemd service supervision  
**Timezone**: UTC (all kiosks run in UTC)  
**Variants**: All variants (production + dev) auto-update  
**Notifications**: Logs only (no email/Slack)

---

## Executive Summary

This plan implements a three-part update system:

1. **Kiosk App Supervision**: Convert the kiosk app from exec'd login shell to a supervised systemd user service with automatic restart
2. **RadiantWave App Updates**: Hourly manifest checks trigger updates via Attic binary cache
3. **System Updates**: NixOS updates on boot and nightly at 8PM UTC

All updates show Plymouth splash during transition and auto-rollback on failure.

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                         BOOT SEQUENCE                          │
├─────────────────────────────────────────────────────────────────┤
│  1. Plymouth splash (held indefinitely)                        │
│  2. getty autologin as kiosk on tty1                           │
│  3. loginShellInit runs:                                        │
│     a. Set environment variables                               │
│     b. Wait for network (up to 10s)                            │
│     c. Start radiantwave-audio-init user service               │
│     d. Start kiosk-app.service (systemd user service)          │
│     e. Quit Plymouth                                            │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                      KIOSK APP SERVICE                         │
├─────────────────────────────────────────────────────────────────┤
│  kiosk-app.service (systemd user service)                      │
│  ├─ Runs: dbus-run-session cage -- radiantwave -d -s           │
│  ├─ Restart=always, RestartSec=5                               │
│  ├─ Logs to /tmp/kiosk.log                                     │
│  └─ On failure: auto-restart (3 attempts)                      │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                     UPDATE CHECKERS                            │
├─────────────────────────────────────────────────────────────────┤
│  radiantwave-update-checker.service (hourly)                   │
│  ├─ Checks: https://attic.radiantwavetech.com/manifest.json    │
│  ├─ If new version: trigger radiantwave-upgrade.service        │
│  └─ Runs as: root (system service)                             │
│                                                                │
│  system-update-checker.service (on boot + 8PM UTC)             │
│  ├─ Triggers: nixos-upgrade.service                            │
│  └─ Runs as: root (system service)                             │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                     UPGRADE PROCESS                            │
├─────────────────────────────────────────────────────────────────┤
│  radiantwave-upgrade.service (triggered by update checker)     │
│  ├─ 1. Show Plymouth splash                                    │
│  ├─ 2. Stop kiosk-app.service                                  │
│  ├─ 3. Update flake.lock (git pull)                            │
│  ├─ 4. Run nixos-rebuild switch                                │
│  ├─ 5. Restart kiosk-app.service                               │
│  ├─ 6. Quit Plymouth                                            │
│  └─ 7. Health check (if fails: auto-rollback)                  │
└─────────────────────────────────────────────────────────────────┘
```

---

## Component Details

### 1. Kiosk App Systemd Service

**File**: `kiosk-app.nix` (new module)

**Purpose**: Supervise the cage/radiantwave process with automatic restart

**Key Configuration**:
```nix
{ config, lib, pkgs, ... }:

{
  # System service (runs as root, starts user service)
  systemd.services.kiosk-app = {
    description = "RadiantWave Kiosk Application";
    after = [ "graphical.target" ];
    wantedBy = [ "graphical.target" ];
    
    # Start the user service
    serviceConfig = {
      ExecStart = "${pkgs.systemd}/bin/systemctl --user start kiosk-app-session.service";
      ExecStop = "${pkgs.systemd}/bin/systemctl --user stop kiosk-app-session.service";
      Type = "oneshot";
      RemainAfterExit = true;
    };
  };

  # User service (actual cage/radiantwave process)
  systemd.user.services.kiosk-app-session = {
    description = "RadiantWave Kiosk Session";
    after = [ "graphical-session.target" ];
    
    serviceConfig = {
      ExecStart = "${pkgs.dbus}/bin/dbus-run-session ${pkgs.cage}/bin/cage -- ${config.radiantApp}/bin/radiantwave -d -s";
      Restart = "always";
      RestartSec = 5;
      
      # Logging
      StandardOutput = "journal";
      StandardError = "journal";
      
      # Environment
      Environment = [
        "GODEBUG=cgocheck=0"
        "WAYLAND_DISPLAY=wayland-0"
        "XDG_RUNTIME_DIR=/run/user/1000"
      ];
    };
    
    # Restart policy
    unitConfig = {
      StartLimitIntervalSec = 60;
      StartLimitBurst = 3;
    };
  };
}
```

**Changes to `configuration.nix`**:
- Remove the `exec` line from `environment.loginShellInit`
- Add `systemctl --user start kiosk-app-session.service` instead
- Keep Plymouth quit after service start

**Updated `loginShellInit`**:
```nix
environment.loginShellInit = ''
  # ... existing env setup and network wait ...
  
  # Start audio init
  systemctl --user start radiantwave-audio-init.service
  
  # Start kiosk app service
  systemctl --user start kiosk-app-session.service
  
  # Wait a moment for cage to start
  sleep 2
  
  # Quit Plymouth
  sudo -n plymouth quit --wait
  
  # Keep shell alive (don't exec)
  # The systemd service manages the kiosk app
'';
```

---

### 2. Attic Cache Configuration

**File**: `attic-cache.nix` (new module)

**Purpose**: Configure Nix to use Attic as a binary cache

**Key Configuration**:
```nix
{ config, lib, pkgs, ... }:

{
  # Add Attic to trusted caches
  nix.settings = {
    trusted-caches = [
      "https://attic.radiantwavetech.com/radiantwave"
    ];
    
    # Extra substituters (for pulling)
    extra-substituters = [
      "https://attic.radiantwavetech.com/radiantwave"
    ];
    
    # Public keys (Attic uses Ed25519)
    trusted-public-keys = [
      "radiantwave:PLACEHOLDER_PUBLIC_KEY"
    ];
  };
  
  # Ensure nix can use the cache without authentication
  # (Attic is configured as public/open)
}
```

**Note**: Replace `PLACEHOLDER_PUBLIC_KEY` with the actual Attic public key after server setup.

---

### 3. RadiantWave Update Checker

**File**: `radiantwave-update-checker.nix` (new module)

**Purpose**: Check Attic manifest hourly for new RadiantWave versions

**Key Configuration**:
```nix
{ config, lib, pkgs, ... }:

{
  systemd.services.radiantwave-update-checker = {
    description = "Check for RadiantWave updates";
    
    # Run hourly
    startAt = "hourly";
    
    serviceConfig = {
      Type = "oneshot";
      
      # Check manifest and trigger upgrade if needed
      ExecStart = pkgs.writeScript "check-radiantwave-update" ''
        #!/bin/bash
        set -euo pipefail
        
        MANIFEST_URL="https://attic.radiantwavetech.com/manifest.json"
        CURRENT_VERSION_FILE="/etc/radiantwave-version"
        
        # Fetch manifest
        MANIFEST=$(curl -sf "$MANIFEST_URL" || exit 0)
        
        # Extract version
        NEW_VERSION=$(echo "$MANIFEST" | ${pkgs.jq}/bin/jq -r '.version')
        
        # Get current version
        if [ -f "$CURRENT_VERSION_FILE" ]; then
          CURRENT_VERSION=$(cat "$CURRENT_VERSION_FILE")
        else
          CURRENT_VERSION="0.0.0"
        fi
        
        # Compare versions
        if [ "$NEW_VERSION" != "$CURRENT_VERSION" ]; then
          echo "Update available: $CURRENT_VERSION -> $NEW_VERSION"
          # Trigger upgrade
          systemctl start radiantwave-upgrade.service
        else
          echo "Already up to date: $CURRENT_VERSION"
        fi
      '';
    };
  };
  
  # Upgrade service (triggered by checker)
  systemd.services.radiantwave-upgrade = {
    description = "Upgrade RadiantWave";
    
    serviceConfig = {
      Type = "oneshot";
      
      # Show Plymouth during upgrade
      ExecStartPre = "${pkgs.plymouth}/bin/plymouth show-splash || true";
      
      # Stop kiosk
      ExecStart = "${pkgs.systemd}/bin/systemctl stop kiosk-app.service";
      
      # Update flake.lock and rebuild
      ExecStart = pkgs.writeScript "upgrade-radiantwave" ''
        #!/bin/bash
        set -euo pipefail
        
        # Update flake.lock
        cd ${config.system.build.toplevel}  # or git repo path
        git pull origin main
        
        # Update radiantwave input
        nix flake update radiantwave
        
        # Rebuild and switch
        nixos-rebuild switch --flake .#${config.networking.hostName}
        
        # Update version file
        NEW_VERSION=$(curl -sf https://attic.radiantwavetech.com/manifest.json | ${pkgs.jq}/bin/jq -r '.version')
        echo "$NEW_VERSION" > /etc/radiantwave-version
      '';
      
      # Restart kiosk
      ExecStart = "${pkgs.systemd}/bin/systemctl start kiosk-app.service";
      
      # Quit Plymouth
      ExecStartPost = "${pkgs.plymouth}/bin/plymouth quit --wait || true";
      
      # Rollback on failure
      FailureAction = "none";  # Handled by healthcheck
      
      # Timeout
      TimeoutStartSec = 300;  # 5 minutes for rebuild
    };
    
    # Run after update checker
    after = [ "radiantwave-update-checker.service" ];
  };
}
```

---

### 4. System Update Configuration

**File**: `system-updates.nix` (modify existing)

**Purpose**: Configure NixOS system updates on boot and nightly at 8PM UTC

**Changes to `configuration.nix`**:
```nix
# Replace existing system.autoUpgrade
system.autoUpgrade = {
  enable = true;
  flake = upgradeFlakeRef;  # e.g., "git+https://gitea...KioskConfig.git#radiantwavekiosk"
  
  # Run on boot and nightly at 8PM UTC (kiosks are in UTC)
  dates = "20:00";  # 8PM UTC daily
  randomizedDelaySec = "10min";  # Avoid thundering herd
  
  flags = [ "--refresh" "-L" ];
  operation = "switch";
};
```

**Note**: The existing health check service (`radiantwave-healthcheck`) already runs after `nixos-upgrade.service` and handles rollback. Keep this as-is.

---

### 5. Update Rollback Mechanism

**File**: `update-rollback.nix` (modify existing health check)

**Purpose**: Auto-rollback on failed updates

**Existing mechanism** (already in `configuration.nix`):
```nix
systemd.services.radiantwave-healthcheck = {
  description = "Health check after upgrade";
  after = [ "nixos-upgrade.service" ];
  wantedBy = [ "nixos-upgrade.service" ];
  
  serviceConfig = {
    Type = "oneshot";
    ExecStartPre = "sleep 90";  # Wait for cage to start
    ExecStart = pkgs.writeScript "healthcheck" ''
      #!/bin/bash
      if ! pgrep -f ".radiantwave-unwrapped"; then
        echo "Health check failed, rolling back..."
        nixos-rebuild switch --rollback
      fi
    '';
  };
};
```

**Enhancement**: Add version file tracking and logging:
```nix
ExecStart = pkgs.writeScript "healthcheck" ''
  #!/bin/bash
  set -euo pipefail
  
  LOG_FILE="/var/log/radiantwave-healthcheck.log"
  
  echo "$(date): Starting health check" >> "$LOG_FILE"
  
  # Wait for cage to start
  sleep 10
  
  # Check if radiantwave is running
  if pgrep -f ".radiantwave-unwrapped" > /dev/null; then
    echo "$(date): Health check passed" >> "$LOG_FILE"
    # Update version file on success
    if [ -f "/tmp/pending-version" ]; then
      cp /tmp/pending-version /etc/radiantwave-version
    fi
  else
    echo "$(date): Health check FAILED, rolling back..." >> "$LOG_FILE"
    nixos-rebuild switch --rollback 2>> "$LOG_FILE"
    echo "$(date): Rollback complete" >> "$LOG_FILE"
  fi
'';
```

---

### 6. CI Changes to Push to Attic

**File**: `.gitea/workflows/build-prod.yml` (modify)

**Purpose**: Push built closures to Attic cache

**Add to workflow**:
```yaml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # ... existing build steps ...
      
      # Push to Attic
      - name: Push to Attic cache
        run: |
          # Install attic-client
          nix profile install github:zedant/attic#attic-client
          
          # Login to Attic (token from secrets)
          attic login production https://attic.radiantwavetech.com production "${{ secrets.ATTIC_TOKEN }}"
          
          # Push closures
          attic push production ./result-kiosk
          attic push production ./result-android
          attic push production ./result-commercial
          
      # Update manifest
      - name: Update manifest
        run: |
          VERSION=$(date +%Y.%m.%d-%H%M)
          STORE_PATH=$(readlink -f ./result-kiosk)
          
          # Create manifest
          cat > manifest.json << EOF
          {
            "version": "$VERSION",
            "storePath": "$STORE_PATH",
            "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
            "commit": "${{ github.sha }}"
          }
          EOF
          
          # Upload manifest to Attic or fileserver
          # (Attic doesn't support arbitrary file uploads, so use a fileserver or git)
```

**Note**: Attic doesn't support arbitrary file uploads. Options for manifest:
1. Host on a simple fileserver (nginx, S3, etc.)
2. Commit to a git repo and serve via GitHub/Gitea Pages
3. Use Attic's labels feature (if available)

---

## Implementation Steps

### Phase 1: Attic Server Setup (User's Responsibility)
- [ ] Set up Attic server at `attic.radiantwavetech.com`
- [ ] Configure public/open access (no authentication)
- [ ] Create cache named `radiantwave`
- [ ] Note the public key for Nix configuration
- [ ] Test pushing a closure: `attic push radiantwave /nix/store/...`

### Phase 2: Manifest Hosting (User's Responsibility)
- [ ] Choose manifest hosting location (fileserver, git, S3)
- [ ] Create initial manifest with current version
- [ ] Ensure manifest is accessible via HTTPS

### Phase 3: Kiosk Configuration Changes
- [ ] Create `kiosk-app.nix` module with systemd service
- [ ] Create `attic-cache.nix` module for Attic configuration
- [ ] Create `radiantwave-update-checker.nix` module
- [ ] Modify `configuration.nix`:
  - Remove `exec` from `loginShellInit`
  - Add `systemctl --user start kiosk-app-session.service`
  - Update `system.autoUpgrade` schedule
- [ ] Modify `rotated_display.nix` to match changes
- [ ] Test locally with `nixos-rebuild build`

### Phase 4: CI Changes
- [ ] Add Attic push step to `build-prod.yml`
- [ ] Add manifest update step
- [ ] Test CI pipeline with manual trigger

### Phase 5: Testing
- [ ] Build ISO with changes
- [ ] Test on VM or physical hardware
- [ ] Verify kiosk starts automatically
- [ ] Test manual update trigger
- [ ] Test automatic update detection
- [ ] Test rollback on failure
- [ ] Test Plymouth splash during updates

---

## Testing Checklist

### Kiosk App Service
- [ ] Kiosk starts automatically on boot
- [ ] Kiosk restarts if crashed (kill -9 radiantwave)
- [ ] Logs appear in `/tmp/kiosk.log` and `journalctl`
- [ ] Audio init completes before kiosk starts

### Update Detection
- [ ] Manifest check runs hourly
- [ ] System update check runs on boot
- [ ] System update check runs at 8PM UTC
- [ ] Checker logs show current vs available version

### Update Process
- [ ] Plymouth splash shows during update
- [ ] Kiosk stops before rebuild
- [ ] Rebuild completes successfully
- [ ] Kiosk restarts after rebuild
- [ ] Plymouth quits after kiosk starts
- [ ] Version file updates on success

### Rollback
- [ ] Health check runs after upgrade
- [ ] Rollback triggers if radiantwave not running
- [ ] Rollback completes successfully
- [ ] Logs show rollback reason

### Network Failure Handling
- [ ] Update checker continues if manifest unreachable
- [ ] Rebuild continues if Attic unreachable (uses local cache)
- [ ] Kiosk restarts even if update fails

---

## File Map (New/Modified)

### New Files
- `kiosk-app.nix` - Systemd service for kiosk app
- `attic-cache.nix` - Attic binary cache configuration
- `radiantwave-update-checker.nix` - Hourly update checker
- `system-updates.nix` - System update schedule (or modify configuration.nix)

### Modified Files
- `configuration.nix` - Remove exec, add service start, update autoUpgrade schedule
- `rotated_display.nix` - Match loginShellInit changes
- `.gitea/workflows/build-prod.yml` - Add Attic push step
- `.gitea/workflows/build-dev.yml` - Add Attic push step (optional)

---

## Rollback Strategy

If the update system itself fails:

1. **Manual rollback**: SSH into kiosk, run `nixos-rebuild switch --rollback`
2. **Boot previous generation**: Select previous generation in systemd-boot (if timeout enabled)
3. **Reinstall from ISO**: Last resort, use zero-touch installer ISO

---

## Monitoring and Debugging

### Logs to Check
- `/tmp/kiosk.log` - Kiosk app output
- `journalctl -u kiosk-app.service` - Systemd service logs
- `journalctl -u radiantwave-update-checker.service` - Update checker logs
- `journalctl -u radiantwave-upgrade.service` - Upgrade process logs
- `/var/log/radiantwave-healthcheck.log` - Health check results

### Manual Commands
```bash
# Check current version
cat /etc/radiantwave-version

# Check manifest manually
curl -sf https://attic.radiantwavetech.com/manifest.json | jq

# Trigger manual update
systemctl start radiantwave-upgrade.service

# Check kiosk service status
systemctl --user status kiosk-app-session.service

# View kiosk logs
journalctl -u kiosk-app.service -f

# Manual rollback
nixos-rebuild switch --rollback
```

---

## Risk Assessment

| Risk | Impact | Mitigation |
|------|--------|------------|
| Attic server unavailable | Updates fail, kiosk continues running | Update checker continues hourly, kiosk unaffected |
| Manifest corrupted | False update trigger | Version comparison logic, checksums |
| Rebuild fails | Kiosk stays on old version | Automatic rollback via health check |
| Kiosk crashes during update | Plymouth stays on screen | Timeout mechanisms, manual intervention |
| Network partition | No updates possible | Kiosk continues on current version |

---

## Success Criteria

- [ ] Kiosk starts automatically on boot (no manual intervention)
- [ ] Kiosk restarts automatically if crashed
- [ ] RadiantWave updates check every hour
- [ ] System updates check on boot and 8PM UTC
- [ ] Updates show Plymouth splash during transition
- [ ] Failed updates auto-rollback within 2 minutes
- [ ] All operations logged for debugging
- [ ] Zero manual intervention required for routine updates

---

## Timeline

**Day 1 (Today)**:
- [ ] Set up Attic server
- [ ] Create manifest hosting
- [ ] Implement kiosk-app.nix
- [ ] Implement attic-cache.nix
- [ ] Test kiosk service locally

**Day 2 (Tomorrow)**:
- [ ] Implement update checker modules
- [ ] Update CI workflows
- [ ] Build test ISO
- [ ] Test on hardware
- [ ] Deploy to production

---

## Questions for Clarification

1. ~~Where will the manifest file be hosted?~~ → **Fileserver (nginx/Caddy)**
2. ~~What timezone is the kiosk in?~~ → **UTC (all kiosks)**
3. ~~Should dev variants also auto-update?~~ → **Yes, all variants**
4. Do you want a manual "check for updates" button in the kiosk UI?
5. ~~Should we notify admins of failed updates?~~ → **Logs only**