KioskConfig/PLAN-update-system.md

RadiantWave Kiosk Update System Plan

Goal: Bulletproof automatic update system for RadiantWave kiosk
Deadline: Production in 2 days
Architecture: Manifest-based app updates + Flake lock system updates + systemd service supervision
Timezone: UTC (all kiosks run in UTC)
Variants: All variants (production + dev) auto-update
Notifications: Logs only (no email/Slack)

---

Executive Summary

This plan implements a three-part update system:

1. Kiosk App Supervision: Convert the kiosk app from exec'd login shell to a supervised systemd user service with automatic restart
2. RadiantWave App Updates: Hourly manifest checks trigger updates via Attic binary cache
3. System Updates: NixOS updates on boot and nightly at 8PM UTC

All updates show Plymouth splash during transition and auto-rollback on failure.

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                         BOOT SEQUENCE                          │
├─────────────────────────────────────────────────────────────────┤
│  1. Plymouth splash (held indefinitely)                        │
│  2. getty autologin as kiosk on tty1                           │
│  3. loginShellInit runs:                                        │
│     a. Set environment variables                               │
│     b. Wait for network (up to 10s)                            │
│     c. Start radiantwave-audio-init user service               │
│     d. Start kiosk-app.service (systemd user service)          │
│     e. Quit Plymouth                                            │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                      KIOSK APP SERVICE                         │
├─────────────────────────────────────────────────────────────────┤
│  kiosk-app.service (systemd user service)                      │
│  ├─ Runs: dbus-run-session cage -- radiantwave -d -s           │
│  ├─ Restart=always, RestartSec=5                               │
│  ├─ Logs to /tmp/kiosk.log                                     │
│  └─ On failure: auto-restart (3 attempts)                      │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                     UPDATE CHECKERS                            │
├─────────────────────────────────────────────────────────────────┤
│  radiantwave-update-checker.service (hourly)                   │
│  ├─ Checks: https://attic.radiantwavetech.com/manifest.json    │
│  ├─ If new version: trigger radiantwave-upgrade.service        │
│  └─ Runs as: root (system service)                             │
│                                                                │
│  system-update-checker.service (on boot + 8PM UTC)             │
│  ├─ Triggers: nixos-upgrade.service                            │
│  └─ Runs as: root (system service)                             │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                     UPGRADE PROCESS                            │
├─────────────────────────────────────────────────────────────────┤
│  radiantwave-upgrade.service (triggered by update checker)     │
│  ├─ 1. Show Plymouth splash                                    │
│  ├─ 2. Stop kiosk-app.service                                  │
│  ├─ 3. Update flake.lock (git pull)                            │
│  ├─ 4. Run nixos-rebuild switch                                │
│  ├─ 5. Restart kiosk-app.service                               │
│  ├─ 6. Quit Plymouth                                            │
│  └─ 7. Health check (if fails: auto-rollback)                  │
└─────────────────────────────────────────────────────────────────┘

---

Component Details

1. Kiosk App Systemd Service

File: kiosk-app.nix (new module)

Purpose: Supervise the cage/radiantwave process with automatic restart

Key Configuration:

{ config, lib, pkgs, ... }:

{
  # System service (runs as root, starts user service)
  systemd.services.kiosk-app = {
    description = "RadiantWave Kiosk Application";
    after = [ "graphical.target" ];
    wantedBy = [ "graphical.target" ];
    
    # Start the user service
    serviceConfig = {
      ExecStart = "${pkgs.systemd}/bin/systemctl --user start kiosk-app-session.service";
      ExecStop = "${pkgs.systemd}/bin/systemctl --user stop kiosk-app-session.service";
      Type = "oneshot";
      RemainAfterExit = true;
    };
  };

  # User service (actual cage/radiantwave process)
  systemd.user.services.kiosk-app-session = {
    description = "RadiantWave Kiosk Session";
    after = [ "graphical-session.target" ];
    
    serviceConfig = {
      ExecStart = "${pkgs.dbus}/bin/dbus-run-session ${pkgs.cage}/bin/cage -- ${config.radiantApp}/bin/radiantwave -d -s";
      Restart = "always";
      RestartSec = 5;
      
      # Logging
      StandardOutput = "journal";
      StandardError = "journal";
      
      # Environment
      Environment = [
        "GODEBUG=cgocheck=0"
        "WAYLAND_DISPLAY=wayland-0"
        "XDG_RUNTIME_DIR=/run/user/1000"
      ];
    };
    
    # Restart policy
    unitConfig = {
      StartLimitIntervalSec = 60;
      StartLimitBurst = 3;
    };
  };
}

Changes to configuration.nix:

• Remove the exec line from environment.loginShellInit
• Add systemctl --user start kiosk-app-session.service instead
• Keep Plymouth quit after service start

Updated loginShellInit:

environment.loginShellInit = ''
  # ... existing env setup and network wait ...
  
  # Start audio init
  systemctl --user start radiantwave-audio-init.service
  
  # Start kiosk app service
  systemctl --user start kiosk-app-session.service
  
  # Wait a moment for cage to start
  sleep 2
  
  # Quit Plymouth
  sudo -n plymouth quit --wait
  
  # Keep shell alive (don't exec)
  # The systemd service manages the kiosk app
'';

---

2. Attic Cache Configuration

File: attic-cache.nix (new module)

Purpose: Configure Nix to use Attic as a binary cache

Key Configuration:

{ config, lib, pkgs, ... }:

{
  # Add Attic to trusted caches
  nix.settings = {
    trusted-caches = [
      "https://attic.radiantwavetech.com/radiantwave"
    ];
    
    # Extra substituters (for pulling)
    extra-substituters = [
      "https://attic.radiantwavetech.com/radiantwave"
    ];
    
    # Public keys (Attic uses Ed25519)
    trusted-public-keys = [
      "radiantwave:PLACEHOLDER_PUBLIC_KEY"
    ];
  };
  
  # Ensure nix can use the cache without authentication
  # (Attic is configured as public/open)
}

Note: Replace PLACEHOLDER_PUBLIC_KEY with the actual Attic public key after server setup.

---

3. RadiantWave Update Checker

File: radiantwave-update-checker.nix (new module)

Purpose: Check Attic manifest hourly for new RadiantWave versions

Key Configuration:

{ config, lib, pkgs, ... }:

{
  systemd.services.radiantwave-update-checker = {
    description = "Check for RadiantWave updates";
    
    # Run hourly
    startAt = "hourly";
    
    serviceConfig = {
      Type = "oneshot";
      
      # Check manifest and trigger upgrade if needed
      ExecStart = pkgs.writeScript "check-radiantwave-update" ''
        #!/bin/bash
        set -euo pipefail
        
        MANIFEST_URL="https://attic.radiantwavetech.com/manifest.json"
        CURRENT_VERSION_FILE="/etc/radiantwave-version"
        
        # Fetch manifest
        MANIFEST=$(curl -sf "$MANIFEST_URL" || exit 0)
        
        # Extract version
        NEW_VERSION=$(echo "$MANIFEST" | ${pkgs.jq}/bin/jq -r '.version')
        
        # Get current version
        if [ -f "$CURRENT_VERSION_FILE" ]; then
          CURRENT_VERSION=$(cat "$CURRENT_VERSION_FILE")
        else
          CURRENT_VERSION="0.0.0"
        fi
        
        # Compare versions
        if [ "$NEW_VERSION" != "$CURRENT_VERSION" ]; then
          echo "Update available: $CURRENT_VERSION -> $NEW_VERSION"
          # Trigger upgrade
          systemctl start radiantwave-upgrade.service
        else
          echo "Already up to date: $CURRENT_VERSION"
        fi
      '';
    };
  };
  
  # Upgrade service (triggered by checker)
  systemd.services.radiantwave-upgrade = {
    description = "Upgrade RadiantWave";
    
    serviceConfig = {
      Type = "oneshot";
      
      # Show Plymouth during upgrade
      ExecStartPre = "${pkgs.plymouth}/bin/plymouth show-splash || true";
      
      # Stop kiosk
      ExecStart = "${pkgs.systemd}/bin/systemctl stop kiosk-app.service";
      
      # Update flake.lock and rebuild
      ExecStart = pkgs.writeScript "upgrade-radiantwave" ''
        #!/bin/bash
        set -euo pipefail
        
        # Update flake.lock
        cd ${config.system.build.toplevel}  # or git repo path
        git pull origin main
        
        # Update radiantwave input
        nix flake update radiantwave
        
        # Rebuild and switch
        nixos-rebuild switch --flake .#${config.networking.hostName}
        
        # Update version file
        NEW_VERSION=$(curl -sf https://attic.radiantwavetech.com/manifest.json | ${pkgs.jq}/bin/jq -r '.version')
        echo "$NEW_VERSION" > /etc/radiantwave-version
      '';
      
      # Restart kiosk
      ExecStart = "${pkgs.systemd}/bin/systemctl start kiosk-app.service";
      
      # Quit Plymouth
      ExecStartPost = "${pkgs.plymouth}/bin/plymouth quit --wait || true";
      
      # Rollback on failure
      FailureAction = "none";  # Handled by healthcheck
      
      # Timeout
      TimeoutStartSec = 300;  # 5 minutes for rebuild
    };
    
    # Run after update checker
    after = [ "radiantwave-update-checker.service" ];
  };
}

---

4. System Update Configuration

File: system-updates.nix (modify existing)

Purpose: Configure NixOS system updates on boot and nightly at 8PM UTC

Changes to configuration.nix:

# Replace existing system.autoUpgrade
system.autoUpgrade = {
  enable = true;
  flake = upgradeFlakeRef;  # e.g., "git+https://gitea...KioskConfig.git#radiantwavekiosk"
  
  # Run on boot and nightly at 8PM UTC (kiosks are in UTC)
  dates = "20:00";  # 8PM UTC daily
  randomizedDelaySec = "10min";  # Avoid thundering herd
  
  flags = [ "--refresh" "-L" ];
  operation = "switch";
};

Note: The existing health check service (radiantwave-healthcheck) already runs after nixos-upgrade.service and handles rollback. Keep this as-is.

---

5. Update Rollback Mechanism

File: update-rollback.nix (modify existing health check)

Purpose: Auto-rollback on failed updates

Existing mechanism (already in configuration.nix):

systemd.services.radiantwave-healthcheck = {
  description = "Health check after upgrade";
  after = [ "nixos-upgrade.service" ];
  wantedBy = [ "nixos-upgrade.service" ];
  
  serviceConfig = {
    Type = "oneshot";
    ExecStartPre = "sleep 90";  # Wait for cage to start
    ExecStart = pkgs.writeScript "healthcheck" ''
      #!/bin/bash
      if ! pgrep -f ".radiantwave-unwrapped"; then
        echo "Health check failed, rolling back..."
        nixos-rebuild switch --rollback
      fi
    '';
  };
};

Enhancement: Add version file tracking and logging:

ExecStart = pkgs.writeScript "healthcheck" ''
  #!/bin/bash
  set -euo pipefail
  
  LOG_FILE="/var/log/radiantwave-healthcheck.log"
  
  echo "$(date): Starting health check" >> "$LOG_FILE"
  
  # Wait for cage to start
  sleep 10
  
  # Check if radiantwave is running
  if pgrep -f ".radiantwave-unwrapped" > /dev/null; then
    echo "$(date): Health check passed" >> "$LOG_FILE"
    # Update version file on success
    if [ -f "/tmp/pending-version" ]; then
      cp /tmp/pending-version /etc/radiantwave-version
    fi
  else
    echo "$(date): Health check FAILED, rolling back..." >> "$LOG_FILE"
    nixos-rebuild switch --rollback 2>> "$LOG_FILE"
    echo "$(date): Rollback complete" >> "$LOG_FILE"
  fi
'';

---

6. CI Changes to Push to Attic

File: .gitea/workflows/build-prod.yml (modify)

Purpose: Push built closures to Attic cache

Add to workflow:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # ... existing build steps ...
      
      # Push to Attic
      - name: Push to Attic cache
        run: |
          # Install attic-client
          nix profile install github:zedant/attic#attic-client
          
          # Login to Attic (token from secrets)
          attic login production https://attic.radiantwavetech.com production "${{ secrets.ATTIC_TOKEN }}"
          
          # Push closures
          attic push production ./result-kiosk
          attic push production ./result-android
          attic push production ./result-commercial
          
      # Update manifest
      - name: Update manifest
        run: |
          VERSION=$(date +%Y.%m.%d-%H%M)
          STORE_PATH=$(readlink -f ./result-kiosk)
          
          # Create manifest
          cat > manifest.json << EOF
          {
            "version": "$VERSION",
            "storePath": "$STORE_PATH",
            "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
            "commit": "${{ github.sha }}"
          }
          EOF
          
          # Upload manifest to Attic or fileserver
          # (Attic doesn't support arbitrary file uploads, so use a fileserver or git)

Note: Attic doesn't support arbitrary file uploads. Options for manifest:

1. Host on a simple fileserver (nginx, S3, etc.)
2. Commit to a git repo and serve via GitHub/Gitea Pages
3. Use Attic's labels feature (if available)

---

Implementation Steps

Phase 1: Attic Server Setup (User's Responsibility)

• Set up Attic server at attic.radiantwavetech.com
• Configure public/open access (no authentication)
• Create cache named radiantwave
• Note the public key for Nix configuration
• Test pushing a closure: attic push radiantwave /nix/store/...

Phase 2: Manifest Hosting (User's Responsibility)

• Choose manifest hosting location (fileserver, git, S3)
• Create initial manifest with current version
• Ensure manifest is accessible via HTTPS

Phase 3: Kiosk Configuration Changes

• Create kiosk-app.nix module with systemd service
• Create attic-cache.nix module for Attic configuration
• Create radiantwave-update-checker.nix module
• Modify configuration.nix:
  • Remove exec from loginShellInit
  • Add systemctl --user start kiosk-app-session.service
  • Update system.autoUpgrade schedule
• Modify rotated_display.nix to match changes
• Test locally with nixos-rebuild build

Phase 4: CI Changes

• Add Attic push step to build-prod.yml
• Add manifest update step
• Test CI pipeline with manual trigger

Phase 5: Testing

• Build ISO with changes
• Test on VM or physical hardware
• Verify kiosk starts automatically
• Test manual update trigger
• Test automatic update detection
• Test rollback on failure
• Test Plymouth splash during updates

---

Testing Checklist

Kiosk App Service

• Kiosk starts automatically on boot
• Kiosk restarts if crashed (kill -9 radiantwave)
• Logs appear in /tmp/kiosk.log and journalctl
• Audio init completes before kiosk starts

Update Detection

• Manifest check runs hourly
• System update check runs on boot
• System update check runs at 8PM UTC
• Checker logs show current vs available version

Update Process

• Plymouth splash shows during update
• Kiosk stops before rebuild
• Rebuild completes successfully
• Kiosk restarts after rebuild
• Plymouth quits after kiosk starts
• Version file updates on success

Rollback

• Health check runs after upgrade
• Rollback triggers if radiantwave not running
• Rollback completes successfully
• Logs show rollback reason

Network Failure Handling

• Update checker continues if manifest unreachable
• Rebuild continues if Attic unreachable (uses local cache)
• Kiosk restarts even if update fails

---

File Map (New/Modified)

New Files

• kiosk-app.nix - Systemd service for kiosk app
• attic-cache.nix - Attic binary cache configuration
• radiantwave-update-checker.nix - Hourly update checker
• system-updates.nix - System update schedule (or modify configuration.nix)

Modified Files

• configuration.nix - Remove exec, add service start, update autoUpgrade schedule
• rotated_display.nix - Match loginShellInit changes
• .gitea/workflows/build-prod.yml - Add Attic push step
• .gitea/workflows/build-dev.yml - Add Attic push step (optional)

---

Rollback Strategy

If the update system itself fails:

1. Manual rollback: SSH into kiosk, run nixos-rebuild switch --rollback
2. Boot previous generation: Select previous generation in systemd-boot (if timeout enabled)
3. Reinstall from ISO: Last resort, use zero-touch installer ISO

---

Monitoring and Debugging

Logs to Check

• /tmp/kiosk.log - Kiosk app output
• journalctl -u kiosk-app.service - Systemd service logs
• journalctl -u radiantwave-update-checker.service - Update checker logs
• journalctl -u radiantwave-upgrade.service - Upgrade process logs
• /var/log/radiantwave-healthcheck.log - Health check results

Manual Commands

# Check current version
cat /etc/radiantwave-version

# Check manifest manually
curl -sf https://attic.radiantwavetech.com/manifest.json | jq

# Trigger manual update
systemctl start radiantwave-upgrade.service

# Check kiosk service status
systemctl --user status kiosk-app-session.service

# View kiosk logs
journalctl -u kiosk-app.service -f

# Manual rollback
nixos-rebuild switch --rollback

---

Risk Assessment

Risk	Impact	Mitigation
Attic server unavailable	Updates fail, kiosk continues running	Update checker continues hourly, kiosk unaffected
Manifest corrupted	False update trigger	Version comparison logic, checksums
Rebuild fails	Kiosk stays on old version	Automatic rollback via health check
Kiosk crashes during update	Plymouth stays on screen	Timeout mechanisms, manual intervention
Network partition	No updates possible	Kiosk continues on current version

---

Success Criteria

• Kiosk starts automatically on boot (no manual intervention)
• Kiosk restarts automatically if crashed
• RadiantWave updates check every hour
• System updates check on boot and 8PM UTC
• Updates show Plymouth splash during transition
• Failed updates auto-rollback within 2 minutes
• All operations logged for debugging
• Zero manual intervention required for routine updates

---

Timeline

Day 1 (Today):

• Set up Attic server
• Create manifest hosting
• Implement kiosk-app.nix
• Implement attic-cache.nix
• Test kiosk service locally

Day 2 (Tomorrow):

• Implement update checker modules
• Update CI workflows
• Build test ISO
• Test on hardware
• Deploy to production

---

Questions for Clarification

1. Where will the manifest file be hosted? → Fileserver (nginx/Caddy)
2. What timezone is the kiosk in? → UTC (all kiosks)
3. Should dev variants also auto-update? → Yes, all variants
4. Do you want a manual "check for updates" button in the kiosk UI?
5. Should we notify admins of failed updates? → Logs only