diff --git a/.gitea/workflows/build-prod.yml b/.gitea/workflows/build-prod.yml index 90effb0..95bbe76 100644 --- a/.gitea/workflows/build-prod.yml +++ b/.gitea/workflows/build-prod.yml @@ -9,11 +9,6 @@ jobs: - name: Checkout run: git clone --depth 1 ssh://git@gitea.radiantwavetech.com:2222/RadiantWaveTech/KioskConfig.git . - - name: Update radiantwave - run: | - . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh - nix flake update radiantwave - - name: Prepare output directory run: mkdir -p /home/user/radiantwave @@ -105,16 +100,6 @@ jobs: # - name: Cleanup Raspberry Pi image # run: rm -f "/home/user/radiantwave/${RPI_IMAGE_FILENAME}" - - name: Commit updated flake.lock - run: | - git config user.email "ci@radiantwavetech.com" - git config user.name "RadiantWave CI" - git remote set-url origin https://oauth2:${{ secrets.GIT_TOKEN }}@gitea.radiantwavetech.com/RadiantWaveTech/kioskconfig.git - - git add flake.lock - git commit -m "chore: update radiantwave-dev to latest" || echo "Nothing to commit" - git push - # - name: Notify # if: always() # uses: https://github.com/dawidd6/action-send-mail@v3 diff --git a/PLAN-update-system.md b/PLAN-update-system.md new file mode 100644 index 0000000..314d610 --- /dev/null +++ b/PLAN-update-system.md @@ -0,0 +1,632 @@ +# RadiantWave Kiosk Update System Plan + +**Goal**: Bulletproof automatic update system for RadiantWave kiosk +**Deadline**: Production in 2 days +**Architecture**: Manifest-based app updates + Flake lock system updates + systemd service supervision +**Timezone**: UTC (all kiosks run in UTC) +**Variants**: All variants (production + dev) auto-update +**Notifications**: Logs only (no email/Slack) + +--- + +## Executive Summary + +This plan implements a three-part update system: + +1. **Kiosk App Supervision**: Convert the kiosk app from exec'd login shell to a supervised systemd user service with automatic restart +2. **RadiantWave App Updates**: Hourly manifest checks trigger updates via Attic binary cache +3. **System Updates**: NixOS updates on boot and nightly at 8PM UTC + +All updates show Plymouth splash during transition and auto-rollback on failure. + +--- + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ BOOT SEQUENCE │ +├─────────────────────────────────────────────────────────────────┤ +│ 1. Plymouth splash (held indefinitely) │ +│ 2. getty autologin as kiosk on tty1 │ +│ 3. loginShellInit runs: │ +│ a. Set environment variables │ +│ b. Wait for network (up to 10s) │ +│ c. Start radiantwave-audio-init user service │ +│ d. Start kiosk-app.service (systemd user service) │ +│ e. Quit Plymouth │ +└─────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────┐ +│ KIOSK APP SERVICE │ +├─────────────────────────────────────────────────────────────────┤ +│ kiosk-app.service (systemd user service) │ +│ ├─ Runs: dbus-run-session cage -- radiantwave -d -s │ +│ ├─ Restart=always, RestartSec=5 │ +│ ├─ Logs to /tmp/kiosk.log │ +│ └─ On failure: auto-restart (3 attempts) │ +└─────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────┐ +│ UPDATE CHECKERS │ +├─────────────────────────────────────────────────────────────────┤ +│ radiantwave-update-checker.service (hourly) │ +│ ├─ Checks: https://attic.radiantwavetech.com/manifest.json │ +│ ├─ If new version: trigger radiantwave-upgrade.service │ +│ └─ Runs as: root (system service) │ +│ │ +│ system-update-checker.service (on boot + 8PM UTC) │ +│ ├─ Triggers: nixos-upgrade.service │ +│ └─ Runs as: root (system service) │ +└─────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────┐ +│ UPGRADE PROCESS │ +├─────────────────────────────────────────────────────────────────┤ +│ radiantwave-upgrade.service (triggered by update checker) │ +│ ├─ 1. Show Plymouth splash │ +│ ├─ 2. Stop kiosk-app.service │ +│ ├─ 3. Update flake.lock (git pull) │ +│ ├─ 4. Run nixos-rebuild switch │ +│ ├─ 5. Restart kiosk-app.service │ +│ ├─ 6. Quit Plymouth │ +│ └─ 7. Health check (if fails: auto-rollback) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Component Details + +### 1. Kiosk App Systemd Service + +**File**: `kiosk-app.nix` (new module) + +**Purpose**: Supervise the cage/radiantwave process with automatic restart + +**Key Configuration**: +```nix +{ config, lib, pkgs, ... }: + +{ + # System service (runs as root, starts user service) + systemd.services.kiosk-app = { + description = "RadiantWave Kiosk Application"; + after = [ "graphical.target" ]; + wantedBy = [ "graphical.target" ]; + + # Start the user service + serviceConfig = { + ExecStart = "${pkgs.systemd}/bin/systemctl --user start kiosk-app-session.service"; + ExecStop = "${pkgs.systemd}/bin/systemctl --user stop kiosk-app-session.service"; + Type = "oneshot"; + RemainAfterExit = true; + }; + }; + + # User service (actual cage/radiantwave process) + systemd.user.services.kiosk-app-session = { + description = "RadiantWave Kiosk Session"; + after = [ "graphical-session.target" ]; + + serviceConfig = { + ExecStart = "${pkgs.dbus}/bin/dbus-run-session ${pkgs.cage}/bin/cage -- ${config.radiantApp}/bin/radiantwave -d -s"; + Restart = "always"; + RestartSec = 5; + + # Logging + StandardOutput = "journal"; + StandardError = "journal"; + + # Environment + Environment = [ + "GODEBUG=cgocheck=0" + "WAYLAND_DISPLAY=wayland-0" + "XDG_RUNTIME_DIR=/run/user/1000" + ]; + }; + + # Restart policy + unitConfig = { + StartLimitIntervalSec = 60; + StartLimitBurst = 3; + }; + }; +} +``` + +**Changes to `configuration.nix`**: +- Remove the `exec` line from `environment.loginShellInit` +- Add `systemctl --user start kiosk-app-session.service` instead +- Keep Plymouth quit after service start + +**Updated `loginShellInit`**: +```nix +environment.loginShellInit = '' + # ... existing env setup and network wait ... + + # Start audio init + systemctl --user start radiantwave-audio-init.service + + # Start kiosk app service + systemctl --user start kiosk-app-session.service + + # Wait a moment for cage to start + sleep 2 + + # Quit Plymouth + sudo -n plymouth quit --wait + + # Keep shell alive (don't exec) + # The systemd service manages the kiosk app +''; +``` + +--- + +### 2. Attic Cache Configuration + +**File**: `attic-cache.nix` (new module) + +**Purpose**: Configure Nix to use Attic as a binary cache + +**Key Configuration**: +```nix +{ config, lib, pkgs, ... }: + +{ + # Add Attic to trusted caches + nix.settings = { + trusted-caches = [ + "https://attic.radiantwavetech.com/radiantwave" + ]; + + # Extra substituters (for pulling) + extra-substituters = [ + "https://attic.radiantwavetech.com/radiantwave" + ]; + + # Public keys (Attic uses Ed25519) + trusted-public-keys = [ + "radiantwave:PLACEHOLDER_PUBLIC_KEY" + ]; + }; + + # Ensure nix can use the cache without authentication + # (Attic is configured as public/open) +} +``` + +**Note**: Replace `PLACEHOLDER_PUBLIC_KEY` with the actual Attic public key after server setup. + +--- + +### 3. RadiantWave Update Checker + +**File**: `radiantwave-update-checker.nix` (new module) + +**Purpose**: Check Attic manifest hourly for new RadiantWave versions + +**Key Configuration**: +```nix +{ config, lib, pkgs, ... }: + +{ + systemd.services.radiantwave-update-checker = { + description = "Check for RadiantWave updates"; + + # Run hourly + startAt = "hourly"; + + serviceConfig = { + Type = "oneshot"; + + # Check manifest and trigger upgrade if needed + ExecStart = pkgs.writeScript "check-radiantwave-update" '' + #!/bin/bash + set -euo pipefail + + MANIFEST_URL="https://attic.radiantwavetech.com/manifest.json" + CURRENT_VERSION_FILE="/etc/radiantwave-version" + + # Fetch manifest + MANIFEST=$(curl -sf "$MANIFEST_URL" || exit 0) + + # Extract version + NEW_VERSION=$(echo "$MANIFEST" | ${pkgs.jq}/bin/jq -r '.version') + + # Get current version + if [ -f "$CURRENT_VERSION_FILE" ]; then + CURRENT_VERSION=$(cat "$CURRENT_VERSION_FILE") + else + CURRENT_VERSION="0.0.0" + fi + + # Compare versions + if [ "$NEW_VERSION" != "$CURRENT_VERSION" ]; then + echo "Update available: $CURRENT_VERSION -> $NEW_VERSION" + # Trigger upgrade + systemctl start radiantwave-upgrade.service + else + echo "Already up to date: $CURRENT_VERSION" + fi + ''; + }; + }; + + # Upgrade service (triggered by checker) + systemd.services.radiantwave-upgrade = { + description = "Upgrade RadiantWave"; + + serviceConfig = { + Type = "oneshot"; + + # Show Plymouth during upgrade + ExecStartPre = "${pkgs.plymouth}/bin/plymouth show-splash || true"; + + # Stop kiosk + ExecStart = "${pkgs.systemd}/bin/systemctl stop kiosk-app.service"; + + # Update flake.lock and rebuild + ExecStart = pkgs.writeScript "upgrade-radiantwave" '' + #!/bin/bash + set -euo pipefail + + # Update flake.lock + cd ${config.system.build.toplevel} # or git repo path + git pull origin main + + # Update radiantwave input + nix flake update radiantwave + + # Rebuild and switch + nixos-rebuild switch --flake .#${config.networking.hostName} + + # Update version file + NEW_VERSION=$(curl -sf https://attic.radiantwavetech.com/manifest.json | ${pkgs.jq}/bin/jq -r '.version') + echo "$NEW_VERSION" > /etc/radiantwave-version + ''; + + # Restart kiosk + ExecStart = "${pkgs.systemd}/bin/systemctl start kiosk-app.service"; + + # Quit Plymouth + ExecStartPost = "${pkgs.plymouth}/bin/plymouth quit --wait || true"; + + # Rollback on failure + FailureAction = "none"; # Handled by healthcheck + + # Timeout + TimeoutStartSec = 300; # 5 minutes for rebuild + }; + + # Run after update checker + after = [ "radiantwave-update-checker.service" ]; + }; +} +``` + +--- + +### 4. System Update Configuration + +**File**: `system-updates.nix` (modify existing) + +**Purpose**: Configure NixOS system updates on boot and nightly at 8PM UTC + +**Changes to `configuration.nix`**: +```nix +# Replace existing system.autoUpgrade +system.autoUpgrade = { + enable = true; + flake = upgradeFlakeRef; # e.g., "git+https://gitea...KioskConfig.git#radiantwavekiosk" + + # Run on boot and nightly at 8PM UTC (kiosks are in UTC) + dates = "20:00"; # 8PM UTC daily + randomizedDelaySec = "10min"; # Avoid thundering herd + + flags = [ "--refresh" "-L" ]; + operation = "switch"; +}; +``` + +**Note**: The existing health check service (`radiantwave-healthcheck`) already runs after `nixos-upgrade.service` and handles rollback. Keep this as-is. + +--- + +### 5. Update Rollback Mechanism + +**File**: `update-rollback.nix` (modify existing health check) + +**Purpose**: Auto-rollback on failed updates + +**Existing mechanism** (already in `configuration.nix`): +```nix +systemd.services.radiantwave-healthcheck = { + description = "Health check after upgrade"; + after = [ "nixos-upgrade.service" ]; + wantedBy = [ "nixos-upgrade.service" ]; + + serviceConfig = { + Type = "oneshot"; + ExecStartPre = "sleep 90"; # Wait for cage to start + ExecStart = pkgs.writeScript "healthcheck" '' + #!/bin/bash + if ! pgrep -f ".radiantwave-unwrapped"; then + echo "Health check failed, rolling back..." + nixos-rebuild switch --rollback + fi + ''; + }; +}; +``` + +**Enhancement**: Add version file tracking and logging: +```nix +ExecStart = pkgs.writeScript "healthcheck" '' + #!/bin/bash + set -euo pipefail + + LOG_FILE="/var/log/radiantwave-healthcheck.log" + + echo "$(date): Starting health check" >> "$LOG_FILE" + + # Wait for cage to start + sleep 10 + + # Check if radiantwave is running + if pgrep -f ".radiantwave-unwrapped" > /dev/null; then + echo "$(date): Health check passed" >> "$LOG_FILE" + # Update version file on success + if [ -f "/tmp/pending-version" ]; then + cp /tmp/pending-version /etc/radiantwave-version + fi + else + echo "$(date): Health check FAILED, rolling back..." >> "$LOG_FILE" + nixos-rebuild switch --rollback 2>> "$LOG_FILE" + echo "$(date): Rollback complete" >> "$LOG_FILE" + fi +''; +``` + +--- + +### 6. CI Changes to Push to Attic + +**File**: `.gitea/workflows/build-prod.yml` (modify) + +**Purpose**: Push built closures to Attic cache + +**Add to workflow**: +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + # ... existing build steps ... + + # Push to Attic + - name: Push to Attic cache + run: | + # Install attic-client + nix profile install github:zedant/attic#attic-client + + # Login to Attic (token from secrets) + attic login production https://attic.radiantwavetech.com production "${{ secrets.ATTIC_TOKEN }}" + + # Push closures + attic push production ./result-kiosk + attic push production ./result-android + attic push production ./result-commercial + + # Update manifest + - name: Update manifest + run: | + VERSION=$(date +%Y.%m.%d-%H%M) + STORE_PATH=$(readlink -f ./result-kiosk) + + # Create manifest + cat > manifest.json << EOF + { + "version": "$VERSION", + "storePath": "$STORE_PATH", + "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", + "commit": "${{ github.sha }}" + } + EOF + + # Upload manifest to Attic or fileserver + # (Attic doesn't support arbitrary file uploads, so use a fileserver or git) +``` + +**Note**: Attic doesn't support arbitrary file uploads. Options for manifest: +1. Host on a simple fileserver (nginx, S3, etc.) +2. Commit to a git repo and serve via GitHub/Gitea Pages +3. Use Attic's labels feature (if available) + +--- + +## Implementation Steps + +### Phase 1: Attic Server Setup (User's Responsibility) +- [ ] Set up Attic server at `attic.radiantwavetech.com` +- [ ] Configure public/open access (no authentication) +- [ ] Create cache named `radiantwave` +- [ ] Note the public key for Nix configuration +- [ ] Test pushing a closure: `attic push radiantwave /nix/store/...` + +### Phase 2: Manifest Hosting (User's Responsibility) +- [ ] Choose manifest hosting location (fileserver, git, S3) +- [ ] Create initial manifest with current version +- [ ] Ensure manifest is accessible via HTTPS + +### Phase 3: Kiosk Configuration Changes +- [ ] Create `kiosk-app.nix` module with systemd service +- [ ] Create `attic-cache.nix` module for Attic configuration +- [ ] Create `radiantwave-update-checker.nix` module +- [ ] Modify `configuration.nix`: + - Remove `exec` from `loginShellInit` + - Add `systemctl --user start kiosk-app-session.service` + - Update `system.autoUpgrade` schedule +- [ ] Modify `rotated_display.nix` to match changes +- [ ] Test locally with `nixos-rebuild build` + +### Phase 4: CI Changes +- [ ] Add Attic push step to `build-prod.yml` +- [ ] Add manifest update step +- [ ] Test CI pipeline with manual trigger + +### Phase 5: Testing +- [ ] Build ISO with changes +- [ ] Test on VM or physical hardware +- [ ] Verify kiosk starts automatically +- [ ] Test manual update trigger +- [ ] Test automatic update detection +- [ ] Test rollback on failure +- [ ] Test Plymouth splash during updates + +--- + +## Testing Checklist + +### Kiosk App Service +- [ ] Kiosk starts automatically on boot +- [ ] Kiosk restarts if crashed (kill -9 radiantwave) +- [ ] Logs appear in `/tmp/kiosk.log` and `journalctl` +- [ ] Audio init completes before kiosk starts + +### Update Detection +- [ ] Manifest check runs hourly +- [ ] System update check runs on boot +- [ ] System update check runs at 8PM UTC +- [ ] Checker logs show current vs available version + +### Update Process +- [ ] Plymouth splash shows during update +- [ ] Kiosk stops before rebuild +- [ ] Rebuild completes successfully +- [ ] Kiosk restarts after rebuild +- [ ] Plymouth quits after kiosk starts +- [ ] Version file updates on success + +### Rollback +- [ ] Health check runs after upgrade +- [ ] Rollback triggers if radiantwave not running +- [ ] Rollback completes successfully +- [ ] Logs show rollback reason + +### Network Failure Handling +- [ ] Update checker continues if manifest unreachable +- [ ] Rebuild continues if Attic unreachable (uses local cache) +- [ ] Kiosk restarts even if update fails + +--- + +## File Map (New/Modified) + +### New Files +- `kiosk-app.nix` - Systemd service for kiosk app +- `attic-cache.nix` - Attic binary cache configuration +- `radiantwave-update-checker.nix` - Hourly update checker +- `system-updates.nix` - System update schedule (or modify configuration.nix) + +### Modified Files +- `configuration.nix` - Remove exec, add service start, update autoUpgrade schedule +- `rotated_display.nix` - Match loginShellInit changes +- `.gitea/workflows/build-prod.yml` - Add Attic push step +- `.gitea/workflows/build-dev.yml` - Add Attic push step (optional) + +--- + +## Rollback Strategy + +If the update system itself fails: + +1. **Manual rollback**: SSH into kiosk, run `nixos-rebuild switch --rollback` +2. **Boot previous generation**: Select previous generation in systemd-boot (if timeout enabled) +3. **Reinstall from ISO**: Last resort, use zero-touch installer ISO + +--- + +## Monitoring and Debugging + +### Logs to Check +- `/tmp/kiosk.log` - Kiosk app output +- `journalctl -u kiosk-app.service` - Systemd service logs +- `journalctl -u radiantwave-update-checker.service` - Update checker logs +- `journalctl -u radiantwave-upgrade.service` - Upgrade process logs +- `/var/log/radiantwave-healthcheck.log` - Health check results + +### Manual Commands +```bash +# Check current version +cat /etc/radiantwave-version + +# Check manifest manually +curl -sf https://attic.radiantwavetech.com/manifest.json | jq + +# Trigger manual update +systemctl start radiantwave-upgrade.service + +# Check kiosk service status +systemctl --user status kiosk-app-session.service + +# View kiosk logs +journalctl -u kiosk-app.service -f + +# Manual rollback +nixos-rebuild switch --rollback +``` + +--- + +## Risk Assessment + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Attic server unavailable | Updates fail, kiosk continues running | Update checker continues hourly, kiosk unaffected | +| Manifest corrupted | False update trigger | Version comparison logic, checksums | +| Rebuild fails | Kiosk stays on old version | Automatic rollback via health check | +| Kiosk crashes during update | Plymouth stays on screen | Timeout mechanisms, manual intervention | +| Network partition | No updates possible | Kiosk continues on current version | + +--- + +## Success Criteria + +- [ ] Kiosk starts automatically on boot (no manual intervention) +- [ ] Kiosk restarts automatically if crashed +- [ ] RadiantWave updates check every hour +- [ ] System updates check on boot and 8PM UTC +- [ ] Updates show Plymouth splash during transition +- [ ] Failed updates auto-rollback within 2 minutes +- [ ] All operations logged for debugging +- [ ] Zero manual intervention required for routine updates + +--- + +## Timeline + +**Day 1 (Today)**: +- [ ] Set up Attic server +- [ ] Create manifest hosting +- [ ] Implement kiosk-app.nix +- [ ] Implement attic-cache.nix +- [ ] Test kiosk service locally + +**Day 2 (Tomorrow)**: +- [ ] Implement update checker modules +- [ ] Update CI workflows +- [ ] Build test ISO +- [ ] Test on hardware +- [ ] Deploy to production + +--- + +## Questions for Clarification + +1. ~~Where will the manifest file be hosted?~~ → **Fileserver (nginx/Caddy)** +2. ~~What timezone is the kiosk in?~~ → **UTC (all kiosks)** +3. ~~Should dev variants also auto-update?~~ → **Yes, all variants** +4. Do you want a manual "check for updates" button in the kiosk UI? +5. ~~Should we notify admins of failed updates?~~ → **Logs only**