Remove unused build steps

This commit is contained in:
NeilClack
2026-08-17 13:36:37 -05:00
parent e09758a70e
commit b584016db2
2 changed files with 632 additions and 15 deletions
+632
View File
@@ -0,0 +1,632 @@
# RadiantWave Kiosk Update System Plan
**Goal**: Bulletproof automatic update system for RadiantWave kiosk
**Deadline**: Production in 2 days
**Architecture**: Manifest-based app updates + Flake lock system updates + systemd service supervision
**Timezone**: UTC (all kiosks run in UTC)
**Variants**: All variants (production + dev) auto-update
**Notifications**: Logs only (no email/Slack)
---
## Executive Summary
This plan implements a three-part update system:
1. **Kiosk App Supervision**: Convert the kiosk app from exec'd login shell to a supervised systemd user service with automatic restart
2. **RadiantWave App Updates**: Hourly manifest checks trigger updates via Attic binary cache
3. **System Updates**: NixOS updates on boot and nightly at 8PM UTC
All updates show Plymouth splash during transition and auto-rollback on failure.
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ BOOT SEQUENCE │
├─────────────────────────────────────────────────────────────────┤
│ 1. Plymouth splash (held indefinitely) │
│ 2. getty autologin as kiosk on tty1 │
│ 3. loginShellInit runs: │
│ a. Set environment variables │
│ b. Wait for network (up to 10s) │
│ c. Start radiantwave-audio-init user service │
│ d. Start kiosk-app.service (systemd user service) │
│ e. Quit Plymouth │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ KIOSK APP SERVICE │
├─────────────────────────────────────────────────────────────────┤
│ kiosk-app.service (systemd user service) │
│ ├─ Runs: dbus-run-session cage -- radiantwave -d -s │
│ ├─ Restart=always, RestartSec=5 │
│ ├─ Logs to /tmp/kiosk.log │
│ └─ On failure: auto-restart (3 attempts) │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ UPDATE CHECKERS │
├─────────────────────────────────────────────────────────────────┤
│ radiantwave-update-checker.service (hourly) │
│ ├─ Checks: https://attic.radiantwavetech.com/manifest.json │
│ ├─ If new version: trigger radiantwave-upgrade.service │
│ └─ Runs as: root (system service) │
│ │
│ system-update-checker.service (on boot + 8PM UTC) │
│ ├─ Triggers: nixos-upgrade.service │
│ └─ Runs as: root (system service) │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ UPGRADE PROCESS │
├─────────────────────────────────────────────────────────────────┤
│ radiantwave-upgrade.service (triggered by update checker) │
│ ├─ 1. Show Plymouth splash │
│ ├─ 2. Stop kiosk-app.service │
│ ├─ 3. Update flake.lock (git pull) │
│ ├─ 4. Run nixos-rebuild switch │
│ ├─ 5. Restart kiosk-app.service │
│ ├─ 6. Quit Plymouth │
│ └─ 7. Health check (if fails: auto-rollback) │
└─────────────────────────────────────────────────────────────────┘
```
---
## Component Details
### 1. Kiosk App Systemd Service
**File**: `kiosk-app.nix` (new module)
**Purpose**: Supervise the cage/radiantwave process with automatic restart
**Key Configuration**:
```nix
{ config, lib, pkgs, ... }:
{
# System service (runs as root, starts user service)
systemd.services.kiosk-app = {
description = "RadiantWave Kiosk Application";
after = [ "graphical.target" ];
wantedBy = [ "graphical.target" ];
# Start the user service
serviceConfig = {
ExecStart = "${pkgs.systemd}/bin/systemctl --user start kiosk-app-session.service";
ExecStop = "${pkgs.systemd}/bin/systemctl --user stop kiosk-app-session.service";
Type = "oneshot";
RemainAfterExit = true;
};
};
# User service (actual cage/radiantwave process)
systemd.user.services.kiosk-app-session = {
description = "RadiantWave Kiosk Session";
after = [ "graphical-session.target" ];
serviceConfig = {
ExecStart = "${pkgs.dbus}/bin/dbus-run-session ${pkgs.cage}/bin/cage -- ${config.radiantApp}/bin/radiantwave -d -s";
Restart = "always";
RestartSec = 5;
# Logging
StandardOutput = "journal";
StandardError = "journal";
# Environment
Environment = [
"GODEBUG=cgocheck=0"
"WAYLAND_DISPLAY=wayland-0"
"XDG_RUNTIME_DIR=/run/user/1000"
];
};
# Restart policy
unitConfig = {
StartLimitIntervalSec = 60;
StartLimitBurst = 3;
};
};
}
```
**Changes to `configuration.nix`**:
- Remove the `exec` line from `environment.loginShellInit`
- Add `systemctl --user start kiosk-app-session.service` instead
- Keep Plymouth quit after service start
**Updated `loginShellInit`**:
```nix
environment.loginShellInit = ''
# ... existing env setup and network wait ...
# Start audio init
systemctl --user start radiantwave-audio-init.service
# Start kiosk app service
systemctl --user start kiosk-app-session.service
# Wait a moment for cage to start
sleep 2
# Quit Plymouth
sudo -n plymouth quit --wait
# Keep shell alive (don't exec)
# The systemd service manages the kiosk app
'';
```
---
### 2. Attic Cache Configuration
**File**: `attic-cache.nix` (new module)
**Purpose**: Configure Nix to use Attic as a binary cache
**Key Configuration**:
```nix
{ config, lib, pkgs, ... }:
{
# Add Attic to trusted caches
nix.settings = {
trusted-caches = [
"https://attic.radiantwavetech.com/radiantwave"
];
# Extra substituters (for pulling)
extra-substituters = [
"https://attic.radiantwavetech.com/radiantwave"
];
# Public keys (Attic uses Ed25519)
trusted-public-keys = [
"radiantwave:PLACEHOLDER_PUBLIC_KEY"
];
};
# Ensure nix can use the cache without authentication
# (Attic is configured as public/open)
}
```
**Note**: Replace `PLACEHOLDER_PUBLIC_KEY` with the actual Attic public key after server setup.
---
### 3. RadiantWave Update Checker
**File**: `radiantwave-update-checker.nix` (new module)
**Purpose**: Check Attic manifest hourly for new RadiantWave versions
**Key Configuration**:
```nix
{ config, lib, pkgs, ... }:
{
systemd.services.radiantwave-update-checker = {
description = "Check for RadiantWave updates";
# Run hourly
startAt = "hourly";
serviceConfig = {
Type = "oneshot";
# Check manifest and trigger upgrade if needed
ExecStart = pkgs.writeScript "check-radiantwave-update" ''
#!/bin/bash
set -euo pipefail
MANIFEST_URL="https://attic.radiantwavetech.com/manifest.json"
CURRENT_VERSION_FILE="/etc/radiantwave-version"
# Fetch manifest
MANIFEST=$(curl -sf "$MANIFEST_URL" || exit 0)
# Extract version
NEW_VERSION=$(echo "$MANIFEST" | ${pkgs.jq}/bin/jq -r '.version')
# Get current version
if [ -f "$CURRENT_VERSION_FILE" ]; then
CURRENT_VERSION=$(cat "$CURRENT_VERSION_FILE")
else
CURRENT_VERSION="0.0.0"
fi
# Compare versions
if [ "$NEW_VERSION" != "$CURRENT_VERSION" ]; then
echo "Update available: $CURRENT_VERSION -> $NEW_VERSION"
# Trigger upgrade
systemctl start radiantwave-upgrade.service
else
echo "Already up to date: $CURRENT_VERSION"
fi
'';
};
};
# Upgrade service (triggered by checker)
systemd.services.radiantwave-upgrade = {
description = "Upgrade RadiantWave";
serviceConfig = {
Type = "oneshot";
# Show Plymouth during upgrade
ExecStartPre = "${pkgs.plymouth}/bin/plymouth show-splash || true";
# Stop kiosk
ExecStart = "${pkgs.systemd}/bin/systemctl stop kiosk-app.service";
# Update flake.lock and rebuild
ExecStart = pkgs.writeScript "upgrade-radiantwave" ''
#!/bin/bash
set -euo pipefail
# Update flake.lock
cd ${config.system.build.toplevel} # or git repo path
git pull origin main
# Update radiantwave input
nix flake update radiantwave
# Rebuild and switch
nixos-rebuild switch --flake .#${config.networking.hostName}
# Update version file
NEW_VERSION=$(curl -sf https://attic.radiantwavetech.com/manifest.json | ${pkgs.jq}/bin/jq -r '.version')
echo "$NEW_VERSION" > /etc/radiantwave-version
'';
# Restart kiosk
ExecStart = "${pkgs.systemd}/bin/systemctl start kiosk-app.service";
# Quit Plymouth
ExecStartPost = "${pkgs.plymouth}/bin/plymouth quit --wait || true";
# Rollback on failure
FailureAction = "none"; # Handled by healthcheck
# Timeout
TimeoutStartSec = 300; # 5 minutes for rebuild
};
# Run after update checker
after = [ "radiantwave-update-checker.service" ];
};
}
```
---
### 4. System Update Configuration
**File**: `system-updates.nix` (modify existing)
**Purpose**: Configure NixOS system updates on boot and nightly at 8PM UTC
**Changes to `configuration.nix`**:
```nix
# Replace existing system.autoUpgrade
system.autoUpgrade = {
enable = true;
flake = upgradeFlakeRef; # e.g., "git+https://gitea...KioskConfig.git#radiantwavekiosk"
# Run on boot and nightly at 8PM UTC (kiosks are in UTC)
dates = "20:00"; # 8PM UTC daily
randomizedDelaySec = "10min"; # Avoid thundering herd
flags = [ "--refresh" "-L" ];
operation = "switch";
};
```
**Note**: The existing health check service (`radiantwave-healthcheck`) already runs after `nixos-upgrade.service` and handles rollback. Keep this as-is.
---
### 5. Update Rollback Mechanism
**File**: `update-rollback.nix` (modify existing health check)
**Purpose**: Auto-rollback on failed updates
**Existing mechanism** (already in `configuration.nix`):
```nix
systemd.services.radiantwave-healthcheck = {
description = "Health check after upgrade";
after = [ "nixos-upgrade.service" ];
wantedBy = [ "nixos-upgrade.service" ];
serviceConfig = {
Type = "oneshot";
ExecStartPre = "sleep 90"; # Wait for cage to start
ExecStart = pkgs.writeScript "healthcheck" ''
#!/bin/bash
if ! pgrep -f ".radiantwave-unwrapped"; then
echo "Health check failed, rolling back..."
nixos-rebuild switch --rollback
fi
'';
};
};
```
**Enhancement**: Add version file tracking and logging:
```nix
ExecStart = pkgs.writeScript "healthcheck" ''
#!/bin/bash
set -euo pipefail
LOG_FILE="/var/log/radiantwave-healthcheck.log"
echo "$(date): Starting health check" >> "$LOG_FILE"
# Wait for cage to start
sleep 10
# Check if radiantwave is running
if pgrep -f ".radiantwave-unwrapped" > /dev/null; then
echo "$(date): Health check passed" >> "$LOG_FILE"
# Update version file on success
if [ -f "/tmp/pending-version" ]; then
cp /tmp/pending-version /etc/radiantwave-version
fi
else
echo "$(date): Health check FAILED, rolling back..." >> "$LOG_FILE"
nixos-rebuild switch --rollback 2>> "$LOG_FILE"
echo "$(date): Rollback complete" >> "$LOG_FILE"
fi
'';
```
---
### 6. CI Changes to Push to Attic
**File**: `.gitea/workflows/build-prod.yml` (modify)
**Purpose**: Push built closures to Attic cache
**Add to workflow**:
```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
# ... existing build steps ...
# Push to Attic
- name: Push to Attic cache
run: |
# Install attic-client
nix profile install github:zedant/attic#attic-client
# Login to Attic (token from secrets)
attic login production https://attic.radiantwavetech.com production "${{ secrets.ATTIC_TOKEN }}"
# Push closures
attic push production ./result-kiosk
attic push production ./result-android
attic push production ./result-commercial
# Update manifest
- name: Update manifest
run: |
VERSION=$(date +%Y.%m.%d-%H%M)
STORE_PATH=$(readlink -f ./result-kiosk)
# Create manifest
cat > manifest.json << EOF
{
"version": "$VERSION",
"storePath": "$STORE_PATH",
"timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"commit": "${{ github.sha }}"
}
EOF
# Upload manifest to Attic or fileserver
# (Attic doesn't support arbitrary file uploads, so use a fileserver or git)
```
**Note**: Attic doesn't support arbitrary file uploads. Options for manifest:
1. Host on a simple fileserver (nginx, S3, etc.)
2. Commit to a git repo and serve via GitHub/Gitea Pages
3. Use Attic's labels feature (if available)
---
## Implementation Steps
### Phase 1: Attic Server Setup (User's Responsibility)
- [ ] Set up Attic server at `attic.radiantwavetech.com`
- [ ] Configure public/open access (no authentication)
- [ ] Create cache named `radiantwave`
- [ ] Note the public key for Nix configuration
- [ ] Test pushing a closure: `attic push radiantwave /nix/store/...`
### Phase 2: Manifest Hosting (User's Responsibility)
- [ ] Choose manifest hosting location (fileserver, git, S3)
- [ ] Create initial manifest with current version
- [ ] Ensure manifest is accessible via HTTPS
### Phase 3: Kiosk Configuration Changes
- [ ] Create `kiosk-app.nix` module with systemd service
- [ ] Create `attic-cache.nix` module for Attic configuration
- [ ] Create `radiantwave-update-checker.nix` module
- [ ] Modify `configuration.nix`:
- Remove `exec` from `loginShellInit`
- Add `systemctl --user start kiosk-app-session.service`
- Update `system.autoUpgrade` schedule
- [ ] Modify `rotated_display.nix` to match changes
- [ ] Test locally with `nixos-rebuild build`
### Phase 4: CI Changes
- [ ] Add Attic push step to `build-prod.yml`
- [ ] Add manifest update step
- [ ] Test CI pipeline with manual trigger
### Phase 5: Testing
- [ ] Build ISO with changes
- [ ] Test on VM or physical hardware
- [ ] Verify kiosk starts automatically
- [ ] Test manual update trigger
- [ ] Test automatic update detection
- [ ] Test rollback on failure
- [ ] Test Plymouth splash during updates
---
## Testing Checklist
### Kiosk App Service
- [ ] Kiosk starts automatically on boot
- [ ] Kiosk restarts if crashed (kill -9 radiantwave)
- [ ] Logs appear in `/tmp/kiosk.log` and `journalctl`
- [ ] Audio init completes before kiosk starts
### Update Detection
- [ ] Manifest check runs hourly
- [ ] System update check runs on boot
- [ ] System update check runs at 8PM UTC
- [ ] Checker logs show current vs available version
### Update Process
- [ ] Plymouth splash shows during update
- [ ] Kiosk stops before rebuild
- [ ] Rebuild completes successfully
- [ ] Kiosk restarts after rebuild
- [ ] Plymouth quits after kiosk starts
- [ ] Version file updates on success
### Rollback
- [ ] Health check runs after upgrade
- [ ] Rollback triggers if radiantwave not running
- [ ] Rollback completes successfully
- [ ] Logs show rollback reason
### Network Failure Handling
- [ ] Update checker continues if manifest unreachable
- [ ] Rebuild continues if Attic unreachable (uses local cache)
- [ ] Kiosk restarts even if update fails
---
## File Map (New/Modified)
### New Files
- `kiosk-app.nix` - Systemd service for kiosk app
- `attic-cache.nix` - Attic binary cache configuration
- `radiantwave-update-checker.nix` - Hourly update checker
- `system-updates.nix` - System update schedule (or modify configuration.nix)
### Modified Files
- `configuration.nix` - Remove exec, add service start, update autoUpgrade schedule
- `rotated_display.nix` - Match loginShellInit changes
- `.gitea/workflows/build-prod.yml` - Add Attic push step
- `.gitea/workflows/build-dev.yml` - Add Attic push step (optional)
---
## Rollback Strategy
If the update system itself fails:
1. **Manual rollback**: SSH into kiosk, run `nixos-rebuild switch --rollback`
2. **Boot previous generation**: Select previous generation in systemd-boot (if timeout enabled)
3. **Reinstall from ISO**: Last resort, use zero-touch installer ISO
---
## Monitoring and Debugging
### Logs to Check
- `/tmp/kiosk.log` - Kiosk app output
- `journalctl -u kiosk-app.service` - Systemd service logs
- `journalctl -u radiantwave-update-checker.service` - Update checker logs
- `journalctl -u radiantwave-upgrade.service` - Upgrade process logs
- `/var/log/radiantwave-healthcheck.log` - Health check results
### Manual Commands
```bash
# Check current version
cat /etc/radiantwave-version
# Check manifest manually
curl -sf https://attic.radiantwavetech.com/manifest.json | jq
# Trigger manual update
systemctl start radiantwave-upgrade.service
# Check kiosk service status
systemctl --user status kiosk-app-session.service
# View kiosk logs
journalctl -u kiosk-app.service -f
# Manual rollback
nixos-rebuild switch --rollback
```
---
## Risk Assessment
| Risk | Impact | Mitigation |
|------|--------|------------|
| Attic server unavailable | Updates fail, kiosk continues running | Update checker continues hourly, kiosk unaffected |
| Manifest corrupted | False update trigger | Version comparison logic, checksums |
| Rebuild fails | Kiosk stays on old version | Automatic rollback via health check |
| Kiosk crashes during update | Plymouth stays on screen | Timeout mechanisms, manual intervention |
| Network partition | No updates possible | Kiosk continues on current version |
---
## Success Criteria
- [ ] Kiosk starts automatically on boot (no manual intervention)
- [ ] Kiosk restarts automatically if crashed
- [ ] RadiantWave updates check every hour
- [ ] System updates check on boot and 8PM UTC
- [ ] Updates show Plymouth splash during transition
- [ ] Failed updates auto-rollback within 2 minutes
- [ ] All operations logged for debugging
- [ ] Zero manual intervention required for routine updates
---
## Timeline
**Day 1 (Today)**:
- [ ] Set up Attic server
- [ ] Create manifest hosting
- [ ] Implement kiosk-app.nix
- [ ] Implement attic-cache.nix
- [ ] Test kiosk service locally
**Day 2 (Tomorrow)**:
- [ ] Implement update checker modules
- [ ] Update CI workflows
- [ ] Build test ISO
- [ ] Test on hardware
- [ ] Deploy to production
---
## Questions for Clarification
1. ~~Where will the manifest file be hosted?~~ → **Fileserver (nginx/Caddy)**
2. ~~What timezone is the kiosk in?~~ → **UTC (all kiosks)**
3. ~~Should dev variants also auto-update?~~ → **Yes, all variants**
4. Do you want a manual "check for updates" button in the kiosk UI?
5. ~~Should we notify admins of failed updates?~~ → **Logs only**