Backup & Restore Guide

This guide covers all aspects of backing up and restoring your STING installation, including encrypted backups, automated offsite sync, retention policies, and disaster recovery.

Overview

STING’s backup system provides:

Encrypted backups — AES-256-CBC encryption with PBKDF2 key derivation
Automated scheduling — Cron-based backup with configurable retention
Remote sync — Offsite backup via SSH, rsync, or S3
Separate retention — Independent local (14 days) and remote (30 days) retention
Health monitoring — Alerts for stale or corrupted backups
Vault integration — Separate HashiCorp Vault snapshot backups
Safe restore — Atomic restore with automatic rollback on failure

Quick Start

All backup commands require sudo:

# Create a standard backup
sudo msting backup

# Create an encrypted backup (recommended)
sudo msting backup --encrypt

# Check backup status
sudo msting status

# Restore from a backup
sudo msting restore /opt/sting-backups/sting_backup_20260314.tar.gz

What Gets Backed Up

Component	Description	Included In
PostgreSQL Database	All three databases (sting_app, kratos_db, sting_messaging)	Main backup
Configuration Files	`config.yml`, `docker-compose.yml`, Kratos config	Main backup
Environment Files	All `env/*.env` files with service credentials	Main backup
SSL Certificates	TLS certificates and keys	Main backup
Docker Volumes	config_data, vault_data, chroma_data, knowledge_data, logs	Main backup
Vault Secrets	HashiCorp Vault data	Separate Vault backup

Backup Locations

Location	Purpose
`/opt/sting-backups/`	Main backup storage (configurable)
`/opt/sting-ce/conf/secrets/backup_key.txt`	Encryption key (600 root:root)
`/var/log/sting-backup-sync.log`	Remote sync log

Configuration

All backup settings live in /opt/sting-ce/conf/config.yml under the backup: section. The backup-sync script reads these values at runtime — no hardcoded paths or credentials.

backup:
  enabled: true
  default_directory: /opt/sting-backups
  compression_level: 5

  # Retention policy for main backup rotation
  retention:
    count: 5              # Keep last N backups
    max_age_days: 30      # Maximum age in days (0 = no limit)

  # Local retention (applied by backup-sync cron)
  local_retention:
    days: 14              # Keep local backups for N days (0 = use count only)

  # Files and patterns to exclude from backup
  exclude_patterns:
    - "*.tmp"
    - "*.log"
    - "node_modules"
    - "venv"
    - ".git"
    - "__pycache__"
    - "*.pyc"
    - "frontend/node_modules"
    - "models"

  # Encryption settings
  encryption:
    enabled: false        # Enable AES-256-CBC encryption
    keychain: true        # Store key in system keychain

  # Backup verification
  verification:
    enabled: true         # Verify backup integrity after creation
    on_create: true       # Verify when creating backup
    scheduled: false      # Run verification in scheduled jobs

  # Remote/offsite backup configuration
  remote:
    enabled: false        # Enable offsite sync
    type: "ssh"           # ssh, rsync, s3
    destination: ""       # Hostname or S3 bucket
    user: ""              # SSH/rsync user
    password: ""          # FTP password (or use VAULT)
    port: 22              # SSH port
    ssh_key: ""           # Path to SSH private key
    path: ""              # Remote directory (relative to home for ssh/rsync)
    schedule: "0 3 * * *" # Cron schedule for automated sync
    ssl: true             # Use SSL/TLS for FTP
    retention:
      count: 10           # Keep more backups offsite
      days: 30            # Remote retention in days

  # Vault backup settings
  vault:
    backup_enabled: true
    directory: /vault/backups
    retention_days: 7

  # Notification settings
  notifications:
    enabled: false
    webhook_url: ""       # Webhook for alerts
    email: ""             # Email for alerts

Backup Commands

msting Commands

# Standard backup
sudo msting backup

# Encrypted backup (AES-256-CBC with PBKDF2, 100K iterations)
sudo msting backup --encrypt

# Restore from backup (atomic with auto-rollback)
sudo msting restore /opt/sting-backups/sting_backup_20260314.tar.gz

# Restore from encrypted backup
sudo msting restore /opt/sting-backups/sting_backup_20260314.tar.gz.enc

Backup Wrapper (Advanced)

# Standard backup
./scripts/backup/backup-wrapper.sh backup

# Encrypted backup
./scripts/backup/backup-wrapper.sh backup --encrypt

# Backup with custom retention
./scripts/backup/backup-wrapper.sh backup --retention 10 --days 14

# Verify backup integrity
./scripts/backup/backup-wrapper.sh verify

# Show backup status and statistics
./scripts/backup/backup-wrapper.sh status

# Rotate old backups manually
./scripts/backup/backup-wrapper.sh rotate

# Backup Vault only
./scripts/backup/backup-wrapper.sh vault

Encryption

How It Works

Encrypted backups use AES-256-CBC with PBKDF2 key derivation (100,000 iterations):

# Encryption (performed automatically by msting backup --encrypt)
openssl enc -aes-256-cbc -salt -pbkdf2 -iter 100000 -pass file:<keyfile> -in backup.tar.gz -out backup.tar.gz.enc

# Decryption (performed automatically by msting restore)
openssl enc -d -aes-256-cbc -pbkdf2 -iter 100000 -pass file:<keyfile> -in backup.tar.gz.enc -out backup.tar.gz

Encryption Key Storage

The encryption key is stored at:

/opt/sting-ce/conf/secrets/backup_key.txt

Permissions: 600 root:root — only root can read
Persists across reboots — stored on disk (not in kernel keyring)
Auto-generated on first encrypted backup via openssl rand -base64 32

The encryption key is required to restore encrypted backups. If the key is lost, encrypted backups cannot be decrypted. Always store a copy of this key in a separate secure location (password manager, encrypted USB, etc.).

Exporting the Key

# View the current encryption key (as root)
sudo cat /opt/sting-ce/conf/secrets/backup_key.txt

# Copy to a secure location
sudo cp /opt/sting-ce/conf/secrets/backup_key.txt /secure/location/

Store the key separately from your backups — if backups and key are in the same location, a single point of failure defeats the purpose of encryption.

Remote/Offsite Sync

Remote sync is handled by the backup-sync.sh script, which reads all settings from config.yml. It runs as a cron job and performs:

Create a fresh encrypted backup
Prune local backups beyond local_retention.days
Transfer to remote destination
Verify transfer (size match)
Sync the encryption key alongside the backup
Prune remote backups beyond remote.retention.days

Logs go to /var/log/sting-backup-sync.log and system journal (sting-backup-sync tag).

SSH (Pipe-Based)

Best for NAS devices or servers where rsync may be restricted. Transfers files over a standard SSH connection using pipes.

backup:
  remote:
    enabled: true
    type: "ssh"
    destination: "my-nas-hostname"
    user: "backup-user"
    port: 22
    ssh_key: "/root/.ssh/id_ed25519_backup"
    path: "sting-backups"    # Relative to home directory
    retention:
      days: 30

Setup steps:

# 1. Generate an SSH key for the backup cron
sudo ssh-keygen -t ed25519 -C "sting-backup" -f /root/.ssh/id_ed25519_backup -N ""

# 2. Display the public key
sudo cat /root/.ssh/id_ed25519_backup.pub

# 3. Add the public key to the remote server's ~/.ssh/authorized_keys

# 4. Test the connection
sudo ssh -i /root/.ssh/id_ed25519_backup -p 22 backup-user@my-nas-hostname "echo connected"

rsync

For servers with rsync available. More efficient for large backups (incremental transfer).

backup:
  remote:
    enabled: true
    type: "rsync"
    destination: "backup.server.com"
    user: "sting"
    port: 22
    ssh_key: "/root/.ssh/id_ed25519_backup"
    path: "/backups/sting"
    retention:
      days: 30

Amazon S3

backup:
  remote:
    enabled: true
    type: "s3"
    destination: "s3://my-backup-bucket/sting"

Requires AWS CLI configured with appropriate credentials.

Automated Backups

Installing the Cron Job

When remote sync is configured, the backup-sync cron runs on the schedule defined in config.yml (remote.schedule, default: 0 3 * * * = daily at 3 AM).

# Install automated backups
sudo /opt/sting-ce/scripts/backup/setup-backup-cron.sh install

# View current cron configuration
sudo crontab -l

# Remove automated backups
sudo /opt/sting-ce/scripts/backup/setup-backup-cron.sh remove

Default Schedule

Job	Schedule	Description
Backup + Remote Sync	`0 3 * * *`	Encrypted backup + offsite sync at 3 AM
Vault Backup	`0 /6 * *`	Vault snapshot every 6 hours
Freshness Check	`0 * * * *`	Hourly backup age check
Health Check	`0 6 * * *`	Daily comprehensive check
Integrity Check	`0 7 * * 0`	Weekly archive verification

Retention Policy

STING enforces separate local and remote retention:

Scope	Default	Config Key	Enforced By
Local count	5 backups	`backup.retention.count`	`msting backup` rotation
Local age	14 days	`backup.local_retention.days`	`backup-sync.sh` cron
Remote age	30 days	`backup.remote.retention.days`	`backup-sync.sh` cron
Remote count	10 backups	`backup.remote.retention.count`	`backup-sync.sh` cron

Both count-based and age-based retention are enforced. Older backups are removed first, then excess backups are trimmed to the count limit.

Monitoring

Health Check Commands

# Run all health checks
./scripts/backup/backup-monitor.sh check

# Check only backup freshness
./scripts/backup/backup-monitor.sh freshness

# Verify backup integrity
./scripts/backup/backup-monitor.sh integrity

# Check Vault backup status
./scripts/backup/backup-monitor.sh vault

# Generate JSON report
./scripts/backup/backup-monitor.sh report

Alert Thresholds

Warning: Backup older than 48 hours
Critical: Backup older than 7 days (168 hours)
Size Warning: Backup smaller than 10 MB

# Custom thresholds
MAX_BACKUP_AGE_HOURS=24 ./scripts/backup/backup-monitor.sh check
CRITICAL_BACKUP_AGE_HOURS=72 ./scripts/backup/backup-monitor.sh check

Restore Process

The restore process is atomic with automatic rollback: if anything fails during restore, the system reverts to its pre-restore state.

Standard Restore

# Stop services
sudo msting stop

# Restore from backup
sudo msting restore /opt/sting-backups/sting_backup_20260314.tar.gz

# Verify encryption keys
sudo msting encryption-keys status

# Start services
sudo msting start

Encrypted Restore

# Ensure the encryption key is in place
sudo ls -la /opt/sting-ce/conf/secrets/backup_key.txt

# Restore (decryption is automatic)
sudo msting restore /opt/sting-backups/sting_backup_20260314.tar.gz.enc

What Restore Does

Creates a rollback snapshot of the current state
Extracts backup to a temporary directory
Validates essential files are present (database dump, config, env files)
Atomically swaps configuration and data
Restores PostgreSQL databases
Restores Docker volumes
If any step fails, automatically rolls back to the snapshot

Disaster Recovery

Scenario: Complete Server Loss

Provision a new server and install STING
Retrieve the encryption key from your secure backup location
Place the key at /opt/sting-ce/conf/secrets/backup_key.txt (chmod 600)
Retrieve the encrypted backup from offsite storage
Run sudo msting restore <backup-file>.enc
Verify with sudo msting status

Scenario: Database Corruption

sudo msting stop
sudo msting restore /opt/sting-backups/sting_backup_20260314.tar.gz
sudo msting encryption-keys status
sudo msting start

Scenario: Encryption Key Lost

If the encryption key is lost:

Encrypted backups cannot be restored — there is no recovery mechanism
Unencrypted backups (.tar.gz) are still usable
Database records, user accounts, and configuration are only recoverable from unencrypted backups

Prevention: Always maintain a copy of the encryption key in a separate secure location (password manager, encrypted USB drive, offline vault).

Scenario: Remote Sync Failure

# Check the sync log
sudo cat /var/log/sting-backup-sync.log

# Test SSH connectivity
sudo ssh -i /root/.ssh/id_ed25519_backup -p <port> <user>@<host> "echo connected"

# Run sync manually
sudo /opt/sting-ce/scripts/backup/backup-sync.sh

Troubleshooting

Backup Fails to Create

# Check Docker is running
docker ps

# Check disk space
df -h /opt/sting-backups

# Check backup logs
sudo cat /opt/sting-ce/logs/backup/backup_$(date +%Y%m%d).log

Encrypted Backup Won’t Decrypt

# Verify the key file exists and is readable
sudo ls -la /opt/sting-ce/conf/secrets/backup_key.txt

# Test manual decryption
sudo openssl enc -d -aes-256-cbc -pbkdf2 -iter 100000 \
  -pass file:/opt/sting-ce/conf/secrets/backup_key.txt \
  -in backup.tar.gz.enc -out /tmp/test_backup.tar.gz

# If the key has changed, you need the original key that was used to encrypt

Remote Sync Fails

# Check the sync log
sudo tail -50 /var/log/sting-backup-sync.log

# Test SSH connection
sudo ssh -i /root/.ssh/id_ed25519_backup -p <port> <user>@<host> "hostname"

# Verify remote directory exists and is writable
sudo ssh -i /root/.ssh/id_ed25519_backup -p <port> <user>@<host> "touch ~/sting-backups/test && rm ~/sting-backups/test"

# Run sync manually to see errors
sudo /opt/sting-ce/scripts/backup/backup-sync.sh

Restore Fails

The restore process creates a rollback snapshot automatically. If restore fails:

# Check what happened
sudo cat /opt/sting-ce/logs/backup/restore_*.log

# The system should have auto-rolled back
# Verify services are running
sudo msting status

Best Practices

Recommended Configuration

Component	Frequency	Retention	Location
Encrypted Backup	Daily (3 AM)	14 days local, 30 days remote	Local + offsite
Encryption Key	Export after generation	Forever	Password manager + offline
Vault Snapshot	Every 6 hours	7 days	Local
Health Check	Daily (6 AM)	N/A	Monitoring
Restore Test	Monthly	N/A	Staging

Security

Encryption key: Store in a password manager AND on an encrypted USB drive
Remote sync: Use SSH key authentication, never passwords
Key permissions: backup_key.txt must be 600 root:root
Separate storage: Keep keys and backups in different locations
Before upgrades: Always verify backups are current and key is backed up

Testing

Monthly: Test restore to a staging environment
Quarterly: Full disaster recovery drill
Before Upgrades: Run sudo msting backup --encrypt and verify

Last updated: March 15, 2026