Overview

STING CE uses a layered data architecture combining relational storage, key-value caching, vector embeddings, and secrets management. Each technology is purpose-selected for its workload.

┌────────────────────────────────────────────────────────────┐
│                     Application Services                    │
└─────┬──────────┬──────────────┬──────────────┬─────────────┘
      │          │              │              │
┌─────▼────┐ ┌──▼───┐  ┌──────▼──────┐  ┌────▼────┐
│PostgreSQL│ │Redis │  │  ChromaDB   │  │  Vault  │
│  3 DBs   │ │Cache │  │  Vectors    │  │ Secrets │
│  :5432   │ │:6379 │  │   :8000     │  │  :8200  │
└──────────┘ └──────┘  └─────────────┘  └─────────┘

PostgreSQL 16

A single PostgreSQL 16 instance (container sting-ce-db, internal port 5432, exposed as 5433) hosts three logically separated databases with dedicated users.

Database Separation

Each user has the minimum permissions required for its service. This limits the blast radius if any single service is compromised.

Extensions

• uuid-ossp — UUID generation for primary keys
• pgcrypto — cryptographic functions for hashing and encryption at the database level

Migrations

Database migrations live in STING/database/migrations/ as sequentially numbered SQL files. Complex schema changes use companion shell scripts for data transformation.

STING/database/migrations/
├── 001_initial_schema.sql
├── 002_add_honey_jar_tables.sql
├── 003_messaging_schema.sql
└── ...

Redis

Redis (container sting-ce-redis, port 6379) provides fast in-memory data operations across four use cases:

Redis is configured as an ephemeral cache — persistent data always lives in PostgreSQL.

ChromaDB 0.5.20

ChromaDB (container sting-ce-chroma, port 8000) stores vector embeddings for semantic search within Honey Jars (knowledge bases).

How It Works

1. Document ingestion — when a user uploads content to a Honey Jar, the knowledge service extracts text and splits it into chunks.
2. Embedding generation — each chunk is converted into a vector embedding using the configured LLM provider.
3. Storage — embeddings are stored in ChromaDB, indexed by collection (one per Honey Jar).
4. Search — user queries are embedded and compared against stored vectors using cosine similarity.

Integration

The knowledge service (sting-ce-knowledge, port 8090) acts as the intermediary between application services and ChromaDB. No other service communicates with ChromaDB directly.

User query → Knowledge Service → ChromaDB (vector search)
                                → PostgreSQL (metadata)
                                → Merged ranked results

File Storage

STING CE stores files on local disk within Docker volumes:

Files are referenced by metadata records in PostgreSQL. The actual binary content lives on the filesystem, keeping the database lean.

HashiCorp Vault

Vault (container sting-ce-vault, port 8200) manages all sensitive configuration:

Key Principles

• No hardcoded credentials — all secrets are fetched from Vault at runtime.
• Auto-unseal — Vault is configured for auto-unseal on startup; manual unseal via sudo msting unseal if needed.
• Key management — API keys can be managed with sudo msting vault-secret list and sudo msting vault-secret <provider> <key>.

Data Flow: Document Upload

A typical document upload to a Honey Jar illustrates how the storage layers work together:

1. User uploads document via frontend
2. Flask API receives file, validates, stores to disk
3. Knowledge service processes document:
   a. Extracts text content
   b. Splits into chunks
   c. Generates vector embeddings (via external-ai-service)
   d. Stores embeddings in ChromaDB
   e. Stores metadata in PostgreSQL (sting_app)
4. Redis cache is invalidated for affected Honey Jar
5. Document is now searchable via semantic similarity

Backup and Recovery

The msting backup command creates a comprehensive backup including:

• All three PostgreSQL databases (pg_dump)
• Redis snapshot (if persistent)
• ChromaDB data volume
• Configuration files and environment variables
• Vault data (encrypted)

Backups can optionally be encrypted with --encrypt. Restore with sudo msting restore <file>.