# Charybdis - Architecture

**Version**: 2.0
**Last Updated**: 2026-05-06
**Status**: Living Document

> For the product vision and roadmap, see [VISION.md](../VISION.md).

---

## Executive Summary

**Charybdis** is a security-native platform engineering tool that unifies software catalog, vulnerability management, and compliance posture in a single event-driven platform. Built in Rust, deployed as a single binary.

### Core Value Proposition

**Problem**:
- Platform engineers run 5+ disconnected tools (Backstage, DefectDojo, Dependency-Track, license scanners, compliance spreadsheets)
- Software catalogs rely on static YAML files that drift from reality
- Security posture is invisible at the catalog level — vulns live in separate tools with no link to services
- Compliance evidence is assembled manually from fragmented data sources

**Solution**:
- **Dynamic software catalog** — event-driven, gRPC-native, no static YAML
- **Native vulnerability management** — ingest scan results (SARIF, CycloneDX, SPDX), triage, assess, track
- **Security gates & compliance** — severity thresholds, license policies, compliance framework mappings
- **Plugin system for integrations** — Slack, Jira, GitHub, custom tools react to events
- **Single binary deployment** — Rust + PostgreSQL, ~50MB RAM, sub-millisecond latency

### Use Case Examples

**Service registration with auto-provisioning:**
```
CI/CD Pipeline (Python) → gRPC CreateEntity(kind=Component)
                                     ↓
                          Charybdis persists entity
                                     ↓
                          Event: EntityCreated published
                                     ↓
                ┌────────────────────┴────────────────────┐
                ↓                                         ↓
    Jira Plugin                               Slack Plugin
    - Creates onboarding epic                 - Notifies #platform channel
                ↓                                         ↓
                └────────────────────┬────────────────────┘
                                     ↓
                    Entity visible via gRPC API / Backstage YAML adapter
```

**Vulnerability ingestion with security gates:**
```
Scanner (Trivy) → CI/CD → gRPC IngestScan(sarif_report)
                                     ↓
                          Charybdis parses SARIF
                          Creates Vulnerability entities linked to Component
                                     ↓
                          Rules engine evaluates auto-assessment
                          Security gate checks thresholds
                                     ↓
                ┌────────────────────┴────────────────────┐
                ↓                                         ↓
    Gate PASSED                               Gate FAILED
    - Vulns tracked                           - Slack alert to #security
    - Dashboard updated                       - Jira ticket created
                                              - CI/CD pipeline blocked
```

---

## Architectural Principles

### 1. **No Database Migrations Ever**

The database schema is created once on first startup and **never changes**.

**How it works**:
- Entities stored as protobuf bytes in `entity_data BYTEA` column
- Plugins add new entity types via protobuf `oneof` variants
- Build system regenerates code, database schema remains static

**Benefits**:
- Deploy new plugins without downtime
- No migration scripts to manage
- Forward/backward compatibility built-in
- Easy rollback (protobuf versioning)

### 2. **gRPC First, Everything Else is Adapter**

Charybdis provides **only gRPC APIs**. All other interfaces (REST, YAML, GraphQL) are adapters on top.

**Rationale**:
- **Strong typing**: Protobuf ensures type safety across all languages
- **Performance**: Binary protocol, efficient serialization
- **Multi-language**: Official gRPC clients for 10+ languages
- **Streaming**: Built-in support for real-time updates (future)
- **Code generation**: Automatic client/server code generation

**What Charybdis provides**:
- ✅ gRPC API with protobuf definitions
- ✅ YAML adapter endpoints (for Backstage compatibility/migration)
- ❌ REST API (teams can add if needed via gateway)
- ❌ GraphQL (teams can add if needed)

### 3. **Plugin-Based Extensibility**

Plugins are **compile-time integrated** Rust crates that:

1. **Define entity types** via protobuf schemas
2. **React to events** via EventHandler trait
3. **Call external APIs** to synchronize state
4. **Store references** in entity annotations
5. **(Future) Extend gRPC API** with custom endpoints

**Plugin characteristics**:
- Not hot-swappable (require rebuild)
- Type-safe at compile time
- No runtime dependency resolution
- Configurable via YAML config file
- Can be developed by community

### 4. **Event-Driven Orchestration**

Events are published **after** entities are persisted:

```
CRUD Operation → Persist to DB → Publish Event → Plugins React
```

**Event flow**:
1. Entity created/updated/deleted in database
2. Event published to event bus with full entity data
3. All subscribed plugin handlers receive event
4. Plugins filter events they care about
5. Plugins perform asynchronous actions
6. Plugins update entity annotations (via repository)

**Error handling**:
- Plugin errors **do not** fail the CRUD operation
- Entity is already persisted before event publishing
- Plugins log errors for observability
- Retries handled by event bus backend

### 5. **Backstage Compatibility by Design**

Charybdis entity model is **100% compatible** with Backstage's descriptor format.

**How it works**:
- Core protobuf types mirror Backstage YAML structure
- YAML adapter endpoints serve entities in Backstage format
- Backstage reads Charybdis as a dynamic location provider
- No changes needed to Backstage frontend

**Key compatibility points**:
- Entity `kind`, `apiVersion`, `metadata`, `spec` structure
- Annotation format (reverse-DNS style)
- Relationship types (`dependsOn`, `partOf`, etc.)
- All standard Backstage entity kinds supported

---

## System Architecture

### Component Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                        Client Layer                             │
│  CI/CD Pipelines (Python, Go, Node.js, Bash, etc.)            │
│  Security Scanners (SARIF, CycloneDX, SPDX output)            │
│  IaC Tools (Terraform, Pulumi, Crossplane)                     │
│  Backstage (via YAML adapter, optional)                        │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 │ gRPC / YAML HTTP
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Charybdis Core                              │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │            gRPC API                                       │  │
│  │  EntityService: CRUD + List (with field masks)            │  │
│  │  ScanService: IngestScan (SARIF, CycloneDX, SPDX)       │  │
│  │  AssessmentService: Triage, Accept, Remediate             │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │         Security Engine (native)                          │  │
│  │  - Vulnerability tracking per entity                      │  │
│  │  - Rules engine (auto-assessment)                         │  │
│  │  - Security gates (severity thresholds)                   │  │
│  │  - License compliance                                     │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │         Adapters                                          │  │
│  │  - YAML Adapter (Backstage compat / migration)            │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │            Repository Layer                               │  │
│  │  - PostgreSQL with protobuf storage                       │  │
│  │  - JSONB annotations for fast queries                     │  │
│  │  - Zero-migration schema                                  │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │              Event Bus System                             │  │
│  │  - Memory backend (dev)                                   │  │
│  │  - Redis backend (prod, future)                           │  │
│  └────────────────────┬─────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                       │
                       │ Entity / Vulnerability / Gate events
                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Plugin Layer (integrations)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐ │
│  │ Slack/Teams   │  │ Jira/GitHub  │  │ Custom Plugins         │ │
│  │ Notifications │  │ Issue Track  │  │ Your integrations      │ │
│  └──────┬────────┘  └──────┬───────┘  └──────────┬─────────────┘ │
└─────────┼──────────────────┼─────────────────────┼──────────────┘
          ▼                  ▼                     ▼
   Slack/Teams API      Jira/GitHub API      External APIs
```

### Data Flow

#### Entity Creation Flow

```
1. Client (CI/CD)
   ↓ gRPC CreateEntityRequest
2. Charybdis gRPC API
   ↓ Validate entity
   ↓ Generate UUID, timestamps
3. Entity Repository
   ↓ Serialize to protobuf bytes
   ↓ INSERT INTO entities
4. PostgreSQL
   ↓ Entity persisted
5. Event Bus
   ↓ Publish EntityCreated event (with full entity data)
6. Plugin Handlers (parallel)
   ├─ DefectDojo: Create product → Store product_id in annotations
   ├─ DependencyTrack: Create project → Store project_uuid in annotations
   └─ Custom: Custom actions
7. Repository.update_annotations()
   ↓ Merge annotations from plugins
   ↓ UPDATE entities SET annotations = ...
8. PostgreSQL
   ↓ Entity updated with external tool IDs
9. Return CreateEntityResponse to client
```

#### Backstage Integration Flow

```
1. Backstage Catalog Backend
   ↓ HTTP GET /yaml/locations
2. Charybdis YAML Adapter
   ↓ Query repository.list_all()
   ↓ Convert entities to Location YAML
3. Backstage receives location list
   ↓ Fetches each entity: GET /yaml/entities/:id
4. Charybdis YAML Adapter
   ↓ repository.get_by_id()
   ↓ Convert entity protobuf → Backstage YAML
5. Backstage ingests entities
   ↓ Displays in UI with all annotations
   ↓ Shows linked DefectDojo products, etc.
```

---

## Core Entity Model

### Protobuf Schema

```protobuf
message Entity {
  // System-managed fields
  string id = 1;                        // UUID (auto-generated)
  google.protobuf.Timestamp created_at = 21;
  google.protobuf.Timestamp updated_at = 22;
  
  // Backstage-compatible fields
  string api_version = 2;               // "backstage.io/v1alpha1"
  string kind = 3;                      // "Component", "API", "User", etc.
  
  // Polymorphic metadata (identifying information)
  oneof metadata {
    // Core types (built-in)
    ComponentMetadata component_metadata = 10;
    APIMetadata api_metadata = 11;
    UserMetadata user_metadata = 12;
    GroupMetadata group_metadata = 13;
    SystemMetadata system_metadata = 14;
    DomainMetadata domain_metadata = 15;
    ResourceMetadata resource_metadata = 16;
    LocationMetadata location_metadata = 17;
    TemplateMetadata template_metadata = 18;
    
    // Plugin types (dynamically added)
    // Example: DefectDojoProductMetadata defectdojo_product_metadata = 100;
  }
  
  // Polymorphic spec (configuration and behavior)
  oneof spec {
    // Core types (built-in)
    ComponentSpec component_spec = 10;
    APISpec api_spec = 11;
    UserSpec user_spec = 12;
    GroupSpec group_spec = 13;
    SystemSpec system_spec = 14;
    DomainSpec domain_spec = 15;
    ResourceSpec resource_spec = 16;
    LocationSpec location_spec = 17;
    TemplateSpec template_spec = 18;
    
    // Plugin types (dynamically added)
    // Example: DefectDojoProductSpec defectdojo_product_spec = 100;
  }
  
  // Annotations - external tool references
  // Format: "tool.com/resource-type-attribute"
  // Examples:
  //   "defectdojo.com/product-id": "42"
  //   "dependencytrack.com/project-uuid": "550e8400-..."
  //   "github.com/repo-slug": "myorg/myrepo"
  map<string, string> annotations = 20;
}
```

### Backstage Entity Kinds Mapping

All Backstage entity kinds must be supported:

| Backstage Kind | Charybdis Proto Type | Status |
|----------------|---------------------|--------|
| Component | ComponentMetadata/Spec | ✅ Done |
| Service | ServiceMetadata/Spec | ✅ Done |
| API | APIMetadata/Spec | ✅ Done |
| User | UserMetadata/Spec | ✅ Done |
| Group | GroupMetadata/Spec | ✅ Done |
| System | SystemMetadata/Spec | ✅ Done |
| Domain | DomainMetadata/Spec | ✅ Done |
| Resource | ResourceMetadata/Spec | ✅ Done |
| Location | LocationMetadata/Spec | Not planned (entities are event-driven, not file-based) |
| Template | TemplateMetadata/Spec | Phase 3 (Scaffolder) |

---

## YAML Adapter Layer

### Purpose

Serve entities in Backstage-compatible YAML format over HTTP.

### API Specification

#### 1. List Locations

**Endpoint**: `GET /yaml/locations`

**Purpose**: Return a dynamic Location entity that lists all entities in Charybdis.

**Response Format** (Backstage Location YAML):
```yaml
apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: charybdis-all-entities
  description: Dynamic location managed by Charybdis
spec:
  type: charybdis
  targets:
    - http://charybdis.company.com/yaml/entities/service-payment
    - http://charybdis.company.com/yaml/entities/service-auth
    - http://charybdis.company.com/yaml/entities/api-payments-v1
    - http://charybdis.company.com/yaml/entities/user-john-doe
    # ... all entities
```

**Implementation Notes**:
- Query `SELECT id, kind FROM entities`
- Generate URL for each entity: `/yaml/entities/{kind}-{id}`
- Return as Backstage Location YAML
- Cache response (invalidate on entity CRUD)

#### 2. Get Entity by ID

**Endpoint**: `GET /yaml/entities/:id`

**Purpose**: Return a specific entity in Backstage YAML format.

**Response Format** (Backstage Component YAML):
```yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  namespace: production
  description: Core payment processing service
  annotations:
    github.com/repo-slug: myorg/payment-service
    defectdojo.com/product-id: "42"
    dependencytrack.com/project-uuid: "550e8400-e29b-41d4-a716-446655440000"
  tags:
    - payments
    - critical
  links:
    - url: https://dashboard.company.com/payment-service
      title: Dashboard
      icon: dashboard
spec:
  type: service
  lifecycle: production
  owner: payments-team
  system: payment-system
  dependsOn:
    - component:auth-service
  providesApis:
    - payments-api-v1
```

**Implementation Notes**:
- Query `repository.get_by_id(id)`
- Convert Entity protobuf → Backstage YAML
- Map `oneof metadata/spec` to appropriate YAML structure
- Include all annotations
- Return as `text/yaml` content type

#### 3. Configuration in Backstage

**Backstage app-config.yaml**:
```yaml
catalog:
  locations:
    - type: url
      target: http://charybdis.company.com/yaml/locations
      rules:
        - allow: [Component, API, User, Group, System, Domain, Resource]
```

**How it works**:
1. Backstage fetches `/yaml/locations` on catalog refresh
2. Parses Location YAML to get entity URLs
3. Fetches each entity URL (`/yaml/entities/:id`)
4. Ingests entities into Backstage catalog
5. Displays in UI with all metadata and annotations

---

## Plugin System

### Plugin Architecture

Charybdis supports **two types of plugins**, both compile-time integrated:

#### **1. Event-Driven Plugins**
React to entity lifecycle events (create, update, delete):
- **Example**: DefectDojo, DependencyTrack
- **Implement**: `EventDrivenPlugin` trait
- **Provide**: `ResourceHandler` implementations
- **Triggered by**: Entity CRUD operations

#### **2. Sync Plugins**
Pull data from external sources on a schedule:
- **Example**: Okta, Keycloak, Active Directory
- **Implement**: `SyncPlugin` trait
- **Scheduled by**: Cron expressions
- **Triggered by**: Time-based schedule or manual API call

### Generic Plugin Utilities (2025-01-15)

All plugins have access to reusable utilities in `src/plugins/`:

#### **PluginHttpClient** 
Generic HTTP client with multiple auth methods:
- Token authentication (DefectDojo)
- Bearer authentication (modern APIs)
- API Key authentication (DependencyTrack)
- Basic authentication (JIRA, Bitbucket)
- Methods: `get()`, `post()`, `put()`, `patch()`, `delete()`

#### **AnnotationHelper**
Consistent API for storing/retrieving plugin metadata:
- `set_id()` / `get_id()` - Store external resource IDs
- `set()` / `get()` - Store arbitrary metadata
- Enforces naming: `{plugin}.com/{resource}-id`
- Prevents annotation conflicts between plugins

#### **FieldMapper**
Maps entity fields to external tool formats:
- Direct field paths: `"metadata.name"`
- Static values: `{"value": "Web Application"}`
- Complex mappings with entity resolution
- Repository integration for cross-entity queries

#### **DateUtils**
Common date/time operations:
- `today()`, `today_plus_days(n)` - Date formatting
- `engagement_date_range(days)` - For time-bound resources
- `iso_timestamp()` - ISO 8601 timestamps

### Plugin Responsibilities

Plugins can:

1. **Define custom entity types** (via protobuf)
2. **React to entity events** (via ResourceHandler trait)
3. **Call external APIs** (using PluginHttpClient)
4. **Store metadata** (using AnnotationHelper)
5. **Map fields** (using FieldMapper)
6. **Schedule syncs** (SyncPlugins with cron)
7. **(Future) Extend gRPC API** (custom services)

### Plugin Lifecycle

```
1. Development
   ├─ Define protobuf schema (metadata/spec messages)
   ├─ Implement EventHandler trait
   ├─ Implement external API client
   └─ Add configuration schema

2. Registration
   ├─ Add to plugins.toml (enabled = true)
   └─ Add to Cargo.toml dependencies

3. Build Time
   ├─ build.rs reads plugins.toml
   ├─ Generates entities.proto with plugin types
   ├─ Compiles all protobuf to Rust code
   └─ Links plugin into binary

4. Runtime
   ├─ Load plugin configuration from config file
   ├─ Initialize plugin handler
   ├─ Subscribe to event bus
   └─ React to entity events
```

### Plugin Configuration System

**File**: `config.toml` (TOML format with `${VAR}` env var substitution)

```toml
# Server configuration
[server]
grpc_host = "[::1]"
grpc_port = 50051

[server.yaml_adapter]
enabled = true
host = "0.0.0.0"
port = 8080

# Database
[database]
url = "${DATABASE_URL}"
max_connections = 10

# Plugin configurations
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_API_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
engagement_type = "CI/CD"

[plugins.keycloak]
enabled = false
# base_url = "https://keycloak.company.com"
# realm = "master"

[plugins.dependencytrack]
enabled = false
# base_url = "https://dependencytrack.company.com"
# api_key = "${DEPENDENCYTRACK_API_KEY}"
```

See `config.toml.example` for the full reference with all options documented.

**Configuration loading** (`src/config.rs`):

```rust
pub struct Config {
    pub server: ServerConfig,
    pub database: DatabaseConfig,
    pub security: SecurityConfig,
    pub telemetry: TelemetryConfig,
    pub plugins: PluginsConfig,
}

impl Config {
    pub fn from_file(path: &str) -> Result<Self> {
        let content = std::fs::read_to_string(path)?;
        // Substitute ${VAR} with environment variable values
        let content = substitute_env_vars(&content);
        let config: Config = toml::from_str(&content)?;
        Ok(config)
    }
}
```

**Each plugin implements**:

```rust
pub trait Plugin {
    fn name(&self) -> &str;
    fn plugin_type(&self) -> PluginType;
    fn load_config(&mut self, config: PluginConfig) -> Result<()>;
    fn validate_config(&self) -> Result<()>;
}
```

### Event Handler Behavior Configuration

Plugins can configure different behaviors based on:
- Entity kind
- Event type (created/updated/deleted)
- Custom conditions

**Example**: DefectDojo plugin config (`config.toml`)

```toml
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_API_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
engagement_type = "CI/CD"
```

**Handler Implementation**:

```rust
impl EventHandler for DefectDojoHandler {
    async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
        // Check if plugin enabled
        if !self.config.enabled {
            return Ok(());
        }
        
        // Match event type to configured behavior
        let behavior = match event.event_type {
            EntityEventType::Created => &self.config.on_entity_created,
            EntityEventType::Updated => &self.config.on_entity_updated,
            EntityEventType::Deleted => &self.config.on_entity_deleted,
        };
        
        // Find matching rule for this entity kind
        for rule in behavior {
            if rule.matches(&event) {
                return self.execute_action(&rule.action, event).await;
            }
        }
        
        Ok(())
    }
}
```

---

## Future: Plugin API Extensions

**Future capability**: Plugins can expose custom gRPC endpoints.

### Example: DefectDojo Query Plugin

**Plugin extends API with custom service**:

```protobuf
// plugins/defectdojo/proto/defectdojo_service.proto

service DefectDojoService {
  // Get findings for a specific entity
  rpc GetEntityFindings(GetEntityFindingsRequest) returns (GetEntityFindingsResponse);
  
  // Get all active vulnerabilities
  rpc GetActiveVulnerabilities(GetActiveVulnerabilitiesRequest) returns (GetActiveVulnerabilitiesResponse);
  
  // Sync entity with DefectDojo
  rpc SyncEntity(SyncEntityRequest) returns (SyncEntityResponse);
}

message GetEntityFindingsRequest {
  string entity_id = 1;
  string severity_filter = 2;  // "Critical", "High", etc.
  bool active_only = 3;
}

message GetEntityFindingsResponse {
  repeated Finding findings = 1;
  int32 total_count = 2;
}

message Finding {
  int32 id = 1;
  string title = 2;
  string severity = 3;
  string description = 4;
  google.protobuf.Timestamp date = 5;
  bool active = 6;
}
```

**Implementation**:

```rust
pub struct DefectDojoServiceImpl {
    client: DefectDojoClient,
    repository: EntityRepository,
}

#[async_trait]
impl DefectDojoService for DefectDojoServiceImpl {
    async fn get_entity_findings(
        &self,
        request: Request<GetEntityFindingsRequest>,
    ) -> Result<Response<GetEntityFindingsResponse>, Status> {
        let req = request.into_inner();
        
        // Get entity to retrieve DefectDojo product ID
        let entity = self.repository.get_by_id(&req.entity_id).await?;
        let product_id = entity
            .annotations
            .get("defectdojo.com/product-id")
            .ok_or_else(|| Status::not_found("No DefectDojo product linked"))?;
        
        // Fetch findings from DefectDojo API
        let findings = self.client
            .get_findings(product_id, &req.severity_filter, req.active_only)
            .await?;
        
        Ok(Response::new(GetEntityFindingsResponse {
            findings: findings.into_iter().map(|f| f.into()).collect(),
            total_count: findings.len() as i32,
        }))
    }
}
```

**Registration in main.rs**:

```rust
// Register plugin's custom gRPC service
if defectdojo_plugin.is_enabled() {
    let defectdojo_service = DefectDojoServiceServer::new(
        DefectDojoServiceImpl::new(defectdojo_client, repository.clone())
    );
    
    server = server.add_service(defectdojo_service);
}
```

**Client usage**:

```bash
grpcurl -d '{
  "entity_id": "service-payment",
  "severity_filter": "Critical",
  "active_only": true
}' localhost:50051 charybdis.plugins.defectdojo.DefectDojoService/GetEntityFindings
```

---

## Technology Decisions

### Why gRPC Only?

**Decision**: Charybdis core provides only gRPC. Teams add REST/GraphQL as needed.

**Rationale**:
- **Strong typing**: Protobuf schemas ensure correctness across languages
- **Performance**: Binary protocol, efficient for CI/CD use cases
- **Multi-language**: Official clients for Python, Go, Node.js, Java, Rust, C++, C#, etc.
- **Streaming**: Built-in bidirectional streaming (future: real-time updates)
- **Code generation**: Automatic client generation eliminates boilerplate
- **Focus**: Core team maintains one high-quality API surface

**Teams can add REST if needed**:
```
grpc-gateway or Envoy → Charybdis gRPC
```

### Why PostgreSQL + JSONB?

**Decision**: Use PostgreSQL with protobuf BYTEA storage and JSONB annotations.

**Rationale**:
- **Scale**: Handles 1000s of entities easily (tested: 170k resources)
- **JSONB performance**: GIN indexes enable fast annotation queries
- **Familiarity**: Most teams already run PostgreSQL
- **Reliability**: ACID transactions, mature ecosystem
- **Future-proof**: Can add read replicas, partitioning if needed

**Alternative considered**: MongoDB
- **Verdict**: PostgreSQL with JSONB provides same flexibility with better consistency guarantees

### Why Compile-Time Plugins?

**Decision**: Plugins are compile-time integrated, not runtime loaded.

**Rationale**:
- **Type safety**: Rust compiler ensures correctness
- **Performance**: No dynamic loading overhead
- **Simplicity**: No plugin version compatibility matrix
- **Security**: No arbitrary code execution
- **Trade-off**: Requires rebuild to add plugins (acceptable for infrastructure tool)

**Community plugins**: Published as crates, teams include in their build.

### Why Event-Driven?

**Decision**: Plugins react to events after entity persistence.

**Rationale**:
- **Decoupling**: Plugins can't break core CRUD operations
- **Resilience**: Plugin failures are logged, not propagated
- **Async**: Long-running operations don't block API
- **Extensibility**: Add plugins without modifying core
- **Observability**: Events provide audit trail

**Trade-off**: Plugins can't prevent entity creation (validation must be in core or client).

---

## Scale Estimates

### Reference Deployment

**Company**: ~100 engineers  
**Backstage Entities**:
- Components: ~1,000
- APIs: ~1,000
- Users: ~700
- Groups: ~200
- Systems: ~50
- Domains: ~150
- Resources: ~170,000

**Charybdis Requirements**:
- PostgreSQL: ~500MB storage (with JSONB annotations)
- Memory: ~256MB for Charybdis service
- CPU: Minimal (mostly I/O bound)

**10x Scale** (1,000 engineers, 1.7M resources):
- PostgreSQL: ~5GB storage
- Memory: ~512MB
- CPU: Still minimal with proper indexing

**Bottlenecks**:
- JSONB queries on annotations (solved with GIN indexes)
- Event bus throughput (solved with Redis/RabbitMQ backend)
- Plugin external API rate limits (solved with plugin-level queuing)

---

## Development Roadmap

> See [VISION.md](VISION.md) for the full product vision and strategic roadmap.

### Phase 0: Catalog Foundation (Done)

- ✅ Zero-migration database architecture
- ✅ gRPC API with protobuf (full entity CRUD)
- ✅ Event bus system (MemoryEventBus)
- ✅ Event dispatcher architecture (generic + plugin-specific)
- ✅ All Backstage entity kinds (Component, System, API, User, Group, Domain, Resource)
- ✅ YAML adapter endpoints (Backstage compatibility)
- ✅ gRPC field masks for partial updates
- ✅ OpenTelemetry observability (traces, metrics, logs)
- ✅ mTLS + RBAC security framework
- ✅ Config loading from TOML with env var substitution
- ✅ Plugin trait architecture (EventDriven + Sync)
- ✅ Generic plugin utilities (HttpClient, AnnotationHelper, FieldMapper, DateUtils)
- ✅ DefectDojo plugin (products, engagements, owner resolution)
- ✅ Keycloak plugin (user/group sync with annotations)

### Phase 1: Security Core (Current Focus)

**Goal**: Native vulnerability management and scan ingestion.

- [ ] Vulnerability/Observation entity kind (protobuf + storage)
- [ ] ScanService gRPC endpoint for scan ingestion
- [ ] SARIF parser (covers majority of modern scanners)
- [ ] CycloneDX parser (SBOMs + vulnerability data)
- [ ] SPDX parser (license data)
- [ ] Assessment workflow (triage, accept risk, remediate)
- [ ] Rules engine for auto-assessment
- [ ] Security gates (severity thresholds per product)
- [ ] License tracking and policy engine

### Phase 2: Compliance & Integrations

**Goal**: Compliance frameworks and integration plugins.

- [ ] Compliance framework mappings (NIS2, SOC2, DORA)
- [ ] VEX document support (CSAF, OpenVEX)
- [ ] Export/reporting (PDF, Excel)
- [ ] Notification plugins (Slack, Teams, email)
- [ ] Issue tracker plugins (Jira, GitHub, GitLab)
- [ ] Redis event bus backend

### Phase 3: Scaffolder & Ecosystem

**Goal**: Service scaffolding and community growth.

- [ ] Service scaffolder (Git-native templates)
- [ ] Event-driven provisioning on scaffold
- [ ] Plugin SDK documentation
- [ ] Helm chart & 1-click deploy
- [ ] Community plugin registry

---

## Success Criteria

### Phase 1 Success (Security Core)

- [ ] Ingest SARIF scan results via gRPC
- [ ] Vulnerabilities linked to catalog entities
- [ ] Rules engine auto-assesses based on severity/component
- [ ] Security gates block on threshold violations
- [ ] Assessment workflow (triage → accept/remediate)
- [ ] License data ingested from CycloneDX/SPDX

### Production Success

- [ ] 1,000+ entities managed with security posture
- [ ] <100ms p99 latency for CRUD operations
- [ ] Single `helm install` deployment
- [ ] Documentation complete
- [ ] 3+ integration plugins in production

---

## Key Decisions Summary

| Decision | Choice | Rationale |
|----------|--------|-----------|
| **API Protocol** | gRPC only | Strong typing, multi-language, performance |
| **Frontend** | Backstage YAML adapter | Compatible with existing Backstage deployments |
| **Security features** | Native (core) | First-class, not plugins. Vulns, compliance, gates in the core |
| **External integrations** | Plugins | Slack, Jira, etc. as event-driven plugins |
| **Database** | PostgreSQL + JSONB | Scale, familiarity, ACID |
| **Plugin Loading** | Compile-time | Type safety, performance, security |
| **Event Model** | Post-persistence | Resilience, decoupling |
| **Schema Evolution** | Protobuf only | Zero migrations |
| **Configuration** | TOML + env vars | Flexibility, Rust-native |
| **Deployment** | Single binary + Docker + K8s | Cloud-native, minimal footprint |

---

## References

- [Backstage Descriptor Format](https://backstage.io/docs/features/software-catalog/descriptor-format)
- [gRPC Documentation](https://grpc.io/docs/)
- [Protocol Buffers Guide](https://protobuf.dev/)
- [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html)

---

## Document Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2025-01-03 | Initial comprehensive architecture document |
| 1.1 | 2025-01-15 | Added dual plugin architecture, generic utilities, updated roadmap |
| 2.0 | 2026-04-02 | Pivoted to security-native platform engineering tool. Extracted product vision to VISION.md. Added native vulnerability management, compliance frameworks to architecture. Updated roadmap and success criteria. |

---

**This document covers architecture and technical decisions.**
For product vision and strategic direction, see [VISION.md](VISION.md).
All code and documentation must align with these documents.