guillaume.grabe/charybdis
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1302ce412c246afdc8931d4431d42c878311d30e
charybdis/docs/architecture.md
T
Guillaume GRABÉ 051a080dfa initial-commit
2026-05-12 17:06:43 +02:00

34 KiB
Raw Blame History

Charybdis - Architecture

Version: 2.0 Last Updated: 2026-05-06 Status: Living Document

For the product vision and roadmap, see VISION.md.

Executive Summary

Charybdis is a security-native platform engineering tool that unifies software catalog, vulnerability management, and compliance posture in a single event-driven platform. Built in Rust, deployed as a single binary.

Core Value Proposition

Problem:

  • Platform engineers run 5+ disconnected tools (Backstage, DefectDojo, Dependency-Track, license scanners, compliance spreadsheets)
  • Software catalogs rely on static YAML files that drift from reality
  • Security posture is invisible at the catalog level — vulns live in separate tools with no link to services
  • Compliance evidence is assembled manually from fragmented data sources

Solution:

  • Dynamic software catalog — event-driven, gRPC-native, no static YAML
  • Native vulnerability management — ingest scan results (SARIF, CycloneDX, SPDX), triage, assess, track
  • Security gates & compliance — severity thresholds, license policies, compliance framework mappings
  • Plugin system for integrations — Slack, Jira, GitHub, custom tools react to events
  • Single binary deployment — Rust + PostgreSQL, ~50MB RAM, sub-millisecond latency

Use Case Examples

Service registration with auto-provisioning:

CI/CD Pipeline (Python) → gRPC CreateEntity(kind=Component)
                                     ↓
                          Charybdis persists entity
                                     ↓
                          Event: EntityCreated published
                                     ↓
                ┌────────────────────┴────────────────────┐
                ↓                                         ↓
    Jira Plugin                               Slack Plugin
    - Creates onboarding epic                 - Notifies #platform channel
                ↓                                         ↓
                └────────────────────┬────────────────────┘
                                     ↓
                    Entity visible via gRPC API / Backstage YAML adapter

Vulnerability ingestion with security gates:

Scanner (Trivy) → CI/CD → gRPC IngestScan(sarif_report)
                                     ↓
                          Charybdis parses SARIF
                          Creates Vulnerability entities linked to Component
                                     ↓
                          Rules engine evaluates auto-assessment
                          Security gate checks thresholds
                                     ↓
                ┌────────────────────┴────────────────────┐
                ↓                                         ↓
    Gate PASSED                               Gate FAILED
    - Vulns tracked                           - Slack alert to #security
    - Dashboard updated                       - Jira ticket created
                                              - CI/CD pipeline blocked

Architectural Principles

1. No Database Migrations Ever

The database schema is created once on first startup and never changes.

How it works:

  • Entities stored as protobuf bytes in entity_data BYTEA column
  • Plugins add new entity types via protobuf oneof variants
  • Build system regenerates code, database schema remains static

Benefits:

  • Deploy new plugins without downtime
  • No migration scripts to manage
  • Forward/backward compatibility built-in
  • Easy rollback (protobuf versioning)

2. gRPC First, Everything Else is Adapter

Charybdis provides only gRPC APIs. All other interfaces (REST, YAML, GraphQL) are adapters on top.

Rationale:

  • Strong typing: Protobuf ensures type safety across all languages
  • Performance: Binary protocol, efficient serialization
  • Multi-language: Official gRPC clients for 10+ languages
  • Streaming: Built-in support for real-time updates (future)
  • Code generation: Automatic client/server code generation

What Charybdis provides:

  • ✅ gRPC API with protobuf definitions
  • ✅ YAML adapter endpoints (for Backstage compatibility/migration)
  • ❌ REST API (teams can add if needed via gateway)
  • ❌ GraphQL (teams can add if needed)

3. Plugin-Based Extensibility

Plugins are compile-time integrated Rust crates that:

  1. Define entity types via protobuf schemas
  2. React to events via EventHandler trait
  3. Call external APIs to synchronize state
  4. Store references in entity annotations
  5. (Future) Extend gRPC API with custom endpoints

Plugin characteristics:

  • Not hot-swappable (require rebuild)
  • Type-safe at compile time
  • No runtime dependency resolution
  • Configurable via YAML config file
  • Can be developed by community

4. Event-Driven Orchestration

Events are published after entities are persisted:

CRUD Operation → Persist to DB → Publish Event → Plugins React

Event flow:

  1. Entity created/updated/deleted in database
  2. Event published to event bus with full entity data
  3. All subscribed plugin handlers receive event
  4. Plugins filter events they care about
  5. Plugins perform asynchronous actions
  6. Plugins update entity annotations (via repository)

Error handling:

  • Plugin errors do not fail the CRUD operation
  • Entity is already persisted before event publishing
  • Plugins log errors for observability
  • Retries handled by event bus backend

5. Backstage Compatibility by Design

Charybdis entity model is 100% compatible with Backstage's descriptor format.

How it works:

  • Core protobuf types mirror Backstage YAML structure
  • YAML adapter endpoints serve entities in Backstage format
  • Backstage reads Charybdis as a dynamic location provider
  • No changes needed to Backstage frontend

Key compatibility points:

  • Entity kind, apiVersion, metadata, spec structure
  • Annotation format (reverse-DNS style)
  • Relationship types (dependsOn, partOf, etc.)
  • All standard Backstage entity kinds supported

System Architecture

Component Overview

┌─────────────────────────────────────────────────────────────────┐
│                        Client Layer                             │
│  CI/CD Pipelines (Python, Go, Node.js, Bash, etc.)            │
│  Security Scanners (SARIF, CycloneDX, SPDX output)            │
│  IaC Tools (Terraform, Pulumi, Crossplane)                     │
│  Backstage (via YAML adapter, optional)                        │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 │ gRPC / YAML HTTP
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Charybdis Core                              │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │            gRPC API                                       │  │
│  │  EntityService: CRUD + List (with field masks)            │  │
│  │  ScanService: IngestScan (SARIF, CycloneDX, SPDX)       │  │
│  │  AssessmentService: Triage, Accept, Remediate             │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │         Security Engine (native)                          │  │
│  │  - Vulnerability tracking per entity                      │  │
│  │  - Rules engine (auto-assessment)                         │  │
│  │  - Security gates (severity thresholds)                   │  │
│  │  - License compliance                                     │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │         Adapters                                          │  │
│  │  - YAML Adapter (Backstage compat / migration)            │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │            Repository Layer                               │  │
│  │  - PostgreSQL with protobuf storage                       │  │
│  │  - JSONB annotations for fast queries                     │  │
│  │  - Zero-migration schema                                  │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                         │
│  ┌────────────────────▼─────────────────────────────────────┐  │
│  │              Event Bus System                             │  │
│  │  - Memory backend (dev)                                   │  │
│  │  - Redis backend (prod, future)                           │  │
│  └────────────────────┬─────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                       │
                       │ Entity / Vulnerability / Gate events
                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Plugin Layer (integrations)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐ │
│  │ Slack/Teams   │  │ Jira/GitHub  │  │ Custom Plugins         │ │
│  │ Notifications │  │ Issue Track  │  │ Your integrations      │ │
│  └──────┬────────┘  └──────┬───────┘  └──────────┬─────────────┘ │
└─────────┼──────────────────┼─────────────────────┼──────────────┘
          ▼                  ▼                     ▼
   Slack/Teams API      Jira/GitHub API      External APIs

Data Flow

Entity Creation Flow

1. Client (CI/CD)
   ↓ gRPC CreateEntityRequest
2. Charybdis gRPC API
   ↓ Validate entity
   ↓ Generate UUID, timestamps
3. Entity Repository
   ↓ Serialize to protobuf bytes
   ↓ INSERT INTO entities
4. PostgreSQL
   ↓ Entity persisted
5. Event Bus
   ↓ Publish EntityCreated event (with full entity data)
6. Plugin Handlers (parallel)
   ├─ DefectDojo: Create product → Store product_id in annotations
   ├─ DependencyTrack: Create project → Store project_uuid in annotations
   └─ Custom: Custom actions
7. Repository.update_annotations()
   ↓ Merge annotations from plugins
   ↓ UPDATE entities SET annotations = ...
8. PostgreSQL
   ↓ Entity updated with external tool IDs
9. Return CreateEntityResponse to client

Backstage Integration Flow

1. Backstage Catalog Backend
   ↓ HTTP GET /yaml/locations
2. Charybdis YAML Adapter
   ↓ Query repository.list_all()
   ↓ Convert entities to Location YAML
3. Backstage receives location list
   ↓ Fetches each entity: GET /yaml/entities/:id
4. Charybdis YAML Adapter
   ↓ repository.get_by_id()
   ↓ Convert entity protobuf → Backstage YAML
5. Backstage ingests entities
   ↓ Displays in UI with all annotations
   ↓ Shows linked DefectDojo products, etc.

Core Entity Model

Protobuf Schema

message Entity {
  // System-managed fields
  string id = 1;                        // UUID (auto-generated)
  google.protobuf.Timestamp created_at = 21;
  google.protobuf.Timestamp updated_at = 22;
  
  // Backstage-compatible fields
  string api_version = 2;               // "backstage.io/v1alpha1"
  string kind = 3;                      // "Component", "API", "User", etc.
  
  // Polymorphic metadata (identifying information)
  oneof metadata {
    // Core types (built-in)
    ComponentMetadata component_metadata = 10;
    APIMetadata api_metadata = 11;
    UserMetadata user_metadata = 12;
    GroupMetadata group_metadata = 13;
    SystemMetadata system_metadata = 14;
    DomainMetadata domain_metadata = 15;
    ResourceMetadata resource_metadata = 16;
    LocationMetadata location_metadata = 17;
    TemplateMetadata template_metadata = 18;
    
    // Plugin types (dynamically added)
    // Example: DefectDojoProductMetadata defectdojo_product_metadata = 100;
  }
  
  // Polymorphic spec (configuration and behavior)
  oneof spec {
    // Core types (built-in)
    ComponentSpec component_spec = 10;
    APISpec api_spec = 11;
    UserSpec user_spec = 12;
    GroupSpec group_spec = 13;
    SystemSpec system_spec = 14;
    DomainSpec domain_spec = 15;
    ResourceSpec resource_spec = 16;
    LocationSpec location_spec = 17;
    TemplateSpec template_spec = 18;
    
    // Plugin types (dynamically added)
    // Example: DefectDojoProductSpec defectdojo_product_spec = 100;
  }
  
  // Annotations - external tool references
  // Format: "tool.com/resource-type-attribute"
  // Examples:
  //   "defectdojo.com/product-id": "42"
  //   "dependencytrack.com/project-uuid": "550e8400-..."
  //   "github.com/repo-slug": "myorg/myrepo"
  map<string, string> annotations = 20;
}

Backstage Entity Kinds Mapping

All Backstage entity kinds must be supported:

Backstage Kind Charybdis Proto Type Status
Component ComponentMetadata/Spec ✅ Done
Service ServiceMetadata/Spec ✅ Done
API APIMetadata/Spec ✅ Done
User UserMetadata/Spec ✅ Done
Group GroupMetadata/Spec ✅ Done
System SystemMetadata/Spec ✅ Done
Domain DomainMetadata/Spec ✅ Done
Resource ResourceMetadata/Spec ✅ Done
Location LocationMetadata/Spec Not planned (entities are event-driven, not file-based)
Template TemplateMetadata/Spec Phase 3 (Scaffolder)

YAML Adapter Layer

Purpose

Serve entities in Backstage-compatible YAML format over HTTP.

API Specification

1. List Locations

Endpoint: GET /yaml/locations

Purpose: Return a dynamic Location entity that lists all entities in Charybdis.

Response Format (Backstage Location YAML):

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: charybdis-all-entities
  description: Dynamic location managed by Charybdis
spec:
  type: charybdis
  targets:
    - http://charybdis.company.com/yaml/entities/service-payment
    - http://charybdis.company.com/yaml/entities/service-auth
    - http://charybdis.company.com/yaml/entities/api-payments-v1
    - http://charybdis.company.com/yaml/entities/user-john-doe
    # ... all entities

Implementation Notes:

  • Query SELECT id, kind FROM entities
  • Generate URL for each entity: /yaml/entities/{kind}-{id}
  • Return as Backstage Location YAML
  • Cache response (invalidate on entity CRUD)

2. Get Entity by ID

Endpoint: GET /yaml/entities/:id

Purpose: Return a specific entity in Backstage YAML format.

Response Format (Backstage Component YAML):

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  namespace: production
  description: Core payment processing service
  annotations:
    github.com/repo-slug: myorg/payment-service
    defectdojo.com/product-id: "42"
    dependencytrack.com/project-uuid: "550e8400-e29b-41d4-a716-446655440000"
  tags:
    - payments
    - critical
  links:
    - url: https://dashboard.company.com/payment-service
      title: Dashboard
      icon: dashboard
spec:
  type: service
  lifecycle: production
  owner: payments-team
  system: payment-system
  dependsOn:
    - component:auth-service
  providesApis:
    - payments-api-v1

Implementation Notes:

  • Query repository.get_by_id(id)
  • Convert Entity protobuf → Backstage YAML
  • Map oneof metadata/spec to appropriate YAML structure
  • Include all annotations
  • Return as text/yaml content type

3. Configuration in Backstage

Backstage app-config.yaml:

catalog:
  locations:
    - type: url
      target: http://charybdis.company.com/yaml/locations
      rules:
        - allow: [Component, API, User, Group, System, Domain, Resource]

How it works:

  1. Backstage fetches /yaml/locations on catalog refresh
  2. Parses Location YAML to get entity URLs
  3. Fetches each entity URL (/yaml/entities/:id)
  4. Ingests entities into Backstage catalog
  5. Displays in UI with all metadata and annotations

Plugin System

Plugin Architecture

Charybdis supports two types of plugins, both compile-time integrated:

1. Event-Driven Plugins

React to entity lifecycle events (create, update, delete):

  • Example: DefectDojo, DependencyTrack
  • Implement: EventDrivenPlugin trait
  • Provide: ResourceHandler implementations
  • Triggered by: Entity CRUD operations

2. Sync Plugins

Pull data from external sources on a schedule:

  • Example: Okta, Keycloak, Active Directory
  • Implement: SyncPlugin trait
  • Scheduled by: Cron expressions
  • Triggered by: Time-based schedule or manual API call

Generic Plugin Utilities (2025-01-15)

All plugins have access to reusable utilities in src/plugins/:

PluginHttpClient

Generic HTTP client with multiple auth methods:

  • Token authentication (DefectDojo)
  • Bearer authentication (modern APIs)
  • API Key authentication (DependencyTrack)
  • Basic authentication (JIRA, Bitbucket)
  • Methods: get(), post(), put(), patch(), delete()

AnnotationHelper

Consistent API for storing/retrieving plugin metadata:

  • set_id() / get_id() - Store external resource IDs
  • set() / get() - Store arbitrary metadata
  • Enforces naming: {plugin}.com/{resource}-id
  • Prevents annotation conflicts between plugins

FieldMapper

Maps entity fields to external tool formats:

  • Direct field paths: "metadata.name"
  • Static values: {"value": "Web Application"}
  • Complex mappings with entity resolution
  • Repository integration for cross-entity queries

DateUtils

Common date/time operations:

  • today(), today_plus_days(n) - Date formatting
  • engagement_date_range(days) - For time-bound resources
  • iso_timestamp() - ISO 8601 timestamps

Plugin Responsibilities

Plugins can:

  1. Define custom entity types (via protobuf)
  2. React to entity events (via ResourceHandler trait)
  3. Call external APIs (using PluginHttpClient)
  4. Store metadata (using AnnotationHelper)
  5. Map fields (using FieldMapper)
  6. Schedule syncs (SyncPlugins with cron)
  7. (Future) Extend gRPC API (custom services)

Plugin Lifecycle

1. Development
   ├─ Define protobuf schema (metadata/spec messages)
   ├─ Implement EventHandler trait
   ├─ Implement external API client
   └─ Add configuration schema

2. Registration
   ├─ Add to plugins.toml (enabled = true)
   └─ Add to Cargo.toml dependencies

3. Build Time
   ├─ build.rs reads plugins.toml
   ├─ Generates entities.proto with plugin types
   ├─ Compiles all protobuf to Rust code
   └─ Links plugin into binary

4. Runtime
   ├─ Load plugin configuration from config file
   ├─ Initialize plugin handler
   ├─ Subscribe to event bus
   └─ React to entity events

Plugin Configuration System

File: config.toml (TOML format with ${VAR} env var substitution)

# Server configuration
[server]
grpc_host = "[::1]"
grpc_port = 50051

[server.yaml_adapter]
enabled = true
host = "0.0.0.0"
port = 8080

# Database
[database]
url = "${DATABASE_URL}"
max_connections = 10

# Plugin configurations
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_API_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
engagement_type = "CI/CD"

[plugins.keycloak]
enabled = false
# base_url = "https://keycloak.company.com"
# realm = "master"

[plugins.dependencytrack]
enabled = false
# base_url = "https://dependencytrack.company.com"
# api_key = "${DEPENDENCYTRACK_API_KEY}"

See config.toml.example for the full reference with all options documented.

Configuration loading (src/config.rs):

pub struct Config {
    pub server: ServerConfig,
    pub database: DatabaseConfig,
    pub security: SecurityConfig,
    pub telemetry: TelemetryConfig,
    pub plugins: PluginsConfig,
}

impl Config {
    pub fn from_file(path: &str) -> Result<Self> {
        let content = std::fs::read_to_string(path)?;
        // Substitute ${VAR} with environment variable values
        let content = substitute_env_vars(&content);
        let config: Config = toml::from_str(&content)?;
        Ok(config)
    }
}

Each plugin implements:

pub trait Plugin {
    fn name(&self) -> &str;
    fn plugin_type(&self) -> PluginType;
    fn load_config(&mut self, config: PluginConfig) -> Result<()>;
    fn validate_config(&self) -> Result<()>;
}

Event Handler Behavior Configuration

Plugins can configure different behaviors based on:

  • Entity kind
  • Event type (created/updated/deleted)
  • Custom conditions

Example: DefectDojo plugin config (config.toml)

[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_API_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
engagement_type = "CI/CD"

Handler Implementation:

impl EventHandler for DefectDojoHandler {
    async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
        // Check if plugin enabled
        if !self.config.enabled {
            return Ok(());
        }
        
        // Match event type to configured behavior
        let behavior = match event.event_type {
            EntityEventType::Created => &self.config.on_entity_created,
            EntityEventType::Updated => &self.config.on_entity_updated,
            EntityEventType::Deleted => &self.config.on_entity_deleted,
        };
        
        // Find matching rule for this entity kind
        for rule in behavior {
            if rule.matches(&event) {
                return self.execute_action(&rule.action, event).await;
            }
        }
        
        Ok(())
    }
}

Future: Plugin API Extensions

Future capability: Plugins can expose custom gRPC endpoints.

Example: DefectDojo Query Plugin

Plugin extends API with custom service:

// plugins/defectdojo/proto/defectdojo_service.proto

service DefectDojoService {
  // Get findings for a specific entity
  rpc GetEntityFindings(GetEntityFindingsRequest) returns (GetEntityFindingsResponse);
  
  // Get all active vulnerabilities
  rpc GetActiveVulnerabilities(GetActiveVulnerabilitiesRequest) returns (GetActiveVulnerabilitiesResponse);
  
  // Sync entity with DefectDojo
  rpc SyncEntity(SyncEntityRequest) returns (SyncEntityResponse);
}

message GetEntityFindingsRequest {
  string entity_id = 1;
  string severity_filter = 2;  // "Critical", "High", etc.
  bool active_only = 3;
}

message GetEntityFindingsResponse {
  repeated Finding findings = 1;
  int32 total_count = 2;
}

message Finding {
  int32 id = 1;
  string title = 2;
  string severity = 3;
  string description = 4;
  google.protobuf.Timestamp date = 5;
  bool active = 6;
}

Implementation:

pub struct DefectDojoServiceImpl {
    client: DefectDojoClient,
    repository: EntityRepository,
}

#[async_trait]
impl DefectDojoService for DefectDojoServiceImpl {
    async fn get_entity_findings(
        &self,
        request: Request<GetEntityFindingsRequest>,
    ) -> Result<Response<GetEntityFindingsResponse>, Status> {
        let req = request.into_inner();
        
        // Get entity to retrieve DefectDojo product ID
        let entity = self.repository.get_by_id(&req.entity_id).await?;
        let product_id = entity
            .annotations
            .get("defectdojo.com/product-id")
            .ok_or_else(|| Status::not_found("No DefectDojo product linked"))?;
        
        // Fetch findings from DefectDojo API
        let findings = self.client
            .get_findings(product_id, &req.severity_filter, req.active_only)
            .await?;
        
        Ok(Response::new(GetEntityFindingsResponse {
            findings: findings.into_iter().map(|f| f.into()).collect(),
            total_count: findings.len() as i32,
        }))
    }
}

Registration in main.rs:

// Register plugin's custom gRPC service
if defectdojo_plugin.is_enabled() {
    let defectdojo_service = DefectDojoServiceServer::new(
        DefectDojoServiceImpl::new(defectdojo_client, repository.clone())
    );
    
    server = server.add_service(defectdojo_service);
}

Client usage:

grpcurl -d '{
  "entity_id": "service-payment",
  "severity_filter": "Critical",
  "active_only": true
}' localhost:50051 charybdis.plugins.defectdojo.DefectDojoService/GetEntityFindings

Technology Decisions

Why gRPC Only?

Decision: Charybdis core provides only gRPC. Teams add REST/GraphQL as needed.

Rationale:

  • Strong typing: Protobuf schemas ensure correctness across languages
  • Performance: Binary protocol, efficient for CI/CD use cases
  • Multi-language: Official clients for Python, Go, Node.js, Java, Rust, C++, C#, etc.
  • Streaming: Built-in bidirectional streaming (future: real-time updates)
  • Code generation: Automatic client generation eliminates boilerplate
  • Focus: Core team maintains one high-quality API surface

Teams can add REST if needed:

grpc-gateway or Envoy → Charybdis gRPC

Why PostgreSQL + JSONB?

Decision: Use PostgreSQL with protobuf BYTEA storage and JSONB annotations.

Rationale:

  • Scale: Handles 1000s of entities easily (tested: 170k resources)
  • JSONB performance: GIN indexes enable fast annotation queries
  • Familiarity: Most teams already run PostgreSQL
  • Reliability: ACID transactions, mature ecosystem
  • Future-proof: Can add read replicas, partitioning if needed

Alternative considered: MongoDB

  • Verdict: PostgreSQL with JSONB provides same flexibility with better consistency guarantees

Why Compile-Time Plugins?

Decision: Plugins are compile-time integrated, not runtime loaded.

Rationale:

  • Type safety: Rust compiler ensures correctness
  • Performance: No dynamic loading overhead
  • Simplicity: No plugin version compatibility matrix
  • Security: No arbitrary code execution
  • Trade-off: Requires rebuild to add plugins (acceptable for infrastructure tool)

Community plugins: Published as crates, teams include in their build.

Why Event-Driven?

Decision: Plugins react to events after entity persistence.

Rationale:

  • Decoupling: Plugins can't break core CRUD operations
  • Resilience: Plugin failures are logged, not propagated
  • Async: Long-running operations don't block API
  • Extensibility: Add plugins without modifying core
  • Observability: Events provide audit trail

Trade-off: Plugins can't prevent entity creation (validation must be in core or client).

Scale Estimates

Reference Deployment

Company: ~100 engineers
Backstage Entities:

  • Components: ~1,000
  • APIs: ~1,000
  • Users: ~700
  • Groups: ~200
  • Systems: ~50
  • Domains: ~150
  • Resources: ~170,000

Charybdis Requirements:

  • PostgreSQL: ~500MB storage (with JSONB annotations)
  • Memory: ~256MB for Charybdis service
  • CPU: Minimal (mostly I/O bound)

10x Scale (1,000 engineers, 1.7M resources):

  • PostgreSQL: ~5GB storage
  • Memory: ~512MB
  • CPU: Still minimal with proper indexing

Bottlenecks:

  • JSONB queries on annotations (solved with GIN indexes)
  • Event bus throughput (solved with Redis/RabbitMQ backend)
  • Plugin external API rate limits (solved with plugin-level queuing)

Development Roadmap

See VISION.md for the full product vision and strategic roadmap.

Phase 0: Catalog Foundation (Done)

  • ✅ Zero-migration database architecture
  • ✅ gRPC API with protobuf (full entity CRUD)
  • ✅ Event bus system (MemoryEventBus)
  • ✅ Event dispatcher architecture (generic + plugin-specific)
  • ✅ All Backstage entity kinds (Component, System, API, User, Group, Domain, Resource)
  • ✅ YAML adapter endpoints (Backstage compatibility)
  • ✅ gRPC field masks for partial updates
  • ✅ OpenTelemetry observability (traces, metrics, logs)
  • ✅ mTLS + RBAC security framework
  • ✅ Config loading from TOML with env var substitution
  • ✅ Plugin trait architecture (EventDriven + Sync)
  • ✅ Generic plugin utilities (HttpClient, AnnotationHelper, FieldMapper, DateUtils)
  • ✅ DefectDojo plugin (products, engagements, owner resolution)
  • ✅ Keycloak plugin (user/group sync with annotations)

Phase 1: Security Core (Current Focus)

Goal: Native vulnerability management and scan ingestion.

  • Vulnerability/Observation entity kind (protobuf + storage)
  • ScanService gRPC endpoint for scan ingestion
  • SARIF parser (covers majority of modern scanners)
  • CycloneDX parser (SBOMs + vulnerability data)
  • SPDX parser (license data)
  • Assessment workflow (triage, accept risk, remediate)
  • Rules engine for auto-assessment
  • Security gates (severity thresholds per product)
  • License tracking and policy engine

Phase 2: Compliance & Integrations

Goal: Compliance frameworks and integration plugins.

  • Compliance framework mappings (NIS2, SOC2, DORA)
  • VEX document support (CSAF, OpenVEX)
  • Export/reporting (PDF, Excel)
  • Notification plugins (Slack, Teams, email)
  • Issue tracker plugins (Jira, GitHub, GitLab)
  • Redis event bus backend

Phase 3: Scaffolder & Ecosystem

Goal: Service scaffolding and community growth.

  • Service scaffolder (Git-native templates)
  • Event-driven provisioning on scaffold
  • Plugin SDK documentation
  • Helm chart & 1-click deploy
  • Community plugin registry

Success Criteria

Phase 1 Success (Security Core)

  • Ingest SARIF scan results via gRPC
  • Vulnerabilities linked to catalog entities
  • Rules engine auto-assesses based on severity/component
  • Security gates block on threshold violations
  • Assessment workflow (triage → accept/remediate)
  • License data ingested from CycloneDX/SPDX

Production Success

  • 1,000+ entities managed with security posture
  • <100ms p99 latency for CRUD operations
  • Single helm install deployment
  • Documentation complete
  • 3+ integration plugins in production

Key Decisions Summary

Decision Choice Rationale
API Protocol gRPC only Strong typing, multi-language, performance
Frontend Backstage YAML adapter Compatible with existing Backstage deployments
Security features Native (core) First-class, not plugins. Vulns, compliance, gates in the core
External integrations Plugins Slack, Jira, etc. as event-driven plugins
Database PostgreSQL + JSONB Scale, familiarity, ACID
Plugin Loading Compile-time Type safety, performance, security
Event Model Post-persistence Resilience, decoupling
Schema Evolution Protobuf only Zero migrations
Configuration TOML + env vars Flexibility, Rust-native
Deployment Single binary + Docker + K8s Cloud-native, minimal footprint

References

Document Version History

Version Date Changes
1.0 2025-01-03 Initial comprehensive architecture document
1.1 2025-01-15 Added dual plugin architecture, generic utilities, updated roadmap
2.0 2026-04-02 Pivoted to security-native platform engineering tool. Extracted product vision to VISION.md. Added native vulnerability management, compliance frameworks to architecture. Updated roadmap and success criteria.

This document covers architecture and technical decisions. For product vision and strategic direction, see VISION.md. All code and documentation must align with these documents.

Reference in New Issue View Git Blame Copy Permalink