charybdis/docs/core-concepts.md

Core Concepts

Charybdis is a security-native platform engineering tool that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.

This page explains the key concepts you need to understand.

Architecture Overview

graph TB
    subgraph "Sources"
        A1[CI/CD Pipelines]
        A2[Security Scanners]
        A3[IaC Tools]
    end

    subgraph "Charybdis Core"
        B1[gRPC API]
        B3[Authentication & Authorization]
        C1[Entity Service]
        C4[Vulnerability Engine]
        C5[Security Gates & Rules]
        C2[Event Bus]
    end

    subgraph "Integrations - Plugins"
        D1[Slack / Teams]
        D2[Jira / GitHub]
        D3[Custom Plugins]
    end

    subgraph "Data Layer"
        E1[(PostgreSQL)]
    end

    A1 -.gRPC.-> B1
    A2 -.Scan Results.-> B1
    A3 -.gRPC.-> B1

    B1 --> B3
    B3 --> C1
    B3 --> C4
    C4 --> C5
    C1 --> E1
    C4 --> E1
    C1 --> C2
    C4 --> C2

    C2 -.Events.-> D1
    C2 -.Events.-> D2
    C2 -.Events.-> D3

    style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
    style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
    style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff

Entities

An entity is the core data model in Charybdis, representing any cataloged item in your software ecosystem.

Entity Structure

Every entity has three main parts:

classDiagram
    class Entity {
        +string id
        +string kind
        +metadata
        +spec
        +annotations
        +timestamps
    }
    
    class Metadata {
        +string name
        +string namespace
        +string description
        +labels
        +links
        +tags
    }
    
    class Spec {
        +string type
        +string lifecycle
        +string owner
        +dependencies
    }
    
    Entity --> Metadata
    Entity --> Spec

Entity Kinds

Charybdis supports all standard Backstage entity kinds:

Kind	Description	Example
Service	Individual microservices or applications	payment-api
Component	Reusable libraries, SDKs, modules	auth-sdk
System	Collections of services working together	e-commerce-platform
API	Interfaces exposed by components	payments-rest-api
User	Individual people	john.doe
Group	Teams and organizational units	team-payments
Domain	Business domains	payments, shipping
Resource	Infrastructure resources	payments-db, cache-cluster

Example: Service

{
  "kind": "Service",
  "service_metadata": {
    "name": "payment-api",
    "namespace": "production",
    "description": "Payment processing service",
    "labels": { "team": "payments" }
  },
  "service_spec": {
    "type": "service",
    "lifecycle": "production",
    "owner": "team-payments",
    "system": "payment-system"
  }
}

Example: Component

{
  "kind": "Component",
  "component_metadata": {
    "name": "auth-sdk",
    "namespace": "shared"
  },
  "component_spec": {
    "type": "library",
    "lifecycle": "production",
    "owner": "platform-team"
  }
}

Example: System

{
  "kind": "System",
  "system_metadata": {
    "name": "e-commerce-platform",
    "namespace": "production",
    "description": "Complete e-commerce system"
  },
  "system_spec": {
    "owner": "platform-team",
    "domain": "retail"
  }
}

Metadata Fields

Field	Type	Description	Required
name	string	Entity name (unique within namespace)	✅
namespace	string	Logical grouping (e.g., "production", "staging")	✅
description	string	Human-readable description	❌
labels	map	Key-value pairs for categorization	❌
tags	array	Search tags	❌
links	array	External URLs (dashboards, docs, etc.)	❌

Annotations

Annotations store integration-specific metadata:

{
  "annotations": {
    "github.com/repo-slug": "myorg/payment-service",
    "defectdojo.com/product-id": "123",
    "dependencytrack.com/project-uuid": "550e8400...",
    "pagerduty.com/service-id": "PXYZ123",
    "grafana.com/dashboard-url": "https://..."
  }
}

Best Practices:

• Use domain-style keys (tool.com/key)
• Store tool-specific IDs
• Keep values as strings
• Use for integration metadata only

Events

Charybdis uses an event-driven architecture to trigger actions when entities change.

Event Flow

sequenceDiagram
    participant Client
    participant API as Entity Service
    participant Bus as Event Bus
    participant Plugin1 as DefectDojo Plugin
    participant Plugin2 as Dependency-Track
    
    Client->>API: CreateEntity(service)
    API->>API: Store entity
    API->>Bus: Emit EntityCreated event
    Bus->>Plugin1: Handle event
    Bus->>Plugin2: Handle event
    Plugin1-->>Plugin1: Create DefectDojo product
    Plugin2-->>Plugin2: Create DT project
    API-->>Client: Return created entity
    
    Note over Bus,Plugin2: Asynchronous processing

Event Types

Event	Trigger	Plugins Receive
EntityCreated	New entity created	Full entity data
EntityUpdated	Entity modified	Updated entity + changes
EntityDeleted	Entity removed	Entity ID + metadata

Event Structure

pub enum EntityEvent {
    Created {
        entity: Entity,
        timestamp: DateTime<Utc>,
    },
    Updated {
        entity: Entity,
        previous: Entity,
        timestamp: DateTime<Utc>,
    },
    Deleted {
        id: String,
        metadata: Metadata,
        timestamp: DateTime<Utc>,
    },
}

Vulnerabilities & Security (Phase 1 — Planned)

Note

: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See VISION.md for the roadmap.

Security is a first-class concept in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.

Vulnerability Lifecycle

sequenceDiagram
    participant Scanner
    participant API as Charybdis API
    participant Rules as Rules Engine
    participant Gate as Security Gate
    participant Bus as Event Bus
    participant Plugin as Slack / Jira

    Scanner->>API: IngestScan(SARIF report)
    API->>API: Parse & create Vulnerability entities
    API->>API: Link vulnerabilities to Component
    API->>Rules: Evaluate auto-assessment rules
    Rules-->>API: Auto-assess (e.g., accept known low-risk)
    API->>Gate: Check security gate thresholds
    alt Gate Passed
        Gate-->>API: OK
    else Gate Failed
        Gate->>Bus: GateFailed event
        Bus->>Plugin: Alert #security channel
    end
    API->>Bus: VulnerabilitiesIngested event
    Bus->>Plugin: Notify / create tickets

Scan Ingestion

Charybdis ingests scan results natively. You don't need an external vulnerability management tool.

Format	Coverage	Use Case
SARIF	60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.)	SAST, DAST, secrets
CycloneDX	SBOMs + vulnerability data	SCA, license
SPDX	License and package data	License compliance

Assessments

Each vulnerability can be assessed:

Status	Meaning
Open	New, unreviewed vulnerability
In Triage	Under review by security team
Accepted	Risk accepted with justification
Remediated	Fixed, pending verification
False Positive	Not a real vulnerability
Auto-Assessed	Automatically assessed by rules engine

Security Gates

Security gates define thresholds per product:

payment-api:
  critical: 0     # No critical vulns allowed
  high: 5         # Up to 5 high
  medium: 20      # Up to 20 medium

When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).

Rules Engine

Rules auto-assess vulnerabilities based on patterns:

• Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
• Scanner source (e.g., "all informational from ZAP → auto-accept")
• Known patterns (e.g., "CVE-XXXX already accepted org-wide")

License Compliance (Phase 2)

Track licenses across your dependency tree:

• Ingest license data from CycloneDX/SPDX
• Define license policies (allowed, restricted, banned)
• Flag violations per component
• Compliance reporting

Storage Model

Charybdis uses PostgreSQL with JSONB for schema-less storage.

Database Schema

CREATE TABLE entities (
    id UUID PRIMARY KEY,
    kind TEXT NOT NULL,
    entity_data BYTEA NOT NULL,         -- Protobuf binary
    annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

CREATE INDEX idx_entities_kind ON entities(kind);
CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);

Why Protobuf + JSONB?

Advantages:

• ✅ No schema migrations — new entity kinds via protobuf oneof, schema never changes
• ✅ Protobuf binary storage for compact, versioned entity data
• ✅ JSONB annotations for fast querying with GIN indexes
• ✅ Forward/backward compatibility built-in

Example Query:

-- Find entities with specific annotation
SELECT * FROM entities 
WHERE annotations->>'defectdojo.com/product-id' = '42';

Security Model

Charybdis implements defense-in-depth security.

Authentication: mTLS

sequenceDiagram
    participant Client
    participant TLS as TLS Layer
    participant Auth as Auth Interceptor
    participant Service as Entity Service
    
    Client->>TLS: Connect with client certificate
    TLS->>TLS: Validate certificate
    TLS->>Auth: Extract certificate
    Auth->>Auth: Parse identity (CN, OU, O)
    Auth->>Auth: Map to role
    Client->>Auth: Request (with identity)
    Auth->>Auth: Check permissions
    alt Authorized
        Auth->>Service: Forward request
        Service-->>Client: Response
    else Denied
        Auth-->>Client: PermissionDenied error
    end

Authorization: RBAC

Role Mapping:

Certificate (CN, OU, O) → RBAC Role → Permissions

Default Roles:

Role	Certificate OU	Permissions
platform	platform-team	Full access (CRUD + list)
automation	automation	Create, read, update, list
plugin	plugins	Read, list only

Permissions

Permission	Operations	Required For
entity:create	Create new entities	CreateEntity
entity:read	Get entity by ID	GetEntity
entity:update	Modify entities	UpdateEntity
entity:delete	Remove entities	DeleteEntity
entity:list	List all entities	ListEntities

Plugin System

Core features (catalog, vulnerabilities, compliance) are native. Plugins handle integrations with external systems.

Plugin Architecture

graph LR
    A[Entity/Vuln Event] --> B[Event Bus]
    B --> C{Plugin Manager}
    C --> D[Slack Plugin]
    C --> E[Jira Plugin]
    C --> F[Custom Plugin]

    D --> G[Slack API]
    E --> H[Jira API]
    F --> I[Your Tool API]

    style C fill:#4A90E2,stroke:#2E5C8A

What's Native vs. Plugin

Native (core)	Status	Plugin (integration)	Status
Software catalog	Done	DefectDojo sync	Done
Vulnerability management	Phase 1	Keycloak user sync	Done
Security gates & rules	Phase 1	Slack / Teams notifications	Planned
Assessment workflow	Phase 1	Jira / GitHub issue creation	Planned
License compliance	Phase 2	Custom integrations	Framework ready
TechDocs	Future

Plugin Lifecycle

1. Configuration - Load plugin settings from plugins.toml
2. Initialization - Plugin registers event handlers
3. Event Processing - Plugin receives events asynchronously (entity, vulnerability, gate events)
4. External Integration - Plugin calls external tool APIs
5. Error Handling - Failed plugins don't affect core service

Plugin Configuration

Example plugins.toml:

[plugins.slack]
enabled = true
webhook_url = "${SLACK_WEBHOOK_URL}"

[[plugins.slack.on_gate_failed]]
channel = "#security-alerts"

[[plugins.slack.on_entity_created]]
channel = "#platform"

See Plugin Guide for details.

Data Consistency

Eventual Consistency

Charybdis uses eventual consistency for plugin integrations:

• Entity CRUD operations are immediately consistent
• Plugin synchronization is eventually consistent
• Events are processed asynchronously

graph LR
    A[CreateEntity] -->|Immediate| B[Entity Stored]
    B -->|Async| C[Event Emitted]
    C -->|Async| D[Plugin Processing]
    D -->|Eventual| E[External Tool Synced]
    
    style B fill:#00C851
    style E fill:#ffbb33

Guarantees

Operation	Consistency	Guarantee
Entity CRUD	Strong	Immediate
Entity queries	Strong	Read-your-writes
Event delivery	At-least-once	May retry
Plugin sync	Eventual	Best-effort

Performance Characteristics

Scalability

• Entities: Tested with 100,000+ entities
• Throughput: 1,000+ requests/second
• Latency: Sub-millisecond average
• Concurrency: Tokio async runtime

Resource Usage

Typical resource consumption:

Component	CPU	Memory	Storage
Charybdis	< 5%	~50 MB	Minimal
PostgreSQL	~10%	~256 MB	Depends on entity count

Optimization Tips

1. Index annotations used for frequent queries
2. Use connection pooling (built-in)
3. Enable query caching in PostgreSQL
4. Monitor event bus queue depth

Backstage Migration

Charybdis includes a YAML adapter for teams migrating from Backstage. This is a migration path, not the primary interface.

How It Works

graph LR
    A[Charybdis Entity] --> B[YAML Adapter]
    B --> C[Backstage YAML Format]
    C --> D[Backstage Catalog]

    style B fill:#4A90E2,stroke:#2E5C8A

Point Backstage at Charybdis and stop maintaining catalog-info.yaml files:

catalog:
  locations:
    - type: url
      target: http://charybdis:8080/yaml/locations

Recommended Migration Path

1. Start with Charybdis + YAML adapter feeding your existing Backstage
2. Adopt Charybdis gRPC API for CI/CD integrations and security scanning
3. Leverage event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
4. Optionally retire Backstage when Charybdis covers your catalog needs

Best Practices

Entity Design

✅ DO:

• Use descriptive names
• Group by namespace
• Add relevant labels
• Include documentation links
• Set appropriate owners

❌ DON'T:

• Store sensitive data in metadata
• Use very long descriptions
• Create deeply nested hierarchies
• Duplicate data across entities

Naming Conventions

<entity-type>-<purpose>-<environment>

Examples:
- payment-api-prod
- user-service-staging
- auth-library
- e-commerce-system

Metadata Organization

{
  "labels": {
    "team": "payments",           // Ownership
    "tier": "critical",            // Importance
    "environment": "production"    // Deployment
  },
  "tags": ["pci-compliant", "public-api"],
  "links": [
    { "url": "...", "title": "Dashboard" },
    { "url": "...", "title": "Documentation" }
  ]
}

Next Steps

• Read the Vision & Roadmap to understand where Charybdis is going
• Configure Security for production (mTLS + RBAC)
• Explore Plugins for external integrations
• Review the Architecture for technical deep dive

---

Questions? Open an issue.