charybdis/docs/core-concepts.md

# Core Concepts

Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.

This page explains the key concepts you need to understand.

## Architecture Overview

```mermaid
graph TB
    subgraph "Sources"
        A1[CI/CD Pipelines]
        A2[Security Scanners]
        A3[IaC Tools]
    end

    subgraph "Charybdis Core"
        B1[gRPC API]
        B3[Authentication & Authorization]
        C1[Entity Service]
        C4[Vulnerability Engine]
        C5[Security Gates & Rules]
        C2[Event Bus]
    end

    subgraph "Integrations - Plugins"
        D1[Slack / Teams]
        D2[Jira / GitHub]
        D3[Custom Plugins]
    end

    subgraph "Data Layer"
        E1[(PostgreSQL)]
    end

    A1 -.gRPC.-> B1
    A2 -.Scan Results.-> B1
    A3 -.gRPC.-> B1

    B1 --> B3
    B3 --> C1
    B3 --> C4
    C4 --> C5
    C1 --> E1
    C4 --> E1
    C1 --> C2
    C4 --> C2

    C2 -.Events.-> D1
    C2 -.Events.-> D2
    C2 -.Events.-> D3

    style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
    style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
    style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
```

## Entities

An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.

### Entity Structure

Every entity has three main parts:

```mermaid
classDiagram
    class Entity {
        +string id
        +string kind
        +metadata
        +spec
        +annotations
        +timestamps
    }
    
    class Metadata {
        +string name
        +string namespace
        +string description
        +labels
        +links
        +tags
    }
    
    class Spec {
        +string type
        +string lifecycle
        +string owner
        +dependencies
    }
    
    Entity --> Metadata
    Entity --> Spec
```

### Entity Kinds

Charybdis supports all standard Backstage entity kinds:

| Kind | Description | Example |
|------|-------------|---------|
| **Service** | Individual microservices or applications | `payment-api` |
| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
| **System** | Collections of services working together | `e-commerce-platform` |
| **API** | Interfaces exposed by components | `payments-rest-api` |
| **User** | Individual people | `john.doe` |
| **Group** | Teams and organizational units | `team-payments` |
| **Domain** | Business domains | `payments`, `shipping` |
| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |

#### Example: Service

```json
{
  "kind": "Service",
  "service_metadata": {
    "name": "payment-api",
    "namespace": "production",
    "description": "Payment processing service",
    "labels": { "team": "payments" }
  },
  "service_spec": {
    "type": "service",
    "lifecycle": "production",
    "owner": "team-payments",
    "system": "payment-system"
  }
}
```

#### Example: Component

```json
{
  "kind": "Component",
  "component_metadata": {
    "name": "auth-sdk",
    "namespace": "shared"
  },
  "component_spec": {
    "type": "library",
    "lifecycle": "production",
    "owner": "platform-team"
  }
}
```

#### Example: System

```json
{
  "kind": "System",
  "system_metadata": {
    "name": "e-commerce-platform",
    "namespace": "production",
    "description": "Complete e-commerce system"
  },
  "system_spec": {
    "owner": "platform-team",
    "domain": "retail"
  }
}
```

### Metadata Fields

| Field | Type | Description | Required |
|-------|------|-------------|----------|
| `name` | string | Entity name (unique within namespace) | ✅ |
| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
| `description` | string | Human-readable description | ❌ |
| `labels` | map | Key-value pairs for categorization | ❌ |
| `tags` | array | Search tags | ❌ |
| `links` | array | External URLs (dashboards, docs, etc.) | ❌ |

### Annotations

Annotations store integration-specific metadata:

```json
{
  "annotations": {
    "github.com/repo-slug": "myorg/payment-service",
    "defectdojo.com/product-id": "123",
    "dependencytrack.com/project-uuid": "550e8400...",
    "pagerduty.com/service-id": "PXYZ123",
    "grafana.com/dashboard-url": "https://..."
  }
}
```

**Best Practices**:
- Use domain-style keys (`tool.com/key`)
- Store tool-specific IDs
- Keep values as strings
- Use for integration metadata only

## Events

Charybdis uses an **event-driven architecture** to trigger actions when entities change.

### Event Flow

```mermaid
sequenceDiagram
    participant Client
    participant API as Entity Service
    participant Bus as Event Bus
    participant Plugin1 as DefectDojo Plugin
    participant Plugin2 as Dependency-Track
    
    Client->>API: CreateEntity(service)
    API->>API: Store entity
    API->>Bus: Emit EntityCreated event
    Bus->>Plugin1: Handle event
    Bus->>Plugin2: Handle event
    Plugin1-->>Plugin1: Create DefectDojo product
    Plugin2-->>Plugin2: Create DT project
    API-->>Client: Return created entity
    
    Note over Bus,Plugin2: Asynchronous processing
```

### Event Types

| Event | Trigger | Plugins Receive |
|-------|---------|----------------|
| `EntityCreated` | New entity created | Full entity data |
| `EntityUpdated` | Entity modified | Updated entity + changes |
| `EntityDeleted` | Entity removed | Entity ID + metadata |

### Event Structure

```rust
pub enum EntityEvent {
    Created {
        entity: Entity,
        timestamp: DateTime<Utc>,
    },
    Updated {
        entity: Entity,
        previous: Entity,
        timestamp: DateTime<Utc>,
    },
    Deleted {
        id: String,
        metadata: Metadata,
        timestamp: DateTime<Utc>,
    },
}
```

## Vulnerabilities & Security (Phase 1 — Planned)

> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.

Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.

### Vulnerability Lifecycle

```mermaid
sequenceDiagram
    participant Scanner
    participant API as Charybdis API
    participant Rules as Rules Engine
    participant Gate as Security Gate
    participant Bus as Event Bus
    participant Plugin as Slack / Jira

    Scanner->>API: IngestScan(SARIF report)
    API->>API: Parse & create Vulnerability entities
    API->>API: Link vulnerabilities to Component
    API->>Rules: Evaluate auto-assessment rules
    Rules-->>API: Auto-assess (e.g., accept known low-risk)
    API->>Gate: Check security gate thresholds
    alt Gate Passed
        Gate-->>API: OK
    else Gate Failed
        Gate->>Bus: GateFailed event
        Bus->>Plugin: Alert #security channel
    end
    API->>Bus: VulnerabilitiesIngested event
    Bus->>Plugin: Notify / create tickets
```

### Scan Ingestion

Charybdis ingests scan results natively. You don't need an external vulnerability management tool.

| Format | Coverage | Use Case |
|--------|----------|----------|
| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
| **CycloneDX** | SBOMs + vulnerability data | SCA, license |
| **SPDX** | License and package data | License compliance |

### Assessments

Each vulnerability can be assessed:

| Status | Meaning |
|--------|---------|
| **Open** | New, unreviewed vulnerability |
| **In Triage** | Under review by security team |
| **Accepted** | Risk accepted with justification |
| **Remediated** | Fixed, pending verification |
| **False Positive** | Not a real vulnerability |
| **Auto-Assessed** | Automatically assessed by rules engine |

### Security Gates

Security gates define thresholds per product:

```
payment-api:
  critical: 0     # No critical vulns allowed
  high: 5         # Up to 5 high
  medium: 20      # Up to 20 medium
```

When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).

### Rules Engine

Rules auto-assess vulnerabilities based on patterns:

- Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
- Scanner source (e.g., "all informational from ZAP → auto-accept")
- Known patterns (e.g., "CVE-XXXX already accepted org-wide")

### License Compliance (Phase 2)

Track licenses across your dependency tree:

- Ingest license data from CycloneDX/SPDX
- Define license policies (allowed, restricted, banned)
- Flag violations per component
- Compliance reporting

## Storage Model

Charybdis uses PostgreSQL with JSONB for schema-less storage.

### Database Schema

```sql
CREATE TABLE entities (
    id UUID PRIMARY KEY,
    kind TEXT NOT NULL,
    entity_data BYTEA NOT NULL,         -- Protobuf binary
    annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

CREATE INDEX idx_entities_kind ON entities(kind);
CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
```

### Why Protobuf + JSONB?

**Advantages**:
- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
- ✅ Protobuf binary storage for compact, versioned entity data
- ✅ JSONB annotations for fast querying with GIN indexes
- ✅ Forward/backward compatibility built-in

**Example Query**:
```sql
-- Find entities with specific annotation
SELECT * FROM entities 
WHERE annotations->>'defectdojo.com/product-id' = '42';
```

## Security Model

Charybdis implements defense-in-depth security.

### Authentication: mTLS

```mermaid
sequenceDiagram
    participant Client
    participant TLS as TLS Layer
    participant Auth as Auth Interceptor
    participant Service as Entity Service
    
    Client->>TLS: Connect with client certificate
    TLS->>TLS: Validate certificate
    TLS->>Auth: Extract certificate
    Auth->>Auth: Parse identity (CN, OU, O)
    Auth->>Auth: Map to role
    Client->>Auth: Request (with identity)
    Auth->>Auth: Check permissions
    alt Authorized
        Auth->>Service: Forward request
        Service-->>Client: Response
    else Denied
        Auth-->>Client: PermissionDenied error
    end
```

### Authorization: RBAC

**Role Mapping**:

```
Certificate (CN, OU, O) → RBAC Role → Permissions
```

**Default Roles**:

| Role | Certificate OU | Permissions |
|------|---------------|-------------|
| `platform` | `platform-team` | Full access (CRUD + list) |
| `automation` | `automation` | Create, read, update, list |
| `plugin` | `plugins` | Read, list only |

### Permissions

| Permission | Operations | Required For |
|------------|-----------|--------------|
| `entity:create` | Create new entities | CreateEntity |
| `entity:read` | Get entity by ID | GetEntity |
| `entity:update` | Modify entities | UpdateEntity |
| `entity:delete` | Remove entities | DeleteEntity |
| `entity:list` | List all entities | ListEntities |

## Plugin System

Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.

### Plugin Architecture

```mermaid
graph LR
    A[Entity/Vuln Event] --> B[Event Bus]
    B --> C{Plugin Manager}
    C --> D[Slack Plugin]
    C --> E[Jira Plugin]
    C --> F[Custom Plugin]

    D --> G[Slack API]
    E --> H[Jira API]
    F --> I[Your Tool API]

    style C fill:#4A90E2,stroke:#2E5C8A
```

### What's Native vs. Plugin

| Native (core) | Status | Plugin (integration) | Status |
|---|---|---|---|
| Software catalog | Done | DefectDojo sync | Done |
| Vulnerability management | Phase 1 | Keycloak user sync | Done |
| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
| License compliance | Phase 2 | Custom integrations | Framework ready |
| TechDocs | Future | | |

### Plugin Lifecycle

1. **Configuration** - Load plugin settings from `plugins.toml`
2. **Initialization** - Plugin registers event handlers
3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
4. **External Integration** - Plugin calls external tool APIs
5. **Error Handling** - Failed plugins don't affect core service

### Plugin Configuration

Example `plugins.toml`:

```toml
[plugins.slack]
enabled = true
webhook_url = "${SLACK_WEBHOOK_URL}"

[[plugins.slack.on_gate_failed]]
channel = "#security-alerts"

[[plugins.slack.on_entity_created]]
channel = "#platform"
```

See [Plugin Guide](plugins.md) for details.

## Data Consistency

### Eventual Consistency

Charybdis uses **eventual consistency** for plugin integrations:

- Entity CRUD operations are **immediately consistent**
- Plugin synchronization is **eventually consistent**
- Events are processed **asynchronously**

```mermaid
graph LR
    A[CreateEntity] -->|Immediate| B[Entity Stored]
    B -->|Async| C[Event Emitted]
    C -->|Async| D[Plugin Processing]
    D -->|Eventual| E[External Tool Synced]
    
    style B fill:#00C851
    style E fill:#ffbb33
```

### Guarantees

| Operation | Consistency | Guarantee |
|-----------|-------------|-----------|
| Entity CRUD | Strong | Immediate |
| Entity queries | Strong | Read-your-writes |
| Event delivery | At-least-once | May retry |
| Plugin sync | Eventual | Best-effort |

## Performance Characteristics

### Scalability

- **Entities**: Tested with 100,000+ entities
- **Throughput**: 1,000+ requests/second
- **Latency**: Sub-millisecond average
- **Concurrency**: Tokio async runtime

### Resource Usage

Typical resource consumption:

| Component | CPU | Memory | Storage |
|-----------|-----|--------|---------|
| Charybdis | < 5% | ~50 MB | Minimal |
| PostgreSQL | ~10% | ~256 MB | Depends on entity count |

### Optimization Tips

1. **Index annotations** used for frequent queries
2. **Use connection pooling** (built-in)
3. **Enable query caching** in PostgreSQL
4. **Monitor event bus** queue depth

## Backstage Migration

Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.

### How It Works

```mermaid
graph LR
    A[Charybdis Entity] --> B[YAML Adapter]
    B --> C[Backstage YAML Format]
    C --> D[Backstage Catalog]

    style B fill:#4A90E2,stroke:#2E5C8A
```

Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:

```yaml
catalog:
  locations:
    - type: url
      target: http://charybdis:8080/yaml/locations
```

### Recommended Migration Path

1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
4. **Optionally retire** Backstage when Charybdis covers your catalog needs

## Best Practices

### Entity Design

✅ **DO**:
- Use descriptive names
- Group by namespace
- Add relevant labels
- Include documentation links
- Set appropriate owners

❌ **DON'T**:
- Store sensitive data in metadata
- Use very long descriptions
- Create deeply nested hierarchies
- Duplicate data across entities

### Naming Conventions

```
<entity-type>-<purpose>-<environment>

Examples:
- payment-api-prod
- user-service-staging
- auth-library
- e-commerce-system
```

### Metadata Organization

```json
{
  "labels": {
    "team": "payments",           // Ownership
    "tier": "critical",            // Importance
    "environment": "production"    // Deployment
  },
  "tags": ["pci-compliant", "public-api"],
  "links": [
    { "url": "...", "title": "Dashboard" },
    { "url": "...", "title": "Documentation" }
  ]
}
```

## Next Steps

- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
- Configure [Security](security.md) for production (mTLS + RBAC)
- Explore [Plugins](plugins.md) for external integrations
- Review the [Architecture](architecture.md) for technical deep dive

---

**Questions?** [Open an issue](../../issues).