doc: update and cleanup

2026-06-09 11:26:47 +02:00
parent ab13af48e7
commit f1b1543f29
11 changed files with 1113 additions and 3376 deletions
@@ -14,7 +14,7 @@ Contributions are welcome! Whether it's a bug report, feature idea, documentatio
 ```bash
 # Clone the repo
-git clone https://github.com/YOUR-ORG/charybdis.git
+git clone <your-charybdis-repo-url>
 cd charybdis
 # Copy config
@@ -1,41 +1,26 @@
 # Charybdis
-**The security-native platform engineering tool.**
+**A security-native platform engineering tool.**
-One platform for your software catalog, vulnerability management, and compliance posture. Event-driven. Single binary. No YAML files to maintain.
+One Rust binary for your software catalog. Event-driven, gRPC-native, no YAML to maintain. Designed to grow into a unified catalog + vulnerability management + compliance platform.
-## Why Charybdis?
+## What Works Today
-Platform engineers today run Backstage for the catalog, DefectDojo for vulnerabilities, Dependency-Track for SBOMs, a license scanner, and a compliance spreadsheet. Five tools, five sources of truth, none of them talking to each other.
+- **Software catalog** — gRPC API for entity CRUD (Component, System, API, User, Group, Domain, Resource, Finding). Real-time event bus on every change.
 - **Vulnerability ingestion** — gRPC `IngestionService` ingests SARIF reports. Findings are deduplicated, reconciled against existing state (new / unchanged / resolved / reopened), and dry-runnable for MR-level diffs.
 - **Backstage YAML adapter** — HTTP endpoint exposing entities as Backstage-compatible Location YAML, for migration or coexistence.
 - **mTLS + RBAC** — Certificate-based auth with fine-grained, OU-mapped roles.
 - **OpenTelemetry** — Traces, metrics, and logs out of the box (console + OTLP).
 - **Plugin system** — Event-driven (DefectDojo) and sync (Keycloak) plugins. Compile-time integrated.
 - **PostgreSQL storage** — Zero-migration: entities stored as protobuf blobs in a fixed schema. New plugins add `oneof` variants, never columns.
-Charybdis unifies this:
+For everything else (assessment workflows, security gates, license policies, compliance frameworks, more parsers, more plugins), see [VISION.md](VISION.md) and [TODO.md](TODO.md).
 - **Software Catalog** — Event-driven, gRPC-native. Services register from CI/CD or IaC. No static YAML, no polling, always accurate.
 - **Vulnerability Management** — Ingest scan results natively (SARIF, CycloneDX, SPDX). Triage, assess, track. No external vuln tool needed.
 - **Compliance & Licenses** — Security gates, license policies, compliance framework mappings. Built-in, not bolted on.
 All of this in a **single Rust binary** that uses ~50MB of RAM.
 ## How It Works
 ```
 CI/CD Pipeline ──gRPC──> Charybdis
 Scanner results ───────>   ├── Catalogs the service (event-driven)
 IaC tools ─────────────>   ├── Ingests vulnerabilities natively
                           ├── Evaluates security gates
                           └── Fires events to plugins (Slack, Jira, ...)
 ```
 **One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.**
 ## Quick Start
 ```bash
 # 1. Start PostgreSQL
-docker run -d \
+docker run -d -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 postgres:15
  -e POSTGRES_PASSWORD=mysecretpassword \
  -p 5432:5432 \
  postgres:15
 # 2. Configure
 cp config.toml.example config.toml
@@ -62,187 +47,66 @@ grpcurl -plaintext -d '{
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```
-The entity is stored, events are fired, and plugins react automatically.
+Detailed walkthrough: [docs/getting-started.md](docs/getting-started.md).
-## Key Features
+## How It Works
 ### Software Catalog
 - **gRPC API** for programmatic entity management from CI/CD, scripts, or any tool
 - **Event-driven plugins** auto-provision external tools when entities change
 - **Zero-migration storage** — PostgreSQL with protobuf + JSONB. No schema changes, ever
 - **Rich entity model** — Component, System, API, User, Group, Domain, Resource
 - **Backstage compatible** — Built-in YAML adapter for migration or coexistence
 ### Security (Native — Phase 1)
 Security is a first-class concept in Charybdis, not a plugin. These features are under active development:
 - **Vulnerability ingestion** — Push scan results via gRPC (SARIF, CycloneDX, SPDX)
 - **Assessment workflow** — Triage, accept risk, remediate, auto-assess via rules
 - **Security gates** — Severity thresholds per product, block deployments on violations
 - **License compliance** — Track licenses, enforce policies, flag violations
 - **Compliance frameworks** — Map vulnerabilities to NIS2, SOC2, DORA requirements
 ### Platform
 - **mTLS + RBAC** — Certificate-based auth with fine-grained permissions
 - **OpenTelemetry** — Full observability (traces, metrics, logs) out of the box
 - **Single binary** — Deploy one Rust binary + PostgreSQL. That's it.
 - **Sub-millisecond latency** — Tested at 170k+ entities
 ## Architecture
 ```mermaid
 graph TB
    subgraph "Sources"
        A1[CI/CD Pipelines]
        A2[IaC Tools]
        A3[Security Scanners]
    end
    subgraph "Charybdis Core"
        B1[gRPC API]
        B2[Software Catalog]
        B3[Vuln Management]
        B4[Security Gates]
        B5[Event Bus]
    end
    subgraph "Integrations - Plugins"
        C1[Slack / Teams]
        C2[Jira / GitHub Issues]
        C3[Custom Plugins]
    end
    subgraph "Frontend"
        D2[Backstage - optional]
    end
    A1 -->|gRPC| B1
    A2 -->|gRPC| B1
    A3 -->|Scan Results| B1
    B1 --> B2
    B1 --> B3
    B3 --> B4
    B2 --> B5
    B3 --> B5
    B5 --> C1
    B5 --> C2
    B5 --> C3
    B2 -.->|YAML Adapter| D2
 ```
 ## Entity Model
 | Kind | Description | Example |
 |------|-------------|---------|
 | **Service** | Individual microservices or applications | `payment-api`, `auth-service` |
 | **Component** | Reusable libraries, SDKs, modules | `auth-sdk`, `logging-lib` |
 | **System** | Collections of components working together | `payment-system` |
 | **API** | Interfaces exposed by components | `payments-rest-api` |
 | **User** | Individual people | `john.doe` |
 | **Group** | Teams and organizational units | `team-payments` |
 | **Domain** | Business domains | `payments`, `shipping` |
 | **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
 Each entity will carry its security posture natively once Phase 1 is complete: vulnerabilities, license status, compliance state, and assessment history.
 ## Plugin System
 Core features (catalog, vulns, compliance) are **native**. Plugins handle **integrations** with external systems:
 ```
-EntityCreated ──> Event Bus ──> DefectDojo Plugin ──> Creates product
+CI/CD Pipeline ──gRPC──> Charybdis
-                            ──> Slack Plugin ──> Notifies channel (planned)
+Scanner output ────────>   ├── Catalogs the entity
-                            ──> Custom Plugin ──> Your logic
+                           ├── Reconciles vulnerabilities (SARIF)
                           └── Fires events to plugins (DefectDojo, ...)
 ```
-Two plugin types:
+See [docs/architecture.md](docs/architecture.md) for the full design (protobuf schema, storage model, event bus, plugin lifecycle).
 - **Event-Driven** — React to entity/vulnerability events in real-time
 - **Sync** — Pull data from external sources on a schedule (e.g., sync users from Keycloak)
-**Current plugins:**
+## Entity Kinds
 - DefectDojo — Auto-create products, engagements, and assign owners (Done)
 - Keycloak — Sync users and groups with annotations (Done)
 - Dependency-Track — Scaffolded
 - Jira, Slack, GitHub — Planned
-Build your own with the `EventDrivenPlugin` or `SyncPlugin` traits.
+| Kind | Purpose |
 |---|---|
 | **Component** | Services, libraries, applications |
 | **System** | Collections of components |
 | **API** | Interfaces exposed by components |
 | **User**, **Group** | People and teams |
 | **Domain** | Business domains |
 | **Resource** | Infrastructure resources |
 | **Finding** | Security findings ingested from scanners |
-## Backstage Compatibility
+Conceptual details: [docs/core-concepts.md](docs/core-concepts.md).
-Already using Backstage? Charybdis works as a **drop-in dynamic backend**. Point Backstage at Charybdis's YAML adapter and stop maintaining `catalog-info.yaml` files:
+## Plugins
-```yaml
+Two plugin types, both compile-time integrated:
 # backstage app-config.yaml
 catalog:
  locations:
    - type: url
      target: http://charybdis:8080/yaml/locations
      rules:
        - allow: [Component, System, Service, API, User, Group]
 ```
-Or use Charybdis standalone via its gRPC API — no Backstage needed.
+- **Event-driven** — React to entity lifecycle events. *Shipped: DefectDojo (auto-create products, engagements, member assignment).*
 - **Sync** — Pull data from external systems on a schedule. *Shipped: Keycloak (users + groups).*
-## vs. Alternatives
+Plugin model, configuration, and writing your own: [plugins/README.md](plugins/README.md).
 | | Charybdis | Backstage | Port / Cortex / OpsLevel |
 |---|---|---|---|
 | **Type** | Open source | Open source | Commercial SaaS |
 | **Catalog** | Event-driven, real-time | Static YAML, polling | Varies |
 | **Security** | Native (first-class) | Plugins (fragmented) | Limited / add-on |
 | **Deployment** | Single binary + PG | Node.js cluster + PG + plugins | Hosted |
 | **Performance** | ~50MB RAM, sub-ms | ~1GB+ RAM | N/A |
 | **Compliance** | Native frameworks | Manual | Some |
 | **Cost** | Free | Free (+ operational cost) | $$$$ |
 ## Project Status
 | Module | Status | Phase |
 |--------|--------|-------|
 | Core gRPC API | Done | 0 |
 | PostgreSQL storage | Done | 0 |
 | Event bus system | Done | 0 |
 | Plugin framework | Done | 0 |
 | mTLS + RBAC | Done | 0 |
 | OpenTelemetry | Done | 0 |
 | Backstage YAML adapter | Done | 0 |
 | DefectDojo plugin | Done | 0 |
 | Keycloak plugin | Done | 0 |
 | Vulnerability ingestion | Planned | 1 |
 | SARIF / CycloneDX / SPDX parsers | Planned | 1 |
 | Security gates & rules engine | Planned | 1 |
 | Compliance frameworks | Planned | 3 |
 See [VISION.md](VISION.md) for the full roadmap.
 ## Documentation
- [Vision & Roadmap](VISION.md) — Where Charybdis is going
+- [Getting Started](docs/getting-started.md) — installation and first entity
- [Getting Started](docs/getting-started.md) — Installation and first steps
+- [Core Concepts](docs/core-concepts.md) — entities, events, annotations
- [Core Concepts](docs/core-concepts.md) — Entities, events, and architecture
+- [Architecture](docs/architecture.md) — protobuf schema, storage, event bus
 - [Security](docs/security.md) — mTLS and RBAC configuration
- [Architecture](docs/architecture.md) — Deep dive into design decisions
+- [Plugins](plugins/README.md) — DefectDojo, Keycloak, building your own
- [Plugin Guide](plugins/README.md) — Building and using plugins
+- [Demo](deploy/DEMO.md) — full stack demo with DefectDojo
- [Demo](deploy/DEMO.md) — Full stack demo with DefectDojo
+- [Vision & Roadmap](VISION.md) — where Charybdis is going
 ## Technology Stack
 | | |
 |---|---|
-| **Language** | Rust |
+| Language | Rust |
-| **API** | gRPC + Protocol Buffers |
+| API | gRPC + Protocol Buffers |
-| **Database** | PostgreSQL 14+ (protobuf + JSONB, zero-migration) |
+| Database | PostgreSQL 14+ (protobuf + JSONB, zero-migration) |
-| **Frontend** | Backstage YAML adapter (compatible) |
+| TLS | rustls (no OpenSSL dependency) |
-| **Security** | mTLS (rustls) + RBAC |
+| Observability | OpenTelemetry (traces, metrics, logs) |
 | **Observability** | OpenTelemetry (traces, metrics, logs) |
 ## Contributing
-Contributions are welcome! Whether it's a new plugin, a security parser, or documentation improvements — we'd love your help.
+See [CONTRIBUTING.md](CONTRIBUTING.md).
 ## License
-[Apache-2.0](LICENSE)
+[Apache-2.0](LICENSE.md)
 ---
 **One platform. Catalog. Security. Compliance. Built in Rust.**
@@ -1,7 +1,7 @@
 # Charybdis - TODO List
-**Last Updated**: 2026-05-06
+**Last Updated**: 2026-06-09
-**Status**: Phase 0 done. Phase 1 (Security Core) in progress — assessment & gates remaining.
+**Status**: Phase 0 done. Phase 1 (Security Core) in progress — ingestion + reconciliation shipped; assessment & gates remaining.
 > Aligned with [VISION.md](VISION.md) roadmap.
@@ -12,12 +12,21 @@
 > **Goal**: Native vulnerability management and scan ingestion. Replace DefectDojo for finding lifecycle management.
 > **Architecture**: Core features (reconciliation, dedup, dry-run) in `src/`. Parsers extensible via `ScannerParser` trait. Plugins contribute parsers via `contributed_parsers()`.
 ### Done
 - [x] `Finding` entity kind (proto + storage)
 - [x] `IngestionService` gRPC endpoint (`ImportScan` + `DryRunScan`)
 - [x] `ScannerParser` trait + `ParserRegistry`
 - [x] SARIF parser (built-in)
 - [x] Fingerprint-based deduplication (scanner-provided when present, sha256 fallback)
 - [x] Reconciliation engine (new / unchanged / resolved / reopened buckets, scoped to `(component, lifecycle)`)
 ### Remaining
 - [ ] CycloneDX VEX parser (vulnerability data from SBOMs)
 - [ ] `Plugin::contributed_parsers()` default impl on base trait
- [ ] Publish events: FindingCreated, FindingResolved, FindingReopened (for downstream plugins)
+- [ ] Publish events: `FindingCreated`, `FindingResolved`, `FindingReopened` (for downstream plugins)
- [ ] Assessment workflow (triage, accept risk, remediate)
+- [ ] Assessment workflow (triage, accept risk, remediate — the `ACCEPTED` / `FALSE_POSITIVE` state values exist but no API sets them)
 - [ ] Rules engine for auto-assessment
 - [ ] Security gates (severity thresholds per product)
 - [ ] License tracking and policy engine
@@ -288,9 +288,8 @@ Backstage will discover all entities from Charybdis automatically. See the `dock
 ## Support
- GitHub Issues: [github.com/charybdis-catalog/charybdis/issues](https://github.com/charybdis-catalog/charybdis/issues)
+- Documentation: [../docs/](../docs/)
- Documentation: [docs/](docs/)
+- Open an issue on the project's Gitea/GitHub repository
 - Discord: [Join our community](https://discord.gg/...)
 ---
@@ -1,126 +1,54 @@
 # Core Concepts
-Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.
+Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes.
-This page explains the key concepts you need to understand.
+For a higher-level overview see the [README](../README.md). For deep technical details see [architecture.md](architecture.md).
 ## Architecture Overview
 ```mermaid
 graph TB
    subgraph "Sources"
        A1[CI/CD Pipelines]
        A2[Security Scanners]
        A3[IaC Tools]
    end
    subgraph "Charybdis Core"
        B1[gRPC API]
        B3[Authentication & Authorization]
        C1[Entity Service]
        C4[Vulnerability Engine]
        C5[Security Gates & Rules]
        C2[Event Bus]
    end
    subgraph "Integrations - Plugins"
        D1[Slack / Teams]
        D2[Jira / GitHub]
        D3[Custom Plugins]
    end
    subgraph "Data Layer"
        E1[(PostgreSQL)]
    end
    A1 -.gRPC.-> B1
    A2 -.Scan Results.-> B1
    A3 -.gRPC.-> B1
    B1 --> B3
    B3 --> C1
    B3 --> C4
    C4 --> C5
    C1 --> E1
    C4 --> E1
    C1 --> C2
    C4 --> C2
    C2 -.Events.-> D1
    C2 -.Events.-> D2
    C2 -.Events.-> D3
    style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
    style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
    style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
 ```
 ## Entities
-An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.
+An **entity** is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding.
-### Entity Structure
+### Entity structure
-Every entity has three main parts:
+Every entity has the same shape:
-```mermaid
+- `id` — UUID assigned by the server
-classDiagram
+- `kind` — one of the supported kinds (see below)
-    class Entity {
+- `metadata` — identifying data (name, description, labels, tags, links)
-        +string id
+- `spec` — kind-specific configuration (type, lifecycle, owner, dependencies, ...)
-        +string kind
+- `annotations` — arbitrary `string → string` map, typically used by plugins to store external IDs
-        +metadata
+- `created_at`, `updated_at` — server-managed timestamps
        +spec
        +annotations
        +timestamps
    }
-    class Metadata {
+`metadata` and `spec` are protobuf `oneof` fields whose variant matches `kind`. So a `Component` entity carries `component_metadata` + `component_spec`, a `User` carries `user_metadata` + `user_spec`, and so on. The on-wire JSON uses the variant name directly.
        +string name
        +string namespace
        +string description
        +labels
        +links
        +tags
    }
-    class Spec {
+### Entity kinds
        +string type
        +string lifecycle
        +string owner
        +dependencies
    }
-    Entity --> Metadata
+| Kind | Purpose |
-    Entity --> Spec
+|---|---|
-```
+| `Component` | Services, applications, libraries, websites |
 | `System` | Collections of components working together |
 | `API` | Interfaces exposed by components |
 | `User` | Individual people |
 | `Group` | Teams and organizational units |
 | `Domain` | Business domains |
 | `Resource` | Infrastructure resources (databases, caches, queues) |
 | `Finding` | Security findings ingested from scanners (see [Findings](#findings)) |
-### Entity Kinds
+Per-kind protobuf definitions: `proto/core/*.proto`.
-Charybdis supports all standard Backstage entity kinds:
+### Example: Component
 | Kind | Description | Example |
 |------|-------------|---------|
 | **Service** | Individual microservices or applications | `payment-api` |
 | **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
 | **System** | Collections of services working together | `e-commerce-platform` |
 | **API** | Interfaces exposed by components | `payments-rest-api` |
 | **User** | Individual people | `john.doe` |
 | **Group** | Teams and organizational units | `team-payments` |
 | **Domain** | Business domains | `payments`, `shipping` |
 | **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
 #### Example: Service
 ```json
 {
-  "kind": "Service",
+  "kind": "Component",
-  "service_metadata": {
+  "component_metadata": {
    "name": "payment-api",
    "namespace": "production",
    "description": "Payment processing service",
-    "labels": { "team": "payments" }
+    "labels": { "team": "payments", "tier": "critical" },
    "tags": ["api", "pci-dss"]
  },
-  "service_spec": {
+  "component_spec": {
    "type": "service",
    "lifecycle": "production",
    "owner": "team-payments",
@@ -129,500 +57,136 @@ Charybdis supports all standard Backstage entity kinds:
 }
 ```
-#### Example: Component
+### Common metadata fields
-```json
+| Field | Type | Notes |
-{
+|---|---|---|
-  "kind": "Component",
+| `name` | string | Required. Unique with `kind` (composite index `idx_entities_kind_name`). |
-  "component_metadata": {
+| `namespace` | string | Logical grouping, e.g. `production`. |
-    "name": "auth-sdk",
+| `description` | string | Human-readable. |
-    "namespace": "shared"
+| `labels` | `map<string,string>` | Categorization. |
-  },
+| `tags` | `repeated string` | Free-form classification. |
-  "component_spec": {
+| `links` | `repeated Link` | External URLs (dashboards, docs). |
    "type": "library",
    "lifecycle": "production",
    "owner": "platform-team"
  }
 }
 ```
-#### Example: System
+## Annotations
-```json
+Annotations are an open `map<string, string>` on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB.
 {
  "kind": "System",
  "system_metadata": {
    "name": "e-commerce-platform",
    "namespace": "production",
    "description": "Complete e-commerce system"
  },
  "system_spec": {
    "owner": "platform-team",
    "domain": "retail"
  }
 }
 ```
 ### Metadata Fields
 | Field | Type | Description | Required |
 |-------|------|-------------|----------|
 | `name` | string | Entity name (unique within namespace) | ✅ |
 | `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
 | `description` | string | Human-readable description | ❌ |
 | `labels` | map | Key-value pairs for categorization | ❌ |
 | `tags` | array | Search tags | ❌ |
 | `links` | array | External URLs (dashboards, docs, etc.) | ❌ |
 ### Annotations
 Annotations store integration-specific metadata:
 ```json
 {
  "annotations": {
    "github.com/repo-slug": "myorg/payment-service",
    "defectdojo.com/product-id": "123",
-    "dependencytrack.com/project-uuid": "550e8400...",
+    "keycloak.com/email": "alice@example.com",
-    "pagerduty.com/service-id": "PXYZ123",
+    "pagerduty.com/service-id": "PXYZ123"
    "grafana.com/dashboard-url": "https://..."
  }
 }
 ```
-**Best Practices**:
+Convention: reverse-DNS keys (`tool.com/key`). Values are always strings.
- Use domain-style keys (`tool.com/key`)
+
- Store tool-specific IDs
+`EntityRepository::update_annotations()` performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers.
 - Keep values as strings
 - Use for integration metadata only
 ## Events
-Charybdis uses an **event-driven architecture** to trigger actions when entities change.
+Charybdis emits an `EntityEvent` on every CRUD operation, delivered to subscribers via the in-memory event bus.
-### Event Flow
+```
-
+Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers
 ```mermaid
 sequenceDiagram
    participant Client
    participant API as Entity Service
    participant Bus as Event Bus
    participant Plugin1 as DefectDojo Plugin
    participant Plugin2 as Dependency-Track
    Client->>API: CreateEntity(service)
    API->>API: Store entity
    API->>Bus: Emit EntityCreated event
    Bus->>Plugin1: Handle event
    Bus->>Plugin2: Handle event
    Plugin1-->>Plugin1: Create DefectDojo product
    Plugin2-->>Plugin2: Create DT project
    API-->>Client: Return created entity
    Note over Bus,Plugin2: Asynchronous processing
 ```
-### Event Types
+### Event types
-| Event | Trigger | Plugins Receive |
+| Type | When |
-|-------|---------|----------------|
+|---|---|
-| `EntityCreated` | New entity created | Full entity data |
+| `Created` | Successful `CreateEntity` |
-| `EntityUpdated` | Entity modified | Updated entity + changes |
+| `Updated` | Successful `UpdateEntity` (full or partial) |
-| `EntityDeleted` | Entity removed | Entity ID + metadata |
+| `Deleted` | Successful `DeleteEntity` |
-### Event Structure
+### Event payload
 ```rust
-pub enum EntityEvent {
+pub struct EntityEvent {
-    Created {
+    pub event_id: Uuid,
-        entity: Entity,
+    pub entity_id: Uuid,
-        timestamp: DateTime<Utc>,
+    pub event_type: EntityEventType,   // Created | Updated | Deleted
-    },
+    pub timestamp: DateTime<Utc>,
-    Updated {
+    pub metadata: HashMap<String, String>,
-        entity: Entity,
+    pub entity_data: Option<Arc<Entity>>,  // full entity for Created/Updated
        previous: Entity,
        timestamp: DateTime<Utc>,
    },
    Deleted {
        id: String,
        metadata: Metadata,
        timestamp: DateTime<Utc>,
    },
 }
 ```
-## Vulnerabilities & Security (Phase 1 — Planned)
+### Delivery semantics
-> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.
+- **Asynchronous** — `CreateEntity` returns to the client before plugin handlers complete.
 - **At-least-once intent**, but the default `MemoryEventBus` is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in [VISION.md](../VISION.md).
 - **Per-handler timeout** — 30 s. Slow plugins don't block the bus.
 - **Panic isolation** — a panicking handler doesn't take down others.
-Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.
+Plugin lifecycle and how to write a handler: [plugins/README.md](../plugins/README.md).
-### Vulnerability Lifecycle
+## Findings
-```mermaid
+`Finding` is a first-class entity kind for security findings produced by scanners. Findings are created by the `IngestionService`, not directly by clients.
 sequenceDiagram
    participant Scanner
    participant API as Charybdis API
    participant Rules as Rules Engine
    participant Gate as Security Gate
    participant Bus as Event Bus
    participant Plugin as Slack / Jira
-    Scanner->>API: IngestScan(SARIF report)
+Each finding is scoped to a `(component_ref, lifecycle)` pair — so production and staging environments track findings independently without duplicating the underlying `Component`.
    API->>API: Parse & create Vulnerability entities
    API->>API: Link vulnerabilities to Component
    API->>Rules: Evaluate auto-assessment rules
    Rules-->>API: Auto-assess (e.g., accept known low-risk)
    API->>Gate: Check security gate thresholds
    alt Gate Passed
        Gate-->>API: OK
    else Gate Failed
        Gate->>Bus: GateFailed event
        Bus->>Plugin: Alert #security channel
    end
    API->>Bus: VulnerabilitiesIngested event
    Bus->>Plugin: Notify / create tickets
 ```
-### Scan Ingestion
+The ingestion flow:
-Charybdis ingests scan results natively. You don't need an external vulnerability management tool.
+1. Client calls `IngestionService.ImportScan(component_ref, lifecycle, format, data)`.
 2. The configured parser (currently SARIF) normalizes scanner output into `NormalizedFinding` records.
 3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`) and diffs incoming vs existing findings for that scope.
 4. The result is one of:
   - **New** — fingerprint not seen before → create a `Finding` entity.
   - **Unchanged** — already active → bump `last_seen`.
   - **Resolved** — previously active, not in this scan → mark `Resolved`.
   - **Reopened** — previously resolved or false-positive, detected again → mark `Reopened`.
 5. `DryRunScan` runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments.
-| Format | Coverage | Use Case |
+Finding state values (see `proto/core/finding.proto`):
 |--------|----------|----------|
 | **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
 | **CycloneDX** | SBOMs + vulnerability data | SCA, license |
 | **SPDX** | License and package data | License compliance |
-### Assessments
+| State | Meaning |
 |---|---|
 | `ACTIVE` | Currently detected |
 | `RESOLVED` | No longer detected (auto-set on re-import) |
 | `ACCEPTED` | Risk accepted by human decision *(state value defined; assessment workflow is planned)* |
 | `FALSE_POSITIVE` | Marked as not a real issue *(state value defined; assessment workflow is planned)* |
 | `REOPENED` | Was resolved, detected again |
-Each vulnerability can be assessed:
+Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).
 | Status | Meaning |
 |--------|---------|
 | **Open** | New, unreviewed vulnerability |
 | **In Triage** | Under review by security team |
 | **Accepted** | Risk accepted with justification |
 | **Remediated** | Fixed, pending verification |
 | **False Positive** | Not a real vulnerability |
 | **Auto-Assessed** | Automatically assessed by rules engine |
 ### Security Gates
 Security gates define thresholds per product:
 ```
 payment-api:
  critical: 0     # No critical vulns allowed
  high: 5         # Up to 5 high
  medium: 20      # Up to 20 medium
 ```
 When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).
 ### Rules Engine
 Rules auto-assess vulnerabilities based on patterns:
 - Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
 - Scanner source (e.g., "all informational from ZAP → auto-accept")
 - Known patterns (e.g., "CVE-XXXX already accepted org-wide")
 ### License Compliance (Phase 2)
 Track licenses across your dependency tree:
 - Ingest license data from CycloneDX/SPDX
 - Define license policies (allowed, restricted, banned)
 - Flag violations per component
 - Compliance reporting
 ## Storage Model
-Charybdis uses PostgreSQL with JSONB for schema-less storage.
+Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a `oneof` variant in protobuf, not a column migration.
 ### Database Schema
 ```sql
 CREATE TABLE entities (
    id          UUID PRIMARY KEY,
-    kind TEXT NOT NULL,
+    kind        VARCHAR NOT NULL,
-    entity_data BYTEA NOT NULL,         -- Protobuf binary
+    name        VARCHAR,
-    annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
+    entity_data BYTEA NOT NULL,            -- protobuf-encoded Entity
-    created_at TIMESTAMP NOT NULL,
+    annotations JSONB NOT NULL DEFAULT '{}',
-    updated_at TIMESTAMP NOT NULL
+    created_at  TIMESTAMPTZ NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL
 );
 CREATE INDEX idx_entities_kind            ON entities(kind);
-CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
+CREATE INDEX idx_entities_kind_name       ON entities(kind, name);
 CREATE INDEX idx_entities_annotations     ON entities USING GIN (annotations);
 CREATE INDEX idx_entities_created_at_id   ON entities(created_at DESC, id DESC);
 ```
-### Why Protobuf + JSONB?
+Why this design:
 - **No migrations** — new plugins extend the protobuf `oneof` variants; database schema is unchanged.
 - **Compact storage** — protobuf binary is smaller than JSON.
 - **Forward/backward compatible** — proto field numbers protect against version drift.
 - **Fast annotation queries** — JSONB + GIN index supports `annotations->>'key' = 'value'` lookups in O(log n).
-**Advantages**:
+Detailed schema reasoning, scaling notes, and field-mask paths: [architecture.md](architecture.md).
 - ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
 - ✅ Protobuf binary storage for compact, versioned entity data
 - ✅ JSONB annotations for fast querying with GIN indexes
 - ✅ Forward/backward compatibility built-in
-**Example Query**:
+## See Also
 ```sql
 -- Find entities with specific annotation
 SELECT * FROM entities 
 WHERE annotations->>'defectdojo.com/product-id' = '42';
 ```
-## Security Model
+- [Architecture](architecture.md) — protobuf schema, scaling, design decisions
-
+- [Security](security.md) — mTLS + RBAC configuration
-Charybdis implements defense-in-depth security.
+- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
-
+- [Vision & Roadmap](../VISION.md) — planned features
 ### Authentication: mTLS
 ```mermaid
 sequenceDiagram
    participant Client
    participant TLS as TLS Layer
    participant Auth as Auth Interceptor
    participant Service as Entity Service
    Client->>TLS: Connect with client certificate
    TLS->>TLS: Validate certificate
    TLS->>Auth: Extract certificate
    Auth->>Auth: Parse identity (CN, OU, O)
    Auth->>Auth: Map to role
    Client->>Auth: Request (with identity)
    Auth->>Auth: Check permissions
    alt Authorized
        Auth->>Service: Forward request
        Service-->>Client: Response
    else Denied
        Auth-->>Client: PermissionDenied error
    end
 ```
 ### Authorization: RBAC
 **Role Mapping**:
 ```
 Certificate (CN, OU, O) → RBAC Role → Permissions
 ```
 **Default Roles**:
 | Role | Certificate OU | Permissions |
 |------|---------------|-------------|
 | `platform` | `platform-team` | Full access (CRUD + list) |
 | `automation` | `automation` | Create, read, update, list |
 | `plugin` | `plugins` | Read, list only |
 ### Permissions
 | Permission | Operations | Required For |
 |------------|-----------|--------------|
 | `entity:create` | Create new entities | CreateEntity |
 | `entity:read` | Get entity by ID | GetEntity |
 | `entity:update` | Modify entities | UpdateEntity |
 | `entity:delete` | Remove entities | DeleteEntity |
 | `entity:list` | List all entities | ListEntities |
 ## Plugin System
 Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.
 ### Plugin Architecture
 ```mermaid
 graph LR
    A[Entity/Vuln Event] --> B[Event Bus]
    B --> C{Plugin Manager}
    C --> D[Slack Plugin]
    C --> E[Jira Plugin]
    C --> F[Custom Plugin]
    D --> G[Slack API]
    E --> H[Jira API]
    F --> I[Your Tool API]
    style C fill:#4A90E2,stroke:#2E5C8A
 ```
 ### What's Native vs. Plugin
 | Native (core) | Status | Plugin (integration) | Status |
 |---|---|---|---|
 | Software catalog | Done | DefectDojo sync | Done |
 | Vulnerability management | Phase 1 | Keycloak user sync | Done |
 | Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
 | Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
 | License compliance | Phase 2 | Custom integrations | Framework ready |
 | TechDocs | Future | | |
 ### Plugin Lifecycle
 1. **Configuration** - Load plugin settings from `plugins.toml`
 2. **Initialization** - Plugin registers event handlers
 3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
 4. **External Integration** - Plugin calls external tool APIs
 5. **Error Handling** - Failed plugins don't affect core service
 ### Plugin Configuration
 Example `plugins.toml`:
 ```toml
 [plugins.slack]
 enabled = true
 webhook_url = "${SLACK_WEBHOOK_URL}"
 [[plugins.slack.on_gate_failed]]
 channel = "#security-alerts"
 [[plugins.slack.on_entity_created]]
 channel = "#platform"
 ```
 See [Plugin Guide](plugins.md) for details.
 ## Data Consistency
 ### Eventual Consistency
 Charybdis uses **eventual consistency** for plugin integrations:
 - Entity CRUD operations are **immediately consistent**
 - Plugin synchronization is **eventually consistent**
 - Events are processed **asynchronously**
 ```mermaid
 graph LR
    A[CreateEntity] -->|Immediate| B[Entity Stored]
    B -->|Async| C[Event Emitted]
    C -->|Async| D[Plugin Processing]
    D -->|Eventual| E[External Tool Synced]
    style B fill:#00C851
    style E fill:#ffbb33
 ```
 ### Guarantees
 | Operation | Consistency | Guarantee |
 |-----------|-------------|-----------|
 | Entity CRUD | Strong | Immediate |
 | Entity queries | Strong | Read-your-writes |
 | Event delivery | At-least-once | May retry |
 | Plugin sync | Eventual | Best-effort |
 ## Performance Characteristics
 ### Scalability
 - **Entities**: Tested with 100,000+ entities
 - **Throughput**: 1,000+ requests/second
 - **Latency**: Sub-millisecond average
 - **Concurrency**: Tokio async runtime
 ### Resource Usage
 Typical resource consumption:
 | Component | CPU | Memory | Storage |
 |-----------|-----|--------|---------|
 | Charybdis | < 5% | ~50 MB | Minimal |
 | PostgreSQL | ~10% | ~256 MB | Depends on entity count |
 ### Optimization Tips
 1. **Index annotations** used for frequent queries
 2. **Use connection pooling** (built-in)
 3. **Enable query caching** in PostgreSQL
 4. **Monitor event bus** queue depth
 ## Backstage Migration
 Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.
 ### How It Works
 ```mermaid
 graph LR
    A[Charybdis Entity] --> B[YAML Adapter]
    B --> C[Backstage YAML Format]
    C --> D[Backstage Catalog]
    style B fill:#4A90E2,stroke:#2E5C8A
 ```
 Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:
 ```yaml
 catalog:
  locations:
    - type: url
      target: http://charybdis:8080/yaml/locations
 ```
 ### Recommended Migration Path
 1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
 2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
 3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
 4. **Optionally retire** Backstage when Charybdis covers your catalog needs
 ## Best Practices
 ### Entity Design
 ✅ **DO**:
 - Use descriptive names
 - Group by namespace
 - Add relevant labels
 - Include documentation links
 - Set appropriate owners
 ❌ **DON'T**:
 - Store sensitive data in metadata
 - Use very long descriptions
 - Create deeply nested hierarchies
 - Duplicate data across entities
 ### Naming Conventions
 ```
 <entity-type>-<purpose>-<environment>
 Examples:
 - payment-api-prod
 - user-service-staging
 - auth-library
 - e-commerce-system
 ```
 ### Metadata Organization
 ```json
 {
  "labels": {
    "team": "payments",           // Ownership
    "tier": "critical",            // Importance
    "environment": "production"    // Deployment
  },
  "tags": ["pci-compliant", "public-api"],
  "links": [
    { "url": "...", "title": "Dashboard" },
    { "url": "...", "title": "Documentation" }
  ]
 }
 ```
 ## Next Steps
 - Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
 - Configure [Security](security.md) for production (mTLS + RBAC)
 - Explore [Plugins](plugins.md) for external integrations
 - Review the [Architecture](architecture.md) for technical deep dive
 ---
 **Questions?** [Open an issue](../../issues).
@@ -1,39 +1,26 @@
 # Getting Started with Charybdis
-This guide will get you from zero to a running Charybdis instance in minutes. By the end, you'll have a working software catalog that can register services via gRPC, ingest scan results, and auto-trigger integrations via event-driven plugins.
+This guide gets you from zero to a running Charybdis instance: a software catalog with a gRPC API, optional Backstage YAML adapter, and plugin integrations on entity events.
 ## Prerequisites
- **Rust** - 1.70 or later ([install](https://rustup.rs/))
+- **Rust** 1.70+ ([install](https://rustup.rs/))
- **PostgreSQL** - 14 or later
+- **PostgreSQL** 14+
- **grpcurl** - For testing (optional, [install](https://github.com/fullstorydev/grpcurl))
+- **protoc** + `libprotobuf-dev` (for the build script)
 - **grpcurl** for testing (optional, [install](https://github.com/fullstorydev/grpcurl))
-## Installation
+## Install
 ### Option 1: From Source
 ```bash
-# Clone the repository
+git clone <your-charybdis-repo-url>
 git clone https://github.com/charybdis-catalog/charybdis.git
 cd charybdis
 # Build
 cargo build --release
-
+# Binary at target/release/charybdis-server
 # The binary will be at target/release/charybdis
 ```
 ### Option 2: Docker (Coming Soon)
 ```bash
 docker pull charybdis/charybdis:latest
 ```
 ## Quick Start
-### Step 1: Start PostgreSQL
+### 1. Start PostgreSQL
 Using Docker:
 ```bash
 docker run -d \
@@ -43,54 +30,22 @@ docker run -d \
  postgres:15
 ```
-Or use an existing PostgreSQL instance.
+### 2. Configure
 ### Step 2: Configure Charybdis
 **Recommended: Use config.toml**
 Copy the example configuration:
 ```bash
 cp config.toml.example config.toml
 ```
 Edit `config.toml` and set your database URL:
 ```toml
 [database]
 url = "${DATABASE_URL}"
 ```
 Set the environment variable:
 ```bash
 export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
 ```
-**Alternative: Environment Variables Only (Legacy)**
+`config.toml` reads `${DATABASE_URL}` from the environment. See [Configuration](#configuration) below for all options.
-If you prefer environment variables:
+### 3. Run
 ```bash
 # Database
 export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
 # Disable security for quick start
 export SECURITY_MTLS_ENABLED=false
 export SECURITY_RBAC_ENABLED=false
 # Logging
 export RUST_LOG=info,charybdis=debug
 ```
 ### Step 3: Run Charybdis
 ```bash
 cargo run
 ```
-You should see:
+Expected startup logs:
 ```
 INFO  charybdis: Database ready
@@ -98,40 +53,34 @@ INFO  charybdis: Event bus started successfully
 INFO  charybdis: EntityService server listening on [::1]:50051
 ```
-### Step 4: Verify It's Working
+### 4. Verify
 Test with grpcurl:
 ```bash
 # List available services
 grpcurl -plaintext localhost:50051 list
 # Output:
 # charybdis.entities.EntityService
 # charybdis.ingestion.IngestionService
 # grpc.reflection.v1.ServerReflection
 ```
-Congratulations! Charybdis is running! 🎉
+## Create Your First Entity
 ## Creating Your First Entity
 ### Using grpcurl
 Create a service entity:
 ```bash
-grpcurl -plaintext \
+grpcurl -plaintext -d '{
  -d '{
  "entity": {
-      "kind": "Service",
+    "kind": "Component",
-      "service_metadata": {
+    "component_metadata": {
-        "name": "payment-service",
+      "name": "payment-api",
      "namespace": "production",
-        "description": "Core payment processing service"
+      "description": "Payment processing service",
      "tags": ["api", "critical"]
    },
    "component_spec": {
      "type": "service",
      "lifecycle": "production",
      "owner": "team-payments"
    }
  }
-  }' \
+}' localhost:50051 charybdis.entities.EntityService/CreateEntity
  localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```
 Response:
@@ -140,117 +89,60 @@ Response:
 {
  "entity": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
-    "kind": "Service",
+    "kind": "Component",
-    "serviceMetadata": {
+    "componentMetadata": { "...": "..." },
-      "name": "payment-service",
+    "createdAt": "2026-06-01T10:00:00Z",
-      "namespace": "production",
+    "updatedAt": "2026-06-01T10:00:00Z"
      "description": "Core payment processing service"
    },
    "createdAt": "2025-11-04T10:00:00Z",
    "updatedAt": "2025-11-04T10:00:00Z"
  }
 }
 ```
-### List All Entities
+### Retrieve and list
 ```bash
 grpcurl -plaintext -d '{}' \
  localhost:50051 charybdis.entities.EntityService/ListEntities
 ```
 ### Get Entity by ID
 ```bash
 # Get by ID
 grpcurl -plaintext \
  -d '{"id": "550e8400-e29b-41d4-a716-446655440000"}' \
  localhost:50051 charybdis.entities.EntityService/GetEntity
 # List all
 grpcurl -plaintext -d '{}' \
  localhost:50051 charybdis.entities.EntityService/ListEntities
 # Filter by kind and name
 grpcurl -plaintext \
  -d '{"kind": "Component", "name": "payment-api"}' \
  localhost:50051 charybdis.entities.EntityService/ListEntities
 ```
-## Entity Types
+## Entity Kinds
-Charybdis supports three main entity types:
+The valid `kind` values are: `Component`, `System`, `API`, `User`, `Group`, `Domain`, `Resource`, `Finding`. Each kind uses a matching `<kind>_metadata` + `<kind>_spec` payload.
-### Service
+Conceptual reference: [core-concepts.md](core-concepts.md). Per-kind protobuf definitions: `proto/core/*.proto`.
-Individual microservices or applications:
+### Component example with metadata and annotations
 ```bash
 grpcurl -plaintext -d '{
  "entity": {
    "kind": "Service",
    "service_metadata": {
      "name": "user-api",
      "namespace": "production",
      "description": "User management API"
    }
  }
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```
 ### System
 Collections of related services:
 ```bash
 grpcurl -plaintext -d '{
  "entity": {
    "kind": "System",
    "system_metadata": {
      "name": "payment-system",
      "namespace": "production",
      "description": "Complete payment processing system"
    }
  }
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```
 ### Component
 Reusable components or libraries:
 ```bash
 grpcurl -plaintext -d '{
  "entity": {
    "kind": "Component",
    "component_spec": {
      "type": "library",
      "lifecycle": "production",
      "owner": "platform-team"
    },
    "component_metadata": {
      "name": "auth-library",
      "namespace": "shared",
      "description": "Shared authentication library"
    }
  }
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```
 ## Adding Metadata
 Entities support rich metadata:
 ```bash
 grpcurl -plaintext -d '{
  "entity": {
    "kind": "Service",
    "service_metadata": {
      "name": "payment-service",
      "namespace": "production",
      "description": "Payment processing",
-      "labels": {
+      "labels": { "team": "payments", "tier": "critical" },
-        "team": "payments",
+      "links": [{
        "tier": "critical"
      },
      "links": [
        {
        "url": "https://dashboard.company.com/payments",
        "title": "Dashboard",
        "icon": "dashboard"
-        }
+      }],
-      ],
+      "tags": ["payments", "pci-compliant"]
-      "tags": ["payments", "pci-compliant", "critical"]
+    },
    "component_spec": {
      "type": "service",
      "lifecycle": "production",
      "owner": "team-payments"
    },
    "annotations": {
      "github.com/repo-slug": "myorg/payment-service",
@@ -260,105 +152,85 @@ grpcurl -plaintext -d '{
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```
-## Integrating with CI/CD
+## Registering Entities from CI
-### Example: GitHub Actions
+Charybdis is designed to be called from pipelines. With `grpcurl` available in your runner:
 Create a workflow to register services automatically:
 ```yaml
-name: Register Service
+# Example pipeline step (Gitea Actions / GitHub Actions syntax)
-on:
+- name: Register service in Charybdis
  push:
    branches: [main]
 jobs:
  register:
    runs-on: ubuntu-latest
    steps:
      - name: Register in Charybdis
  run: |
          grpcurl -plaintext \
            -d '{
              "entity": {
                "kind": "Service",
                "service_metadata": {
                  "name": "${{ github.event.repository.name }}",
                  "namespace": "production",
                  "description": "${{ github.event.repository.description }}"
                },
                "annotations": {
                  "github.com/repo-slug": "${{ github.repository }}"
                }
              }
            }' \
            your-charybdis-host:50051 \
            charybdis.entities.EntityService/CreateEntity
 ```
 ### Example: GitLab CI
 ```yaml
 register_service:
  stage: deploy
  script:
    - |
    grpcurl -plaintext \
      -d "{
        \"entity\": {
-            \"kind\": \"Service\",
+          \"kind\": \"Component\",
-            \"service_metadata\": {
+          \"component_metadata\": {
-              \"name\": \"${CI_PROJECT_NAME}\",
+            \"name\": \"$CI_PROJECT_NAME\",
-              \"namespace\": \"${CI_ENVIRONMENT_NAME}\"
+            \"namespace\": \"production\"
          },
          \"component_spec\": {
            \"type\": \"service\",
            \"lifecycle\": \"production\",
            \"owner\": \"$CI_PROJECT_NAMESPACE\"
          },
          \"annotations\": {
            \"repo-slug\": \"$CI_PROJECT_PATH\"
          }
        }
      }" \
-        your-charybdis-host:50051 \
+      charybdis.internal:50051 \
      charybdis.entities.EntityService/CreateEntity
 ```
 In production, secure the endpoint with mTLS (see [Enabling Security](#enabling-security)).
 ## Enabling Security
-For production use, enable mTLS and RBAC:
+Charybdis ships with mTLS + RBAC disabled for local exploration. For shared or production environments, enable both.
-### Step 1: Generate Certificates
+### 1. Generate dev certificates
 ```bash
-# Use the provided test script
+./deploy/scripts/generate-dev-certs.sh
 ./test-mtls-rbac.sh
 ```
-This creates:
+This writes `deploy/certs/` with:
- `certs/ca.pem` - Certificate Authority
+- `ca.pem` — CA
- `certs/server-cert.pem` / `server-key.pem` - Server certificate
+- `server-cert.pem` / `server-key.pem` — server
- `certs/admin-cert.pem` / `admin-key.pem` - Admin client certificate
+- `admin-cert.pem` / `admin-key.pem` — admin client
 - (and per-role client certs)
-### Step 2: Enable Security
+### 2. Enable in `config.toml`
-```bash
+```toml
-export SECURITY_MTLS_ENABLED=true
+[security.mtls]
-export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem
+enabled = true
-export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem
+server_cert = "./deploy/certs/server-cert.pem"
-export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem
+server_key  = "./deploy/certs/server-key.pem"
-export SECURITY_RBAC_ENABLED=true
+client_ca_cert = "./deploy/certs/ca.pem"
 [security.rbac]
 enabled = true
 ```
-### Step 3: Test with mTLS
+Restart Charybdis. RBAC defaults map cert OUs to roles (`platform-team` → full access, `automation` → CRUD without delete, `plugins` → read-only).
 ### 3. Call with mTLS
 ```bash
 grpcurl \
-  -cacert certs/ca.pem \
+  -cacert deploy/certs/ca.pem \
-  -cert certs/admin-cert.pem \
+  -cert deploy/certs/admin-cert.pem \
-  -key certs/admin-key.pem \
+  -key deploy/certs/admin-key.pem \
  -d '{}' \
  localhost:50051 charybdis.entities.EntityService/ListEntities
 ```
-See the [Security Guide](security.md) for detailed configuration.
+Full reference (custom role mappings, audit logging, reverse-proxy mode): [security.md](security.md).
 ## Backstage Migration (Optional)
-If you currently use Backstage and want to migrate gradually, the built-in YAML adapter serves entities in Backstage format:
+If you currently run Backstage, Charybdis serves entities in Backstage's Location YAML format:
 ```yaml
 # backstage app-config.yaml
@@ -367,78 +239,36 @@ catalog:
    - type: url
      target: http://your-charybdis-host:8080/yaml/locations
      rules:
-        - allow: [Component, System, Service]
+        - allow: [Component, System, API, User, Group]
 ```
-Backstage will automatically discover and import entities from Charybdis. You can run both in parallel — entities managed via gRPC are immediately visible in Backstage.
+Backstage discovers entities by polling the endpoint. Entities created via gRPC are visible on the next poll. The YAML adapter is served by an HTTP listener separate from the gRPC port (default `:8080`).
 ## Troubleshooting
 ### Port Already in Use
 ```
 Error: transport error
 ```
 **Solution**: Check if another process is using port 50051:
 ```bash
 lsof -ti:50051
 ```
 Kill the process or change the port:
 ```bash
 export GRPC_PORT=50052
 ```
 ### Database Connection Failed
 ```
 Error: password authentication failed
 ```
 **Solution**: Verify your DATABASE_URL:
 ```bash
 # Test connection
 psql "$DATABASE_URL" -c "SELECT 1;"
 ```
 ### Permission Denied (with security enabled)
 ```
 Code: PermissionDenied
 Message: Role 'X' does not have permission 'Y'
 ```
 **Solution**: Check your certificate and role mappings. See [Security Guide](security.md).
 ## Next Steps
 Now that you have Charybdis running:
 1. Learn about [Core Concepts](core-concepts.md) — entities, vulnerabilities, events
 2. Read the [Vision & Roadmap](../VISION.md) — where Charybdis is going
 3. Configure [Security](security.md) for production (mTLS + RBAC)
 4. Explore [Plugins](../plugins/README.md) for external integrations
 ## Configuration
-Charybdis supports two configuration methods:
+Charybdis loads its config from (in order):
-### 1. Configuration File (Recommended)
+1. `./config.toml`
 2. `./charybdis.toml`
 3. `/etc/charybdis/config.toml`
 4. Environment variables (fallback)
-Use `config.toml` for structured configuration:
+### `config.toml` skeleton
 ```toml
 # config.toml
 [server]
 grpc_host = "[::1]"
 grpc_port = 50051
 [server.yaml_adapter]
 enabled = true
 host = "0.0.0.0"
 port = 8080
 [database]
-url = "${DATABASE_URL}"  # Environment variable substitution
+url = "${DATABASE_URL}"
 max_connections = 10
 connection_timeout_secs = 30
 [security.mtls]
 enabled = false
@@ -450,59 +280,52 @@ enabled = false
 service_name = "charybdis"
 environment = "development"
 enable_console = true
 ```
 **Benefits:**
 - ✅ Organized by section (server, database, security, telemetry, plugins)
 - ✅ Environment variable substitution with `${VAR_NAME}`
 - ✅ Comments and documentation inline
 - ✅ Easy to version control (excluding secrets)
 - ✅ No need to export dozens of environment variables
 **Using Environment Variables in config.toml:**
 ```toml
 [database]
 url = "${DATABASE_URL}"  # Will be substituted at runtime
 [plugins.defectdojo]
-api_key = "${DEFECTDOJO_API_KEY}"  # Secrets stay in environment
+enabled = false
 # see plugins/README.md for the full plugin reference
 ```
-Then set only the secrets:
+`${VAR}` and `${VAR:-default}` substitution works in any string value — keep secrets in the environment, not in the file.
-```bash
+### Environment-variable fallback
 export DATABASE_URL="postgresql://..."
 export DEFECTDOJO_API_KEY="secret-key"
 ```
-### 2. Environment Variables (Legacy)
+If no config file is found, these env vars are read:
 If `config.toml` is not found, Charybdis falls back to environment variables:
 | Variable | Default | Description |
-|----------|---------|-------------|
+|---|---|---|
 | `DATABASE_URL` | (required) | PostgreSQL connection string |
-| `GRPC_HOST` | `[::1]` | gRPC server bind address |
+| `GRPC_HOST` | `[::1]` | gRPC bind address |
-| `GRPC_PORT` | `50051` | gRPC server port |
+| `GRPC_PORT` | `50051` | gRPC port |
-| `RUST_LOG` | `info` | Logging level |
+| `RUST_LOG` | `info` | Logging level filter |
-| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS authentication |
+| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS |
-| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC authorization |
+| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC |
-| `OTEL_ENABLE_CONSOLE` | `true` | Enable console logging |
+| `OTEL_ENABLE_CONSOLE` | `true` | Console exporter |
 | `OTEL_SERVICE_NAME` | `charybdis` | Service name for telemetry |
-### Configuration File Locations
+See `config.toml.example` for the complete template.
-Charybdis looks for configuration files in this order:
+## Troubleshooting
-1. `./config.toml` (current directory)
+### Port already in use
-2. `./charybdis.toml`
+```
-3. `/etc/charybdis/config.toml` (Linux/Unix)
+Error: transport error
 ```
 Find and free port 50051: `lsof -ti:50051 | xargs kill`, or set `GRPC_PORT=50052`.
-If none are found, it uses environment variables.
+### Database connection failed
 Verify the URL: `psql "$DATABASE_URL" -c "SELECT 1;"`.
-See `config.toml.example` for a complete configuration template with all options documented.
+### Permission denied with security enabled
 ```
 Code: PermissionDenied
 Message: Role 'X' does not have permission 'Y'
 ```
 Inspect your cert subject (OU determines the role) and the `[security.rbac.permissions]` table in `config.toml`. Reference: [security.md](security.md).
---
+## Next Steps
-**Need help?** Check the [troubleshooting guide](troubleshooting.md) or [open an issue](../../issues).
+- [Core Concepts](core-concepts.md) — entity model, events, annotations
 - [Architecture](architecture.md) — protobuf schema, storage, event bus
 - [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
 - [Vision & Roadmap](../VISION.md) — where Charybdis is going
@@ -1,125 +0,0 @@
 # Charybdis Documentation
 **The security-native platform engineering tool.**
 ## What is Charybdis?
 Charybdis is a platform engineering tool that unifies **software catalog**, **vulnerability management**, and **compliance posture** in a single event-driven platform. Built in Rust, deployed as a single binary.
 Instead of running Backstage + DefectDojo + Dependency-Track + a license scanner + a compliance spreadsheet, you run Charybdis.
 ### The Problems It Solves
 1. **Fragmented tooling** — Your software catalog, vulnerability data, license info, and compliance evidence live in 5 different tools that don't talk to each other. Charybdis unifies them.
 2. **Static catalog data** — Traditional catalogs rely on YAML files that go stale within weeks. Charybdis is event-driven — CI/CD pipelines and IaC tools register and update entities via gRPC, so the catalog is always accurate.
 3. **Security as an afterthought** — In Backstage, security is a plugin. In Charybdis, every entity carries its vulnerability posture, license status, and compliance state natively.
 4. **Manual provisioning** — New service? Manually create entries in every tool. With Charybdis, one gRPC call catalogs the service and event-driven plugins handle the rest.
 5. **Compliance evidence assembly** — Compliance reporting pulls from real vulnerability and license data, not spreadsheets.
 ## How It Works
 ```
 CI/CD or Scanner ──gRPC──> Charybdis
                             ├── Catalogs the service (event-driven)       ← Done
                             ├── Fires events to plugins (DefectDojo, ...) ← Done
                             ├── Ingests scan results (SARIF, CycloneDX)   ← Phase 1
                             ├── Evaluates security gates & rules          ← Phase 1
                             └── Exposes catalog via YAML adapter (Backstage) ← Done
 ```
 **One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.**
 ## Key Concepts
 | Concept | Description |
 |---------|-------------|
 | **Entity** | Anything in your software ecosystem: services, systems, components, APIs, users, groups, domains, resources |
 | **Vulnerability** | *(Phase 1)* A security finding linked to an entity, ingested from scanner output (SARIF, CycloneDX) |
 | **Assessment** | *(Phase 1)* The triage decision on a vulnerability: accept risk, remediate, auto-assessed by rules |
 | **Security Gate** | *(Phase 1)* Severity thresholds per product — blocks deployments when violated |
 | **Event Bus** | Publishes lifecycle events when entities or vulnerabilities change |
 | **Plugin** | Reacts to events to integrate with external systems (Slack, Jira, GitHub, custom) |
 | **Annotations** | Key-value metadata on entities for external references (e.g. `github.com/repo-slug`) |
 ## Architecture
 ```mermaid
 graph LR
    A[CI/CD / Scanners] -->|gRPC| B[Charybdis]
    B -->|Native| C[Software Catalog]
    B -->|Native| D[Vuln Management]
    B -->|Native| E[Compliance]
    B -->|Events| F[Plugins: Slack / Jira / Custom]
    B -->|YAML| G[Backstage - optional]
    style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
    style D fill:#E24A4A,stroke:#8A2E2E,color:#fff
    style E fill:#4AE28A,stroke:#2E8A5C,color:#fff
 ```
 ## Documentation
 ### Getting Started
 - [Installation & Quick Start](getting-started.md) — Get Charybdis running and register your first entity
 - [Demo Stack](../deploy/DEMO.md) — Full demo with DefectDojo
 ### Understanding Charybdis
 - [Core Concepts](core-concepts.md) — Entities, vulnerabilities, events, and data model
 - [Vision & Roadmap](../VISION.md) — Where Charybdis is going and why
 - [Architecture](architecture.md) — Technical design decisions
 ### Configuration
 - [Security](security.md) — mTLS authentication and RBAC authorization
 - [Plugin Configuration](PLUGIN_CONFIGURATION_GUIDE.md) — Setting up and configuring plugins
 ### Extending Charybdis
 - [Plugin Development](../plugins/README.md) — Build your own integration plugins
 ## Quick Example
 Register a service from your CI/CD pipeline:
 ```bash
 grpcurl -plaintext -d '{
  "entity": {
    "kind": "Component",
    "component_metadata": {
      "name": "payment-api",
      "description": "Payment processing service"
    },
    "component_spec": {
      "type": "service",
      "lifecycle": "production",
      "owner": "team-payments"
    }
  }
 }' charybdis:50051 charybdis.entities.EntityService/CreateEntity
 ```
 **What happens next:**
 - Entity stored in PostgreSQL with a UUID
 - `EntityCreated` event published to the event bus
 - Plugins react (e.g., DefectDojo creates a product automatically)
 - Entity available via gRPC and YAML adapter
 No YAML file to write. No PR to open. No manual provisioning.
 ## Backstage Migration
 Already using Backstage? Charybdis provides a YAML adapter for gradual migration. Point Backstage at Charybdis as a catalog source — entities registered via gRPC are immediately available in Backstage.
 ```yaml
 # backstage app-config.yaml
 catalog:
  locations:
    - type: url
      target: http://charybdis:8080/yaml/locations
 ```
 ---
 **Ready to get started?** Head to the [Getting Started Guide](getting-started.md).
@@ -67,17 +67,19 @@ export SECURITY_RBAC_ENABLED=true
 #### Development Certificates
-Use the provided test script:
+Use the provided dev-cert script:
 ```bash
-./test-mtls-rbac.sh
+./deploy/scripts/generate-dev-certs.sh
 ```
-This generates:
+This generates (under `deploy/certs/`):
- `ca.pem` / `ca-key.pem` - Certificate Authority
+- `ca.pem` / `ca-key.pem` — Certificate Authority
- `server-cert.pem` / `server-key.pem` - Server certificate  
+- `server-cert.pem` / `server-key.pem` — Server certificate
- `admin-cert.pem` / `admin-key.pem` - Admin client (OU=platform-team)
+- `admin-cert.pem` / `admin-key.pem` — Admin client (OU=platform-team)
- Various plugin certificates (OU=plugins)
+- Per-role client certs (OU=automation, OU=plugins)
 For an individual client cert without regenerating everything, use `./deploy/scripts/generate-client-cert.sh`.
 #### Production Certificates
@@ -494,10 +496,9 @@ Charybdis security features support compliance requirements:
 ## Next Steps
- 🔌 Configure [Plugins](plugins.md) with proper certificates
+- [Plugins](../plugins/README.md) — configure plugins to use mTLS client certs
- 🚀 Review [Deployment Guide](deployment.md) for production
+- [Architecture](architecture.md) — interceptor placement and observability hooks
 - 📊 Set up [Monitoring](monitoring.md) for security events
 ---
-**Security Questions?** Open a [security issue](../../security) (for vulnerabilities, use private disclosure).
+**Security Questions?** Open an issue (for vulnerabilities, use private disclosure).
@@ -1,340 +1,513 @@
 # Charybdis Plugins
-This directory contains plugin implementations that extend Charybdis with integrations to external security and development tools.
+Plugins extend Charybdis with integrations to external tools (security scanners, issue trackers, identity providers). Plugins are **compile-time integrated**: they live as Rust crates under `plugins/`, declare protobuf extensions in `plugins.toml`, and are linked into the `charybdis-server` binary at build time.
-## What are Plugins?
+This document covers:
 - The plugin model (event-driven vs sync, traits, lifecycle)
 - Configuring shipped plugins (DefectDojo, Keycloak)
 - Writing a new plugin
-Charybdis supports **two types of plugins**, both compile-time integrated:
+## Table of Contents
-### **1. Event-Driven Plugins**
+1. [Plugin Types](#plugin-types)
-React to entity lifecycle events (create, update, delete):
+2. [Available Plugins](#available-plugins)
- **Example**: DefectDojo, DependencyTrack
+3. [Plugin Configuration](#plugin-configuration)
- **Implement**: `EventDrivenPlugin` trait + `ResourceHandler`
+4. [Field Mapping System](#field-mapping-system)
- **Triggered by**: Entity CRUD operations
+5. [Entity Resolution](#entity-resolution)
- **Use case**: Auto-create resources in external tools when entities are created
+6. [DefectDojo Reference](#defectdojo-reference)
 7. [Keycloak Reference](#keycloak-reference)
 8. [Writing a New Plugin](#writing-a-new-plugin)
 9. [Plugin Lifecycle](#plugin-lifecycle)
 10. [Annotations](#annotations)
 11. [Troubleshooting](#troubleshooting)
-### **2. Sync Plugins**
+## Plugin Types
 Pull data from external sources on a schedule:
 - **Example**: Okta, Keycloak, Active Directory
 - **Implement**: `SyncPlugin` trait
 - **Triggered by**: Cron schedule or manual API call
 - **Use case**: Sync users/groups from identity providers
-## Generic Utilities
+### Event-Driven Plugins
 React to entity lifecycle events (`Created`, `Updated`, `Deleted`) — typically push data to an external system.
-All plugins have access to reusable utilities in `src/plugins/`:
+- Implement `EventDrivenPlugin` + one or more `ResourceHandler` per resource kind.
 - Example: **DefectDojo** — Component creation → DefectDojo product + engagement.
- **`PluginHttpClient`** - Multi-auth HTTP client (Token, Bearer, API Key, Basic Auth)
+### Sync Plugins
- **`AnnotationHelper`** - Consistent API for storing/retrieving plugin metadata
+Pull data from an external system on a schedule, creating or updating entities in Charybdis.
 - **`FieldMapper`** - Maps entity fields to external tool formats
 - **`DateUtils`** - Common date/time operations
-## Plugin Structure
+- Implement `SyncPlugin` (`schedule`, `on_startup`, `manual_trigger`).
 - Example: **Keycloak** — periodic sync of users and groups.
-Each plugin follows this structure:
+Both types implement the base `Plugin` trait (`name`, `validate_config`, `health_check`).
 ```
 plugins/
 └── my_plugin/
    ├── Cargo.toml              # Plugin crate definition
    ├── README.md               # Plugin-specific documentation
    ├── proto/
    │   └── my_plugin.proto     # Protobuf entity definitions
    └── src/
        ├── lib.rs              # Plugin entry point
        ├── handler.rs          # Event handler implementation
        ├── client.rs           # External API client
        └── config.rs           # Plugin configuration
 ```
 ## Available Plugins
-### DefectDojo
+| Plugin | Type | Status |
- **Type**: Event-Driven
+|---|---|---|
- **Purpose**: Security vulnerability management integration
+| DefectDojo | Event-driven | Shipped |
- **External API**: DefectDojo REST API v2
+| Keycloak | Sync | Shipped |
 - **Handlers**: ProductHandler, EngagementHandler, ProductTypeHandler, ProductMemberHandler
 - **Features**:
  - Automatic product creation on Component/Service creation
  - Product updates on entity changes
  - Auto-create CI/CD engagements
  - Owner resolution (Group → members → email annotation → DefectDojo user lookup)
  - Configurable OIDC provider mapping (Keycloak, Okta, Azure AD)
  - Annotation storage (`defectdojo.com/product-id`, `defectdojo.com/engagement-id`, `defectdojo.com/owner-member-ids`)
-### Keycloak
+Planned integrations (Slack, Jira, GitHub, Dependency-Track) are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).
 - **Type**: Sync
 - **Purpose**: Identity provider sync (users and groups)
 - **External API**: Keycloak Admin REST API
 - **Features**:
  - User sync with profile and annotations (display_name, email, picture)
  - Group sync with hierarchy (parent, children, members)
  - Configurable annotations (e.g., `keycloak.com/email` for OIDC mapping)
  - On-startup sync, manual trigger via gRPC
 ### DependencyTrack (Scaffolded)
 - **Type**: Event-Driven
 - **Purpose**: Software supply chain security
 - **External API**: DependencyTrack REST API
 - **Status**: Scaffolded (proto + crate structure), handler logic not implemented
 - **Planned**:
  - Auto-create projects for components
  - SBOM ingestion
 ## Creating a New Plugin
 ### Quick Start
 1. **Create plugin directory structure:**
   ```bash
   mkdir -p plugins/my_plugin/{proto,src}
   ```
 2. **Add to plugins.toml:**
   ```toml
   [plugins.my_plugin]
   enabled = true
   proto_path = "plugins/my_plugin/proto"
   metadata_field_number = 102  # Use next available number
   spec_field_number = 102
   description = "My custom integration"
   ```
 3. **Define protobuf schema:**
   Create `plugins/my_plugin/proto/my_plugin.proto`:
   ```protobuf
   syntax = "proto3";
   package charybdis.plugins.my_plugin;
   message MyPluginMetadata {
     string name = 1;
     string description = 2;
   }
   message MyPluginSpec {
     string integration_type = 1;
     bool enabled = 2;
   }
   ```
 4. **Create plugin crate:**
   Create `plugins/my_plugin/Cargo.toml`:
   ```toml
   [package]
   name = "charybdis-plugin-my-plugin"
   version = "0.1.0"
   edition = "2021"
   [dependencies]
   charybdis = { path = "../.." }
   async-trait = "0.1"
   tokio = { version = "1.0", features = ["full"] }
   tracing = "0.1"
   ```
 5. **Implement event handler:**
   Create `plugins/my_plugin/src/lib.rs`:
   ```rust
   use async_trait::async_trait;
   use charybdis::events::{EventHandler, EntityEvent, EventResult};
   pub struct MyPluginHandler;
   #[async_trait]
   impl EventHandler for MyPluginHandler {
       async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
           // React to entity changes
           Ok(())
       }
   }
   ```
 6. **Build and test:**
   ```bash
   cargo build
   cargo test
   ```
 ## Plugin Field Number Allocation
 Field numbers must be unique across all plugins to avoid protobuf conflicts:
 | Range    | Allocation            |
 |----------|-----------------------|
 | 1-99     | Core entity types     |
 | 100      | DefectDojo            |
 | 101      | DependencyTrack       |
 | 102-199  | Available for plugins |
 When creating a new plugin, use the next available number in the 102+ range.
 ## Plugin Configuration
-Plugins are configured in `config.toml` with `${VAR}` env var substitution for secrets:
+Plugins are configured in `config.toml` under `[plugins.<name>]`. Use `${VAR}` for secrets — never commit tokens.
 ```toml
 [plugins.defectdojo]
 enabled = true
-base_url = "${DEFECTDOJO_API_URL}"
+base_url = "${DEFECTDOJO_URL}"
 api_token = "${DEFECTDOJO_API_TOKEN}"
 [plugins.defectdojo.default_engagement]
 auto_create = true
 name = "CI/CD Pipeline"
 [plugins.keycloak]
 enabled = true
 base_url = "${KEYCLOAK_URL}"
-realm = "master"
+realm = "charybdis"
 client_id = "charybdis-sync"
 client_secret = "${KEYCLOAK_CLIENT_SECRET}"
 ```
-See `config.toml.example` for all options.
+When `enabled = false`, the plugin is loaded but does not register handlers and does not react to events. See `config.toml.example` for the full reference.
 ## Field Mapping System
 Event-driven plugins (currently DefectDojo) use a configurable mapper to translate Charybdis entity fields into external-tool API payloads. Mappings are defined per resource type under `[plugins.<name>.field_mappings.<resource>]`.
 Three mapping types are supported:
 ### 1. Direct field mapping (string)
 Dot-notation path into the entity.
 ```toml
 name = "metadata.name"
 description = "metadata.description"
 email = "spec.profile.email"
 tags = "metadata.tags"
 ```
 ### 2. Static value (object with `value` key)
 ```toml
 business_criticality = { value = "high" }
 is_active = { value = true }
 priority = { value = 100 }
 ```
 Supports strings, booleans, numbers, objects, and arrays.
 ### 3. Entity resolution (object)
 Resolve a reference into a related entity and extract a field from it.
 ```toml
 product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
 }
 ```
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `from` | string | yes | Source field path |
 | `resolve_entity` | string | yes | Entity kind to look up (`User`, `Component`, ...) |
 | `extract` | string | yes | Field path to extract from the resolved entity |
 | `resolve_array` | bool | no | Source field is an array of references |
 | `lookup_entity` | string | no | Find an entity that references the source |
 ## Entity Resolution
 Entity resolution lets a plugin navigate the entity graph at mapping time. The mapper performs a database lookup, extracts the target field, and substitutes it into the payload.
 ```
 1. Source field            spec.owner = "user:john.doe"
 2. Resolve entity          User entity for "john.doe"
 3. Extract target field    annotations.defectdojo.com/user-id = "123"
 4. Result                  product_manager = 123
 ```
 ### Array resolution
 ```toml
 user_ids = {
  from = "spec.members",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id",
  resolve_array = true
 }
 ```
 `Group.spec.members = ["user:john", "user:jane"]` → `user_ids = [123, 124]`.
 ### Linked entity lookup
 ```toml
 engagement_product = {
  from = "id",
  lookup_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
 }
 ```
 Finds a `Component` that references the source entity and extracts its annotation.
 ### Error handling
 Resolution failures (entity not found, missing annotation, invalid reference) log a warning and omit the field rather than failing the whole operation.
 ## DefectDojo Reference
 Maps Charybdis entities to DefectDojo resources via the REST API v2. Handlers shipped: `ProductHandler`, `EngagementHandler`, `ProductTypeHandler`, `ProductMemberHandler`.
 ### Connection
 ```toml
 [plugins.defectdojo]
 enabled = true
 base_url = "${DEFECTDOJO_URL}"
 api_token = "${DEFECTDOJO_API_TOKEN}"
 default_product_type_id = 1
 auto_create_users = true
 auto_create_product_types = false
 ```
 To obtain the API token: in DefectDojo, **User Profile → API Key**.
 ### Default engagement (auto-created with each product)
 ```toml
 [plugins.defectdojo.default_engagement]
 auto_create = true
 name = "CI/CD Pipeline"
 description = "Automated security scans from CI/CD pipeline"
 engagement_type = "CI/CD"
 status = "In Progress"
 duration_days = 365
 deduplication_on_engagement = true
 ```
 ### Owner resolution
 When a `Component` has an `owner` (typically a team name), Charybdis resolves it to DefectDojo users:
 1. Find `Group` entity matching the owner.
 2. Read `Group.spec.members` (usernames).
 3. Look up each `User` entity.
 4. Extract the user's email from the configured annotation.
 5. Find the matching user in DefectDojo by email.
 ```toml
 [plugins.defectdojo.owner_resolution]
 user_email_annotation = "keycloak.com/email"   # depends on your IdP
 defectdojo_lookup_field = "email"              # "email" or "username"
 assign_all_members = true                       # all group members, or just first match
 ```
 ### Product mapping (Component → DefectDojo Product)
 ```toml
 [plugins.defectdojo.field_mappings.product]
 name = "metadata.name"
 description = "metadata.description"
 product_type_id = { value = 1 }
 tags = "metadata.tags"
 business_criticality = { value = "high" }       # very high | high | medium | low | very low | none
 platform = { value = "web" }                    # web | mobile | desktop | iot | ...
 lifecycle = { value = "production" }
 origin = { value = "internal" }
 external_audience = { value = true }
 internet_accessible = { value = true }
 product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
 }
 ```
 ### User mapping (User → DefectDojo User)
 ```toml
 [plugins.defectdojo.field_mappings.user]
 username = "metadata.name"
 email = "spec.profile.email"
 first_name = "spec.profile.displayName"
 last_name = "spec.profile.displayName"
 is_active = { value = true }
 ```
 ### Product Type mapping (System → DefectDojo Product Type)
 ```toml
 [plugins.defectdojo.field_mappings.product_type]
 name = "metadata.name"
 description = "metadata.description"
 critical_product = { value = false }
 key_product = { value = true }
 ```
 ### Product Member mapping (Group → DefectDojo Product Member)
 ```toml
 [plugins.defectdojo.field_mappings.product_member]
 product_id = {
  from = "spec.parent",
  resolve_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
 }
 user_id = {
  from = "spec.members",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id",
  resolve_array = true
 }
 role_name = { value = "Reader" }    # Owner | Maintainer | Writer | Reader | API_Importer
 ```
 ### Engagement mapping (Resource → DefectDojo Engagement)
 ```toml
 [plugins.defectdojo.field_mappings.engagement]
 name = "metadata.name"
 description = "metadata.description"
 product_id = {
  from = "spec.owner",
  resolve_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
 }
 lead_id = {
  from = "spec.dependsOn",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
 }
 target_start = "spec.target_start"
 target_end = "spec.target_end"
 status = { value = "In Progress" }
 engagement_type = { value = "CI/CD" }
 version = "metadata.annotations.version"
 commit_hash = "metadata.annotations.commit_hash"
 branch_tag = "metadata.annotations.branch"
 build_id = "metadata.annotations.build_id"
 source_code_management_uri = "metadata.annotations.repo_url"
 deduplication_on_engagement = { value = true }
 api_test = { value = true }
 pen_test = { value = false }
 ```
 ### Annotations written by the plugin
 - `defectdojo.com/product-id`
 - `defectdojo.com/engagement-id`
 - `defectdojo.com/owner-member-ids`
 ## Keycloak Reference
 Sync plugin: pulls users and groups from a Keycloak realm via the Admin REST API on a schedule, on startup, or via manual trigger.
 ```toml
 [plugins.keycloak]
 enabled = true
 base_url = "${KEYCLOAK_URL}"
 realm = "charybdis"
 client_id = "charybdis-sync"
 client_secret = "${KEYCLOAK_CLIENT_SECRET}"
 [plugins.keycloak.sync]
 schedule = "0 */5 * * * *"   # optional cron; omit to sync only on startup/trigger
 on_startup = true
 manual_trigger = true
 sync_users = true
 sync_groups = true
 namespace = "keycloak"
 page_size = 100
 ```
 The service account behind `client_id`/`client_secret` must have `view-users` and `view-groups` on the realm.
 Synced entities carry annotations such as `keycloak.com/email`, `keycloak.com/id`, used by other plugins (e.g., the DefectDojo owner-resolution chain above).
 ## Writing a New Plugin
 ### 1. Scaffold directory
 ```bash
 mkdir -p plugins/my_plugin/{proto,src}
 ```
 ### 2. Register in `plugins.toml`
 ```toml
 [plugins.my_plugin]
 enabled = true
 proto_path = "plugins/my_plugin/proto"
 metadata_field_number = 103   # next available — see registry below
 spec_field_number = 203
 description = "My custom integration"
 ```
 ### 3. Define the protobuf schema
 `plugins/my_plugin/proto/my_plugin.proto`:
 ```protobuf
 syntax = "proto3";
 package charybdis.plugins.my_plugin;
 message MyPluginMetadata {
  string name = 1;
  string description = 2;
 }
 message MyPluginSpec {
  string integration_type = 1;
  bool enabled = 2;
 }
 ```
 `build.rs` regenerates `proto/entities.proto` automatically on next build.
 ### 4. Create the crate
 `plugins/my_plugin/Cargo.toml`:
 ```toml
 [package]
 name = "charybdis-my-plugin-plugin"
 version = "0.1.0"
 edition = "2024"
 [lib]
 name = "charybdis_my_plugin"
 path = "src/lib.rs"
 [dependencies]
 charybdis = { path = "../.." }
 async-trait = "0.1"
 anyhow = "1.0"
 serde = { version = "1.0", features = ["derive"] }
 tokio = { version = "1.48", features = ["full"] }
 tracing = "0.1"
 ```
 Add the crate to the workspace `members` in the top-level `Cargo.toml`.
 ### 5. Implement the plugin
 Event-driven example:
 ```rust
 use async_trait::async_trait;
 use charybdis::plugins::{EventDrivenPlugin, Plugin, PluginConfig, PluginType, ResourceHandler};
 use std::sync::Arc;
 pub struct MyPlugin { /* config, client, handlers */ }
 #[async_trait]
 impl Plugin for MyPlugin {
    fn name(&self) -> &str { "my_plugin" }
    fn plugin_type(&self) -> PluginType { PluginType::EventDriven }
    fn load_config(&mut self, _: PluginConfig) -> anyhow::Result<()> { Ok(()) }
    fn validate_config(&self) -> anyhow::Result<()> { Ok(()) }
 }
 #[async_trait]
 impl EventDrivenPlugin for MyPlugin {
    fn resource_handlers(&self) -> Vec<Arc<dyn ResourceHandler>> {
        vec![/* your handlers */]
    }
 }
 ```
 ### 6. Register in `charybdis-server`
 Add a loader block in `charybdis-server/src/main.rs` (follow the DefectDojo block as a template) and declare the dependency in `charybdis-server/Cargo.toml`.
 ### Field number registry
 | Range | Allocation |
 |---|---|
 | 1–99 | Core entity types |
 | 100 | DefectDojo |
 | 101 | DependencyTrack |
 | 102 | Keycloak |
 | 103+ | Available |
 Field numbers MUST be unique across plugins — protobuf wire format depends on it.
 ## Plugin Lifecycle
-1. **Build Time:**
+### Build time
-   - `build.rs` reads `plugins.toml`
+1. `build.rs` reads `plugins.toml`.
-   - Generates `proto/entities.proto` with plugin types
+2. Generates `proto/entities.proto` with plugin-contributed `oneof` variants.
-   - Compiles all protobuf files
+3. Compiles all protobuf files (`tonic-prost-build`).
   - Generates Rust code
-2. **Runtime:**
+### Runtime
-   - Plugin handlers registered with event bus
+1. `charybdis-server` loads each enabled plugin from `config.toml`.
-   - Events published on entity CRUD operations
+2. Event-driven plugins register their `ResourceHandler`s with the dispatcher.
-   - Handlers react asynchronously
+3. Sync plugins register with the cron scheduler (and optionally run once at startup).
-   - External APIs called as needed
+4. Entity CRUD operations publish `EntityEvent` to the in-memory event bus.
 5. The dispatcher fans out events to matching handlers (filtered by `trigger_kinds`).
 6. Handlers call external APIs and write annotations back to Charybdis.
-3. **Event Flow:**
+### Event flow
-   ```
+```
-   Client creates entity
+Client → gRPC CreateEntity → DB INSERT → EntityEvent::Created → Dispatcher
-   → Entity stored in database
+                                                                  ↓
-   → Event published to event bus
+                                          ResourceHandler.handle_create(entity)
-   → Plugin handler receives event
+                                                                  ↓
-   → Plugin calls external API
+                                          POST /api/products → DefectDojo
-   → Plugin stores external ID in annotations
+                                                                  ↓
-   ```
+                                          update_annotations("defectdojo.com/product-id")
 ## Using Annotations
 Plugins store external tool IDs in entity annotations:
 ```rust
 // Store external ID
 entity.annotations.insert(
    "my-plugin.com/resource-id".to_string(),
    "ext-12345".to_string(),
 );
 // Query by external ID in SQL
 SELECT * FROM entities 
 WHERE annotations->>'my-plugin.com/resource-id' = 'ext-12345';
 ```
-### Annotation Naming Convention
+## Annotations
 Plugins store external IDs in entity annotations using reverse-DNS keys:
 Use reverse-DNS style:
 - `defectdojo.com/product-id`
- `dependencytrack.com/project-uuid`
+- `defectdojo.com/engagement-id`
- `github.com/repo-slug`
+- `keycloak.com/id`
- `{tool}.com/{resource}-{attribute}`
+- `keycloak.com/email`
-## Best Practices
+Query by annotation via JSONB in PostgreSQL:
-### 1. Error Handling
+```sql
- Don't panic - return errors
+SELECT * FROM entities
- Log but continue on non-critical failures
+WHERE annotations->>'defectdojo.com/product-id' = '456';
 - Implement retry logic for transient errors
 ### 2. Idempotency
 - Check if resource exists before creating
 - Make operations safe to retry
 - Handle duplicate creation gracefully
 ### 3. Performance
 - Don't block event handlers
 - Use `tokio::spawn` for long operations
 - Batch operations when possible
 ### 4. Testing
 - Unit test event handlers
 - Mock external API clients
 - Integration tests with real APIs (optional)
 ### 5. Documentation
 - Document required environment variables
 - Provide example configurations
 - Explain entity model and annotations
 ## Contributing Plugins
 We welcome plugin contributions! To contribute:
 1. Fork the repository
 2. Create your plugin following the structure above
 3. Add comprehensive tests
 4. Document configuration and usage
 5. Submit a pull request
 ### Plugin Requirements
 - [ ] Protobuf definitions with Metadata and Spec messages
 - [ ] Event handler implementation
 - [ ] External API client (if applicable)
 - [ ] Configuration via environment variables
 - [ ] README with setup instructions
 - [ ] Unit tests for event handler
 - [ ] Example usage in documentation
 ## Plugin Distribution Models
 ### In-Tree (Current)
 Plugins live in the `plugins/` directory and are enabled via `plugins.toml`.
 **Pros:**
 - Easy to discover
 - Consistent quality
 - Tested together
 **Cons:**
 - Requires core repo access
 - All plugins built together
 ### External Crates (Future)
 Plugins distributed as separate Rust crates.
 **Example:**
 ```toml
 [dependencies]
 charybdis-plugin-custom = "0.1"
 ```
-**Pros:**
+Use `EntityRepository::update_annotations()` for atomic JSONB merge — it avoids the read-modify-write race when multiple handlers write to the same entity.
 - Independent versioning
 - Community contributions
 - Optional dependencies
-**Cons:**
+## Troubleshooting
 - Discovery harder
 - Compatibility challenges
-### Plugin Marketplace (Future)
+### Plugin not reacting to events
-Central registry of available plugins (like Backstage).
+- `enabled = true` in `config.toml`?
 - Plugin registered in `charybdis-server/src/main.rs`?
 - Entity kind matches the handler's `trigger_kinds`?
 - Event bus running? (`info!("Event bus started")` in startup logs)
-## Resources
+### Field mapping not applied
 - Path uses dot notation, e.g. `metadata.name`, not `meta.name`.
 - Static value uses `{ value = "..." }`, not a bare string when you want a literal.
 - Field actually exists on the entity (inspect via the YAML adapter or `GetEntity` gRPC).
- [Plugin Configuration Guide](../docs/PLUGIN_CONFIGURATION_GUIDE.md)
+### Entity resolution returns nothing
- [Core Concepts](../docs/core-concepts.md)
+- Resolved entity exists in the DB.
- [Architecture](../docs/architecture.md)
+- Resolved entity has the `extract` annotation set.
- [Protobuf Style Guide](https://protobuf.dev/programming-guides/style/)
+- `resolve_entity` uses the PascalCase kind (`"User"`, not `"user"`).
 - For arrays, `resolve_array = true` is set.
-## Support
+### DefectDojo API errors
 - `400 Bad Request: Invalid product type ID` → check `default_product_type_id` or your `product_type_id` mapping.
 - `401 Unauthorized` → check `DEFECTDOJO_API_TOKEN`.
 - `409 Conflict: user already exists` → expected, the plugin reuses existing users.
- Open an issue for bug reports
+### Performance
- Discussions for questions
+- Minimize entity-resolution chain depth.
- PRs for contributions
+- Prefer static values when the data isn't entity-bound.
 - Check network latency to the external API — handlers run synchronously per event.
-## License
+## See Also
-Same as Charybdis core (see LICENSE file in repository root)
+- [Core Concepts](../docs/core-concepts.md) — entity model, events, annotations.
 - [Architecture](../docs/architecture.md) — protobuf schema, storage, scaling.
 - [Field Mapper Tests](../src/plugins/field_mapper.rs) — reference behavior.