doc: update and cleanup

2026-06-09 11:26:47 +02:00
parent ab13af48e7
commit f1b1543f29
11 changed files with 1113 additions and 3376 deletions
@@ -14,7 +14,7 @@ Contributions are welcome! Whether it's a bug report, feature idea, documentatio

 ```bash
 # Clone the repo
-git clone https://github.com/YOUR-ORG/charybdis.git
+git clone <your-charybdis-repo-url>
 cd charybdis

 # Copy config
@@ -1,41 +1,26 @@
 # Charybdis

-**The security-native platform engineering tool.**
+**A security-native platform engineering tool.**

-One platform for your software catalog, vulnerability management, and compliance posture. Event-driven. Single binary. No YAML files to maintain.
+One Rust binary for your software catalog. Event-driven, gRPC-native, no YAML to maintain. Designed to grow into a unified catalog + vulnerability management + compliance platform.

-## Why Charybdis?
+## What Works Today

-Platform engineers today run Backstage for the catalog, DefectDojo for vulnerabilities, Dependency-Track for SBOMs, a license scanner, and a compliance spreadsheet. Five tools, five sources of truth, none of them talking to each other.
+- **Software catalog** — gRPC API for entity CRUD (Component, System, API, User, Group, Domain, Resource, Finding). Real-time event bus on every change.
+- **Vulnerability ingestion** — gRPC `IngestionService` ingests SARIF reports. Findings are deduplicated, reconciled against existing state (new / unchanged / resolved / reopened), and dry-runnable for MR-level diffs.
+- **Backstage YAML adapter** — HTTP endpoint exposing entities as Backstage-compatible Location YAML, for migration or coexistence.
+- **mTLS + RBAC** — Certificate-based auth with fine-grained, OU-mapped roles.
+- **OpenTelemetry** — Traces, metrics, and logs out of the box (console + OTLP).
+- **Plugin system** — Event-driven (DefectDojo) and sync (Keycloak) plugins. Compile-time integrated.
+- **PostgreSQL storage** — Zero-migration: entities stored as protobuf blobs in a fixed schema. New plugins add `oneof` variants, never columns.

-Charybdis unifies this:
-
- **Software Catalog** — Event-driven, gRPC-native. Services register from CI/CD or IaC. No static YAML, no polling, always accurate.
- **Vulnerability Management** — Ingest scan results natively (SARIF, CycloneDX, SPDX). Triage, assess, track. No external vuln tool needed.
- **Compliance & Licenses** — Security gates, license policies, compliance framework mappings. Built-in, not bolted on.
-
-All of this in a **single Rust binary** that uses ~50MB of RAM.
-
-## How It Works
-
-```
-CI/CD Pipeline ──gRPC──> Charybdis
-Scanner results ───────>   ├── Catalogs the service (event-driven)
-IaC tools ─────────────>   ├── Ingests vulnerabilities natively
-                           ├── Evaluates security gates
-                           └── Fires events to plugins (Slack, Jira, ...)
-```
-
-**One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.**
+For everything else (assessment workflows, security gates, license policies, compliance frameworks, more parsers, more plugins), see [VISION.md](VISION.md) and [TODO.md](TODO.md).

 ## Quick Start

 ```bash
 # 1. Start PostgreSQL
-docker run -d \
-  -e POSTGRES_PASSWORD=mysecretpassword \
-  -p 5432:5432 \
-  postgres:15
+docker run -d -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 postgres:15

 # 2. Configure
 cp config.toml.example config.toml
@@ -62,187 +47,66 @@ grpcurl -plaintext -d '{
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```

-The entity is stored, events are fired, and plugins react automatically.
+Detailed walkthrough: [docs/getting-started.md](docs/getting-started.md).

-## Key Features
-
-### Software Catalog
- **gRPC API** for programmatic entity management from CI/CD, scripts, or any tool
- **Event-driven plugins** auto-provision external tools when entities change
- **Zero-migration storage** — PostgreSQL with protobuf + JSONB. No schema changes, ever
- **Rich entity model** — Component, System, API, User, Group, Domain, Resource
- **Backstage compatible** — Built-in YAML adapter for migration or coexistence
-
-### Security (Native — Phase 1)
-Security is a first-class concept in Charybdis, not a plugin. These features are under active development:
- **Vulnerability ingestion** — Push scan results via gRPC (SARIF, CycloneDX, SPDX)
- **Assessment workflow** — Triage, accept risk, remediate, auto-assess via rules
- **Security gates** — Severity thresholds per product, block deployments on violations
- **License compliance** — Track licenses, enforce policies, flag violations
- **Compliance frameworks** — Map vulnerabilities to NIS2, SOC2, DORA requirements
-
-### Platform
- **mTLS + RBAC** — Certificate-based auth with fine-grained permissions
- **OpenTelemetry** — Full observability (traces, metrics, logs) out of the box
- **Single binary** — Deploy one Rust binary + PostgreSQL. That's it.
- **Sub-millisecond latency** — Tested at 170k+ entities
-
-## Architecture
-
-```mermaid
-graph TB
-    subgraph "Sources"
-        A1[CI/CD Pipelines]
-        A2[IaC Tools]
-        A3[Security Scanners]
-    end
-
-    subgraph "Charybdis Core"
-        B1[gRPC API]
-        B2[Software Catalog]
-        B3[Vuln Management]
-        B4[Security Gates]
-        B5[Event Bus]
-    end
-
-    subgraph "Integrations - Plugins"
-        C1[Slack / Teams]
-        C2[Jira / GitHub Issues]
-        C3[Custom Plugins]
-    end
-
-    subgraph "Frontend"
-        D2[Backstage - optional]
-    end
-
-    A1 -->|gRPC| B1
-    A2 -->|gRPC| B1
-    A3 -->|Scan Results| B1
-    B1 --> B2
-    B1 --> B3
-    B3 --> B4
-    B2 --> B5
-    B3 --> B5
-    B5 --> C1
-    B5 --> C2
-    B5 --> C3
-    B2 -.->|YAML Adapter| D2
-```
-
-## Entity Model
-
-| Kind | Description | Example |
-|------|-------------|---------|
-| **Service** | Individual microservices or applications | `payment-api`, `auth-service` |
-| **Component** | Reusable libraries, SDKs, modules | `auth-sdk`, `logging-lib` |
-| **System** | Collections of components working together | `payment-system` |
-| **API** | Interfaces exposed by components | `payments-rest-api` |
-| **User** | Individual people | `john.doe` |
-| **Group** | Teams and organizational units | `team-payments` |
-| **Domain** | Business domains | `payments`, `shipping` |
-| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
-
-Each entity will carry its security posture natively once Phase 1 is complete: vulnerabilities, license status, compliance state, and assessment history.
-
-## Plugin System
-
-Core features (catalog, vulns, compliance) are **native**. Plugins handle **integrations** with external systems:
+## How It Works

 ```
-EntityCreated ──> Event Bus ──> DefectDojo Plugin ──> Creates product
-                            ──> Slack Plugin ──> Notifies channel (planned)
-                            ──> Custom Plugin ──> Your logic
+CI/CD Pipeline ──gRPC──> Charybdis
+Scanner output ────────>   ├── Catalogs the entity
+                           ├── Reconciles vulnerabilities (SARIF)
+                           └── Fires events to plugins (DefectDojo, ...)
 ```

-Two plugin types:
- **Event-Driven** — React to entity/vulnerability events in real-time
- **Sync** — Pull data from external sources on a schedule (e.g., sync users from Keycloak)
+See [docs/architecture.md](docs/architecture.md) for the full design (protobuf schema, storage model, event bus, plugin lifecycle).

-**Current plugins:**
- DefectDojo — Auto-create products, engagements, and assign owners (Done)
- Keycloak — Sync users and groups with annotations (Done)
- Dependency-Track — Scaffolded
- Jira, Slack, GitHub — Planned
+## Entity Kinds

-Build your own with the `EventDrivenPlugin` or `SyncPlugin` traits.
+| Kind | Purpose |
+|---|---|
+| **Component** | Services, libraries, applications |
+| **System** | Collections of components |
+| **API** | Interfaces exposed by components |
+| **User**, **Group** | People and teams |
+| **Domain** | Business domains |
+| **Resource** | Infrastructure resources |
+| **Finding** | Security findings ingested from scanners |

-## Backstage Compatibility
+Conceptual details: [docs/core-concepts.md](docs/core-concepts.md).

-Already using Backstage? Charybdis works as a **drop-in dynamic backend**. Point Backstage at Charybdis's YAML adapter and stop maintaining `catalog-info.yaml` files:
+## Plugins

-```yaml
-# backstage app-config.yaml
-catalog:
-  locations:
-    - type: url
-      target: http://charybdis:8080/yaml/locations
-      rules:
-        - allow: [Component, System, Service, API, User, Group]
-```
+Two plugin types, both compile-time integrated:

-Or use Charybdis standalone via its gRPC API — no Backstage needed.
+- **Event-driven** — React to entity lifecycle events. *Shipped: DefectDojo (auto-create products, engagements, member assignment).*
+- **Sync** — Pull data from external systems on a schedule. *Shipped: Keycloak (users + groups).*

-## vs. Alternatives
-
-| | Charybdis | Backstage | Port / Cortex / OpsLevel |
-|---|---|---|---|
-| **Type** | Open source | Open source | Commercial SaaS |
-| **Catalog** | Event-driven, real-time | Static YAML, polling | Varies |
-| **Security** | Native (first-class) | Plugins (fragmented) | Limited / add-on |
-| **Deployment** | Single binary + PG | Node.js cluster + PG + plugins | Hosted |
-| **Performance** | ~50MB RAM, sub-ms | ~1GB+ RAM | N/A |
-| **Compliance** | Native frameworks | Manual | Some |
-| **Cost** | Free | Free (+ operational cost) | $$$$ |
-
-## Project Status
-
-| Module | Status | Phase |
-|--------|--------|-------|
-| Core gRPC API | Done | 0 |
-| PostgreSQL storage | Done | 0 |
-| Event bus system | Done | 0 |
-| Plugin framework | Done | 0 |
-| mTLS + RBAC | Done | 0 |
-| OpenTelemetry | Done | 0 |
-| Backstage YAML adapter | Done | 0 |
-| DefectDojo plugin | Done | 0 |
-| Keycloak plugin | Done | 0 |
-| Vulnerability ingestion | Planned | 1 |
-| SARIF / CycloneDX / SPDX parsers | Planned | 1 |
-| Security gates & rules engine | Planned | 1 |
-| Compliance frameworks | Planned | 3 |
-
-See [VISION.md](VISION.md) for the full roadmap.
+Plugin model, configuration, and writing your own: [plugins/README.md](plugins/README.md).

 ## Documentation

- [Vision & Roadmap](VISION.md) — Where Charybdis is going
- [Getting Started](docs/getting-started.md) — Installation and first steps
- [Core Concepts](docs/core-concepts.md) — Entities, events, and architecture
+- [Getting Started](docs/getting-started.md) — installation and first entity
+- [Core Concepts](docs/core-concepts.md) — entities, events, annotations
+- [Architecture](docs/architecture.md) — protobuf schema, storage, event bus
 - [Security](docs/security.md) — mTLS and RBAC configuration
- [Architecture](docs/architecture.md) — Deep dive into design decisions
- [Plugin Guide](plugins/README.md) — Building and using plugins
- [Demo](deploy/DEMO.md) — Full stack demo with DefectDojo
+- [Plugins](plugins/README.md) — DefectDojo, Keycloak, building your own
+- [Demo](deploy/DEMO.md) — full stack demo with DefectDojo
+- [Vision & Roadmap](VISION.md) — where Charybdis is going

 ## Technology Stack

 | | |
 |---|---|
-| **Language** | Rust |
-| **API** | gRPC + Protocol Buffers |
-| **Database** | PostgreSQL 14+ (protobuf + JSONB, zero-migration) |
-| **Frontend** | Backstage YAML adapter (compatible) |
-| **Security** | mTLS (rustls) + RBAC |
-| **Observability** | OpenTelemetry (traces, metrics, logs) |
+| Language | Rust |
+| API | gRPC + Protocol Buffers |
+| Database | PostgreSQL 14+ (protobuf + JSONB, zero-migration) |
+| TLS | rustls (no OpenSSL dependency) |
+| Observability | OpenTelemetry (traces, metrics, logs) |

 ## Contributing

-Contributions are welcome! Whether it's a new plugin, a security parser, or documentation improvements — we'd love your help.
+See [CONTRIBUTING.md](CONTRIBUTING.md).

 ## License

-[Apache-2.0](LICENSE)
-
---
-
-**One platform. Catalog. Security. Compliance. Built in Rust.**
+[Apache-2.0](LICENSE.md)
@@ -1,7 +1,7 @@
 # Charybdis - TODO List

-**Last Updated**: 2026-05-06
-**Status**: Phase 0 done. Phase 1 (Security Core) in progress — assessment & gates remaining.
+**Last Updated**: 2026-06-09
+**Status**: Phase 0 done. Phase 1 (Security Core) in progress — ingestion + reconciliation shipped; assessment & gates remaining.

 > Aligned with [VISION.md](VISION.md) roadmap.

@@ -12,12 +12,21 @@
 > **Goal**: Native vulnerability management and scan ingestion. Replace DefectDojo for finding lifecycle management.
 > **Architecture**: Core features (reconciliation, dedup, dry-run) in `src/`. Parsers extensible via `ScannerParser` trait. Plugins contribute parsers via `contributed_parsers()`.

+### Done
+
+- [x] `Finding` entity kind (proto + storage)
+- [x] `IngestionService` gRPC endpoint (`ImportScan` + `DryRunScan`)
+- [x] `ScannerParser` trait + `ParserRegistry`
+- [x] SARIF parser (built-in)
+- [x] Fingerprint-based deduplication (scanner-provided when present, sha256 fallback)
+- [x] Reconciliation engine (new / unchanged / resolved / reopened buckets, scoped to `(component, lifecycle)`)
+
 ### Remaining

 - [ ] CycloneDX VEX parser (vulnerability data from SBOMs)
 - [ ] `Plugin::contributed_parsers()` default impl on base trait
- [ ] Publish events: FindingCreated, FindingResolved, FindingReopened (for downstream plugins)
- [ ] Assessment workflow (triage, accept risk, remediate)
+- [ ] Publish events: `FindingCreated`, `FindingResolved`, `FindingReopened` (for downstream plugins)
+- [ ] Assessment workflow (triage, accept risk, remediate — the `ACCEPTED` / `FALSE_POSITIVE` state values exist but no API sets them)
 - [ ] Rules engine for auto-assessment
 - [ ] Security gates (severity thresholds per product)
 - [ ] License tracking and policy engine
@@ -288,9 +288,8 @@ Backstage will discover all entities from Charybdis automatically. See the `dock

 ## Support

- GitHub Issues: [github.com/charybdis-catalog/charybdis/issues](https://github.com/charybdis-catalog/charybdis/issues)
- Documentation: [docs/](docs/)
- Discord: [Join our community](https://discord.gg/...)
+- Documentation: [../docs/](../docs/)
+- Open an issue on the project's Gitea/GitHub repository

 ---

@@ -1,126 +1,54 @@
 # Core Concepts

-Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.
+Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes.

-This page explains the key concepts you need to understand.
-
-## Architecture Overview
-
-```mermaid
-graph TB
-    subgraph "Sources"
-        A1[CI/CD Pipelines]
-        A2[Security Scanners]
-        A3[IaC Tools]
-    end
-
-    subgraph "Charybdis Core"
-        B1[gRPC API]
-        B3[Authentication & Authorization]
-        C1[Entity Service]
-        C4[Vulnerability Engine]
-        C5[Security Gates & Rules]
-        C2[Event Bus]
-    end
-
-    subgraph "Integrations - Plugins"
-        D1[Slack / Teams]
-        D2[Jira / GitHub]
-        D3[Custom Plugins]
-    end
-
-    subgraph "Data Layer"
-        E1[(PostgreSQL)]
-    end
-
-    A1 -.gRPC.-> B1
-    A2 -.Scan Results.-> B1
-    A3 -.gRPC.-> B1
-
-    B1 --> B3
-    B3 --> C1
-    B3 --> C4
-    C4 --> C5
-    C1 --> E1
-    C4 --> E1
-    C1 --> C2
-    C4 --> C2
-
-    C2 -.Events.-> D1
-    C2 -.Events.-> D2
-    C2 -.Events.-> D3
-
-    style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
-    style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
-    style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
-```
+For a higher-level overview see the [README](../README.md). For deep technical details see [architecture.md](architecture.md).

 ## Entities

-An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.
+An **entity** is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding.

-### Entity Structure
+### Entity structure

-Every entity has three main parts:
+Every entity has the same shape:

-```mermaid
-classDiagram
-    class Entity {
-        +string id
-        +string kind
-        +metadata
-        +spec
-        +annotations
-        +timestamps
-    }
-    
-    class Metadata {
-        +string name
-        +string namespace
-        +string description
-        +labels
-        +links
-        +tags
-    }
-    
-    class Spec {
-        +string type
-        +string lifecycle
-        +string owner
-        +dependencies
-    }
-    
-    Entity --> Metadata
-    Entity --> Spec
-```
+- `id` — UUID assigned by the server
+- `kind` — one of the supported kinds (see below)
+- `metadata` — identifying data (name, description, labels, tags, links)
+- `spec` — kind-specific configuration (type, lifecycle, owner, dependencies, ...)
+- `annotations` — arbitrary `string → string` map, typically used by plugins to store external IDs
+- `created_at`, `updated_at` — server-managed timestamps

-### Entity Kinds
+`metadata` and `spec` are protobuf `oneof` fields whose variant matches `kind`. So a `Component` entity carries `component_metadata` + `component_spec`, a `User` carries `user_metadata` + `user_spec`, and so on. The on-wire JSON uses the variant name directly.

-Charybdis supports all standard Backstage entity kinds:
+### Entity kinds

-| Kind | Description | Example |
-|------|-------------|---------|
-| **Service** | Individual microservices or applications | `payment-api` |
-| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
-| **System** | Collections of services working together | `e-commerce-platform` |
-| **API** | Interfaces exposed by components | `payments-rest-api` |
-| **User** | Individual people | `john.doe` |
-| **Group** | Teams and organizational units | `team-payments` |
-| **Domain** | Business domains | `payments`, `shipping` |
-| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
+| Kind | Purpose |
+|---|---|
+| `Component` | Services, applications, libraries, websites |
+| `System` | Collections of components working together |
+| `API` | Interfaces exposed by components |
+| `User` | Individual people |
+| `Group` | Teams and organizational units |
+| `Domain` | Business domains |
+| `Resource` | Infrastructure resources (databases, caches, queues) |
+| `Finding` | Security findings ingested from scanners (see [Findings](#findings)) |

-#### Example: Service
+Per-kind protobuf definitions: `proto/core/*.proto`.
+
+### Example: Component

 ```json
 {
-  "kind": "Service",
-  "service_metadata": {
+  "kind": "Component",
+  "component_metadata": {
    "name": "payment-api",
    "namespace": "production",
    "description": "Payment processing service",
-    "labels": { "team": "payments" }
+    "labels": { "team": "payments", "tier": "critical" },
+    "tags": ["api", "pci-dss"]
  },
-  "service_spec": {
+  "component_spec": {
    "type": "service",
    "lifecycle": "production",
    "owner": "team-payments",
@@ -129,500 +57,136 @@ Charybdis supports all standard Backstage entity kinds:
 }
 ```

-#### Example: Component
+### Common metadata fields

-```json
-{
-  "kind": "Component",
-  "component_metadata": {
-    "name": "auth-sdk",
-    "namespace": "shared"
-  },
-  "component_spec": {
-    "type": "library",
-    "lifecycle": "production",
-    "owner": "platform-team"
-  }
-}
-```
+| Field | Type | Notes |
+|---|---|---|
+| `name` | string | Required. Unique with `kind` (composite index `idx_entities_kind_name`). |
+| `namespace` | string | Logical grouping, e.g. `production`. |
+| `description` | string | Human-readable. |
+| `labels` | `map<string,string>` | Categorization. |
+| `tags` | `repeated string` | Free-form classification. |
+| `links` | `repeated Link` | External URLs (dashboards, docs). |

-#### Example: System
+## Annotations

-```json
-{
-  "kind": "System",
-  "system_metadata": {
-    "name": "e-commerce-platform",
-    "namespace": "production",
-    "description": "Complete e-commerce system"
-  },
-  "system_spec": {
-    "owner": "platform-team",
-    "domain": "retail"
-  }
-}
-```
-
-### Metadata Fields
-
-| Field | Type | Description | Required |
-|-------|------|-------------|----------|
-| `name` | string | Entity name (unique within namespace) | ✅ |
-| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
-| `description` | string | Human-readable description | ❌ |
-| `labels` | map | Key-value pairs for categorization | ❌ |
-| `tags` | array | Search tags | ❌ |
-| `links` | array | External URLs (dashboards, docs, etc.) | ❌ |
-
-### Annotations
-
-Annotations store integration-specific metadata:
+Annotations are an open `map<string, string>` on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB.

 ```json
 {
  "annotations": {
    "github.com/repo-slug": "myorg/payment-service",
    "defectdojo.com/product-id": "123",
-    "dependencytrack.com/project-uuid": "550e8400...",
-    "pagerduty.com/service-id": "PXYZ123",
-    "grafana.com/dashboard-url": "https://..."
+    "keycloak.com/email": "alice@example.com",
+    "pagerduty.com/service-id": "PXYZ123"
  }
 }
 ```

-**Best Practices**:
- Use domain-style keys (`tool.com/key`)
- Store tool-specific IDs
- Keep values as strings
- Use for integration metadata only
+Convention: reverse-DNS keys (`tool.com/key`). Values are always strings.
+
+`EntityRepository::update_annotations()` performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers.

 ## Events

-Charybdis uses an **event-driven architecture** to trigger actions when entities change.
+Charybdis emits an `EntityEvent` on every CRUD operation, delivered to subscribers via the in-memory event bus.

-### Event Flow
-
-```mermaid
-sequenceDiagram
-    participant Client
-    participant API as Entity Service
-    participant Bus as Event Bus
-    participant Plugin1 as DefectDojo Plugin
-    participant Plugin2 as Dependency-Track
-    
-    Client->>API: CreateEntity(service)
-    API->>API: Store entity
-    API->>Bus: Emit EntityCreated event
-    Bus->>Plugin1: Handle event
-    Bus->>Plugin2: Handle event
-    Plugin1-->>Plugin1: Create DefectDojo product
-    Plugin2-->>Plugin2: Create DT project
-    API-->>Client: Return created entity
-    
-    Note over Bus,Plugin2: Asynchronous processing
+```
+Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers
 ```

-### Event Types
+### Event types

-| Event | Trigger | Plugins Receive |
-|-------|---------|----------------|
-| `EntityCreated` | New entity created | Full entity data |
-| `EntityUpdated` | Entity modified | Updated entity + changes |
-| `EntityDeleted` | Entity removed | Entity ID + metadata |
+| Type | When |
+|---|---|
+| `Created` | Successful `CreateEntity` |
+| `Updated` | Successful `UpdateEntity` (full or partial) |
+| `Deleted` | Successful `DeleteEntity` |

-### Event Structure
+### Event payload

 ```rust
-pub enum EntityEvent {
-    Created {
-        entity: Entity,
-        timestamp: DateTime<Utc>,
-    },
-    Updated {
-        entity: Entity,
-        previous: Entity,
-        timestamp: DateTime<Utc>,
-    },
-    Deleted {
-        id: String,
-        metadata: Metadata,
-        timestamp: DateTime<Utc>,
-    },
+pub struct EntityEvent {
+    pub event_id: Uuid,
+    pub entity_id: Uuid,
+    pub event_type: EntityEventType,   // Created | Updated | Deleted
+    pub timestamp: DateTime<Utc>,
+    pub metadata: HashMap<String, String>,
+    pub entity_data: Option<Arc<Entity>>,  // full entity for Created/Updated
 }
 ```

-## Vulnerabilities & Security (Phase 1 — Planned)
+### Delivery semantics

-> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.
+- **Asynchronous** — `CreateEntity` returns to the client before plugin handlers complete.
+- **At-least-once intent**, but the default `MemoryEventBus` is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in [VISION.md](../VISION.md).
+- **Per-handler timeout** — 30 s. Slow plugins don't block the bus.
+- **Panic isolation** — a panicking handler doesn't take down others.

-Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.
+Plugin lifecycle and how to write a handler: [plugins/README.md](../plugins/README.md).

-### Vulnerability Lifecycle
+## Findings

-```mermaid
-sequenceDiagram
-    participant Scanner
-    participant API as Charybdis API
-    participant Rules as Rules Engine
-    participant Gate as Security Gate
-    participant Bus as Event Bus
-    participant Plugin as Slack / Jira
+`Finding` is a first-class entity kind for security findings produced by scanners. Findings are created by the `IngestionService`, not directly by clients.

-    Scanner->>API: IngestScan(SARIF report)
-    API->>API: Parse & create Vulnerability entities
-    API->>API: Link vulnerabilities to Component
-    API->>Rules: Evaluate auto-assessment rules
-    Rules-->>API: Auto-assess (e.g., accept known low-risk)
-    API->>Gate: Check security gate thresholds
-    alt Gate Passed
-        Gate-->>API: OK
-    else Gate Failed
-        Gate->>Bus: GateFailed event
-        Bus->>Plugin: Alert #security channel
-    end
-    API->>Bus: VulnerabilitiesIngested event
-    Bus->>Plugin: Notify / create tickets
-```
+Each finding is scoped to a `(component_ref, lifecycle)` pair — so production and staging environments track findings independently without duplicating the underlying `Component`.

-### Scan Ingestion
+The ingestion flow:

-Charybdis ingests scan results natively. You don't need an external vulnerability management tool.
+1. Client calls `IngestionService.ImportScan(component_ref, lifecycle, format, data)`.
+2. The configured parser (currently SARIF) normalizes scanner output into `NormalizedFinding` records.
+3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`) and diffs incoming vs existing findings for that scope.
+4. The result is one of:
+   - **New** — fingerprint not seen before → create a `Finding` entity.
+   - **Unchanged** — already active → bump `last_seen`.
+   - **Resolved** — previously active, not in this scan → mark `Resolved`.
+   - **Reopened** — previously resolved or false-positive, detected again → mark `Reopened`.
+5. `DryRunScan` runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments.

-| Format | Coverage | Use Case |
-|--------|----------|----------|
-| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
-| **CycloneDX** | SBOMs + vulnerability data | SCA, license |
-| **SPDX** | License and package data | License compliance |
+Finding state values (see `proto/core/finding.proto`):

-### Assessments
+| State | Meaning |
+|---|---|
+| `ACTIVE` | Currently detected |
+| `RESOLVED` | No longer detected (auto-set on re-import) |
+| `ACCEPTED` | Risk accepted by human decision *(state value defined; assessment workflow is planned)* |
+| `FALSE_POSITIVE` | Marked as not a real issue *(state value defined; assessment workflow is planned)* |
+| `REOPENED` | Was resolved, detected again |

-Each vulnerability can be assessed:
-
-| Status | Meaning |
-|--------|---------|
-| **Open** | New, unreviewed vulnerability |
-| **In Triage** | Under review by security team |
-| **Accepted** | Risk accepted with justification |
-| **Remediated** | Fixed, pending verification |
-| **False Positive** | Not a real vulnerability |
-| **Auto-Assessed** | Automatically assessed by rules engine |
-
-### Security Gates
-
-Security gates define thresholds per product:
-
-```
-payment-api:
-  critical: 0     # No critical vulns allowed
-  high: 5         # Up to 5 high
-  medium: 20      # Up to 20 medium
-```
-
-When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).
-
-### Rules Engine
-
-Rules auto-assess vulnerabilities based on patterns:
-
- Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
- Scanner source (e.g., "all informational from ZAP → auto-accept")
- Known patterns (e.g., "CVE-XXXX already accepted org-wide")
-
-### License Compliance (Phase 2)
-
-Track licenses across your dependency tree:
-
- Ingest license data from CycloneDX/SPDX
- Define license policies (allowed, restricted, banned)
- Flag violations per component
- Compliance reporting
+Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).

 ## Storage Model

-Charybdis uses PostgreSQL with JSONB for schema-less storage.
-
-### Database Schema
+Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a `oneof` variant in protobuf, not a column migration.

 ```sql
 CREATE TABLE entities (
-    id UUID PRIMARY KEY,
-    kind TEXT NOT NULL,
-    entity_data BYTEA NOT NULL,         -- Protobuf binary
-    annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
-    created_at TIMESTAMP NOT NULL,
-    updated_at TIMESTAMP NOT NULL
+    id          UUID PRIMARY KEY,
+    kind        VARCHAR NOT NULL,
+    name        VARCHAR,
+    entity_data BYTEA NOT NULL,            -- protobuf-encoded Entity
+    annotations JSONB NOT NULL DEFAULT '{}',
+    created_at  TIMESTAMPTZ NOT NULL,
+    updated_at  TIMESTAMPTZ NOT NULL
 );

-CREATE INDEX idx_entities_kind ON entities(kind);
-CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
+CREATE INDEX idx_entities_kind            ON entities(kind);
+CREATE INDEX idx_entities_kind_name       ON entities(kind, name);
+CREATE INDEX idx_entities_annotations     ON entities USING GIN (annotations);
+CREATE INDEX idx_entities_created_at_id   ON entities(created_at DESC, id DESC);
 ```

-### Why Protobuf + JSONB?
+Why this design:
+- **No migrations** — new plugins extend the protobuf `oneof` variants; database schema is unchanged.
+- **Compact storage** — protobuf binary is smaller than JSON.
+- **Forward/backward compatible** — proto field numbers protect against version drift.
+- **Fast annotation queries** — JSONB + GIN index supports `annotations->>'key' = 'value'` lookups in O(log n).

-**Advantages**:
- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
- ✅ Protobuf binary storage for compact, versioned entity data
- ✅ JSONB annotations for fast querying with GIN indexes
- ✅ Forward/backward compatibility built-in
+Detailed schema reasoning, scaling notes, and field-mask paths: [architecture.md](architecture.md).

-**Example Query**:
-```sql
-- Find entities with specific annotation
-SELECT * FROM entities 
-WHERE annotations->>'defectdojo.com/product-id' = '42';
-```
+## See Also

-## Security Model
-
-Charybdis implements defense-in-depth security.
-
-### Authentication: mTLS
-
-```mermaid
-sequenceDiagram
-    participant Client
-    participant TLS as TLS Layer
-    participant Auth as Auth Interceptor
-    participant Service as Entity Service
-    
-    Client->>TLS: Connect with client certificate
-    TLS->>TLS: Validate certificate
-    TLS->>Auth: Extract certificate
-    Auth->>Auth: Parse identity (CN, OU, O)
-    Auth->>Auth: Map to role
-    Client->>Auth: Request (with identity)
-    Auth->>Auth: Check permissions
-    alt Authorized
-        Auth->>Service: Forward request
-        Service-->>Client: Response
-    else Denied
-        Auth-->>Client: PermissionDenied error
-    end
-```
-
-### Authorization: RBAC
-
-**Role Mapping**:
-
-```
-Certificate (CN, OU, O) → RBAC Role → Permissions
-```
-
-**Default Roles**:
-
-| Role | Certificate OU | Permissions |
-|------|---------------|-------------|
-| `platform` | `platform-team` | Full access (CRUD + list) |
-| `automation` | `automation` | Create, read, update, list |
-| `plugin` | `plugins` | Read, list only |
-
-### Permissions
-
-| Permission | Operations | Required For |
-|------------|-----------|--------------|
-| `entity:create` | Create new entities | CreateEntity |
-| `entity:read` | Get entity by ID | GetEntity |
-| `entity:update` | Modify entities | UpdateEntity |
-| `entity:delete` | Remove entities | DeleteEntity |
-| `entity:list` | List all entities | ListEntities |
-
-## Plugin System
-
-Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.
-
-### Plugin Architecture
-
-```mermaid
-graph LR
-    A[Entity/Vuln Event] --> B[Event Bus]
-    B --> C{Plugin Manager}
-    C --> D[Slack Plugin]
-    C --> E[Jira Plugin]
-    C --> F[Custom Plugin]
-
-    D --> G[Slack API]
-    E --> H[Jira API]
-    F --> I[Your Tool API]
-
-    style C fill:#4A90E2,stroke:#2E5C8A
-```
-
-### What's Native vs. Plugin
-
-| Native (core) | Status | Plugin (integration) | Status |
-|---|---|---|---|
-| Software catalog | Done | DefectDojo sync | Done |
-| Vulnerability management | Phase 1 | Keycloak user sync | Done |
-| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
-| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
-| License compliance | Phase 2 | Custom integrations | Framework ready |
-| TechDocs | Future | | |
-
-### Plugin Lifecycle
-
-1. **Configuration** - Load plugin settings from `plugins.toml`
-2. **Initialization** - Plugin registers event handlers
-3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
-4. **External Integration** - Plugin calls external tool APIs
-5. **Error Handling** - Failed plugins don't affect core service
-
-### Plugin Configuration
-
-Example `plugins.toml`:
-
-```toml
-[plugins.slack]
-enabled = true
-webhook_url = "${SLACK_WEBHOOK_URL}"
-
-[[plugins.slack.on_gate_failed]]
-channel = "#security-alerts"
-
-[[plugins.slack.on_entity_created]]
-channel = "#platform"
-```
-
-See [Plugin Guide](plugins.md) for details.
-
-## Data Consistency
-
-### Eventual Consistency
-
-Charybdis uses **eventual consistency** for plugin integrations:
-
- Entity CRUD operations are **immediately consistent**
- Plugin synchronization is **eventually consistent**
- Events are processed **asynchronously**
-
-```mermaid
-graph LR
-    A[CreateEntity] -->|Immediate| B[Entity Stored]
-    B -->|Async| C[Event Emitted]
-    C -->|Async| D[Plugin Processing]
-    D -->|Eventual| E[External Tool Synced]
-    
-    style B fill:#00C851
-    style E fill:#ffbb33
-```
-
-### Guarantees
-
-| Operation | Consistency | Guarantee |
-|-----------|-------------|-----------|
-| Entity CRUD | Strong | Immediate |
-| Entity queries | Strong | Read-your-writes |
-| Event delivery | At-least-once | May retry |
-| Plugin sync | Eventual | Best-effort |
-
-## Performance Characteristics
-
-### Scalability
-
- **Entities**: Tested with 100,000+ entities
- **Throughput**: 1,000+ requests/second
- **Latency**: Sub-millisecond average
- **Concurrency**: Tokio async runtime
-
-### Resource Usage
-
-Typical resource consumption:
-
-| Component | CPU | Memory | Storage |
-|-----------|-----|--------|---------|
-| Charybdis | < 5% | ~50 MB | Minimal |
-| PostgreSQL | ~10% | ~256 MB | Depends on entity count |
-
-### Optimization Tips
-
-1. **Index annotations** used for frequent queries
-2. **Use connection pooling** (built-in)
-3. **Enable query caching** in PostgreSQL
-4. **Monitor event bus** queue depth
-
-## Backstage Migration
-
-Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.
-
-### How It Works
-
-```mermaid
-graph LR
-    A[Charybdis Entity] --> B[YAML Adapter]
-    B --> C[Backstage YAML Format]
-    C --> D[Backstage Catalog]
-
-    style B fill:#4A90E2,stroke:#2E5C8A
-```
-
-Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:
-
-```yaml
-catalog:
-  locations:
-    - type: url
-      target: http://charybdis:8080/yaml/locations
-```
-
-### Recommended Migration Path
-
-1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
-2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
-3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
-4. **Optionally retire** Backstage when Charybdis covers your catalog needs
-
-## Best Practices
-
-### Entity Design
-
-✅ **DO**:
- Use descriptive names
- Group by namespace
- Add relevant labels
- Include documentation links
- Set appropriate owners
-
-❌ **DON'T**:
- Store sensitive data in metadata
- Use very long descriptions
- Create deeply nested hierarchies
- Duplicate data across entities
-
-### Naming Conventions
-
-```
-<entity-type>-<purpose>-<environment>
-
-Examples:
- payment-api-prod
- user-service-staging
- auth-library
- e-commerce-system
-```
-
-### Metadata Organization
-
-```json
-{
-  "labels": {
-    "team": "payments",           // Ownership
-    "tier": "critical",            // Importance
-    "environment": "production"    // Deployment
-  },
-  "tags": ["pci-compliant", "public-api"],
-  "links": [
-    { "url": "...", "title": "Dashboard" },
-    { "url": "...", "title": "Documentation" }
-  ]
-}
-```
-
-## Next Steps
-
- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
- Configure [Security](security.md) for production (mTLS + RBAC)
- Explore [Plugins](plugins.md) for external integrations
- Review the [Architecture](architecture.md) for technical deep dive
-
---
-
-**Questions?** [Open an issue](../../issues).
+- [Architecture](architecture.md) — protobuf schema, scaling, design decisions
+- [Security](security.md) — mTLS + RBAC configuration
+- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
+- [Vision & Roadmap](../VISION.md) — planned features
@@ -1,39 +1,26 @@
 # Getting Started with Charybdis

-This guide will get you from zero to a running Charybdis instance in minutes. By the end, you'll have a working software catalog that can register services via gRPC, ingest scan results, and auto-trigger integrations via event-driven plugins.
+This guide gets you from zero to a running Charybdis instance: a software catalog with a gRPC API, optional Backstage YAML adapter, and plugin integrations on entity events.

 ## Prerequisites

- **Rust** - 1.70 or later ([install](https://rustup.rs/))
- **PostgreSQL** - 14 or later
- **grpcurl** - For testing (optional, [install](https://github.com/fullstorydev/grpcurl))
+- **Rust** 1.70+ ([install](https://rustup.rs/))
+- **PostgreSQL** 14+
+- **protoc** + `libprotobuf-dev` (for the build script)
+- **grpcurl** for testing (optional, [install](https://github.com/fullstorydev/grpcurl))

-## Installation
-
-### Option 1: From Source
+## Install

 ```bash
-# Clone the repository
-git clone https://github.com/charybdis-catalog/charybdis.git
+git clone <your-charybdis-repo-url>
 cd charybdis
-
-# Build
 cargo build --release
-
-# The binary will be at target/release/charybdis
-```
-
-### Option 2: Docker (Coming Soon)
-
-```bash
-docker pull charybdis/charybdis:latest
+# Binary at target/release/charybdis-server
 ```

 ## Quick Start

-### Step 1: Start PostgreSQL
-
-Using Docker:
+### 1. Start PostgreSQL

 ```bash
 docker run -d \
@@ -43,54 +30,22 @@ docker run -d \
  postgres:15
 ```

-Or use an existing PostgreSQL instance.
-
-### Step 2: Configure Charybdis
-
-**Recommended: Use config.toml**
-
-Copy the example configuration:
+### 2. Configure

 ```bash
 cp config.toml.example config.toml
-```
-
-Edit `config.toml` and set your database URL:
-
-```toml
-[database]
-url = "${DATABASE_URL}"
-```
-
-Set the environment variable:
-
-```bash
 export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
 ```

-**Alternative: Environment Variables Only (Legacy)**
+`config.toml` reads `${DATABASE_URL}` from the environment. See [Configuration](#configuration) below for all options.

-If you prefer environment variables:
-
-```bash
-# Database
-export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
-
-# Disable security for quick start
-export SECURITY_MTLS_ENABLED=false
-export SECURITY_RBAC_ENABLED=false
-
-# Logging
-export RUST_LOG=info,charybdis=debug
-```
-
-### Step 3: Run Charybdis
+### 3. Run

 ```bash
 cargo run
 ```

-You should see:
+Expected startup logs:

 ```
 INFO  charybdis: Database ready
@@ -98,40 +53,34 @@ INFO  charybdis: Event bus started successfully
 INFO  charybdis: EntityService server listening on [::1]:50051
 ```

-### Step 4: Verify It's Working
-
-Test with grpcurl:
+### 4. Verify

 ```bash
-# List available services
 grpcurl -plaintext localhost:50051 list
-
-# Output:
 # charybdis.entities.EntityService
+# charybdis.ingestion.IngestionService
 # grpc.reflection.v1.ServerReflection
 ```

-Congratulations! Charybdis is running! 🎉
-
-## Creating Your First Entity
-
-### Using grpcurl
-
-Create a service entity:
+## Create Your First Entity

 ```bash
-grpcurl -plaintext \
-  -d '{
-    "entity": {
-      "kind": "Service",
-      "service_metadata": {
-        "name": "payment-service",
-        "namespace": "production",
-        "description": "Core payment processing service"
-      }
+grpcurl -plaintext -d '{
+  "entity": {
+    "kind": "Component",
+    "component_metadata": {
+      "name": "payment-api",
+      "namespace": "production",
+      "description": "Payment processing service",
+      "tags": ["api", "critical"]
+    },
+    "component_spec": {
+      "type": "service",
+      "lifecycle": "production",
+      "owner": "team-payments"
    }
-  }' \
-  localhost:50051 charybdis.entities.EntityService/CreateEntity
+  }
+}' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```

 Response:
@@ -140,117 +89,60 @@ Response:
 {
  "entity": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
-    "kind": "Service",
-    "serviceMetadata": {
-      "name": "payment-service",
-      "namespace": "production",
-      "description": "Core payment processing service"
-    },
-    "createdAt": "2025-11-04T10:00:00Z",
-    "updatedAt": "2025-11-04T10:00:00Z"
+    "kind": "Component",
+    "componentMetadata": { "...": "..." },
+    "createdAt": "2026-06-01T10:00:00Z",
+    "updatedAt": "2026-06-01T10:00:00Z"
  }
 }
 ```

-### List All Entities
-
-```bash
-grpcurl -plaintext -d '{}' \
-  localhost:50051 charybdis.entities.EntityService/ListEntities
-```
-
-### Get Entity by ID
+### Retrieve and list

 ```bash
+# Get by ID
 grpcurl -plaintext \
  -d '{"id": "550e8400-e29b-41d4-a716-446655440000"}' \
  localhost:50051 charybdis.entities.EntityService/GetEntity
+
+# List all
+grpcurl -plaintext -d '{}' \
+  localhost:50051 charybdis.entities.EntityService/ListEntities
+
+# Filter by kind and name
+grpcurl -plaintext \
+  -d '{"kind": "Component", "name": "payment-api"}' \
+  localhost:50051 charybdis.entities.EntityService/ListEntities
 ```

-## Entity Types
+## Entity Kinds

-Charybdis supports three main entity types:
+The valid `kind` values are: `Component`, `System`, `API`, `User`, `Group`, `Domain`, `Resource`, `Finding`. Each kind uses a matching `<kind>_metadata` + `<kind>_spec` payload.

-### Service
+Conceptual reference: [core-concepts.md](core-concepts.md). Per-kind protobuf definitions: `proto/core/*.proto`.

-Individual microservices or applications:
-
-```bash
-grpcurl -plaintext -d '{
-  "entity": {
-    "kind": "Service",
-    "service_metadata": {
-      "name": "user-api",
-      "namespace": "production",
-      "description": "User management API"
-    }
-  }
-}' localhost:50051 charybdis.entities.EntityService/CreateEntity
-```
-
-### System
-
-Collections of related services:
-
-```bash
-grpcurl -plaintext -d '{
-  "entity": {
-    "kind": "System",
-    "system_metadata": {
-      "name": "payment-system",
-      "namespace": "production",
-      "description": "Complete payment processing system"
-    }
-  }
-}' localhost:50051 charybdis.entities.EntityService/CreateEntity
-```
-
-### Component
-
-Reusable components or libraries:
+### Component example with metadata and annotations

 ```bash
 grpcurl -plaintext -d '{
  "entity": {
    "kind": "Component",
-    "component_spec": {
-      "type": "library",
-      "lifecycle": "production",
-      "owner": "platform-team"
-    },
    "component_metadata": {
-      "name": "auth-library",
-      "namespace": "shared",
-      "description": "Shared authentication library"
-    }
-  }
-}' localhost:50051 charybdis.entities.EntityService/CreateEntity
-```
-
-## Adding Metadata
-
-Entities support rich metadata:
-
-```bash
-grpcurl -plaintext -d '{
-  "entity": {
-    "kind": "Service",
-    "service_metadata": {
      "name": "payment-service",
      "namespace": "production",
      "description": "Payment processing",
-      "labels": {
-        "team": "payments",
-        "tier": "critical"
-      },
-      "links": [
-        {
-          "url": "https://dashboard.company.com/payments",
-          "title": "Dashboard",
-          "icon": "dashboard"
-        }
-      ],
-      "tags": ["payments", "pci-compliant", "critical"]
+      "labels": { "team": "payments", "tier": "critical" },
+      "links": [{
+        "url": "https://dashboard.company.com/payments",
+        "title": "Dashboard",
+        "icon": "dashboard"
+      }],
+      "tags": ["payments", "pci-compliant"]
+    },
+    "component_spec": {
+      "type": "service",
+      "lifecycle": "production",
+      "owner": "team-payments"
    },
    "annotations": {
      "github.com/repo-slug": "myorg/payment-service",
@@ -260,105 +152,85 @@ grpcurl -plaintext -d '{
 }' localhost:50051 charybdis.entities.EntityService/CreateEntity
 ```

-## Integrating with CI/CD
+## Registering Entities from CI

-### Example: GitHub Actions
-
-Create a workflow to register services automatically:
+Charybdis is designed to be called from pipelines. With `grpcurl` available in your runner:

 ```yaml
-name: Register Service
-on:
-  push:
-    branches: [main]
-
-jobs:
-  register:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Register in Charybdis
-        run: |
-          grpcurl -plaintext \
-            -d '{
-              "entity": {
-                "kind": "Service",
-                "service_metadata": {
-                  "name": "${{ github.event.repository.name }}",
-                  "namespace": "production",
-                  "description": "${{ github.event.repository.description }}"
-                },
-                "annotations": {
-                  "github.com/repo-slug": "${{ github.repository }}"
-                }
-              }
-            }' \
-            your-charybdis-host:50051 \
-            charybdis.entities.EntityService/CreateEntity
-```
-
-### Example: GitLab CI
-
-```yaml
-register_service:
-  stage: deploy
-  script:
-    - |
-      grpcurl -plaintext \
-        -d "{
-          \"entity\": {
-            \"kind\": \"Service\",
-            \"service_metadata\": {
-              \"name\": \"${CI_PROJECT_NAME}\",
-              \"namespace\": \"${CI_ENVIRONMENT_NAME}\"
-            }
+# Example pipeline step (Gitea Actions / GitHub Actions syntax)
+- name: Register service in Charybdis
+  run: |
+    grpcurl -plaintext \
+      -d "{
+        \"entity\": {
+          \"kind\": \"Component\",
+          \"component_metadata\": {
+            \"name\": \"$CI_PROJECT_NAME\",
+            \"namespace\": \"production\"
+          },
+          \"component_spec\": {
+            \"type\": \"service\",
+            \"lifecycle\": \"production\",
+            \"owner\": \"$CI_PROJECT_NAMESPACE\"
+          },
+          \"annotations\": {
+            \"repo-slug\": \"$CI_PROJECT_PATH\"
          }
-        }" \
-        your-charybdis-host:50051 \
-        charybdis.entities.EntityService/CreateEntity
+        }
+      }" \
+      charybdis.internal:50051 \
+      charybdis.entities.EntityService/CreateEntity
 ```

+In production, secure the endpoint with mTLS (see [Enabling Security](#enabling-security)).
+
 ## Enabling Security

-For production use, enable mTLS and RBAC:
+Charybdis ships with mTLS + RBAC disabled for local exploration. For shared or production environments, enable both.

-### Step 1: Generate Certificates
+### 1. Generate dev certificates

 ```bash
-# Use the provided test script
-./test-mtls-rbac.sh
+./deploy/scripts/generate-dev-certs.sh
 ```

-This creates:
- `certs/ca.pem` - Certificate Authority
- `certs/server-cert.pem` / `server-key.pem` - Server certificate
- `certs/admin-cert.pem` / `admin-key.pem` - Admin client certificate
+This writes `deploy/certs/` with:
+- `ca.pem` — CA
+- `server-cert.pem` / `server-key.pem` — server
+- `admin-cert.pem` / `admin-key.pem` — admin client
+- (and per-role client certs)

-### Step 2: Enable Security
+### 2. Enable in `config.toml`

-```bash
-export SECURITY_MTLS_ENABLED=true
-export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem
-export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem
-export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem
-export SECURITY_RBAC_ENABLED=true
+```toml
+[security.mtls]
+enabled = true
+server_cert = "./deploy/certs/server-cert.pem"
+server_key  = "./deploy/certs/server-key.pem"
+client_ca_cert = "./deploy/certs/ca.pem"
+
+[security.rbac]
+enabled = true
 ```

-### Step 3: Test with mTLS
+Restart Charybdis. RBAC defaults map cert OUs to roles (`platform-team` → full access, `automation` → CRUD without delete, `plugins` → read-only).
+
+### 3. Call with mTLS

 ```bash
 grpcurl \
-  -cacert certs/ca.pem \
-  -cert certs/admin-cert.pem \
-  -key certs/admin-key.pem \
+  -cacert deploy/certs/ca.pem \
+  -cert deploy/certs/admin-cert.pem \
+  -key deploy/certs/admin-key.pem \
  -d '{}' \
  localhost:50051 charybdis.entities.EntityService/ListEntities
 ```

-See the [Security Guide](security.md) for detailed configuration.
+Full reference (custom role mappings, audit logging, reverse-proxy mode): [security.md](security.md).

 ## Backstage Migration (Optional)

-If you currently use Backstage and want to migrate gradually, the built-in YAML adapter serves entities in Backstage format:
+If you currently run Backstage, Charybdis serves entities in Backstage's Location YAML format:

 ```yaml
 # backstage app-config.yaml
@@ -367,78 +239,36 @@ catalog:
    - type: url
      target: http://your-charybdis-host:8080/yaml/locations
      rules:
-        - allow: [Component, System, Service]
+        - allow: [Component, System, API, User, Group]
 ```

-Backstage will automatically discover and import entities from Charybdis. You can run both in parallel — entities managed via gRPC are immediately visible in Backstage.
-
-## Troubleshooting
-
-### Port Already in Use
-
-```
-Error: transport error
-```
-
-**Solution**: Check if another process is using port 50051:
-
-```bash
-lsof -ti:50051
-```
-
-Kill the process or change the port:
-
-```bash
-export GRPC_PORT=50052
-```
-
-### Database Connection Failed
-
-```
-Error: password authentication failed
-```
-
-**Solution**: Verify your DATABASE_URL:
-
-```bash
-# Test connection
-psql "$DATABASE_URL" -c "SELECT 1;"
-```
-
-### Permission Denied (with security enabled)
-
-```
-Code: PermissionDenied
-Message: Role 'X' does not have permission 'Y'
-```
-
-**Solution**: Check your certificate and role mappings. See [Security Guide](security.md).
-
-## Next Steps
-
-Now that you have Charybdis running:
-
-1. Learn about [Core Concepts](core-concepts.md) — entities, vulnerabilities, events
-2. Read the [Vision & Roadmap](../VISION.md) — where Charybdis is going
-3. Configure [Security](security.md) for production (mTLS + RBAC)
-4. Explore [Plugins](../plugins/README.md) for external integrations
+Backstage discovers entities by polling the endpoint. Entities created via gRPC are visible on the next poll. The YAML adapter is served by an HTTP listener separate from the gRPC port (default `:8080`).

 ## Configuration

-Charybdis supports two configuration methods:
+Charybdis loads its config from (in order):

-### 1. Configuration File (Recommended)
+1. `./config.toml`
+2. `./charybdis.toml`
+3. `/etc/charybdis/config.toml`
+4. Environment variables (fallback)

-Use `config.toml` for structured configuration:
+### `config.toml` skeleton

 ```toml
-# config.toml
 [server]
 grpc_host = "[::1]"
 grpc_port = 50051

+[server.yaml_adapter]
+enabled = true
+host = "0.0.0.0"
+port = 8080
+
 [database]
-url = "${DATABASE_URL}"  # Environment variable substitution
+url = "${DATABASE_URL}"
+max_connections = 10
+connection_timeout_secs = 30

 [security.mtls]
 enabled = false
@@ -450,59 +280,52 @@ enabled = false
 service_name = "charybdis"
 environment = "development"
 enable_console = true
-```
-
-**Benefits:**
- ✅ Organized by section (server, database, security, telemetry, plugins)
- ✅ Environment variable substitution with `${VAR_NAME}`
- ✅ Comments and documentation inline
- ✅ Easy to version control (excluding secrets)
- ✅ No need to export dozens of environment variables
-
-**Using Environment Variables in config.toml:**
-
-```toml
-[database]
-url = "${DATABASE_URL}"  # Will be substituted at runtime

 [plugins.defectdojo]
-api_key = "${DEFECTDOJO_API_KEY}"  # Secrets stay in environment
+enabled = false
+# see plugins/README.md for the full plugin reference
 ```

-Then set only the secrets:
+`${VAR}` and `${VAR:-default}` substitution works in any string value — keep secrets in the environment, not in the file.

-```bash
-export DATABASE_URL="postgresql://..."
-export DEFECTDOJO_API_KEY="secret-key"
-```
+### Environment-variable fallback

-### 2. Environment Variables (Legacy)
-
-If `config.toml` is not found, Charybdis falls back to environment variables:
+If no config file is found, these env vars are read:

 | Variable | Default | Description |
-|----------|---------|-------------|
+|---|---|---|
 | `DATABASE_URL` | (required) | PostgreSQL connection string |
-| `GRPC_HOST` | `[::1]` | gRPC server bind address |
-| `GRPC_PORT` | `50051` | gRPC server port |
-| `RUST_LOG` | `info` | Logging level |
-| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS authentication |
-| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC authorization |
-| `OTEL_ENABLE_CONSOLE` | `true` | Enable console logging |
+| `GRPC_HOST` | `[::1]` | gRPC bind address |
+| `GRPC_PORT` | `50051` | gRPC port |
+| `RUST_LOG` | `info` | Logging level filter |
+| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS |
+| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC |
+| `OTEL_ENABLE_CONSOLE` | `true` | Console exporter |
 | `OTEL_SERVICE_NAME` | `charybdis` | Service name for telemetry |

-### Configuration File Locations
+See `config.toml.example` for the complete template.

-Charybdis looks for configuration files in this order:
+## Troubleshooting

-1. `./config.toml` (current directory)
-2. `./charybdis.toml`
-3. `/etc/charybdis/config.toml` (Linux/Unix)
+### Port already in use
+```
+Error: transport error
+```
+Find and free port 50051: `lsof -ti:50051 | xargs kill`, or set `GRPC_PORT=50052`.

-If none are found, it uses environment variables.
+### Database connection failed
+Verify the URL: `psql "$DATABASE_URL" -c "SELECT 1;"`.

-See `config.toml.example` for a complete configuration template with all options documented.
+### Permission denied with security enabled
+```
+Code: PermissionDenied
+Message: Role 'X' does not have permission 'Y'
+```
+Inspect your cert subject (OU determines the role) and the `[security.rbac.permissions]` table in `config.toml`. Reference: [security.md](security.md).

---
+## Next Steps

-**Need help?** Check the [troubleshooting guide](troubleshooting.md) or [open an issue](../../issues).
+- [Core Concepts](core-concepts.md) — entity model, events, annotations
+- [Architecture](architecture.md) — protobuf schema, storage, event bus
+- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
+- [Vision & Roadmap](../VISION.md) — where Charybdis is going
@@ -1,125 +0,0 @@
-# Charybdis Documentation
-
-**The security-native platform engineering tool.**
-
-## What is Charybdis?
-
-Charybdis is a platform engineering tool that unifies **software catalog**, **vulnerability management**, and **compliance posture** in a single event-driven platform. Built in Rust, deployed as a single binary.
-
-Instead of running Backstage + DefectDojo + Dependency-Track + a license scanner + a compliance spreadsheet, you run Charybdis.
-
-### The Problems It Solves
-
-1. **Fragmented tooling** — Your software catalog, vulnerability data, license info, and compliance evidence live in 5 different tools that don't talk to each other. Charybdis unifies them.
-
-2. **Static catalog data** — Traditional catalogs rely on YAML files that go stale within weeks. Charybdis is event-driven — CI/CD pipelines and IaC tools register and update entities via gRPC, so the catalog is always accurate.
-
-3. **Security as an afterthought** — In Backstage, security is a plugin. In Charybdis, every entity carries its vulnerability posture, license status, and compliance state natively.
-
-4. **Manual provisioning** — New service? Manually create entries in every tool. With Charybdis, one gRPC call catalogs the service and event-driven plugins handle the rest.
-
-5. **Compliance evidence assembly** — Compliance reporting pulls from real vulnerability and license data, not spreadsheets.
-
-## How It Works
-
-```
-CI/CD or Scanner ──gRPC──> Charybdis
-                             ├── Catalogs the service (event-driven)       ← Done
-                             ├── Fires events to plugins (DefectDojo, ...) ← Done
-                             ├── Ingests scan results (SARIF, CycloneDX)   ← Phase 1
-                             ├── Evaluates security gates & rules          ← Phase 1
-                             └── Exposes catalog via YAML adapter (Backstage) ← Done
-```
-
-**One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.**
-
-## Key Concepts
-
-| Concept | Description |
-|---------|-------------|
-| **Entity** | Anything in your software ecosystem: services, systems, components, APIs, users, groups, domains, resources |
-| **Vulnerability** | *(Phase 1)* A security finding linked to an entity, ingested from scanner output (SARIF, CycloneDX) |
-| **Assessment** | *(Phase 1)* The triage decision on a vulnerability: accept risk, remediate, auto-assessed by rules |
-| **Security Gate** | *(Phase 1)* Severity thresholds per product — blocks deployments when violated |
-| **Event Bus** | Publishes lifecycle events when entities or vulnerabilities change |
-| **Plugin** | Reacts to events to integrate with external systems (Slack, Jira, GitHub, custom) |
-| **Annotations** | Key-value metadata on entities for external references (e.g. `github.com/repo-slug`) |
-
-## Architecture
-
-```mermaid
-graph LR
-    A[CI/CD / Scanners] -->|gRPC| B[Charybdis]
-    B -->|Native| C[Software Catalog]
-    B -->|Native| D[Vuln Management]
-    B -->|Native| E[Compliance]
-    B -->|Events| F[Plugins: Slack / Jira / Custom]
-    B -->|YAML| G[Backstage - optional]
-
-    style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
-    style D fill:#E24A4A,stroke:#8A2E2E,color:#fff
-    style E fill:#4AE28A,stroke:#2E8A5C,color:#fff
-```
-
-## Documentation
-
-### Getting Started
- [Installation & Quick Start](getting-started.md) — Get Charybdis running and register your first entity
- [Demo Stack](../deploy/DEMO.md) — Full demo with DefectDojo
-
-### Understanding Charybdis
- [Core Concepts](core-concepts.md) — Entities, vulnerabilities, events, and data model
- [Vision & Roadmap](../VISION.md) — Where Charybdis is going and why
- [Architecture](architecture.md) — Technical design decisions
-
-### Configuration
- [Security](security.md) — mTLS authentication and RBAC authorization
- [Plugin Configuration](PLUGIN_CONFIGURATION_GUIDE.md) — Setting up and configuring plugins
-
-### Extending Charybdis
- [Plugin Development](../plugins/README.md) — Build your own integration plugins
-
-## Quick Example
-
-Register a service from your CI/CD pipeline:
-
-```bash
-grpcurl -plaintext -d '{
-  "entity": {
-    "kind": "Component",
-    "component_metadata": {
-      "name": "payment-api",
-      "description": "Payment processing service"
-    },
-    "component_spec": {
-      "type": "service",
-      "lifecycle": "production",
-      "owner": "team-payments"
-    }
-  }
-}' charybdis:50051 charybdis.entities.EntityService/CreateEntity
-```
-
-**What happens next:**
- Entity stored in PostgreSQL with a UUID
- `EntityCreated` event published to the event bus
- Plugins react (e.g., DefectDojo creates a product automatically)
- Entity available via gRPC and YAML adapter
-
-No YAML file to write. No PR to open. No manual provisioning.
-
-## Backstage Migration
-
-Already using Backstage? Charybdis provides a YAML adapter for gradual migration. Point Backstage at Charybdis as a catalog source — entities registered via gRPC are immediately available in Backstage.
-
-```yaml
-# backstage app-config.yaml
-catalog:
-  locations:
-    - type: url
-      target: http://charybdis:8080/yaml/locations
-```
-
---
-
-**Ready to get started?** Head to the [Getting Started Guide](getting-started.md).
@@ -67,17 +67,19 @@ export SECURITY_RBAC_ENABLED=true

 #### Development Certificates

-Use the provided test script:
+Use the provided dev-cert script:

 ```bash
-./test-mtls-rbac.sh
+./deploy/scripts/generate-dev-certs.sh
 ```

-This generates:
- `ca.pem` / `ca-key.pem` - Certificate Authority
- `server-cert.pem` / `server-key.pem` - Server certificate  
- `admin-cert.pem` / `admin-key.pem` - Admin client (OU=platform-team)
- Various plugin certificates (OU=plugins)
+This generates (under `deploy/certs/`):
+- `ca.pem` / `ca-key.pem` — Certificate Authority
+- `server-cert.pem` / `server-key.pem` — Server certificate
+- `admin-cert.pem` / `admin-key.pem` — Admin client (OU=platform-team)
+- Per-role client certs (OU=automation, OU=plugins)
+
+For an individual client cert without regenerating everything, use `./deploy/scripts/generate-client-cert.sh`.

 #### Production Certificates

@@ -494,10 +496,9 @@ Charybdis security features support compliance requirements:

 ## Next Steps

- 🔌 Configure [Plugins](plugins.md) with proper certificates
- 🚀 Review [Deployment Guide](deployment.md) for production
- 📊 Set up [Monitoring](monitoring.md) for security events
+- [Plugins](../plugins/README.md) — configure plugins to use mTLS client certs
+- [Architecture](architecture.md) — interceptor placement and observability hooks

 ---

-**Security Questions?** Open a [security issue](../../security) (for vulnerabilities, use private disclosure).
+**Security Questions?** Open an issue (for vulnerabilities, use private disclosure).
@@ -1,340 +1,513 @@
 # Charybdis Plugins

-This directory contains plugin implementations that extend Charybdis with integrations to external security and development tools.
+Plugins extend Charybdis with integrations to external tools (security scanners, issue trackers, identity providers). Plugins are **compile-time integrated**: they live as Rust crates under `plugins/`, declare protobuf extensions in `plugins.toml`, and are linked into the `charybdis-server` binary at build time.

-## What are Plugins?
+This document covers:
+- The plugin model (event-driven vs sync, traits, lifecycle)
+- Configuring shipped plugins (DefectDojo, Keycloak)
+- Writing a new plugin

-Charybdis supports **two types of plugins**, both compile-time integrated:
+## Table of Contents

-### **1. Event-Driven Plugins**
-React to entity lifecycle events (create, update, delete):
- **Example**: DefectDojo, DependencyTrack
- **Implement**: `EventDrivenPlugin` trait + `ResourceHandler`
- **Triggered by**: Entity CRUD operations
- **Use case**: Auto-create resources in external tools when entities are created
+1. [Plugin Types](#plugin-types)
+2. [Available Plugins](#available-plugins)
+3. [Plugin Configuration](#plugin-configuration)
+4. [Field Mapping System](#field-mapping-system)
+5. [Entity Resolution](#entity-resolution)
+6. [DefectDojo Reference](#defectdojo-reference)
+7. [Keycloak Reference](#keycloak-reference)
+8. [Writing a New Plugin](#writing-a-new-plugin)
+9. [Plugin Lifecycle](#plugin-lifecycle)
+10. [Annotations](#annotations)
+11. [Troubleshooting](#troubleshooting)

-### **2. Sync Plugins**
-Pull data from external sources on a schedule:
- **Example**: Okta, Keycloak, Active Directory
- **Implement**: `SyncPlugin` trait
- **Triggered by**: Cron schedule or manual API call
- **Use case**: Sync users/groups from identity providers
+## Plugin Types

-## Generic Utilities
+### Event-Driven Plugins
+React to entity lifecycle events (`Created`, `Updated`, `Deleted`) — typically push data to an external system.

-All plugins have access to reusable utilities in `src/plugins/`:
+- Implement `EventDrivenPlugin` + one or more `ResourceHandler` per resource kind.
+- Example: **DefectDojo** — Component creation → DefectDojo product + engagement.

- **`PluginHttpClient`** - Multi-auth HTTP client (Token, Bearer, API Key, Basic Auth)
- **`AnnotationHelper`** - Consistent API for storing/retrieving plugin metadata
- **`FieldMapper`** - Maps entity fields to external tool formats
- **`DateUtils`** - Common date/time operations
+### Sync Plugins
+Pull data from an external system on a schedule, creating or updating entities in Charybdis.

-## Plugin Structure
+- Implement `SyncPlugin` (`schedule`, `on_startup`, `manual_trigger`).
+- Example: **Keycloak** — periodic sync of users and groups.

-Each plugin follows this structure:
-
-```
-plugins/
-└── my_plugin/
-    ├── Cargo.toml              # Plugin crate definition
-    ├── README.md               # Plugin-specific documentation
-    ├── proto/
-    │   └── my_plugin.proto     # Protobuf entity definitions
-    └── src/
-        ├── lib.rs              # Plugin entry point
-        ├── handler.rs          # Event handler implementation
-        ├── client.rs           # External API client
-        └── config.rs           # Plugin configuration
-```
+Both types implement the base `Plugin` trait (`name`, `validate_config`, `health_check`).

 ## Available Plugins

-### DefectDojo
- **Type**: Event-Driven
- **Purpose**: Security vulnerability management integration
- **External API**: DefectDojo REST API v2
- **Handlers**: ProductHandler, EngagementHandler, ProductTypeHandler, ProductMemberHandler
- **Features**:
-  - Automatic product creation on Component/Service creation
-  - Product updates on entity changes
-  - Auto-create CI/CD engagements
-  - Owner resolution (Group → members → email annotation → DefectDojo user lookup)
-  - Configurable OIDC provider mapping (Keycloak, Okta, Azure AD)
-  - Annotation storage (`defectdojo.com/product-id`, `defectdojo.com/engagement-id`, `defectdojo.com/owner-member-ids`)
+| Plugin | Type | Status |
+|---|---|---|
+| DefectDojo | Event-driven | Shipped |
+| Keycloak | Sync | Shipped |

-### Keycloak
- **Type**: Sync
- **Purpose**: Identity provider sync (users and groups)
- **External API**: Keycloak Admin REST API
- **Features**:
-  - User sync with profile and annotations (display_name, email, picture)
-  - Group sync with hierarchy (parent, children, members)
-  - Configurable annotations (e.g., `keycloak.com/email` for OIDC mapping)
-  - On-startup sync, manual trigger via gRPC
-
-### DependencyTrack (Scaffolded)
- **Type**: Event-Driven
- **Purpose**: Software supply chain security
- **External API**: DependencyTrack REST API
- **Status**: Scaffolded (proto + crate structure), handler logic not implemented
- **Planned**:
-  - Auto-create projects for components
-  - SBOM ingestion
-
-## Creating a New Plugin
-
-### Quick Start
-
-1. **Create plugin directory structure:**
-   ```bash
-   mkdir -p plugins/my_plugin/{proto,src}
-   ```
-
-2. **Add to plugins.toml:**
-   ```toml
-   [plugins.my_plugin]
-   enabled = true
-   proto_path = "plugins/my_plugin/proto"
-   metadata_field_number = 102  # Use next available number
-   spec_field_number = 102
-   description = "My custom integration"
-   ```
-
-3. **Define protobuf schema:**
-   Create `plugins/my_plugin/proto/my_plugin.proto`:
-   ```protobuf
-   syntax = "proto3";
-   package charybdis.plugins.my_plugin;
-
-   message MyPluginMetadata {
-     string name = 1;
-     string description = 2;
-   }
-
-   message MyPluginSpec {
-     string integration_type = 1;
-     bool enabled = 2;
-   }
-   ```
-
-4. **Create plugin crate:**
-   Create `plugins/my_plugin/Cargo.toml`:
-   ```toml
-   [package]
-   name = "charybdis-plugin-my-plugin"
-   version = "0.1.0"
-   edition = "2021"
-
-   [dependencies]
-   charybdis = { path = "../.." }
-   async-trait = "0.1"
-   tokio = { version = "1.0", features = ["full"] }
-   tracing = "0.1"
-   ```
-
-5. **Implement event handler:**
-   Create `plugins/my_plugin/src/lib.rs`:
-   ```rust
-   use async_trait::async_trait;
-   use charybdis::events::{EventHandler, EntityEvent, EventResult};
-
-   pub struct MyPluginHandler;
-
-   #[async_trait]
-   impl EventHandler for MyPluginHandler {
-       async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
-           // React to entity changes
-           Ok(())
-       }
-   }
-   ```
-
-6. **Build and test:**
-   ```bash
-   cargo build
-   cargo test
-   ```
-
-## Plugin Field Number Allocation
-
-Field numbers must be unique across all plugins to avoid protobuf conflicts:
-
-| Range    | Allocation            |
-|----------|-----------------------|
-| 1-99     | Core entity types     |
-| 100      | DefectDojo            |
-| 101      | DependencyTrack       |
-| 102-199  | Available for plugins |
-
-When creating a new plugin, use the next available number in the 102+ range.
+Planned integrations (Slack, Jira, GitHub, Dependency-Track) are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).

 ## Plugin Configuration

-Plugins are configured in `config.toml` with `${VAR}` env var substitution for secrets:
+Plugins are configured in `config.toml` under `[plugins.<name>]`. Use `${VAR}` for secrets — never commit tokens.

 ```toml
 [plugins.defectdojo]
 enabled = true
-base_url = "${DEFECTDOJO_API_URL}"
+base_url = "${DEFECTDOJO_URL}"
 api_token = "${DEFECTDOJO_API_TOKEN}"

-[plugins.defectdojo.default_engagement]
-auto_create = true
-name = "CI/CD Pipeline"
-
 [plugins.keycloak]
 enabled = true
 base_url = "${KEYCLOAK_URL}"
-realm = "master"
+realm = "charybdis"
 client_id = "charybdis-sync"
 client_secret = "${KEYCLOAK_CLIENT_SECRET}"
 ```

-See `config.toml.example` for all options.
+When `enabled = false`, the plugin is loaded but does not register handlers and does not react to events. See `config.toml.example` for the full reference.
+
+## Field Mapping System
+
+Event-driven plugins (currently DefectDojo) use a configurable mapper to translate Charybdis entity fields into external-tool API payloads. Mappings are defined per resource type under `[plugins.<name>.field_mappings.<resource>]`.
+
+Three mapping types are supported:
+
+### 1. Direct field mapping (string)
+Dot-notation path into the entity.
+
+```toml
+name = "metadata.name"
+description = "metadata.description"
+email = "spec.profile.email"
+tags = "metadata.tags"
+```
+
+### 2. Static value (object with `value` key)
+
+```toml
+business_criticality = { value = "high" }
+is_active = { value = true }
+priority = { value = 100 }
+```
+
+Supports strings, booleans, numbers, objects, and arrays.
+
+### 3. Entity resolution (object)
+Resolve a reference into a related entity and extract a field from it.
+
+```toml
+product_manager = {
+  from = "spec.owner",
+  resolve_entity = "User",
+  extract = "annotations.defectdojo.com/user-id"
+}
+```
+
+| Parameter | Type | Required | Description |
+|---|---|---|---|
+| `from` | string | yes | Source field path |
+| `resolve_entity` | string | yes | Entity kind to look up (`User`, `Component`, ...) |
+| `extract` | string | yes | Field path to extract from the resolved entity |
+| `resolve_array` | bool | no | Source field is an array of references |
+| `lookup_entity` | string | no | Find an entity that references the source |
+
+## Entity Resolution
+
+Entity resolution lets a plugin navigate the entity graph at mapping time. The mapper performs a database lookup, extracts the target field, and substitutes it into the payload.
+
+```
+1. Source field            spec.owner = "user:john.doe"
+2. Resolve entity          User entity for "john.doe"
+3. Extract target field    annotations.defectdojo.com/user-id = "123"
+4. Result                  product_manager = 123
+```
+
+### Array resolution
+
+```toml
+user_ids = {
+  from = "spec.members",
+  resolve_entity = "User",
+  extract = "annotations.defectdojo.com/user-id",
+  resolve_array = true
+}
+```
+
+`Group.spec.members = ["user:john", "user:jane"]` → `user_ids = [123, 124]`.
+
+### Linked entity lookup
+
+```toml
+engagement_product = {
+  from = "id",
+  lookup_entity = "Component",
+  extract = "annotations.defectdojo.com/product-id"
+}
+```
+
+Finds a `Component` that references the source entity and extracts its annotation.
+
+### Error handling
+Resolution failures (entity not found, missing annotation, invalid reference) log a warning and omit the field rather than failing the whole operation.
+
+## DefectDojo Reference
+
+Maps Charybdis entities to DefectDojo resources via the REST API v2. Handlers shipped: `ProductHandler`, `EngagementHandler`, `ProductTypeHandler`, `ProductMemberHandler`.
+
+### Connection
+
+```toml
+[plugins.defectdojo]
+enabled = true
+base_url = "${DEFECTDOJO_URL}"
+api_token = "${DEFECTDOJO_API_TOKEN}"
+default_product_type_id = 1
+auto_create_users = true
+auto_create_product_types = false
+```
+
+To obtain the API token: in DefectDojo, **User Profile → API Key**.
+
+### Default engagement (auto-created with each product)
+
+```toml
+[plugins.defectdojo.default_engagement]
+auto_create = true
+name = "CI/CD Pipeline"
+description = "Automated security scans from CI/CD pipeline"
+engagement_type = "CI/CD"
+status = "In Progress"
+duration_days = 365
+deduplication_on_engagement = true
+```
+
+### Owner resolution
+When a `Component` has an `owner` (typically a team name), Charybdis resolves it to DefectDojo users:
+
+1. Find `Group` entity matching the owner.
+2. Read `Group.spec.members` (usernames).
+3. Look up each `User` entity.
+4. Extract the user's email from the configured annotation.
+5. Find the matching user in DefectDojo by email.
+
+```toml
+[plugins.defectdojo.owner_resolution]
+user_email_annotation = "keycloak.com/email"   # depends on your IdP
+defectdojo_lookup_field = "email"              # "email" or "username"
+assign_all_members = true                       # all group members, or just first match
+```
+
+### Product mapping (Component → DefectDojo Product)
+
+```toml
+[plugins.defectdojo.field_mappings.product]
+name = "metadata.name"
+description = "metadata.description"
+product_type_id = { value = 1 }
+tags = "metadata.tags"
+business_criticality = { value = "high" }       # very high | high | medium | low | very low | none
+platform = { value = "web" }                    # web | mobile | desktop | iot | ...
+lifecycle = { value = "production" }
+origin = { value = "internal" }
+external_audience = { value = true }
+internet_accessible = { value = true }
+
+product_manager = {
+  from = "spec.owner",
+  resolve_entity = "User",
+  extract = "annotations.defectdojo.com/user-id"
+}
+```
+
+### User mapping (User → DefectDojo User)
+
+```toml
+[plugins.defectdojo.field_mappings.user]
+username = "metadata.name"
+email = "spec.profile.email"
+first_name = "spec.profile.displayName"
+last_name = "spec.profile.displayName"
+is_active = { value = true }
+```
+
+### Product Type mapping (System → DefectDojo Product Type)
+
+```toml
+[plugins.defectdojo.field_mappings.product_type]
+name = "metadata.name"
+description = "metadata.description"
+critical_product = { value = false }
+key_product = { value = true }
+```
+
+### Product Member mapping (Group → DefectDojo Product Member)
+
+```toml
+[plugins.defectdojo.field_mappings.product_member]
+product_id = {
+  from = "spec.parent",
+  resolve_entity = "Component",
+  extract = "annotations.defectdojo.com/product-id"
+}
+
+user_id = {
+  from = "spec.members",
+  resolve_entity = "User",
+  extract = "annotations.defectdojo.com/user-id",
+  resolve_array = true
+}
+
+role_name = { value = "Reader" }    # Owner | Maintainer | Writer | Reader | API_Importer
+```
+
+### Engagement mapping (Resource → DefectDojo Engagement)
+
+```toml
+[plugins.defectdojo.field_mappings.engagement]
+name = "metadata.name"
+description = "metadata.description"
+
+product_id = {
+  from = "spec.owner",
+  resolve_entity = "Component",
+  extract = "annotations.defectdojo.com/product-id"
+}
+
+lead_id = {
+  from = "spec.dependsOn",
+  resolve_entity = "User",
+  extract = "annotations.defectdojo.com/user-id"
+}
+
+target_start = "spec.target_start"
+target_end = "spec.target_end"
+status = { value = "In Progress" }
+engagement_type = { value = "CI/CD" }
+version = "metadata.annotations.version"
+commit_hash = "metadata.annotations.commit_hash"
+branch_tag = "metadata.annotations.branch"
+build_id = "metadata.annotations.build_id"
+source_code_management_uri = "metadata.annotations.repo_url"
+
+deduplication_on_engagement = { value = true }
+api_test = { value = true }
+pen_test = { value = false }
+```
+
+### Annotations written by the plugin
+- `defectdojo.com/product-id`
+- `defectdojo.com/engagement-id`
+- `defectdojo.com/owner-member-ids`
+
+## Keycloak Reference
+
+Sync plugin: pulls users and groups from a Keycloak realm via the Admin REST API on a schedule, on startup, or via manual trigger.
+
+```toml
+[plugins.keycloak]
+enabled = true
+base_url = "${KEYCLOAK_URL}"
+realm = "charybdis"
+client_id = "charybdis-sync"
+client_secret = "${KEYCLOAK_CLIENT_SECRET}"
+
+[plugins.keycloak.sync]
+schedule = "0 */5 * * * *"   # optional cron; omit to sync only on startup/trigger
+on_startup = true
+manual_trigger = true
+sync_users = true
+sync_groups = true
+namespace = "keycloak"
+page_size = 100
+```
+
+The service account behind `client_id`/`client_secret` must have `view-users` and `view-groups` on the realm.
+
+Synced entities carry annotations such as `keycloak.com/email`, `keycloak.com/id`, used by other plugins (e.g., the DefectDojo owner-resolution chain above).
+
+## Writing a New Plugin
+
+### 1. Scaffold directory
+
+```bash
+mkdir -p plugins/my_plugin/{proto,src}
+```
+
+### 2. Register in `plugins.toml`
+
+```toml
+[plugins.my_plugin]
+enabled = true
+proto_path = "plugins/my_plugin/proto"
+metadata_field_number = 103   # next available — see registry below
+spec_field_number = 203
+description = "My custom integration"
+```
+
+### 3. Define the protobuf schema
+
+`plugins/my_plugin/proto/my_plugin.proto`:
+
+```protobuf
+syntax = "proto3";
+package charybdis.plugins.my_plugin;
+
+message MyPluginMetadata {
+  string name = 1;
+  string description = 2;
+}
+
+message MyPluginSpec {
+  string integration_type = 1;
+  bool enabled = 2;
+}
+```
+
+`build.rs` regenerates `proto/entities.proto` automatically on next build.
+
+### 4. Create the crate
+
+`plugins/my_plugin/Cargo.toml`:
+
+```toml
+[package]
+name = "charybdis-my-plugin-plugin"
+version = "0.1.0"
+edition = "2024"
+
+[lib]
+name = "charybdis_my_plugin"
+path = "src/lib.rs"
+
+[dependencies]
+charybdis = { path = "../.." }
+async-trait = "0.1"
+anyhow = "1.0"
+serde = { version = "1.0", features = ["derive"] }
+tokio = { version = "1.48", features = ["full"] }
+tracing = "0.1"
+```
+
+Add the crate to the workspace `members` in the top-level `Cargo.toml`.
+
+### 5. Implement the plugin
+
+Event-driven example:
+
+```rust
+use async_trait::async_trait;
+use charybdis::plugins::{EventDrivenPlugin, Plugin, PluginConfig, PluginType, ResourceHandler};
+use std::sync::Arc;
+
+pub struct MyPlugin { /* config, client, handlers */ }
+
+#[async_trait]
+impl Plugin for MyPlugin {
+    fn name(&self) -> &str { "my_plugin" }
+    fn plugin_type(&self) -> PluginType { PluginType::EventDriven }
+    fn load_config(&mut self, _: PluginConfig) -> anyhow::Result<()> { Ok(()) }
+    fn validate_config(&self) -> anyhow::Result<()> { Ok(()) }
+}
+
+#[async_trait]
+impl EventDrivenPlugin for MyPlugin {
+    fn resource_handlers(&self) -> Vec<Arc<dyn ResourceHandler>> {
+        vec![/* your handlers */]
+    }
+}
+```
+
+### 6. Register in `charybdis-server`
+
+Add a loader block in `charybdis-server/src/main.rs` (follow the DefectDojo block as a template) and declare the dependency in `charybdis-server/Cargo.toml`.
+
+### Field number registry
+
+| Range | Allocation |
+|---|---|
+| 1–99 | Core entity types |
+| 100 | DefectDojo |
+| 101 | DependencyTrack |
+| 102 | Keycloak |
+| 103+ | Available |
+
+Field numbers MUST be unique across plugins — protobuf wire format depends on it.

 ## Plugin Lifecycle

-1. **Build Time:**
-   - `build.rs` reads `plugins.toml`
-   - Generates `proto/entities.proto` with plugin types
-   - Compiles all protobuf files
-   - Generates Rust code
+### Build time
+1. `build.rs` reads `plugins.toml`.
+2. Generates `proto/entities.proto` with plugin-contributed `oneof` variants.
+3. Compiles all protobuf files (`tonic-prost-build`).

-2. **Runtime:**
-   - Plugin handlers registered with event bus
-   - Events published on entity CRUD operations
-   - Handlers react asynchronously
-   - External APIs called as needed
+### Runtime
+1. `charybdis-server` loads each enabled plugin from `config.toml`.
+2. Event-driven plugins register their `ResourceHandler`s with the dispatcher.
+3. Sync plugins register with the cron scheduler (and optionally run once at startup).
+4. Entity CRUD operations publish `EntityEvent` to the in-memory event bus.
+5. The dispatcher fans out events to matching handlers (filtered by `trigger_kinds`).
+6. Handlers call external APIs and write annotations back to Charybdis.

-3. **Event Flow:**
-   ```
-   Client creates entity
-   → Entity stored in database
-   → Event published to event bus
-   → Plugin handler receives event
-   → Plugin calls external API
-   → Plugin stores external ID in annotations
-   ```
-
-## Using Annotations
-
-Plugins store external tool IDs in entity annotations:
-
-```rust
-// Store external ID
-entity.annotations.insert(
-    "my-plugin.com/resource-id".to_string(),
-    "ext-12345".to_string(),
-);
-
-// Query by external ID in SQL
-SELECT * FROM entities 
-WHERE annotations->>'my-plugin.com/resource-id' = 'ext-12345';
+### Event flow
+```
+Client → gRPC CreateEntity → DB INSERT → EntityEvent::Created → Dispatcher
+                                                                  ↓
+                                          ResourceHandler.handle_create(entity)
+                                                                  ↓
+                                          POST /api/products → DefectDojo
+                                                                  ↓
+                                          update_annotations("defectdojo.com/product-id")
 ```

-### Annotation Naming Convention
+## Annotations
+
+Plugins store external IDs in entity annotations using reverse-DNS keys:

-Use reverse-DNS style:
 - `defectdojo.com/product-id`
- `dependencytrack.com/project-uuid`
- `github.com/repo-slug`
- `{tool}.com/{resource}-{attribute}`
+- `defectdojo.com/engagement-id`
+- `keycloak.com/id`
+- `keycloak.com/email`

-## Best Practices
+Query by annotation via JSONB in PostgreSQL:

-### 1. Error Handling
- Don't panic - return errors
- Log but continue on non-critical failures
- Implement retry logic for transient errors
-
-### 2. Idempotency
- Check if resource exists before creating
- Make operations safe to retry
- Handle duplicate creation gracefully
-
-### 3. Performance
- Don't block event handlers
- Use `tokio::spawn` for long operations
- Batch operations when possible
-
-### 4. Testing
- Unit test event handlers
- Mock external API clients
- Integration tests with real APIs (optional)
-
-### 5. Documentation
- Document required environment variables
- Provide example configurations
- Explain entity model and annotations
-
-## Contributing Plugins
-
-We welcome plugin contributions! To contribute:
-
-1. Fork the repository
-2. Create your plugin following the structure above
-3. Add comprehensive tests
-4. Document configuration and usage
-5. Submit a pull request
-
-### Plugin Requirements
-
- [ ] Protobuf definitions with Metadata and Spec messages
- [ ] Event handler implementation
- [ ] External API client (if applicable)
- [ ] Configuration via environment variables
- [ ] README with setup instructions
- [ ] Unit tests for event handler
- [ ] Example usage in documentation
-
-## Plugin Distribution Models
-
-### In-Tree (Current)
-Plugins live in the `plugins/` directory and are enabled via `plugins.toml`.
-
-**Pros:**
- Easy to discover
- Consistent quality
- Tested together
-
-**Cons:**
- Requires core repo access
- All plugins built together
-
-### External Crates (Future)
-Plugins distributed as separate Rust crates.
-
-**Example:**
-```toml
-[dependencies]
-charybdis-plugin-custom = "0.1"
+```sql
+SELECT * FROM entities
+WHERE annotations->>'defectdojo.com/product-id' = '456';
 ```

-**Pros:**
- Independent versioning
- Community contributions
- Optional dependencies
+Use `EntityRepository::update_annotations()` for atomic JSONB merge — it avoids the read-modify-write race when multiple handlers write to the same entity.

-**Cons:**
- Discovery harder
- Compatibility challenges
+## Troubleshooting

-### Plugin Marketplace (Future)
-Central registry of available plugins (like Backstage).
+### Plugin not reacting to events
+- `enabled = true` in `config.toml`?
+- Plugin registered in `charybdis-server/src/main.rs`?
+- Entity kind matches the handler's `trigger_kinds`?
+- Event bus running? (`info!("Event bus started")` in startup logs)

-## Resources
+### Field mapping not applied
+- Path uses dot notation, e.g. `metadata.name`, not `meta.name`.
+- Static value uses `{ value = "..." }`, not a bare string when you want a literal.
+- Field actually exists on the entity (inspect via the YAML adapter or `GetEntity` gRPC).

- [Plugin Configuration Guide](../docs/PLUGIN_CONFIGURATION_GUIDE.md)
- [Core Concepts](../docs/core-concepts.md)
- [Architecture](../docs/architecture.md)
- [Protobuf Style Guide](https://protobuf.dev/programming-guides/style/)
+### Entity resolution returns nothing
+- Resolved entity exists in the DB.
+- Resolved entity has the `extract` annotation set.
+- `resolve_entity` uses the PascalCase kind (`"User"`, not `"user"`).
+- For arrays, `resolve_array = true` is set.

-## Support
+### DefectDojo API errors
+- `400 Bad Request: Invalid product type ID` → check `default_product_type_id` or your `product_type_id` mapping.
+- `401 Unauthorized` → check `DEFECTDOJO_API_TOKEN`.
+- `409 Conflict: user already exists` → expected, the plugin reuses existing users.

- Open an issue for bug reports
- Discussions for questions
- PRs for contributions
+### Performance
+- Minimize entity-resolution chain depth.
+- Prefer static values when the data isn't entity-bound.
+- Check network latency to the external API — handlers run synchronously per event.

-## License
+## See Also

-Same as Charybdis core (see LICENSE file in repository root)
+- [Core Concepts](../docs/core-concepts.md) — entity model, events, annotations.
+- [Architecture](../docs/architecture.md) — protobuf schema, storage, scaling.
+- [Field Mapper Tests](../src/plugins/field_mapper.rs) — reference behavior.