diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dd3c28a..6274831 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,7 +14,7 @@ Contributions are welcome! Whether it's a bug report, feature idea, documentatio ```bash # Clone the repo -git clone https://github.com/YOUR-ORG/charybdis.git +git clone cd charybdis # Copy config diff --git a/README.md b/README.md index 11e2cfe..009770a 100644 --- a/README.md +++ b/README.md @@ -1,41 +1,26 @@ # Charybdis -**The security-native platform engineering tool.** +**A security-native platform engineering tool.** -One platform for your software catalog, vulnerability management, and compliance posture. Event-driven. Single binary. No YAML files to maintain. +One Rust binary for your software catalog. Event-driven, gRPC-native, no YAML to maintain. Designed to grow into a unified catalog + vulnerability management + compliance platform. -## Why Charybdis? +## What Works Today -Platform engineers today run Backstage for the catalog, DefectDojo for vulnerabilities, Dependency-Track for SBOMs, a license scanner, and a compliance spreadsheet. Five tools, five sources of truth, none of them talking to each other. +- **Software catalog** — gRPC API for entity CRUD (Component, System, API, User, Group, Domain, Resource, Finding). Real-time event bus on every change. +- **Vulnerability ingestion** — gRPC `IngestionService` ingests SARIF reports. Findings are deduplicated, reconciled against existing state (new / unchanged / resolved / reopened), and dry-runnable for MR-level diffs. +- **Backstage YAML adapter** — HTTP endpoint exposing entities as Backstage-compatible Location YAML, for migration or coexistence. +- **mTLS + RBAC** — Certificate-based auth with fine-grained, OU-mapped roles. +- **OpenTelemetry** — Traces, metrics, and logs out of the box (console + OTLP). +- **Plugin system** — Event-driven (DefectDojo) and sync (Keycloak) plugins. Compile-time integrated. +- **PostgreSQL storage** — Zero-migration: entities stored as protobuf blobs in a fixed schema. New plugins add `oneof` variants, never columns. -Charybdis unifies this: - -- **Software Catalog** — Event-driven, gRPC-native. Services register from CI/CD or IaC. No static YAML, no polling, always accurate. -- **Vulnerability Management** — Ingest scan results natively (SARIF, CycloneDX, SPDX). Triage, assess, track. No external vuln tool needed. -- **Compliance & Licenses** — Security gates, license policies, compliance framework mappings. Built-in, not bolted on. - -All of this in a **single Rust binary** that uses ~50MB of RAM. - -## How It Works - -``` -CI/CD Pipeline ──gRPC──> Charybdis -Scanner results ───────> ├── Catalogs the service (event-driven) -IaC tools ─────────────> ├── Ingests vulnerabilities natively - ├── Evaluates security gates - └── Fires events to plugins (Slack, Jira, ...) -``` - -**One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.** +For everything else (assessment workflows, security gates, license policies, compliance frameworks, more parsers, more plugins), see [VISION.md](VISION.md) and [TODO.md](TODO.md). ## Quick Start ```bash # 1. Start PostgreSQL -docker run -d \ - -e POSTGRES_PASSWORD=mysecretpassword \ - -p 5432:5432 \ - postgres:15 +docker run -d -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 postgres:15 # 2. Configure cp config.toml.example config.toml @@ -62,187 +47,66 @@ grpcurl -plaintext -d '{ }' localhost:50051 charybdis.entities.EntityService/CreateEntity ``` -The entity is stored, events are fired, and plugins react automatically. +Detailed walkthrough: [docs/getting-started.md](docs/getting-started.md). -## Key Features - -### Software Catalog -- **gRPC API** for programmatic entity management from CI/CD, scripts, or any tool -- **Event-driven plugins** auto-provision external tools when entities change -- **Zero-migration storage** — PostgreSQL with protobuf + JSONB. No schema changes, ever -- **Rich entity model** — Component, System, API, User, Group, Domain, Resource -- **Backstage compatible** — Built-in YAML adapter for migration or coexistence - -### Security (Native — Phase 1) -Security is a first-class concept in Charybdis, not a plugin. These features are under active development: -- **Vulnerability ingestion** — Push scan results via gRPC (SARIF, CycloneDX, SPDX) -- **Assessment workflow** — Triage, accept risk, remediate, auto-assess via rules -- **Security gates** — Severity thresholds per product, block deployments on violations -- **License compliance** — Track licenses, enforce policies, flag violations -- **Compliance frameworks** — Map vulnerabilities to NIS2, SOC2, DORA requirements - -### Platform -- **mTLS + RBAC** — Certificate-based auth with fine-grained permissions -- **OpenTelemetry** — Full observability (traces, metrics, logs) out of the box -- **Single binary** — Deploy one Rust binary + PostgreSQL. That's it. -- **Sub-millisecond latency** — Tested at 170k+ entities - -## Architecture - -```mermaid -graph TB - subgraph "Sources" - A1[CI/CD Pipelines] - A2[IaC Tools] - A3[Security Scanners] - end - - subgraph "Charybdis Core" - B1[gRPC API] - B2[Software Catalog] - B3[Vuln Management] - B4[Security Gates] - B5[Event Bus] - end - - subgraph "Integrations - Plugins" - C1[Slack / Teams] - C2[Jira / GitHub Issues] - C3[Custom Plugins] - end - - subgraph "Frontend" - D2[Backstage - optional] - end - - A1 -->|gRPC| B1 - A2 -->|gRPC| B1 - A3 -->|Scan Results| B1 - B1 --> B2 - B1 --> B3 - B3 --> B4 - B2 --> B5 - B3 --> B5 - B5 --> C1 - B5 --> C2 - B5 --> C3 - B2 -.->|YAML Adapter| D2 -``` - -## Entity Model - -| Kind | Description | Example | -|------|-------------|---------| -| **Service** | Individual microservices or applications | `payment-api`, `auth-service` | -| **Component** | Reusable libraries, SDKs, modules | `auth-sdk`, `logging-lib` | -| **System** | Collections of components working together | `payment-system` | -| **API** | Interfaces exposed by components | `payments-rest-api` | -| **User** | Individual people | `john.doe` | -| **Group** | Teams and organizational units | `team-payments` | -| **Domain** | Business domains | `payments`, `shipping` | -| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` | - -Each entity will carry its security posture natively once Phase 1 is complete: vulnerabilities, license status, compliance state, and assessment history. - -## Plugin System - -Core features (catalog, vulns, compliance) are **native**. Plugins handle **integrations** with external systems: +## How It Works ``` -EntityCreated ──> Event Bus ──> DefectDojo Plugin ──> Creates product - ──> Slack Plugin ──> Notifies channel (planned) - ──> Custom Plugin ──> Your logic +CI/CD Pipeline ──gRPC──> Charybdis +Scanner output ────────> ├── Catalogs the entity + ├── Reconciles vulnerabilities (SARIF) + └── Fires events to plugins (DefectDojo, ...) ``` -Two plugin types: -- **Event-Driven** — React to entity/vulnerability events in real-time -- **Sync** — Pull data from external sources on a schedule (e.g., sync users from Keycloak) +See [docs/architecture.md](docs/architecture.md) for the full design (protobuf schema, storage model, event bus, plugin lifecycle). -**Current plugins:** -- DefectDojo — Auto-create products, engagements, and assign owners (Done) -- Keycloak — Sync users and groups with annotations (Done) -- Dependency-Track — Scaffolded -- Jira, Slack, GitHub — Planned +## Entity Kinds -Build your own with the `EventDrivenPlugin` or `SyncPlugin` traits. +| Kind | Purpose | +|---|---| +| **Component** | Services, libraries, applications | +| **System** | Collections of components | +| **API** | Interfaces exposed by components | +| **User**, **Group** | People and teams | +| **Domain** | Business domains | +| **Resource** | Infrastructure resources | +| **Finding** | Security findings ingested from scanners | -## Backstage Compatibility +Conceptual details: [docs/core-concepts.md](docs/core-concepts.md). -Already using Backstage? Charybdis works as a **drop-in dynamic backend**. Point Backstage at Charybdis's YAML adapter and stop maintaining `catalog-info.yaml` files: +## Plugins -```yaml -# backstage app-config.yaml -catalog: - locations: - - type: url - target: http://charybdis:8080/yaml/locations - rules: - - allow: [Component, System, Service, API, User, Group] -``` +Two plugin types, both compile-time integrated: -Or use Charybdis standalone via its gRPC API — no Backstage needed. +- **Event-driven** — React to entity lifecycle events. *Shipped: DefectDojo (auto-create products, engagements, member assignment).* +- **Sync** — Pull data from external systems on a schedule. *Shipped: Keycloak (users + groups).* -## vs. Alternatives - -| | Charybdis | Backstage | Port / Cortex / OpsLevel | -|---|---|---|---| -| **Type** | Open source | Open source | Commercial SaaS | -| **Catalog** | Event-driven, real-time | Static YAML, polling | Varies | -| **Security** | Native (first-class) | Plugins (fragmented) | Limited / add-on | -| **Deployment** | Single binary + PG | Node.js cluster + PG + plugins | Hosted | -| **Performance** | ~50MB RAM, sub-ms | ~1GB+ RAM | N/A | -| **Compliance** | Native frameworks | Manual | Some | -| **Cost** | Free | Free (+ operational cost) | $$$$ | - -## Project Status - -| Module | Status | Phase | -|--------|--------|-------| -| Core gRPC API | Done | 0 | -| PostgreSQL storage | Done | 0 | -| Event bus system | Done | 0 | -| Plugin framework | Done | 0 | -| mTLS + RBAC | Done | 0 | -| OpenTelemetry | Done | 0 | -| Backstage YAML adapter | Done | 0 | -| DefectDojo plugin | Done | 0 | -| Keycloak plugin | Done | 0 | -| Vulnerability ingestion | Planned | 1 | -| SARIF / CycloneDX / SPDX parsers | Planned | 1 | -| Security gates & rules engine | Planned | 1 | -| Compliance frameworks | Planned | 3 | - -See [VISION.md](VISION.md) for the full roadmap. +Plugin model, configuration, and writing your own: [plugins/README.md](plugins/README.md). ## Documentation -- [Vision & Roadmap](VISION.md) — Where Charybdis is going -- [Getting Started](docs/getting-started.md) — Installation and first steps -- [Core Concepts](docs/core-concepts.md) — Entities, events, and architecture +- [Getting Started](docs/getting-started.md) — installation and first entity +- [Core Concepts](docs/core-concepts.md) — entities, events, annotations +- [Architecture](docs/architecture.md) — protobuf schema, storage, event bus - [Security](docs/security.md) — mTLS and RBAC configuration -- [Architecture](docs/architecture.md) — Deep dive into design decisions -- [Plugin Guide](plugins/README.md) — Building and using plugins -- [Demo](deploy/DEMO.md) — Full stack demo with DefectDojo +- [Plugins](plugins/README.md) — DefectDojo, Keycloak, building your own +- [Demo](deploy/DEMO.md) — full stack demo with DefectDojo +- [Vision & Roadmap](VISION.md) — where Charybdis is going ## Technology Stack | | | |---|---| -| **Language** | Rust | -| **API** | gRPC + Protocol Buffers | -| **Database** | PostgreSQL 14+ (protobuf + JSONB, zero-migration) | -| **Frontend** | Backstage YAML adapter (compatible) | -| **Security** | mTLS (rustls) + RBAC | -| **Observability** | OpenTelemetry (traces, metrics, logs) | +| Language | Rust | +| API | gRPC + Protocol Buffers | +| Database | PostgreSQL 14+ (protobuf + JSONB, zero-migration) | +| TLS | rustls (no OpenSSL dependency) | +| Observability | OpenTelemetry (traces, metrics, logs) | ## Contributing -Contributions are welcome! Whether it's a new plugin, a security parser, or documentation improvements — we'd love your help. +See [CONTRIBUTING.md](CONTRIBUTING.md). ## License -[Apache-2.0](LICENSE) - ---- - -**One platform. Catalog. Security. Compliance. Built in Rust.** +[Apache-2.0](LICENSE.md) diff --git a/TODO.md b/TODO.md index 52ffc92..cb00f26 100644 --- a/TODO.md +++ b/TODO.md @@ -1,7 +1,7 @@ # Charybdis - TODO List -**Last Updated**: 2026-05-06 -**Status**: Phase 0 done. Phase 1 (Security Core) in progress — assessment & gates remaining. +**Last Updated**: 2026-06-09 +**Status**: Phase 0 done. Phase 1 (Security Core) in progress — ingestion + reconciliation shipped; assessment & gates remaining. > Aligned with [VISION.md](VISION.md) roadmap. @@ -12,12 +12,21 @@ > **Goal**: Native vulnerability management and scan ingestion. Replace DefectDojo for finding lifecycle management. > **Architecture**: Core features (reconciliation, dedup, dry-run) in `src/`. Parsers extensible via `ScannerParser` trait. Plugins contribute parsers via `contributed_parsers()`. +### Done + +- [x] `Finding` entity kind (proto + storage) +- [x] `IngestionService` gRPC endpoint (`ImportScan` + `DryRunScan`) +- [x] `ScannerParser` trait + `ParserRegistry` +- [x] SARIF parser (built-in) +- [x] Fingerprint-based deduplication (scanner-provided when present, sha256 fallback) +- [x] Reconciliation engine (new / unchanged / resolved / reopened buckets, scoped to `(component, lifecycle)`) + ### Remaining - [ ] CycloneDX VEX parser (vulnerability data from SBOMs) - [ ] `Plugin::contributed_parsers()` default impl on base trait -- [ ] Publish events: FindingCreated, FindingResolved, FindingReopened (for downstream plugins) -- [ ] Assessment workflow (triage, accept risk, remediate) +- [ ] Publish events: `FindingCreated`, `FindingResolved`, `FindingReopened` (for downstream plugins) +- [ ] Assessment workflow (triage, accept risk, remediate — the `ACCEPTED` / `FALSE_POSITIVE` state values exist but no API sets them) - [ ] Rules engine for auto-assessment - [ ] Security gates (severity thresholds per product) - [ ] License tracking and policy engine diff --git a/deploy/DEMO.md b/deploy/DEMO.md index 515f41d..a71d2a3 100644 --- a/deploy/DEMO.md +++ b/deploy/DEMO.md @@ -288,9 +288,8 @@ Backstage will discover all entities from Charybdis automatically. See the `dock ## Support -- GitHub Issues: [github.com/charybdis-catalog/charybdis/issues](https://github.com/charybdis-catalog/charybdis/issues) -- Documentation: [docs/](docs/) -- Discord: [Join our community](https://discord.gg/...) +- Documentation: [../docs/](../docs/) +- Open an issue on the project's Gitea/GitHub repository --- diff --git a/docs/PLUGIN_CONFIGURATION_GUIDE.md b/docs/PLUGIN_CONFIGURATION_GUIDE.md deleted file mode 100644 index 89cdf1f..0000000 --- a/docs/PLUGIN_CONFIGURATION_GUIDE.md +++ /dev/null @@ -1,1009 +0,0 @@ -# Charybdis Plugin Configuration Guide - -Complete reference for configuring and using the Charybdis plugin system. - -## Table of Contents - -1. [Overview](#overview) -2. [Plugin System Architecture](#plugin-system-architecture) -3. [Configuration File Structure](#configuration-file-structure) -4. [Field Mapping System](#field-mapping-system) -5. [Entity Resolution](#entity-resolution) -6. [Plugin Configuration Reference](#plugin-configuration-reference) -7. [Environment Variables](#environment-variables) -8. [Best Practices](#best-practices) -9. [Troubleshooting](#troubleshooting) -10. [Examples](#examples) - -## Overview - -Charybdis plugins extend the core platform with integrations to external tools and services. Plugins react to entity lifecycle events (Create, Update, Delete) and automatically provision or sync resources in external systems. - -### Plugin Types - -**Event-Driven Plugins** (Current Implementation): -- React to entity lifecycle events -- Push data to external systems -- One-way sync: Charybdis → External Tool - -**Sync Plugins** (Future): -- Pull data from external systems on schedule -- Create/update entities in Charybdis -- One-way sync: External Tool → Charybdis - -## Plugin System Architecture - -``` -┌─────────────────────────────────────────┐ -│ Charybdis Core │ -│ ┌────────────────────────────────┐ │ -│ │ Entity Repository │ │ -│ │ (PostgreSQL) │ │ -│ └──────────┬─────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌────────────────────────────────┐ │ -│ │ Event Bus │ │ -│ │ (In-Memory / Future: Redis) │ │ -│ └──────────┬─────────────────────┘ │ -│ │ EntityEvent │ -│ │ (Created/Updated/Deleted) │ -└─────────────┼────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────┐ -│ Event Dispatcher │ -│ (Routes events to plugins) │ -└──────────┬──────────────────────────────┘ - │ - ├─────────────────┐ - ▼ ▼ -┌──────────────────┐ ┌──────────────────┐ -│ DefectDojo │ │ Future Plugins │ -│ Plugin │ │ (DependencyTrack)│ -│ │ │ │ -│ ┌──────────────┐ │ └──────────────────┘ -│ │ProductHandler│ │ -│ │UserHandler │ │ -│ │... │ │ -│ └──────────────┘ │ -└─────────┬────────┘ - │ HTTP/REST - ▼ -┌──────────────────┐ -│ External Tool │ -│ (DefectDojo) │ -└──────────────────┘ -``` - -### Key Components - -1. **Plugin**: Top-level plugin interface -2. **ResourceHandler**: Handles specific resource types (e.g., Products, Users) -3. **FieldMapper**: Maps Charybdis entity fields to external tool fields -4. **Event Dispatcher**: Routes entity events to appropriate handlers -5. **Plugin Manager**: Manages plugin lifecycle and registration - -## Configuration File Structure - -Plugin configuration lives in your main `config.toml` file under the `[plugins.]` section. - -### Basic Structure - -```toml -[plugins.] -enabled = true # Enable/disable plugin -base_url = "${API_URL}" # External tool API URL -api_token = "${API_TOKEN}" # Authentication token - # Plugin-specific settings - -[plugins..field_mappings.] - = # Field mappings per resource type -``` - -### Configuration Sections - -#### 1. Plugin Enable/Disable - -```toml -[plugins.defectdojo] -enabled = true # or false to disable -``` - -When disabled, the plugin: -- Will NOT be compiled into the binary -- Will NOT react to entity events -- Will NOT appear in plugin registry - -#### 2. Connection Settings - -```toml -[plugins.defectdojo] -base_url = "${DEFECTDOJO_URL}" -api_token = "${DEFECTDOJO_API_TOKEN}" -``` - -**Security Best Practice**: Always use environment variables for sensitive data (tokens, passwords, secrets). - -#### 3. Plugin-Specific Options - -Each plugin may have unique configuration options: - -```toml -[plugins.defectdojo] -default_product_type_id = 1 # Default product type for new products -auto_create_users = true # Auto-create missing users -auto_create_product_types = false # Require manual product type creation -``` - -#### 4. Field Mappings - -Field mappings define how Charybdis entity fields map to external tool API fields. - -```toml -[plugins.defectdojo.field_mappings.product] -name = "metadata.name" # Direct field mapping -description = "metadata.description" # Direct field mapping -business_criticality = { value = "high" } # Static value -tags = "metadata.tags" # Array mapping -``` - -## Field Mapping System - -### Mapping Types - -The field mapping system supports three types of mappings: - -#### 1. Direct Field Mapping (String) - -Maps a field from the Charybdis entity using dot notation: - -```toml -name = "metadata.name" -description = "metadata.description" -email = "spec.profile.email" -tags = "metadata.tags" -``` - -**Syntax**: `""` - -**Examples**: -- `"metadata.name"` → `entity.metadata.name` -- `"spec.owner"` → `entity.spec.owner` -- `"metadata.annotations.version"` → `entity.metadata.annotations["version"]` - -#### 2. Static Value Mapping (Object with `value` key) - -Provides a static value regardless of entity content: - -```toml -business_criticality = { value = "high" } -platform = { value = "web" } -is_active = { value = true } -priority = { value = 100 } -lifecycle = { value = "production" } -``` - -**Syntax**: `{ value = }` - -**Supported Types**: -- String: `{ value = "text" }` -- Boolean: `{ value = true }` -- Number: `{ value = 42 }` -- Object: `{ value = { key = "value" } }` -- Array: `{ value = ["item1", "item2"] }` - -#### 3. Complex Mapping with Entity Resolution (Object) - -Resolves entity references and extracts data from related entities: - -```toml -product_manager = { - from = "spec.owner", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} -``` - -**Syntax**: -```toml - = { - from = "", # Source field path - resolve_entity = "", # Entity kind to resolve to - extract = "", # Field to extract from resolved entity - resolve_array = , # Optional: resolve array of entities - lookup_entity = "" # Optional: lookup linked entity -} -``` - -**Parameters**: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `from` | string | ✓ | Source field path in dot notation | -| `resolve_entity` | string | ✓ | Entity kind to resolve (User, Component, etc.) | -| `extract` | string | ✓ | Field path to extract from resolved entity | -| `resolve_array` | boolean | ✗ | Set to true if `from` contains array of entity IDs | -| `lookup_entity` | string | ✗ | Find linked entity by source ID | - -## Entity Resolution - -Entity resolution is the most powerful feature of the field mapping system. It allows plugins to navigate entity relationships and extract data from related entities. - -### How Entity Resolution Works - -``` -1. Start with source entity (e.g., Component) - entity.spec.owner = "user:john.doe" - -2. Extract source value - from = "spec.owner" → "user:john.doe" - -3. Resolve entity reference - resolve_entity = "User" → Query for User entity with id "john.doe" - -4. Extract target field - extract = "annotations.defectdojo.com/user-id" → "123" - -5. Result - product_manager = 123 -``` - -### Single Entity Resolution - -Resolve one entity reference: - -```toml -[plugins.defectdojo.field_mappings.product] -product_manager = { - from = "spec.owner", # Component.spec.owner = "user:john.doe" - resolve_entity = "User", # Find User entity - extract = "annotations.defectdojo.com/user-id" # Get DD user ID → "123" -} -``` - -### Array Entity Resolution - -Resolve multiple entity references: - -```toml -[plugins.defectdojo.field_mappings.product_member] -user_ids = { - from = "spec.members", # Group.spec.members = ["user:john", "user:jane"] - resolve_entity = "User", # Find each User entity - extract = "annotations.defectdojo.com/user-id", # Get DD user ID for each - resolve_array = true # Process as array -} -``` - -**Result**: `user_ids = [123, 124]` - -### Linked Entity Lookup - -Find an entity linked to the source entity: - -```toml -engagement_product = { - from = "id", # Start with engagement entity ID - lookup_entity = "Component", # Find Component where... - extract = "annotations.defectdojo.com/product-id" # Component links to this engagement -} -``` - -This searches for a Component entity that references the source entity. - -### Nested Entity Resolution - -Chain multiple resolutions: - -```toml -team_lead_email = { - from = "spec.owner", # Component.spec.owner = "group:platform-team" - resolve_entity = "Group", # Find Group entity - extract = "spec.parent" # Get parent = "user:tech-lead" -} - -# Then in a second mapping: -lead_id = { - from = "spec.parent", # From previous resolution - resolve_entity = "User", # Find User entity - extract = "annotations.defectdojo.com/user-id" -} -``` - -### Entity Resolution Error Handling - -The field mapper handles errors gracefully: - -- **Entity not found**: Logs warning, field is omitted -- **Annotation missing**: Logs warning, field is omitted -- **Invalid entity reference**: Logs warning, field is omitted -- **Array with missing entities**: Logs warning, includes found entities only - -**In `map_all()` mode**: Failed field mappings log warnings but don't fail entire operation. - -## Plugin Configuration Reference - -### DefectDojo Plugin - -Complete configuration reference for the DefectDojo plugin. - -#### Connection Settings - -```toml -[plugins.defectdojo] -enabled = true -base_url = "${DEFECTDOJO_URL}" -api_token = "${DEFECTDOJO_API_TOKEN}" -``` - -#### Plugin Options - -```toml -[plugins.defectdojo] -# Default product type ID for new products (required if not mapped) -default_product_type_id = 1 - -# Automatically create users in DefectDojo when referenced -# If false, referenced users must exist in DefectDojo -auto_create_users = true - -# Automatically create product types in DefectDojo -# If false, product types must be created manually -auto_create_product_types = false -``` - -#### Resource Type: Product - -Maps **Component** entities to DefectDojo Products. - -```toml -[plugins.defectdojo.field_mappings.product] -# Required fields -name = "metadata.name" # Product name -description = "metadata.description" # Product description -product_type_id = { value = 1 } # Product type ID - -# Optional fields -tags = "metadata.tags" # Product tags (array) -business_criticality = { value = "high" } # very high, high, medium, low, very low, none -platform = { value = "web" } # web, mobile, desktop, iot, etc. -lifecycle = { value = "production" } # production, development, retirement -origin = { value = "internal" } # internal, external, third party -user_records = { value = 1000000 } # Number of user records -revenue = { value = "1000000" } # Revenue amount -external_audience = { value = true } # Accessible to external users -internet_accessible = { value = true } # Accessible via internet - -# User references (with entity resolution) -product_manager = { - from = "spec.owner", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} - -technical_contact = { - from = "metadata.annotations.technical-contact", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} - -team_manager = { - from = "metadata.annotations.team-manager", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} -``` - -#### Resource Type: User - -Maps **User** entities to DefectDojo Users. - -```toml -[plugins.defectdojo.field_mappings.user] -# Required fields -username = "metadata.name" # Username (unique) -email = "spec.profile.email" # Email address (unique) - -# Optional fields -first_name = "spec.profile.displayName" # First name -last_name = "spec.profile.displayName" # Last name (can use same field) -is_active = { value = true } # User active status -``` - -#### Resource Type: Product Type - -Maps **System** entities to DefectDojo Product Types. - -```toml -[plugins.defectdojo.field_mappings.product_type] -# Required fields -name = "metadata.name" # Product type name - -# Optional fields -description = "metadata.description" # Product type description -critical_product = { value = false } # Requires extra review -key_product = { value = true } # Important but not critical -``` - -#### Resource Type: Product Member - -Maps **Group** entities to DefectDojo Product Members (user-product-role assignments). - -**Note**: Only Group entities with specific structure are processed. - -```toml -[plugins.defectdojo.field_mappings.product_member] -# Product reference (resolve from Group parent) -product_id = { - from = "spec.parent", # Group.spec.parent = "component:my-service" - resolve_entity = "Component", # Find Component entity - extract = "annotations.defectdojo.com/product-id" # Get DD product ID -} - -# User reference (resolve from Group members) -user_id = { - from = "spec.members", # Group.spec.members = ["user:john", "user:jane"] - resolve_entity = "User", # Find User entities - extract = "annotations.defectdojo.com/user-id", # Get DD user IDs - resolve_array = true # Process as array -} - -# Role assignment -role_name = { value = "Reader" } # Owner, Maintainer, Reader, Writer, API_Importer -``` - -**DefectDojo Roles**: -- **Owner**: Full control over product -- **Maintainer**: Can modify product settings -- **Writer**: Can add findings and tests -- **Reader**: Read-only access -- **API_Importer**: Can import findings via API - -#### Resource Type: Engagement - -Maps **Resource** entities to DefectDojo Engagements (security assessments). - -```toml -[plugins.defectdojo.field_mappings.engagement] -# Required fields -name = "metadata.name" # Engagement name -description = "metadata.description" # Engagement description - -# Product reference -product_id = { - from = "spec.owner", # Resource.spec.owner = "component:my-service" - resolve_entity = "Component", # Find Component entity - extract = "annotations.defectdojo.com/product-id" -} - -# Date fields (ISO 8601 format: YYYY-MM-DD) -target_start = "spec.target_start" # Start date -target_end = "spec.target_end" # End date - -# Optional fields -status = { value = "In Progress" } # Not Started, In Progress, Completed, Cancelled -engagement_type = { value = "CI/CD" } # Interactive, CI/CD, etc. - -# Lead user reference -lead_id = { - from = "spec.dependsOn", # Resource.spec.dependsOn = ["user:lead"] - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} - -# Version control fields -version = "metadata.annotations.version" -commit_hash = "metadata.annotations.commit_hash" -branch_tag = "metadata.annotations.branch" -build_id = "metadata.annotations.build_id" -source_code_management_uri = "metadata.annotations.repo_url" - -# Test type flags -deduplication_on_engagement = { value = true } -threat_model = { value = false } -api_test = { value = true } -pen_test = { value = false } -check_list = { value = false } -``` - -## Environment Variables - -### Required Variables - -```bash -# DefectDojo Plugin -export DEFECTDOJO_URL="https://defectdojo.example.com" -export DEFECTDOJO_API_TOKEN="your-api-token-here" - -# Database -export DATABASE_URL="postgresql://user:pass@localhost/charybdis" - -# Optional: Telemetry -export OTLP_ENDPOINT="http://localhost:4317" -``` - -### Obtaining DefectDojo API Token - -1. Log into DefectDojo UI -2. Go to **User Profile** → **API Key** -3. Click **Generate** or copy existing key -4. Set environment variable: `export DEFECTDOJO_API_TOKEN=""` - -### Security Considerations - -**Never commit secrets to git**: -- ✓ Use environment variables for tokens/passwords -- ✓ Add `.env` to `.gitignore` -- ✓ Use secret management (HashiCorp Vault, AWS Secrets Manager) -- ✗ Don't hardcode secrets in `config.toml` - -## Best Practices - -### 1. Field Mapping Design - -**Start Simple**: -```toml -# Start with required fields only -name = "metadata.name" -description = "metadata.description" -``` - -**Add Optional Fields Gradually**: -```toml -# Add optional fields as needed -tags = "metadata.tags" -business_criticality = { value = "medium" } -``` - -**Use Entity Resolution Last**: -```toml -# Complex mappings last (requires other entities to exist) -product_manager = { - from = "spec.owner", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} -``` - -### 2. Entity Creation Order - -For entity resolution to work, create entities in this order: - -1. **Users** (no dependencies) -2. **Product Types** (no dependencies) -3. **Components** (may reference Users) -4. **Groups** (references Components and Users) -5. **Resources/Engagements** (references Components and Users) - -### 3. Testing Strategy - -**Test in Isolation**: -1. Create a test Component without references -2. Verify Product created in DefectDojo -3. Check annotation added: `defectdojo.com/product-id` - -**Test with References**: -1. Create User entity -2. Wait for DefectDojo sync -3. Create Component with `spec.owner = "user:john.doe"` -4. Verify product_manager set correctly in DefectDojo - -**Test Updates**: -1. Update Component name -2. Verify Product name updated in DefectDojo - -**Test Deletes**: -1. Delete Component -2. Verify Product deleted in DefectDojo - -### 4. Error Handling - -**Check Logs**: Plugin operations are logged with `tracing`: -``` -INFO DefectDojo plugin: Creating DefectDojo product for entity: component-123 -INFO DefectDojo plugin: Created DefectDojo product 456 for entity component-123 -WARN DefectDojo plugin: Entity component-789 has no DefectDojo product ID, skipping deletion -ERROR DefectDojo plugin: Failed to create product: API error 400: Invalid product type ID -``` - -**Log Levels**: -- `INFO`: Normal operations -- `WARN`: Non-critical issues (missing annotations, skipped operations) -- `ERROR`: Critical failures (API errors, invalid configuration) - -### 5. Performance Optimization - -**Minimize Entity Resolution**: -- Cache resolved IDs when possible -- Use static values when appropriate -- Avoid deep resolution chains - -**Batch Operations**: -- Create multiple entities in succession -- Plugin handles each independently -- No manual batching needed - -### 6. Configuration Management - -**Use Configuration Profiles**: - -```toml -# config.development.toml -[plugins.defectdojo] -base_url = "https://defectdojo-dev.example.com" -auto_create_users = true - -# config.production.toml -[plugins.defectdojo] -base_url = "https://defectdojo.example.com" -auto_create_users = false # Require manual user creation -``` - -**Environment-Specific Settings**: -```bash -# Development -export CONFIG_FILE="config.development.toml" - -# Production -export CONFIG_FILE="config.production.toml" -``` - -## Troubleshooting - -### Plugin Not Reacting to Events - -**Check**: -1. Plugin enabled: `plugins.defectdojo.enabled = true` -2. Plugin compiled: Run `cargo build` -3. Entity kind matches trigger: Component → product handler -4. Event bus running: Check logs for event publications - -### Field Mapping Not Working - -**Check**: -1. Field path correct: `"metadata.name"` not `"meta.name"` -2. Field exists in entity: Use YAML adapter to inspect entity -3. Static value syntax: `{ value = "text" }` not `"text"` -4. Entity resolution: Resolved entity exists and has required annotation - -### Entity Resolution Fails - -**Check**: -1. Referenced entity exists: `user:john.doe` exists in database -2. Referenced entity has annotation: `defectdojo.com/user-id` present -3. Entity kind correct: `resolve_entity = "User"` not `"user"` -4. Array syntax: `resolve_array = true` for array fields - -### API Errors - -**Common Errors**: - -``` -400 Bad Request: Invalid product type ID -→ Check default_product_type_id or product_type_id mapping - -401 Unauthorized: Invalid token -→ Check DEFECTDOJO_API_TOKEN is correct - -404 Not Found: Product not found -→ Entity may not have been created yet - -409 Conflict: User already exists -→ Normal for users (plugin detects and reuses) -``` - -### Missing Annotations - -If annotations not created: -1. Check entity update succeeded: `repository.update()` logs -2. Verify annotation key: `defectdojo.com/product-id` -3. Check database: `SELECT annotations FROM entities WHERE id = '...'` - -### Performance Issues - -If plugin causing slowness: -1. Check DefectDojo API response times -2. Reduce entity resolution depth -3. Use static values where possible -4. Check network connectivity to DefectDojo - -## Examples - -### Example 1: Basic Product Creation - -**Entity** (Component): -```yaml -apiVersion: backstage.io/v1alpha1 -kind: Component -metadata: - name: payment-service - description: Payment processing microservice - tags: - - payment - - critical - - pci-dss -spec: - type: service - lifecycle: production - owner: platform-team -``` - -**Configuration**: -```toml -[plugins.defectdojo.field_mappings.product] -name = "metadata.name" -description = "metadata.description" -tags = "metadata.tags" -product_type_id = { value = 1 } -business_criticality = { value = "very high" } -platform = { value = "web" } -lifecycle = "spec.lifecycle" -``` - -**Result in DefectDojo**: -```json -{ - "id": 456, - "name": "payment-service", - "description": "Payment processing microservice", - "tags": ["payment", "critical", "pci-dss"], - "prod_type": 1, - "business_criticality": "very high", - "platform": "web", - "lifecycle": "production" -} -``` - -**Updated Entity**: -```yaml -metadata: - annotations: - defectdojo.com/product-id: "456" -``` - -### Example 2: Product with User References - -**Entities**: - -User: -```yaml -apiVersion: backstage.io/v1alpha1 -kind: User -metadata: - name: john.doe -spec: - profile: - displayName: John Doe - email: john.doe@example.com - memberOf: - - platform-team -``` - -Component: -```yaml -apiVersion: backstage.io/v1alpha1 -kind: Component -metadata: - name: payment-service - annotations: - technical-contact: user:jane.smith -spec: - owner: user:john.doe -``` - -**Configuration**: -```toml -[plugins.defectdojo.field_mappings.product] -name = "metadata.name" -product_type_id = { value = 1 } - -product_manager = { - from = "spec.owner", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} - -technical_contact = { - from = "metadata.annotations.technical-contact", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} -``` - -**Resolution Process**: -1. Component created → Product handler triggered -2. Field mapper resolves `spec.owner = "user:john.doe"` -3. Queries for User entity with id "john.doe" -4. Extracts `annotations.defectdojo.com/user-id = "123"` -5. Sets `product_manager: 123` in API call - -### Example 3: Product Member Assignment - -**Entities**: - -Component (already has DD product ID): -```yaml -metadata: - annotations: - defectdojo.com/product-id: "456" -``` - -Users (already have DD user IDs): -```yaml -# john.doe -metadata: - annotations: - defectdojo.com/user-id: "123" - -# jane.smith -metadata: - annotations: - defectdojo.com/user-id: "124" -``` - -Group: -```yaml -apiVersion: backstage.io/v1alpha1 -kind: Group -metadata: - name: payment-service-security-team -spec: - type: team - parent: component:payment-service - members: - - user:john.doe - - user:jane.smith -``` - -**Configuration**: -```toml -[plugins.defectdojo.field_mappings.product_member] -product_id = { - from = "spec.parent", - resolve_entity = "Component", - extract = "annotations.defectdojo.com/product-id" -} - -user_id = { - from = "spec.members", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id", - resolve_array = true -} - -role_name = { value = "Reader" } -``` - -**Result**: -- Creates Product Member: Product 456 + User 123 + Role "Reader" -- Creates Product Member: Product 456 + User 124 + Role "Reader" - -### Example 4: Engagement with Full Metadata - -**Entity**: -```yaml -apiVersion: backstage.io/v1alpha1 -kind: Resource -metadata: - name: payment-service-q1-2025-assessment - description: Q1 2025 Security Assessment - annotations: - version: "2.3.0" - commit_hash: "abc123def456" - branch: "release/2.3" - build_id: "build-789" - repo_url: "https://github.com/example/payment-service" -spec: - type: security-assessment - owner: component:payment-service - dependsOn: - - user:security-lead - target_start: "2025-01-01" - target_end: "2025-03-31" -``` - -**Configuration**: -```toml -[plugins.defectdojo.field_mappings.engagement] -name = "metadata.name" -description = "metadata.description" - -product_id = { - from = "spec.owner", - resolve_entity = "Component", - extract = "annotations.defectdojo.com/product-id" -} - -lead_id = { - from = "spec.dependsOn", - resolve_entity = "User", - extract = "annotations.defectdojo.com/user-id" -} - -target_start = "spec.target_start" -target_end = "spec.target_end" -version = "metadata.annotations.version" -commit_hash = "metadata.annotations.commit_hash" -branch_tag = "metadata.annotations.branch" -build_id = "metadata.annotations.build_id" -source_code_management_uri = "metadata.annotations.repo_url" - -status = { value = "In Progress" } -engagement_type = { value = "CI/CD" } -deduplication_on_engagement = { value = true } -api_test = { value = true } -pen_test = { value = false } -``` - -**Result in DefectDojo**: -```json -{ - "id": 789, - "name": "payment-service-q1-2025-assessment", - "description": "Q1 2025 Security Assessment", - "product": 456, - "lead": 123, - "target_start": "2025-01-01", - "target_end": "2025-03-31", - "version": "2.3.0", - "commit_hash": "abc123def456", - "branch_tag": "release/2.3", - "build_id": "build-789", - "source_code_management_uri": "https://github.com/example/payment-service", - "status": "In Progress", - "engagement_type": "CI/CD", - "deduplication_on_engagement": true, - "api_test": true, - "pen_test": false -} -``` - -## Advanced Topics - -### Custom Field Mapping Logic - -For complex scenarios not covered by built-in mapping types, implement custom resource handlers. - -### Plugin Development - -See `docs/PLUGIN_DEVELOPMENT_GUIDE.md` (future) for creating new plugins. - -### Entity Resolution Performance - -Entity resolution requires database queries. For high-throughput scenarios: -- Use static values where possible -- Cache resolved IDs in memory -- Consider implementing a resolution cache layer - -### Multi-Environment Configuration - -```bash -# Development -cargo run -- --config config.development.toml - -# Staging -cargo run -- --config config.staging.toml - -# Production -cargo run -- --config config.production.toml -``` - -## Related Documentation - -- [DefectDojo Plugin README](../plugins/defectdojo/README.md) -- [Plugin Implementation Guide](./DEFECTDOJO_PLUGIN_IMPLEMENTATION.md) -- [Charybdis Architecture](architecture.md) -- [Field Mapper Tests](../src/plugins/field_mapper.rs#tests) - -## Support - -For issues or questions: -1. Check this documentation -2. Review plugin logs -3. Test with minimal configuration -4. Open issue on GitHub with: - - Configuration (sanitized) - - Entity YAML - - Error logs - - Expected vs actual behavior diff --git a/docs/architecture.md b/docs/architecture.md index 8ec4f87..d3643b4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,965 +1,419 @@ -# Charybdis - Architecture +# Architecture -**Version**: 2.0 -**Last Updated**: 2026-05-06 -**Status**: Living Document +This document describes the implementation of Charybdis: its protobuf schema, storage model, event flow, plugin lifecycle, and the rationale behind the major design decisions. -> For the product vision and roadmap, see [VISION.md](../VISION.md). - ---- - -## Executive Summary - -**Charybdis** is a security-native platform engineering tool that unifies software catalog, vulnerability management, and compliance posture in a single event-driven platform. Built in Rust, deployed as a single binary. - -### Core Value Proposition - -**Problem**: -- Platform engineers run 5+ disconnected tools (Backstage, DefectDojo, Dependency-Track, license scanners, compliance spreadsheets) -- Software catalogs rely on static YAML files that drift from reality -- Security posture is invisible at the catalog level — vulns live in separate tools with no link to services -- Compliance evidence is assembled manually from fragmented data sources - -**Solution**: -- **Dynamic software catalog** — event-driven, gRPC-native, no static YAML -- **Native vulnerability management** — ingest scan results (SARIF, CycloneDX, SPDX), triage, assess, track -- **Security gates & compliance** — severity thresholds, license policies, compliance framework mappings -- **Plugin system for integrations** — Slack, Jira, GitHub, custom tools react to events -- **Single binary deployment** — Rust + PostgreSQL, ~50MB RAM, sub-millisecond latency - -### Use Case Examples - -**Service registration with auto-provisioning:** -``` -CI/CD Pipeline (Python) → gRPC CreateEntity(kind=Component) - ↓ - Charybdis persists entity - ↓ - Event: EntityCreated published - ↓ - ┌────────────────────┴────────────────────┐ - ↓ ↓ - Jira Plugin Slack Plugin - - Creates onboarding epic - Notifies #platform channel - ↓ ↓ - └────────────────────┬────────────────────┘ - ↓ - Entity visible via gRPC API / Backstage YAML adapter -``` - -**Vulnerability ingestion with security gates:** -``` -Scanner (Trivy) → CI/CD → gRPC IngestScan(sarif_report) - ↓ - Charybdis parses SARIF - Creates Vulnerability entities linked to Component - ↓ - Rules engine evaluates auto-assessment - Security gate checks thresholds - ↓ - ┌────────────────────┴────────────────────┐ - ↓ ↓ - Gate PASSED Gate FAILED - - Vulns tracked - Slack alert to #security - - Dashboard updated - Jira ticket created - - CI/CD pipeline blocked -``` - ---- +For the product vision and roadmap, see [VISION.md](../VISION.md). +For end-user concepts (entities, annotations, events), see [core-concepts.md](core-concepts.md). ## Architectural Principles -### 1. **No Database Migrations Ever** +### 1. Zero database migrations -The database schema is created once on first startup and **never changes**. +The PostgreSQL schema is created on first startup and **never changes**. -**How it works**: -- Entities stored as protobuf bytes in `entity_data BYTEA` column -- Plugins add new entity types via protobuf `oneof` variants -- Build system regenerates code, database schema remains static +- Entities are serialized to protobuf and stored in `entity_data BYTEA`. +- New entity kinds are added as new `oneof` variants in the protobuf schema, not as new columns. +- Plugins extend the schema at build time by registering protobuf field numbers (see [plugin lifecycle](#plugin-lifecycle)). -**Benefits**: -- Deploy new plugins without downtime -- No migration scripts to manage -- Forward/backward compatibility built-in -- Easy rollback (protobuf versioning) +Deploying a new plugin requires a rebuild but no migration script. Rolling back is the same: rebuild without the plugin, the stored bytes for that variant are simply ignored. -### 2. **gRPC First, Everything Else is Adapter** +### 2. gRPC is the only API -Charybdis provides **only gRPC APIs**. All other interfaces (REST, YAML, GraphQL) are adapters on top. +Charybdis exposes two gRPC services and one HTTP adapter: -**Rationale**: -- **Strong typing**: Protobuf ensures type safety across all languages -- **Performance**: Binary protocol, efficient serialization -- **Multi-language**: Official gRPC clients for 10+ languages -- **Streaming**: Built-in support for real-time updates (future) -- **Code generation**: Automatic client/server code generation +- `EntityService` (`proto/entities.proto`) — full entity CRUD with field-mask partial updates. +- `IngestionService` (`proto/ingestion.proto`) — `ImportScan` / `DryRunScan` for security findings. +- HTTP YAML adapter — Backstage-compatible Location YAML on a separate port. -**What Charybdis provides**: -- ✅ gRPC API with protobuf definitions -- ✅ YAML adapter endpoints (for Backstage compatibility/migration) -- ❌ REST API (teams can add if needed via gateway) -- ❌ GraphQL (teams can add if needed) +No REST, no GraphQL in core. Teams that need them should put grpc-gateway or Envoy in front. -### 3. **Plugin-Based Extensibility** +### 3. Plugins are compile-time integrated -Plugins are **compile-time integrated** Rust crates that: +Plugins are Rust crates linked at build time into `charybdis-server`. Pros: type safety, no dynamic loader, no runtime version skew. Cons: adding a plugin requires a rebuild — acceptable for an infrastructure tool. -1. **Define entity types** via protobuf schemas -2. **React to events** via EventHandler trait -3. **Call external APIs** to synchronize state -4. **Store references** in entity annotations -5. **(Future) Extend gRPC API** with custom endpoints +### 4. Event-driven, post-persistence -**Plugin characteristics**: -- Not hot-swappable (require rebuild) -- Type-safe at compile time -- No runtime dependency resolution -- Configurable via YAML config file -- Can be developed by community - -### 4. **Event-Driven Orchestration** - -Events are published **after** entities are persisted: +Events fire **after** the database write succeeds. ``` -CRUD Operation → Persist to DB → Publish Event → Plugins React +CRUD request → validate → DB write → publish event → plugins react (async) ``` -**Event flow**: -1. Entity created/updated/deleted in database -2. Event published to event bus with full entity data -3. All subscribed plugin handlers receive event -4. Plugins filter events they care about -5. Plugins perform asynchronous actions -6. Plugins update entity annotations (via repository) +Plugin failures don't fail the original request. The entity is already persisted; plugins log their errors and the event bus moves on. -**Error handling**: -- Plugin errors **do not** fail the CRUD operation -- Entity is already persisted before event publishing -- Plugins log errors for observability -- Retries handled by event bus backend +### 5. Backstage-compatible entity shape -### 5. **Backstage Compatibility by Design** +The protobuf entity model maps 1:1 onto Backstage's `apiVersion / kind / metadata / spec` structure. The YAML adapter renders this directly. This is a deliberate compatibility choice — teams can run Charybdis as a dynamic backend behind their existing Backstage frontend during migration. -Charybdis entity model is **100% compatible** with Backstage's descriptor format. - -**How it works**: -- Core protobuf types mirror Backstage YAML structure -- YAML adapter endpoints serve entities in Backstage format -- Backstage reads Charybdis as a dynamic location provider -- No changes needed to Backstage frontend - -**Key compatibility points**: -- Entity `kind`, `apiVersion`, `metadata`, `spec` structure -- Annotation format (reverse-DNS style) -- Relationship types (`dependsOn`, `partOf`, etc.) -- All standard Backstage entity kinds supported - ---- - -## System Architecture - -### Component Overview +## System Layout ``` -┌─────────────────────────────────────────────────────────────────┐ -│ Client Layer │ -│ CI/CD Pipelines (Python, Go, Node.js, Bash, etc.) │ -│ Security Scanners (SARIF, CycloneDX, SPDX output) │ -│ IaC Tools (Terraform, Pulumi, Crossplane) │ -│ Backstage (via YAML adapter, optional) │ -└────────────────┬────────────────────────────────────────────────┘ - │ - │ gRPC / YAML HTTP - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Charybdis Core │ -│ ┌──────────────────────────────────────────────────────────┐ │ -│ │ gRPC API │ │ -│ │ EntityService: CRUD + List (with field masks) │ │ -│ │ ScanService: IngestScan (SARIF, CycloneDX, SPDX) │ │ -│ │ AssessmentService: Triage, Accept, Remediate │ │ -│ └────────────────────┬─────────────────────────────────────┘ │ -│ │ │ -│ ┌────────────────────▼─────────────────────────────────────┐ │ -│ │ Security Engine (native) │ │ -│ │ - Vulnerability tracking per entity │ │ -│ │ - Rules engine (auto-assessment) │ │ -│ │ - Security gates (severity thresholds) │ │ -│ │ - License compliance │ │ -│ └────────────────────┬─────────────────────────────────────┘ │ -│ │ │ -│ ┌────────────────────▼─────────────────────────────────────┐ │ -│ │ Adapters │ │ -│ │ - YAML Adapter (Backstage compat / migration) │ │ -│ └────────────────────┬─────────────────────────────────────┘ │ -│ │ │ -│ ┌────────────────────▼─────────────────────────────────────┐ │ -│ │ Repository Layer │ │ -│ │ - PostgreSQL with protobuf storage │ │ -│ │ - JSONB annotations for fast queries │ │ -│ │ - Zero-migration schema │ │ -│ └────────────────────┬─────────────────────────────────────┘ │ -│ │ │ -│ ┌────────────────────▼─────────────────────────────────────┐ │ -│ │ Event Bus System │ │ -│ │ - Memory backend (dev) │ │ -│ │ - Redis backend (prod, future) │ │ -│ └────────────────────┬─────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────┐ +│ Clients │ +│ CI/CD pipelines · Scanners (SARIF) · IaC tools · Backstage │ +└─────────────────┬────────────────────────────────────────────┘ + │ gRPC (mTLS optional) HTTP (YAML) + ▼ ▼ +┌──────────────────────────────────────────────────────────────┐ +│ charybdis-server │ +│ ┌────────────────────────┐ ┌──────────────────────────┐ │ +│ │ EntityService │ │ IngestionService │ │ +│ │ CreateEntity │ │ ImportScan │ │ +│ │ GetEntity │ │ DryRunScan │ │ +│ │ UpdateEntity (FM) │ └──────────┬───────────────┘ │ +│ │ DeleteEntity │ │ │ +│ │ ListEntities │ ▼ │ +│ └──────────┬─────────────┘ ┌──────────────────────────┐ │ +│ │ │ Findings pipeline │ │ +│ │ │ Parser registry │ │ +│ │ │ Reconciliation engine │ │ +│ │ │ Fingerprint dedup │ │ +│ │ └──────────┬───────────────┘ │ +│ │ │ │ +│ ▼ ▼ │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ AuthInterceptor (mTLS + RBAC) │ │ +│ └────────────────────┬───────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────▼───────────────────────────────────┐ │ +│ │ EntityRepository (PostgreSQL, protobuf + JSONB) │ │ +│ └────────────────────┬───────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────▼───────────────────────────────────┐ │ +│ │ EventBus (MemoryEventBus) │ │ +│ │ → EventDispatcher → plugin ResourceHandlers │ │ +│ └────────────────────┬───────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────▼───────────────────────────────────┐ │ +│ │ YAML Adapter (axum HTTP, separate port) │ │ +│ └────────────────────────────────────────────────────────┘ │ +│ │ +│ Telemetry: OpenTelemetry traces/metrics/logs (console+OTLP) │ +└──────────────────────────────────────────────────────────────┘ │ - │ Entity / Vulnerability / Gate events - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Plugin Layer (integrations) │ -│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │ -│ │ Slack/Teams │ │ Jira/GitHub │ │ Custom Plugins │ │ -│ │ Notifications │ │ Issue Track │ │ Your integrations │ │ -│ └──────┬────────┘ └──────┬───────┘ └──────────┬─────────────┘ │ -└─────────┼──────────────────┼─────────────────────┼──────────────┘ - ▼ ▼ ▼ - Slack/Teams API Jira/GitHub API External APIs + ▼ external API calls + Plugins: DefectDojo, Keycloak ``` -### Data Flow +## Entity Schema -#### Entity Creation Flow - -``` -1. Client (CI/CD) - ↓ gRPC CreateEntityRequest -2. Charybdis gRPC API - ↓ Validate entity - ↓ Generate UUID, timestamps -3. Entity Repository - ↓ Serialize to protobuf bytes - ↓ INSERT INTO entities -4. PostgreSQL - ↓ Entity persisted -5. Event Bus - ↓ Publish EntityCreated event (with full entity data) -6. Plugin Handlers (parallel) - ├─ DefectDojo: Create product → Store product_id in annotations - ├─ DependencyTrack: Create project → Store project_uuid in annotations - └─ Custom: Custom actions -7. Repository.update_annotations() - ↓ Merge annotations from plugins - ↓ UPDATE entities SET annotations = ... -8. PostgreSQL - ↓ Entity updated with external tool IDs -9. Return CreateEntityResponse to client -``` - -#### Backstage Integration Flow - -``` -1. Backstage Catalog Backend - ↓ HTTP GET /yaml/locations -2. Charybdis YAML Adapter - ↓ Query repository.list_all() - ↓ Convert entities to Location YAML -3. Backstage receives location list - ↓ Fetches each entity: GET /yaml/entities/:id -4. Charybdis YAML Adapter - ↓ repository.get_by_id() - ↓ Convert entity protobuf → Backstage YAML -5. Backstage ingests entities - ↓ Displays in UI with all annotations - ↓ Shows linked DefectDojo products, etc. -``` - ---- - -## Core Entity Model - -### Protobuf Schema +Defined in `proto/entities.proto` (generated by `build.rs` from `entities.proto.template` + `plugins.toml`). ```protobuf message Entity { - // System-managed fields - string id = 1; // UUID (auto-generated) + string id = 1; // UUID, server-assigned + string kind = 2; // "Component", "Finding", ... + + oneof metadata { + charybdis.core.ComponentMetadata component_metadata = 12; + charybdis.core.ServiceMetadata service_metadata = 10; + charybdis.core.SystemMetadata system_metadata = 11; + charybdis.core.ApiMetadata api_metadata = 13; + charybdis.core.UserMetadata user_metadata = 14; + charybdis.core.GroupMetadata group_metadata = 15; + charybdis.core.DomainMetadata domain_metadata = 16; + charybdis.core.ResourceMetadata resource_metadata = 17; + charybdis.core.FindingMetadata finding_metadata = 24; + + // Plugin-contributed variants (field number 100+) + charybdis.plugins.defectdojo.DefectdojoMetadata defectdojo_metadata = 100; + charybdis.plugins.dependencytrack.DependencytrackMetadata dependencytrack_metadata = 101; + charybdis.plugins.keycloak.KeycloakMetadata keycloak_metadata = 102; + } + + oneof spec { /* matching variants */ } + + map annotations = 20; google.protobuf.Timestamp created_at = 21; google.protobuf.Timestamp updated_at = 22; - - // Backstage-compatible fields - string api_version = 2; // "backstage.io/v1alpha1" - string kind = 3; // "Component", "API", "User", etc. - - // Polymorphic metadata (identifying information) - oneof metadata { - // Core types (built-in) - ComponentMetadata component_metadata = 10; - APIMetadata api_metadata = 11; - UserMetadata user_metadata = 12; - GroupMetadata group_metadata = 13; - SystemMetadata system_metadata = 14; - DomainMetadata domain_metadata = 15; - ResourceMetadata resource_metadata = 16; - LocationMetadata location_metadata = 17; - TemplateMetadata template_metadata = 18; - - // Plugin types (dynamically added) - // Example: DefectDojoProductMetadata defectdojo_product_metadata = 100; - } - - // Polymorphic spec (configuration and behavior) - oneof spec { - // Core types (built-in) - ComponentSpec component_spec = 10; - APISpec api_spec = 11; - UserSpec user_spec = 12; - GroupSpec group_spec = 13; - SystemSpec system_spec = 14; - DomainSpec domain_spec = 15; - ResourceSpec resource_spec = 16; - LocationSpec location_spec = 17; - TemplateSpec template_spec = 18; - - // Plugin types (dynamically added) - // Example: DefectDojoProductSpec defectdojo_product_spec = 100; - } - - // Annotations - external tool references - // Format: "tool.com/resource-type-attribute" - // Examples: - // "defectdojo.com/product-id": "42" - // "dependencytrack.com/project-uuid": "550e8400-..." - // "github.com/repo-slug": "myorg/myrepo" - map annotations = 20; } ``` -### Backstage Entity Kinds Mapping +Field numbers are governed by `plugins.toml` — core types use 10–49, plugins claim slots starting at 100. The registry MUST be kept consistent: protobuf wire format depends on field-number stability. -All Backstage entity kinds must be supported: +### Validation -| Backstage Kind | Charybdis Proto Type | Status | -|----------------|---------------------|--------| -| Component | ComponentMetadata/Spec | ✅ Done | -| Service | ServiceMetadata/Spec | ✅ Done | -| API | APIMetadata/Spec | ✅ Done | -| User | UserMetadata/Spec | ✅ Done | -| Group | GroupMetadata/Spec | ✅ Done | -| System | SystemMetadata/Spec | ✅ Done | -| Domain | DomainMetadata/Spec | ✅ Done | -| Resource | ResourceMetadata/Spec | ✅ Done | -| Location | LocationMetadata/Spec | Not planned (entities are event-driven, not file-based) | -| Template | TemplateMetadata/Spec | Phase 3 (Scaffolder) | +Performed in `src/lib.rs::validate_entity()`: +- `kind` is non-empty and one of the known values (`VALID_KINDS`). +- `metadata` is present. +- The metadata variant matches `kind` (e.g., `kind=Component` requires `component_metadata`). ---- +Plugin metadata variants are accepted without a kind cross-check — they identify themselves via the variant tag. -## YAML Adapter Layer +### Field masks -### Purpose +`UpdateEntity` accepts a `google.protobuf.FieldMask`. `EntityRepository::partial_update()` (in `src/database.rs`) walks the mask paths and patches only the named fields. Supported paths include: -Serve entities in Backstage-compatible YAML format over HTTP. +- Top-level: `kind`, `annotations`, `metadata`, `spec` +- Nested annotation key: `annotations.` +- Per-variant fields: `component_metadata.name`, `component_spec.lifecycle`, `service_spec.owner`, ..., `user_spec.profile.email`, `group_spec.profile.display_name`, etc. -### API Specification +The full path enumeration lives in the `apply_field_mask` match in `database.rs`. New kinds extend this match. -#### 1. List Locations +## Storage Model -**Endpoint**: `GET /yaml/locations` +```sql +CREATE TABLE entities ( + id UUID PRIMARY KEY, + kind VARCHAR NOT NULL, + name VARCHAR, + entity_data BYTEA NOT NULL, -- prost-encoded Entity message + annotations JSONB NOT NULL DEFAULT '{}', + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); -**Purpose**: Return a dynamic Location entity that lists all entities in Charybdis. +CREATE INDEX idx_entities_kind ON entities(kind); +CREATE INDEX idx_entities_kind_name ON entities(kind, name); +CREATE INDEX idx_entities_annotations ON entities USING GIN (annotations); +CREATE INDEX idx_entities_created_at_id ON entities(created_at DESC, id DESC); +``` + +Created by `database::ensure_schema()` on startup. Idempotent. + +- `entity_data` carries the source of truth (the full protobuf blob). +- `kind`, `name`, `annotations` are denormalized for indexed lookups — kept in sync by the repository. +- `idx_entities_kind_name` powers O(log n) `get_by_kind_and_name` (used heavily by ingestion to resolve `component_ref` → UUID). +- `idx_entities_annotations` (GIN) powers `annotations->>'key' = 'value'` filters for plugins. +- `idx_entities_created_at_id` powers stable cursor pagination — `ListEntities` encodes `(created_at, id)` as a base64 cursor, scanning `WHERE (created_at, id) < (cursor) ORDER BY created_at DESC, id DESC LIMIT N+1` to detect a next page without `OFFSET` scans. + +### Atomic annotation merge + +`update_annotations()` uses PostgreSQL's `||` JSONB operator: + +```sql +UPDATE entities +SET annotations = annotations || $1::jsonb, updated_at = $2 +WHERE id = $3 +RETURNING entity_data, annotations; +``` + +The merge happens at SQL level, eliminating the read-modify-write race when multiple plugin handlers write to the same entity concurrently. The returned `entity_data` is then re-encoded with the merged annotations so the protobuf blob stays consistent. + +## Event Bus + +Defined in `src/events/`. The default backend is in-process (`MemoryEventBus`). + +### Flow + +``` +EntityService.CreateEntity + ↓ +EntityRepository.create() → DB row written + ↓ +EntityEvent::created(uuid) + .with_metadata("entity_kind", kind) + .with_entity_data(Arc) + ↓ +EventBus.publish() + ↓ +EventDispatcher (subscribed) + ↓ +for each plugin's ResourceHandler: + if handler.trigger_kinds().contains(entity.kind): + handler.handle_create(entity) // 30s timeout, panic-isolated +``` + +### Properties + +- **At-least-once intent** — handlers may run more than once if a redelivery mechanism is added later. Handlers should be idempotent. +- **Best-effort delivery** — `MemoryEventBus` does not survive a server restart. A durable backend (Redis) is planned in [VISION.md](../VISION.md). +- **30s per-handler timeout** — slow plugins don't block the bus. +- **Panic isolation** — a panicking handler doesn't take down others or the dispatcher. + +## Findings Pipeline + +`IngestionService.ImportScan` triggers the following pipeline (`src/findings/`): + +1. **Resolve `component_ref` → UUID** (`MyIngestionService::resolve_component_id`) — accepts a UUID directly, or a name (looked up via `get_by_kind_and_name("Component", name)`). +2. **Pick a parser** by `format` from `ParserRegistry` (`src/scanners/mod.rs`). Today, only SARIF is registered. +3. **Parse** to `Vec` (`SarifParser` in `src/scanners/sarif.rs`). +4. **Reconcile** against existing findings for `(component_id, lifecycle)` (`ReconciliationEngine::reconcile`): + - Compute fingerprint per finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`). Line numbers are deliberately excluded — they shift too easily. + - Bucket each incoming finding as **New** (unknown fingerprint), **Unchanged** (active and re-seen), or **Reopened** (was resolved/false-positive, seen again). + - Anything in DB with state `ACTIVE`/`REOPENED` whose fingerprint is absent from this scan becomes **Resolved**. +5. **Apply** (`ReconciliationEngine::apply`): + - Create new findings as `Finding` entities (`build_finding_entity`). + - Bump `charybdis.io/last-seen` and `charybdis.io/scan-id` annotations on unchanged findings (atomic merge). + - Mark resolved findings as `RESOLVED` with `resolved_at` timestamp. + - Mark reopened findings as `REOPENED`. +6. **Respond** with `ReconciliationSummary { total_parsed, new_count, unchanged_count, resolved_count, reopened_count }` plus per-bucket finding lists. + +`DryRunScan` runs steps 1–4 and returns the summary without persisting. This is what makes MR/PR-level diff comments possible — a CI job can call `DryRunScan` and report "this change introduces X new findings" before merge. + +Additional parsers (CycloneDX VEX, SPDX), assessment workflow (triage/accept/remediate), rules engine, and security gates are tracked in [VISION.md](../VISION.md). State enum values for `ACCEPTED` and `FALSE_POSITIVE` already exist in the protobuf; the workflow that sets them does not. + +## YAML Adapter + +Backstage compatibility layer (`src/adapters/yaml/`). HTTP server on a separate port from gRPC. Two endpoints: + +### `GET /yaml/locations` + +Returns a single Backstage `Location` entity listing every entity in Charybdis as a target URL. -**Response Format** (Backstage Location YAML): ```yaml apiVersion: backstage.io/v1alpha1 kind: Location metadata: name: charybdis-all-entities - description: Dynamic location managed by Charybdis spec: type: charybdis targets: - - http://charybdis.company.com/yaml/entities/service-payment - - http://charybdis.company.com/yaml/entities/service-auth - - http://charybdis.company.com/yaml/entities/api-payments-v1 - - http://charybdis.company.com/yaml/entities/user-john-doe - # ... all entities + - http://charybdis.example.com/yaml/entities/ + # ... one per entity ``` -**Implementation Notes**: -- Query `SELECT id, kind FROM entities` -- Generate URL for each entity: `/yaml/entities/{kind}-{id}` -- Return as Backstage Location YAML -- Cache response (invalidate on entity CRUD) +Cached for 30 s. `YamlAdapterState::invalidate_cache()` is exposed for explicit invalidation on entity-CRUD events (not wired by default — it's a hook). -#### 2. Get Entity by ID +### `GET /yaml/entities/:id` -**Endpoint**: `GET /yaml/entities/:id` +Renders a single entity as Backstage YAML. -**Purpose**: Return a specific entity in Backstage YAML format. - -**Response Format** (Backstage Component YAML): ```yaml apiVersion: backstage.io/v1alpha1 kind: Component metadata: name: payment-service namespace: production - description: Core payment processing service annotations: - github.com/repo-slug: myorg/payment-service defectdojo.com/product-id: "42" - dependencytrack.com/project-uuid: "550e8400-e29b-41d4-a716-446655440000" - tags: - - payments - - critical - links: - - url: https://dashboard.company.com/payment-service - title: Dashboard - icon: dashboard + tags: [payments, critical] spec: type: service lifecycle: production - owner: payments-team + owner: team-payments system: payment-system - dependsOn: - - component:auth-service - providesApis: - - payments-api-v1 ``` -**Implementation Notes**: -- Query `repository.get_by_id(id)` -- Convert Entity protobuf → Backstage YAML -- Map `oneof metadata/spec` to appropriate YAML structure -- Include all annotations -- Return as `text/yaml` content type +### Configuration in Backstage -#### 3. Configuration in Backstage - -**Backstage app-config.yaml**: ```yaml catalog: locations: - type: url - target: http://charybdis.company.com/yaml/locations + target: http://charybdis.example.com/yaml/locations rules: - - allow: [Component, API, User, Group, System, Domain, Resource] + - allow: [Component, System, API, User, Group, Domain, Resource] ``` -**How it works**: -1. Backstage fetches `/yaml/locations` on catalog refresh -2. Parses Location YAML to get entity URLs -3. Fetches each entity URL (`/yaml/entities/:id`) -4. Ingests entities into Backstage catalog -5. Displays in UI with all metadata and annotations +Backstage polls the locations URL on its own schedule, then fetches each entity URL. ---- +## Plugin Lifecycle -## Plugin System +### Build time -### Plugin Architecture +1. `build.rs` reads `plugins.toml`. +2. For each enabled plugin, the corresponding `plugins//proto/.proto` is added to the compile set. +3. `entities.proto.template` is expanded with plugin imports and `oneof` variants using the field numbers from `plugins.toml`. +4. `tonic-prost-build` compiles every proto file and writes the descriptor set for gRPC reflection. -Charybdis supports **two types of plugins**, both compile-time integrated: +### Runtime -#### **1. Event-Driven Plugins** -React to entity lifecycle events (create, update, delete): -- **Example**: DefectDojo, DependencyTrack -- **Implement**: `EventDrivenPlugin` trait -- **Provide**: `ResourceHandler` implementations -- **Triggered by**: Entity CRUD operations +1. `charybdis-server` (entry point: `charybdis-server/src/main.rs`) loads `config.toml`. +2. For each plugin block with `enabled = true`, the corresponding plugin crate is instantiated (`DefectDojoPlugin::new`, `KeycloakPlugin::new`, ...). +3. Event-driven plugins are registered with the `PluginManager`; their `ResourceHandler`s are wrapped in an `EventDispatcher` subscribed to the event bus. +4. Sync plugins register with the cron scheduler. If `on_startup = true`, an initial sync runs in a background task. +5. mTLS + RBAC are wired into a tonic interceptor (when enabled). +6. The YAML adapter is spawned as a separate axum task on its own port. +7. The gRPC server is started with `EntityServiceServer + IngestionServiceServer + reflection`. -#### **2. Sync Plugins** -Pull data from external sources on a schedule: -- **Example**: Okta, Keycloak, Active Directory -- **Implement**: `SyncPlugin` trait -- **Scheduled by**: Cron expressions -- **Triggered by**: Time-based schedule or manual API call +The two `main.rs` arrangement (one in `src/main.rs` for the library convenience, one in `charybdis-server/src/main.rs` for the real binary) exists because the root `charybdis` crate cannot depend on plugin crates without creating a cyclic dependency. `charybdis-server` is the seam that links plugins to the core. -### Generic Plugin Utilities (2025-01-15) +Plugin trait reference, configuration, field mapping system, and "writing a new plugin": [../plugins/README.md](../plugins/README.md). -All plugins have access to reusable utilities in `src/plugins/`: +## Security -#### **PluginHttpClient** -Generic HTTP client with multiple auth methods: -- Token authentication (DefectDojo) -- Bearer authentication (modern APIs) -- API Key authentication (DependencyTrack) -- Basic authentication (JIRA, Bitbucket) -- Methods: `get()`, `post()`, `put()`, `patch()`, `delete()` +mTLS + RBAC are off by default for local development, on by configuration for shared and production environments. The `AuthInterceptor` (`src/security/interceptor.rs`): -#### **AnnotationHelper** -Consistent API for storing/retrieving plugin metadata: -- `set_id()` / `get_id()` - Store external resource IDs -- `set()` / `get()` - Store arbitrary metadata -- Enforces naming: `{plugin}.com/{resource}-id` -- Prevents annotation conflicts between plugins +1. Extracts the client certificate from tonic's `TlsConnectInfo` (direct mTLS) or from an `x-forwarded-client-cert` header (reverse-proxy mode). +2. Parses the certificate via `x509-parser` into a `ClientIdentity { common_name, organization, organizational_unit, serial }`. +3. Maps the identity to a role via `RbacEngine::map_identity_to_role()` (subject-match rules from config). +4. Looks up the required permission for the gRPC method (`method_to_permission` is hard-coded). +5. Checks the role's permission list; denies on miss, logs allowed/denied to the audit log. -#### **FieldMapper** -Maps entity fields to external tool formats: -- Direct field paths: `"metadata.name"` -- Static values: `{"value": "Web Application"}` -- Complex mappings with entity resolution -- Repository integration for cross-entity queries +Configuration reference, role mapping, audit logging, reverse-proxy mode: [security.md](security.md). -#### **DateUtils** -Common date/time operations: -- `today()`, `today_plus_days(n)` - Date formatting -- `engagement_date_range(days)` - For time-bound resources -- `iso_timestamp()` - ISO 8601 timestamps +## Observability -### Plugin Responsibilities +OpenTelemetry setup in `src/telemetry/`. Three signals: +- **Traces** — every gRPC method is `#[instrument]`-ed; spans carry `entity.id`, `entity.kind`, `client.cn`, `role`, etc. +- **Metrics** — `Metrics::record_entity_operation(op, kind, duration)` from each handler. +- **Logs** — structured via `tracing-subscriber`, JSON or console. -Plugins can: - -1. **Define custom entity types** (via protobuf) -2. **React to entity events** (via ResourceHandler trait) -3. **Call external APIs** (using PluginHttpClient) -4. **Store metadata** (using AnnotationHelper) -5. **Map fields** (using FieldMapper) -6. **Schedule syncs** (SyncPlugins with cron) -7. **(Future) Extend gRPC API** (custom services) - -### Plugin Lifecycle - -``` -1. Development - ├─ Define protobuf schema (metadata/spec messages) - ├─ Implement EventHandler trait - ├─ Implement external API client - └─ Add configuration schema - -2. Registration - ├─ Add to plugins.toml (enabled = true) - └─ Add to Cargo.toml dependencies - -3. Build Time - ├─ build.rs reads plugins.toml - ├─ Generates entities.proto with plugin types - ├─ Compiles all protobuf to Rust code - └─ Links plugin into binary - -4. Runtime - ├─ Load plugin configuration from config file - ├─ Initialize plugin handler - ├─ Subscribe to event bus - └─ React to entity events -``` - -### Plugin Configuration System - -**File**: `config.toml` (TOML format with `${VAR}` env var substitution) - -```toml -# Server configuration -[server] -grpc_host = "[::1]" -grpc_port = 50051 - -[server.yaml_adapter] -enabled = true -host = "0.0.0.0" -port = 8080 - -# Database -[database] -url = "${DATABASE_URL}" -max_connections = 10 - -# Plugin configurations -[plugins.defectdojo] -enabled = true -base_url = "${DEFECTDOJO_API_URL}" -api_token = "${DEFECTDOJO_API_TOKEN}" - -[plugins.defectdojo.default_engagement] -auto_create = true -name = "CI/CD Pipeline" -engagement_type = "CI/CD" - -[plugins.keycloak] -enabled = false -# base_url = "https://keycloak.company.com" -# realm = "master" - -[plugins.dependencytrack] -enabled = false -# base_url = "https://dependencytrack.company.com" -# api_key = "${DEPENDENCYTRACK_API_KEY}" -``` - -See `config.toml.example` for the full reference with all options documented. - -**Configuration loading** (`src/config.rs`): - -```rust -pub struct Config { - pub server: ServerConfig, - pub database: DatabaseConfig, - pub security: SecurityConfig, - pub telemetry: TelemetryConfig, - pub plugins: PluginsConfig, -} - -impl Config { - pub fn from_file(path: &str) -> Result { - let content = std::fs::read_to_string(path)?; - // Substitute ${VAR} with environment variable values - let content = substitute_env_vars(&content); - let config: Config = toml::from_str(&content)?; - Ok(config) - } -} -``` - -**Each plugin implements**: - -```rust -pub trait Plugin { - fn name(&self) -> &str; - fn plugin_type(&self) -> PluginType; - fn load_config(&mut self, config: PluginConfig) -> Result<()>; - fn validate_config(&self) -> Result<()>; -} -``` - -### Event Handler Behavior Configuration - -Plugins can configure different behaviors based on: -- Entity kind -- Event type (created/updated/deleted) -- Custom conditions - -**Example**: DefectDojo plugin config (`config.toml`) - -```toml -[plugins.defectdojo] -enabled = true -base_url = "${DEFECTDOJO_API_URL}" -api_token = "${DEFECTDOJO_API_TOKEN}" - -[plugins.defectdojo.default_engagement] -auto_create = true -name = "CI/CD Pipeline" -engagement_type = "CI/CD" -``` - -**Handler Implementation**: - -```rust -impl EventHandler for DefectDojoHandler { - async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> { - // Check if plugin enabled - if !self.config.enabled { - return Ok(()); - } - - // Match event type to configured behavior - let behavior = match event.event_type { - EntityEventType::Created => &self.config.on_entity_created, - EntityEventType::Updated => &self.config.on_entity_updated, - EntityEventType::Deleted => &self.config.on_entity_deleted, - }; - - // Find matching rule for this entity kind - for rule in behavior { - if rule.matches(&event) { - return self.execute_action(&rule.action, event).await; - } - } - - Ok(()) - } -} -``` - ---- - -## Future: Plugin API Extensions - -**Future capability**: Plugins can expose custom gRPC endpoints. - -### Example: DefectDojo Query Plugin - -**Plugin extends API with custom service**: - -```protobuf -// plugins/defectdojo/proto/defectdojo_service.proto - -service DefectDojoService { - // Get findings for a specific entity - rpc GetEntityFindings(GetEntityFindingsRequest) returns (GetEntityFindingsResponse); - - // Get all active vulnerabilities - rpc GetActiveVulnerabilities(GetActiveVulnerabilitiesRequest) returns (GetActiveVulnerabilitiesResponse); - - // Sync entity with DefectDojo - rpc SyncEntity(SyncEntityRequest) returns (SyncEntityResponse); -} - -message GetEntityFindingsRequest { - string entity_id = 1; - string severity_filter = 2; // "Critical", "High", etc. - bool active_only = 3; -} - -message GetEntityFindingsResponse { - repeated Finding findings = 1; - int32 total_count = 2; -} - -message Finding { - int32 id = 1; - string title = 2; - string severity = 3; - string description = 4; - google.protobuf.Timestamp date = 5; - bool active = 6; -} -``` - -**Implementation**: - -```rust -pub struct DefectDojoServiceImpl { - client: DefectDojoClient, - repository: EntityRepository, -} - -#[async_trait] -impl DefectDojoService for DefectDojoServiceImpl { - async fn get_entity_findings( - &self, - request: Request, - ) -> Result, Status> { - let req = request.into_inner(); - - // Get entity to retrieve DefectDojo product ID - let entity = self.repository.get_by_id(&req.entity_id).await?; - let product_id = entity - .annotations - .get("defectdojo.com/product-id") - .ok_or_else(|| Status::not_found("No DefectDojo product linked"))?; - - // Fetch findings from DefectDojo API - let findings = self.client - .get_findings(product_id, &req.severity_filter, req.active_only) - .await?; - - Ok(Response::new(GetEntityFindingsResponse { - findings: findings.into_iter().map(|f| f.into()).collect(), - total_count: findings.len() as i32, - })) - } -} -``` - -**Registration in main.rs**: - -```rust -// Register plugin's custom gRPC service -if defectdojo_plugin.is_enabled() { - let defectdojo_service = DefectDojoServiceServer::new( - DefectDojoServiceImpl::new(defectdojo_client, repository.clone()) - ); - - server = server.add_service(defectdojo_service); -} -``` - -**Client usage**: - -```bash -grpcurl -d '{ - "entity_id": "service-payment", - "severity_filter": "Critical", - "active_only": true -}' localhost:50051 charybdis.plugins.defectdojo.DefectDojoService/GetEntityFindings -``` - ---- +Console exporter is on by default. OTLP exporter (traces/metrics/logs) is enabled via `[telemetry.otlp]` block in `config.toml`. ## Technology Decisions -### Why gRPC Only? +### Why gRPC only -**Decision**: Charybdis core provides only gRPC. Teams add REST/GraphQL as needed. +Strong typing across languages, binary efficiency for high-frequency CI/CD calls, official clients in every language platform engineers use, and one well-maintained surface area instead of three. Teams that need REST add grpc-gateway or Envoy. -**Rationale**: -- **Strong typing**: Protobuf schemas ensure correctness across languages -- **Performance**: Binary protocol, efficient for CI/CD use cases -- **Multi-language**: Official clients for Python, Go, Node.js, Java, Rust, C++, C#, etc. -- **Streaming**: Built-in bidirectional streaming (future: real-time updates) -- **Code generation**: Automatic client generation eliminates boilerplate -- **Focus**: Core team maintains one high-quality API surface +### Why PostgreSQL + protobuf + JSONB -**Teams can add REST if needed**: -``` -grpc-gateway or Envoy → Charybdis gRPC -``` +- ACID guarantees + a mature ecosystem most teams already operate. +- Protobuf blob = compact, versioned, no schema migrations for new kinds. +- JSONB + GIN = fast `annotations->>'key'` queries without bespoke tables per integration. -### Why PostgreSQL + JSONB? +The combination gives schema flexibility without sacrificing transactional safety. Considered MongoDB; rejected because the schema flexibility is already obtained via protobuf and PostgreSQL's consistency story is stronger. -**Decision**: Use PostgreSQL with protobuf BYTEA storage and JSONB annotations. +### Why compile-time plugins -**Rationale**: -- **Scale**: Handles 1000s of entities easily (tested: 170k resources) -- **JSONB performance**: GIN indexes enable fast annotation queries -- **Familiarity**: Most teams already run PostgreSQL -- **Reliability**: ACID transactions, mature ecosystem -- **Future-proof**: Can add read replicas, partitioning if needed +- Rust type system enforces handler signatures; no runtime trait-object surprises. +- No dynamic loader to maintain. +- No version-skew matrix between plugin and core. +- The cost is a rebuild to add a plugin — acceptable for infrastructure tooling that's already redeployed in pipelines. -**Alternative considered**: MongoDB -- **Verdict**: PostgreSQL with JSONB provides same flexibility with better consistency guarantees +### Why post-persistence events -### Why Compile-Time Plugins? +- Plugin failures can't roll back a successful entity write. +- Long-running external API calls don't block the gRPC response. +- New plugins drop in without touching CRUD code paths. -**Decision**: Plugins are compile-time integrated, not runtime loaded. +The trade-off: plugins can't veto an entity creation. Validation that must block creation belongs in core (`validate_entity`) or in the client. -**Rationale**: -- **Type safety**: Rust compiler ensures correctness -- **Performance**: No dynamic loading overhead -- **Simplicity**: No plugin version compatibility matrix -- **Security**: No arbitrary code execution -- **Trade-off**: Requires rebuild to add plugins (acceptable for infrastructure tool) +### Why rustls (no OpenSSL) -**Community plugins**: Published as crates, teams include in their build. +The entire TLS stack (mTLS server, tonic transport, reqwest HTTP client in plugins) uses `rustls`. No `openssl-sys`, no `pkg-config`, no `libssl-dev` — the binary is statically linkable, builds reproducibly in slim containers, and avoids a class of CVEs from OpenSSL ABI breakage. -### Why Event-Driven? +## Scale Reference -**Decision**: Plugins react to events after entity persistence. +These are *target* numbers based on a 100-engineer reference org, not benchmarks: -**Rationale**: -- **Decoupling**: Plugins can't break core CRUD operations -- **Resilience**: Plugin failures are logged, not propagated -- **Async**: Long-running operations don't block API -- **Extensibility**: Add plugins without modifying core -- **Observability**: Events provide audit trail +| Dimension | Reference value | +|---|---| +| Components / APIs | ~1k each | +| Users / Groups | ~700 / ~200 | +| Systems / Domains | ~50 / ~150 | +| Resources | ~170k | +| PostgreSQL storage | ~500 MB | +| Charybdis memory | ~256 MB | -**Trade-off**: Plugins can't prevent entity creation (validation must be in core or client). +Bottlenecks anticipated (none observed in production yet): +- JSONB annotation queries → mitigated by the GIN index. +- Event bus throughput → mitigated by a durable backend (planned). +- Plugin external-API rate limits → per-plugin concern, not core. ---- +## File Map -## Scale Estimates - -### Reference Deployment - -**Company**: ~100 engineers -**Backstage Entities**: -- Components: ~1,000 -- APIs: ~1,000 -- Users: ~700 -- Groups: ~200 -- Systems: ~50 -- Domains: ~150 -- Resources: ~170,000 - -**Charybdis Requirements**: -- PostgreSQL: ~500MB storage (with JSONB annotations) -- Memory: ~256MB for Charybdis service -- CPU: Minimal (mostly I/O bound) - -**10x Scale** (1,000 engineers, 1.7M resources): -- PostgreSQL: ~5GB storage -- Memory: ~512MB -- CPU: Still minimal with proper indexing - -**Bottlenecks**: -- JSONB queries on annotations (solved with GIN indexes) -- Event bus throughput (solved with Redis/RabbitMQ backend) -- Plugin external API rate limits (solved with plugin-level queuing) - ---- - -## Development Roadmap - -> See [VISION.md](VISION.md) for the full product vision and strategic roadmap. - -### Phase 0: Catalog Foundation (Done) - -- ✅ Zero-migration database architecture -- ✅ gRPC API with protobuf (full entity CRUD) -- ✅ Event bus system (MemoryEventBus) -- ✅ Event dispatcher architecture (generic + plugin-specific) -- ✅ All Backstage entity kinds (Component, System, API, User, Group, Domain, Resource) -- ✅ YAML adapter endpoints (Backstage compatibility) -- ✅ gRPC field masks for partial updates -- ✅ OpenTelemetry observability (traces, metrics, logs) -- ✅ mTLS + RBAC security framework -- ✅ Config loading from TOML with env var substitution -- ✅ Plugin trait architecture (EventDriven + Sync) -- ✅ Generic plugin utilities (HttpClient, AnnotationHelper, FieldMapper, DateUtils) -- ✅ DefectDojo plugin (products, engagements, owner resolution) -- ✅ Keycloak plugin (user/group sync with annotations) - -### Phase 1: Security Core (Current Focus) - -**Goal**: Native vulnerability management and scan ingestion. - -- [ ] Vulnerability/Observation entity kind (protobuf + storage) -- [ ] ScanService gRPC endpoint for scan ingestion -- [ ] SARIF parser (covers majority of modern scanners) -- [ ] CycloneDX parser (SBOMs + vulnerability data) -- [ ] SPDX parser (license data) -- [ ] Assessment workflow (triage, accept risk, remediate) -- [ ] Rules engine for auto-assessment -- [ ] Security gates (severity thresholds per product) -- [ ] License tracking and policy engine - -### Phase 2: Compliance & Integrations - -**Goal**: Compliance frameworks and integration plugins. - -- [ ] Compliance framework mappings (NIS2, SOC2, DORA) -- [ ] VEX document support (CSAF, OpenVEX) -- [ ] Export/reporting (PDF, Excel) -- [ ] Notification plugins (Slack, Teams, email) -- [ ] Issue tracker plugins (Jira, GitHub, GitLab) -- [ ] Redis event bus backend - -### Phase 3: Scaffolder & Ecosystem - -**Goal**: Service scaffolding and community growth. - -- [ ] Service scaffolder (Git-native templates) -- [ ] Event-driven provisioning on scaffold -- [ ] Plugin SDK documentation -- [ ] Helm chart & 1-click deploy -- [ ] Community plugin registry - ---- - -## Success Criteria - -### Phase 1 Success (Security Core) - -- [ ] Ingest SARIF scan results via gRPC -- [ ] Vulnerabilities linked to catalog entities -- [ ] Rules engine auto-assesses based on severity/component -- [ ] Security gates block on threshold violations -- [ ] Assessment workflow (triage → accept/remediate) -- [ ] License data ingested from CycloneDX/SPDX - -### Production Success - -- [ ] 1,000+ entities managed with security posture -- [ ] <100ms p99 latency for CRUD operations -- [ ] Single `helm install` deployment -- [ ] Documentation complete -- [ ] 3+ integration plugins in production - ---- - -## Key Decisions Summary - -| Decision | Choice | Rationale | -|----------|--------|-----------| -| **API Protocol** | gRPC only | Strong typing, multi-language, performance | -| **Frontend** | Backstage YAML adapter | Compatible with existing Backstage deployments | -| **Security features** | Native (core) | First-class, not plugins. Vulns, compliance, gates in the core | -| **External integrations** | Plugins | Slack, Jira, etc. as event-driven plugins | -| **Database** | PostgreSQL + JSONB | Scale, familiarity, ACID | -| **Plugin Loading** | Compile-time | Type safety, performance, security | -| **Event Model** | Post-persistence | Resilience, decoupling | -| **Schema Evolution** | Protobuf only | Zero migrations | -| **Configuration** | TOML + env vars | Flexibility, Rust-native | -| **Deployment** | Single binary + Docker + K8s | Cloud-native, minimal footprint | - ---- +| Path | Responsibility | +|---|---| +| `proto/entities.proto` | Generated entity schema (do not hand-edit) | +| `proto/entities.proto.template` | Template the generator expands | +| `proto/core/*.proto` | Per-kind metadata + spec messages | +| `proto/ingestion.proto` | `IngestionService` | +| `plugins.toml` | Plugin registry + protobuf field-number allocation | +| `build.rs` | Generates `entities.proto`, compiles all protos | +| `src/lib.rs` | `MyEntityService` (gRPC EntityService impl) + validation | +| `src/database.rs` | `EntityRepository`, schema bootstrap, field-mask paths | +| `src/findings/` | Ingestion + reconciliation + fingerprint | +| `src/scanners/` | `ScannerParser` trait + SARIF parser | +| `src/events/` | EventBus, EventDispatcher | +| `src/plugins/` | Plugin traits, utilities (HTTP client, annotation helper, field mapper), dispatcher | +| `src/security/` | mTLS, RBAC, AuthInterceptor | +| `src/adapters/yaml/` | Backstage YAML adapter | +| `src/telemetry/` | OpenTelemetry setup | +| `charybdis-server/src/main.rs` | Binary entry point — wires plugins, services, mTLS, YAML adapter | +| `plugins//` | Plugin crates (`defectdojo`, `keycloak`, `dependencytrack`) | ## References @@ -967,19 +421,3 @@ grpc-gateway or Envoy → Charybdis gRPC - [gRPC Documentation](https://grpc.io/docs/) - [Protocol Buffers Guide](https://protobuf.dev/) - [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html) - ---- - -## Document Version History - -| Version | Date | Changes | -|---------|------|---------| -| 1.0 | 2025-01-03 | Initial comprehensive architecture document | -| 1.1 | 2025-01-15 | Added dual plugin architecture, generic utilities, updated roadmap | -| 2.0 | 2026-04-02 | Pivoted to security-native platform engineering tool. Extracted product vision to VISION.md. Added native vulnerability management, compliance frameworks to architecture. Updated roadmap and success criteria. | - ---- - -**This document covers architecture and technical decisions.** -For product vision and strategic direction, see [VISION.md](VISION.md). -All code and documentation must align with these documents. diff --git a/docs/core-concepts.md b/docs/core-concepts.md index be0de79..adbca8e 100644 --- a/docs/core-concepts.md +++ b/docs/core-concepts.md @@ -1,126 +1,54 @@ # Core Concepts -Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically. +Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes. -This page explains the key concepts you need to understand. - -## Architecture Overview - -```mermaid -graph TB - subgraph "Sources" - A1[CI/CD Pipelines] - A2[Security Scanners] - A3[IaC Tools] - end - - subgraph "Charybdis Core" - B1[gRPC API] - B3[Authentication & Authorization] - C1[Entity Service] - C4[Vulnerability Engine] - C5[Security Gates & Rules] - C2[Event Bus] - end - - subgraph "Integrations - Plugins" - D1[Slack / Teams] - D2[Jira / GitHub] - D3[Custom Plugins] - end - - subgraph "Data Layer" - E1[(PostgreSQL)] - end - - A1 -.gRPC.-> B1 - A2 -.Scan Results.-> B1 - A3 -.gRPC.-> B1 - - B1 --> B3 - B3 --> C1 - B3 --> C4 - C4 --> C5 - C1 --> E1 - C4 --> E1 - C1 --> C2 - C4 --> C2 - - C2 -.Events.-> D1 - C2 -.Events.-> D2 - C2 -.Events.-> D3 - - style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff - style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff - style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff -``` +For a higher-level overview see the [README](../README.md). For deep technical details see [architecture.md](architecture.md). ## Entities -An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem. +An **entity** is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding. -### Entity Structure +### Entity structure -Every entity has three main parts: +Every entity has the same shape: -```mermaid -classDiagram - class Entity { - +string id - +string kind - +metadata - +spec - +annotations - +timestamps - } - - class Metadata { - +string name - +string namespace - +string description - +labels - +links - +tags - } - - class Spec { - +string type - +string lifecycle - +string owner - +dependencies - } - - Entity --> Metadata - Entity --> Spec -``` +- `id` — UUID assigned by the server +- `kind` — one of the supported kinds (see below) +- `metadata` — identifying data (name, description, labels, tags, links) +- `spec` — kind-specific configuration (type, lifecycle, owner, dependencies, ...) +- `annotations` — arbitrary `string → string` map, typically used by plugins to store external IDs +- `created_at`, `updated_at` — server-managed timestamps -### Entity Kinds +`metadata` and `spec` are protobuf `oneof` fields whose variant matches `kind`. So a `Component` entity carries `component_metadata` + `component_spec`, a `User` carries `user_metadata` + `user_spec`, and so on. The on-wire JSON uses the variant name directly. -Charybdis supports all standard Backstage entity kinds: +### Entity kinds -| Kind | Description | Example | -|------|-------------|---------| -| **Service** | Individual microservices or applications | `payment-api` | -| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` | -| **System** | Collections of services working together | `e-commerce-platform` | -| **API** | Interfaces exposed by components | `payments-rest-api` | -| **User** | Individual people | `john.doe` | -| **Group** | Teams and organizational units | `team-payments` | -| **Domain** | Business domains | `payments`, `shipping` | -| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` | +| Kind | Purpose | +|---|---| +| `Component` | Services, applications, libraries, websites | +| `System` | Collections of components working together | +| `API` | Interfaces exposed by components | +| `User` | Individual people | +| `Group` | Teams and organizational units | +| `Domain` | Business domains | +| `Resource` | Infrastructure resources (databases, caches, queues) | +| `Finding` | Security findings ingested from scanners (see [Findings](#findings)) | -#### Example: Service +Per-kind protobuf definitions: `proto/core/*.proto`. + +### Example: Component ```json { - "kind": "Service", - "service_metadata": { + "kind": "Component", + "component_metadata": { "name": "payment-api", "namespace": "production", "description": "Payment processing service", - "labels": { "team": "payments" } + "labels": { "team": "payments", "tier": "critical" }, + "tags": ["api", "pci-dss"] }, - "service_spec": { + "component_spec": { "type": "service", "lifecycle": "production", "owner": "team-payments", @@ -129,500 +57,136 @@ Charybdis supports all standard Backstage entity kinds: } ``` -#### Example: Component +### Common metadata fields -```json -{ - "kind": "Component", - "component_metadata": { - "name": "auth-sdk", - "namespace": "shared" - }, - "component_spec": { - "type": "library", - "lifecycle": "production", - "owner": "platform-team" - } -} -``` +| Field | Type | Notes | +|---|---|---| +| `name` | string | Required. Unique with `kind` (composite index `idx_entities_kind_name`). | +| `namespace` | string | Logical grouping, e.g. `production`. | +| `description` | string | Human-readable. | +| `labels` | `map` | Categorization. | +| `tags` | `repeated string` | Free-form classification. | +| `links` | `repeated Link` | External URLs (dashboards, docs). | -#### Example: System +## Annotations -```json -{ - "kind": "System", - "system_metadata": { - "name": "e-commerce-platform", - "namespace": "production", - "description": "Complete e-commerce system" - }, - "system_spec": { - "owner": "platform-team", - "domain": "retail" - } -} -``` - -### Metadata Fields - -| Field | Type | Description | Required | -|-------|------|-------------|----------| -| `name` | string | Entity name (unique within namespace) | ✅ | -| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ | -| `description` | string | Human-readable description | ❌ | -| `labels` | map | Key-value pairs for categorization | ❌ | -| `tags` | array | Search tags | ❌ | -| `links` | array | External URLs (dashboards, docs, etc.) | ❌ | - -### Annotations - -Annotations store integration-specific metadata: +Annotations are an open `map` on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB. ```json { "annotations": { "github.com/repo-slug": "myorg/payment-service", "defectdojo.com/product-id": "123", - "dependencytrack.com/project-uuid": "550e8400...", - "pagerduty.com/service-id": "PXYZ123", - "grafana.com/dashboard-url": "https://..." + "keycloak.com/email": "alice@example.com", + "pagerduty.com/service-id": "PXYZ123" } } ``` -**Best Practices**: -- Use domain-style keys (`tool.com/key`) -- Store tool-specific IDs -- Keep values as strings -- Use for integration metadata only +Convention: reverse-DNS keys (`tool.com/key`). Values are always strings. + +`EntityRepository::update_annotations()` performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers. ## Events -Charybdis uses an **event-driven architecture** to trigger actions when entities change. +Charybdis emits an `EntityEvent` on every CRUD operation, delivered to subscribers via the in-memory event bus. -### Event Flow - -```mermaid -sequenceDiagram - participant Client - participant API as Entity Service - participant Bus as Event Bus - participant Plugin1 as DefectDojo Plugin - participant Plugin2 as Dependency-Track - - Client->>API: CreateEntity(service) - API->>API: Store entity - API->>Bus: Emit EntityCreated event - Bus->>Plugin1: Handle event - Bus->>Plugin2: Handle event - Plugin1-->>Plugin1: Create DefectDojo product - Plugin2-->>Plugin2: Create DT project - API-->>Client: Return created entity - - Note over Bus,Plugin2: Asynchronous processing +``` +Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers ``` -### Event Types +### Event types -| Event | Trigger | Plugins Receive | -|-------|---------|----------------| -| `EntityCreated` | New entity created | Full entity data | -| `EntityUpdated` | Entity modified | Updated entity + changes | -| `EntityDeleted` | Entity removed | Entity ID + metadata | +| Type | When | +|---|---| +| `Created` | Successful `CreateEntity` | +| `Updated` | Successful `UpdateEntity` (full or partial) | +| `Deleted` | Successful `DeleteEntity` | -### Event Structure +### Event payload ```rust -pub enum EntityEvent { - Created { - entity: Entity, - timestamp: DateTime, - }, - Updated { - entity: Entity, - previous: Entity, - timestamp: DateTime, - }, - Deleted { - id: String, - metadata: Metadata, - timestamp: DateTime, - }, +pub struct EntityEvent { + pub event_id: Uuid, + pub entity_id: Uuid, + pub event_type: EntityEventType, // Created | Updated | Deleted + pub timestamp: DateTime, + pub metadata: HashMap, + pub entity_data: Option>, // full entity for Created/Updated } ``` -## Vulnerabilities & Security (Phase 1 — Planned) +### Delivery semantics -> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap. +- **Asynchronous** — `CreateEntity` returns to the client before plugin handlers complete. +- **At-least-once intent**, but the default `MemoryEventBus` is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in [VISION.md](../VISION.md). +- **Per-handler timeout** — 30 s. Slow plugins don't block the bus. +- **Panic isolation** — a panicking handler doesn't take down others. -Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core. +Plugin lifecycle and how to write a handler: [plugins/README.md](../plugins/README.md). -### Vulnerability Lifecycle +## Findings -```mermaid -sequenceDiagram - participant Scanner - participant API as Charybdis API - participant Rules as Rules Engine - participant Gate as Security Gate - participant Bus as Event Bus - participant Plugin as Slack / Jira +`Finding` is a first-class entity kind for security findings produced by scanners. Findings are created by the `IngestionService`, not directly by clients. - Scanner->>API: IngestScan(SARIF report) - API->>API: Parse & create Vulnerability entities - API->>API: Link vulnerabilities to Component - API->>Rules: Evaluate auto-assessment rules - Rules-->>API: Auto-assess (e.g., accept known low-risk) - API->>Gate: Check security gate thresholds - alt Gate Passed - Gate-->>API: OK - else Gate Failed - Gate->>Bus: GateFailed event - Bus->>Plugin: Alert #security channel - end - API->>Bus: VulnerabilitiesIngested event - Bus->>Plugin: Notify / create tickets -``` +Each finding is scoped to a `(component_ref, lifecycle)` pair — so production and staging environments track findings independently without duplicating the underlying `Component`. -### Scan Ingestion +The ingestion flow: -Charybdis ingests scan results natively. You don't need an external vulnerability management tool. +1. Client calls `IngestionService.ImportScan(component_ref, lifecycle, format, data)`. +2. The configured parser (currently SARIF) normalizes scanner output into `NormalizedFinding` records. +3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`) and diffs incoming vs existing findings for that scope. +4. The result is one of: + - **New** — fingerprint not seen before → create a `Finding` entity. + - **Unchanged** — already active → bump `last_seen`. + - **Resolved** — previously active, not in this scan → mark `Resolved`. + - **Reopened** — previously resolved or false-positive, detected again → mark `Reopened`. +5. `DryRunScan` runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments. -| Format | Coverage | Use Case | -|--------|----------|----------| -| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets | -| **CycloneDX** | SBOMs + vulnerability data | SCA, license | -| **SPDX** | License and package data | License compliance | +Finding state values (see `proto/core/finding.proto`): -### Assessments +| State | Meaning | +|---|---| +| `ACTIVE` | Currently detected | +| `RESOLVED` | No longer detected (auto-set on re-import) | +| `ACCEPTED` | Risk accepted by human decision *(state value defined; assessment workflow is planned)* | +| `FALSE_POSITIVE` | Marked as not a real issue *(state value defined; assessment workflow is planned)* | +| `REOPENED` | Was resolved, detected again | -Each vulnerability can be assessed: - -| Status | Meaning | -|--------|---------| -| **Open** | New, unreviewed vulnerability | -| **In Triage** | Under review by security team | -| **Accepted** | Risk accepted with justification | -| **Remediated** | Fixed, pending verification | -| **False Positive** | Not a real vulnerability | -| **Auto-Assessed** | Automatically assessed by rules engine | - -### Security Gates - -Security gates define thresholds per product: - -``` -payment-api: - critical: 0 # No critical vulns allowed - high: 5 # Up to 5 high - medium: 20 # Up to 20 medium -``` - -When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets). - -### Rules Engine - -Rules auto-assess vulnerabilities based on patterns: - -- Severity + component combination (e.g., "low severity in test dependencies → auto-accept") -- Scanner source (e.g., "all informational from ZAP → auto-accept") -- Known patterns (e.g., "CVE-XXXX already accepted org-wide") - -### License Compliance (Phase 2) - -Track licenses across your dependency tree: - -- Ingest license data from CycloneDX/SPDX -- Define license policies (allowed, restricted, banned) -- Flag violations per component -- Compliance reporting +Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md). ## Storage Model -Charybdis uses PostgreSQL with JSONB for schema-less storage. - -### Database Schema +Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a `oneof` variant in protobuf, not a column migration. ```sql CREATE TABLE entities ( - id UUID PRIMARY KEY, - kind TEXT NOT NULL, - entity_data BYTEA NOT NULL, -- Protobuf binary - annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed) - created_at TIMESTAMP NOT NULL, - updated_at TIMESTAMP NOT NULL + id UUID PRIMARY KEY, + kind VARCHAR NOT NULL, + name VARCHAR, + entity_data BYTEA NOT NULL, -- protobuf-encoded Entity + annotations JSONB NOT NULL DEFAULT '{}', + created_at TIMESTAMPTZ NOT NULL, + updated_at TIMESTAMPTZ NOT NULL ); -CREATE INDEX idx_entities_kind ON entities(kind); -CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations); +CREATE INDEX idx_entities_kind ON entities(kind); +CREATE INDEX idx_entities_kind_name ON entities(kind, name); +CREATE INDEX idx_entities_annotations ON entities USING GIN (annotations); +CREATE INDEX idx_entities_created_at_id ON entities(created_at DESC, id DESC); ``` -### Why Protobuf + JSONB? +Why this design: +- **No migrations** — new plugins extend the protobuf `oneof` variants; database schema is unchanged. +- **Compact storage** — protobuf binary is smaller than JSON. +- **Forward/backward compatible** — proto field numbers protect against version drift. +- **Fast annotation queries** — JSONB + GIN index supports `annotations->>'key' = 'value'` lookups in O(log n). -**Advantages**: -- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes -- ✅ Protobuf binary storage for compact, versioned entity data -- ✅ JSONB annotations for fast querying with GIN indexes -- ✅ Forward/backward compatibility built-in +Detailed schema reasoning, scaling notes, and field-mask paths: [architecture.md](architecture.md). -**Example Query**: -```sql --- Find entities with specific annotation -SELECT * FROM entities -WHERE annotations->>'defectdojo.com/product-id' = '42'; -``` +## See Also -## Security Model - -Charybdis implements defense-in-depth security. - -### Authentication: mTLS - -```mermaid -sequenceDiagram - participant Client - participant TLS as TLS Layer - participant Auth as Auth Interceptor - participant Service as Entity Service - - Client->>TLS: Connect with client certificate - TLS->>TLS: Validate certificate - TLS->>Auth: Extract certificate - Auth->>Auth: Parse identity (CN, OU, O) - Auth->>Auth: Map to role - Client->>Auth: Request (with identity) - Auth->>Auth: Check permissions - alt Authorized - Auth->>Service: Forward request - Service-->>Client: Response - else Denied - Auth-->>Client: PermissionDenied error - end -``` - -### Authorization: RBAC - -**Role Mapping**: - -``` -Certificate (CN, OU, O) → RBAC Role → Permissions -``` - -**Default Roles**: - -| Role | Certificate OU | Permissions | -|------|---------------|-------------| -| `platform` | `platform-team` | Full access (CRUD + list) | -| `automation` | `automation` | Create, read, update, list | -| `plugin` | `plugins` | Read, list only | - -### Permissions - -| Permission | Operations | Required For | -|------------|-----------|--------------| -| `entity:create` | Create new entities | CreateEntity | -| `entity:read` | Get entity by ID | GetEntity | -| `entity:update` | Modify entities | UpdateEntity | -| `entity:delete` | Remove entities | DeleteEntity | -| `entity:list` | List all entities | ListEntities | - -## Plugin System - -Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems. - -### Plugin Architecture - -```mermaid -graph LR - A[Entity/Vuln Event] --> B[Event Bus] - B --> C{Plugin Manager} - C --> D[Slack Plugin] - C --> E[Jira Plugin] - C --> F[Custom Plugin] - - D --> G[Slack API] - E --> H[Jira API] - F --> I[Your Tool API] - - style C fill:#4A90E2,stroke:#2E5C8A -``` - -### What's Native vs. Plugin - -| Native (core) | Status | Plugin (integration) | Status | -|---|---|---|---| -| Software catalog | Done | DefectDojo sync | Done | -| Vulnerability management | Phase 1 | Keycloak user sync | Done | -| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned | -| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned | -| License compliance | Phase 2 | Custom integrations | Framework ready | -| TechDocs | Future | | | - -### Plugin Lifecycle - -1. **Configuration** - Load plugin settings from `plugins.toml` -2. **Initialization** - Plugin registers event handlers -3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events) -4. **External Integration** - Plugin calls external tool APIs -5. **Error Handling** - Failed plugins don't affect core service - -### Plugin Configuration - -Example `plugins.toml`: - -```toml -[plugins.slack] -enabled = true -webhook_url = "${SLACK_WEBHOOK_URL}" - -[[plugins.slack.on_gate_failed]] -channel = "#security-alerts" - -[[plugins.slack.on_entity_created]] -channel = "#platform" -``` - -See [Plugin Guide](plugins.md) for details. - -## Data Consistency - -### Eventual Consistency - -Charybdis uses **eventual consistency** for plugin integrations: - -- Entity CRUD operations are **immediately consistent** -- Plugin synchronization is **eventually consistent** -- Events are processed **asynchronously** - -```mermaid -graph LR - A[CreateEntity] -->|Immediate| B[Entity Stored] - B -->|Async| C[Event Emitted] - C -->|Async| D[Plugin Processing] - D -->|Eventual| E[External Tool Synced] - - style B fill:#00C851 - style E fill:#ffbb33 -``` - -### Guarantees - -| Operation | Consistency | Guarantee | -|-----------|-------------|-----------| -| Entity CRUD | Strong | Immediate | -| Entity queries | Strong | Read-your-writes | -| Event delivery | At-least-once | May retry | -| Plugin sync | Eventual | Best-effort | - -## Performance Characteristics - -### Scalability - -- **Entities**: Tested with 100,000+ entities -- **Throughput**: 1,000+ requests/second -- **Latency**: Sub-millisecond average -- **Concurrency**: Tokio async runtime - -### Resource Usage - -Typical resource consumption: - -| Component | CPU | Memory | Storage | -|-----------|-----|--------|---------| -| Charybdis | < 5% | ~50 MB | Minimal | -| PostgreSQL | ~10% | ~256 MB | Depends on entity count | - -### Optimization Tips - -1. **Index annotations** used for frequent queries -2. **Use connection pooling** (built-in) -3. **Enable query caching** in PostgreSQL -4. **Monitor event bus** queue depth - -## Backstage Migration - -Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface. - -### How It Works - -```mermaid -graph LR - A[Charybdis Entity] --> B[YAML Adapter] - B --> C[Backstage YAML Format] - C --> D[Backstage Catalog] - - style B fill:#4A90E2,stroke:#2E5C8A -``` - -Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files: - -```yaml -catalog: - locations: - - type: url - target: http://charybdis:8080/yaml/locations -``` - -### Recommended Migration Path - -1. **Start** with Charybdis + YAML adapter feeding your existing Backstage -2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning -3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.) -4. **Optionally retire** Backstage when Charybdis covers your catalog needs - -## Best Practices - -### Entity Design - -✅ **DO**: -- Use descriptive names -- Group by namespace -- Add relevant labels -- Include documentation links -- Set appropriate owners - -❌ **DON'T**: -- Store sensitive data in metadata -- Use very long descriptions -- Create deeply nested hierarchies -- Duplicate data across entities - -### Naming Conventions - -``` --- - -Examples: -- payment-api-prod -- user-service-staging -- auth-library -- e-commerce-system -``` - -### Metadata Organization - -```json -{ - "labels": { - "team": "payments", // Ownership - "tier": "critical", // Importance - "environment": "production" // Deployment - }, - "tags": ["pci-compliant", "public-api"], - "links": [ - { "url": "...", "title": "Dashboard" }, - { "url": "...", "title": "Documentation" } - ] -} -``` - -## Next Steps - -- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going -- Configure [Security](security.md) for production (mTLS + RBAC) -- Explore [Plugins](plugins.md) for external integrations -- Review the [Architecture](architecture.md) for technical deep dive - ---- - -**Questions?** [Open an issue](../../issues). +- [Architecture](architecture.md) — protobuf schema, scaling, design decisions +- [Security](security.md) — mTLS + RBAC configuration +- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own +- [Vision & Roadmap](../VISION.md) — planned features diff --git a/docs/getting-started.md b/docs/getting-started.md index 61116aa..2c340d9 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,39 +1,26 @@ # Getting Started with Charybdis -This guide will get you from zero to a running Charybdis instance in minutes. By the end, you'll have a working software catalog that can register services via gRPC, ingest scan results, and auto-trigger integrations via event-driven plugins. +This guide gets you from zero to a running Charybdis instance: a software catalog with a gRPC API, optional Backstage YAML adapter, and plugin integrations on entity events. ## Prerequisites -- **Rust** - 1.70 or later ([install](https://rustup.rs/)) -- **PostgreSQL** - 14 or later -- **grpcurl** - For testing (optional, [install](https://github.com/fullstorydev/grpcurl)) +- **Rust** 1.70+ ([install](https://rustup.rs/)) +- **PostgreSQL** 14+ +- **protoc** + `libprotobuf-dev` (for the build script) +- **grpcurl** for testing (optional, [install](https://github.com/fullstorydev/grpcurl)) -## Installation - -### Option 1: From Source +## Install ```bash -# Clone the repository -git clone https://github.com/charybdis-catalog/charybdis.git +git clone cd charybdis - -# Build cargo build --release - -# The binary will be at target/release/charybdis -``` - -### Option 2: Docker (Coming Soon) - -```bash -docker pull charybdis/charybdis:latest +# Binary at target/release/charybdis-server ``` ## Quick Start -### Step 1: Start PostgreSQL - -Using Docker: +### 1. Start PostgreSQL ```bash docker run -d \ @@ -43,54 +30,22 @@ docker run -d \ postgres:15 ``` -Or use an existing PostgreSQL instance. - -### Step 2: Configure Charybdis - -**Recommended: Use config.toml** - -Copy the example configuration: +### 2. Configure ```bash cp config.toml.example config.toml -``` - -Edit `config.toml` and set your database URL: - -```toml -[database] -url = "${DATABASE_URL}" -``` - -Set the environment variable: - -```bash export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres" ``` -**Alternative: Environment Variables Only (Legacy)** +`config.toml` reads `${DATABASE_URL}` from the environment. See [Configuration](#configuration) below for all options. -If you prefer environment variables: - -```bash -# Database -export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres" - -# Disable security for quick start -export SECURITY_MTLS_ENABLED=false -export SECURITY_RBAC_ENABLED=false - -# Logging -export RUST_LOG=info,charybdis=debug -``` - -### Step 3: Run Charybdis +### 3. Run ```bash cargo run ``` -You should see: +Expected startup logs: ``` INFO charybdis: Database ready @@ -98,40 +53,34 @@ INFO charybdis: Event bus started successfully INFO charybdis: EntityService server listening on [::1]:50051 ``` -### Step 4: Verify It's Working - -Test with grpcurl: +### 4. Verify ```bash -# List available services grpcurl -plaintext localhost:50051 list - -# Output: # charybdis.entities.EntityService +# charybdis.ingestion.IngestionService # grpc.reflection.v1.ServerReflection ``` -Congratulations! Charybdis is running! 🎉 - -## Creating Your First Entity - -### Using grpcurl - -Create a service entity: +## Create Your First Entity ```bash -grpcurl -plaintext \ - -d '{ - "entity": { - "kind": "Service", - "service_metadata": { - "name": "payment-service", - "namespace": "production", - "description": "Core payment processing service" - } +grpcurl -plaintext -d '{ + "entity": { + "kind": "Component", + "component_metadata": { + "name": "payment-api", + "namespace": "production", + "description": "Payment processing service", + "tags": ["api", "critical"] + }, + "component_spec": { + "type": "service", + "lifecycle": "production", + "owner": "team-payments" } - }' \ - localhost:50051 charybdis.entities.EntityService/CreateEntity + } +}' localhost:50051 charybdis.entities.EntityService/CreateEntity ``` Response: @@ -140,117 +89,60 @@ Response: { "entity": { "id": "550e8400-e29b-41d4-a716-446655440000", - "kind": "Service", - "serviceMetadata": { - "name": "payment-service", - "namespace": "production", - "description": "Core payment processing service" - }, - "createdAt": "2025-11-04T10:00:00Z", - "updatedAt": "2025-11-04T10:00:00Z" + "kind": "Component", + "componentMetadata": { "...": "..." }, + "createdAt": "2026-06-01T10:00:00Z", + "updatedAt": "2026-06-01T10:00:00Z" } } ``` -### List All Entities - -```bash -grpcurl -plaintext -d '{}' \ - localhost:50051 charybdis.entities.EntityService/ListEntities -``` - -### Get Entity by ID +### Retrieve and list ```bash +# Get by ID grpcurl -plaintext \ -d '{"id": "550e8400-e29b-41d4-a716-446655440000"}' \ localhost:50051 charybdis.entities.EntityService/GetEntity + +# List all +grpcurl -plaintext -d '{}' \ + localhost:50051 charybdis.entities.EntityService/ListEntities + +# Filter by kind and name +grpcurl -plaintext \ + -d '{"kind": "Component", "name": "payment-api"}' \ + localhost:50051 charybdis.entities.EntityService/ListEntities ``` -## Entity Types +## Entity Kinds -Charybdis supports three main entity types: +The valid `kind` values are: `Component`, `System`, `API`, `User`, `Group`, `Domain`, `Resource`, `Finding`. Each kind uses a matching `_metadata` + `_spec` payload. -### Service +Conceptual reference: [core-concepts.md](core-concepts.md). Per-kind protobuf definitions: `proto/core/*.proto`. -Individual microservices or applications: - -```bash -grpcurl -plaintext -d '{ - "entity": { - "kind": "Service", - "service_metadata": { - "name": "user-api", - "namespace": "production", - "description": "User management API" - } - } -}' localhost:50051 charybdis.entities.EntityService/CreateEntity -``` - -### System - -Collections of related services: - -```bash -grpcurl -plaintext -d '{ - "entity": { - "kind": "System", - "system_metadata": { - "name": "payment-system", - "namespace": "production", - "description": "Complete payment processing system" - } - } -}' localhost:50051 charybdis.entities.EntityService/CreateEntity -``` - -### Component - -Reusable components or libraries: +### Component example with metadata and annotations ```bash grpcurl -plaintext -d '{ "entity": { "kind": "Component", - "component_spec": { - "type": "library", - "lifecycle": "production", - "owner": "platform-team" - }, "component_metadata": { - "name": "auth-library", - "namespace": "shared", - "description": "Shared authentication library" - } - } -}' localhost:50051 charybdis.entities.EntityService/CreateEntity -``` - -## Adding Metadata - -Entities support rich metadata: - -```bash -grpcurl -plaintext -d '{ - "entity": { - "kind": "Service", - "service_metadata": { "name": "payment-service", "namespace": "production", "description": "Payment processing", - "labels": { - "team": "payments", - "tier": "critical" - }, - "links": [ - { - "url": "https://dashboard.company.com/payments", - "title": "Dashboard", - "icon": "dashboard" - } - ], - "tags": ["payments", "pci-compliant", "critical"] + "labels": { "team": "payments", "tier": "critical" }, + "links": [{ + "url": "https://dashboard.company.com/payments", + "title": "Dashboard", + "icon": "dashboard" + }], + "tags": ["payments", "pci-compliant"] + }, + "component_spec": { + "type": "service", + "lifecycle": "production", + "owner": "team-payments" }, "annotations": { "github.com/repo-slug": "myorg/payment-service", @@ -260,105 +152,85 @@ grpcurl -plaintext -d '{ }' localhost:50051 charybdis.entities.EntityService/CreateEntity ``` -## Integrating with CI/CD +## Registering Entities from CI -### Example: GitHub Actions - -Create a workflow to register services automatically: +Charybdis is designed to be called from pipelines. With `grpcurl` available in your runner: ```yaml -name: Register Service -on: - push: - branches: [main] - -jobs: - register: - runs-on: ubuntu-latest - steps: - - name: Register in Charybdis - run: | - grpcurl -plaintext \ - -d '{ - "entity": { - "kind": "Service", - "service_metadata": { - "name": "${{ github.event.repository.name }}", - "namespace": "production", - "description": "${{ github.event.repository.description }}" - }, - "annotations": { - "github.com/repo-slug": "${{ github.repository }}" - } - } - }' \ - your-charybdis-host:50051 \ - charybdis.entities.EntityService/CreateEntity -``` - -### Example: GitLab CI - -```yaml -register_service: - stage: deploy - script: - - | - grpcurl -plaintext \ - -d "{ - \"entity\": { - \"kind\": \"Service\", - \"service_metadata\": { - \"name\": \"${CI_PROJECT_NAME}\", - \"namespace\": \"${CI_ENVIRONMENT_NAME}\" - } +# Example pipeline step (Gitea Actions / GitHub Actions syntax) +- name: Register service in Charybdis + run: | + grpcurl -plaintext \ + -d "{ + \"entity\": { + \"kind\": \"Component\", + \"component_metadata\": { + \"name\": \"$CI_PROJECT_NAME\", + \"namespace\": \"production\" + }, + \"component_spec\": { + \"type\": \"service\", + \"lifecycle\": \"production\", + \"owner\": \"$CI_PROJECT_NAMESPACE\" + }, + \"annotations\": { + \"repo-slug\": \"$CI_PROJECT_PATH\" } - }" \ - your-charybdis-host:50051 \ - charybdis.entities.EntityService/CreateEntity + } + }" \ + charybdis.internal:50051 \ + charybdis.entities.EntityService/CreateEntity ``` +In production, secure the endpoint with mTLS (see [Enabling Security](#enabling-security)). + ## Enabling Security -For production use, enable mTLS and RBAC: +Charybdis ships with mTLS + RBAC disabled for local exploration. For shared or production environments, enable both. -### Step 1: Generate Certificates +### 1. Generate dev certificates ```bash -# Use the provided test script -./test-mtls-rbac.sh +./deploy/scripts/generate-dev-certs.sh ``` -This creates: -- `certs/ca.pem` - Certificate Authority -- `certs/server-cert.pem` / `server-key.pem` - Server certificate -- `certs/admin-cert.pem` / `admin-key.pem` - Admin client certificate +This writes `deploy/certs/` with: +- `ca.pem` — CA +- `server-cert.pem` / `server-key.pem` — server +- `admin-cert.pem` / `admin-key.pem` — admin client +- (and per-role client certs) -### Step 2: Enable Security +### 2. Enable in `config.toml` -```bash -export SECURITY_MTLS_ENABLED=true -export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem -export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem -export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem -export SECURITY_RBAC_ENABLED=true +```toml +[security.mtls] +enabled = true +server_cert = "./deploy/certs/server-cert.pem" +server_key = "./deploy/certs/server-key.pem" +client_ca_cert = "./deploy/certs/ca.pem" + +[security.rbac] +enabled = true ``` -### Step 3: Test with mTLS +Restart Charybdis. RBAC defaults map cert OUs to roles (`platform-team` → full access, `automation` → CRUD without delete, `plugins` → read-only). + +### 3. Call with mTLS ```bash grpcurl \ - -cacert certs/ca.pem \ - -cert certs/admin-cert.pem \ - -key certs/admin-key.pem \ + -cacert deploy/certs/ca.pem \ + -cert deploy/certs/admin-cert.pem \ + -key deploy/certs/admin-key.pem \ -d '{}' \ localhost:50051 charybdis.entities.EntityService/ListEntities ``` -See the [Security Guide](security.md) for detailed configuration. +Full reference (custom role mappings, audit logging, reverse-proxy mode): [security.md](security.md). ## Backstage Migration (Optional) -If you currently use Backstage and want to migrate gradually, the built-in YAML adapter serves entities in Backstage format: +If you currently run Backstage, Charybdis serves entities in Backstage's Location YAML format: ```yaml # backstage app-config.yaml @@ -367,78 +239,36 @@ catalog: - type: url target: http://your-charybdis-host:8080/yaml/locations rules: - - allow: [Component, System, Service] + - allow: [Component, System, API, User, Group] ``` -Backstage will automatically discover and import entities from Charybdis. You can run both in parallel — entities managed via gRPC are immediately visible in Backstage. - -## Troubleshooting - -### Port Already in Use - -``` -Error: transport error -``` - -**Solution**: Check if another process is using port 50051: - -```bash -lsof -ti:50051 -``` - -Kill the process or change the port: - -```bash -export GRPC_PORT=50052 -``` - -### Database Connection Failed - -``` -Error: password authentication failed -``` - -**Solution**: Verify your DATABASE_URL: - -```bash -# Test connection -psql "$DATABASE_URL" -c "SELECT 1;" -``` - -### Permission Denied (with security enabled) - -``` -Code: PermissionDenied -Message: Role 'X' does not have permission 'Y' -``` - -**Solution**: Check your certificate and role mappings. See [Security Guide](security.md). - -## Next Steps - -Now that you have Charybdis running: - -1. Learn about [Core Concepts](core-concepts.md) — entities, vulnerabilities, events -2. Read the [Vision & Roadmap](../VISION.md) — where Charybdis is going -3. Configure [Security](security.md) for production (mTLS + RBAC) -4. Explore [Plugins](../plugins/README.md) for external integrations +Backstage discovers entities by polling the endpoint. Entities created via gRPC are visible on the next poll. The YAML adapter is served by an HTTP listener separate from the gRPC port (default `:8080`). ## Configuration -Charybdis supports two configuration methods: +Charybdis loads its config from (in order): -### 1. Configuration File (Recommended) +1. `./config.toml` +2. `./charybdis.toml` +3. `/etc/charybdis/config.toml` +4. Environment variables (fallback) -Use `config.toml` for structured configuration: +### `config.toml` skeleton ```toml -# config.toml [server] grpc_host = "[::1]" grpc_port = 50051 +[server.yaml_adapter] +enabled = true +host = "0.0.0.0" +port = 8080 + [database] -url = "${DATABASE_URL}" # Environment variable substitution +url = "${DATABASE_URL}" +max_connections = 10 +connection_timeout_secs = 30 [security.mtls] enabled = false @@ -450,59 +280,52 @@ enabled = false service_name = "charybdis" environment = "development" enable_console = true -``` - -**Benefits:** -- ✅ Organized by section (server, database, security, telemetry, plugins) -- ✅ Environment variable substitution with `${VAR_NAME}` -- ✅ Comments and documentation inline -- ✅ Easy to version control (excluding secrets) -- ✅ No need to export dozens of environment variables - -**Using Environment Variables in config.toml:** - -```toml -[database] -url = "${DATABASE_URL}" # Will be substituted at runtime [plugins.defectdojo] -api_key = "${DEFECTDOJO_API_KEY}" # Secrets stay in environment +enabled = false +# see plugins/README.md for the full plugin reference ``` -Then set only the secrets: +`${VAR}` and `${VAR:-default}` substitution works in any string value — keep secrets in the environment, not in the file. -```bash -export DATABASE_URL="postgresql://..." -export DEFECTDOJO_API_KEY="secret-key" -``` +### Environment-variable fallback -### 2. Environment Variables (Legacy) - -If `config.toml` is not found, Charybdis falls back to environment variables: +If no config file is found, these env vars are read: | Variable | Default | Description | -|----------|---------|-------------| +|---|---|---| | `DATABASE_URL` | (required) | PostgreSQL connection string | -| `GRPC_HOST` | `[::1]` | gRPC server bind address | -| `GRPC_PORT` | `50051` | gRPC server port | -| `RUST_LOG` | `info` | Logging level | -| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS authentication | -| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC authorization | -| `OTEL_ENABLE_CONSOLE` | `true` | Enable console logging | +| `GRPC_HOST` | `[::1]` | gRPC bind address | +| `GRPC_PORT` | `50051` | gRPC port | +| `RUST_LOG` | `info` | Logging level filter | +| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS | +| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC | +| `OTEL_ENABLE_CONSOLE` | `true` | Console exporter | | `OTEL_SERVICE_NAME` | `charybdis` | Service name for telemetry | -### Configuration File Locations +See `config.toml.example` for the complete template. -Charybdis looks for configuration files in this order: +## Troubleshooting -1. `./config.toml` (current directory) -2. `./charybdis.toml` -3. `/etc/charybdis/config.toml` (Linux/Unix) +### Port already in use +``` +Error: transport error +``` +Find and free port 50051: `lsof -ti:50051 | xargs kill`, or set `GRPC_PORT=50052`. -If none are found, it uses environment variables. +### Database connection failed +Verify the URL: `psql "$DATABASE_URL" -c "SELECT 1;"`. -See `config.toml.example` for a complete configuration template with all options documented. +### Permission denied with security enabled +``` +Code: PermissionDenied +Message: Role 'X' does not have permission 'Y' +``` +Inspect your cert subject (OU determines the role) and the `[security.rbac.permissions]` table in `config.toml`. Reference: [security.md](security.md). ---- +## Next Steps -**Need help?** Check the [troubleshooting guide](troubleshooting.md) or [open an issue](../../issues). +- [Core Concepts](core-concepts.md) — entity model, events, annotations +- [Architecture](architecture.md) — protobuf schema, storage, event bus +- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own +- [Vision & Roadmap](../VISION.md) — where Charybdis is going diff --git a/docs/index.md b/docs/index.md deleted file mode 100644 index 2d40ad2..0000000 --- a/docs/index.md +++ /dev/null @@ -1,125 +0,0 @@ -# Charybdis Documentation - -**The security-native platform engineering tool.** - -## What is Charybdis? - -Charybdis is a platform engineering tool that unifies **software catalog**, **vulnerability management**, and **compliance posture** in a single event-driven platform. Built in Rust, deployed as a single binary. - -Instead of running Backstage + DefectDojo + Dependency-Track + a license scanner + a compliance spreadsheet, you run Charybdis. - -### The Problems It Solves - -1. **Fragmented tooling** — Your software catalog, vulnerability data, license info, and compliance evidence live in 5 different tools that don't talk to each other. Charybdis unifies them. - -2. **Static catalog data** — Traditional catalogs rely on YAML files that go stale within weeks. Charybdis is event-driven — CI/CD pipelines and IaC tools register and update entities via gRPC, so the catalog is always accurate. - -3. **Security as an afterthought** — In Backstage, security is a plugin. In Charybdis, every entity carries its vulnerability posture, license status, and compliance state natively. - -4. **Manual provisioning** — New service? Manually create entries in every tool. With Charybdis, one gRPC call catalogs the service and event-driven plugins handle the rest. - -5. **Compliance evidence assembly** — Compliance reporting pulls from real vulnerability and license data, not spreadsheets. - -## How It Works - -``` -CI/CD or Scanner ──gRPC──> Charybdis - ├── Catalogs the service (event-driven) ← Done - ├── Fires events to plugins (DefectDojo, ...) ← Done - ├── Ingests scan results (SARIF, CycloneDX) ← Phase 1 - ├── Evaluates security gates & rules ← Phase 1 - └── Exposes catalog via YAML adapter (Backstage) ← Done -``` - -**One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.** - -## Key Concepts - -| Concept | Description | -|---------|-------------| -| **Entity** | Anything in your software ecosystem: services, systems, components, APIs, users, groups, domains, resources | -| **Vulnerability** | *(Phase 1)* A security finding linked to an entity, ingested from scanner output (SARIF, CycloneDX) | -| **Assessment** | *(Phase 1)* The triage decision on a vulnerability: accept risk, remediate, auto-assessed by rules | -| **Security Gate** | *(Phase 1)* Severity thresholds per product — blocks deployments when violated | -| **Event Bus** | Publishes lifecycle events when entities or vulnerabilities change | -| **Plugin** | Reacts to events to integrate with external systems (Slack, Jira, GitHub, custom) | -| **Annotations** | Key-value metadata on entities for external references (e.g. `github.com/repo-slug`) | - -## Architecture - -```mermaid -graph LR - A[CI/CD / Scanners] -->|gRPC| B[Charybdis] - B -->|Native| C[Software Catalog] - B -->|Native| D[Vuln Management] - B -->|Native| E[Compliance] - B -->|Events| F[Plugins: Slack / Jira / Custom] - B -->|YAML| G[Backstage - optional] - - style B fill:#4A90E2,stroke:#2E5C8A,color:#fff - style D fill:#E24A4A,stroke:#8A2E2E,color:#fff - style E fill:#4AE28A,stroke:#2E8A5C,color:#fff -``` - -## Documentation - -### Getting Started -- [Installation & Quick Start](getting-started.md) — Get Charybdis running and register your first entity -- [Demo Stack](../deploy/DEMO.md) — Full demo with DefectDojo - -### Understanding Charybdis -- [Core Concepts](core-concepts.md) — Entities, vulnerabilities, events, and data model -- [Vision & Roadmap](../VISION.md) — Where Charybdis is going and why -- [Architecture](architecture.md) — Technical design decisions - -### Configuration -- [Security](security.md) — mTLS authentication and RBAC authorization -- [Plugin Configuration](PLUGIN_CONFIGURATION_GUIDE.md) — Setting up and configuring plugins - -### Extending Charybdis -- [Plugin Development](../plugins/README.md) — Build your own integration plugins - -## Quick Example - -Register a service from your CI/CD pipeline: - -```bash -grpcurl -plaintext -d '{ - "entity": { - "kind": "Component", - "component_metadata": { - "name": "payment-api", - "description": "Payment processing service" - }, - "component_spec": { - "type": "service", - "lifecycle": "production", - "owner": "team-payments" - } - } -}' charybdis:50051 charybdis.entities.EntityService/CreateEntity -``` - -**What happens next:** -- Entity stored in PostgreSQL with a UUID -- `EntityCreated` event published to the event bus -- Plugins react (e.g., DefectDojo creates a product automatically) -- Entity available via gRPC and YAML adapter - -No YAML file to write. No PR to open. No manual provisioning. - -## Backstage Migration - -Already using Backstage? Charybdis provides a YAML adapter for gradual migration. Point Backstage at Charybdis as a catalog source — entities registered via gRPC are immediately available in Backstage. - -```yaml -# backstage app-config.yaml -catalog: - locations: - - type: url - target: http://charybdis:8080/yaml/locations -``` - ---- - -**Ready to get started?** Head to the [Getting Started Guide](getting-started.md). diff --git a/docs/security.md b/docs/security.md index e5312c4..e0e0a63 100644 --- a/docs/security.md +++ b/docs/security.md @@ -67,17 +67,19 @@ export SECURITY_RBAC_ENABLED=true #### Development Certificates -Use the provided test script: +Use the provided dev-cert script: ```bash -./test-mtls-rbac.sh +./deploy/scripts/generate-dev-certs.sh ``` -This generates: -- `ca.pem` / `ca-key.pem` - Certificate Authority -- `server-cert.pem` / `server-key.pem` - Server certificate -- `admin-cert.pem` / `admin-key.pem` - Admin client (OU=platform-team) -- Various plugin certificates (OU=plugins) +This generates (under `deploy/certs/`): +- `ca.pem` / `ca-key.pem` — Certificate Authority +- `server-cert.pem` / `server-key.pem` — Server certificate +- `admin-cert.pem` / `admin-key.pem` — Admin client (OU=platform-team) +- Per-role client certs (OU=automation, OU=plugins) + +For an individual client cert without regenerating everything, use `./deploy/scripts/generate-client-cert.sh`. #### Production Certificates @@ -494,10 +496,9 @@ Charybdis security features support compliance requirements: ## Next Steps -- 🔌 Configure [Plugins](plugins.md) with proper certificates -- 🚀 Review [Deployment Guide](deployment.md) for production -- 📊 Set up [Monitoring](monitoring.md) for security events +- [Plugins](../plugins/README.md) — configure plugins to use mTLS client certs +- [Architecture](architecture.md) — interceptor placement and observability hooks --- -**Security Questions?** Open a [security issue](../../security) (for vulnerabilities, use private disclosure). +**Security Questions?** Open an issue (for vulnerabilities, use private disclosure). diff --git a/plugins/README.md b/plugins/README.md index 10b8088..f7a59fb 100644 --- a/plugins/README.md +++ b/plugins/README.md @@ -1,340 +1,513 @@ # Charybdis Plugins -This directory contains plugin implementations that extend Charybdis with integrations to external security and development tools. +Plugins extend Charybdis with integrations to external tools (security scanners, issue trackers, identity providers). Plugins are **compile-time integrated**: they live as Rust crates under `plugins/`, declare protobuf extensions in `plugins.toml`, and are linked into the `charybdis-server` binary at build time. -## What are Plugins? +This document covers: +- The plugin model (event-driven vs sync, traits, lifecycle) +- Configuring shipped plugins (DefectDojo, Keycloak) +- Writing a new plugin -Charybdis supports **two types of plugins**, both compile-time integrated: +## Table of Contents -### **1. Event-Driven Plugins** -React to entity lifecycle events (create, update, delete): -- **Example**: DefectDojo, DependencyTrack -- **Implement**: `EventDrivenPlugin` trait + `ResourceHandler` -- **Triggered by**: Entity CRUD operations -- **Use case**: Auto-create resources in external tools when entities are created +1. [Plugin Types](#plugin-types) +2. [Available Plugins](#available-plugins) +3. [Plugin Configuration](#plugin-configuration) +4. [Field Mapping System](#field-mapping-system) +5. [Entity Resolution](#entity-resolution) +6. [DefectDojo Reference](#defectdojo-reference) +7. [Keycloak Reference](#keycloak-reference) +8. [Writing a New Plugin](#writing-a-new-plugin) +9. [Plugin Lifecycle](#plugin-lifecycle) +10. [Annotations](#annotations) +11. [Troubleshooting](#troubleshooting) -### **2. Sync Plugins** -Pull data from external sources on a schedule: -- **Example**: Okta, Keycloak, Active Directory -- **Implement**: `SyncPlugin` trait -- **Triggered by**: Cron schedule or manual API call -- **Use case**: Sync users/groups from identity providers +## Plugin Types -## Generic Utilities +### Event-Driven Plugins +React to entity lifecycle events (`Created`, `Updated`, `Deleted`) — typically push data to an external system. -All plugins have access to reusable utilities in `src/plugins/`: +- Implement `EventDrivenPlugin` + one or more `ResourceHandler` per resource kind. +- Example: **DefectDojo** — Component creation → DefectDojo product + engagement. -- **`PluginHttpClient`** - Multi-auth HTTP client (Token, Bearer, API Key, Basic Auth) -- **`AnnotationHelper`** - Consistent API for storing/retrieving plugin metadata -- **`FieldMapper`** - Maps entity fields to external tool formats -- **`DateUtils`** - Common date/time operations +### Sync Plugins +Pull data from an external system on a schedule, creating or updating entities in Charybdis. -## Plugin Structure +- Implement `SyncPlugin` (`schedule`, `on_startup`, `manual_trigger`). +- Example: **Keycloak** — periodic sync of users and groups. -Each plugin follows this structure: - -``` -plugins/ -└── my_plugin/ - ├── Cargo.toml # Plugin crate definition - ├── README.md # Plugin-specific documentation - ├── proto/ - │ └── my_plugin.proto # Protobuf entity definitions - └── src/ - ├── lib.rs # Plugin entry point - ├── handler.rs # Event handler implementation - ├── client.rs # External API client - └── config.rs # Plugin configuration -``` +Both types implement the base `Plugin` trait (`name`, `validate_config`, `health_check`). ## Available Plugins -### DefectDojo -- **Type**: Event-Driven -- **Purpose**: Security vulnerability management integration -- **External API**: DefectDojo REST API v2 -- **Handlers**: ProductHandler, EngagementHandler, ProductTypeHandler, ProductMemberHandler -- **Features**: - - Automatic product creation on Component/Service creation - - Product updates on entity changes - - Auto-create CI/CD engagements - - Owner resolution (Group → members → email annotation → DefectDojo user lookup) - - Configurable OIDC provider mapping (Keycloak, Okta, Azure AD) - - Annotation storage (`defectdojo.com/product-id`, `defectdojo.com/engagement-id`, `defectdojo.com/owner-member-ids`) +| Plugin | Type | Status | +|---|---|---| +| DefectDojo | Event-driven | Shipped | +| Keycloak | Sync | Shipped | -### Keycloak -- **Type**: Sync -- **Purpose**: Identity provider sync (users and groups) -- **External API**: Keycloak Admin REST API -- **Features**: - - User sync with profile and annotations (display_name, email, picture) - - Group sync with hierarchy (parent, children, members) - - Configurable annotations (e.g., `keycloak.com/email` for OIDC mapping) - - On-startup sync, manual trigger via gRPC - -### DependencyTrack (Scaffolded) -- **Type**: Event-Driven -- **Purpose**: Software supply chain security -- **External API**: DependencyTrack REST API -- **Status**: Scaffolded (proto + crate structure), handler logic not implemented -- **Planned**: - - Auto-create projects for components - - SBOM ingestion - -## Creating a New Plugin - -### Quick Start - -1. **Create plugin directory structure:** - ```bash - mkdir -p plugins/my_plugin/{proto,src} - ``` - -2. **Add to plugins.toml:** - ```toml - [plugins.my_plugin] - enabled = true - proto_path = "plugins/my_plugin/proto" - metadata_field_number = 102 # Use next available number - spec_field_number = 102 - description = "My custom integration" - ``` - -3. **Define protobuf schema:** - Create `plugins/my_plugin/proto/my_plugin.proto`: - ```protobuf - syntax = "proto3"; - package charybdis.plugins.my_plugin; - - message MyPluginMetadata { - string name = 1; - string description = 2; - } - - message MyPluginSpec { - string integration_type = 1; - bool enabled = 2; - } - ``` - -4. **Create plugin crate:** - Create `plugins/my_plugin/Cargo.toml`: - ```toml - [package] - name = "charybdis-plugin-my-plugin" - version = "0.1.0" - edition = "2021" - - [dependencies] - charybdis = { path = "../.." } - async-trait = "0.1" - tokio = { version = "1.0", features = ["full"] } - tracing = "0.1" - ``` - -5. **Implement event handler:** - Create `plugins/my_plugin/src/lib.rs`: - ```rust - use async_trait::async_trait; - use charybdis::events::{EventHandler, EntityEvent, EventResult}; - - pub struct MyPluginHandler; - - #[async_trait] - impl EventHandler for MyPluginHandler { - async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> { - // React to entity changes - Ok(()) - } - } - ``` - -6. **Build and test:** - ```bash - cargo build - cargo test - ``` - -## Plugin Field Number Allocation - -Field numbers must be unique across all plugins to avoid protobuf conflicts: - -| Range | Allocation | -|----------|-----------------------| -| 1-99 | Core entity types | -| 100 | DefectDojo | -| 101 | DependencyTrack | -| 102-199 | Available for plugins | - -When creating a new plugin, use the next available number in the 102+ range. +Planned integrations (Slack, Jira, GitHub, Dependency-Track) are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md). ## Plugin Configuration -Plugins are configured in `config.toml` with `${VAR}` env var substitution for secrets: +Plugins are configured in `config.toml` under `[plugins.]`. Use `${VAR}` for secrets — never commit tokens. ```toml [plugins.defectdojo] enabled = true -base_url = "${DEFECTDOJO_API_URL}" +base_url = "${DEFECTDOJO_URL}" api_token = "${DEFECTDOJO_API_TOKEN}" -[plugins.defectdojo.default_engagement] -auto_create = true -name = "CI/CD Pipeline" - [plugins.keycloak] enabled = true base_url = "${KEYCLOAK_URL}" -realm = "master" +realm = "charybdis" client_id = "charybdis-sync" client_secret = "${KEYCLOAK_CLIENT_SECRET}" ``` -See `config.toml.example` for all options. +When `enabled = false`, the plugin is loaded but does not register handlers and does not react to events. See `config.toml.example` for the full reference. + +## Field Mapping System + +Event-driven plugins (currently DefectDojo) use a configurable mapper to translate Charybdis entity fields into external-tool API payloads. Mappings are defined per resource type under `[plugins..field_mappings.]`. + +Three mapping types are supported: + +### 1. Direct field mapping (string) +Dot-notation path into the entity. + +```toml +name = "metadata.name" +description = "metadata.description" +email = "spec.profile.email" +tags = "metadata.tags" +``` + +### 2. Static value (object with `value` key) + +```toml +business_criticality = { value = "high" } +is_active = { value = true } +priority = { value = 100 } +``` + +Supports strings, booleans, numbers, objects, and arrays. + +### 3. Entity resolution (object) +Resolve a reference into a related entity and extract a field from it. + +```toml +product_manager = { + from = "spec.owner", + resolve_entity = "User", + extract = "annotations.defectdojo.com/user-id" +} +``` + +| Parameter | Type | Required | Description | +|---|---|---|---| +| `from` | string | yes | Source field path | +| `resolve_entity` | string | yes | Entity kind to look up (`User`, `Component`, ...) | +| `extract` | string | yes | Field path to extract from the resolved entity | +| `resolve_array` | bool | no | Source field is an array of references | +| `lookup_entity` | string | no | Find an entity that references the source | + +## Entity Resolution + +Entity resolution lets a plugin navigate the entity graph at mapping time. The mapper performs a database lookup, extracts the target field, and substitutes it into the payload. + +``` +1. Source field spec.owner = "user:john.doe" +2. Resolve entity User entity for "john.doe" +3. Extract target field annotations.defectdojo.com/user-id = "123" +4. Result product_manager = 123 +``` + +### Array resolution + +```toml +user_ids = { + from = "spec.members", + resolve_entity = "User", + extract = "annotations.defectdojo.com/user-id", + resolve_array = true +} +``` + +`Group.spec.members = ["user:john", "user:jane"]` → `user_ids = [123, 124]`. + +### Linked entity lookup + +```toml +engagement_product = { + from = "id", + lookup_entity = "Component", + extract = "annotations.defectdojo.com/product-id" +} +``` + +Finds a `Component` that references the source entity and extracts its annotation. + +### Error handling +Resolution failures (entity not found, missing annotation, invalid reference) log a warning and omit the field rather than failing the whole operation. + +## DefectDojo Reference + +Maps Charybdis entities to DefectDojo resources via the REST API v2. Handlers shipped: `ProductHandler`, `EngagementHandler`, `ProductTypeHandler`, `ProductMemberHandler`. + +### Connection + +```toml +[plugins.defectdojo] +enabled = true +base_url = "${DEFECTDOJO_URL}" +api_token = "${DEFECTDOJO_API_TOKEN}" +default_product_type_id = 1 +auto_create_users = true +auto_create_product_types = false +``` + +To obtain the API token: in DefectDojo, **User Profile → API Key**. + +### Default engagement (auto-created with each product) + +```toml +[plugins.defectdojo.default_engagement] +auto_create = true +name = "CI/CD Pipeline" +description = "Automated security scans from CI/CD pipeline" +engagement_type = "CI/CD" +status = "In Progress" +duration_days = 365 +deduplication_on_engagement = true +``` + +### Owner resolution +When a `Component` has an `owner` (typically a team name), Charybdis resolves it to DefectDojo users: + +1. Find `Group` entity matching the owner. +2. Read `Group.spec.members` (usernames). +3. Look up each `User` entity. +4. Extract the user's email from the configured annotation. +5. Find the matching user in DefectDojo by email. + +```toml +[plugins.defectdojo.owner_resolution] +user_email_annotation = "keycloak.com/email" # depends on your IdP +defectdojo_lookup_field = "email" # "email" or "username" +assign_all_members = true # all group members, or just first match +``` + +### Product mapping (Component → DefectDojo Product) + +```toml +[plugins.defectdojo.field_mappings.product] +name = "metadata.name" +description = "metadata.description" +product_type_id = { value = 1 } +tags = "metadata.tags" +business_criticality = { value = "high" } # very high | high | medium | low | very low | none +platform = { value = "web" } # web | mobile | desktop | iot | ... +lifecycle = { value = "production" } +origin = { value = "internal" } +external_audience = { value = true } +internet_accessible = { value = true } + +product_manager = { + from = "spec.owner", + resolve_entity = "User", + extract = "annotations.defectdojo.com/user-id" +} +``` + +### User mapping (User → DefectDojo User) + +```toml +[plugins.defectdojo.field_mappings.user] +username = "metadata.name" +email = "spec.profile.email" +first_name = "spec.profile.displayName" +last_name = "spec.profile.displayName" +is_active = { value = true } +``` + +### Product Type mapping (System → DefectDojo Product Type) + +```toml +[plugins.defectdojo.field_mappings.product_type] +name = "metadata.name" +description = "metadata.description" +critical_product = { value = false } +key_product = { value = true } +``` + +### Product Member mapping (Group → DefectDojo Product Member) + +```toml +[plugins.defectdojo.field_mappings.product_member] +product_id = { + from = "spec.parent", + resolve_entity = "Component", + extract = "annotations.defectdojo.com/product-id" +} + +user_id = { + from = "spec.members", + resolve_entity = "User", + extract = "annotations.defectdojo.com/user-id", + resolve_array = true +} + +role_name = { value = "Reader" } # Owner | Maintainer | Writer | Reader | API_Importer +``` + +### Engagement mapping (Resource → DefectDojo Engagement) + +```toml +[plugins.defectdojo.field_mappings.engagement] +name = "metadata.name" +description = "metadata.description" + +product_id = { + from = "spec.owner", + resolve_entity = "Component", + extract = "annotations.defectdojo.com/product-id" +} + +lead_id = { + from = "spec.dependsOn", + resolve_entity = "User", + extract = "annotations.defectdojo.com/user-id" +} + +target_start = "spec.target_start" +target_end = "spec.target_end" +status = { value = "In Progress" } +engagement_type = { value = "CI/CD" } +version = "metadata.annotations.version" +commit_hash = "metadata.annotations.commit_hash" +branch_tag = "metadata.annotations.branch" +build_id = "metadata.annotations.build_id" +source_code_management_uri = "metadata.annotations.repo_url" + +deduplication_on_engagement = { value = true } +api_test = { value = true } +pen_test = { value = false } +``` + +### Annotations written by the plugin +- `defectdojo.com/product-id` +- `defectdojo.com/engagement-id` +- `defectdojo.com/owner-member-ids` + +## Keycloak Reference + +Sync plugin: pulls users and groups from a Keycloak realm via the Admin REST API on a schedule, on startup, or via manual trigger. + +```toml +[plugins.keycloak] +enabled = true +base_url = "${KEYCLOAK_URL}" +realm = "charybdis" +client_id = "charybdis-sync" +client_secret = "${KEYCLOAK_CLIENT_SECRET}" + +[plugins.keycloak.sync] +schedule = "0 */5 * * * *" # optional cron; omit to sync only on startup/trigger +on_startup = true +manual_trigger = true +sync_users = true +sync_groups = true +namespace = "keycloak" +page_size = 100 +``` + +The service account behind `client_id`/`client_secret` must have `view-users` and `view-groups` on the realm. + +Synced entities carry annotations such as `keycloak.com/email`, `keycloak.com/id`, used by other plugins (e.g., the DefectDojo owner-resolution chain above). + +## Writing a New Plugin + +### 1. Scaffold directory + +```bash +mkdir -p plugins/my_plugin/{proto,src} +``` + +### 2. Register in `plugins.toml` + +```toml +[plugins.my_plugin] +enabled = true +proto_path = "plugins/my_plugin/proto" +metadata_field_number = 103 # next available — see registry below +spec_field_number = 203 +description = "My custom integration" +``` + +### 3. Define the protobuf schema + +`plugins/my_plugin/proto/my_plugin.proto`: + +```protobuf +syntax = "proto3"; +package charybdis.plugins.my_plugin; + +message MyPluginMetadata { + string name = 1; + string description = 2; +} + +message MyPluginSpec { + string integration_type = 1; + bool enabled = 2; +} +``` + +`build.rs` regenerates `proto/entities.proto` automatically on next build. + +### 4. Create the crate + +`plugins/my_plugin/Cargo.toml`: + +```toml +[package] +name = "charybdis-my-plugin-plugin" +version = "0.1.0" +edition = "2024" + +[lib] +name = "charybdis_my_plugin" +path = "src/lib.rs" + +[dependencies] +charybdis = { path = "../.." } +async-trait = "0.1" +anyhow = "1.0" +serde = { version = "1.0", features = ["derive"] } +tokio = { version = "1.48", features = ["full"] } +tracing = "0.1" +``` + +Add the crate to the workspace `members` in the top-level `Cargo.toml`. + +### 5. Implement the plugin + +Event-driven example: + +```rust +use async_trait::async_trait; +use charybdis::plugins::{EventDrivenPlugin, Plugin, PluginConfig, PluginType, ResourceHandler}; +use std::sync::Arc; + +pub struct MyPlugin { /* config, client, handlers */ } + +#[async_trait] +impl Plugin for MyPlugin { + fn name(&self) -> &str { "my_plugin" } + fn plugin_type(&self) -> PluginType { PluginType::EventDriven } + fn load_config(&mut self, _: PluginConfig) -> anyhow::Result<()> { Ok(()) } + fn validate_config(&self) -> anyhow::Result<()> { Ok(()) } +} + +#[async_trait] +impl EventDrivenPlugin for MyPlugin { + fn resource_handlers(&self) -> Vec> { + vec![/* your handlers */] + } +} +``` + +### 6. Register in `charybdis-server` + +Add a loader block in `charybdis-server/src/main.rs` (follow the DefectDojo block as a template) and declare the dependency in `charybdis-server/Cargo.toml`. + +### Field number registry + +| Range | Allocation | +|---|---| +| 1–99 | Core entity types | +| 100 | DefectDojo | +| 101 | DependencyTrack | +| 102 | Keycloak | +| 103+ | Available | + +Field numbers MUST be unique across plugins — protobuf wire format depends on it. ## Plugin Lifecycle -1. **Build Time:** - - `build.rs` reads `plugins.toml` - - Generates `proto/entities.proto` with plugin types - - Compiles all protobuf files - - Generates Rust code +### Build time +1. `build.rs` reads `plugins.toml`. +2. Generates `proto/entities.proto` with plugin-contributed `oneof` variants. +3. Compiles all protobuf files (`tonic-prost-build`). -2. **Runtime:** - - Plugin handlers registered with event bus - - Events published on entity CRUD operations - - Handlers react asynchronously - - External APIs called as needed +### Runtime +1. `charybdis-server` loads each enabled plugin from `config.toml`. +2. Event-driven plugins register their `ResourceHandler`s with the dispatcher. +3. Sync plugins register with the cron scheduler (and optionally run once at startup). +4. Entity CRUD operations publish `EntityEvent` to the in-memory event bus. +5. The dispatcher fans out events to matching handlers (filtered by `trigger_kinds`). +6. Handlers call external APIs and write annotations back to Charybdis. -3. **Event Flow:** - ``` - Client creates entity - → Entity stored in database - → Event published to event bus - → Plugin handler receives event - → Plugin calls external API - → Plugin stores external ID in annotations - ``` - -## Using Annotations - -Plugins store external tool IDs in entity annotations: - -```rust -// Store external ID -entity.annotations.insert( - "my-plugin.com/resource-id".to_string(), - "ext-12345".to_string(), -); - -// Query by external ID in SQL -SELECT * FROM entities -WHERE annotations->>'my-plugin.com/resource-id' = 'ext-12345'; +### Event flow +``` +Client → gRPC CreateEntity → DB INSERT → EntityEvent::Created → Dispatcher + ↓ + ResourceHandler.handle_create(entity) + ↓ + POST /api/products → DefectDojo + ↓ + update_annotations("defectdojo.com/product-id") ``` -### Annotation Naming Convention +## Annotations + +Plugins store external IDs in entity annotations using reverse-DNS keys: -Use reverse-DNS style: - `defectdojo.com/product-id` -- `dependencytrack.com/project-uuid` -- `github.com/repo-slug` -- `{tool}.com/{resource}-{attribute}` +- `defectdojo.com/engagement-id` +- `keycloak.com/id` +- `keycloak.com/email` -## Best Practices +Query by annotation via JSONB in PostgreSQL: -### 1. Error Handling -- Don't panic - return errors -- Log but continue on non-critical failures -- Implement retry logic for transient errors - -### 2. Idempotency -- Check if resource exists before creating -- Make operations safe to retry -- Handle duplicate creation gracefully - -### 3. Performance -- Don't block event handlers -- Use `tokio::spawn` for long operations -- Batch operations when possible - -### 4. Testing -- Unit test event handlers -- Mock external API clients -- Integration tests with real APIs (optional) - -### 5. Documentation -- Document required environment variables -- Provide example configurations -- Explain entity model and annotations - -## Contributing Plugins - -We welcome plugin contributions! To contribute: - -1. Fork the repository -2. Create your plugin following the structure above -3. Add comprehensive tests -4. Document configuration and usage -5. Submit a pull request - -### Plugin Requirements - -- [ ] Protobuf definitions with Metadata and Spec messages -- [ ] Event handler implementation -- [ ] External API client (if applicable) -- [ ] Configuration via environment variables -- [ ] README with setup instructions -- [ ] Unit tests for event handler -- [ ] Example usage in documentation - -## Plugin Distribution Models - -### In-Tree (Current) -Plugins live in the `plugins/` directory and are enabled via `plugins.toml`. - -**Pros:** -- Easy to discover -- Consistent quality -- Tested together - -**Cons:** -- Requires core repo access -- All plugins built together - -### External Crates (Future) -Plugins distributed as separate Rust crates. - -**Example:** -```toml -[dependencies] -charybdis-plugin-custom = "0.1" +```sql +SELECT * FROM entities +WHERE annotations->>'defectdojo.com/product-id' = '456'; ``` -**Pros:** -- Independent versioning -- Community contributions -- Optional dependencies +Use `EntityRepository::update_annotations()` for atomic JSONB merge — it avoids the read-modify-write race when multiple handlers write to the same entity. -**Cons:** -- Discovery harder -- Compatibility challenges +## Troubleshooting -### Plugin Marketplace (Future) -Central registry of available plugins (like Backstage). +### Plugin not reacting to events +- `enabled = true` in `config.toml`? +- Plugin registered in `charybdis-server/src/main.rs`? +- Entity kind matches the handler's `trigger_kinds`? +- Event bus running? (`info!("Event bus started")` in startup logs) -## Resources +### Field mapping not applied +- Path uses dot notation, e.g. `metadata.name`, not `meta.name`. +- Static value uses `{ value = "..." }`, not a bare string when you want a literal. +- Field actually exists on the entity (inspect via the YAML adapter or `GetEntity` gRPC). -- [Plugin Configuration Guide](../docs/PLUGIN_CONFIGURATION_GUIDE.md) -- [Core Concepts](../docs/core-concepts.md) -- [Architecture](../docs/architecture.md) -- [Protobuf Style Guide](https://protobuf.dev/programming-guides/style/) +### Entity resolution returns nothing +- Resolved entity exists in the DB. +- Resolved entity has the `extract` annotation set. +- `resolve_entity` uses the PascalCase kind (`"User"`, not `"user"`). +- For arrays, `resolve_array = true` is set. -## Support +### DefectDojo API errors +- `400 Bad Request: Invalid product type ID` → check `default_product_type_id` or your `product_type_id` mapping. +- `401 Unauthorized` → check `DEFECTDOJO_API_TOKEN`. +- `409 Conflict: user already exists` → expected, the plugin reuses existing users. -- Open an issue for bug reports -- Discussions for questions -- PRs for contributions +### Performance +- Minimize entity-resolution chain depth. +- Prefer static values when the data isn't entity-bound. +- Check network latency to the external API — handlers run synchronously per event. -## License +## See Also -Same as Charybdis core (see LICENSE file in repository root) \ No newline at end of file +- [Core Concepts](../docs/core-concepts.md) — entity model, events, annotations. +- [Architecture](../docs/architecture.md) — protobuf schema, storage, scaling. +- [Field Mapper Tests](../src/plugins/field_mapper.rs) — reference behavior.