# Architecture This document describes the implementation of Charybdis: its protobuf schema, storage model, event flow, plugin lifecycle, and the rationale behind the major design decisions. For the product vision and roadmap, see [VISION.md](../VISION.md). For end-user concepts (entities, annotations, events), see [core-concepts.md](core-concepts.md). ## Architectural Principles ### 1. Zero database migrations The PostgreSQL schema is created on first startup and **never changes**. - Entities are serialized to protobuf and stored in `entity_data BYTEA`. - New entity kinds are added as new `oneof` variants in the protobuf schema, not as new columns. - Plugins extend the schema at build time by registering protobuf field numbers (see [plugin lifecycle](#plugin-lifecycle)). Deploying a new plugin requires a rebuild but no migration script. Rolling back is the same: rebuild without the plugin, the stored bytes for that variant are simply ignored. ### 2. gRPC is the only API Charybdis exposes two gRPC services and one HTTP adapter: - `EntityService` (`proto/entities.proto`) — full entity CRUD with field-mask partial updates. - `IngestionService` (`proto/ingestion.proto`) — `ImportScan` / `DryRunScan` for security findings. - HTTP YAML adapter — Backstage-compatible Location YAML on a separate port. No REST, no GraphQL in core. Teams that need them should put grpc-gateway or Envoy in front. ### 3. Plugins are compile-time integrated Plugins are Rust crates linked at build time into `charybdis-server`. Pros: type safety, no dynamic loader, no runtime version skew. Cons: adding a plugin requires a rebuild — acceptable for an infrastructure tool. ### 4. Event-driven, post-persistence Events fire **after** the database write succeeds. ``` CRUD request → validate → DB write → publish event → plugins react (async) ``` Plugin failures don't fail the original request. The entity is already persisted; plugins log their errors and the event bus moves on. ### 5. Backstage-compatible entity shape The protobuf entity model maps 1:1 onto Backstage's `apiVersion / kind / metadata / spec` structure. The YAML adapter renders this directly. This is a deliberate compatibility choice — teams can run Charybdis as a dynamic backend behind their existing Backstage frontend during migration. ## System Layout ``` ┌──────────────────────────────────────────────────────────────┐ │ Clients │ │ CI/CD pipelines · Scanners (SARIF) · IaC tools · Backstage │ └─────────────────┬────────────────────────────────────────────┘ │ gRPC (mTLS optional) HTTP (YAML) ▼ ▼ ┌──────────────────────────────────────────────────────────────┐ │ charybdis-server │ │ ┌────────────────────────┐ ┌──────────────────────────┐ │ │ │ EntityService │ │ IngestionService │ │ │ │ CreateEntity │ │ ImportScan │ │ │ │ GetEntity │ │ DryRunScan │ │ │ │ UpdateEntity (FM) │ └──────────┬───────────────┘ │ │ │ DeleteEntity │ │ │ │ │ ListEntities │ ▼ │ │ └──────────┬─────────────┘ ┌──────────────────────────┐ │ │ │ │ Findings pipeline │ │ │ │ │ Parser registry │ │ │ │ │ Reconciliation engine │ │ │ │ │ Fingerprint dedup │ │ │ │ └──────────┬───────────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ AuthInterceptor (mTLS + RBAC) │ │ │ └────────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌────────────────────▼───────────────────────────────────┐ │ │ │ EntityRepository (PostgreSQL, protobuf + JSONB) │ │ │ └────────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌────────────────────▼───────────────────────────────────┐ │ │ │ EventBus (MemoryEventBus) │ │ │ │ → EventDispatcher → plugin ResourceHandlers │ │ │ └────────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌────────────────────▼───────────────────────────────────┐ │ │ │ YAML Adapter (axum HTTP, separate port) │ │ │ └────────────────────────────────────────────────────────┘ │ │ │ │ Telemetry: OpenTelemetry traces/metrics/logs (console+OTLP) │ └──────────────────────────────────────────────────────────────┘ │ ▼ external API calls Plugins: DefectDojo, Keycloak ``` ## Entity Schema Defined in `proto/entities.proto` (generated by `build.rs` from `entities.proto.template` + `plugins.toml`). ```protobuf message Entity { string id = 1; // UUID, server-assigned string kind = 2; // "Component", "Finding", ... oneof metadata { charybdis.core.ComponentMetadata component_metadata = 12; charybdis.core.ServiceMetadata service_metadata = 10; charybdis.core.SystemMetadata system_metadata = 11; charybdis.core.ApiMetadata api_metadata = 13; charybdis.core.UserMetadata user_metadata = 14; charybdis.core.GroupMetadata group_metadata = 15; charybdis.core.DomainMetadata domain_metadata = 16; charybdis.core.ResourceMetadata resource_metadata = 17; charybdis.core.FindingMetadata finding_metadata = 24; // Plugin-contributed variants (field number 100+) charybdis.plugins.defectdojo.DefectdojoMetadata defectdojo_metadata = 100; charybdis.plugins.dependencytrack.DependencytrackMetadata dependencytrack_metadata = 101; charybdis.plugins.keycloak.KeycloakMetadata keycloak_metadata = 102; } oneof spec { /* matching variants */ } map annotations = 20; google.protobuf.Timestamp created_at = 21; google.protobuf.Timestamp updated_at = 22; } ``` Field numbers are governed by `plugins.toml` — core types use 10–49, plugins claim slots starting at 100. The registry MUST be kept consistent: protobuf wire format depends on field-number stability. ### Validation Performed in `src/lib.rs::validate_entity()`: - `kind` is non-empty and one of the known values (`VALID_KINDS`). - `metadata` is present. - The metadata variant matches `kind` (e.g., `kind=Component` requires `component_metadata`). Plugin metadata variants are accepted without a kind cross-check — they identify themselves via the variant tag. ### Field masks `UpdateEntity` accepts a `google.protobuf.FieldMask`. `EntityRepository::partial_update()` (in `src/database.rs`) walks the mask paths and patches only the named fields. Supported paths include: - Top-level: `kind`, `annotations`, `metadata`, `spec` - Nested annotation key: `annotations.` - Per-variant fields: `component_metadata.name`, `component_spec.lifecycle`, `service_spec.owner`, ..., `user_spec.profile.email`, `group_spec.profile.display_name`, etc. The full path enumeration lives in the `apply_field_mask` match in `database.rs`. New kinds extend this match. ## Storage Model ```sql CREATE TABLE entities ( id UUID PRIMARY KEY, kind VARCHAR NOT NULL, name VARCHAR, entity_data BYTEA NOT NULL, -- prost-encoded Entity message annotations JSONB NOT NULL DEFAULT '{}', created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); CREATE INDEX idx_entities_kind ON entities(kind); CREATE INDEX idx_entities_kind_name ON entities(kind, name); CREATE INDEX idx_entities_annotations ON entities USING GIN (annotations); CREATE INDEX idx_entities_created_at_id ON entities(created_at DESC, id DESC); ``` Created by `database::ensure_schema()` on startup. Idempotent. - `entity_data` carries the source of truth (the full protobuf blob). - `kind`, `name`, `annotations` are denormalized for indexed lookups — kept in sync by the repository. - `idx_entities_kind_name` powers O(log n) `get_by_kind_and_name` (used heavily by ingestion to resolve `component_ref` → UUID). - `idx_entities_annotations` (GIN) powers `annotations->>'key' = 'value'` filters for plugins. - `idx_entities_created_at_id` powers stable cursor pagination — `ListEntities` encodes `(created_at, id)` as a base64 cursor, scanning `WHERE (created_at, id) < (cursor) ORDER BY created_at DESC, id DESC LIMIT N+1` to detect a next page without `OFFSET` scans. ### Atomic annotation merge `update_annotations()` uses PostgreSQL's `||` JSONB operator: ```sql UPDATE entities SET annotations = annotations || $1::jsonb, updated_at = $2 WHERE id = $3 RETURNING entity_data, annotations; ``` The merge happens at SQL level, eliminating the read-modify-write race when multiple plugin handlers write to the same entity concurrently. The returned `entity_data` is then re-encoded with the merged annotations so the protobuf blob stays consistent. ## Event Bus Defined in `src/events/`. The default backend is in-process (`MemoryEventBus`). ### Flow ``` EntityService.CreateEntity ↓ EntityRepository.create() → DB row written ↓ EntityEvent::created(uuid) .with_metadata("entity_kind", kind) .with_entity_data(Arc) ↓ EventBus.publish() ↓ EventDispatcher (subscribed) ↓ for each plugin's ResourceHandler: if handler.trigger_kinds().contains(entity.kind): handler.handle_create(entity) // 30s timeout, panic-isolated ``` ### Properties - **At-least-once intent** — handlers may run more than once if a redelivery mechanism is added later. Handlers should be idempotent. - **Best-effort delivery** — `MemoryEventBus` does not survive a server restart. A durable backend (Redis) is planned in [VISION.md](../VISION.md). - **30s per-handler timeout** — slow plugins don't block the bus. - **Panic isolation** — a panicking handler doesn't take down others or the dispatcher. ## Findings Pipeline `IngestionService.ImportScan` triggers the following pipeline (`src/findings/`): 1. **Resolve `component_ref` → UUID** (`MyIngestionService::resolve_component_id`) — accepts a UUID directly, or a name (looked up via `get_by_kind_and_name("Component", name)`). 2. **Pick a parser** by `format` from `ParserRegistry` (`src/scanners/mod.rs`). Today, only SARIF is registered. 3. **Parse** to `Vec` (`SarifParser` in `src/scanners/sarif.rs`). 4. **Reconcile** against existing findings for `(component_id, lifecycle)` (`ReconciliationEngine::reconcile`): - Compute fingerprint per finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`). Line numbers are deliberately excluded — they shift too easily. - Bucket each incoming finding as **New** (unknown fingerprint), **Unchanged** (active and re-seen), or **Reopened** (was resolved/false-positive, seen again). - Anything in DB with state `ACTIVE`/`REOPENED` whose fingerprint is absent from this scan becomes **Resolved**. 5. **Apply** (`ReconciliationEngine::apply`): - Create new findings as `Finding` entities (`build_finding_entity`). - Bump `charybdis.io/last-seen` and `charybdis.io/scan-id` annotations on unchanged findings (atomic merge). - Mark resolved findings as `RESOLVED` with `resolved_at` timestamp. - Mark reopened findings as `REOPENED`. 6. **Respond** with `ReconciliationSummary { total_parsed, new_count, unchanged_count, resolved_count, reopened_count }` plus per-bucket finding lists. `DryRunScan` runs steps 1–4 and returns the summary without persisting. This is what makes MR/PR-level diff comments possible — a CI job can call `DryRunScan` and report "this change introduces X new findings" before merge. Additional parsers (CycloneDX VEX, SPDX), assessment workflow (triage/accept/remediate), rules engine, and security gates are tracked in [VISION.md](../VISION.md). State enum values for `ACCEPTED` and `FALSE_POSITIVE` already exist in the protobuf; the workflow that sets them does not. ## YAML Adapter Backstage compatibility layer (`src/adapters/yaml/`). HTTP server on a separate port from gRPC. Two endpoints: ### `GET /yaml/locations` Returns a single Backstage `Location` entity listing every entity in Charybdis as a target URL. ```yaml apiVersion: backstage.io/v1alpha1 kind: Location metadata: name: charybdis-all-entities spec: type: charybdis targets: - http://charybdis.example.com/yaml/entities/ # ... one per entity ``` Cached for 30 s. `YamlAdapterState::invalidate_cache()` is exposed for explicit invalidation on entity-CRUD events (not wired by default — it's a hook). ### `GET /yaml/entities/:id` Renders a single entity as Backstage YAML. ```yaml apiVersion: backstage.io/v1alpha1 kind: Component metadata: name: payment-service namespace: production annotations: defectdojo.com/product-id: "42" tags: [payments, critical] spec: type: service lifecycle: production owner: team-payments system: payment-system ``` ### Configuration in Backstage ```yaml catalog: locations: - type: url target: http://charybdis.example.com/yaml/locations rules: - allow: [Component, System, API, User, Group, Domain, Resource] ``` Backstage polls the locations URL on its own schedule, then fetches each entity URL. ## Plugin Lifecycle ### Build time 1. `build.rs` reads `plugins.toml`. 2. For each enabled plugin, the corresponding `plugins//proto/.proto` is added to the compile set. 3. `entities.proto.template` is expanded with plugin imports and `oneof` variants using the field numbers from `plugins.toml`. 4. `tonic-prost-build` compiles every proto file and writes the descriptor set for gRPC reflection. ### Runtime 1. `charybdis-server` (entry point: `charybdis-server/src/main.rs`) loads `config.toml`. 2. For each plugin block with `enabled = true`, the corresponding plugin crate is instantiated (`DefectDojoPlugin::new`, `KeycloakPlugin::new`, ...). 3. Event-driven plugins are registered with the `PluginManager`; their `ResourceHandler`s are wrapped in an `EventDispatcher` subscribed to the event bus. 4. Sync plugins register with the cron scheduler. If `on_startup = true`, an initial sync runs in a background task. 5. mTLS + RBAC are wired into a tonic interceptor (when enabled). 6. The YAML adapter is spawned as a separate axum task on its own port. 7. The gRPC server is started with `EntityServiceServer + IngestionServiceServer + reflection`. The two `main.rs` arrangement (one in `src/main.rs` for the library convenience, one in `charybdis-server/src/main.rs` for the real binary) exists because the root `charybdis` crate cannot depend on plugin crates without creating a cyclic dependency. `charybdis-server` is the seam that links plugins to the core. Plugin trait reference, configuration, field mapping system, and "writing a new plugin": [../plugins/README.md](../plugins/README.md). ## Security mTLS + RBAC are off by default for local development, on by configuration for shared and production environments. The `AuthInterceptor` (`src/security/interceptor.rs`): 1. Extracts the client certificate from tonic's `TlsConnectInfo` (direct mTLS) or from an `x-forwarded-client-cert` header (reverse-proxy mode). 2. Parses the certificate via `x509-parser` into a `ClientIdentity { common_name, organization, organizational_unit, serial }`. 3. Maps the identity to a role via `RbacEngine::map_identity_to_role()` (subject-match rules from config). 4. Looks up the required permission for the gRPC method (`method_to_permission` is hard-coded). 5. Checks the role's permission list; denies on miss, logs allowed/denied to the audit log. Configuration reference, role mapping, audit logging, reverse-proxy mode: [security.md](security.md). ## Observability OpenTelemetry setup in `src/telemetry/`. Three signals: - **Traces** — every gRPC method is `#[instrument]`-ed; spans carry `entity.id`, `entity.kind`, `client.cn`, `role`, etc. - **Metrics** — `Metrics::record_entity_operation(op, kind, duration)` from each handler. - **Logs** — structured via `tracing-subscriber`, JSON or console. Console exporter is on by default. OTLP exporter (traces/metrics/logs) is enabled via `[telemetry.otlp]` block in `config.toml`. ## Technology Decisions ### Why gRPC only Strong typing across languages, binary efficiency for high-frequency CI/CD calls, official clients in every language platform engineers use, and one well-maintained surface area instead of three. Teams that need REST add grpc-gateway or Envoy. ### Why PostgreSQL + protobuf + JSONB - ACID guarantees + a mature ecosystem most teams already operate. - Protobuf blob = compact, versioned, no schema migrations for new kinds. - JSONB + GIN = fast `annotations->>'key'` queries without bespoke tables per integration. The combination gives schema flexibility without sacrificing transactional safety. Considered MongoDB; rejected because the schema flexibility is already obtained via protobuf and PostgreSQL's consistency story is stronger. ### Why compile-time plugins - Rust type system enforces handler signatures; no runtime trait-object surprises. - No dynamic loader to maintain. - No version-skew matrix between plugin and core. - The cost is a rebuild to add a plugin — acceptable for infrastructure tooling that's already redeployed in pipelines. ### Why post-persistence events - Plugin failures can't roll back a successful entity write. - Long-running external API calls don't block the gRPC response. - New plugins drop in without touching CRUD code paths. The trade-off: plugins can't veto an entity creation. Validation that must block creation belongs in core (`validate_entity`) or in the client. ### Why rustls (no OpenSSL) The entire TLS stack (mTLS server, tonic transport, reqwest HTTP client in plugins) uses `rustls`. No `openssl-sys`, no `pkg-config`, no `libssl-dev` — the binary is statically linkable, builds reproducibly in slim containers, and avoids a class of CVEs from OpenSSL ABI breakage. ## Scale Reference These are *target* numbers based on a 100-engineer reference org, not benchmarks: | Dimension | Reference value | |---|---| | Components / APIs | ~1k each | | Users / Groups | ~700 / ~200 | | Systems / Domains | ~50 / ~150 | | Resources | ~170k | | PostgreSQL storage | ~500 MB | | Charybdis memory | ~256 MB | Bottlenecks anticipated (none observed in production yet): - JSONB annotation queries → mitigated by the GIN index. - Event bus throughput → mitigated by a durable backend (planned). - Plugin external-API rate limits → per-plugin concern, not core. ## File Map | Path | Responsibility | |---|---| | `proto/entities.proto` | Generated entity schema (do not hand-edit) | | `proto/entities.proto.template` | Template the generator expands | | `proto/core/*.proto` | Per-kind metadata + spec messages | | `proto/ingestion.proto` | `IngestionService` | | `plugins.toml` | Plugin registry + protobuf field-number allocation | | `build.rs` | Generates `entities.proto`, compiles all protos | | `src/lib.rs` | `MyEntityService` (gRPC EntityService impl) + validation | | `src/database.rs` | `EntityRepository`, schema bootstrap, field-mask paths | | `src/findings/` | Ingestion + reconciliation + fingerprint | | `src/scanners/` | `ScannerParser` trait + SARIF parser | | `src/events/` | EventBus, EventDispatcher | | `src/plugins/` | Plugin traits, utilities (HTTP client, annotation helper, field mapper), dispatcher | | `src/security/` | mTLS, RBAC, AuthInterceptor | | `src/adapters/yaml/` | Backstage YAML adapter | | `src/telemetry/` | OpenTelemetry setup | | `charybdis-server/src/main.rs` | Binary entry point — wires plugins, services, mTLS, YAML adapter | | `plugins//` | Plugin crates (`defectdojo`, `keycloak`, `dependencytrack`) | ## References - [Backstage Descriptor Format](https://backstage.io/docs/features/software-catalog/descriptor-format) - [gRPC Documentation](https://grpc.io/docs/) - [Protocol Buffers Guide](https://protobuf.dev/) - [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html)