charybdis/docs/architecture.md

Architecture

This document describes the implementation of Charybdis: its protobuf schema, storage model, event flow, plugin lifecycle, and the rationale behind the major design decisions.

For the product vision and roadmap, see VISION.md. For end-user concepts (entities, annotations, events), see core-concepts.md.

Architectural Principles

1. Zero database migrations

The PostgreSQL schema is created on first startup and never changes.

• Entities are serialized to protobuf and stored in entity_data BYTEA.
• New entity kinds are added as new oneof variants in the protobuf schema, not as new columns.
• Plugins extend the schema at build time by registering protobuf field numbers (see plugin lifecycle).

Deploying a new plugin requires a rebuild but no migration script. Rolling back is the same: rebuild without the plugin, the stored bytes for that variant are simply ignored.

2. gRPC is the only API

Charybdis exposes two gRPC services and one HTTP adapter:

• EntityService (proto/entities.proto) — full entity CRUD with field-mask partial updates.
• IngestionService (proto/ingestion.proto) — ImportScan / DryRunScan for security findings.
• HTTP YAML adapter — Backstage-compatible Location YAML on a separate port.

No REST, no GraphQL in core. Teams that need them should put grpc-gateway or Envoy in front.

3. Plugins are compile-time integrated

Plugins are Rust crates linked at build time into charybdis-server. Pros: type safety, no dynamic loader, no runtime version skew. Cons: adding a plugin requires a rebuild — acceptable for an infrastructure tool.

4. Event-driven, post-persistence

Events fire after the database write succeeds.

CRUD request → validate → DB write → publish event → plugins react (async)

Plugin failures don't fail the original request. The entity is already persisted; plugins log their errors and the event bus moves on.

5. Backstage-compatible entity shape

The protobuf entity model maps 1:1 onto Backstage's apiVersion / kind / metadata / spec structure. The YAML adapter renders this directly. This is a deliberate compatibility choice — teams can run Charybdis as a dynamic backend behind their existing Backstage frontend during migration.

System Layout

┌──────────────────────────────────────────────────────────────┐
│  Clients                                                      │
│  CI/CD pipelines · Scanners (SARIF) · IaC tools · Backstage   │
└─────────────────┬────────────────────────────────────────────┘
                  │ gRPC (mTLS optional)        HTTP (YAML)
                  ▼                              ▼
┌──────────────────────────────────────────────────────────────┐
│  charybdis-server                                             │
│  ┌────────────────────────┐  ┌──────────────────────────┐    │
│  │ EntityService          │  │ IngestionService          │    │
│  │ CreateEntity           │  │ ImportScan                │    │
│  │ GetEntity              │  │ DryRunScan                │    │
│  │ UpdateEntity (FM)      │  └──────────┬───────────────┘    │
│  │ DeleteEntity           │             │                    │
│  │ ListEntities           │             ▼                    │
│  └──────────┬─────────────┘  ┌──────────────────────────┐    │
│             │                 │ Findings pipeline         │    │
│             │                 │  Parser registry          │    │
│             │                 │  Reconciliation engine    │    │
│             │                 │  Fingerprint dedup        │    │
│             │                 └──────────┬───────────────┘    │
│             │                            │                    │
│             ▼                            ▼                    │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ AuthInterceptor (mTLS + RBAC)                          │  │
│  └────────────────────┬───────────────────────────────────┘  │
│                       │                                       │
│  ┌────────────────────▼───────────────────────────────────┐  │
│  │ EntityRepository (PostgreSQL, protobuf + JSONB)        │  │
│  └────────────────────┬───────────────────────────────────┘  │
│                       │                                       │
│  ┌────────────────────▼───────────────────────────────────┐  │
│  │ EventBus (MemoryEventBus)                              │  │
│  │   → EventDispatcher → plugin ResourceHandlers          │  │
│  └────────────────────┬───────────────────────────────────┘  │
│                       │                                       │
│  ┌────────────────────▼───────────────────────────────────┐  │
│  │ YAML Adapter (axum HTTP, separate port)                │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                               │
│  Telemetry: OpenTelemetry traces/metrics/logs (console+OTLP) │
└──────────────────────────────────────────────────────────────┘
                       │
                       ▼ external API calls
              Plugins: DefectDojo, Keycloak

Entity Schema

Defined in proto/entities.proto (generated by build.rs from entities.proto.template + plugins.toml).

message Entity {
  string id = 1;                         // UUID, server-assigned
  string kind = 2;                       // "Component", "Finding", ...

  oneof metadata {
    charybdis.core.ComponentMetadata component_metadata = 12;
    charybdis.core.ServiceMetadata   service_metadata   = 10;
    charybdis.core.SystemMetadata    system_metadata    = 11;
    charybdis.core.ApiMetadata       api_metadata       = 13;
    charybdis.core.UserMetadata      user_metadata      = 14;
    charybdis.core.GroupMetadata     group_metadata     = 15;
    charybdis.core.DomainMetadata    domain_metadata    = 16;
    charybdis.core.ResourceMetadata  resource_metadata  = 17;
    charybdis.core.FindingMetadata   finding_metadata   = 24;

    // Plugin-contributed variants (field number 100+)
    charybdis.plugins.defectdojo.DefectdojoMetadata           defectdojo_metadata = 100;
    charybdis.plugins.dependencytrack.DependencytrackMetadata dependencytrack_metadata = 101;
    charybdis.plugins.keycloak.KeycloakMetadata               keycloak_metadata = 102;
  }

  oneof spec { /* matching variants */ }

  map<string, string> annotations = 20;
  google.protobuf.Timestamp created_at = 21;
  google.protobuf.Timestamp updated_at = 22;
}

Field numbers are governed by plugins.toml — core types use 10–49, plugins claim slots starting at 100. The registry MUST be kept consistent: protobuf wire format depends on field-number stability.

Validation

Performed in src/lib.rs::validate_entity():

• kind is non-empty and one of the known values (VALID_KINDS).
• metadata is present.
• The metadata variant matches kind (e.g., kind=Component requires component_metadata).

Plugin metadata variants are accepted without a kind cross-check — they identify themselves via the variant tag.

Field masks

UpdateEntity accepts a google.protobuf.FieldMask. EntityRepository::partial_update() (in src/database.rs) walks the mask paths and patches only the named fields. Supported paths include:

• Top-level: kind, annotations, metadata, spec
• Nested annotation key: annotations.<key>
• Per-variant fields: component_metadata.name, component_spec.lifecycle, service_spec.owner, ..., user_spec.profile.email, group_spec.profile.display_name, etc.

The full path enumeration lives in the apply_field_mask match in database.rs. New kinds extend this match.

Storage Model

CREATE TABLE entities (
    id          UUID PRIMARY KEY,
    kind        VARCHAR NOT NULL,
    name        VARCHAR,
    entity_data BYTEA NOT NULL,            -- prost-encoded Entity message
    annotations JSONB NOT NULL DEFAULT '{}',
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_entities_kind            ON entities(kind);
CREATE INDEX idx_entities_kind_name       ON entities(kind, name);
CREATE INDEX idx_entities_annotations     ON entities USING GIN (annotations);
CREATE INDEX idx_entities_created_at_id   ON entities(created_at DESC, id DESC);

Created by database::ensure_schema() on startup. Idempotent.

• entity_data carries the source of truth (the full protobuf blob).
• kind, name, annotations are denormalized for indexed lookups — kept in sync by the repository.
• idx_entities_kind_name powers O(log n) get_by_kind_and_name (used heavily by ingestion to resolve component_ref → UUID).
• idx_entities_annotations (GIN) powers annotations->>'key' = 'value' filters for plugins.
• idx_entities_created_at_id powers stable cursor pagination — ListEntities encodes (created_at, id) as a base64 cursor, scanning WHERE (created_at, id) < (cursor) ORDER BY created_at DESC, id DESC LIMIT N+1 to detect a next page without OFFSET scans.

Atomic annotation merge

update_annotations() uses PostgreSQL's || JSONB operator:

UPDATE entities
SET annotations = annotations || $1::jsonb, updated_at = $2
WHERE id = $3
RETURNING entity_data, annotations;

The merge happens at SQL level, eliminating the read-modify-write race when multiple plugin handlers write to the same entity concurrently. The returned entity_data is then re-encoded with the merged annotations so the protobuf blob stays consistent.

Event Bus

Defined in src/events/. The default backend is in-process (MemoryEventBus).

Flow

EntityService.CreateEntity
  ↓
EntityRepository.create() → DB row written
  ↓
EntityEvent::created(uuid)
  .with_metadata("entity_kind", kind)
  .with_entity_data(Arc<Entity>)
  ↓
EventBus.publish()
  ↓
EventDispatcher (subscribed)
  ↓
for each plugin's ResourceHandler:
  if handler.trigger_kinds().contains(entity.kind):
    handler.handle_create(entity)  // 30s timeout, panic-isolated

Properties

• At-least-once intent — handlers may run more than once if a redelivery mechanism is added later. Handlers should be idempotent.
• Best-effort delivery — MemoryEventBus does not survive a server restart. A durable backend (Redis) is planned in VISION.md.
• 30s per-handler timeout — slow plugins don't block the bus.
• Panic isolation — a panicking handler doesn't take down others or the dispatcher.

Findings Pipeline

IngestionService.ImportScan triggers the following pipeline (src/findings/):

1. Resolve component_ref → UUID (MyIngestionService::resolve_component_id) — accepts a UUID directly, or a name (looked up via get_by_kind_and_name("Component", name)).
2. Pick a parser by format from ParserRegistry (src/scanners/mod.rs). Today, only SARIF is registered.
3. Parse to Vec<NormalizedFinding> (SarifParser in src/scanners/sarif.rs).
4. Reconcile against existing findings for (component_id, lifecycle) (ReconciliationEngine::reconcile):
   • Compute fingerprint per finding (scanner-provided when present, else sha256(scanner | rule_id | file_path)). Line numbers are deliberately excluded — they shift too easily.
   • Bucket each incoming finding as New (unknown fingerprint), Unchanged (active and re-seen), or Reopened (was resolved/false-positive, seen again).
   • Anything in DB with state ACTIVE/REOPENED whose fingerprint is absent from this scan becomes Resolved.
5. Apply (ReconciliationEngine::apply):
   • Create new findings as Finding entities (build_finding_entity).
   • Bump charybdis.io/last-seen and charybdis.io/scan-id annotations on unchanged findings (atomic merge).
   • Mark resolved findings as RESOLVED with resolved_at timestamp.
   • Mark reopened findings as REOPENED.
6. Respond with ReconciliationSummary { total_parsed, new_count, unchanged_count, resolved_count, reopened_count } plus per-bucket finding lists.

DryRunScan runs steps 1–4 and returns the summary without persisting. This is what makes MR/PR-level diff comments possible — a CI job can call DryRunScan and report "this change introduces X new findings" before merge.

Additional parsers (CycloneDX VEX, SPDX), assessment workflow (triage/accept/remediate), rules engine, and security gates are tracked in VISION.md. State enum values for ACCEPTED and FALSE_POSITIVE already exist in the protobuf; the workflow that sets them does not.

YAML Adapter

Backstage compatibility layer (src/adapters/yaml/). HTTP server on a separate port from gRPC. Two endpoints:

GET /yaml/locations

Returns a single Backstage Location entity listing every entity in Charybdis as a target URL.

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: charybdis-all-entities
spec:
  type: charybdis
  targets:
    - http://charybdis.example.com/yaml/entities/<id>
    # ... one per entity

Cached for 30 s. YamlAdapterState::invalidate_cache() is exposed for explicit invalidation on entity-CRUD events (not wired by default — it's a hook).

GET /yaml/entities/:id

Renders a single entity as Backstage YAML.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  namespace: production
  annotations:
    defectdojo.com/product-id: "42"
  tags: [payments, critical]
spec:
  type: service
  lifecycle: production
  owner: team-payments
  system: payment-system

Configuration in Backstage

catalog:
  locations:
    - type: url
      target: http://charybdis.example.com/yaml/locations
      rules:
        - allow: [Component, System, API, User, Group, Domain, Resource]

Backstage polls the locations URL on its own schedule, then fetches each entity URL.

Plugin Lifecycle

Build time

1. build.rs reads plugins.toml.
2. For each enabled plugin, the corresponding plugins/<name>/proto/<name>.proto is added to the compile set.
3. entities.proto.template is expanded with plugin imports and oneof variants using the field numbers from plugins.toml.
4. tonic-prost-build compiles every proto file and writes the descriptor set for gRPC reflection.

Runtime

1. charybdis-server (entry point: charybdis-server/src/main.rs) loads config.toml.
2. For each plugin block with enabled = true, the corresponding plugin crate is instantiated (DefectDojoPlugin::new, KeycloakPlugin::new, ...).
3. Event-driven plugins are registered with the PluginManager; their ResourceHandlers are wrapped in an EventDispatcher subscribed to the event bus.
4. Sync plugins register with the cron scheduler. If on_startup = true, an initial sync runs in a background task.
5. mTLS + RBAC are wired into a tonic interceptor (when enabled).
6. The YAML adapter is spawned as a separate axum task on its own port.
7. The gRPC server is started with EntityServiceServer + IngestionServiceServer + reflection.

The two main.rs arrangement (one in src/main.rs for the library convenience, one in charybdis-server/src/main.rs for the real binary) exists because the root charybdis crate cannot depend on plugin crates without creating a cyclic dependency. charybdis-server is the seam that links plugins to the core.

Plugin trait reference, configuration, field mapping system, and "writing a new plugin": ../plugins/README.md.

Security

mTLS + RBAC are off by default for local development, on by configuration for shared and production environments. The AuthInterceptor (src/security/interceptor.rs):

1. Extracts the client certificate from tonic's TlsConnectInfo (direct mTLS) or from an x-forwarded-client-cert header (reverse-proxy mode).
2. Parses the certificate via x509-parser into a ClientIdentity { common_name, organization, organizational_unit, serial }.
3. Maps the identity to a role via RbacEngine::map_identity_to_role() (subject-match rules from config).
4. Looks up the required permission for the gRPC method (method_to_permission is hard-coded).
5. Checks the role's permission list; denies on miss, logs allowed/denied to the audit log.

Configuration reference, role mapping, audit logging, reverse-proxy mode: security.md.

Observability

OpenTelemetry setup in src/telemetry/. Three signals:

• Traces — every gRPC method is #[instrument]-ed; spans carry entity.id, entity.kind, client.cn, role, etc.
• Metrics — Metrics::record_entity_operation(op, kind, duration) from each handler.
• Logs — structured via tracing-subscriber, JSON or console.

Console exporter is on by default. OTLP exporter (traces/metrics/logs) is enabled via [telemetry.otlp] block in config.toml.

Technology Decisions

Why gRPC only

Strong typing across languages, binary efficiency for high-frequency CI/CD calls, official clients in every language platform engineers use, and one well-maintained surface area instead of three. Teams that need REST add grpc-gateway or Envoy.

Why PostgreSQL + protobuf + JSONB

• ACID guarantees + a mature ecosystem most teams already operate.
• Protobuf blob = compact, versioned, no schema migrations for new kinds.
• JSONB + GIN = fast annotations->>'key' queries without bespoke tables per integration.

The combination gives schema flexibility without sacrificing transactional safety. Considered MongoDB; rejected because the schema flexibility is already obtained via protobuf and PostgreSQL's consistency story is stronger.

Why compile-time plugins

• Rust type system enforces handler signatures; no runtime trait-object surprises.
• No dynamic loader to maintain.
• No version-skew matrix between plugin and core.
• The cost is a rebuild to add a plugin — acceptable for infrastructure tooling that's already redeployed in pipelines.

Why post-persistence events

• Plugin failures can't roll back a successful entity write.
• Long-running external API calls don't block the gRPC response.
• New plugins drop in without touching CRUD code paths.

The trade-off: plugins can't veto an entity creation. Validation that must block creation belongs in core (validate_entity) or in the client.

Why rustls (no OpenSSL)

The entire TLS stack (mTLS server, tonic transport, reqwest HTTP client in plugins) uses rustls. No openssl-sys, no pkg-config, no libssl-dev — the binary is statically linkable, builds reproducibly in slim containers, and avoids a class of CVEs from OpenSSL ABI breakage.

Scale Reference

These are target numbers based on a 100-engineer reference org, not benchmarks:

Dimension	Reference value
Components / APIs	~1k each
Users / Groups	~700 / ~200
Systems / Domains	~50 / ~150
Resources	~170k
PostgreSQL storage	~500 MB
Charybdis memory	~256 MB

Bottlenecks anticipated (none observed in production yet):

• JSONB annotation queries → mitigated by the GIN index.
• Event bus throughput → mitigated by a durable backend (planned).
• Plugin external-API rate limits → per-plugin concern, not core.

File Map

Path	Responsibility
proto/entities.proto	Generated entity schema (do not hand-edit)
proto/entities.proto.template	Template the generator expands
proto/core/*.proto	Per-kind metadata + spec messages
proto/ingestion.proto	IngestionService
plugins.toml	Plugin registry + protobuf field-number allocation
build.rs	Generates entities.proto, compiles all protos
src/lib.rs	MyEntityService (gRPC EntityService impl) + validation
src/database.rs	EntityRepository, schema bootstrap, field-mask paths
src/findings/	Ingestion + reconciliation + fingerprint
src/scanners/	ScannerParser trait + SARIF parser
src/events/	EventBus, EventDispatcher
src/plugins/	Plugin traits, utilities (HTTP client, annotation helper, field mapper), dispatcher
src/security/	mTLS, RBAC, AuthInterceptor
src/adapters/yaml/	Backstage YAML adapter
src/telemetry/	OpenTelemetry setup
charybdis-server/src/main.rs	Binary entry point — wires plugins, services, mTLS, YAML adapter
plugins/<name>/	Plugin crates (defectdojo, keycloak, dependencytrack)

References

• Backstage Descriptor Format
• gRPC Documentation
• Protocol Buffers Guide
• PostgreSQL JSONB