charybdis/docs/core-concepts.md

Core Concepts

Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes.

For a higher-level overview see the README. For deep technical details see architecture.md.

Entities

An entity is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding.

Entity structure

Every entity has the same shape:

• id — UUID assigned by the server
• kind — one of the supported kinds (see below)
• metadata — identifying data (name, description, labels, tags, links)
• spec — kind-specific configuration (type, lifecycle, owner, dependencies, ...)
• annotations — arbitrary string → string map, typically used by plugins to store external IDs
• created_at, updated_at — server-managed timestamps

metadata and spec are protobuf oneof fields whose variant matches kind. So a Component entity carries component_metadata + component_spec, a User carries user_metadata + user_spec, and so on. The on-wire JSON uses the variant name directly.

Entity kinds

Kind	Purpose
Component	Services, applications, libraries, websites
System	Collections of components working together
API	Interfaces exposed by components
User	Individual people
Group	Teams and organizational units
Domain	Business domains
Resource	Infrastructure resources (databases, caches, queues)
Finding	Security findings ingested from scanners (see Findings)

Per-kind protobuf definitions: proto/core/*.proto.

Example: Component

{
  "kind": "Component",
  "component_metadata": {
    "name": "payment-api",
    "namespace": "production",
    "description": "Payment processing service",
    "labels": { "team": "payments", "tier": "critical" },
    "tags": ["api", "pci-dss"]
  },
  "component_spec": {
    "type": "service",
    "lifecycle": "production",
    "owner": "team-payments",
    "system": "payment-system"
  }
}

Common metadata fields

Field	Type	Notes
name	string	Required. Unique with kind (composite index idx_entities_kind_name).
namespace	string	Logical grouping, e.g. production.
description	string	Human-readable.
labels	map<string,string>	Categorization.
tags	repeated string	Free-form classification.
links	repeated Link	External URLs (dashboards, docs).

Annotations

Annotations are an open map<string, string> on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB.

{
  "annotations": {
    "github.com/repo-slug": "myorg/payment-service",
    "defectdojo.com/product-id": "123",
    "keycloak.com/email": "alice@example.com",
    "pagerduty.com/service-id": "PXYZ123"
  }
}

Convention: reverse-DNS keys (tool.com/key). Values are always strings.

EntityRepository::update_annotations() performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers.

Events

Charybdis emits an EntityEvent on every CRUD operation, delivered to subscribers via the in-memory event bus.

Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers

Event types

Type	When
Created	Successful CreateEntity
Updated	Successful UpdateEntity (full or partial)
Deleted	Successful DeleteEntity

Event payload

pub struct EntityEvent {
    pub event_id: Uuid,
    pub entity_id: Uuid,
    pub event_type: EntityEventType,   // Created | Updated | Deleted
    pub timestamp: DateTime<Utc>,
    pub metadata: HashMap<String, String>,
    pub entity_data: Option<Arc<Entity>>,  // full entity for Created/Updated
}

Delivery semantics

• Asynchronous — CreateEntity returns to the client before plugin handlers complete.
• At-least-once intent, but the default MemoryEventBus is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in VISION.md.
• Per-handler timeout — 30 s. Slow plugins don't block the bus.
• Panic isolation — a panicking handler doesn't take down others.

Plugin lifecycle and how to write a handler: plugins/README.md.

Findings

Finding is a first-class entity kind for security findings produced by scanners. Findings are created by the IngestionService, not directly by clients.

Each finding is scoped to a (component_ref, lifecycle) pair — so production and staging environments track findings independently without duplicating the underlying Component.

The ingestion flow:

1. Client calls IngestionService.ImportScan(component_ref, lifecycle, format, data).
2. The configured parser (currently SARIF) normalizes scanner output into NormalizedFinding records.
3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else sha256(scanner | rule_id | file_path)) and diffs incoming vs existing findings for that scope.
4. The result is one of:
   • New — fingerprint not seen before → create a Finding entity.
   • Unchanged — already active → bump last_seen.
   • Resolved — previously active, not in this scan → mark Resolved.
   • Reopened — previously resolved or false-positive, detected again → mark Reopened.
5. DryRunScan runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments.

Finding state values (see proto/core/finding.proto):

State	Meaning
ACTIVE	Currently detected
RESOLVED	No longer detected (auto-set on re-import)
ACCEPTED	Risk accepted by human decision (state value defined; assessment workflow is planned)
FALSE_POSITIVE	Marked as not a real issue (state value defined; assessment workflow is planned)
REOPENED	Was resolved, detected again

Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in VISION.md and TODO.md.

Storage Model

Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a oneof variant in protobuf, not a column migration.

CREATE TABLE entities (
    id          UUID PRIMARY KEY,
    kind        VARCHAR NOT NULL,
    name        VARCHAR,
    entity_data BYTEA NOT NULL,            -- protobuf-encoded Entity
    annotations JSONB NOT NULL DEFAULT '{}',
    created_at  TIMESTAMPTZ NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL
);

CREATE INDEX idx_entities_kind            ON entities(kind);
CREATE INDEX idx_entities_kind_name       ON entities(kind, name);
CREATE INDEX idx_entities_annotations     ON entities USING GIN (annotations);
CREATE INDEX idx_entities_created_at_id   ON entities(created_at DESC, id DESC);

Why this design:

• No migrations — new plugins extend the protobuf oneof variants; database schema is unchanged.
• Compact storage — protobuf binary is smaller than JSON.
• Forward/backward compatible — proto field numbers protect against version drift.
• Fast annotation queries — JSONB + GIN index supports annotations->>'key' = 'value' lookups in O(log n).

Detailed schema reasoning, scaling notes, and field-mask paths: architecture.md.

See Also

• Architecture — protobuf schema, scaling, design decisions
• Security — mTLS + RBAC configuration
• Plugins — DefectDojo, Keycloak, writing your own
• Vision & Roadmap — planned features