charybdis/docs/core-concepts.md

# Core Concepts

Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes.

For a higher-level overview see the [README](../README.md). For deep technical details see [architecture.md](architecture.md).

## Entities

An **entity** is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding.

### Entity structure

Every entity has the same shape:

- `id` — UUID assigned by the server
- `kind` — one of the supported kinds (see below)
- `metadata` — identifying data (name, description, labels, tags, links)
- `spec` — kind-specific configuration (type, lifecycle, owner, dependencies, ...)
- `annotations` — arbitrary `string → string` map, typically used by plugins to store external IDs
- `created_at`, `updated_at` — server-managed timestamps

`metadata` and `spec` are protobuf `oneof` fields whose variant matches `kind`. So a `Component` entity carries `component_metadata` + `component_spec`, a `User` carries `user_metadata` + `user_spec`, and so on. The on-wire JSON uses the variant name directly.

### Entity kinds

| Kind | Purpose |
|---|---|
| `Component` | Services, applications, libraries, websites |
| `System` | Collections of components working together |
| `API` | Interfaces exposed by components |
| `User` | Individual people |
| `Group` | Teams and organizational units |
| `Domain` | Business domains |
| `Resource` | Infrastructure resources (databases, caches, queues) |
| `Finding` | Security findings ingested from scanners (see [Findings](#findings)) |

Per-kind protobuf definitions: `proto/core/*.proto`.

### Example: Component

```json
{
  "kind": "Component",
  "component_metadata": {
    "name": "payment-api",
    "namespace": "production",
    "description": "Payment processing service",
    "labels": { "team": "payments", "tier": "critical" },
    "tags": ["api", "pci-dss"]
  },
  "component_spec": {
    "type": "service",
    "lifecycle": "production",
    "owner": "team-payments",
    "system": "payment-system"
  }
}
```

### Common metadata fields

| Field | Type | Notes |
|---|---|---|
| `name` | string | Required. Unique with `kind` (composite index `idx_entities_kind_name`). |
| `namespace` | string | Logical grouping, e.g. `production`. |
| `description` | string | Human-readable. |
| `labels` | `map<string,string>` | Categorization. |
| `tags` | `repeated string` | Free-form classification. |
| `links` | `repeated Link` | External URLs (dashboards, docs). |

## Annotations

Annotations are an open `map<string, string>` on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB.

```json
{
  "annotations": {
    "github.com/repo-slug": "myorg/payment-service",
    "defectdojo.com/product-id": "123",
    "keycloak.com/email": "alice@example.com",
    "pagerduty.com/service-id": "PXYZ123"
  }
}
```

Convention: reverse-DNS keys (`tool.com/key`). Values are always strings.

`EntityRepository::update_annotations()` performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers.

## Events

Charybdis emits an `EntityEvent` on every CRUD operation, delivered to subscribers via the in-memory event bus.

```
Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers
```

### Event types

| Type | When |
|---|---|
| `Created` | Successful `CreateEntity` |
| `Updated` | Successful `UpdateEntity` (full or partial) |
| `Deleted` | Successful `DeleteEntity` |

### Event payload

```rust
pub struct EntityEvent {
    pub event_id: Uuid,
    pub entity_id: Uuid,
    pub event_type: EntityEventType,   // Created | Updated | Deleted
    pub timestamp: DateTime<Utc>,
    pub metadata: HashMap<String, String>,
    pub entity_data: Option<Arc<Entity>>,  // full entity for Created/Updated
}
```

### Delivery semantics

- **Asynchronous** — `CreateEntity` returns to the client before plugin handlers complete.
- **At-least-once intent**, but the default `MemoryEventBus` is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in [VISION.md](../VISION.md).
- **Per-handler timeout** — 30 s. Slow plugins don't block the bus.
- **Panic isolation** — a panicking handler doesn't take down others.

Plugin lifecycle and how to write a handler: [plugins/README.md](../plugins/README.md).

## Findings

`Finding` is a first-class entity kind for security findings produced by scanners. Findings are created by the `IngestionService`, not directly by clients.

Each finding is scoped to a `(component_ref, lifecycle)` pair — so production and staging environments track findings independently without duplicating the underlying `Component`.

The ingestion flow:

1. Client calls `IngestionService.ImportScan(component_ref, lifecycle, format, data)`.
2. The configured parser (currently SARIF) normalizes scanner output into `NormalizedFinding` records.
3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`) and diffs incoming vs existing findings for that scope.
4. The result is one of:
   - **New** — fingerprint not seen before → create a `Finding` entity.
   - **Unchanged** — already active → bump `last_seen`.
   - **Resolved** — previously active, not in this scan → mark `Resolved`.
   - **Reopened** — previously resolved or false-positive, detected again → mark `Reopened`.
5. `DryRunScan` runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments.

Finding state values (see `proto/core/finding.proto`):

| State | Meaning |
|---|---|
| `ACTIVE` | Currently detected |
| `RESOLVED` | No longer detected (auto-set on re-import) |
| `ACCEPTED` | Risk accepted by human decision *(state value defined; assessment workflow is planned)* |
| `FALSE_POSITIVE` | Marked as not a real issue *(state value defined; assessment workflow is planned)* |
| `REOPENED` | Was resolved, detected again |

Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).

## Storage Model

Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a `oneof` variant in protobuf, not a column migration.

```sql
CREATE TABLE entities (
    id          UUID PRIMARY KEY,
    kind        VARCHAR NOT NULL,
    name        VARCHAR,
    entity_data BYTEA NOT NULL,            -- protobuf-encoded Entity
    annotations JSONB NOT NULL DEFAULT '{}',
    created_at  TIMESTAMPTZ NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL
);

CREATE INDEX idx_entities_kind            ON entities(kind);
CREATE INDEX idx_entities_kind_name       ON entities(kind, name);
CREATE INDEX idx_entities_annotations     ON entities USING GIN (annotations);
CREATE INDEX idx_entities_created_at_id   ON entities(created_at DESC, id DESC);
```

Why this design:
- **No migrations** — new plugins extend the protobuf `oneof` variants; database schema is unchanged.
- **Compact storage** — protobuf binary is smaller than JSON.
- **Forward/backward compatible** — proto field numbers protect against version drift.
- **Fast annotation queries** — JSONB + GIN index supports `annotations->>'key' = 'value'` lookups in O(log n).

Detailed schema reasoning, scaling notes, and field-mask paths: [architecture.md](architecture.md).

## See Also

- [Architecture](architecture.md) — protobuf schema, scaling, design decisions
- [Security](security.md) — mTLS + RBAC configuration
- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
- [Vision & Roadmap](../VISION.md) — planned features