Public Access
424 lines
22 KiB
Markdown
424 lines
22 KiB
Markdown
# Architecture
|
||
|
||
This document describes the implementation of Charybdis: its protobuf schema, storage model, event flow, plugin lifecycle, and the rationale behind the major design decisions.
|
||
|
||
For the product vision and roadmap, see [VISION.md](../VISION.md).
|
||
For end-user concepts (entities, annotations, events), see [core-concepts.md](core-concepts.md).
|
||
|
||
## Architectural Principles
|
||
|
||
### 1. Zero database migrations
|
||
|
||
The PostgreSQL schema is created on first startup and **never changes**.
|
||
|
||
- Entities are serialized to protobuf and stored in `entity_data BYTEA`.
|
||
- New entity kinds are added as new `oneof` variants in the protobuf schema, not as new columns.
|
||
- Plugins extend the schema at build time by registering protobuf field numbers (see [plugin lifecycle](#plugin-lifecycle)).
|
||
|
||
Deploying a new plugin requires a rebuild but no migration script. Rolling back is the same: rebuild without the plugin, the stored bytes for that variant are simply ignored.
|
||
|
||
### 2. gRPC is the only API
|
||
|
||
Charybdis exposes two gRPC services and one HTTP adapter:
|
||
|
||
- `EntityService` (`proto/entities.proto`) — full entity CRUD with field-mask partial updates.
|
||
- `IngestionService` (`proto/ingestion.proto`) — `ImportScan` / `DryRunScan` for security findings.
|
||
- HTTP YAML adapter — Backstage-compatible Location YAML on a separate port.
|
||
|
||
No REST, no GraphQL in core. Teams that need them should put grpc-gateway or Envoy in front.
|
||
|
||
### 3. Plugins are compile-time integrated
|
||
|
||
Plugins are Rust crates linked at build time into `charybdis-server`. Pros: type safety, no dynamic loader, no runtime version skew. Cons: adding a plugin requires a rebuild — acceptable for an infrastructure tool.
|
||
|
||
### 4. Event-driven, post-persistence
|
||
|
||
Events fire **after** the database write succeeds.
|
||
|
||
```
|
||
CRUD request → validate → DB write → publish event → plugins react (async)
|
||
```
|
||
|
||
Plugin failures don't fail the original request. The entity is already persisted; plugins log their errors and the event bus moves on.
|
||
|
||
### 5. Backstage-compatible entity shape
|
||
|
||
The protobuf entity model maps 1:1 onto Backstage's `apiVersion / kind / metadata / spec` structure. The YAML adapter renders this directly. This is a deliberate compatibility choice — teams can run Charybdis as a dynamic backend behind their existing Backstage frontend during migration.
|
||
|
||
## System Layout
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ Clients │
|
||
│ CI/CD pipelines · Scanners (SARIF) · IaC tools · Backstage │
|
||
└─────────────────┬────────────────────────────────────────────┘
|
||
│ gRPC (mTLS optional) HTTP (YAML)
|
||
▼ ▼
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ charybdis-server │
|
||
│ ┌────────────────────────┐ ┌──────────────────────────┐ │
|
||
│ │ EntityService │ │ IngestionService │ │
|
||
│ │ CreateEntity │ │ ImportScan │ │
|
||
│ │ GetEntity │ │ DryRunScan │ │
|
||
│ │ UpdateEntity (FM) │ └──────────┬───────────────┘ │
|
||
│ │ DeleteEntity │ │ │
|
||
│ │ ListEntities │ ▼ │
|
||
│ └──────────┬─────────────┘ ┌──────────────────────────┐ │
|
||
│ │ │ Findings pipeline │ │
|
||
│ │ │ Parser registry │ │
|
||
│ │ │ Reconciliation engine │ │
|
||
│ │ │ Fingerprint dedup │ │
|
||
│ │ └──────────┬───────────────┘ │
|
||
│ │ │ │
|
||
│ ▼ ▼ │
|
||
│ ┌────────────────────────────────────────────────────────┐ │
|
||
│ │ AuthInterceptor (mTLS + RBAC) │ │
|
||
│ └────────────────────┬───────────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌────────────────────▼───────────────────────────────────┐ │
|
||
│ │ EntityRepository (PostgreSQL, protobuf + JSONB) │ │
|
||
│ └────────────────────┬───────────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌────────────────────▼───────────────────────────────────┐ │
|
||
│ │ EventBus (MemoryEventBus) │ │
|
||
│ │ → EventDispatcher → plugin ResourceHandlers │ │
|
||
│ └────────────────────┬───────────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌────────────────────▼───────────────────────────────────┐ │
|
||
│ │ YAML Adapter (axum HTTP, separate port) │ │
|
||
│ └────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ Telemetry: OpenTelemetry traces/metrics/logs (console+OTLP) │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼ external API calls
|
||
Plugins: DefectDojo, Keycloak
|
||
```
|
||
|
||
## Entity Schema
|
||
|
||
Defined in `proto/entities.proto` (generated by `build.rs` from `entities.proto.template` + `plugins.toml`).
|
||
|
||
```protobuf
|
||
message Entity {
|
||
string id = 1; // UUID, server-assigned
|
||
string kind = 2; // "Component", "Finding", ...
|
||
|
||
oneof metadata {
|
||
charybdis.core.ComponentMetadata component_metadata = 12;
|
||
charybdis.core.ServiceMetadata service_metadata = 10;
|
||
charybdis.core.SystemMetadata system_metadata = 11;
|
||
charybdis.core.ApiMetadata api_metadata = 13;
|
||
charybdis.core.UserMetadata user_metadata = 14;
|
||
charybdis.core.GroupMetadata group_metadata = 15;
|
||
charybdis.core.DomainMetadata domain_metadata = 16;
|
||
charybdis.core.ResourceMetadata resource_metadata = 17;
|
||
charybdis.core.FindingMetadata finding_metadata = 24;
|
||
|
||
// Plugin-contributed variants (field number 100+)
|
||
charybdis.plugins.defectdojo.DefectdojoMetadata defectdojo_metadata = 100;
|
||
charybdis.plugins.dependencytrack.DependencytrackMetadata dependencytrack_metadata = 101;
|
||
charybdis.plugins.keycloak.KeycloakMetadata keycloak_metadata = 102;
|
||
}
|
||
|
||
oneof spec { /* matching variants */ }
|
||
|
||
map<string, string> annotations = 20;
|
||
google.protobuf.Timestamp created_at = 21;
|
||
google.protobuf.Timestamp updated_at = 22;
|
||
}
|
||
```
|
||
|
||
Field numbers are governed by `plugins.toml` — core types use 10–49, plugins claim slots starting at 100. The registry MUST be kept consistent: protobuf wire format depends on field-number stability.
|
||
|
||
### Validation
|
||
|
||
Performed in `src/lib.rs::validate_entity()`:
|
||
- `kind` is non-empty and one of the known values (`VALID_KINDS`).
|
||
- `metadata` is present.
|
||
- The metadata variant matches `kind` (e.g., `kind=Component` requires `component_metadata`).
|
||
|
||
Plugin metadata variants are accepted without a kind cross-check — they identify themselves via the variant tag.
|
||
|
||
### Field masks
|
||
|
||
`UpdateEntity` accepts a `google.protobuf.FieldMask`. `EntityRepository::partial_update()` (in `src/database.rs`) walks the mask paths and patches only the named fields. Supported paths include:
|
||
|
||
- Top-level: `kind`, `annotations`, `metadata`, `spec`
|
||
- Nested annotation key: `annotations.<key>`
|
||
- Per-variant fields: `component_metadata.name`, `component_spec.lifecycle`, `service_spec.owner`, ..., `user_spec.profile.email`, `group_spec.profile.display_name`, etc.
|
||
|
||
The full path enumeration lives in the `apply_field_mask` match in `database.rs`. New kinds extend this match.
|
||
|
||
## Storage Model
|
||
|
||
```sql
|
||
CREATE TABLE entities (
|
||
id UUID PRIMARY KEY,
|
||
kind VARCHAR NOT NULL,
|
||
name VARCHAR,
|
||
entity_data BYTEA NOT NULL, -- prost-encoded Entity message
|
||
annotations JSONB NOT NULL DEFAULT '{}',
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
|
||
CREATE INDEX idx_entities_kind ON entities(kind);
|
||
CREATE INDEX idx_entities_kind_name ON entities(kind, name);
|
||
CREATE INDEX idx_entities_annotations ON entities USING GIN (annotations);
|
||
CREATE INDEX idx_entities_created_at_id ON entities(created_at DESC, id DESC);
|
||
```
|
||
|
||
Created by `database::ensure_schema()` on startup. Idempotent.
|
||
|
||
- `entity_data` carries the source of truth (the full protobuf blob).
|
||
- `kind`, `name`, `annotations` are denormalized for indexed lookups — kept in sync by the repository.
|
||
- `idx_entities_kind_name` powers O(log n) `get_by_kind_and_name` (used heavily by ingestion to resolve `component_ref` → UUID).
|
||
- `idx_entities_annotations` (GIN) powers `annotations->>'key' = 'value'` filters for plugins.
|
||
- `idx_entities_created_at_id` powers stable cursor pagination — `ListEntities` encodes `(created_at, id)` as a base64 cursor, scanning `WHERE (created_at, id) < (cursor) ORDER BY created_at DESC, id DESC LIMIT N+1` to detect a next page without `OFFSET` scans.
|
||
|
||
### Atomic annotation merge
|
||
|
||
`update_annotations()` uses PostgreSQL's `||` JSONB operator:
|
||
|
||
```sql
|
||
UPDATE entities
|
||
SET annotations = annotations || $1::jsonb, updated_at = $2
|
||
WHERE id = $3
|
||
RETURNING entity_data, annotations;
|
||
```
|
||
|
||
The merge happens at SQL level, eliminating the read-modify-write race when multiple plugin handlers write to the same entity concurrently. The returned `entity_data` is then re-encoded with the merged annotations so the protobuf blob stays consistent.
|
||
|
||
## Event Bus
|
||
|
||
Defined in `src/events/`. The default backend is in-process (`MemoryEventBus`).
|
||
|
||
### Flow
|
||
|
||
```
|
||
EntityService.CreateEntity
|
||
↓
|
||
EntityRepository.create() → DB row written
|
||
↓
|
||
EntityEvent::created(uuid)
|
||
.with_metadata("entity_kind", kind)
|
||
.with_entity_data(Arc<Entity>)
|
||
↓
|
||
EventBus.publish()
|
||
↓
|
||
EventDispatcher (subscribed)
|
||
↓
|
||
for each plugin's ResourceHandler:
|
||
if handler.trigger_kinds().contains(entity.kind):
|
||
handler.handle_create(entity) // 30s timeout, panic-isolated
|
||
```
|
||
|
||
### Properties
|
||
|
||
- **At-least-once intent** — handlers may run more than once if a redelivery mechanism is added later. Handlers should be idempotent.
|
||
- **Best-effort delivery** — `MemoryEventBus` does not survive a server restart. A durable backend (Redis) is planned in [VISION.md](../VISION.md).
|
||
- **30s per-handler timeout** — slow plugins don't block the bus.
|
||
- **Panic isolation** — a panicking handler doesn't take down others or the dispatcher.
|
||
|
||
## Findings Pipeline
|
||
|
||
`IngestionService.ImportScan` triggers the following pipeline (`src/findings/`):
|
||
|
||
1. **Resolve `component_ref` → UUID** (`MyIngestionService::resolve_component_id`) — accepts a UUID directly, or a name (looked up via `get_by_kind_and_name("Component", name)`).
|
||
2. **Pick a parser** by `format` from `ParserRegistry` (`src/scanners/mod.rs`). Today, only SARIF is registered.
|
||
3. **Parse** to `Vec<NormalizedFinding>` (`SarifParser` in `src/scanners/sarif.rs`).
|
||
4. **Reconcile** against existing findings for `(component_id, lifecycle)` (`ReconciliationEngine::reconcile`):
|
||
- Compute fingerprint per finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`). Line numbers are deliberately excluded — they shift too easily.
|
||
- Bucket each incoming finding as **New** (unknown fingerprint), **Unchanged** (active and re-seen), or **Reopened** (was resolved/false-positive, seen again).
|
||
- Anything in DB with state `ACTIVE`/`REOPENED` whose fingerprint is absent from this scan becomes **Resolved**.
|
||
5. **Apply** (`ReconciliationEngine::apply`):
|
||
- Create new findings as `Finding` entities (`build_finding_entity`).
|
||
- Bump `charybdis.io/last-seen` and `charybdis.io/scan-id` annotations on unchanged findings (atomic merge).
|
||
- Mark resolved findings as `RESOLVED` with `resolved_at` timestamp.
|
||
- Mark reopened findings as `REOPENED`.
|
||
6. **Respond** with `ReconciliationSummary { total_parsed, new_count, unchanged_count, resolved_count, reopened_count }` plus per-bucket finding lists.
|
||
|
||
`DryRunScan` runs steps 1–4 and returns the summary without persisting. This is what makes MR/PR-level diff comments possible — a CI job can call `DryRunScan` and report "this change introduces X new findings" before merge.
|
||
|
||
Additional parsers (CycloneDX VEX, SPDX), assessment workflow (triage/accept/remediate), rules engine, and security gates are tracked in [VISION.md](../VISION.md). State enum values for `ACCEPTED` and `FALSE_POSITIVE` already exist in the protobuf; the workflow that sets them does not.
|
||
|
||
## YAML Adapter
|
||
|
||
Backstage compatibility layer (`src/adapters/yaml/`). HTTP server on a separate port from gRPC. Two endpoints:
|
||
|
||
### `GET /yaml/locations`
|
||
|
||
Returns a single Backstage `Location` entity listing every entity in Charybdis as a target URL.
|
||
|
||
```yaml
|
||
apiVersion: backstage.io/v1alpha1
|
||
kind: Location
|
||
metadata:
|
||
name: charybdis-all-entities
|
||
spec:
|
||
type: charybdis
|
||
targets:
|
||
- http://charybdis.example.com/yaml/entities/<id>
|
||
# ... one per entity
|
||
```
|
||
|
||
Cached for 30 s. `YamlAdapterState::invalidate_cache()` is exposed for explicit invalidation on entity-CRUD events (not wired by default — it's a hook).
|
||
|
||
### `GET /yaml/entities/:id`
|
||
|
||
Renders a single entity as Backstage YAML.
|
||
|
||
```yaml
|
||
apiVersion: backstage.io/v1alpha1
|
||
kind: Component
|
||
metadata:
|
||
name: payment-service
|
||
namespace: production
|
||
annotations:
|
||
defectdojo.com/product-id: "42"
|
||
tags: [payments, critical]
|
||
spec:
|
||
type: service
|
||
lifecycle: production
|
||
owner: team-payments
|
||
system: payment-system
|
||
```
|
||
|
||
### Configuration in Backstage
|
||
|
||
```yaml
|
||
catalog:
|
||
locations:
|
||
- type: url
|
||
target: http://charybdis.example.com/yaml/locations
|
||
rules:
|
||
- allow: [Component, System, API, User, Group, Domain, Resource]
|
||
```
|
||
|
||
Backstage polls the locations URL on its own schedule, then fetches each entity URL.
|
||
|
||
## Plugin Lifecycle
|
||
|
||
### Build time
|
||
|
||
1. `build.rs` reads `plugins.toml`.
|
||
2. For each enabled plugin, the corresponding `plugins/<name>/proto/<name>.proto` is added to the compile set.
|
||
3. `entities.proto.template` is expanded with plugin imports and `oneof` variants using the field numbers from `plugins.toml`.
|
||
4. `tonic-prost-build` compiles every proto file and writes the descriptor set for gRPC reflection.
|
||
|
||
### Runtime
|
||
|
||
1. `charybdis-server` (entry point: `charybdis-server/src/main.rs`) loads `config.toml`.
|
||
2. For each plugin block with `enabled = true`, the corresponding plugin crate is instantiated (`DefectDojoPlugin::new`, `KeycloakPlugin::new`, ...).
|
||
3. Event-driven plugins are registered with the `PluginManager`; their `ResourceHandler`s are wrapped in an `EventDispatcher` subscribed to the event bus.
|
||
4. Sync plugins register with the cron scheduler. If `on_startup = true`, an initial sync runs in a background task.
|
||
5. mTLS + RBAC are wired into a tonic interceptor (when enabled).
|
||
6. The YAML adapter is spawned as a separate axum task on its own port.
|
||
7. The gRPC server is started with `EntityServiceServer + IngestionServiceServer + reflection`.
|
||
|
||
The two `main.rs` arrangement (one in `src/main.rs` for the library convenience, one in `charybdis-server/src/main.rs` for the real binary) exists because the root `charybdis` crate cannot depend on plugin crates without creating a cyclic dependency. `charybdis-server` is the seam that links plugins to the core.
|
||
|
||
Plugin trait reference, configuration, field mapping system, and "writing a new plugin": [../plugins/README.md](../plugins/README.md).
|
||
|
||
## Security
|
||
|
||
mTLS + RBAC are off by default for local development, on by configuration for shared and production environments. The `AuthInterceptor` (`src/security/interceptor.rs`):
|
||
|
||
1. Extracts the client certificate from tonic's `TlsConnectInfo` (direct mTLS) or from an `x-forwarded-client-cert` header (reverse-proxy mode).
|
||
2. Parses the certificate via `x509-parser` into a `ClientIdentity { common_name, organization, organizational_unit, serial }`.
|
||
3. Maps the identity to a role via `RbacEngine::map_identity_to_role()` (subject-match rules from config).
|
||
4. Looks up the required permission for the gRPC method (`method_to_permission` is hard-coded).
|
||
5. Checks the role's permission list; denies on miss, logs allowed/denied to the audit log.
|
||
|
||
Configuration reference, role mapping, audit logging, reverse-proxy mode: [security.md](security.md).
|
||
|
||
## Observability
|
||
|
||
OpenTelemetry setup in `src/telemetry/`. Three signals:
|
||
- **Traces** — every gRPC method is `#[instrument]`-ed; spans carry `entity.id`, `entity.kind`, `client.cn`, `role`, etc.
|
||
- **Metrics** — `Metrics::record_entity_operation(op, kind, duration)` from each handler.
|
||
- **Logs** — structured via `tracing-subscriber`, JSON or console.
|
||
|
||
Console exporter is on by default. OTLP exporter (traces/metrics/logs) is enabled via `[telemetry.otlp]` block in `config.toml`.
|
||
|
||
## Technology Decisions
|
||
|
||
### Why gRPC only
|
||
|
||
Strong typing across languages, binary efficiency for high-frequency CI/CD calls, official clients in every language platform engineers use, and one well-maintained surface area instead of three. Teams that need REST add grpc-gateway or Envoy.
|
||
|
||
### Why PostgreSQL + protobuf + JSONB
|
||
|
||
- ACID guarantees + a mature ecosystem most teams already operate.
|
||
- Protobuf blob = compact, versioned, no schema migrations for new kinds.
|
||
- JSONB + GIN = fast `annotations->>'key'` queries without bespoke tables per integration.
|
||
|
||
The combination gives schema flexibility without sacrificing transactional safety. Considered MongoDB; rejected because the schema flexibility is already obtained via protobuf and PostgreSQL's consistency story is stronger.
|
||
|
||
### Why compile-time plugins
|
||
|
||
- Rust type system enforces handler signatures; no runtime trait-object surprises.
|
||
- No dynamic loader to maintain.
|
||
- No version-skew matrix between plugin and core.
|
||
- The cost is a rebuild to add a plugin — acceptable for infrastructure tooling that's already redeployed in pipelines.
|
||
|
||
### Why post-persistence events
|
||
|
||
- Plugin failures can't roll back a successful entity write.
|
||
- Long-running external API calls don't block the gRPC response.
|
||
- New plugins drop in without touching CRUD code paths.
|
||
|
||
The trade-off: plugins can't veto an entity creation. Validation that must block creation belongs in core (`validate_entity`) or in the client.
|
||
|
||
### Why rustls (no OpenSSL)
|
||
|
||
The entire TLS stack (mTLS server, tonic transport, reqwest HTTP client in plugins) uses `rustls`. No `openssl-sys`, no `pkg-config`, no `libssl-dev` — the binary is statically linkable, builds reproducibly in slim containers, and avoids a class of CVEs from OpenSSL ABI breakage.
|
||
|
||
## Scale Reference
|
||
|
||
These are *target* numbers based on a 100-engineer reference org, not benchmarks:
|
||
|
||
| Dimension | Reference value |
|
||
|---|---|
|
||
| Components / APIs | ~1k each |
|
||
| Users / Groups | ~700 / ~200 |
|
||
| Systems / Domains | ~50 / ~150 |
|
||
| Resources | ~170k |
|
||
| PostgreSQL storage | ~500 MB |
|
||
| Charybdis memory | ~256 MB |
|
||
|
||
Bottlenecks anticipated (none observed in production yet):
|
||
- JSONB annotation queries → mitigated by the GIN index.
|
||
- Event bus throughput → mitigated by a durable backend (planned).
|
||
- Plugin external-API rate limits → per-plugin concern, not core.
|
||
|
||
## File Map
|
||
|
||
| Path | Responsibility |
|
||
|---|---|
|
||
| `proto/entities.proto` | Generated entity schema (do not hand-edit) |
|
||
| `proto/entities.proto.template` | Template the generator expands |
|
||
| `proto/core/*.proto` | Per-kind metadata + spec messages |
|
||
| `proto/ingestion.proto` | `IngestionService` |
|
||
| `plugins.toml` | Plugin registry + protobuf field-number allocation |
|
||
| `build.rs` | Generates `entities.proto`, compiles all protos |
|
||
| `src/lib.rs` | `MyEntityService` (gRPC EntityService impl) + validation |
|
||
| `src/database.rs` | `EntityRepository`, schema bootstrap, field-mask paths |
|
||
| `src/findings/` | Ingestion + reconciliation + fingerprint |
|
||
| `src/scanners/` | `ScannerParser` trait + SARIF parser |
|
||
| `src/events/` | EventBus, EventDispatcher |
|
||
| `src/plugins/` | Plugin traits, utilities (HTTP client, annotation helper, field mapper), dispatcher |
|
||
| `src/security/` | mTLS, RBAC, AuthInterceptor |
|
||
| `src/adapters/yaml/` | Backstage YAML adapter |
|
||
| `src/telemetry/` | OpenTelemetry setup |
|
||
| `charybdis-server/src/main.rs` | Binary entry point — wires plugins, services, mTLS, YAML adapter |
|
||
| `plugins/<name>/` | Plugin crates (`defectdojo`, `keycloak`, `dependencytrack`) |
|
||
|
||
## References
|
||
|
||
- [Backstage Descriptor Format](https://backstage.io/docs/features/software-catalog/descriptor-format)
|
||
- [gRPC Documentation](https://grpc.io/docs/)
|
||
- [Protocol Buffers Guide](https://protobuf.dev/)
|
||
- [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html)
|