guillaume.grabe/charybdis
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

doc: update and cleanup
CI / Check (push) Successful in 13m13s
Details
CI / Format (push) Successful in 48s
Details
CI / Clippy (push) Successful in 12m13s
Details
CI / Test (push) Successful in 12m45s
Details

Browse Source
This commit is contained in:
Guillaume GRABÉ
2026-06-09 11:26:47 +02:00
parent ab13af48e7
commit f1b1543f29
11 changed files with 1113 additions and 3376 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/core-concepts.md
+115 -551
View File
@@ -1,126 +1,54 @@
# Core Concepts
Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.
Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes.
This page explains the key concepts you need to understand.
## Architecture Overview
```mermaid
graph TB
subgraph "Sources"
A1[CI/CD Pipelines]
A2[Security Scanners]
A3[IaC Tools]
end
subgraph "Charybdis Core"
B1[gRPC API]
B3[Authentication & Authorization]
C1[Entity Service]
C4[Vulnerability Engine]
C5[Security Gates & Rules]
C2[Event Bus]
end
subgraph "Integrations - Plugins"
D1[Slack / Teams]
D2[Jira / GitHub]
D3[Custom Plugins]
end
subgraph "Data Layer"
E1[(PostgreSQL)]
end
A1 -.gRPC.-> B1
A2 -.Scan Results.-> B1
A3 -.gRPC.-> B1
B1 --> B3
B3 --> C1
B3 --> C4
C4 --> C5
C1 --> E1
C4 --> E1
C1 --> C2
C4 --> C2
C2 -.Events.-> D1
C2 -.Events.-> D2
C2 -.Events.-> D3
style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
```
For a higher-level overview see the [README](../README.md). For deep technical details see [architecture.md](architecture.md).
## Entities
An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.
An **entity** is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding.
### Entity Structure
### Entity structure
Every entity has three main parts:
Every entity has the same shape:
```mermaid
classDiagram
class Entity {
+string id
+string kind
+metadata
+spec
+annotations
+timestamps
}
class Metadata {
+string name
+string namespace
+string description
+labels
+links
+tags
}
class Spec {
+string type
+string lifecycle
+string owner
+dependencies
}
Entity --> Metadata
Entity --> Spec
```
- `id` — UUID assigned by the server
- `kind` — one of the supported kinds (see below)
- `metadata` — identifying data (name, description, labels, tags, links)
- `spec` — kind-specific configuration (type, lifecycle, owner, dependencies, ...)
- `annotations` — arbitrary `string → string` map, typically used by plugins to store external IDs
- `created_at`, `updated_at` — server-managed timestamps
### Entity Kinds
`metadata` and `spec` are protobuf `oneof` fields whose variant matches `kind`. So a `Component` entity carries `component_metadata` + `component_spec`, a `User` carries `user_metadata` + `user_spec`, and so on. The on-wire JSON uses the variant name directly.
Charybdis supports all standard Backstage entity kinds:
### Entity kinds
| Kind | Description | Example |
|------|-------------|---------|
| **Service** | Individual microservices or applications | `payment-api` |
| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
| **System** | Collections of services working together | `e-commerce-platform` |
| **API** | Interfaces exposed by components | `payments-rest-api` |
| **User** | Individual people | `john.doe` |
| **Group** | Teams and organizational units | `team-payments` |
| **Domain** | Business domains | `payments`, `shipping` |
| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
| Kind | Purpose |
|---|---|
| `Component` | Services, applications, libraries, websites |
| `System` | Collections of components working together |
| `API` | Interfaces exposed by components |
| `User` | Individual people |
| `Group` | Teams and organizational units |
| `Domain` | Business domains |
| `Resource` | Infrastructure resources (databases, caches, queues) |
| `Finding` | Security findings ingested from scanners (see [Findings](#findings)) |
#### Example: Service
Per-kind protobuf definitions: `proto/core/*.proto`.
### Example: Component
```json
{
"kind": "Service",
"service_metadata": {
"kind": "Component",
"component_metadata": {
"name": "payment-api",
"namespace": "production",
"description": "Payment processing service",
"labels": { "team": "payments" }
"labels": { "team": "payments", "tier": "critical" },
"tags": ["api", "pci-dss"]
},
"service_spec": {
"component_spec": {
"type": "service",
"lifecycle": "production",
"owner": "team-payments",
@@ -129,500 +57,136 @@ Charybdis supports all standard Backstage entity kinds:
}
```
#### Example: Component
### Common metadata fields
```json
{
"kind": "Component",
"component_metadata": {
"name": "auth-sdk",
"namespace": "shared"
},
"component_spec": {
"type": "library",
"lifecycle": "production",
"owner": "platform-team"
}
}
```
| Field | Type | Notes |
|---|---|---|
| `name` | string | Required. Unique with `kind` (composite index `idx_entities_kind_name`). |
| `namespace` | string | Logical grouping, e.g. `production`. |
| `description` | string | Human-readable. |
| `labels` | `map<string,string>` | Categorization. |
| `tags` | `repeated string` | Free-form classification. |
| `links` | `repeated Link` | External URLs (dashboards, docs). |
#### Example: System
## Annotations
```json
{
"kind": "System",
"system_metadata": {
"name": "e-commerce-platform",
"namespace": "production",
"description": "Complete e-commerce system"
},
"system_spec": {
"owner": "platform-team",
"domain": "retail"
}
}
```
### Metadata Fields
| Field | Type | Description | Required |
|-------|------|-------------|----------|
| `name` | string | Entity name (unique within namespace) | ✅ |
| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
| `description` | string | Human-readable description | ❌ |
| `labels` | map | Key-value pairs for categorization | ❌ |
| `tags` | array | Search tags | ❌ |
| `links` | array | External URLs (dashboards, docs, etc.) | ❌ |
### Annotations
Annotations store integration-specific metadata:
Annotations are an open `map<string, string>` on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB.
```json
{
"annotations": {
"github.com/repo-slug": "myorg/payment-service",
"defectdojo.com/product-id": "123",
"dependencytrack.com/project-uuid": "550e8400...",
"pagerduty.com/service-id": "PXYZ123",
"grafana.com/dashboard-url": "https://..."
"keycloak.com/email": "alice@example.com",
"pagerduty.com/service-id": "PXYZ123"
}
}
```
**Best Practices**:
- Use domain-style keys (`tool.com/key`)
- Store tool-specific IDs
- Keep values as strings
- Use for integration metadata only
Convention: reverse-DNS keys (`tool.com/key`). Values are always strings.
`EntityRepository::update_annotations()` performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers.
## Events
Charybdis uses an **event-driven architecture** to trigger actions when entities change.
Charybdis emits an `EntityEvent` on every CRUD operation, delivered to subscribers via the in-memory event bus.
### Event Flow
```mermaid
sequenceDiagram
participant Client
participant API as Entity Service
participant Bus as Event Bus
participant Plugin1 as DefectDojo Plugin
participant Plugin2 as Dependency-Track
Client->>API: CreateEntity(service)
API->>API: Store entity
API->>Bus: Emit EntityCreated event
Bus->>Plugin1: Handle event
Bus->>Plugin2: Handle event
Plugin1-->>Plugin1: Create DefectDojo product
Plugin2-->>Plugin2: Create DT project
API-->>Client: Return created entity
Note over Bus,Plugin2: Asynchronous processing
```
Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers
```
### Event Types
### Event types
| Event | Trigger | Plugins Receive |
|-------|---------|----------------|
| `EntityCreated` | New entity created | Full entity data |
| `EntityUpdated` | Entity modified | Updated entity + changes |
| `EntityDeleted` | Entity removed | Entity ID + metadata |
| Type | When |
|---|---|
| `Created` | Successful `CreateEntity` |
| `Updated` | Successful `UpdateEntity` (full or partial) |
| `Deleted` | Successful `DeleteEntity` |
### Event Structure
### Event payload
```rust
pub enum EntityEvent {
Created {
entity: Entity,
timestamp: DateTime<Utc>,
},
Updated {
entity: Entity,
previous: Entity,
timestamp: DateTime<Utc>,
},
Deleted {
id: String,
metadata: Metadata,
timestamp: DateTime<Utc>,
},
pub struct EntityEvent {
pub event_id: Uuid,
pub entity_id: Uuid,
pub event_type: EntityEventType, // Created | Updated | Deleted
pub timestamp: DateTime<Utc>,
pub metadata: HashMap<String, String>,
pub entity_data: Option<Arc<Entity>>, // full entity for Created/Updated
}
```
## Vulnerabilities & Security (Phase 1 — Planned)
### Delivery semantics
> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.
- **Asynchronous** — `CreateEntity` returns to the client before plugin handlers complete.
- **At-least-once intent**, but the default `MemoryEventBus` is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in [VISION.md](../VISION.md).
- **Per-handler timeout** — 30 s. Slow plugins don't block the bus.
- **Panic isolation** — a panicking handler doesn't take down others.
Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.
Plugin lifecycle and how to write a handler: [plugins/README.md](../plugins/README.md).
### Vulnerability Lifecycle
## Findings
```mermaid
sequenceDiagram
participant Scanner
participant API as Charybdis API
participant Rules as Rules Engine
participant Gate as Security Gate
participant Bus as Event Bus
participant Plugin as Slack / Jira
`Finding` is a first-class entity kind for security findings produced by scanners. Findings are created by the `IngestionService`, not directly by clients.
Scanner->>API: IngestScan(SARIF report)
API->>API: Parse & create Vulnerability entities
API->>API: Link vulnerabilities to Component
API->>Rules: Evaluate auto-assessment rules
Rules-->>API: Auto-assess (e.g., accept known low-risk)
API->>Gate: Check security gate thresholds
alt Gate Passed
Gate-->>API: OK
else Gate Failed
Gate->>Bus: GateFailed event
Bus->>Plugin: Alert #security channel
end
API->>Bus: VulnerabilitiesIngested event
Bus->>Plugin: Notify / create tickets
```
Each finding is scoped to a `(component_ref, lifecycle)` pair — so production and staging environments track findings independently without duplicating the underlying `Component`.
### Scan Ingestion
The ingestion flow:
Charybdis ingests scan results natively. You don't need an external vulnerability management tool.
1. Client calls `IngestionService.ImportScan(component_ref, lifecycle, format, data)`.
2. The configured parser (currently SARIF) normalizes scanner output into `NormalizedFinding` records.
3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`) and diffs incoming vs existing findings for that scope.
4. The result is one of:
- **New** — fingerprint not seen before → create a `Finding` entity.
- **Unchanged** — already active → bump `last_seen`.
- **Resolved** — previously active, not in this scan → mark `Resolved`.
- **Reopened** — previously resolved or false-positive, detected again → mark `Reopened`.
5. `DryRunScan` runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments.
| Format | Coverage | Use Case |
|--------|----------|----------|
| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
| **CycloneDX** | SBOMs + vulnerability data | SCA, license |
| **SPDX** | License and package data | License compliance |
Finding state values (see `proto/core/finding.proto`):
### Assessments
| State | Meaning |
|---|---|
| `ACTIVE` | Currently detected |
| `RESOLVED` | No longer detected (auto-set on re-import) |
| `ACCEPTED` | Risk accepted by human decision *(state value defined; assessment workflow is planned)* |
| `FALSE_POSITIVE` | Marked as not a real issue *(state value defined; assessment workflow is planned)* |
| `REOPENED` | Was resolved, detected again |
Each vulnerability can be assessed:
| Status | Meaning |
|--------|---------|
| **Open** | New, unreviewed vulnerability |
| **In Triage** | Under review by security team |
| **Accepted** | Risk accepted with justification |
| **Remediated** | Fixed, pending verification |
| **False Positive** | Not a real vulnerability |
| **Auto-Assessed** | Automatically assessed by rules engine |
### Security Gates
Security gates define thresholds per product:
```
payment-api:
critical: 0 # No critical vulns allowed
high: 5 # Up to 5 high
medium: 20 # Up to 20 medium
```
When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).
### Rules Engine
Rules auto-assess vulnerabilities based on patterns:
- Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
- Scanner source (e.g., "all informational from ZAP → auto-accept")
- Known patterns (e.g., "CVE-XXXX already accepted org-wide")
### License Compliance (Phase 2)
Track licenses across your dependency tree:
- Ingest license data from CycloneDX/SPDX
- Define license policies (allowed, restricted, banned)
- Flag violations per component
- Compliance reporting
Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).
## Storage Model
Charybdis uses PostgreSQL with JSONB for schema-less storage.
### Database Schema
Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a `oneof` variant in protobuf, not a column migration.
```sql
CREATE TABLE entities (
id UUID PRIMARY KEY,
kind TEXT NOT NULL,
entity_data BYTEA NOT NULL, -- Protobuf binary
annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
id UUID PRIMARY KEY,
kind VARCHAR NOT NULL,
name VARCHAR,
entity_data BYTEA NOT NULL, -- protobuf-encoded Entity
annotations JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_entities_kind ON entities(kind);
CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
CREATE INDEX idx_entities_kind ON entities(kind);
CREATE INDEX idx_entities_kind_name ON entities(kind, name);
CREATE INDEX idx_entities_annotations ON entities USING GIN (annotations);
CREATE INDEX idx_entities_created_at_id ON entities(created_at DESC, id DESC);
```
### Why Protobuf + JSONB?
Why this design:
- **No migrations** — new plugins extend the protobuf `oneof` variants; database schema is unchanged.
- **Compact storage** — protobuf binary is smaller than JSON.
- **Forward/backward compatible** — proto field numbers protect against version drift.
- **Fast annotation queries** — JSONB + GIN index supports `annotations->>'key' = 'value'` lookups in O(log n).
**Advantages**:
- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
- ✅ Protobuf binary storage for compact, versioned entity data
- ✅ JSONB annotations for fast querying with GIN indexes
- ✅ Forward/backward compatibility built-in
Detailed schema reasoning, scaling notes, and field-mask paths: [architecture.md](architecture.md).
**Example Query**:
```sql
-- Find entities with specific annotation
SELECT * FROM entities
WHERE annotations->>'defectdojo.com/product-id' = '42';
```
## See Also
## Security Model
Charybdis implements defense-in-depth security.
### Authentication: mTLS
```mermaid
sequenceDiagram
participant Client
participant TLS as TLS Layer
participant Auth as Auth Interceptor
participant Service as Entity Service
Client->>TLS: Connect with client certificate
TLS->>TLS: Validate certificate
TLS->>Auth: Extract certificate
Auth->>Auth: Parse identity (CN, OU, O)
Auth->>Auth: Map to role
Client->>Auth: Request (with identity)
Auth->>Auth: Check permissions
alt Authorized
Auth->>Service: Forward request
Service-->>Client: Response
else Denied
Auth-->>Client: PermissionDenied error
end
```
### Authorization: RBAC
**Role Mapping**:
```
Certificate (CN, OU, O) → RBAC Role → Permissions
```
**Default Roles**:
| Role | Certificate OU | Permissions |
|------|---------------|-------------|
| `platform` | `platform-team` | Full access (CRUD + list) |
| `automation` | `automation` | Create, read, update, list |
| `plugin` | `plugins` | Read, list only |
### Permissions
| Permission | Operations | Required For |
|------------|-----------|--------------|
| `entity:create` | Create new entities | CreateEntity |
| `entity:read` | Get entity by ID | GetEntity |
| `entity:update` | Modify entities | UpdateEntity |
| `entity:delete` | Remove entities | DeleteEntity |
| `entity:list` | List all entities | ListEntities |
## Plugin System
Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.
### Plugin Architecture
```mermaid
graph LR
A[Entity/Vuln Event] --> B[Event Bus]
B --> C{Plugin Manager}
C --> D[Slack Plugin]
C --> E[Jira Plugin]
C --> F[Custom Plugin]
D --> G[Slack API]
E --> H[Jira API]
F --> I[Your Tool API]
style C fill:#4A90E2,stroke:#2E5C8A
```
### What's Native vs. Plugin
| Native (core) | Status | Plugin (integration) | Status |
|---|---|---|---|
| Software catalog | Done | DefectDojo sync | Done |
| Vulnerability management | Phase 1 | Keycloak user sync | Done |
| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
| License compliance | Phase 2 | Custom integrations | Framework ready |
| TechDocs | Future | | |
### Plugin Lifecycle
1. **Configuration** - Load plugin settings from `plugins.toml`
2. **Initialization** - Plugin registers event handlers
3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
4. **External Integration** - Plugin calls external tool APIs
5. **Error Handling** - Failed plugins don't affect core service
### Plugin Configuration
Example `plugins.toml`:
```toml
[plugins.slack]
enabled = true
webhook_url = "${SLACK_WEBHOOK_URL}"
[[plugins.slack.on_gate_failed]]
channel = "#security-alerts"
[[plugins.slack.on_entity_created]]
channel = "#platform"
```
See [Plugin Guide](plugins.md) for details.
## Data Consistency
### Eventual Consistency
Charybdis uses **eventual consistency** for plugin integrations:
- Entity CRUD operations are **immediately consistent**
- Plugin synchronization is **eventually consistent**
- Events are processed **asynchronously**
```mermaid
graph LR
A[CreateEntity] -->|Immediate| B[Entity Stored]
B -->|Async| C[Event Emitted]
C -->|Async| D[Plugin Processing]
D -->|Eventual| E[External Tool Synced]
style B fill:#00C851
style E fill:#ffbb33
```
### Guarantees
| Operation | Consistency | Guarantee |
|-----------|-------------|-----------|
| Entity CRUD | Strong | Immediate |
| Entity queries | Strong | Read-your-writes |
| Event delivery | At-least-once | May retry |
| Plugin sync | Eventual | Best-effort |
## Performance Characteristics
### Scalability
- **Entities**: Tested with 100,000+ entities
- **Throughput**: 1,000+ requests/second
- **Latency**: Sub-millisecond average
- **Concurrency**: Tokio async runtime
### Resource Usage
Typical resource consumption:
| Component | CPU | Memory | Storage |
|-----------|-----|--------|---------|
| Charybdis | < 5% | ~50 MB | Minimal |
| PostgreSQL | ~10% | ~256 MB | Depends on entity count |
### Optimization Tips
1. **Index annotations** used for frequent queries
2. **Use connection pooling** (built-in)
3. **Enable query caching** in PostgreSQL
4. **Monitor event bus** queue depth
## Backstage Migration
Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.
### How It Works
```mermaid
graph LR
A[Charybdis Entity] --> B[YAML Adapter]
B --> C[Backstage YAML Format]
C --> D[Backstage Catalog]
style B fill:#4A90E2,stroke:#2E5C8A
```
Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:
```yaml
catalog:
locations:
- type: url
target: http://charybdis:8080/yaml/locations
```
### Recommended Migration Path
1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
4. **Optionally retire** Backstage when Charybdis covers your catalog needs
## Best Practices
### Entity Design
✅ **DO**:
- Use descriptive names
- Group by namespace
- Add relevant labels
- Include documentation links
- Set appropriate owners
❌ **DON'T**:
- Store sensitive data in metadata
- Use very long descriptions
- Create deeply nested hierarchies
- Duplicate data across entities
### Naming Conventions
```
<entity-type>-<purpose>-<environment>
Examples:
- payment-api-prod
- user-service-staging
- auth-library
- e-commerce-system
```
### Metadata Organization
```json
{
"labels": {
"team": "payments", // Ownership
"tier": "critical", // Importance
"environment": "production" // Deployment
},
"tags": ["pci-compliant", "public-api"],
"links": [
{ "url": "...", "title": "Dashboard" },
{ "url": "...", "title": "Documentation" }
]
}
```
## Next Steps
- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
- Configure [Security](security.md) for production (mTLS + RBAC)
- Explore [Plugins](plugins.md) for external integrations
- Review the [Architecture](architecture.md) for technical deep dive
---
**Questions?** [Open an issue](../../issues).
- [Architecture](architecture.md) — protobuf schema, scaling, design decisions
- [Security](security.md) — mTLS + RBAC configuration
- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
- [Vision & Roadmap](../VISION.md) — planned features