guillaume.grabe/charybdis
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial-commit

Browse Source
This commit is contained in:
Guillaume GRABÉ
2026-05-12 17:06:43 +02:00
commit 051a080dfa
110 changed files with 26377 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/core-concepts.md
+628
View File
@@ -0,0 +1,628 @@
# Core Concepts
Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.
This page explains the key concepts you need to understand.
## Architecture Overview
```mermaid
graph TB
subgraph "Sources"
A1[CI/CD Pipelines]
A2[Security Scanners]
A3[IaC Tools]
end
subgraph "Charybdis Core"
B1[gRPC API]
B3[Authentication & Authorization]
C1[Entity Service]
C4[Vulnerability Engine]
C5[Security Gates & Rules]
C2[Event Bus]
end
subgraph "Integrations - Plugins"
D1[Slack / Teams]
D2[Jira / GitHub]
D3[Custom Plugins]
end
subgraph "Data Layer"
E1[(PostgreSQL)]
end
A1 -.gRPC.-> B1
A2 -.Scan Results.-> B1
A3 -.gRPC.-> B1
B1 --> B3
B3 --> C1
B3 --> C4
C4 --> C5
C1 --> E1
C4 --> E1
C1 --> C2
C4 --> C2
C2 -.Events.-> D1
C2 -.Events.-> D2
C2 -.Events.-> D3
style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
```
## Entities
An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.
### Entity Structure
Every entity has three main parts:
```mermaid
classDiagram
class Entity {
+string id
+string kind
+metadata
+spec
+annotations
+timestamps
}
class Metadata {
+string name
+string namespace
+string description
+labels
+links
+tags
}
class Spec {
+string type
+string lifecycle
+string owner
+dependencies
}
Entity --> Metadata
Entity --> Spec
```
### Entity Kinds
Charybdis supports all standard Backstage entity kinds:
| Kind | Description | Example |
|------|-------------|---------|
| **Service** | Individual microservices or applications | `payment-api` |
| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
| **System** | Collections of services working together | `e-commerce-platform` |
| **API** | Interfaces exposed by components | `payments-rest-api` |
| **User** | Individual people | `john.doe` |
| **Group** | Teams and organizational units | `team-payments` |
| **Domain** | Business domains | `payments`, `shipping` |
| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
#### Example: Service
```json
{
"kind": "Service",
"service_metadata": {
"name": "payment-api",
"namespace": "production",
"description": "Payment processing service",
"labels": { "team": "payments" }
},
"service_spec": {
"type": "service",
"lifecycle": "production",
"owner": "team-payments",
"system": "payment-system"
}
}
```
#### Example: Component
```json
{
"kind": "Component",
"component_metadata": {
"name": "auth-sdk",
"namespace": "shared"
},
"component_spec": {
"type": "library",
"lifecycle": "production",
"owner": "platform-team"
}
}
```
#### Example: System
```json
{
"kind": "System",
"system_metadata": {
"name": "e-commerce-platform",
"namespace": "production",
"description": "Complete e-commerce system"
},
"system_spec": {
"owner": "platform-team",
"domain": "retail"
}
}
```
### Metadata Fields
| Field | Type | Description | Required |
|-------|------|-------------|----------|
| `name` | string | Entity name (unique within namespace) | ✅ |
| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
| `description` | string | Human-readable description | ❌ |
| `labels` | map | Key-value pairs for categorization | ❌ |
| `tags` | array | Search tags | ❌ |
| `links` | array | External URLs (dashboards, docs, etc.) | ❌ |
### Annotations
Annotations store integration-specific metadata:
```json
{
"annotations": {
"github.com/repo-slug": "myorg/payment-service",
"defectdojo.com/product-id": "123",
"dependencytrack.com/project-uuid": "550e8400...",
"pagerduty.com/service-id": "PXYZ123",
"grafana.com/dashboard-url": "https://..."
}
}
```
**Best Practices**:
- Use domain-style keys (`tool.com/key`)
- Store tool-specific IDs
- Keep values as strings
- Use for integration metadata only
## Events
Charybdis uses an **event-driven architecture** to trigger actions when entities change.
### Event Flow
```mermaid
sequenceDiagram
participant Client
participant API as Entity Service
participant Bus as Event Bus
participant Plugin1 as DefectDojo Plugin
participant Plugin2 as Dependency-Track
Client->>API: CreateEntity(service)
API->>API: Store entity
API->>Bus: Emit EntityCreated event
Bus->>Plugin1: Handle event
Bus->>Plugin2: Handle event
Plugin1-->>Plugin1: Create DefectDojo product
Plugin2-->>Plugin2: Create DT project
API-->>Client: Return created entity
Note over Bus,Plugin2: Asynchronous processing
```
### Event Types
| Event | Trigger | Plugins Receive |
|-------|---------|----------------|
| `EntityCreated` | New entity created | Full entity data |
| `EntityUpdated` | Entity modified | Updated entity + changes |
| `EntityDeleted` | Entity removed | Entity ID + metadata |
### Event Structure
```rust
pub enum EntityEvent {
Created {
entity: Entity,
timestamp: DateTime<Utc>,
},
Updated {
entity: Entity,
previous: Entity,
timestamp: DateTime<Utc>,
},
Deleted {
id: String,
metadata: Metadata,
timestamp: DateTime<Utc>,
},
}
```
## Vulnerabilities & Security (Phase 1 — Planned)
> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.
Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.
### Vulnerability Lifecycle
```mermaid
sequenceDiagram
participant Scanner
participant API as Charybdis API
participant Rules as Rules Engine
participant Gate as Security Gate
participant Bus as Event Bus
participant Plugin as Slack / Jira
Scanner->>API: IngestScan(SARIF report)
API->>API: Parse & create Vulnerability entities
API->>API: Link vulnerabilities to Component
API->>Rules: Evaluate auto-assessment rules
Rules-->>API: Auto-assess (e.g., accept known low-risk)
API->>Gate: Check security gate thresholds
alt Gate Passed
Gate-->>API: OK
else Gate Failed
Gate->>Bus: GateFailed event
Bus->>Plugin: Alert #security channel
end
API->>Bus: VulnerabilitiesIngested event
Bus->>Plugin: Notify / create tickets
```
### Scan Ingestion
Charybdis ingests scan results natively. You don't need an external vulnerability management tool.
| Format | Coverage | Use Case |
|--------|----------|----------|
| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
| **CycloneDX** | SBOMs + vulnerability data | SCA, license |
| **SPDX** | License and package data | License compliance |
### Assessments
Each vulnerability can be assessed:
| Status | Meaning |
|--------|---------|
| **Open** | New, unreviewed vulnerability |
| **In Triage** | Under review by security team |
| **Accepted** | Risk accepted with justification |
| **Remediated** | Fixed, pending verification |
| **False Positive** | Not a real vulnerability |
| **Auto-Assessed** | Automatically assessed by rules engine |
### Security Gates
Security gates define thresholds per product:
```
payment-api:
critical: 0 # No critical vulns allowed
high: 5 # Up to 5 high
medium: 20 # Up to 20 medium
```
When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).
### Rules Engine
Rules auto-assess vulnerabilities based on patterns:
- Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
- Scanner source (e.g., "all informational from ZAP → auto-accept")
- Known patterns (e.g., "CVE-XXXX already accepted org-wide")
### License Compliance (Phase 2)
Track licenses across your dependency tree:
- Ingest license data from CycloneDX/SPDX
- Define license policies (allowed, restricted, banned)
- Flag violations per component
- Compliance reporting
## Storage Model
Charybdis uses PostgreSQL with JSONB for schema-less storage.
### Database Schema
```sql
CREATE TABLE entities (
id UUID PRIMARY KEY,
kind TEXT NOT NULL,
entity_data BYTEA NOT NULL, -- Protobuf binary
annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
CREATE INDEX idx_entities_kind ON entities(kind);
CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
```
### Why Protobuf + JSONB?
**Advantages**:
- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
- ✅ Protobuf binary storage for compact, versioned entity data
- ✅ JSONB annotations for fast querying with GIN indexes
- ✅ Forward/backward compatibility built-in
**Example Query**:
```sql
-- Find entities with specific annotation
SELECT * FROM entities
WHERE annotations->>'defectdojo.com/product-id' = '42';
```
## Security Model
Charybdis implements defense-in-depth security.
### Authentication: mTLS
```mermaid
sequenceDiagram
participant Client
participant TLS as TLS Layer
participant Auth as Auth Interceptor
participant Service as Entity Service
Client->>TLS: Connect with client certificate
TLS->>TLS: Validate certificate
TLS->>Auth: Extract certificate
Auth->>Auth: Parse identity (CN, OU, O)
Auth->>Auth: Map to role
Client->>Auth: Request (with identity)
Auth->>Auth: Check permissions
alt Authorized
Auth->>Service: Forward request
Service-->>Client: Response
else Denied
Auth-->>Client: PermissionDenied error
end
```
### Authorization: RBAC
**Role Mapping**:
```
Certificate (CN, OU, O) → RBAC Role → Permissions
```
**Default Roles**:
| Role | Certificate OU | Permissions |
|------|---------------|-------------|
| `platform` | `platform-team` | Full access (CRUD + list) |
| `automation` | `automation` | Create, read, update, list |
| `plugin` | `plugins` | Read, list only |
### Permissions
| Permission | Operations | Required For |
|------------|-----------|--------------|
| `entity:create` | Create new entities | CreateEntity |
| `entity:read` | Get entity by ID | GetEntity |
| `entity:update` | Modify entities | UpdateEntity |
| `entity:delete` | Remove entities | DeleteEntity |
| `entity:list` | List all entities | ListEntities |
## Plugin System
Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.
### Plugin Architecture
```mermaid
graph LR
A[Entity/Vuln Event] --> B[Event Bus]
B --> C{Plugin Manager}
C --> D[Slack Plugin]
C --> E[Jira Plugin]
C --> F[Custom Plugin]
D --> G[Slack API]
E --> H[Jira API]
F --> I[Your Tool API]
style C fill:#4A90E2,stroke:#2E5C8A
```
### What's Native vs. Plugin
| Native (core) | Status | Plugin (integration) | Status |
|---|---|---|---|
| Software catalog | Done | DefectDojo sync | Done |
| Vulnerability management | Phase 1 | Keycloak user sync | Done |
| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
| License compliance | Phase 2 | Custom integrations | Framework ready |
| TechDocs | Future | | |
### Plugin Lifecycle
1. **Configuration** - Load plugin settings from `plugins.toml`
2. **Initialization** - Plugin registers event handlers
3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
4. **External Integration** - Plugin calls external tool APIs
5. **Error Handling** - Failed plugins don't affect core service
### Plugin Configuration
Example `plugins.toml`:
```toml
[plugins.slack]
enabled = true
webhook_url = "${SLACK_WEBHOOK_URL}"
[[plugins.slack.on_gate_failed]]
channel = "#security-alerts"
[[plugins.slack.on_entity_created]]
channel = "#platform"
```
See [Plugin Guide](plugins.md) for details.
## Data Consistency
### Eventual Consistency
Charybdis uses **eventual consistency** for plugin integrations:
- Entity CRUD operations are **immediately consistent**
- Plugin synchronization is **eventually consistent**
- Events are processed **asynchronously**
```mermaid
graph LR
A[CreateEntity] -->|Immediate| B[Entity Stored]
B -->|Async| C[Event Emitted]
C -->|Async| D[Plugin Processing]
D -->|Eventual| E[External Tool Synced]
style B fill:#00C851
style E fill:#ffbb33
```
### Guarantees
| Operation | Consistency | Guarantee |
|-----------|-------------|-----------|
| Entity CRUD | Strong | Immediate |
| Entity queries | Strong | Read-your-writes |
| Event delivery | At-least-once | May retry |
| Plugin sync | Eventual | Best-effort |
## Performance Characteristics
### Scalability
- **Entities**: Tested with 100,000+ entities
- **Throughput**: 1,000+ requests/second
- **Latency**: Sub-millisecond average
- **Concurrency**: Tokio async runtime
### Resource Usage
Typical resource consumption:
| Component | CPU | Memory | Storage |
|-----------|-----|--------|---------|
| Charybdis | < 5% | ~50 MB | Minimal |
| PostgreSQL | ~10% | ~256 MB | Depends on entity count |
### Optimization Tips
1. **Index annotations** used for frequent queries
2. **Use connection pooling** (built-in)
3. **Enable query caching** in PostgreSQL
4. **Monitor event bus** queue depth
## Backstage Migration
Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.
### How It Works
```mermaid
graph LR
A[Charybdis Entity] --> B[YAML Adapter]
B --> C[Backstage YAML Format]
C --> D[Backstage Catalog]
style B fill:#4A90E2,stroke:#2E5C8A
```
Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:
```yaml
catalog:
locations:
- type: url
target: http://charybdis:8080/yaml/locations
```
### Recommended Migration Path
1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
4. **Optionally retire** Backstage when Charybdis covers your catalog needs
## Best Practices
### Entity Design
✅ **DO**:
- Use descriptive names
- Group by namespace
- Add relevant labels
- Include documentation links
- Set appropriate owners
❌ **DON'T**:
- Store sensitive data in metadata
- Use very long descriptions
- Create deeply nested hierarchies
- Duplicate data across entities
### Naming Conventions
```
<entity-type>-<purpose>-<environment>
Examples:
- payment-api-prod
- user-service-staging
- auth-library
- e-commerce-system
```
### Metadata Organization
```json
{
"labels": {
"team": "payments", // Ownership
"tier": "critical", // Importance
"environment": "production" // Deployment
},
"tags": ["pci-compliant", "public-api"],
"links": [
{ "url": "...", "title": "Dashboard" },
{ "url": "...", "title": "Documentation" }
]
}
```
## Next Steps
- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
- Configure [Security](security.md) for production (mTLS + RBAC)
- Explore [Plugins](plugins.md) for external integrations
- Review the [Architecture](architecture.md) for technical deep dive
---
**Questions?** [Open an issue](../../issues).