Public Access
initial-commit
This commit is contained in:
Guillaume GRABÉ
@@ -0,0 +1,985 @@
|
||||
# Charybdis - Architecture
|
||||
|
||||
**Version**: 2.0
|
||||
**Last Updated**: 2026-05-06
|
||||
**Status**: Living Document
|
||||
|
||||
> For the product vision and roadmap, see [VISION.md](../VISION.md).
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Charybdis** is a security-native platform engineering tool that unifies software catalog, vulnerability management, and compliance posture in a single event-driven platform. Built in Rust, deployed as a single binary.
|
||||
|
||||
### Core Value Proposition
|
||||
|
||||
**Problem**:
|
||||
- Platform engineers run 5+ disconnected tools (Backstage, DefectDojo, Dependency-Track, license scanners, compliance spreadsheets)
|
||||
- Software catalogs rely on static YAML files that drift from reality
|
||||
- Security posture is invisible at the catalog level — vulns live in separate tools with no link to services
|
||||
- Compliance evidence is assembled manually from fragmented data sources
|
||||
|
||||
**Solution**:
|
||||
- **Dynamic software catalog** — event-driven, gRPC-native, no static YAML
|
||||
- **Native vulnerability management** — ingest scan results (SARIF, CycloneDX, SPDX), triage, assess, track
|
||||
- **Security gates & compliance** — severity thresholds, license policies, compliance framework mappings
|
||||
- **Plugin system for integrations** — Slack, Jira, GitHub, custom tools react to events
|
||||
- **Single binary deployment** — Rust + PostgreSQL, ~50MB RAM, sub-millisecond latency
|
||||
|
||||
### Use Case Examples
|
||||
|
||||
**Service registration with auto-provisioning:**
|
||||
```
|
||||
CI/CD Pipeline (Python) → gRPC CreateEntity(kind=Component)
|
||||
↓
|
||||
Charybdis persists entity
|
||||
↓
|
||||
Event: EntityCreated published
|
||||
↓
|
||||
┌────────────────────┴────────────────────┐
|
||||
↓ ↓
|
||||
Jira Plugin Slack Plugin
|
||||
- Creates onboarding epic - Notifies #platform channel
|
||||
↓ ↓
|
||||
└────────────────────┬────────────────────┘
|
||||
↓
|
||||
Entity visible via gRPC API / Backstage YAML adapter
|
||||
```
|
||||
|
||||
**Vulnerability ingestion with security gates:**
|
||||
```
|
||||
Scanner (Trivy) → CI/CD → gRPC IngestScan(sarif_report)
|
||||
↓
|
||||
Charybdis parses SARIF
|
||||
Creates Vulnerability entities linked to Component
|
||||
↓
|
||||
Rules engine evaluates auto-assessment
|
||||
Security gate checks thresholds
|
||||
↓
|
||||
┌────────────────────┴────────────────────┐
|
||||
↓ ↓
|
||||
Gate PASSED Gate FAILED
|
||||
- Vulns tracked - Slack alert to #security
|
||||
- Dashboard updated - Jira ticket created
|
||||
- CI/CD pipeline blocked
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architectural Principles
|
||||
|
||||
### 1. **No Database Migrations Ever**
|
||||
|
||||
The database schema is created once on first startup and **never changes**.
|
||||
|
||||
**How it works**:
|
||||
- Entities stored as protobuf bytes in `entity_data BYTEA` column
|
||||
- Plugins add new entity types via protobuf `oneof` variants
|
||||
- Build system regenerates code, database schema remains static
|
||||
|
||||
**Benefits**:
|
||||
- Deploy new plugins without downtime
|
||||
- No migration scripts to manage
|
||||
- Forward/backward compatibility built-in
|
||||
- Easy rollback (protobuf versioning)
|
||||
|
||||
### 2. **gRPC First, Everything Else is Adapter**
|
||||
|
||||
Charybdis provides **only gRPC APIs**. All other interfaces (REST, YAML, GraphQL) are adapters on top.
|
||||
|
||||
**Rationale**:
|
||||
- **Strong typing**: Protobuf ensures type safety across all languages
|
||||
- **Performance**: Binary protocol, efficient serialization
|
||||
- **Multi-language**: Official gRPC clients for 10+ languages
|
||||
- **Streaming**: Built-in support for real-time updates (future)
|
||||
- **Code generation**: Automatic client/server code generation
|
||||
|
||||
**What Charybdis provides**:
|
||||
- ✅ gRPC API with protobuf definitions
|
||||
- ✅ YAML adapter endpoints (for Backstage compatibility/migration)
|
||||
- ❌ REST API (teams can add if needed via gateway)
|
||||
- ❌ GraphQL (teams can add if needed)
|
||||
|
||||
### 3. **Plugin-Based Extensibility**
|
||||
|
||||
Plugins are **compile-time integrated** Rust crates that:
|
||||
|
||||
1. **Define entity types** via protobuf schemas
|
||||
2. **React to events** via EventHandler trait
|
||||
3. **Call external APIs** to synchronize state
|
||||
4. **Store references** in entity annotations
|
||||
5. **(Future) Extend gRPC API** with custom endpoints
|
||||
|
||||
**Plugin characteristics**:
|
||||
- Not hot-swappable (require rebuild)
|
||||
- Type-safe at compile time
|
||||
- No runtime dependency resolution
|
||||
- Configurable via YAML config file
|
||||
- Can be developed by community
|
||||
|
||||
### 4. **Event-Driven Orchestration**
|
||||
|
||||
Events are published **after** entities are persisted:
|
||||
|
||||
```
|
||||
CRUD Operation → Persist to DB → Publish Event → Plugins React
|
||||
```
|
||||
|
||||
**Event flow**:
|
||||
1. Entity created/updated/deleted in database
|
||||
2. Event published to event bus with full entity data
|
||||
3. All subscribed plugin handlers receive event
|
||||
4. Plugins filter events they care about
|
||||
5. Plugins perform asynchronous actions
|
||||
6. Plugins update entity annotations (via repository)
|
||||
|
||||
**Error handling**:
|
||||
- Plugin errors **do not** fail the CRUD operation
|
||||
- Entity is already persisted before event publishing
|
||||
- Plugins log errors for observability
|
||||
- Retries handled by event bus backend
|
||||
|
||||
### 5. **Backstage Compatibility by Design**
|
||||
|
||||
Charybdis entity model is **100% compatible** with Backstage's descriptor format.
|
||||
|
||||
**How it works**:
|
||||
- Core protobuf types mirror Backstage YAML structure
|
||||
- YAML adapter endpoints serve entities in Backstage format
|
||||
- Backstage reads Charybdis as a dynamic location provider
|
||||
- No changes needed to Backstage frontend
|
||||
|
||||
**Key compatibility points**:
|
||||
- Entity `kind`, `apiVersion`, `metadata`, `spec` structure
|
||||
- Annotation format (reverse-DNS style)
|
||||
- Relationship types (`dependsOn`, `partOf`, etc.)
|
||||
- All standard Backstage entity kinds supported
|
||||
|
||||
---
|
||||
|
||||
## System Architecture
|
||||
|
||||
### Component Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Client Layer │
|
||||
│ CI/CD Pipelines (Python, Go, Node.js, Bash, etc.) │
|
||||
│ Security Scanners (SARIF, CycloneDX, SPDX output) │
|
||||
│ IaC Tools (Terraform, Pulumi, Crossplane) │
|
||||
│ Backstage (via YAML adapter, optional) │
|
||||
└────────────────┬────────────────────────────────────────────────┘
|
||||
│
|
||||
│ gRPC / YAML HTTP
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Charybdis Core │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ gRPC API │ │
|
||||
│ │ EntityService: CRUD + List (with field masks) │ │
|
||||
│ │ ScanService: IngestScan (SARIF, CycloneDX, SPDX) │ │
|
||||
│ │ AssessmentService: Triage, Accept, Remediate │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Security Engine (native) │ │
|
||||
│ │ - Vulnerability tracking per entity │ │
|
||||
│ │ - Rules engine (auto-assessment) │ │
|
||||
│ │ - Security gates (severity thresholds) │ │
|
||||
│ │ - License compliance │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Adapters │ │
|
||||
│ │ - YAML Adapter (Backstage compat / migration) │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Repository Layer │ │
|
||||
│ │ - PostgreSQL with protobuf storage │ │
|
||||
│ │ - JSONB annotations for fast queries │ │
|
||||
│ │ - Zero-migration schema │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Event Bus System │ │
|
||||
│ │ - Memory backend (dev) │ │
|
||||
│ │ - Redis backend (prod, future) │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ Entity / Vulnerability / Gate events
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Plugin Layer (integrations) │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
|
||||
│ │ Slack/Teams │ │ Jira/GitHub │ │ Custom Plugins │ │
|
||||
│ │ Notifications │ │ Issue Track │ │ Your integrations │ │
|
||||
│ └──────┬────────┘ └──────┬───────┘ └──────────┬─────────────┘ │
|
||||
└─────────┼──────────────────┼─────────────────────┼──────────────┘
|
||||
▼ ▼ ▼
|
||||
Slack/Teams API Jira/GitHub API External APIs
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
#### Entity Creation Flow
|
||||
|
||||
```
|
||||
1. Client (CI/CD)
|
||||
↓ gRPC CreateEntityRequest
|
||||
2. Charybdis gRPC API
|
||||
↓ Validate entity
|
||||
↓ Generate UUID, timestamps
|
||||
3. Entity Repository
|
||||
↓ Serialize to protobuf bytes
|
||||
↓ INSERT INTO entities
|
||||
4. PostgreSQL
|
||||
↓ Entity persisted
|
||||
5. Event Bus
|
||||
↓ Publish EntityCreated event (with full entity data)
|
||||
6. Plugin Handlers (parallel)
|
||||
├─ DefectDojo: Create product → Store product_id in annotations
|
||||
├─ DependencyTrack: Create project → Store project_uuid in annotations
|
||||
└─ Custom: Custom actions
|
||||
7. Repository.update_annotations()
|
||||
↓ Merge annotations from plugins
|
||||
↓ UPDATE entities SET annotations = ...
|
||||
8. PostgreSQL
|
||||
↓ Entity updated with external tool IDs
|
||||
9. Return CreateEntityResponse to client
|
||||
```
|
||||
|
||||
#### Backstage Integration Flow
|
||||
|
||||
```
|
||||
1. Backstage Catalog Backend
|
||||
↓ HTTP GET /yaml/locations
|
||||
2. Charybdis YAML Adapter
|
||||
↓ Query repository.list_all()
|
||||
↓ Convert entities to Location YAML
|
||||
3. Backstage receives location list
|
||||
↓ Fetches each entity: GET /yaml/entities/:id
|
||||
4. Charybdis YAML Adapter
|
||||
↓ repository.get_by_id()
|
||||
↓ Convert entity protobuf → Backstage YAML
|
||||
5. Backstage ingests entities
|
||||
↓ Displays in UI with all annotations
|
||||
↓ Shows linked DefectDojo products, etc.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Entity Model
|
||||
|
||||
### Protobuf Schema
|
||||
|
||||
```protobuf
|
||||
message Entity {
|
||||
// System-managed fields
|
||||
string id = 1; // UUID (auto-generated)
|
||||
google.protobuf.Timestamp created_at = 21;
|
||||
google.protobuf.Timestamp updated_at = 22;
|
||||
|
||||
// Backstage-compatible fields
|
||||
string api_version = 2; // "backstage.io/v1alpha1"
|
||||
string kind = 3; // "Component", "API", "User", etc.
|
||||
|
||||
// Polymorphic metadata (identifying information)
|
||||
oneof metadata {
|
||||
// Core types (built-in)
|
||||
ComponentMetadata component_metadata = 10;
|
||||
APIMetadata api_metadata = 11;
|
||||
UserMetadata user_metadata = 12;
|
||||
GroupMetadata group_metadata = 13;
|
||||
SystemMetadata system_metadata = 14;
|
||||
DomainMetadata domain_metadata = 15;
|
||||
ResourceMetadata resource_metadata = 16;
|
||||
LocationMetadata location_metadata = 17;
|
||||
TemplateMetadata template_metadata = 18;
|
||||
|
||||
// Plugin types (dynamically added)
|
||||
// Example: DefectDojoProductMetadata defectdojo_product_metadata = 100;
|
||||
}
|
||||
|
||||
// Polymorphic spec (configuration and behavior)
|
||||
oneof spec {
|
||||
// Core types (built-in)
|
||||
ComponentSpec component_spec = 10;
|
||||
APISpec api_spec = 11;
|
||||
UserSpec user_spec = 12;
|
||||
GroupSpec group_spec = 13;
|
||||
SystemSpec system_spec = 14;
|
||||
DomainSpec domain_spec = 15;
|
||||
ResourceSpec resource_spec = 16;
|
||||
LocationSpec location_spec = 17;
|
||||
TemplateSpec template_spec = 18;
|
||||
|
||||
// Plugin types (dynamically added)
|
||||
// Example: DefectDojoProductSpec defectdojo_product_spec = 100;
|
||||
}
|
||||
|
||||
// Annotations - external tool references
|
||||
// Format: "tool.com/resource-type-attribute"
|
||||
// Examples:
|
||||
// "defectdojo.com/product-id": "42"
|
||||
// "dependencytrack.com/project-uuid": "550e8400-..."
|
||||
// "github.com/repo-slug": "myorg/myrepo"
|
||||
map<string, string> annotations = 20;
|
||||
}
|
||||
```
|
||||
|
||||
### Backstage Entity Kinds Mapping
|
||||
|
||||
All Backstage entity kinds must be supported:
|
||||
|
||||
| Backstage Kind | Charybdis Proto Type | Status |
|
||||
|----------------|---------------------|--------|
|
||||
| Component | ComponentMetadata/Spec | ✅ Done |
|
||||
| Service | ServiceMetadata/Spec | ✅ Done |
|
||||
| API | APIMetadata/Spec | ✅ Done |
|
||||
| User | UserMetadata/Spec | ✅ Done |
|
||||
| Group | GroupMetadata/Spec | ✅ Done |
|
||||
| System | SystemMetadata/Spec | ✅ Done |
|
||||
| Domain | DomainMetadata/Spec | ✅ Done |
|
||||
| Resource | ResourceMetadata/Spec | ✅ Done |
|
||||
| Location | LocationMetadata/Spec | Not planned (entities are event-driven, not file-based) |
|
||||
| Template | TemplateMetadata/Spec | Phase 3 (Scaffolder) |
|
||||
|
||||
---
|
||||
|
||||
## YAML Adapter Layer
|
||||
|
||||
### Purpose
|
||||
|
||||
Serve entities in Backstage-compatible YAML format over HTTP.
|
||||
|
||||
### API Specification
|
||||
|
||||
#### 1. List Locations
|
||||
|
||||
**Endpoint**: `GET /yaml/locations`
|
||||
|
||||
**Purpose**: Return a dynamic Location entity that lists all entities in Charybdis.
|
||||
|
||||
**Response Format** (Backstage Location YAML):
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Location
|
||||
metadata:
|
||||
name: charybdis-all-entities
|
||||
description: Dynamic location managed by Charybdis
|
||||
spec:
|
||||
type: charybdis
|
||||
targets:
|
||||
- http://charybdis.company.com/yaml/entities/service-payment
|
||||
- http://charybdis.company.com/yaml/entities/service-auth
|
||||
- http://charybdis.company.com/yaml/entities/api-payments-v1
|
||||
- http://charybdis.company.com/yaml/entities/user-john-doe
|
||||
# ... all entities
|
||||
```
|
||||
|
||||
**Implementation Notes**:
|
||||
- Query `SELECT id, kind FROM entities`
|
||||
- Generate URL for each entity: `/yaml/entities/{kind}-{id}`
|
||||
- Return as Backstage Location YAML
|
||||
- Cache response (invalidate on entity CRUD)
|
||||
|
||||
#### 2. Get Entity by ID
|
||||
|
||||
**Endpoint**: `GET /yaml/entities/:id`
|
||||
|
||||
**Purpose**: Return a specific entity in Backstage YAML format.
|
||||
|
||||
**Response Format** (Backstage Component YAML):
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: payment-service
|
||||
namespace: production
|
||||
description: Core payment processing service
|
||||
annotations:
|
||||
github.com/repo-slug: myorg/payment-service
|
||||
defectdojo.com/product-id: "42"
|
||||
dependencytrack.com/project-uuid: "550e8400-e29b-41d4-a716-446655440000"
|
||||
tags:
|
||||
- payments
|
||||
- critical
|
||||
links:
|
||||
- url: https://dashboard.company.com/payment-service
|
||||
title: Dashboard
|
||||
icon: dashboard
|
||||
spec:
|
||||
type: service
|
||||
lifecycle: production
|
||||
owner: payments-team
|
||||
system: payment-system
|
||||
dependsOn:
|
||||
- component:auth-service
|
||||
providesApis:
|
||||
- payments-api-v1
|
||||
```
|
||||
|
||||
**Implementation Notes**:
|
||||
- Query `repository.get_by_id(id)`
|
||||
- Convert Entity protobuf → Backstage YAML
|
||||
- Map `oneof metadata/spec` to appropriate YAML structure
|
||||
- Include all annotations
|
||||
- Return as `text/yaml` content type
|
||||
|
||||
#### 3. Configuration in Backstage
|
||||
|
||||
**Backstage app-config.yaml**:
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://charybdis.company.com/yaml/locations
|
||||
rules:
|
||||
- allow: [Component, API, User, Group, System, Domain, Resource]
|
||||
```
|
||||
|
||||
**How it works**:
|
||||
1. Backstage fetches `/yaml/locations` on catalog refresh
|
||||
2. Parses Location YAML to get entity URLs
|
||||
3. Fetches each entity URL (`/yaml/entities/:id`)
|
||||
4. Ingests entities into Backstage catalog
|
||||
5. Displays in UI with all metadata and annotations
|
||||
|
||||
---
|
||||
|
||||
## Plugin System
|
||||
|
||||
### Plugin Architecture
|
||||
|
||||
Charybdis supports **two types of plugins**, both compile-time integrated:
|
||||
|
||||
#### **1. Event-Driven Plugins**
|
||||
React to entity lifecycle events (create, update, delete):
|
||||
- **Example**: DefectDojo, DependencyTrack
|
||||
- **Implement**: `EventDrivenPlugin` trait
|
||||
- **Provide**: `ResourceHandler` implementations
|
||||
- **Triggered by**: Entity CRUD operations
|
||||
|
||||
#### **2. Sync Plugins**
|
||||
Pull data from external sources on a schedule:
|
||||
- **Example**: Okta, Keycloak, Active Directory
|
||||
- **Implement**: `SyncPlugin` trait
|
||||
- **Scheduled by**: Cron expressions
|
||||
- **Triggered by**: Time-based schedule or manual API call
|
||||
|
||||
### Generic Plugin Utilities (2025-01-15)
|
||||
|
||||
All plugins have access to reusable utilities in `src/plugins/`:
|
||||
|
||||
#### **PluginHttpClient**
|
||||
Generic HTTP client with multiple auth methods:
|
||||
- Token authentication (DefectDojo)
|
||||
- Bearer authentication (modern APIs)
|
||||
- API Key authentication (DependencyTrack)
|
||||
- Basic authentication (JIRA, Bitbucket)
|
||||
- Methods: `get()`, `post()`, `put()`, `patch()`, `delete()`
|
||||
|
||||
#### **AnnotationHelper**
|
||||
Consistent API for storing/retrieving plugin metadata:
|
||||
- `set_id()` / `get_id()` - Store external resource IDs
|
||||
- `set()` / `get()` - Store arbitrary metadata
|
||||
- Enforces naming: `{plugin}.com/{resource}-id`
|
||||
- Prevents annotation conflicts between plugins
|
||||
|
||||
#### **FieldMapper**
|
||||
Maps entity fields to external tool formats:
|
||||
- Direct field paths: `"metadata.name"`
|
||||
- Static values: `{"value": "Web Application"}`
|
||||
- Complex mappings with entity resolution
|
||||
- Repository integration for cross-entity queries
|
||||
|
||||
#### **DateUtils**
|
||||
Common date/time operations:
|
||||
- `today()`, `today_plus_days(n)` - Date formatting
|
||||
- `engagement_date_range(days)` - For time-bound resources
|
||||
- `iso_timestamp()` - ISO 8601 timestamps
|
||||
|
||||
### Plugin Responsibilities
|
||||
|
||||
Plugins can:
|
||||
|
||||
1. **Define custom entity types** (via protobuf)
|
||||
2. **React to entity events** (via ResourceHandler trait)
|
||||
3. **Call external APIs** (using PluginHttpClient)
|
||||
4. **Store metadata** (using AnnotationHelper)
|
||||
5. **Map fields** (using FieldMapper)
|
||||
6. **Schedule syncs** (SyncPlugins with cron)
|
||||
7. **(Future) Extend gRPC API** (custom services)
|
||||
|
||||
### Plugin Lifecycle
|
||||
|
||||
```
|
||||
1. Development
|
||||
├─ Define protobuf schema (metadata/spec messages)
|
||||
├─ Implement EventHandler trait
|
||||
├─ Implement external API client
|
||||
└─ Add configuration schema
|
||||
|
||||
2. Registration
|
||||
├─ Add to plugins.toml (enabled = true)
|
||||
└─ Add to Cargo.toml dependencies
|
||||
|
||||
3. Build Time
|
||||
├─ build.rs reads plugins.toml
|
||||
├─ Generates entities.proto with plugin types
|
||||
├─ Compiles all protobuf to Rust code
|
||||
└─ Links plugin into binary
|
||||
|
||||
4. Runtime
|
||||
├─ Load plugin configuration from config file
|
||||
├─ Initialize plugin handler
|
||||
├─ Subscribe to event bus
|
||||
└─ React to entity events
|
||||
```
|
||||
|
||||
### Plugin Configuration System
|
||||
|
||||
**File**: `config.toml` (TOML format with `${VAR}` env var substitution)
|
||||
|
||||
```toml
|
||||
# Server configuration
|
||||
[server]
|
||||
grpc_host = "[::1]"
|
||||
grpc_port = 50051
|
||||
|
||||
[server.yaml_adapter]
|
||||
enabled = true
|
||||
host = "0.0.0.0"
|
||||
port = 8080
|
||||
|
||||
# Database
|
||||
[database]
|
||||
url = "${DATABASE_URL}"
|
||||
max_connections = 10
|
||||
|
||||
# Plugin configurations
|
||||
[plugins.defectdojo]
|
||||
enabled = true
|
||||
base_url = "${DEFECTDOJO_API_URL}"
|
||||
api_token = "${DEFECTDOJO_API_TOKEN}"
|
||||
|
||||
[plugins.defectdojo.default_engagement]
|
||||
auto_create = true
|
||||
name = "CI/CD Pipeline"
|
||||
engagement_type = "CI/CD"
|
||||
|
||||
[plugins.keycloak]
|
||||
enabled = false
|
||||
# base_url = "https://keycloak.company.com"
|
||||
# realm = "master"
|
||||
|
||||
[plugins.dependencytrack]
|
||||
enabled = false
|
||||
# base_url = "https://dependencytrack.company.com"
|
||||
# api_key = "${DEPENDENCYTRACK_API_KEY}"
|
||||
```
|
||||
|
||||
See `config.toml.example` for the full reference with all options documented.
|
||||
|
||||
**Configuration loading** (`src/config.rs`):
|
||||
|
||||
```rust
|
||||
pub struct Config {
|
||||
pub server: ServerConfig,
|
||||
pub database: DatabaseConfig,
|
||||
pub security: SecurityConfig,
|
||||
pub telemetry: TelemetryConfig,
|
||||
pub plugins: PluginsConfig,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
pub fn from_file(path: &str) -> Result<Self> {
|
||||
let content = std::fs::read_to_string(path)?;
|
||||
// Substitute ${VAR} with environment variable values
|
||||
let content = substitute_env_vars(&content);
|
||||
let config: Config = toml::from_str(&content)?;
|
||||
Ok(config)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Each plugin implements**:
|
||||
|
||||
```rust
|
||||
pub trait Plugin {
|
||||
fn name(&self) -> &str;
|
||||
fn plugin_type(&self) -> PluginType;
|
||||
fn load_config(&mut self, config: PluginConfig) -> Result<()>;
|
||||
fn validate_config(&self) -> Result<()>;
|
||||
}
|
||||
```
|
||||
|
||||
### Event Handler Behavior Configuration
|
||||
|
||||
Plugins can configure different behaviors based on:
|
||||
- Entity kind
|
||||
- Event type (created/updated/deleted)
|
||||
- Custom conditions
|
||||
|
||||
**Example**: DefectDojo plugin config (`config.toml`)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo]
|
||||
enabled = true
|
||||
base_url = "${DEFECTDOJO_API_URL}"
|
||||
api_token = "${DEFECTDOJO_API_TOKEN}"
|
||||
|
||||
[plugins.defectdojo.default_engagement]
|
||||
auto_create = true
|
||||
name = "CI/CD Pipeline"
|
||||
engagement_type = "CI/CD"
|
||||
```
|
||||
|
||||
**Handler Implementation**:
|
||||
|
||||
```rust
|
||||
impl EventHandler for DefectDojoHandler {
|
||||
async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
|
||||
// Check if plugin enabled
|
||||
if !self.config.enabled {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Match event type to configured behavior
|
||||
let behavior = match event.event_type {
|
||||
EntityEventType::Created => &self.config.on_entity_created,
|
||||
EntityEventType::Updated => &self.config.on_entity_updated,
|
||||
EntityEventType::Deleted => &self.config.on_entity_deleted,
|
||||
};
|
||||
|
||||
// Find matching rule for this entity kind
|
||||
for rule in behavior {
|
||||
if rule.matches(&event) {
|
||||
return self.execute_action(&rule.action, event).await;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future: Plugin API Extensions
|
||||
|
||||
**Future capability**: Plugins can expose custom gRPC endpoints.
|
||||
|
||||
### Example: DefectDojo Query Plugin
|
||||
|
||||
**Plugin extends API with custom service**:
|
||||
|
||||
```protobuf
|
||||
// plugins/defectdojo/proto/defectdojo_service.proto
|
||||
|
||||
service DefectDojoService {
|
||||
// Get findings for a specific entity
|
||||
rpc GetEntityFindings(GetEntityFindingsRequest) returns (GetEntityFindingsResponse);
|
||||
|
||||
// Get all active vulnerabilities
|
||||
rpc GetActiveVulnerabilities(GetActiveVulnerabilitiesRequest) returns (GetActiveVulnerabilitiesResponse);
|
||||
|
||||
// Sync entity with DefectDojo
|
||||
rpc SyncEntity(SyncEntityRequest) returns (SyncEntityResponse);
|
||||
}
|
||||
|
||||
message GetEntityFindingsRequest {
|
||||
string entity_id = 1;
|
||||
string severity_filter = 2; // "Critical", "High", etc.
|
||||
bool active_only = 3;
|
||||
}
|
||||
|
||||
message GetEntityFindingsResponse {
|
||||
repeated Finding findings = 1;
|
||||
int32 total_count = 2;
|
||||
}
|
||||
|
||||
message Finding {
|
||||
int32 id = 1;
|
||||
string title = 2;
|
||||
string severity = 3;
|
||||
string description = 4;
|
||||
google.protobuf.Timestamp date = 5;
|
||||
bool active = 6;
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```rust
|
||||
pub struct DefectDojoServiceImpl {
|
||||
client: DefectDojoClient,
|
||||
repository: EntityRepository,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl DefectDojoService for DefectDojoServiceImpl {
|
||||
async fn get_entity_findings(
|
||||
&self,
|
||||
request: Request<GetEntityFindingsRequest>,
|
||||
) -> Result<Response<GetEntityFindingsResponse>, Status> {
|
||||
let req = request.into_inner();
|
||||
|
||||
// Get entity to retrieve DefectDojo product ID
|
||||
let entity = self.repository.get_by_id(&req.entity_id).await?;
|
||||
let product_id = entity
|
||||
.annotations
|
||||
.get("defectdojo.com/product-id")
|
||||
.ok_or_else(|| Status::not_found("No DefectDojo product linked"))?;
|
||||
|
||||
// Fetch findings from DefectDojo API
|
||||
let findings = self.client
|
||||
.get_findings(product_id, &req.severity_filter, req.active_only)
|
||||
.await?;
|
||||
|
||||
Ok(Response::new(GetEntityFindingsResponse {
|
||||
findings: findings.into_iter().map(|f| f.into()).collect(),
|
||||
total_count: findings.len() as i32,
|
||||
}))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Registration in main.rs**:
|
||||
|
||||
```rust
|
||||
// Register plugin's custom gRPC service
|
||||
if defectdojo_plugin.is_enabled() {
|
||||
let defectdojo_service = DefectDojoServiceServer::new(
|
||||
DefectDojoServiceImpl::new(defectdojo_client, repository.clone())
|
||||
);
|
||||
|
||||
server = server.add_service(defectdojo_service);
|
||||
}
|
||||
```
|
||||
|
||||
**Client usage**:
|
||||
|
||||
```bash
|
||||
grpcurl -d '{
|
||||
"entity_id": "service-payment",
|
||||
"severity_filter": "Critical",
|
||||
"active_only": true
|
||||
}' localhost:50051 charybdis.plugins.defectdojo.DefectDojoService/GetEntityFindings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Technology Decisions
|
||||
|
||||
### Why gRPC Only?
|
||||
|
||||
**Decision**: Charybdis core provides only gRPC. Teams add REST/GraphQL as needed.
|
||||
|
||||
**Rationale**:
|
||||
- **Strong typing**: Protobuf schemas ensure correctness across languages
|
||||
- **Performance**: Binary protocol, efficient for CI/CD use cases
|
||||
- **Multi-language**: Official clients for Python, Go, Node.js, Java, Rust, C++, C#, etc.
|
||||
- **Streaming**: Built-in bidirectional streaming (future: real-time updates)
|
||||
- **Code generation**: Automatic client generation eliminates boilerplate
|
||||
- **Focus**: Core team maintains one high-quality API surface
|
||||
|
||||
**Teams can add REST if needed**:
|
||||
```
|
||||
grpc-gateway or Envoy → Charybdis gRPC
|
||||
```
|
||||
|
||||
### Why PostgreSQL + JSONB?
|
||||
|
||||
**Decision**: Use PostgreSQL with protobuf BYTEA storage and JSONB annotations.
|
||||
|
||||
**Rationale**:
|
||||
- **Scale**: Handles 1000s of entities easily (tested: 170k resources)
|
||||
- **JSONB performance**: GIN indexes enable fast annotation queries
|
||||
- **Familiarity**: Most teams already run PostgreSQL
|
||||
- **Reliability**: ACID transactions, mature ecosystem
|
||||
- **Future-proof**: Can add read replicas, partitioning if needed
|
||||
|
||||
**Alternative considered**: MongoDB
|
||||
- **Verdict**: PostgreSQL with JSONB provides same flexibility with better consistency guarantees
|
||||
|
||||
### Why Compile-Time Plugins?
|
||||
|
||||
**Decision**: Plugins are compile-time integrated, not runtime loaded.
|
||||
|
||||
**Rationale**:
|
||||
- **Type safety**: Rust compiler ensures correctness
|
||||
- **Performance**: No dynamic loading overhead
|
||||
- **Simplicity**: No plugin version compatibility matrix
|
||||
- **Security**: No arbitrary code execution
|
||||
- **Trade-off**: Requires rebuild to add plugins (acceptable for infrastructure tool)
|
||||
|
||||
**Community plugins**: Published as crates, teams include in their build.
|
||||
|
||||
### Why Event-Driven?
|
||||
|
||||
**Decision**: Plugins react to events after entity persistence.
|
||||
|
||||
**Rationale**:
|
||||
- **Decoupling**: Plugins can't break core CRUD operations
|
||||
- **Resilience**: Plugin failures are logged, not propagated
|
||||
- **Async**: Long-running operations don't block API
|
||||
- **Extensibility**: Add plugins without modifying core
|
||||
- **Observability**: Events provide audit trail
|
||||
|
||||
**Trade-off**: Plugins can't prevent entity creation (validation must be in core or client).
|
||||
|
||||
---
|
||||
|
||||
## Scale Estimates
|
||||
|
||||
### Reference Deployment
|
||||
|
||||
**Company**: ~100 engineers
|
||||
**Backstage Entities**:
|
||||
- Components: ~1,000
|
||||
- APIs: ~1,000
|
||||
- Users: ~700
|
||||
- Groups: ~200
|
||||
- Systems: ~50
|
||||
- Domains: ~150
|
||||
- Resources: ~170,000
|
||||
|
||||
**Charybdis Requirements**:
|
||||
- PostgreSQL: ~500MB storage (with JSONB annotations)
|
||||
- Memory: ~256MB for Charybdis service
|
||||
- CPU: Minimal (mostly I/O bound)
|
||||
|
||||
**10x Scale** (1,000 engineers, 1.7M resources):
|
||||
- PostgreSQL: ~5GB storage
|
||||
- Memory: ~512MB
|
||||
- CPU: Still minimal with proper indexing
|
||||
|
||||
**Bottlenecks**:
|
||||
- JSONB queries on annotations (solved with GIN indexes)
|
||||
- Event bus throughput (solved with Redis/RabbitMQ backend)
|
||||
- Plugin external API rate limits (solved with plugin-level queuing)
|
||||
|
||||
---
|
||||
|
||||
## Development Roadmap
|
||||
|
||||
> See [VISION.md](VISION.md) for the full product vision and strategic roadmap.
|
||||
|
||||
### Phase 0: Catalog Foundation (Done)
|
||||
|
||||
- ✅ Zero-migration database architecture
|
||||
- ✅ gRPC API with protobuf (full entity CRUD)
|
||||
- ✅ Event bus system (MemoryEventBus)
|
||||
- ✅ Event dispatcher architecture (generic + plugin-specific)
|
||||
- ✅ All Backstage entity kinds (Component, System, API, User, Group, Domain, Resource)
|
||||
- ✅ YAML adapter endpoints (Backstage compatibility)
|
||||
- ✅ gRPC field masks for partial updates
|
||||
- ✅ OpenTelemetry observability (traces, metrics, logs)
|
||||
- ✅ mTLS + RBAC security framework
|
||||
- ✅ Config loading from TOML with env var substitution
|
||||
- ✅ Plugin trait architecture (EventDriven + Sync)
|
||||
- ✅ Generic plugin utilities (HttpClient, AnnotationHelper, FieldMapper, DateUtils)
|
||||
- ✅ DefectDojo plugin (products, engagements, owner resolution)
|
||||
- ✅ Keycloak plugin (user/group sync with annotations)
|
||||
|
||||
### Phase 1: Security Core (Current Focus)
|
||||
|
||||
**Goal**: Native vulnerability management and scan ingestion.
|
||||
|
||||
- [ ] Vulnerability/Observation entity kind (protobuf + storage)
|
||||
- [ ] ScanService gRPC endpoint for scan ingestion
|
||||
- [ ] SARIF parser (covers majority of modern scanners)
|
||||
- [ ] CycloneDX parser (SBOMs + vulnerability data)
|
||||
- [ ] SPDX parser (license data)
|
||||
- [ ] Assessment workflow (triage, accept risk, remediate)
|
||||
- [ ] Rules engine for auto-assessment
|
||||
- [ ] Security gates (severity thresholds per product)
|
||||
- [ ] License tracking and policy engine
|
||||
|
||||
### Phase 2: Compliance & Integrations
|
||||
|
||||
**Goal**: Compliance frameworks and integration plugins.
|
||||
|
||||
- [ ] Compliance framework mappings (NIS2, SOC2, DORA)
|
||||
- [ ] VEX document support (CSAF, OpenVEX)
|
||||
- [ ] Export/reporting (PDF, Excel)
|
||||
- [ ] Notification plugins (Slack, Teams, email)
|
||||
- [ ] Issue tracker plugins (Jira, GitHub, GitLab)
|
||||
- [ ] Redis event bus backend
|
||||
|
||||
### Phase 3: Scaffolder & Ecosystem
|
||||
|
||||
**Goal**: Service scaffolding and community growth.
|
||||
|
||||
- [ ] Service scaffolder (Git-native templates)
|
||||
- [ ] Event-driven provisioning on scaffold
|
||||
- [ ] Plugin SDK documentation
|
||||
- [ ] Helm chart & 1-click deploy
|
||||
- [ ] Community plugin registry
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Phase 1 Success (Security Core)
|
||||
|
||||
- [ ] Ingest SARIF scan results via gRPC
|
||||
- [ ] Vulnerabilities linked to catalog entities
|
||||
- [ ] Rules engine auto-assesses based on severity/component
|
||||
- [ ] Security gates block on threshold violations
|
||||
- [ ] Assessment workflow (triage → accept/remediate)
|
||||
- [ ] License data ingested from CycloneDX/SPDX
|
||||
|
||||
### Production Success
|
||||
|
||||
- [ ] 1,000+ entities managed with security posture
|
||||
- [ ] <100ms p99 latency for CRUD operations
|
||||
- [ ] Single `helm install` deployment
|
||||
- [ ] Documentation complete
|
||||
- [ ] 3+ integration plugins in production
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions Summary
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| **API Protocol** | gRPC only | Strong typing, multi-language, performance |
|
||||
| **Frontend** | Backstage YAML adapter | Compatible with existing Backstage deployments |
|
||||
| **Security features** | Native (core) | First-class, not plugins. Vulns, compliance, gates in the core |
|
||||
| **External integrations** | Plugins | Slack, Jira, etc. as event-driven plugins |
|
||||
| **Database** | PostgreSQL + JSONB | Scale, familiarity, ACID |
|
||||
| **Plugin Loading** | Compile-time | Type safety, performance, security |
|
||||
| **Event Model** | Post-persistence | Resilience, decoupling |
|
||||
| **Schema Evolution** | Protobuf only | Zero migrations |
|
||||
| **Configuration** | TOML + env vars | Flexibility, Rust-native |
|
||||
| **Deployment** | Single binary + Docker + K8s | Cloud-native, minimal footprint |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Backstage Descriptor Format](https://backstage.io/docs/features/software-catalog/descriptor-format)
|
||||
- [gRPC Documentation](https://grpc.io/docs/)
|
||||
- [Protocol Buffers Guide](https://protobuf.dev/)
|
||||
- [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html)
|
||||
|
||||
---
|
||||
|
||||
## Document Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0 | 2025-01-03 | Initial comprehensive architecture document |
|
||||
| 1.1 | 2025-01-15 | Added dual plugin architecture, generic utilities, updated roadmap |
|
||||
| 2.0 | 2026-04-02 | Pivoted to security-native platform engineering tool. Extracted product vision to VISION.md. Added native vulnerability management, compliance frameworks to architecture. Updated roadmap and success criteria. |
|
||||
|
||||
---
|
||||
|
||||
**This document covers architecture and technical decisions.**
|
||||
For product vision and strategic direction, see [VISION.md](VISION.md).
|
||||
All code and documentation must align with these documents.
|
||||
Reference in New Issue
Block a user