Public Access
initial-commit
This commit is contained in:
Guillaume GRABÉ
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,985 @@
|
||||
# Charybdis - Architecture
|
||||
|
||||
**Version**: 2.0
|
||||
**Last Updated**: 2026-05-06
|
||||
**Status**: Living Document
|
||||
|
||||
> For the product vision and roadmap, see [VISION.md](../VISION.md).
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Charybdis** is a security-native platform engineering tool that unifies software catalog, vulnerability management, and compliance posture in a single event-driven platform. Built in Rust, deployed as a single binary.
|
||||
|
||||
### Core Value Proposition
|
||||
|
||||
**Problem**:
|
||||
- Platform engineers run 5+ disconnected tools (Backstage, DefectDojo, Dependency-Track, license scanners, compliance spreadsheets)
|
||||
- Software catalogs rely on static YAML files that drift from reality
|
||||
- Security posture is invisible at the catalog level — vulns live in separate tools with no link to services
|
||||
- Compliance evidence is assembled manually from fragmented data sources
|
||||
|
||||
**Solution**:
|
||||
- **Dynamic software catalog** — event-driven, gRPC-native, no static YAML
|
||||
- **Native vulnerability management** — ingest scan results (SARIF, CycloneDX, SPDX), triage, assess, track
|
||||
- **Security gates & compliance** — severity thresholds, license policies, compliance framework mappings
|
||||
- **Plugin system for integrations** — Slack, Jira, GitHub, custom tools react to events
|
||||
- **Single binary deployment** — Rust + PostgreSQL, ~50MB RAM, sub-millisecond latency
|
||||
|
||||
### Use Case Examples
|
||||
|
||||
**Service registration with auto-provisioning:**
|
||||
```
|
||||
CI/CD Pipeline (Python) → gRPC CreateEntity(kind=Component)
|
||||
↓
|
||||
Charybdis persists entity
|
||||
↓
|
||||
Event: EntityCreated published
|
||||
↓
|
||||
┌────────────────────┴────────────────────┐
|
||||
↓ ↓
|
||||
Jira Plugin Slack Plugin
|
||||
- Creates onboarding epic - Notifies #platform channel
|
||||
↓ ↓
|
||||
└────────────────────┬────────────────────┘
|
||||
↓
|
||||
Entity visible via gRPC API / Backstage YAML adapter
|
||||
```
|
||||
|
||||
**Vulnerability ingestion with security gates:**
|
||||
```
|
||||
Scanner (Trivy) → CI/CD → gRPC IngestScan(sarif_report)
|
||||
↓
|
||||
Charybdis parses SARIF
|
||||
Creates Vulnerability entities linked to Component
|
||||
↓
|
||||
Rules engine evaluates auto-assessment
|
||||
Security gate checks thresholds
|
||||
↓
|
||||
┌────────────────────┴────────────────────┐
|
||||
↓ ↓
|
||||
Gate PASSED Gate FAILED
|
||||
- Vulns tracked - Slack alert to #security
|
||||
- Dashboard updated - Jira ticket created
|
||||
- CI/CD pipeline blocked
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architectural Principles
|
||||
|
||||
### 1. **No Database Migrations Ever**
|
||||
|
||||
The database schema is created once on first startup and **never changes**.
|
||||
|
||||
**How it works**:
|
||||
- Entities stored as protobuf bytes in `entity_data BYTEA` column
|
||||
- Plugins add new entity types via protobuf `oneof` variants
|
||||
- Build system regenerates code, database schema remains static
|
||||
|
||||
**Benefits**:
|
||||
- Deploy new plugins without downtime
|
||||
- No migration scripts to manage
|
||||
- Forward/backward compatibility built-in
|
||||
- Easy rollback (protobuf versioning)
|
||||
|
||||
### 2. **gRPC First, Everything Else is Adapter**
|
||||
|
||||
Charybdis provides **only gRPC APIs**. All other interfaces (REST, YAML, GraphQL) are adapters on top.
|
||||
|
||||
**Rationale**:
|
||||
- **Strong typing**: Protobuf ensures type safety across all languages
|
||||
- **Performance**: Binary protocol, efficient serialization
|
||||
- **Multi-language**: Official gRPC clients for 10+ languages
|
||||
- **Streaming**: Built-in support for real-time updates (future)
|
||||
- **Code generation**: Automatic client/server code generation
|
||||
|
||||
**What Charybdis provides**:
|
||||
- ✅ gRPC API with protobuf definitions
|
||||
- ✅ YAML adapter endpoints (for Backstage compatibility/migration)
|
||||
- ❌ REST API (teams can add if needed via gateway)
|
||||
- ❌ GraphQL (teams can add if needed)
|
||||
|
||||
### 3. **Plugin-Based Extensibility**
|
||||
|
||||
Plugins are **compile-time integrated** Rust crates that:
|
||||
|
||||
1. **Define entity types** via protobuf schemas
|
||||
2. **React to events** via EventHandler trait
|
||||
3. **Call external APIs** to synchronize state
|
||||
4. **Store references** in entity annotations
|
||||
5. **(Future) Extend gRPC API** with custom endpoints
|
||||
|
||||
**Plugin characteristics**:
|
||||
- Not hot-swappable (require rebuild)
|
||||
- Type-safe at compile time
|
||||
- No runtime dependency resolution
|
||||
- Configurable via YAML config file
|
||||
- Can be developed by community
|
||||
|
||||
### 4. **Event-Driven Orchestration**
|
||||
|
||||
Events are published **after** entities are persisted:
|
||||
|
||||
```
|
||||
CRUD Operation → Persist to DB → Publish Event → Plugins React
|
||||
```
|
||||
|
||||
**Event flow**:
|
||||
1. Entity created/updated/deleted in database
|
||||
2. Event published to event bus with full entity data
|
||||
3. All subscribed plugin handlers receive event
|
||||
4. Plugins filter events they care about
|
||||
5. Plugins perform asynchronous actions
|
||||
6. Plugins update entity annotations (via repository)
|
||||
|
||||
**Error handling**:
|
||||
- Plugin errors **do not** fail the CRUD operation
|
||||
- Entity is already persisted before event publishing
|
||||
- Plugins log errors for observability
|
||||
- Retries handled by event bus backend
|
||||
|
||||
### 5. **Backstage Compatibility by Design**
|
||||
|
||||
Charybdis entity model is **100% compatible** with Backstage's descriptor format.
|
||||
|
||||
**How it works**:
|
||||
- Core protobuf types mirror Backstage YAML structure
|
||||
- YAML adapter endpoints serve entities in Backstage format
|
||||
- Backstage reads Charybdis as a dynamic location provider
|
||||
- No changes needed to Backstage frontend
|
||||
|
||||
**Key compatibility points**:
|
||||
- Entity `kind`, `apiVersion`, `metadata`, `spec` structure
|
||||
- Annotation format (reverse-DNS style)
|
||||
- Relationship types (`dependsOn`, `partOf`, etc.)
|
||||
- All standard Backstage entity kinds supported
|
||||
|
||||
---
|
||||
|
||||
## System Architecture
|
||||
|
||||
### Component Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Client Layer │
|
||||
│ CI/CD Pipelines (Python, Go, Node.js, Bash, etc.) │
|
||||
│ Security Scanners (SARIF, CycloneDX, SPDX output) │
|
||||
│ IaC Tools (Terraform, Pulumi, Crossplane) │
|
||||
│ Backstage (via YAML adapter, optional) │
|
||||
└────────────────┬────────────────────────────────────────────────┘
|
||||
│
|
||||
│ gRPC / YAML HTTP
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Charybdis Core │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ gRPC API │ │
|
||||
│ │ EntityService: CRUD + List (with field masks) │ │
|
||||
│ │ ScanService: IngestScan (SARIF, CycloneDX, SPDX) │ │
|
||||
│ │ AssessmentService: Triage, Accept, Remediate │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Security Engine (native) │ │
|
||||
│ │ - Vulnerability tracking per entity │ │
|
||||
│ │ - Rules engine (auto-assessment) │ │
|
||||
│ │ - Security gates (severity thresholds) │ │
|
||||
│ │ - License compliance │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Adapters │ │
|
||||
│ │ - YAML Adapter (Backstage compat / migration) │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Repository Layer │ │
|
||||
│ │ - PostgreSQL with protobuf storage │ │
|
||||
│ │ - JSONB annotations for fast queries │ │
|
||||
│ │ - Zero-migration schema │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────▼─────────────────────────────────────┐ │
|
||||
│ │ Event Bus System │ │
|
||||
│ │ - Memory backend (dev) │ │
|
||||
│ │ - Redis backend (prod, future) │ │
|
||||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ Entity / Vulnerability / Gate events
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Plugin Layer (integrations) │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
|
||||
│ │ Slack/Teams │ │ Jira/GitHub │ │ Custom Plugins │ │
|
||||
│ │ Notifications │ │ Issue Track │ │ Your integrations │ │
|
||||
│ └──────┬────────┘ └──────┬───────┘ └──────────┬─────────────┘ │
|
||||
└─────────┼──────────────────┼─────────────────────┼──────────────┘
|
||||
▼ ▼ ▼
|
||||
Slack/Teams API Jira/GitHub API External APIs
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
#### Entity Creation Flow
|
||||
|
||||
```
|
||||
1. Client (CI/CD)
|
||||
↓ gRPC CreateEntityRequest
|
||||
2. Charybdis gRPC API
|
||||
↓ Validate entity
|
||||
↓ Generate UUID, timestamps
|
||||
3. Entity Repository
|
||||
↓ Serialize to protobuf bytes
|
||||
↓ INSERT INTO entities
|
||||
4. PostgreSQL
|
||||
↓ Entity persisted
|
||||
5. Event Bus
|
||||
↓ Publish EntityCreated event (with full entity data)
|
||||
6. Plugin Handlers (parallel)
|
||||
├─ DefectDojo: Create product → Store product_id in annotations
|
||||
├─ DependencyTrack: Create project → Store project_uuid in annotations
|
||||
└─ Custom: Custom actions
|
||||
7. Repository.update_annotations()
|
||||
↓ Merge annotations from plugins
|
||||
↓ UPDATE entities SET annotations = ...
|
||||
8. PostgreSQL
|
||||
↓ Entity updated with external tool IDs
|
||||
9. Return CreateEntityResponse to client
|
||||
```
|
||||
|
||||
#### Backstage Integration Flow
|
||||
|
||||
```
|
||||
1. Backstage Catalog Backend
|
||||
↓ HTTP GET /yaml/locations
|
||||
2. Charybdis YAML Adapter
|
||||
↓ Query repository.list_all()
|
||||
↓ Convert entities to Location YAML
|
||||
3. Backstage receives location list
|
||||
↓ Fetches each entity: GET /yaml/entities/:id
|
||||
4. Charybdis YAML Adapter
|
||||
↓ repository.get_by_id()
|
||||
↓ Convert entity protobuf → Backstage YAML
|
||||
5. Backstage ingests entities
|
||||
↓ Displays in UI with all annotations
|
||||
↓ Shows linked DefectDojo products, etc.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Entity Model
|
||||
|
||||
### Protobuf Schema
|
||||
|
||||
```protobuf
|
||||
message Entity {
|
||||
// System-managed fields
|
||||
string id = 1; // UUID (auto-generated)
|
||||
google.protobuf.Timestamp created_at = 21;
|
||||
google.protobuf.Timestamp updated_at = 22;
|
||||
|
||||
// Backstage-compatible fields
|
||||
string api_version = 2; // "backstage.io/v1alpha1"
|
||||
string kind = 3; // "Component", "API", "User", etc.
|
||||
|
||||
// Polymorphic metadata (identifying information)
|
||||
oneof metadata {
|
||||
// Core types (built-in)
|
||||
ComponentMetadata component_metadata = 10;
|
||||
APIMetadata api_metadata = 11;
|
||||
UserMetadata user_metadata = 12;
|
||||
GroupMetadata group_metadata = 13;
|
||||
SystemMetadata system_metadata = 14;
|
||||
DomainMetadata domain_metadata = 15;
|
||||
ResourceMetadata resource_metadata = 16;
|
||||
LocationMetadata location_metadata = 17;
|
||||
TemplateMetadata template_metadata = 18;
|
||||
|
||||
// Plugin types (dynamically added)
|
||||
// Example: DefectDojoProductMetadata defectdojo_product_metadata = 100;
|
||||
}
|
||||
|
||||
// Polymorphic spec (configuration and behavior)
|
||||
oneof spec {
|
||||
// Core types (built-in)
|
||||
ComponentSpec component_spec = 10;
|
||||
APISpec api_spec = 11;
|
||||
UserSpec user_spec = 12;
|
||||
GroupSpec group_spec = 13;
|
||||
SystemSpec system_spec = 14;
|
||||
DomainSpec domain_spec = 15;
|
||||
ResourceSpec resource_spec = 16;
|
||||
LocationSpec location_spec = 17;
|
||||
TemplateSpec template_spec = 18;
|
||||
|
||||
// Plugin types (dynamically added)
|
||||
// Example: DefectDojoProductSpec defectdojo_product_spec = 100;
|
||||
}
|
||||
|
||||
// Annotations - external tool references
|
||||
// Format: "tool.com/resource-type-attribute"
|
||||
// Examples:
|
||||
// "defectdojo.com/product-id": "42"
|
||||
// "dependencytrack.com/project-uuid": "550e8400-..."
|
||||
// "github.com/repo-slug": "myorg/myrepo"
|
||||
map<string, string> annotations = 20;
|
||||
}
|
||||
```
|
||||
|
||||
### Backstage Entity Kinds Mapping
|
||||
|
||||
All Backstage entity kinds must be supported:
|
||||
|
||||
| Backstage Kind | Charybdis Proto Type | Status |
|
||||
|----------------|---------------------|--------|
|
||||
| Component | ComponentMetadata/Spec | ✅ Done |
|
||||
| Service | ServiceMetadata/Spec | ✅ Done |
|
||||
| API | APIMetadata/Spec | ✅ Done |
|
||||
| User | UserMetadata/Spec | ✅ Done |
|
||||
| Group | GroupMetadata/Spec | ✅ Done |
|
||||
| System | SystemMetadata/Spec | ✅ Done |
|
||||
| Domain | DomainMetadata/Spec | ✅ Done |
|
||||
| Resource | ResourceMetadata/Spec | ✅ Done |
|
||||
| Location | LocationMetadata/Spec | Not planned (entities are event-driven, not file-based) |
|
||||
| Template | TemplateMetadata/Spec | Phase 3 (Scaffolder) |
|
||||
|
||||
---
|
||||
|
||||
## YAML Adapter Layer
|
||||
|
||||
### Purpose
|
||||
|
||||
Serve entities in Backstage-compatible YAML format over HTTP.
|
||||
|
||||
### API Specification
|
||||
|
||||
#### 1. List Locations
|
||||
|
||||
**Endpoint**: `GET /yaml/locations`
|
||||
|
||||
**Purpose**: Return a dynamic Location entity that lists all entities in Charybdis.
|
||||
|
||||
**Response Format** (Backstage Location YAML):
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Location
|
||||
metadata:
|
||||
name: charybdis-all-entities
|
||||
description: Dynamic location managed by Charybdis
|
||||
spec:
|
||||
type: charybdis
|
||||
targets:
|
||||
- http://charybdis.company.com/yaml/entities/service-payment
|
||||
- http://charybdis.company.com/yaml/entities/service-auth
|
||||
- http://charybdis.company.com/yaml/entities/api-payments-v1
|
||||
- http://charybdis.company.com/yaml/entities/user-john-doe
|
||||
# ... all entities
|
||||
```
|
||||
|
||||
**Implementation Notes**:
|
||||
- Query `SELECT id, kind FROM entities`
|
||||
- Generate URL for each entity: `/yaml/entities/{kind}-{id}`
|
||||
- Return as Backstage Location YAML
|
||||
- Cache response (invalidate on entity CRUD)
|
||||
|
||||
#### 2. Get Entity by ID
|
||||
|
||||
**Endpoint**: `GET /yaml/entities/:id`
|
||||
|
||||
**Purpose**: Return a specific entity in Backstage YAML format.
|
||||
|
||||
**Response Format** (Backstage Component YAML):
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: payment-service
|
||||
namespace: production
|
||||
description: Core payment processing service
|
||||
annotations:
|
||||
github.com/repo-slug: myorg/payment-service
|
||||
defectdojo.com/product-id: "42"
|
||||
dependencytrack.com/project-uuid: "550e8400-e29b-41d4-a716-446655440000"
|
||||
tags:
|
||||
- payments
|
||||
- critical
|
||||
links:
|
||||
- url: https://dashboard.company.com/payment-service
|
||||
title: Dashboard
|
||||
icon: dashboard
|
||||
spec:
|
||||
type: service
|
||||
lifecycle: production
|
||||
owner: payments-team
|
||||
system: payment-system
|
||||
dependsOn:
|
||||
- component:auth-service
|
||||
providesApis:
|
||||
- payments-api-v1
|
||||
```
|
||||
|
||||
**Implementation Notes**:
|
||||
- Query `repository.get_by_id(id)`
|
||||
- Convert Entity protobuf → Backstage YAML
|
||||
- Map `oneof metadata/spec` to appropriate YAML structure
|
||||
- Include all annotations
|
||||
- Return as `text/yaml` content type
|
||||
|
||||
#### 3. Configuration in Backstage
|
||||
|
||||
**Backstage app-config.yaml**:
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://charybdis.company.com/yaml/locations
|
||||
rules:
|
||||
- allow: [Component, API, User, Group, System, Domain, Resource]
|
||||
```
|
||||
|
||||
**How it works**:
|
||||
1. Backstage fetches `/yaml/locations` on catalog refresh
|
||||
2. Parses Location YAML to get entity URLs
|
||||
3. Fetches each entity URL (`/yaml/entities/:id`)
|
||||
4. Ingests entities into Backstage catalog
|
||||
5. Displays in UI with all metadata and annotations
|
||||
|
||||
---
|
||||
|
||||
## Plugin System
|
||||
|
||||
### Plugin Architecture
|
||||
|
||||
Charybdis supports **two types of plugins**, both compile-time integrated:
|
||||
|
||||
#### **1. Event-Driven Plugins**
|
||||
React to entity lifecycle events (create, update, delete):
|
||||
- **Example**: DefectDojo, DependencyTrack
|
||||
- **Implement**: `EventDrivenPlugin` trait
|
||||
- **Provide**: `ResourceHandler` implementations
|
||||
- **Triggered by**: Entity CRUD operations
|
||||
|
||||
#### **2. Sync Plugins**
|
||||
Pull data from external sources on a schedule:
|
||||
- **Example**: Okta, Keycloak, Active Directory
|
||||
- **Implement**: `SyncPlugin` trait
|
||||
- **Scheduled by**: Cron expressions
|
||||
- **Triggered by**: Time-based schedule or manual API call
|
||||
|
||||
### Generic Plugin Utilities (2025-01-15)
|
||||
|
||||
All plugins have access to reusable utilities in `src/plugins/`:
|
||||
|
||||
#### **PluginHttpClient**
|
||||
Generic HTTP client with multiple auth methods:
|
||||
- Token authentication (DefectDojo)
|
||||
- Bearer authentication (modern APIs)
|
||||
- API Key authentication (DependencyTrack)
|
||||
- Basic authentication (JIRA, Bitbucket)
|
||||
- Methods: `get()`, `post()`, `put()`, `patch()`, `delete()`
|
||||
|
||||
#### **AnnotationHelper**
|
||||
Consistent API for storing/retrieving plugin metadata:
|
||||
- `set_id()` / `get_id()` - Store external resource IDs
|
||||
- `set()` / `get()` - Store arbitrary metadata
|
||||
- Enforces naming: `{plugin}.com/{resource}-id`
|
||||
- Prevents annotation conflicts between plugins
|
||||
|
||||
#### **FieldMapper**
|
||||
Maps entity fields to external tool formats:
|
||||
- Direct field paths: `"metadata.name"`
|
||||
- Static values: `{"value": "Web Application"}`
|
||||
- Complex mappings with entity resolution
|
||||
- Repository integration for cross-entity queries
|
||||
|
||||
#### **DateUtils**
|
||||
Common date/time operations:
|
||||
- `today()`, `today_plus_days(n)` - Date formatting
|
||||
- `engagement_date_range(days)` - For time-bound resources
|
||||
- `iso_timestamp()` - ISO 8601 timestamps
|
||||
|
||||
### Plugin Responsibilities
|
||||
|
||||
Plugins can:
|
||||
|
||||
1. **Define custom entity types** (via protobuf)
|
||||
2. **React to entity events** (via ResourceHandler trait)
|
||||
3. **Call external APIs** (using PluginHttpClient)
|
||||
4. **Store metadata** (using AnnotationHelper)
|
||||
5. **Map fields** (using FieldMapper)
|
||||
6. **Schedule syncs** (SyncPlugins with cron)
|
||||
7. **(Future) Extend gRPC API** (custom services)
|
||||
|
||||
### Plugin Lifecycle
|
||||
|
||||
```
|
||||
1. Development
|
||||
├─ Define protobuf schema (metadata/spec messages)
|
||||
├─ Implement EventHandler trait
|
||||
├─ Implement external API client
|
||||
└─ Add configuration schema
|
||||
|
||||
2. Registration
|
||||
├─ Add to plugins.toml (enabled = true)
|
||||
└─ Add to Cargo.toml dependencies
|
||||
|
||||
3. Build Time
|
||||
├─ build.rs reads plugins.toml
|
||||
├─ Generates entities.proto with plugin types
|
||||
├─ Compiles all protobuf to Rust code
|
||||
└─ Links plugin into binary
|
||||
|
||||
4. Runtime
|
||||
├─ Load plugin configuration from config file
|
||||
├─ Initialize plugin handler
|
||||
├─ Subscribe to event bus
|
||||
└─ React to entity events
|
||||
```
|
||||
|
||||
### Plugin Configuration System
|
||||
|
||||
**File**: `config.toml` (TOML format with `${VAR}` env var substitution)
|
||||
|
||||
```toml
|
||||
# Server configuration
|
||||
[server]
|
||||
grpc_host = "[::1]"
|
||||
grpc_port = 50051
|
||||
|
||||
[server.yaml_adapter]
|
||||
enabled = true
|
||||
host = "0.0.0.0"
|
||||
port = 8080
|
||||
|
||||
# Database
|
||||
[database]
|
||||
url = "${DATABASE_URL}"
|
||||
max_connections = 10
|
||||
|
||||
# Plugin configurations
|
||||
[plugins.defectdojo]
|
||||
enabled = true
|
||||
base_url = "${DEFECTDOJO_API_URL}"
|
||||
api_token = "${DEFECTDOJO_API_TOKEN}"
|
||||
|
||||
[plugins.defectdojo.default_engagement]
|
||||
auto_create = true
|
||||
name = "CI/CD Pipeline"
|
||||
engagement_type = "CI/CD"
|
||||
|
||||
[plugins.keycloak]
|
||||
enabled = false
|
||||
# base_url = "https://keycloak.company.com"
|
||||
# realm = "master"
|
||||
|
||||
[plugins.dependencytrack]
|
||||
enabled = false
|
||||
# base_url = "https://dependencytrack.company.com"
|
||||
# api_key = "${DEPENDENCYTRACK_API_KEY}"
|
||||
```
|
||||
|
||||
See `config.toml.example` for the full reference with all options documented.
|
||||
|
||||
**Configuration loading** (`src/config.rs`):
|
||||
|
||||
```rust
|
||||
pub struct Config {
|
||||
pub server: ServerConfig,
|
||||
pub database: DatabaseConfig,
|
||||
pub security: SecurityConfig,
|
||||
pub telemetry: TelemetryConfig,
|
||||
pub plugins: PluginsConfig,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
pub fn from_file(path: &str) -> Result<Self> {
|
||||
let content = std::fs::read_to_string(path)?;
|
||||
// Substitute ${VAR} with environment variable values
|
||||
let content = substitute_env_vars(&content);
|
||||
let config: Config = toml::from_str(&content)?;
|
||||
Ok(config)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Each plugin implements**:
|
||||
|
||||
```rust
|
||||
pub trait Plugin {
|
||||
fn name(&self) -> &str;
|
||||
fn plugin_type(&self) -> PluginType;
|
||||
fn load_config(&mut self, config: PluginConfig) -> Result<()>;
|
||||
fn validate_config(&self) -> Result<()>;
|
||||
}
|
||||
```
|
||||
|
||||
### Event Handler Behavior Configuration
|
||||
|
||||
Plugins can configure different behaviors based on:
|
||||
- Entity kind
|
||||
- Event type (created/updated/deleted)
|
||||
- Custom conditions
|
||||
|
||||
**Example**: DefectDojo plugin config (`config.toml`)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo]
|
||||
enabled = true
|
||||
base_url = "${DEFECTDOJO_API_URL}"
|
||||
api_token = "${DEFECTDOJO_API_TOKEN}"
|
||||
|
||||
[plugins.defectdojo.default_engagement]
|
||||
auto_create = true
|
||||
name = "CI/CD Pipeline"
|
||||
engagement_type = "CI/CD"
|
||||
```
|
||||
|
||||
**Handler Implementation**:
|
||||
|
||||
```rust
|
||||
impl EventHandler for DefectDojoHandler {
|
||||
async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
|
||||
// Check if plugin enabled
|
||||
if !self.config.enabled {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Match event type to configured behavior
|
||||
let behavior = match event.event_type {
|
||||
EntityEventType::Created => &self.config.on_entity_created,
|
||||
EntityEventType::Updated => &self.config.on_entity_updated,
|
||||
EntityEventType::Deleted => &self.config.on_entity_deleted,
|
||||
};
|
||||
|
||||
// Find matching rule for this entity kind
|
||||
for rule in behavior {
|
||||
if rule.matches(&event) {
|
||||
return self.execute_action(&rule.action, event).await;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future: Plugin API Extensions
|
||||
|
||||
**Future capability**: Plugins can expose custom gRPC endpoints.
|
||||
|
||||
### Example: DefectDojo Query Plugin
|
||||
|
||||
**Plugin extends API with custom service**:
|
||||
|
||||
```protobuf
|
||||
// plugins/defectdojo/proto/defectdojo_service.proto
|
||||
|
||||
service DefectDojoService {
|
||||
// Get findings for a specific entity
|
||||
rpc GetEntityFindings(GetEntityFindingsRequest) returns (GetEntityFindingsResponse);
|
||||
|
||||
// Get all active vulnerabilities
|
||||
rpc GetActiveVulnerabilities(GetActiveVulnerabilitiesRequest) returns (GetActiveVulnerabilitiesResponse);
|
||||
|
||||
// Sync entity with DefectDojo
|
||||
rpc SyncEntity(SyncEntityRequest) returns (SyncEntityResponse);
|
||||
}
|
||||
|
||||
message GetEntityFindingsRequest {
|
||||
string entity_id = 1;
|
||||
string severity_filter = 2; // "Critical", "High", etc.
|
||||
bool active_only = 3;
|
||||
}
|
||||
|
||||
message GetEntityFindingsResponse {
|
||||
repeated Finding findings = 1;
|
||||
int32 total_count = 2;
|
||||
}
|
||||
|
||||
message Finding {
|
||||
int32 id = 1;
|
||||
string title = 2;
|
||||
string severity = 3;
|
||||
string description = 4;
|
||||
google.protobuf.Timestamp date = 5;
|
||||
bool active = 6;
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```rust
|
||||
pub struct DefectDojoServiceImpl {
|
||||
client: DefectDojoClient,
|
||||
repository: EntityRepository,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl DefectDojoService for DefectDojoServiceImpl {
|
||||
async fn get_entity_findings(
|
||||
&self,
|
||||
request: Request<GetEntityFindingsRequest>,
|
||||
) -> Result<Response<GetEntityFindingsResponse>, Status> {
|
||||
let req = request.into_inner();
|
||||
|
||||
// Get entity to retrieve DefectDojo product ID
|
||||
let entity = self.repository.get_by_id(&req.entity_id).await?;
|
||||
let product_id = entity
|
||||
.annotations
|
||||
.get("defectdojo.com/product-id")
|
||||
.ok_or_else(|| Status::not_found("No DefectDojo product linked"))?;
|
||||
|
||||
// Fetch findings from DefectDojo API
|
||||
let findings = self.client
|
||||
.get_findings(product_id, &req.severity_filter, req.active_only)
|
||||
.await?;
|
||||
|
||||
Ok(Response::new(GetEntityFindingsResponse {
|
||||
findings: findings.into_iter().map(|f| f.into()).collect(),
|
||||
total_count: findings.len() as i32,
|
||||
}))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Registration in main.rs**:
|
||||
|
||||
```rust
|
||||
// Register plugin's custom gRPC service
|
||||
if defectdojo_plugin.is_enabled() {
|
||||
let defectdojo_service = DefectDojoServiceServer::new(
|
||||
DefectDojoServiceImpl::new(defectdojo_client, repository.clone())
|
||||
);
|
||||
|
||||
server = server.add_service(defectdojo_service);
|
||||
}
|
||||
```
|
||||
|
||||
**Client usage**:
|
||||
|
||||
```bash
|
||||
grpcurl -d '{
|
||||
"entity_id": "service-payment",
|
||||
"severity_filter": "Critical",
|
||||
"active_only": true
|
||||
}' localhost:50051 charybdis.plugins.defectdojo.DefectDojoService/GetEntityFindings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Technology Decisions
|
||||
|
||||
### Why gRPC Only?
|
||||
|
||||
**Decision**: Charybdis core provides only gRPC. Teams add REST/GraphQL as needed.
|
||||
|
||||
**Rationale**:
|
||||
- **Strong typing**: Protobuf schemas ensure correctness across languages
|
||||
- **Performance**: Binary protocol, efficient for CI/CD use cases
|
||||
- **Multi-language**: Official clients for Python, Go, Node.js, Java, Rust, C++, C#, etc.
|
||||
- **Streaming**: Built-in bidirectional streaming (future: real-time updates)
|
||||
- **Code generation**: Automatic client generation eliminates boilerplate
|
||||
- **Focus**: Core team maintains one high-quality API surface
|
||||
|
||||
**Teams can add REST if needed**:
|
||||
```
|
||||
grpc-gateway or Envoy → Charybdis gRPC
|
||||
```
|
||||
|
||||
### Why PostgreSQL + JSONB?
|
||||
|
||||
**Decision**: Use PostgreSQL with protobuf BYTEA storage and JSONB annotations.
|
||||
|
||||
**Rationale**:
|
||||
- **Scale**: Handles 1000s of entities easily (tested: 170k resources)
|
||||
- **JSONB performance**: GIN indexes enable fast annotation queries
|
||||
- **Familiarity**: Most teams already run PostgreSQL
|
||||
- **Reliability**: ACID transactions, mature ecosystem
|
||||
- **Future-proof**: Can add read replicas, partitioning if needed
|
||||
|
||||
**Alternative considered**: MongoDB
|
||||
- **Verdict**: PostgreSQL with JSONB provides same flexibility with better consistency guarantees
|
||||
|
||||
### Why Compile-Time Plugins?
|
||||
|
||||
**Decision**: Plugins are compile-time integrated, not runtime loaded.
|
||||
|
||||
**Rationale**:
|
||||
- **Type safety**: Rust compiler ensures correctness
|
||||
- **Performance**: No dynamic loading overhead
|
||||
- **Simplicity**: No plugin version compatibility matrix
|
||||
- **Security**: No arbitrary code execution
|
||||
- **Trade-off**: Requires rebuild to add plugins (acceptable for infrastructure tool)
|
||||
|
||||
**Community plugins**: Published as crates, teams include in their build.
|
||||
|
||||
### Why Event-Driven?
|
||||
|
||||
**Decision**: Plugins react to events after entity persistence.
|
||||
|
||||
**Rationale**:
|
||||
- **Decoupling**: Plugins can't break core CRUD operations
|
||||
- **Resilience**: Plugin failures are logged, not propagated
|
||||
- **Async**: Long-running operations don't block API
|
||||
- **Extensibility**: Add plugins without modifying core
|
||||
- **Observability**: Events provide audit trail
|
||||
|
||||
**Trade-off**: Plugins can't prevent entity creation (validation must be in core or client).
|
||||
|
||||
---
|
||||
|
||||
## Scale Estimates
|
||||
|
||||
### Reference Deployment
|
||||
|
||||
**Company**: ~100 engineers
|
||||
**Backstage Entities**:
|
||||
- Components: ~1,000
|
||||
- APIs: ~1,000
|
||||
- Users: ~700
|
||||
- Groups: ~200
|
||||
- Systems: ~50
|
||||
- Domains: ~150
|
||||
- Resources: ~170,000
|
||||
|
||||
**Charybdis Requirements**:
|
||||
- PostgreSQL: ~500MB storage (with JSONB annotations)
|
||||
- Memory: ~256MB for Charybdis service
|
||||
- CPU: Minimal (mostly I/O bound)
|
||||
|
||||
**10x Scale** (1,000 engineers, 1.7M resources):
|
||||
- PostgreSQL: ~5GB storage
|
||||
- Memory: ~512MB
|
||||
- CPU: Still minimal with proper indexing
|
||||
|
||||
**Bottlenecks**:
|
||||
- JSONB queries on annotations (solved with GIN indexes)
|
||||
- Event bus throughput (solved with Redis/RabbitMQ backend)
|
||||
- Plugin external API rate limits (solved with plugin-level queuing)
|
||||
|
||||
---
|
||||
|
||||
## Development Roadmap
|
||||
|
||||
> See [VISION.md](VISION.md) for the full product vision and strategic roadmap.
|
||||
|
||||
### Phase 0: Catalog Foundation (Done)
|
||||
|
||||
- ✅ Zero-migration database architecture
|
||||
- ✅ gRPC API with protobuf (full entity CRUD)
|
||||
- ✅ Event bus system (MemoryEventBus)
|
||||
- ✅ Event dispatcher architecture (generic + plugin-specific)
|
||||
- ✅ All Backstage entity kinds (Component, System, API, User, Group, Domain, Resource)
|
||||
- ✅ YAML adapter endpoints (Backstage compatibility)
|
||||
- ✅ gRPC field masks for partial updates
|
||||
- ✅ OpenTelemetry observability (traces, metrics, logs)
|
||||
- ✅ mTLS + RBAC security framework
|
||||
- ✅ Config loading from TOML with env var substitution
|
||||
- ✅ Plugin trait architecture (EventDriven + Sync)
|
||||
- ✅ Generic plugin utilities (HttpClient, AnnotationHelper, FieldMapper, DateUtils)
|
||||
- ✅ DefectDojo plugin (products, engagements, owner resolution)
|
||||
- ✅ Keycloak plugin (user/group sync with annotations)
|
||||
|
||||
### Phase 1: Security Core (Current Focus)
|
||||
|
||||
**Goal**: Native vulnerability management and scan ingestion.
|
||||
|
||||
- [ ] Vulnerability/Observation entity kind (protobuf + storage)
|
||||
- [ ] ScanService gRPC endpoint for scan ingestion
|
||||
- [ ] SARIF parser (covers majority of modern scanners)
|
||||
- [ ] CycloneDX parser (SBOMs + vulnerability data)
|
||||
- [ ] SPDX parser (license data)
|
||||
- [ ] Assessment workflow (triage, accept risk, remediate)
|
||||
- [ ] Rules engine for auto-assessment
|
||||
- [ ] Security gates (severity thresholds per product)
|
||||
- [ ] License tracking and policy engine
|
||||
|
||||
### Phase 2: Compliance & Integrations
|
||||
|
||||
**Goal**: Compliance frameworks and integration plugins.
|
||||
|
||||
- [ ] Compliance framework mappings (NIS2, SOC2, DORA)
|
||||
- [ ] VEX document support (CSAF, OpenVEX)
|
||||
- [ ] Export/reporting (PDF, Excel)
|
||||
- [ ] Notification plugins (Slack, Teams, email)
|
||||
- [ ] Issue tracker plugins (Jira, GitHub, GitLab)
|
||||
- [ ] Redis event bus backend
|
||||
|
||||
### Phase 3: Scaffolder & Ecosystem
|
||||
|
||||
**Goal**: Service scaffolding and community growth.
|
||||
|
||||
- [ ] Service scaffolder (Git-native templates)
|
||||
- [ ] Event-driven provisioning on scaffold
|
||||
- [ ] Plugin SDK documentation
|
||||
- [ ] Helm chart & 1-click deploy
|
||||
- [ ] Community plugin registry
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Phase 1 Success (Security Core)
|
||||
|
||||
- [ ] Ingest SARIF scan results via gRPC
|
||||
- [ ] Vulnerabilities linked to catalog entities
|
||||
- [ ] Rules engine auto-assesses based on severity/component
|
||||
- [ ] Security gates block on threshold violations
|
||||
- [ ] Assessment workflow (triage → accept/remediate)
|
||||
- [ ] License data ingested from CycloneDX/SPDX
|
||||
|
||||
### Production Success
|
||||
|
||||
- [ ] 1,000+ entities managed with security posture
|
||||
- [ ] <100ms p99 latency for CRUD operations
|
||||
- [ ] Single `helm install` deployment
|
||||
- [ ] Documentation complete
|
||||
- [ ] 3+ integration plugins in production
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions Summary
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| **API Protocol** | gRPC only | Strong typing, multi-language, performance |
|
||||
| **Frontend** | Backstage YAML adapter | Compatible with existing Backstage deployments |
|
||||
| **Security features** | Native (core) | First-class, not plugins. Vulns, compliance, gates in the core |
|
||||
| **External integrations** | Plugins | Slack, Jira, etc. as event-driven plugins |
|
||||
| **Database** | PostgreSQL + JSONB | Scale, familiarity, ACID |
|
||||
| **Plugin Loading** | Compile-time | Type safety, performance, security |
|
||||
| **Event Model** | Post-persistence | Resilience, decoupling |
|
||||
| **Schema Evolution** | Protobuf only | Zero migrations |
|
||||
| **Configuration** | TOML + env vars | Flexibility, Rust-native |
|
||||
| **Deployment** | Single binary + Docker + K8s | Cloud-native, minimal footprint |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Backstage Descriptor Format](https://backstage.io/docs/features/software-catalog/descriptor-format)
|
||||
- [gRPC Documentation](https://grpc.io/docs/)
|
||||
- [Protocol Buffers Guide](https://protobuf.dev/)
|
||||
- [PostgreSQL JSONB](https://www.postgresql.org/docs/current/datatype-json.html)
|
||||
|
||||
---
|
||||
|
||||
## Document Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0 | 2025-01-03 | Initial comprehensive architecture document |
|
||||
| 1.1 | 2025-01-15 | Added dual plugin architecture, generic utilities, updated roadmap |
|
||||
| 2.0 | 2026-04-02 | Pivoted to security-native platform engineering tool. Extracted product vision to VISION.md. Added native vulnerability management, compliance frameworks to architecture. Updated roadmap and success criteria. |
|
||||
|
||||
---
|
||||
|
||||
**This document covers architecture and technical decisions.**
|
||||
For product vision and strategic direction, see [VISION.md](VISION.md).
|
||||
All code and documentation must align with these documents.
|
||||
@@ -0,0 +1,628 @@
|
||||
# Core Concepts
|
||||
|
||||
Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.
|
||||
|
||||
This page explains the key concepts you need to understand.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Sources"
|
||||
A1[CI/CD Pipelines]
|
||||
A2[Security Scanners]
|
||||
A3[IaC Tools]
|
||||
end
|
||||
|
||||
subgraph "Charybdis Core"
|
||||
B1[gRPC API]
|
||||
B3[Authentication & Authorization]
|
||||
C1[Entity Service]
|
||||
C4[Vulnerability Engine]
|
||||
C5[Security Gates & Rules]
|
||||
C2[Event Bus]
|
||||
end
|
||||
|
||||
subgraph "Integrations - Plugins"
|
||||
D1[Slack / Teams]
|
||||
D2[Jira / GitHub]
|
||||
D3[Custom Plugins]
|
||||
end
|
||||
|
||||
subgraph "Data Layer"
|
||||
E1[(PostgreSQL)]
|
||||
end
|
||||
|
||||
A1 -.gRPC.-> B1
|
||||
A2 -.Scan Results.-> B1
|
||||
A3 -.gRPC.-> B1
|
||||
|
||||
B1 --> B3
|
||||
B3 --> C1
|
||||
B3 --> C4
|
||||
C4 --> C5
|
||||
C1 --> E1
|
||||
C4 --> E1
|
||||
C1 --> C2
|
||||
C4 --> C2
|
||||
|
||||
C2 -.Events.-> D1
|
||||
C2 -.Events.-> D2
|
||||
C2 -.Events.-> D3
|
||||
|
||||
style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
|
||||
style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
|
||||
style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
|
||||
```
|
||||
|
||||
## Entities
|
||||
|
||||
An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.
|
||||
|
||||
### Entity Structure
|
||||
|
||||
Every entity has three main parts:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class Entity {
|
||||
+string id
|
||||
+string kind
|
||||
+metadata
|
||||
+spec
|
||||
+annotations
|
||||
+timestamps
|
||||
}
|
||||
|
||||
class Metadata {
|
||||
+string name
|
||||
+string namespace
|
||||
+string description
|
||||
+labels
|
||||
+links
|
||||
+tags
|
||||
}
|
||||
|
||||
class Spec {
|
||||
+string type
|
||||
+string lifecycle
|
||||
+string owner
|
||||
+dependencies
|
||||
}
|
||||
|
||||
Entity --> Metadata
|
||||
Entity --> Spec
|
||||
```
|
||||
|
||||
### Entity Kinds
|
||||
|
||||
Charybdis supports all standard Backstage entity kinds:
|
||||
|
||||
| Kind | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| **Service** | Individual microservices or applications | `payment-api` |
|
||||
| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
|
||||
| **System** | Collections of services working together | `e-commerce-platform` |
|
||||
| **API** | Interfaces exposed by components | `payments-rest-api` |
|
||||
| **User** | Individual people | `john.doe` |
|
||||
| **Group** | Teams and organizational units | `team-payments` |
|
||||
| **Domain** | Business domains | `payments`, `shipping` |
|
||||
| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
|
||||
|
||||
#### Example: Service
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "payment-api",
|
||||
"namespace": "production",
|
||||
"description": "Payment processing service",
|
||||
"labels": { "team": "payments" }
|
||||
},
|
||||
"service_spec": {
|
||||
"type": "service",
|
||||
"lifecycle": "production",
|
||||
"owner": "team-payments",
|
||||
"system": "payment-system"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Example: Component
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Component",
|
||||
"component_metadata": {
|
||||
"name": "auth-sdk",
|
||||
"namespace": "shared"
|
||||
},
|
||||
"component_spec": {
|
||||
"type": "library",
|
||||
"lifecycle": "production",
|
||||
"owner": "platform-team"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Example: System
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "System",
|
||||
"system_metadata": {
|
||||
"name": "e-commerce-platform",
|
||||
"namespace": "production",
|
||||
"description": "Complete e-commerce system"
|
||||
},
|
||||
"system_spec": {
|
||||
"owner": "platform-team",
|
||||
"domain": "retail"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Metadata Fields
|
||||
|
||||
| Field | Type | Description | Required |
|
||||
|-------|------|-------------|----------|
|
||||
| `name` | string | Entity name (unique within namespace) | ✅ |
|
||||
| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
|
||||
| `description` | string | Human-readable description | ❌ |
|
||||
| `labels` | map | Key-value pairs for categorization | ❌ |
|
||||
| `tags` | array | Search tags | ❌ |
|
||||
| `links` | array | External URLs (dashboards, docs, etc.) | ❌ |
|
||||
|
||||
### Annotations
|
||||
|
||||
Annotations store integration-specific metadata:
|
||||
|
||||
```json
|
||||
{
|
||||
"annotations": {
|
||||
"github.com/repo-slug": "myorg/payment-service",
|
||||
"defectdojo.com/product-id": "123",
|
||||
"dependencytrack.com/project-uuid": "550e8400...",
|
||||
"pagerduty.com/service-id": "PXYZ123",
|
||||
"grafana.com/dashboard-url": "https://..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Best Practices**:
|
||||
- Use domain-style keys (`tool.com/key`)
|
||||
- Store tool-specific IDs
|
||||
- Keep values as strings
|
||||
- Use for integration metadata only
|
||||
|
||||
## Events
|
||||
|
||||
Charybdis uses an **event-driven architecture** to trigger actions when entities change.
|
||||
|
||||
### Event Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant API as Entity Service
|
||||
participant Bus as Event Bus
|
||||
participant Plugin1 as DefectDojo Plugin
|
||||
participant Plugin2 as Dependency-Track
|
||||
|
||||
Client->>API: CreateEntity(service)
|
||||
API->>API: Store entity
|
||||
API->>Bus: Emit EntityCreated event
|
||||
Bus->>Plugin1: Handle event
|
||||
Bus->>Plugin2: Handle event
|
||||
Plugin1-->>Plugin1: Create DefectDojo product
|
||||
Plugin2-->>Plugin2: Create DT project
|
||||
API-->>Client: Return created entity
|
||||
|
||||
Note over Bus,Plugin2: Asynchronous processing
|
||||
```
|
||||
|
||||
### Event Types
|
||||
|
||||
| Event | Trigger | Plugins Receive |
|
||||
|-------|---------|----------------|
|
||||
| `EntityCreated` | New entity created | Full entity data |
|
||||
| `EntityUpdated` | Entity modified | Updated entity + changes |
|
||||
| `EntityDeleted` | Entity removed | Entity ID + metadata |
|
||||
|
||||
### Event Structure
|
||||
|
||||
```rust
|
||||
pub enum EntityEvent {
|
||||
Created {
|
||||
entity: Entity,
|
||||
timestamp: DateTime<Utc>,
|
||||
},
|
||||
Updated {
|
||||
entity: Entity,
|
||||
previous: Entity,
|
||||
timestamp: DateTime<Utc>,
|
||||
},
|
||||
Deleted {
|
||||
id: String,
|
||||
metadata: Metadata,
|
||||
timestamp: DateTime<Utc>,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Vulnerabilities & Security (Phase 1 — Planned)
|
||||
|
||||
> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.
|
||||
|
||||
Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.
|
||||
|
||||
### Vulnerability Lifecycle
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Scanner
|
||||
participant API as Charybdis API
|
||||
participant Rules as Rules Engine
|
||||
participant Gate as Security Gate
|
||||
participant Bus as Event Bus
|
||||
participant Plugin as Slack / Jira
|
||||
|
||||
Scanner->>API: IngestScan(SARIF report)
|
||||
API->>API: Parse & create Vulnerability entities
|
||||
API->>API: Link vulnerabilities to Component
|
||||
API->>Rules: Evaluate auto-assessment rules
|
||||
Rules-->>API: Auto-assess (e.g., accept known low-risk)
|
||||
API->>Gate: Check security gate thresholds
|
||||
alt Gate Passed
|
||||
Gate-->>API: OK
|
||||
else Gate Failed
|
||||
Gate->>Bus: GateFailed event
|
||||
Bus->>Plugin: Alert #security channel
|
||||
end
|
||||
API->>Bus: VulnerabilitiesIngested event
|
||||
Bus->>Plugin: Notify / create tickets
|
||||
```
|
||||
|
||||
### Scan Ingestion
|
||||
|
||||
Charybdis ingests scan results natively. You don't need an external vulnerability management tool.
|
||||
|
||||
| Format | Coverage | Use Case |
|
||||
|--------|----------|----------|
|
||||
| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
|
||||
| **CycloneDX** | SBOMs + vulnerability data | SCA, license |
|
||||
| **SPDX** | License and package data | License compliance |
|
||||
|
||||
### Assessments
|
||||
|
||||
Each vulnerability can be assessed:
|
||||
|
||||
| Status | Meaning |
|
||||
|--------|---------|
|
||||
| **Open** | New, unreviewed vulnerability |
|
||||
| **In Triage** | Under review by security team |
|
||||
| **Accepted** | Risk accepted with justification |
|
||||
| **Remediated** | Fixed, pending verification |
|
||||
| **False Positive** | Not a real vulnerability |
|
||||
| **Auto-Assessed** | Automatically assessed by rules engine |
|
||||
|
||||
### Security Gates
|
||||
|
||||
Security gates define thresholds per product:
|
||||
|
||||
```
|
||||
payment-api:
|
||||
critical: 0 # No critical vulns allowed
|
||||
high: 5 # Up to 5 high
|
||||
medium: 20 # Up to 20 medium
|
||||
```
|
||||
|
||||
When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).
|
||||
|
||||
### Rules Engine
|
||||
|
||||
Rules auto-assess vulnerabilities based on patterns:
|
||||
|
||||
- Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
|
||||
- Scanner source (e.g., "all informational from ZAP → auto-accept")
|
||||
- Known patterns (e.g., "CVE-XXXX already accepted org-wide")
|
||||
|
||||
### License Compliance (Phase 2)
|
||||
|
||||
Track licenses across your dependency tree:
|
||||
|
||||
- Ingest license data from CycloneDX/SPDX
|
||||
- Define license policies (allowed, restricted, banned)
|
||||
- Flag violations per component
|
||||
- Compliance reporting
|
||||
|
||||
## Storage Model
|
||||
|
||||
Charybdis uses PostgreSQL with JSONB for schema-less storage.
|
||||
|
||||
### Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE entities (
|
||||
id UUID PRIMARY KEY,
|
||||
kind TEXT NOT NULL,
|
||||
entity_data BYTEA NOT NULL, -- Protobuf binary
|
||||
annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
|
||||
created_at TIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_entities_kind ON entities(kind);
|
||||
CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
|
||||
```
|
||||
|
||||
### Why Protobuf + JSONB?
|
||||
|
||||
**Advantages**:
|
||||
- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
|
||||
- ✅ Protobuf binary storage for compact, versioned entity data
|
||||
- ✅ JSONB annotations for fast querying with GIN indexes
|
||||
- ✅ Forward/backward compatibility built-in
|
||||
|
||||
**Example Query**:
|
||||
```sql
|
||||
-- Find entities with specific annotation
|
||||
SELECT * FROM entities
|
||||
WHERE annotations->>'defectdojo.com/product-id' = '42';
|
||||
```
|
||||
|
||||
## Security Model
|
||||
|
||||
Charybdis implements defense-in-depth security.
|
||||
|
||||
### Authentication: mTLS
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant TLS as TLS Layer
|
||||
participant Auth as Auth Interceptor
|
||||
participant Service as Entity Service
|
||||
|
||||
Client->>TLS: Connect with client certificate
|
||||
TLS->>TLS: Validate certificate
|
||||
TLS->>Auth: Extract certificate
|
||||
Auth->>Auth: Parse identity (CN, OU, O)
|
||||
Auth->>Auth: Map to role
|
||||
Client->>Auth: Request (with identity)
|
||||
Auth->>Auth: Check permissions
|
||||
alt Authorized
|
||||
Auth->>Service: Forward request
|
||||
Service-->>Client: Response
|
||||
else Denied
|
||||
Auth-->>Client: PermissionDenied error
|
||||
end
|
||||
```
|
||||
|
||||
### Authorization: RBAC
|
||||
|
||||
**Role Mapping**:
|
||||
|
||||
```
|
||||
Certificate (CN, OU, O) → RBAC Role → Permissions
|
||||
```
|
||||
|
||||
**Default Roles**:
|
||||
|
||||
| Role | Certificate OU | Permissions |
|
||||
|------|---------------|-------------|
|
||||
| `platform` | `platform-team` | Full access (CRUD + list) |
|
||||
| `automation` | `automation` | Create, read, update, list |
|
||||
| `plugin` | `plugins` | Read, list only |
|
||||
|
||||
### Permissions
|
||||
|
||||
| Permission | Operations | Required For |
|
||||
|------------|-----------|--------------|
|
||||
| `entity:create` | Create new entities | CreateEntity |
|
||||
| `entity:read` | Get entity by ID | GetEntity |
|
||||
| `entity:update` | Modify entities | UpdateEntity |
|
||||
| `entity:delete` | Remove entities | DeleteEntity |
|
||||
| `entity:list` | List all entities | ListEntities |
|
||||
|
||||
## Plugin System
|
||||
|
||||
Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.
|
||||
|
||||
### Plugin Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Entity/Vuln Event] --> B[Event Bus]
|
||||
B --> C{Plugin Manager}
|
||||
C --> D[Slack Plugin]
|
||||
C --> E[Jira Plugin]
|
||||
C --> F[Custom Plugin]
|
||||
|
||||
D --> G[Slack API]
|
||||
E --> H[Jira API]
|
||||
F --> I[Your Tool API]
|
||||
|
||||
style C fill:#4A90E2,stroke:#2E5C8A
|
||||
```
|
||||
|
||||
### What's Native vs. Plugin
|
||||
|
||||
| Native (core) | Status | Plugin (integration) | Status |
|
||||
|---|---|---|---|
|
||||
| Software catalog | Done | DefectDojo sync | Done |
|
||||
| Vulnerability management | Phase 1 | Keycloak user sync | Done |
|
||||
| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
|
||||
| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
|
||||
| License compliance | Phase 2 | Custom integrations | Framework ready |
|
||||
| TechDocs | Future | | |
|
||||
|
||||
### Plugin Lifecycle
|
||||
|
||||
1. **Configuration** - Load plugin settings from `plugins.toml`
|
||||
2. **Initialization** - Plugin registers event handlers
|
||||
3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
|
||||
4. **External Integration** - Plugin calls external tool APIs
|
||||
5. **Error Handling** - Failed plugins don't affect core service
|
||||
|
||||
### Plugin Configuration
|
||||
|
||||
Example `plugins.toml`:
|
||||
|
||||
```toml
|
||||
[plugins.slack]
|
||||
enabled = true
|
||||
webhook_url = "${SLACK_WEBHOOK_URL}"
|
||||
|
||||
[[plugins.slack.on_gate_failed]]
|
||||
channel = "#security-alerts"
|
||||
|
||||
[[plugins.slack.on_entity_created]]
|
||||
channel = "#platform"
|
||||
```
|
||||
|
||||
See [Plugin Guide](plugins.md) for details.
|
||||
|
||||
## Data Consistency
|
||||
|
||||
### Eventual Consistency
|
||||
|
||||
Charybdis uses **eventual consistency** for plugin integrations:
|
||||
|
||||
- Entity CRUD operations are **immediately consistent**
|
||||
- Plugin synchronization is **eventually consistent**
|
||||
- Events are processed **asynchronously**
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[CreateEntity] -->|Immediate| B[Entity Stored]
|
||||
B -->|Async| C[Event Emitted]
|
||||
C -->|Async| D[Plugin Processing]
|
||||
D -->|Eventual| E[External Tool Synced]
|
||||
|
||||
style B fill:#00C851
|
||||
style E fill:#ffbb33
|
||||
```
|
||||
|
||||
### Guarantees
|
||||
|
||||
| Operation | Consistency | Guarantee |
|
||||
|-----------|-------------|-----------|
|
||||
| Entity CRUD | Strong | Immediate |
|
||||
| Entity queries | Strong | Read-your-writes |
|
||||
| Event delivery | At-least-once | May retry |
|
||||
| Plugin sync | Eventual | Best-effort |
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Scalability
|
||||
|
||||
- **Entities**: Tested with 100,000+ entities
|
||||
- **Throughput**: 1,000+ requests/second
|
||||
- **Latency**: Sub-millisecond average
|
||||
- **Concurrency**: Tokio async runtime
|
||||
|
||||
### Resource Usage
|
||||
|
||||
Typical resource consumption:
|
||||
|
||||
| Component | CPU | Memory | Storage |
|
||||
|-----------|-----|--------|---------|
|
||||
| Charybdis | < 5% | ~50 MB | Minimal |
|
||||
| PostgreSQL | ~10% | ~256 MB | Depends on entity count |
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Index annotations** used for frequent queries
|
||||
2. **Use connection pooling** (built-in)
|
||||
3. **Enable query caching** in PostgreSQL
|
||||
4. **Monitor event bus** queue depth
|
||||
|
||||
## Backstage Migration
|
||||
|
||||
Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.
|
||||
|
||||
### How It Works
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Charybdis Entity] --> B[YAML Adapter]
|
||||
B --> C[Backstage YAML Format]
|
||||
C --> D[Backstage Catalog]
|
||||
|
||||
style B fill:#4A90E2,stroke:#2E5C8A
|
||||
```
|
||||
|
||||
Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:
|
||||
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://charybdis:8080/yaml/locations
|
||||
```
|
||||
|
||||
### Recommended Migration Path
|
||||
|
||||
1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
|
||||
2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
|
||||
3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
|
||||
4. **Optionally retire** Backstage when Charybdis covers your catalog needs
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Entity Design
|
||||
|
||||
✅ **DO**:
|
||||
- Use descriptive names
|
||||
- Group by namespace
|
||||
- Add relevant labels
|
||||
- Include documentation links
|
||||
- Set appropriate owners
|
||||
|
||||
❌ **DON'T**:
|
||||
- Store sensitive data in metadata
|
||||
- Use very long descriptions
|
||||
- Create deeply nested hierarchies
|
||||
- Duplicate data across entities
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
```
|
||||
<entity-type>-<purpose>-<environment>
|
||||
|
||||
Examples:
|
||||
- payment-api-prod
|
||||
- user-service-staging
|
||||
- auth-library
|
||||
- e-commerce-system
|
||||
```
|
||||
|
||||
### Metadata Organization
|
||||
|
||||
```json
|
||||
{
|
||||
"labels": {
|
||||
"team": "payments", // Ownership
|
||||
"tier": "critical", // Importance
|
||||
"environment": "production" // Deployment
|
||||
},
|
||||
"tags": ["pci-compliant", "public-api"],
|
||||
"links": [
|
||||
{ "url": "...", "title": "Dashboard" },
|
||||
{ "url": "...", "title": "Documentation" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
|
||||
- Configure [Security](security.md) for production (mTLS + RBAC)
|
||||
- Explore [Plugins](plugins.md) for external integrations
|
||||
- Review the [Architecture](architecture.md) for technical deep dive
|
||||
|
||||
---
|
||||
|
||||
**Questions?** [Open an issue](../../issues).
|
||||
@@ -0,0 +1,508 @@
|
||||
# Getting Started with Charybdis
|
||||
|
||||
This guide will get you from zero to a running Charybdis instance in minutes. By the end, you'll have a working software catalog that can register services via gRPC, ingest scan results, and auto-trigger integrations via event-driven plugins.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Rust** - 1.70 or later ([install](https://rustup.rs/))
|
||||
- **PostgreSQL** - 14 or later
|
||||
- **grpcurl** - For testing (optional, [install](https://github.com/fullstorydev/grpcurl))
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1: From Source
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/charybdis-catalog/charybdis.git
|
||||
cd charybdis
|
||||
|
||||
# Build
|
||||
cargo build --release
|
||||
|
||||
# The binary will be at target/release/charybdis
|
||||
```
|
||||
|
||||
### Option 2: Docker (Coming Soon)
|
||||
|
||||
```bash
|
||||
docker pull charybdis/charybdis:latest
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Start PostgreSQL
|
||||
|
||||
Using Docker:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name charybdis-postgres \
|
||||
-e POSTGRES_PASSWORD=mysecretpassword \
|
||||
-p 5432:5432 \
|
||||
postgres:15
|
||||
```
|
||||
|
||||
Or use an existing PostgreSQL instance.
|
||||
|
||||
### Step 2: Configure Charybdis
|
||||
|
||||
**Recommended: Use config.toml**
|
||||
|
||||
Copy the example configuration:
|
||||
|
||||
```bash
|
||||
cp config.toml.example config.toml
|
||||
```
|
||||
|
||||
Edit `config.toml` and set your database URL:
|
||||
|
||||
```toml
|
||||
[database]
|
||||
url = "${DATABASE_URL}"
|
||||
```
|
||||
|
||||
Set the environment variable:
|
||||
|
||||
```bash
|
||||
export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
|
||||
```
|
||||
|
||||
**Alternative: Environment Variables Only (Legacy)**
|
||||
|
||||
If you prefer environment variables:
|
||||
|
||||
```bash
|
||||
# Database
|
||||
export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
|
||||
|
||||
# Disable security for quick start
|
||||
export SECURITY_MTLS_ENABLED=false
|
||||
export SECURITY_RBAC_ENABLED=false
|
||||
|
||||
# Logging
|
||||
export RUST_LOG=info,charybdis=debug
|
||||
```
|
||||
|
||||
### Step 3: Run Charybdis
|
||||
|
||||
```bash
|
||||
cargo run
|
||||
```
|
||||
|
||||
You should see:
|
||||
|
||||
```
|
||||
INFO charybdis: Database ready
|
||||
INFO charybdis: Event bus started successfully
|
||||
INFO charybdis: EntityService server listening on [::1]:50051
|
||||
```
|
||||
|
||||
### Step 4: Verify It's Working
|
||||
|
||||
Test with grpcurl:
|
||||
|
||||
```bash
|
||||
# List available services
|
||||
grpcurl -plaintext localhost:50051 list
|
||||
|
||||
# Output:
|
||||
# charybdis.entities.EntityService
|
||||
# grpc.reflection.v1.ServerReflection
|
||||
```
|
||||
|
||||
Congratulations! Charybdis is running! 🎉
|
||||
|
||||
## Creating Your First Entity
|
||||
|
||||
### Using grpcurl
|
||||
|
||||
Create a service entity:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext \
|
||||
-d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "payment-service",
|
||||
"namespace": "production",
|
||||
"description": "Core payment processing service"
|
||||
}
|
||||
}
|
||||
}' \
|
||||
localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"entity": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"kind": "Service",
|
||||
"serviceMetadata": {
|
||||
"name": "payment-service",
|
||||
"namespace": "production",
|
||||
"description": "Core payment processing service"
|
||||
},
|
||||
"createdAt": "2025-11-04T10:00:00Z",
|
||||
"updatedAt": "2025-11-04T10:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### List All Entities
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
```
|
||||
|
||||
### Get Entity by ID
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext \
|
||||
-d '{"id": "550e8400-e29b-41d4-a716-446655440000"}' \
|
||||
localhost:50051 charybdis.entities.EntityService/GetEntity
|
||||
```
|
||||
|
||||
## Entity Types
|
||||
|
||||
Charybdis supports three main entity types:
|
||||
|
||||
### Service
|
||||
|
||||
Individual microservices or applications:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "user-api",
|
||||
"namespace": "production",
|
||||
"description": "User management API"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
### System
|
||||
|
||||
Collections of related services:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "System",
|
||||
"system_metadata": {
|
||||
"name": "payment-system",
|
||||
"namespace": "production",
|
||||
"description": "Complete payment processing system"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
### Component
|
||||
|
||||
Reusable components or libraries:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Component",
|
||||
"component_spec": {
|
||||
"type": "library",
|
||||
"lifecycle": "production",
|
||||
"owner": "platform-team"
|
||||
},
|
||||
"component_metadata": {
|
||||
"name": "auth-library",
|
||||
"namespace": "shared",
|
||||
"description": "Shared authentication library"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
## Adding Metadata
|
||||
|
||||
Entities support rich metadata:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "payment-service",
|
||||
"namespace": "production",
|
||||
"description": "Payment processing",
|
||||
"labels": {
|
||||
"team": "payments",
|
||||
"tier": "critical"
|
||||
},
|
||||
"links": [
|
||||
{
|
||||
"url": "https://dashboard.company.com/payments",
|
||||
"title": "Dashboard",
|
||||
"icon": "dashboard"
|
||||
}
|
||||
],
|
||||
"tags": ["payments", "pci-compliant", "critical"]
|
||||
},
|
||||
"annotations": {
|
||||
"github.com/repo-slug": "myorg/payment-service",
|
||||
"pagerduty.com/service-id": "PXYZ123"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
## Integrating with CI/CD
|
||||
|
||||
### Example: GitHub Actions
|
||||
|
||||
Create a workflow to register services automatically:
|
||||
|
||||
```yaml
|
||||
name: Register Service
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
register:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Register in Charybdis
|
||||
run: |
|
||||
grpcurl -plaintext \
|
||||
-d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "${{ github.event.repository.name }}",
|
||||
"namespace": "production",
|
||||
"description": "${{ github.event.repository.description }}"
|
||||
},
|
||||
"annotations": {
|
||||
"github.com/repo-slug": "${{ github.repository }}"
|
||||
}
|
||||
}
|
||||
}' \
|
||||
your-charybdis-host:50051 \
|
||||
charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
### Example: GitLab CI
|
||||
|
||||
```yaml
|
||||
register_service:
|
||||
stage: deploy
|
||||
script:
|
||||
- |
|
||||
grpcurl -plaintext \
|
||||
-d "{
|
||||
\"entity\": {
|
||||
\"kind\": \"Service\",
|
||||
\"service_metadata\": {
|
||||
\"name\": \"${CI_PROJECT_NAME}\",
|
||||
\"namespace\": \"${CI_ENVIRONMENT_NAME}\"
|
||||
}
|
||||
}
|
||||
}" \
|
||||
your-charybdis-host:50051 \
|
||||
charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
## Enabling Security
|
||||
|
||||
For production use, enable mTLS and RBAC:
|
||||
|
||||
### Step 1: Generate Certificates
|
||||
|
||||
```bash
|
||||
# Use the provided test script
|
||||
./test-mtls-rbac.sh
|
||||
```
|
||||
|
||||
This creates:
|
||||
- `certs/ca.pem` - Certificate Authority
|
||||
- `certs/server-cert.pem` / `server-key.pem` - Server certificate
|
||||
- `certs/admin-cert.pem` / `admin-key.pem` - Admin client certificate
|
||||
|
||||
### Step 2: Enable Security
|
||||
|
||||
```bash
|
||||
export SECURITY_MTLS_ENABLED=true
|
||||
export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem
|
||||
export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem
|
||||
export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem
|
||||
export SECURITY_RBAC_ENABLED=true
|
||||
```
|
||||
|
||||
### Step 3: Test with mTLS
|
||||
|
||||
```bash
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/admin-cert.pem \
|
||||
-key certs/admin-key.pem \
|
||||
-d '{}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
```
|
||||
|
||||
See the [Security Guide](security.md) for detailed configuration.
|
||||
|
||||
## Backstage Migration (Optional)
|
||||
|
||||
If you currently use Backstage and want to migrate gradually, the built-in YAML adapter serves entities in Backstage format:
|
||||
|
||||
```yaml
|
||||
# backstage app-config.yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://your-charybdis-host:8080/yaml/locations
|
||||
rules:
|
||||
- allow: [Component, System, Service]
|
||||
```
|
||||
|
||||
Backstage will automatically discover and import entities from Charybdis. You can run both in parallel — entities managed via gRPC are immediately visible in Backstage.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```
|
||||
Error: transport error
|
||||
```
|
||||
|
||||
**Solution**: Check if another process is using port 50051:
|
||||
|
||||
```bash
|
||||
lsof -ti:50051
|
||||
```
|
||||
|
||||
Kill the process or change the port:
|
||||
|
||||
```bash
|
||||
export GRPC_PORT=50052
|
||||
```
|
||||
|
||||
### Database Connection Failed
|
||||
|
||||
```
|
||||
Error: password authentication failed
|
||||
```
|
||||
|
||||
**Solution**: Verify your DATABASE_URL:
|
||||
|
||||
```bash
|
||||
# Test connection
|
||||
psql "$DATABASE_URL" -c "SELECT 1;"
|
||||
```
|
||||
|
||||
### Permission Denied (with security enabled)
|
||||
|
||||
```
|
||||
Code: PermissionDenied
|
||||
Message: Role 'X' does not have permission 'Y'
|
||||
```
|
||||
|
||||
**Solution**: Check your certificate and role mappings. See [Security Guide](security.md).
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you have Charybdis running:
|
||||
|
||||
1. Learn about [Core Concepts](core-concepts.md) — entities, vulnerabilities, events
|
||||
2. Read the [Vision & Roadmap](../VISION.md) — where Charybdis is going
|
||||
3. Configure [Security](security.md) for production (mTLS + RBAC)
|
||||
4. Explore [Plugins](../plugins/README.md) for external integrations
|
||||
|
||||
## Configuration
|
||||
|
||||
Charybdis supports two configuration methods:
|
||||
|
||||
### 1. Configuration File (Recommended)
|
||||
|
||||
Use `config.toml` for structured configuration:
|
||||
|
||||
```toml
|
||||
# config.toml
|
||||
[server]
|
||||
grpc_host = "[::1]"
|
||||
grpc_port = 50051
|
||||
|
||||
[database]
|
||||
url = "${DATABASE_URL}" # Environment variable substitution
|
||||
|
||||
[security.mtls]
|
||||
enabled = false
|
||||
|
||||
[security.rbac]
|
||||
enabled = false
|
||||
|
||||
[telemetry]
|
||||
service_name = "charybdis"
|
||||
environment = "development"
|
||||
enable_console = true
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Organized by section (server, database, security, telemetry, plugins)
|
||||
- ✅ Environment variable substitution with `${VAR_NAME}`
|
||||
- ✅ Comments and documentation inline
|
||||
- ✅ Easy to version control (excluding secrets)
|
||||
- ✅ No need to export dozens of environment variables
|
||||
|
||||
**Using Environment Variables in config.toml:**
|
||||
|
||||
```toml
|
||||
[database]
|
||||
url = "${DATABASE_URL}" # Will be substituted at runtime
|
||||
|
||||
[plugins.defectdojo]
|
||||
api_key = "${DEFECTDOJO_API_KEY}" # Secrets stay in environment
|
||||
```
|
||||
|
||||
Then set only the secrets:
|
||||
|
||||
```bash
|
||||
export DATABASE_URL="postgresql://..."
|
||||
export DEFECTDOJO_API_KEY="secret-key"
|
||||
```
|
||||
|
||||
### 2. Environment Variables (Legacy)
|
||||
|
||||
If `config.toml` is not found, Charybdis falls back to environment variables:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `DATABASE_URL` | (required) | PostgreSQL connection string |
|
||||
| `GRPC_HOST` | `[::1]` | gRPC server bind address |
|
||||
| `GRPC_PORT` | `50051` | gRPC server port |
|
||||
| `RUST_LOG` | `info` | Logging level |
|
||||
| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS authentication |
|
||||
| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC authorization |
|
||||
| `OTEL_ENABLE_CONSOLE` | `true` | Enable console logging |
|
||||
| `OTEL_SERVICE_NAME` | `charybdis` | Service name for telemetry |
|
||||
|
||||
### Configuration File Locations
|
||||
|
||||
Charybdis looks for configuration files in this order:
|
||||
|
||||
1. `./config.toml` (current directory)
|
||||
2. `./charybdis.toml`
|
||||
3. `/etc/charybdis/config.toml` (Linux/Unix)
|
||||
|
||||
If none are found, it uses environment variables.
|
||||
|
||||
See `config.toml.example` for a complete configuration template with all options documented.
|
||||
|
||||
---
|
||||
|
||||
**Need help?** Check the [troubleshooting guide](troubleshooting.md) or [open an issue](../../issues).
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
# Charybdis Documentation
|
||||
|
||||
**The security-native platform engineering tool.**
|
||||
|
||||
## What is Charybdis?
|
||||
|
||||
Charybdis is a platform engineering tool that unifies **software catalog**, **vulnerability management**, and **compliance posture** in a single event-driven platform. Built in Rust, deployed as a single binary.
|
||||
|
||||
Instead of running Backstage + DefectDojo + Dependency-Track + a license scanner + a compliance spreadsheet, you run Charybdis.
|
||||
|
||||
### The Problems It Solves
|
||||
|
||||
1. **Fragmented tooling** — Your software catalog, vulnerability data, license info, and compliance evidence live in 5 different tools that don't talk to each other. Charybdis unifies them.
|
||||
|
||||
2. **Static catalog data** — Traditional catalogs rely on YAML files that go stale within weeks. Charybdis is event-driven — CI/CD pipelines and IaC tools register and update entities via gRPC, so the catalog is always accurate.
|
||||
|
||||
3. **Security as an afterthought** — In Backstage, security is a plugin. In Charybdis, every entity carries its vulnerability posture, license status, and compliance state natively.
|
||||
|
||||
4. **Manual provisioning** — New service? Manually create entries in every tool. With Charybdis, one gRPC call catalogs the service and event-driven plugins handle the rest.
|
||||
|
||||
5. **Compliance evidence assembly** — Compliance reporting pulls from real vulnerability and license data, not spreadsheets.
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
CI/CD or Scanner ──gRPC──> Charybdis
|
||||
├── Catalogs the service (event-driven) ← Done
|
||||
├── Fires events to plugins (DefectDojo, ...) ← Done
|
||||
├── Ingests scan results (SARIF, CycloneDX) ← Phase 1
|
||||
├── Evaluates security gates & rules ← Phase 1
|
||||
└── Exposes catalog via YAML adapter (Backstage) ← Done
|
||||
```
|
||||
|
||||
**One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.**
|
||||
|
||||
## Key Concepts
|
||||
|
||||
| Concept | Description |
|
||||
|---------|-------------|
|
||||
| **Entity** | Anything in your software ecosystem: services, systems, components, APIs, users, groups, domains, resources |
|
||||
| **Vulnerability** | *(Phase 1)* A security finding linked to an entity, ingested from scanner output (SARIF, CycloneDX) |
|
||||
| **Assessment** | *(Phase 1)* The triage decision on a vulnerability: accept risk, remediate, auto-assessed by rules |
|
||||
| **Security Gate** | *(Phase 1)* Severity thresholds per product — blocks deployments when violated |
|
||||
| **Event Bus** | Publishes lifecycle events when entities or vulnerabilities change |
|
||||
| **Plugin** | Reacts to events to integrate with external systems (Slack, Jira, GitHub, custom) |
|
||||
| **Annotations** | Key-value metadata on entities for external references (e.g. `github.com/repo-slug`) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[CI/CD / Scanners] -->|gRPC| B[Charybdis]
|
||||
B -->|Native| C[Software Catalog]
|
||||
B -->|Native| D[Vuln Management]
|
||||
B -->|Native| E[Compliance]
|
||||
B -->|Events| F[Plugins: Slack / Jira / Custom]
|
||||
B -->|YAML| G[Backstage - optional]
|
||||
|
||||
style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
|
||||
style D fill:#E24A4A,stroke:#8A2E2E,color:#fff
|
||||
style E fill:#4AE28A,stroke:#2E8A5C,color:#fff
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
### Getting Started
|
||||
- [Installation & Quick Start](getting-started.md) — Get Charybdis running and register your first entity
|
||||
- [Demo Stack](../deploy/DEMO.md) — Full demo with DefectDojo
|
||||
|
||||
### Understanding Charybdis
|
||||
- [Core Concepts](core-concepts.md) — Entities, vulnerabilities, events, and data model
|
||||
- [Vision & Roadmap](../VISION.md) — Where Charybdis is going and why
|
||||
- [Architecture](architecture.md) — Technical design decisions
|
||||
|
||||
### Configuration
|
||||
- [Security](security.md) — mTLS authentication and RBAC authorization
|
||||
- [Plugin Configuration](PLUGIN_CONFIGURATION_GUIDE.md) — Setting up and configuring plugins
|
||||
|
||||
### Extending Charybdis
|
||||
- [Plugin Development](../plugins/README.md) — Build your own integration plugins
|
||||
|
||||
## Quick Example
|
||||
|
||||
Register a service from your CI/CD pipeline:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Component",
|
||||
"component_metadata": {
|
||||
"name": "payment-api",
|
||||
"description": "Payment processing service"
|
||||
},
|
||||
"component_spec": {
|
||||
"type": "service",
|
||||
"lifecycle": "production",
|
||||
"owner": "team-payments"
|
||||
}
|
||||
}
|
||||
}' charybdis:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
**What happens next:**
|
||||
- Entity stored in PostgreSQL with a UUID
|
||||
- `EntityCreated` event published to the event bus
|
||||
- Plugins react (e.g., DefectDojo creates a product automatically)
|
||||
- Entity available via gRPC and YAML adapter
|
||||
|
||||
No YAML file to write. No PR to open. No manual provisioning.
|
||||
|
||||
## Backstage Migration
|
||||
|
||||
Already using Backstage? Charybdis provides a YAML adapter for gradual migration. Point Backstage at Charybdis as a catalog source — entities registered via gRPC are immediately available in Backstage.
|
||||
|
||||
```yaml
|
||||
# backstage app-config.yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://charybdis:8080/yaml/locations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Ready to get started?** Head to the [Getting Started Guide](getting-started.md).
|
||||
@@ -0,0 +1,503 @@
|
||||
# Security
|
||||
|
||||
Charybdis provides enterprise-grade security with mTLS authentication and RBAC authorization.
|
||||
|
||||
## Overview
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
A[Client Request] -->|mTLS Handshake| B[TLS Layer]
|
||||
B -->|Certificate| C[Auth Interceptor]
|
||||
C -->|Extract Identity| D{Parse Certificate}
|
||||
D -->|CN, OU, O| E[Role Mapping]
|
||||
E -->|Role| F{Permission Check}
|
||||
F -->|Authorized| G[Entity Service]
|
||||
F -->|Denied| H[PermissionDenied Error]
|
||||
G -->|Audit Log| I[Security Events]
|
||||
|
||||
style C fill:#f9d71c,stroke:#f9a825
|
||||
style F fill:#4A90E2,stroke:#2E5C8A
|
||||
```
|
||||
|
||||
## mTLS Authentication
|
||||
|
||||
Mutual TLS (mTLS) provides certificate-based client authentication.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Client** presents X.509 certificate during TLS handshake
|
||||
2. **Server** validates certificate against trusted CA
|
||||
3. **Charybdis** extracts certificate identity (CN, OU, O)
|
||||
4. **RBAC Engine** maps identity to role
|
||||
5. **Permissions** are checked before allowing the request
|
||||
|
||||
### Certificate Structure
|
||||
|
||||
```
|
||||
Subject: CN=admin-user, OU=platform-team, O=Charybdis-Dev, C=US
|
||||
Issuer: CN=Charybdis Root CA, O=Charybdis-Dev, C=US
|
||||
Validity: Not Before: Nov 3 2025, Not After: Nov 3 2026
|
||||
```
|
||||
|
||||
**Fields Used for Authentication**:
|
||||
- `CN` (Common Name) - User or service identifier
|
||||
- `OU` (Organizational Unit) - Team or role identifier
|
||||
- `O` (Organization) - Organization name
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Enable mTLS
|
||||
export SECURITY_MTLS_ENABLED=true
|
||||
|
||||
# Server certificate and key
|
||||
export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem
|
||||
export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem
|
||||
|
||||
# Client CA for verification
|
||||
export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem
|
||||
|
||||
# Enable RBAC
|
||||
export SECURITY_RBAC_ENABLED=true
|
||||
```
|
||||
|
||||
### Generating Certificates
|
||||
|
||||
#### Development Certificates
|
||||
|
||||
Use the provided test script:
|
||||
|
||||
```bash
|
||||
./test-mtls-rbac.sh
|
||||
```
|
||||
|
||||
This generates:
|
||||
- `ca.pem` / `ca-key.pem` - Certificate Authority
|
||||
- `server-cert.pem` / `server-key.pem` - Server certificate
|
||||
- `admin-cert.pem` / `admin-key.pem` - Admin client (OU=platform-team)
|
||||
- Various plugin certificates (OU=plugins)
|
||||
|
||||
#### Production Certificates
|
||||
|
||||
For production, use a proper PKI:
|
||||
|
||||
**Options**:
|
||||
- **Internal PKI**: HashiCorp Vault, cert-manager, step-ca
|
||||
- **Public CA**: Let's Encrypt (for internet-facing)
|
||||
- **Enterprise CA**: Your organization's certificate authority
|
||||
|
||||
**Requirements**:
|
||||
- Valid for at least 90 days
|
||||
- Issued by trusted CA
|
||||
- Proper subject alternative names (SANs)
|
||||
- Appropriate key usage extensions
|
||||
|
||||
### Certificate Best Practices
|
||||
|
||||
✅ **DO**:
|
||||
- Use short-lived certificates (24-90 days)
|
||||
- Implement automatic rotation
|
||||
- Monitor expiration dates
|
||||
- Use strong key lengths (2048+ bit RSA or 256+ bit ECC)
|
||||
- Store private keys securely
|
||||
- Use separate CAs for dev/prod
|
||||
|
||||
❌ **DON'T**:
|
||||
- Use self-signed certs in production
|
||||
- Share private keys
|
||||
- Use the same certificate across environments
|
||||
- Ignore expiration warnings
|
||||
- Store keys in version control
|
||||
|
||||
## RBAC Authorization
|
||||
|
||||
Role-Based Access Control (RBAC) enforces fine-grained permissions.
|
||||
|
||||
### Role Mapping
|
||||
|
||||
Charybdis maps certificate attributes to roles:
|
||||
|
||||
```
|
||||
Certificate OU → RBAC Role → Permissions
|
||||
```
|
||||
|
||||
### Default Roles
|
||||
|
||||
| Role | Certificate OU | Description | Permissions |
|
||||
|------|---------------|-------------|-------------|
|
||||
| `platform` | `platform-team` | Platform administrators | Full access (CRUD + list) |
|
||||
| `automation` | `automation` | CI/CD pipelines | Create, read, update, list |
|
||||
| `plugin` | `plugins` | Integration plugins | Read, list only |
|
||||
|
||||
### Permissions
|
||||
|
||||
| Permission | Methods | Description |
|
||||
|------------|---------|-------------|
|
||||
| `entity:create` | CreateEntity | Create new entities |
|
||||
| `entity:read` | GetEntity | Read entity by ID |
|
||||
| `entity:update` | UpdateEntity, PartialUpdateEntity | Modify entities |
|
||||
| `entity:delete` | DeleteEntity | Delete entities |
|
||||
| `entity:list` | ListEntities | List all entities |
|
||||
|
||||
### Permission Matrix
|
||||
|
||||
| Role | Create | Read | Update | Delete | List |
|
||||
|------|--------|------|--------|--------|------|
|
||||
| `platform` | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| `automation` | ✅ | ✅ | ✅ | ❌ | ✅ |
|
||||
| `plugin` | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Admin Access (Full Permissions)
|
||||
|
||||
```bash
|
||||
# Create entity
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/admin-cert.pem \
|
||||
-key certs/admin-key.pem \
|
||||
-d '{"entity": {...}}' \
|
||||
localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
|
||||
# ✅ SUCCESS
|
||||
```
|
||||
|
||||
### Automation Access (Limited)
|
||||
|
||||
```bash
|
||||
# CI/CD can create
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/ci-pipeline-cert.pem \
|
||||
-key certs/ci-pipeline-key.pem \
|
||||
-d '{"entity": {...}}' \
|
||||
localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
|
||||
# ✅ SUCCESS
|
||||
|
||||
# But cannot delete
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/ci-pipeline-cert.pem \
|
||||
-key certs/ci-pipeline-key.pem \
|
||||
-d '{"id": "..."}' \
|
||||
localhost:50051 charybdis.entities.EntityService/DeleteEntity
|
||||
|
||||
# ❌ Code: PermissionDenied
|
||||
# Message: Role 'automation' does not have permission 'entity:delete'
|
||||
```
|
||||
|
||||
### Plugin Access (Read-Only)
|
||||
|
||||
```bash
|
||||
# Plugin can list
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/defectdojo-plugin-cert.pem \
|
||||
-key certs/defectdojo-plugin-key.pem \
|
||||
-d '{}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
|
||||
# ✅ SUCCESS
|
||||
|
||||
# But cannot create
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/defectdojo-plugin-cert.pem \
|
||||
-key certs/defectdojo-plugin-key.pem \
|
||||
-d '{"entity": {...}}' \
|
||||
localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
|
||||
# ❌ Code: PermissionDenied
|
||||
# Message: Role 'plugin' does not have permission 'entity:create'
|
||||
```
|
||||
|
||||
## Custom Role Configuration
|
||||
|
||||
### Defining Custom Roles
|
||||
|
||||
Edit `src/main.rs::setup_default_rbac_config()`:
|
||||
|
||||
```rust
|
||||
let role_mappings = vec![
|
||||
// Custom developer role
|
||||
RoleMapping {
|
||||
role: "developer".to_string(),
|
||||
rules: vec![RoleRule {
|
||||
subject: SubjectMatch {
|
||||
cn: None,
|
||||
ou: Some("engineering".to_string()),
|
||||
o: None,
|
||||
},
|
||||
}],
|
||||
},
|
||||
];
|
||||
|
||||
let mut permissions = HashMap::new();
|
||||
permissions.insert(
|
||||
"developer".to_string(),
|
||||
vec![
|
||||
"entity:create".to_string(),
|
||||
"entity:read".to_string(),
|
||||
"entity:list".to_string(),
|
||||
],
|
||||
);
|
||||
```
|
||||
|
||||
### Certificate-Based Mapping
|
||||
|
||||
Map specific certificates to roles:
|
||||
|
||||
```rust
|
||||
// Map by CN (specific user/service)
|
||||
RoleRule {
|
||||
subject: SubjectMatch {
|
||||
cn: Some("jenkins-ci".to_string()),
|
||||
ou: None,
|
||||
o: None,
|
||||
},
|
||||
}
|
||||
|
||||
// Map by Organization
|
||||
RoleRule {
|
||||
subject: SubjectMatch {
|
||||
cn: None,
|
||||
ou: None,
|
||||
o: Some("External-Partner".to_string()),
|
||||
},
|
||||
}
|
||||
|
||||
// Combined matching
|
||||
RoleRule {
|
||||
subject: SubjectMatch {
|
||||
cn: Some("admin".to_string()),
|
||||
ou: Some("platform-team".to_string()),
|
||||
o: Some("MyCompany".to_string()),
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Audit Logging
|
||||
|
||||
All security events are logged for compliance and forensics.
|
||||
|
||||
### Event Types
|
||||
|
||||
| Event | Logged Information |
|
||||
|-------|-------------------|
|
||||
| Authentication Success | Identity (CN, OU, O), timestamp |
|
||||
| Authentication Failure | Reason, timestamp |
|
||||
| Authorization Success | Identity, role, method, duration |
|
||||
| Authorization Denial | Identity, role, method, required permission |
|
||||
| Certificate Error | Error details, certificate info |
|
||||
|
||||
### Log Format
|
||||
|
||||
```
|
||||
2025-11-04T10:16:22Z INFO [AUDIT] Authorization Success
|
||||
identity: CN=admin-user, OU=platform-team
|
||||
role: platform
|
||||
method: /charybdis.entities.EntityService/CreateEntity
|
||||
permission: entity:create
|
||||
duration: 0.08ms
|
||||
|
||||
2025-11-04T10:16:25Z WARN [AUDIT] Authorization Denied
|
||||
identity: CN=defectdojo-plugin, OU=plugins
|
||||
role: plugin
|
||||
method: /charybdis.entities.EntityService/CreateEntity
|
||||
required_permission: entity:create
|
||||
reason: Role 'plugin' does not have permission 'entity:create'
|
||||
duration: 0.05ms
|
||||
```
|
||||
|
||||
### Audit Configuration
|
||||
|
||||
```bash
|
||||
# Enable audit logging (default: true)
|
||||
export SECURITY_RBAC_AUDIT_ENABLED=true
|
||||
|
||||
# Log all requests (default: true)
|
||||
export SECURITY_RBAC_AUDIT_LOG_ALL_REQUESTS=true
|
||||
|
||||
# Log denied requests (default: true)
|
||||
export SECURITY_RBAC_AUDIT_LOG_DENIED=true
|
||||
```
|
||||
|
||||
## Reverse Proxy Deployment
|
||||
|
||||
For environments where mTLS termination happens at a reverse proxy:
|
||||
|
||||
### Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Client] -->|mTLS| B[Reverse Proxy]
|
||||
B -->|HTTP + Cert Header| C[Charybdis]
|
||||
|
||||
style B fill:#4A90E2,stroke:#2E5C8A
|
||||
```
|
||||
|
||||
### Proxy Configuration
|
||||
|
||||
#### Envoy
|
||||
|
||||
```yaml
|
||||
- name: envoy.filters.http.lua
|
||||
typed_config:
|
||||
inline_code: |
|
||||
function envoy_on_request(request_handle)
|
||||
local cert = request_handle:connection():ssl():peerCertificatePresented()
|
||||
if cert then
|
||||
request_handle:headers():add("x-client-cert", cert)
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
#### nginx
|
||||
|
||||
```nginx
|
||||
location / {
|
||||
proxy_set_header X-Client-Cert $ssl_client_cert;
|
||||
proxy_pass http://charybdis:50051;
|
||||
}
|
||||
```
|
||||
|
||||
### Header Format
|
||||
|
||||
Charybdis accepts certificates in these headers:
|
||||
- `x-forwarded-client-cert` (Envoy/Istio standard)
|
||||
- `x-client-cert` (nginx)
|
||||
|
||||
Format: Base64-encoded DER certificate
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Certificate Not Found
|
||||
|
||||
```
|
||||
ERROR: Client certificate required but not provided
|
||||
```
|
||||
|
||||
**Causes**:
|
||||
- Certificate not sent by client
|
||||
- Wrong certificate path
|
||||
- Certificate expired
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Verify certificate is valid
|
||||
openssl x509 -in cert.pem -noout -text
|
||||
|
||||
# Check expiration
|
||||
openssl x509 -in cert.pem -noout -dates
|
||||
|
||||
# Test TLS handshake
|
||||
openssl s_client -connect localhost:50051 \
|
||||
-CAfile ca.pem -cert cert.pem -key key.pem
|
||||
```
|
||||
|
||||
#### 2. Permission Denied
|
||||
|
||||
```
|
||||
Code: PermissionDenied
|
||||
Message: Role 'X' does not have permission 'Y'
|
||||
```
|
||||
|
||||
**Causes**:
|
||||
- Certificate OU doesn't match any role
|
||||
- Role lacks required permission
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# Check certificate attributes
|
||||
openssl x509 -in cert.pem -noout -subject
|
||||
|
||||
# Review role mappings in src/main.rs
|
||||
# Verify permission requirements in docs
|
||||
```
|
||||
|
||||
#### 3. No Role Assigned
|
||||
|
||||
```
|
||||
ERROR: No role assigned to identity: CN=..., OU=...
|
||||
```
|
||||
|
||||
**Causes**:
|
||||
- Certificate OU not configured in role mappings
|
||||
- Typo in OU value
|
||||
|
||||
**Solutions**:
|
||||
- Add role mapping for the OU
|
||||
- Generate new certificate with correct OU
|
||||
- Check role mapping configuration
|
||||
|
||||
## Security Hardening
|
||||
|
||||
### Production Checklist
|
||||
|
||||
- [ ] Use certificates from trusted CA
|
||||
- [ ] Enable mTLS (`SECURITY_MTLS_ENABLED=true`)
|
||||
- [ ] Enable RBAC (`SECURITY_RBAC_ENABLED=true`)
|
||||
- [ ] Implement certificate rotation (90 days max)
|
||||
- [ ] Monitor certificate expiration
|
||||
- [ ] Enable audit logging
|
||||
- [ ] Use TLS 1.3 only
|
||||
- [ ] Implement rate limiting
|
||||
- [ ] Set up intrusion detection
|
||||
- [ ] Regular security audits
|
||||
|
||||
### Certificate Rotation
|
||||
|
||||
Implement automatic rotation:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# rotate-certs.sh
|
||||
|
||||
# Generate new certificate
|
||||
./generate-cert.sh
|
||||
|
||||
# Gracefully restart Charybdis
|
||||
kill -HUP $(pidof charybdis)
|
||||
|
||||
# Verify new cert is active
|
||||
sleep 5
|
||||
openssl s_client -connect localhost:50051 < /dev/null | \
|
||||
openssl x509 -noout -dates
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
Monitor security metrics:
|
||||
|
||||
- Authentication success/failure rate
|
||||
- Authorization denial rate
|
||||
- Certificate expiration warnings
|
||||
- Audit log anomalies
|
||||
- Failed login attempts
|
||||
|
||||
## Compliance
|
||||
|
||||
Charybdis security features support compliance requirements:
|
||||
|
||||
| Standard | Supported Features |
|
||||
|----------|-------------------|
|
||||
| **SOC 2** | Audit logging, access control, encryption in transit |
|
||||
| **ISO 27001** | Authentication, authorization, audit trails |
|
||||
| **PCI DSS** | Encryption, access control, logging |
|
||||
| **HIPAA** | Access control, audit logging, encryption |
|
||||
|
||||
## Next Steps
|
||||
|
||||
- 🔌 Configure [Plugins](plugins.md) with proper certificates
|
||||
- 🚀 Review [Deployment Guide](deployment.md) for production
|
||||
- 📊 Set up [Monitoring](monitoring.md) for security events
|
||||
|
||||
---
|
||||
|
||||
**Security Questions?** Open a [security issue](../../security) (for vulnerabilities, use private disclosure).
|
||||
Reference in New Issue
Block a user