Public Access
doc: update and cleanup
This commit is contained in:
Guillaume GRABÉ
File diff suppressed because it is too large
Load Diff
+299
-861
File diff suppressed because it is too large
Load Diff
+115
-551
@@ -1,126 +1,54 @@
|
||||
# Core Concepts
|
||||
|
||||
Charybdis is a **security-native platform engineering tool** that unifies software catalog, vulnerability management, and compliance posture behind a gRPC API. When entities change or scan results are ingested, the event bus triggers rules evaluation, security gate checks, and plugin integrations automatically.
|
||||
Charybdis is a software catalog with a gRPC API. Every change to an entity emits an event; plugins react to those events to keep external systems in sync. This page explains the entity model, the event flow, and how data is stored — the building blocks every other doc assumes.
|
||||
|
||||
This page explains the key concepts you need to understand.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Sources"
|
||||
A1[CI/CD Pipelines]
|
||||
A2[Security Scanners]
|
||||
A3[IaC Tools]
|
||||
end
|
||||
|
||||
subgraph "Charybdis Core"
|
||||
B1[gRPC API]
|
||||
B3[Authentication & Authorization]
|
||||
C1[Entity Service]
|
||||
C4[Vulnerability Engine]
|
||||
C5[Security Gates & Rules]
|
||||
C2[Event Bus]
|
||||
end
|
||||
|
||||
subgraph "Integrations - Plugins"
|
||||
D1[Slack / Teams]
|
||||
D2[Jira / GitHub]
|
||||
D3[Custom Plugins]
|
||||
end
|
||||
|
||||
subgraph "Data Layer"
|
||||
E1[(PostgreSQL)]
|
||||
end
|
||||
|
||||
A1 -.gRPC.-> B1
|
||||
A2 -.Scan Results.-> B1
|
||||
A3 -.gRPC.-> B1
|
||||
|
||||
B1 --> B3
|
||||
B3 --> C1
|
||||
B3 --> C4
|
||||
C4 --> C5
|
||||
C1 --> E1
|
||||
C4 --> E1
|
||||
C1 --> C2
|
||||
C4 --> C2
|
||||
|
||||
C2 -.Events.-> D1
|
||||
C2 -.Events.-> D2
|
||||
C2 -.Events.-> D3
|
||||
|
||||
style C4 fill:#E24A4A,stroke:#8A2E2E,color:#fff
|
||||
style C5 fill:#E2884A,stroke:#8A5C2E,color:#fff
|
||||
style C2 fill:#4A90E2,stroke:#2E5C8A,color:#fff
|
||||
```
|
||||
For a higher-level overview see the [README](../README.md). For deep technical details see [architecture.md](architecture.md).
|
||||
|
||||
## Entities
|
||||
|
||||
An **entity** is the core data model in Charybdis, representing any cataloged item in your software ecosystem.
|
||||
An **entity** is the unit of data in Charybdis. It represents something in your software ecosystem: a service, a team, an API, a security finding.
|
||||
|
||||
### Entity Structure
|
||||
### Entity structure
|
||||
|
||||
Every entity has three main parts:
|
||||
Every entity has the same shape:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class Entity {
|
||||
+string id
|
||||
+string kind
|
||||
+metadata
|
||||
+spec
|
||||
+annotations
|
||||
+timestamps
|
||||
}
|
||||
|
||||
class Metadata {
|
||||
+string name
|
||||
+string namespace
|
||||
+string description
|
||||
+labels
|
||||
+links
|
||||
+tags
|
||||
}
|
||||
|
||||
class Spec {
|
||||
+string type
|
||||
+string lifecycle
|
||||
+string owner
|
||||
+dependencies
|
||||
}
|
||||
|
||||
Entity --> Metadata
|
||||
Entity --> Spec
|
||||
```
|
||||
- `id` — UUID assigned by the server
|
||||
- `kind` — one of the supported kinds (see below)
|
||||
- `metadata` — identifying data (name, description, labels, tags, links)
|
||||
- `spec` — kind-specific configuration (type, lifecycle, owner, dependencies, ...)
|
||||
- `annotations` — arbitrary `string → string` map, typically used by plugins to store external IDs
|
||||
- `created_at`, `updated_at` — server-managed timestamps
|
||||
|
||||
### Entity Kinds
|
||||
`metadata` and `spec` are protobuf `oneof` fields whose variant matches `kind`. So a `Component` entity carries `component_metadata` + `component_spec`, a `User` carries `user_metadata` + `user_spec`, and so on. The on-wire JSON uses the variant name directly.
|
||||
|
||||
Charybdis supports all standard Backstage entity kinds:
|
||||
### Entity kinds
|
||||
|
||||
| Kind | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| **Service** | Individual microservices or applications | `payment-api` |
|
||||
| **Component** | Reusable libraries, SDKs, modules | `auth-sdk` |
|
||||
| **System** | Collections of services working together | `e-commerce-platform` |
|
||||
| **API** | Interfaces exposed by components | `payments-rest-api` |
|
||||
| **User** | Individual people | `john.doe` |
|
||||
| **Group** | Teams and organizational units | `team-payments` |
|
||||
| **Domain** | Business domains | `payments`, `shipping` |
|
||||
| **Resource** | Infrastructure resources | `payments-db`, `cache-cluster` |
|
||||
| Kind | Purpose |
|
||||
|---|---|
|
||||
| `Component` | Services, applications, libraries, websites |
|
||||
| `System` | Collections of components working together |
|
||||
| `API` | Interfaces exposed by components |
|
||||
| `User` | Individual people |
|
||||
| `Group` | Teams and organizational units |
|
||||
| `Domain` | Business domains |
|
||||
| `Resource` | Infrastructure resources (databases, caches, queues) |
|
||||
| `Finding` | Security findings ingested from scanners (see [Findings](#findings)) |
|
||||
|
||||
#### Example: Service
|
||||
Per-kind protobuf definitions: `proto/core/*.proto`.
|
||||
|
||||
### Example: Component
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"kind": "Component",
|
||||
"component_metadata": {
|
||||
"name": "payment-api",
|
||||
"namespace": "production",
|
||||
"description": "Payment processing service",
|
||||
"labels": { "team": "payments" }
|
||||
"labels": { "team": "payments", "tier": "critical" },
|
||||
"tags": ["api", "pci-dss"]
|
||||
},
|
||||
"service_spec": {
|
||||
"component_spec": {
|
||||
"type": "service",
|
||||
"lifecycle": "production",
|
||||
"owner": "team-payments",
|
||||
@@ -129,500 +57,136 @@ Charybdis supports all standard Backstage entity kinds:
|
||||
}
|
||||
```
|
||||
|
||||
#### Example: Component
|
||||
### Common metadata fields
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "Component",
|
||||
"component_metadata": {
|
||||
"name": "auth-sdk",
|
||||
"namespace": "shared"
|
||||
},
|
||||
"component_spec": {
|
||||
"type": "library",
|
||||
"lifecycle": "production",
|
||||
"owner": "platform-team"
|
||||
}
|
||||
}
|
||||
```
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `name` | string | Required. Unique with `kind` (composite index `idx_entities_kind_name`). |
|
||||
| `namespace` | string | Logical grouping, e.g. `production`. |
|
||||
| `description` | string | Human-readable. |
|
||||
| `labels` | `map<string,string>` | Categorization. |
|
||||
| `tags` | `repeated string` | Free-form classification. |
|
||||
| `links` | `repeated Link` | External URLs (dashboards, docs). |
|
||||
|
||||
#### Example: System
|
||||
## Annotations
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "System",
|
||||
"system_metadata": {
|
||||
"name": "e-commerce-platform",
|
||||
"namespace": "production",
|
||||
"description": "Complete e-commerce system"
|
||||
},
|
||||
"system_spec": {
|
||||
"owner": "platform-team",
|
||||
"domain": "retail"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Metadata Fields
|
||||
|
||||
| Field | Type | Description | Required |
|
||||
|-------|------|-------------|----------|
|
||||
| `name` | string | Entity name (unique within namespace) | ✅ |
|
||||
| `namespace` | string | Logical grouping (e.g., "production", "staging") | ✅ |
|
||||
| `description` | string | Human-readable description | ❌ |
|
||||
| `labels` | map | Key-value pairs for categorization | ❌ |
|
||||
| `tags` | array | Search tags | ❌ |
|
||||
| `links` | array | External URLs (dashboards, docs, etc.) | ❌ |
|
||||
|
||||
### Annotations
|
||||
|
||||
Annotations store integration-specific metadata:
|
||||
Annotations are an open `map<string, string>` on every entity, intended for integration metadata. Plugins write their external IDs here; queries can index into them via PostgreSQL JSONB.
|
||||
|
||||
```json
|
||||
{
|
||||
"annotations": {
|
||||
"github.com/repo-slug": "myorg/payment-service",
|
||||
"defectdojo.com/product-id": "123",
|
||||
"dependencytrack.com/project-uuid": "550e8400...",
|
||||
"pagerduty.com/service-id": "PXYZ123",
|
||||
"grafana.com/dashboard-url": "https://..."
|
||||
"keycloak.com/email": "alice@example.com",
|
||||
"pagerduty.com/service-id": "PXYZ123"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Best Practices**:
|
||||
- Use domain-style keys (`tool.com/key`)
|
||||
- Store tool-specific IDs
|
||||
- Keep values as strings
|
||||
- Use for integration metadata only
|
||||
Convention: reverse-DNS keys (`tool.com/key`). Values are always strings.
|
||||
|
||||
`EntityRepository::update_annotations()` performs an atomic JSONB merge at the SQL level — safe for concurrent writes from multiple handlers.
|
||||
|
||||
## Events
|
||||
|
||||
Charybdis uses an **event-driven architecture** to trigger actions when entities change.
|
||||
Charybdis emits an `EntityEvent` on every CRUD operation, delivered to subscribers via the in-memory event bus.
|
||||
|
||||
### Event Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant API as Entity Service
|
||||
participant Bus as Event Bus
|
||||
participant Plugin1 as DefectDojo Plugin
|
||||
participant Plugin2 as Dependency-Track
|
||||
|
||||
Client->>API: CreateEntity(service)
|
||||
API->>API: Store entity
|
||||
API->>Bus: Emit EntityCreated event
|
||||
Bus->>Plugin1: Handle event
|
||||
Bus->>Plugin2: Handle event
|
||||
Plugin1-->>Plugin1: Create DefectDojo product
|
||||
Plugin2-->>Plugin2: Create DT project
|
||||
API-->>Client: Return created entity
|
||||
|
||||
Note over Bus,Plugin2: Asynchronous processing
|
||||
```
|
||||
Client → CreateEntity → DB INSERT → EntityEvent::Created → Event bus → Subscribers
|
||||
```
|
||||
|
||||
### Event Types
|
||||
### Event types
|
||||
|
||||
| Event | Trigger | Plugins Receive |
|
||||
|-------|---------|----------------|
|
||||
| `EntityCreated` | New entity created | Full entity data |
|
||||
| `EntityUpdated` | Entity modified | Updated entity + changes |
|
||||
| `EntityDeleted` | Entity removed | Entity ID + metadata |
|
||||
| Type | When |
|
||||
|---|---|
|
||||
| `Created` | Successful `CreateEntity` |
|
||||
| `Updated` | Successful `UpdateEntity` (full or partial) |
|
||||
| `Deleted` | Successful `DeleteEntity` |
|
||||
|
||||
### Event Structure
|
||||
### Event payload
|
||||
|
||||
```rust
|
||||
pub enum EntityEvent {
|
||||
Created {
|
||||
entity: Entity,
|
||||
timestamp: DateTime<Utc>,
|
||||
},
|
||||
Updated {
|
||||
entity: Entity,
|
||||
previous: Entity,
|
||||
timestamp: DateTime<Utc>,
|
||||
},
|
||||
Deleted {
|
||||
id: String,
|
||||
metadata: Metadata,
|
||||
timestamp: DateTime<Utc>,
|
||||
},
|
||||
pub struct EntityEvent {
|
||||
pub event_id: Uuid,
|
||||
pub entity_id: Uuid,
|
||||
pub event_type: EntityEventType, // Created | Updated | Deleted
|
||||
pub timestamp: DateTime<Utc>,
|
||||
pub metadata: HashMap<String, String>,
|
||||
pub entity_data: Option<Arc<Entity>>, // full entity for Created/Updated
|
||||
}
|
||||
```
|
||||
|
||||
## Vulnerabilities & Security (Phase 1 — Planned)
|
||||
### Delivery semantics
|
||||
|
||||
> **Note**: The features described in this section are part of Phase 1 (Security Core) and are not yet implemented. This documents the planned architecture. See [VISION.md](../VISION.md) for the roadmap.
|
||||
- **Asynchronous** — `CreateEntity` returns to the client before plugin handlers complete.
|
||||
- **At-least-once intent**, but the default `MemoryEventBus` is in-process; if the server restarts mid-dispatch, events are lost. A durable backend (Redis) is planned in [VISION.md](../VISION.md).
|
||||
- **Per-handler timeout** — 30 s. Slow plugins don't block the bus.
|
||||
- **Panic isolation** — a panicking handler doesn't take down others.
|
||||
|
||||
Security is a **first-class concept** in Charybdis, not a plugin. Once Phase 1 is complete, vulnerability management, scan ingestion, security gates, and assessment workflows will be native to the core.
|
||||
Plugin lifecycle and how to write a handler: [plugins/README.md](../plugins/README.md).
|
||||
|
||||
### Vulnerability Lifecycle
|
||||
## Findings
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Scanner
|
||||
participant API as Charybdis API
|
||||
participant Rules as Rules Engine
|
||||
participant Gate as Security Gate
|
||||
participant Bus as Event Bus
|
||||
participant Plugin as Slack / Jira
|
||||
`Finding` is a first-class entity kind for security findings produced by scanners. Findings are created by the `IngestionService`, not directly by clients.
|
||||
|
||||
Scanner->>API: IngestScan(SARIF report)
|
||||
API->>API: Parse & create Vulnerability entities
|
||||
API->>API: Link vulnerabilities to Component
|
||||
API->>Rules: Evaluate auto-assessment rules
|
||||
Rules-->>API: Auto-assess (e.g., accept known low-risk)
|
||||
API->>Gate: Check security gate thresholds
|
||||
alt Gate Passed
|
||||
Gate-->>API: OK
|
||||
else Gate Failed
|
||||
Gate->>Bus: GateFailed event
|
||||
Bus->>Plugin: Alert #security channel
|
||||
end
|
||||
API->>Bus: VulnerabilitiesIngested event
|
||||
Bus->>Plugin: Notify / create tickets
|
||||
```
|
||||
Each finding is scoped to a `(component_ref, lifecycle)` pair — so production and staging environments track findings independently without duplicating the underlying `Component`.
|
||||
|
||||
### Scan Ingestion
|
||||
The ingestion flow:
|
||||
|
||||
Charybdis ingests scan results natively. You don't need an external vulnerability management tool.
|
||||
1. Client calls `IngestionService.ImportScan(component_ref, lifecycle, format, data)`.
|
||||
2. The configured parser (currently SARIF) normalizes scanner output into `NormalizedFinding` records.
|
||||
3. The reconciliation engine computes a fingerprint for each finding (scanner-provided when present, else `sha256(scanner | rule_id | file_path)`) and diffs incoming vs existing findings for that scope.
|
||||
4. The result is one of:
|
||||
- **New** — fingerprint not seen before → create a `Finding` entity.
|
||||
- **Unchanged** — already active → bump `last_seen`.
|
||||
- **Resolved** — previously active, not in this scan → mark `Resolved`.
|
||||
- **Reopened** — previously resolved or false-positive, detected again → mark `Reopened`.
|
||||
5. `DryRunScan` runs the same pipeline but skips persistence — used for MR/PR-level "this change introduces X new vulnerabilities" comments.
|
||||
|
||||
| Format | Coverage | Use Case |
|
||||
|--------|----------|----------|
|
||||
| **SARIF** | 60%+ of modern scanners (Semgrep, CodeQL, Trivy, etc.) | SAST, DAST, secrets |
|
||||
| **CycloneDX** | SBOMs + vulnerability data | SCA, license |
|
||||
| **SPDX** | License and package data | License compliance |
|
||||
Finding state values (see `proto/core/finding.proto`):
|
||||
|
||||
### Assessments
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `ACTIVE` | Currently detected |
|
||||
| `RESOLVED` | No longer detected (auto-set on re-import) |
|
||||
| `ACCEPTED` | Risk accepted by human decision *(state value defined; assessment workflow is planned)* |
|
||||
| `FALSE_POSITIVE` | Marked as not a real issue *(state value defined; assessment workflow is planned)* |
|
||||
| `REOPENED` | Was resolved, detected again |
|
||||
|
||||
Each vulnerability can be assessed:
|
||||
|
||||
| Status | Meaning |
|
||||
|--------|---------|
|
||||
| **Open** | New, unreviewed vulnerability |
|
||||
| **In Triage** | Under review by security team |
|
||||
| **Accepted** | Risk accepted with justification |
|
||||
| **Remediated** | Fixed, pending verification |
|
||||
| **False Positive** | Not a real vulnerability |
|
||||
| **Auto-Assessed** | Automatically assessed by rules engine |
|
||||
|
||||
### Security Gates
|
||||
|
||||
Security gates define thresholds per product:
|
||||
|
||||
```
|
||||
payment-api:
|
||||
critical: 0 # No critical vulns allowed
|
||||
high: 5 # Up to 5 high
|
||||
medium: 20 # Up to 20 medium
|
||||
```
|
||||
|
||||
When a gate is violated, events fire and plugins react (block CI/CD, alert Slack, create Jira tickets).
|
||||
|
||||
### Rules Engine
|
||||
|
||||
Rules auto-assess vulnerabilities based on patterns:
|
||||
|
||||
- Severity + component combination (e.g., "low severity in test dependencies → auto-accept")
|
||||
- Scanner source (e.g., "all informational from ZAP → auto-accept")
|
||||
- Known patterns (e.g., "CVE-XXXX already accepted org-wide")
|
||||
|
||||
### License Compliance (Phase 2)
|
||||
|
||||
Track licenses across your dependency tree:
|
||||
|
||||
- Ingest license data from CycloneDX/SPDX
|
||||
- Define license policies (allowed, restricted, banned)
|
||||
- Flag violations per component
|
||||
- Compliance reporting
|
||||
Only SARIF is shipped today. Additional parsers, assessment workflows, rules engine, and security gates are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).
|
||||
|
||||
## Storage Model
|
||||
|
||||
Charybdis uses PostgreSQL with JSONB for schema-less storage.
|
||||
|
||||
### Database Schema
|
||||
Charybdis stores every entity as a protobuf blob in a fixed PostgreSQL schema. The schema never changes — adding a new entity kind means adding a `oneof` variant in protobuf, not a column migration.
|
||||
|
||||
```sql
|
||||
CREATE TABLE entities (
|
||||
id UUID PRIMARY KEY,
|
||||
kind TEXT NOT NULL,
|
||||
entity_data BYTEA NOT NULL, -- Protobuf binary
|
||||
annotations JSONB NOT NULL DEFAULT '{}', -- Plugin metadata (indexed)
|
||||
created_at TIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP NOT NULL
|
||||
id UUID PRIMARY KEY,
|
||||
kind VARCHAR NOT NULL,
|
||||
name VARCHAR,
|
||||
entity_data BYTEA NOT NULL, -- protobuf-encoded Entity
|
||||
annotations JSONB NOT NULL DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL,
|
||||
updated_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_entities_kind ON entities(kind);
|
||||
CREATE INDEX idx_entities_annotations ON entities USING GIN(annotations);
|
||||
CREATE INDEX idx_entities_kind ON entities(kind);
|
||||
CREATE INDEX idx_entities_kind_name ON entities(kind, name);
|
||||
CREATE INDEX idx_entities_annotations ON entities USING GIN (annotations);
|
||||
CREATE INDEX idx_entities_created_at_id ON entities(created_at DESC, id DESC);
|
||||
```
|
||||
|
||||
### Why Protobuf + JSONB?
|
||||
Why this design:
|
||||
- **No migrations** — new plugins extend the protobuf `oneof` variants; database schema is unchanged.
|
||||
- **Compact storage** — protobuf binary is smaller than JSON.
|
||||
- **Forward/backward compatible** — proto field numbers protect against version drift.
|
||||
- **Fast annotation queries** — JSONB + GIN index supports `annotations->>'key' = 'value'` lookups in O(log n).
|
||||
|
||||
**Advantages**:
|
||||
- ✅ No schema migrations — new entity kinds via protobuf `oneof`, schema never changes
|
||||
- ✅ Protobuf binary storage for compact, versioned entity data
|
||||
- ✅ JSONB annotations for fast querying with GIN indexes
|
||||
- ✅ Forward/backward compatibility built-in
|
||||
Detailed schema reasoning, scaling notes, and field-mask paths: [architecture.md](architecture.md).
|
||||
|
||||
**Example Query**:
|
||||
```sql
|
||||
-- Find entities with specific annotation
|
||||
SELECT * FROM entities
|
||||
WHERE annotations->>'defectdojo.com/product-id' = '42';
|
||||
```
|
||||
## See Also
|
||||
|
||||
## Security Model
|
||||
|
||||
Charybdis implements defense-in-depth security.
|
||||
|
||||
### Authentication: mTLS
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant TLS as TLS Layer
|
||||
participant Auth as Auth Interceptor
|
||||
participant Service as Entity Service
|
||||
|
||||
Client->>TLS: Connect with client certificate
|
||||
TLS->>TLS: Validate certificate
|
||||
TLS->>Auth: Extract certificate
|
||||
Auth->>Auth: Parse identity (CN, OU, O)
|
||||
Auth->>Auth: Map to role
|
||||
Client->>Auth: Request (with identity)
|
||||
Auth->>Auth: Check permissions
|
||||
alt Authorized
|
||||
Auth->>Service: Forward request
|
||||
Service-->>Client: Response
|
||||
else Denied
|
||||
Auth-->>Client: PermissionDenied error
|
||||
end
|
||||
```
|
||||
|
||||
### Authorization: RBAC
|
||||
|
||||
**Role Mapping**:
|
||||
|
||||
```
|
||||
Certificate (CN, OU, O) → RBAC Role → Permissions
|
||||
```
|
||||
|
||||
**Default Roles**:
|
||||
|
||||
| Role | Certificate OU | Permissions |
|
||||
|------|---------------|-------------|
|
||||
| `platform` | `platform-team` | Full access (CRUD + list) |
|
||||
| `automation` | `automation` | Create, read, update, list |
|
||||
| `plugin` | `plugins` | Read, list only |
|
||||
|
||||
### Permissions
|
||||
|
||||
| Permission | Operations | Required For |
|
||||
|------------|-----------|--------------|
|
||||
| `entity:create` | Create new entities | CreateEntity |
|
||||
| `entity:read` | Get entity by ID | GetEntity |
|
||||
| `entity:update` | Modify entities | UpdateEntity |
|
||||
| `entity:delete` | Remove entities | DeleteEntity |
|
||||
| `entity:list` | List all entities | ListEntities |
|
||||
|
||||
## Plugin System
|
||||
|
||||
Core features (catalog, vulnerabilities, compliance) are **native**. Plugins handle **integrations** with external systems.
|
||||
|
||||
### Plugin Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Entity/Vuln Event] --> B[Event Bus]
|
||||
B --> C{Plugin Manager}
|
||||
C --> D[Slack Plugin]
|
||||
C --> E[Jira Plugin]
|
||||
C --> F[Custom Plugin]
|
||||
|
||||
D --> G[Slack API]
|
||||
E --> H[Jira API]
|
||||
F --> I[Your Tool API]
|
||||
|
||||
style C fill:#4A90E2,stroke:#2E5C8A
|
||||
```
|
||||
|
||||
### What's Native vs. Plugin
|
||||
|
||||
| Native (core) | Status | Plugin (integration) | Status |
|
||||
|---|---|---|---|
|
||||
| Software catalog | Done | DefectDojo sync | Done |
|
||||
| Vulnerability management | Phase 1 | Keycloak user sync | Done |
|
||||
| Security gates & rules | Phase 1 | Slack / Teams notifications | Planned |
|
||||
| Assessment workflow | Phase 1 | Jira / GitHub issue creation | Planned |
|
||||
| License compliance | Phase 2 | Custom integrations | Framework ready |
|
||||
| TechDocs | Future | | |
|
||||
|
||||
### Plugin Lifecycle
|
||||
|
||||
1. **Configuration** - Load plugin settings from `plugins.toml`
|
||||
2. **Initialization** - Plugin registers event handlers
|
||||
3. **Event Processing** - Plugin receives events asynchronously (entity, vulnerability, gate events)
|
||||
4. **External Integration** - Plugin calls external tool APIs
|
||||
5. **Error Handling** - Failed plugins don't affect core service
|
||||
|
||||
### Plugin Configuration
|
||||
|
||||
Example `plugins.toml`:
|
||||
|
||||
```toml
|
||||
[plugins.slack]
|
||||
enabled = true
|
||||
webhook_url = "${SLACK_WEBHOOK_URL}"
|
||||
|
||||
[[plugins.slack.on_gate_failed]]
|
||||
channel = "#security-alerts"
|
||||
|
||||
[[plugins.slack.on_entity_created]]
|
||||
channel = "#platform"
|
||||
```
|
||||
|
||||
See [Plugin Guide](plugins.md) for details.
|
||||
|
||||
## Data Consistency
|
||||
|
||||
### Eventual Consistency
|
||||
|
||||
Charybdis uses **eventual consistency** for plugin integrations:
|
||||
|
||||
- Entity CRUD operations are **immediately consistent**
|
||||
- Plugin synchronization is **eventually consistent**
|
||||
- Events are processed **asynchronously**
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[CreateEntity] -->|Immediate| B[Entity Stored]
|
||||
B -->|Async| C[Event Emitted]
|
||||
C -->|Async| D[Plugin Processing]
|
||||
D -->|Eventual| E[External Tool Synced]
|
||||
|
||||
style B fill:#00C851
|
||||
style E fill:#ffbb33
|
||||
```
|
||||
|
||||
### Guarantees
|
||||
|
||||
| Operation | Consistency | Guarantee |
|
||||
|-----------|-------------|-----------|
|
||||
| Entity CRUD | Strong | Immediate |
|
||||
| Entity queries | Strong | Read-your-writes |
|
||||
| Event delivery | At-least-once | May retry |
|
||||
| Plugin sync | Eventual | Best-effort |
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Scalability
|
||||
|
||||
- **Entities**: Tested with 100,000+ entities
|
||||
- **Throughput**: 1,000+ requests/second
|
||||
- **Latency**: Sub-millisecond average
|
||||
- **Concurrency**: Tokio async runtime
|
||||
|
||||
### Resource Usage
|
||||
|
||||
Typical resource consumption:
|
||||
|
||||
| Component | CPU | Memory | Storage |
|
||||
|-----------|-----|--------|---------|
|
||||
| Charybdis | < 5% | ~50 MB | Minimal |
|
||||
| PostgreSQL | ~10% | ~256 MB | Depends on entity count |
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Index annotations** used for frequent queries
|
||||
2. **Use connection pooling** (built-in)
|
||||
3. **Enable query caching** in PostgreSQL
|
||||
4. **Monitor event bus** queue depth
|
||||
|
||||
## Backstage Migration
|
||||
|
||||
Charybdis includes a YAML adapter for teams migrating from Backstage. This is a **migration path**, not the primary interface.
|
||||
|
||||
### How It Works
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Charybdis Entity] --> B[YAML Adapter]
|
||||
B --> C[Backstage YAML Format]
|
||||
C --> D[Backstage Catalog]
|
||||
|
||||
style B fill:#4A90E2,stroke:#2E5C8A
|
||||
```
|
||||
|
||||
Point Backstage at Charybdis and stop maintaining `catalog-info.yaml` files:
|
||||
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://charybdis:8080/yaml/locations
|
||||
```
|
||||
|
||||
### Recommended Migration Path
|
||||
|
||||
1. **Start** with Charybdis + YAML adapter feeding your existing Backstage
|
||||
2. **Adopt** Charybdis gRPC API for CI/CD integrations and security scanning
|
||||
3. **Leverage** event-driven plugins for auto-provisioning (DefectDojo, Jira, etc.)
|
||||
4. **Optionally retire** Backstage when Charybdis covers your catalog needs
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Entity Design
|
||||
|
||||
✅ **DO**:
|
||||
- Use descriptive names
|
||||
- Group by namespace
|
||||
- Add relevant labels
|
||||
- Include documentation links
|
||||
- Set appropriate owners
|
||||
|
||||
❌ **DON'T**:
|
||||
- Store sensitive data in metadata
|
||||
- Use very long descriptions
|
||||
- Create deeply nested hierarchies
|
||||
- Duplicate data across entities
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
```
|
||||
<entity-type>-<purpose>-<environment>
|
||||
|
||||
Examples:
|
||||
- payment-api-prod
|
||||
- user-service-staging
|
||||
- auth-library
|
||||
- e-commerce-system
|
||||
```
|
||||
|
||||
### Metadata Organization
|
||||
|
||||
```json
|
||||
{
|
||||
"labels": {
|
||||
"team": "payments", // Ownership
|
||||
"tier": "critical", // Importance
|
||||
"environment": "production" // Deployment
|
||||
},
|
||||
"tags": ["pci-compliant", "public-api"],
|
||||
"links": [
|
||||
{ "url": "...", "title": "Dashboard" },
|
||||
{ "url": "...", "title": "Documentation" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Read the [Vision & Roadmap](../VISION.md) to understand where Charybdis is going
|
||||
- Configure [Security](security.md) for production (mTLS + RBAC)
|
||||
- Explore [Plugins](plugins.md) for external integrations
|
||||
- Review the [Architecture](architecture.md) for technical deep dive
|
||||
|
||||
---
|
||||
|
||||
**Questions?** [Open an issue](../../issues).
|
||||
- [Architecture](architecture.md) — protobuf schema, scaling, design decisions
|
||||
- [Security](security.md) — mTLS + RBAC configuration
|
||||
- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
|
||||
- [Vision & Roadmap](../VISION.md) — planned features
|
||||
|
||||
+162
-339
@@ -1,39 +1,26 @@
|
||||
# Getting Started with Charybdis
|
||||
|
||||
This guide will get you from zero to a running Charybdis instance in minutes. By the end, you'll have a working software catalog that can register services via gRPC, ingest scan results, and auto-trigger integrations via event-driven plugins.
|
||||
This guide gets you from zero to a running Charybdis instance: a software catalog with a gRPC API, optional Backstage YAML adapter, and plugin integrations on entity events.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Rust** - 1.70 or later ([install](https://rustup.rs/))
|
||||
- **PostgreSQL** - 14 or later
|
||||
- **grpcurl** - For testing (optional, [install](https://github.com/fullstorydev/grpcurl))
|
||||
- **Rust** 1.70+ ([install](https://rustup.rs/))
|
||||
- **PostgreSQL** 14+
|
||||
- **protoc** + `libprotobuf-dev` (for the build script)
|
||||
- **grpcurl** for testing (optional, [install](https://github.com/fullstorydev/grpcurl))
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1: From Source
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/charybdis-catalog/charybdis.git
|
||||
git clone <your-charybdis-repo-url>
|
||||
cd charybdis
|
||||
|
||||
# Build
|
||||
cargo build --release
|
||||
|
||||
# The binary will be at target/release/charybdis
|
||||
```
|
||||
|
||||
### Option 2: Docker (Coming Soon)
|
||||
|
||||
```bash
|
||||
docker pull charybdis/charybdis:latest
|
||||
# Binary at target/release/charybdis-server
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Start PostgreSQL
|
||||
|
||||
Using Docker:
|
||||
### 1. Start PostgreSQL
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
@@ -43,54 +30,22 @@ docker run -d \
|
||||
postgres:15
|
||||
```
|
||||
|
||||
Or use an existing PostgreSQL instance.
|
||||
|
||||
### Step 2: Configure Charybdis
|
||||
|
||||
**Recommended: Use config.toml**
|
||||
|
||||
Copy the example configuration:
|
||||
### 2. Configure
|
||||
|
||||
```bash
|
||||
cp config.toml.example config.toml
|
||||
```
|
||||
|
||||
Edit `config.toml` and set your database URL:
|
||||
|
||||
```toml
|
||||
[database]
|
||||
url = "${DATABASE_URL}"
|
||||
```
|
||||
|
||||
Set the environment variable:
|
||||
|
||||
```bash
|
||||
export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
|
||||
```
|
||||
|
||||
**Alternative: Environment Variables Only (Legacy)**
|
||||
`config.toml` reads `${DATABASE_URL}` from the environment. See [Configuration](#configuration) below for all options.
|
||||
|
||||
If you prefer environment variables:
|
||||
|
||||
```bash
|
||||
# Database
|
||||
export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
|
||||
|
||||
# Disable security for quick start
|
||||
export SECURITY_MTLS_ENABLED=false
|
||||
export SECURITY_RBAC_ENABLED=false
|
||||
|
||||
# Logging
|
||||
export RUST_LOG=info,charybdis=debug
|
||||
```
|
||||
|
||||
### Step 3: Run Charybdis
|
||||
### 3. Run
|
||||
|
||||
```bash
|
||||
cargo run
|
||||
```
|
||||
|
||||
You should see:
|
||||
Expected startup logs:
|
||||
|
||||
```
|
||||
INFO charybdis: Database ready
|
||||
@@ -98,40 +53,34 @@ INFO charybdis: Event bus started successfully
|
||||
INFO charybdis: EntityService server listening on [::1]:50051
|
||||
```
|
||||
|
||||
### Step 4: Verify It's Working
|
||||
|
||||
Test with grpcurl:
|
||||
### 4. Verify
|
||||
|
||||
```bash
|
||||
# List available services
|
||||
grpcurl -plaintext localhost:50051 list
|
||||
|
||||
# Output:
|
||||
# charybdis.entities.EntityService
|
||||
# charybdis.ingestion.IngestionService
|
||||
# grpc.reflection.v1.ServerReflection
|
||||
```
|
||||
|
||||
Congratulations! Charybdis is running! 🎉
|
||||
|
||||
## Creating Your First Entity
|
||||
|
||||
### Using grpcurl
|
||||
|
||||
Create a service entity:
|
||||
## Create Your First Entity
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext \
|
||||
-d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "payment-service",
|
||||
"namespace": "production",
|
||||
"description": "Core payment processing service"
|
||||
}
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Component",
|
||||
"component_metadata": {
|
||||
"name": "payment-api",
|
||||
"namespace": "production",
|
||||
"description": "Payment processing service",
|
||||
"tags": ["api", "critical"]
|
||||
},
|
||||
"component_spec": {
|
||||
"type": "service",
|
||||
"lifecycle": "production",
|
||||
"owner": "team-payments"
|
||||
}
|
||||
}' \
|
||||
localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
Response:
|
||||
@@ -140,117 +89,60 @@ Response:
|
||||
{
|
||||
"entity": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"kind": "Service",
|
||||
"serviceMetadata": {
|
||||
"name": "payment-service",
|
||||
"namespace": "production",
|
||||
"description": "Core payment processing service"
|
||||
},
|
||||
"createdAt": "2025-11-04T10:00:00Z",
|
||||
"updatedAt": "2025-11-04T10:00:00Z"
|
||||
"kind": "Component",
|
||||
"componentMetadata": { "...": "..." },
|
||||
"createdAt": "2026-06-01T10:00:00Z",
|
||||
"updatedAt": "2026-06-01T10:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### List All Entities
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
```
|
||||
|
||||
### Get Entity by ID
|
||||
### Retrieve and list
|
||||
|
||||
```bash
|
||||
# Get by ID
|
||||
grpcurl -plaintext \
|
||||
-d '{"id": "550e8400-e29b-41d4-a716-446655440000"}' \
|
||||
localhost:50051 charybdis.entities.EntityService/GetEntity
|
||||
|
||||
# List all
|
||||
grpcurl -plaintext -d '{}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
|
||||
# Filter by kind and name
|
||||
grpcurl -plaintext \
|
||||
-d '{"kind": "Component", "name": "payment-api"}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
```
|
||||
|
||||
## Entity Types
|
||||
## Entity Kinds
|
||||
|
||||
Charybdis supports three main entity types:
|
||||
The valid `kind` values are: `Component`, `System`, `API`, `User`, `Group`, `Domain`, `Resource`, `Finding`. Each kind uses a matching `<kind>_metadata` + `<kind>_spec` payload.
|
||||
|
||||
### Service
|
||||
Conceptual reference: [core-concepts.md](core-concepts.md). Per-kind protobuf definitions: `proto/core/*.proto`.
|
||||
|
||||
Individual microservices or applications:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "user-api",
|
||||
"namespace": "production",
|
||||
"description": "User management API"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
### System
|
||||
|
||||
Collections of related services:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "System",
|
||||
"system_metadata": {
|
||||
"name": "payment-system",
|
||||
"namespace": "production",
|
||||
"description": "Complete payment processing system"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
### Component
|
||||
|
||||
Reusable components or libraries:
|
||||
### Component example with metadata and annotations
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Component",
|
||||
"component_spec": {
|
||||
"type": "library",
|
||||
"lifecycle": "production",
|
||||
"owner": "platform-team"
|
||||
},
|
||||
"component_metadata": {
|
||||
"name": "auth-library",
|
||||
"namespace": "shared",
|
||||
"description": "Shared authentication library"
|
||||
}
|
||||
}
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
## Adding Metadata
|
||||
|
||||
Entities support rich metadata:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "payment-service",
|
||||
"namespace": "production",
|
||||
"description": "Payment processing",
|
||||
"labels": {
|
||||
"team": "payments",
|
||||
"tier": "critical"
|
||||
},
|
||||
"links": [
|
||||
{
|
||||
"url": "https://dashboard.company.com/payments",
|
||||
"title": "Dashboard",
|
||||
"icon": "dashboard"
|
||||
}
|
||||
],
|
||||
"tags": ["payments", "pci-compliant", "critical"]
|
||||
"labels": { "team": "payments", "tier": "critical" },
|
||||
"links": [{
|
||||
"url": "https://dashboard.company.com/payments",
|
||||
"title": "Dashboard",
|
||||
"icon": "dashboard"
|
||||
}],
|
||||
"tags": ["payments", "pci-compliant"]
|
||||
},
|
||||
"component_spec": {
|
||||
"type": "service",
|
||||
"lifecycle": "production",
|
||||
"owner": "team-payments"
|
||||
},
|
||||
"annotations": {
|
||||
"github.com/repo-slug": "myorg/payment-service",
|
||||
@@ -260,105 +152,85 @@ grpcurl -plaintext -d '{
|
||||
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
## Integrating with CI/CD
|
||||
## Registering Entities from CI
|
||||
|
||||
### Example: GitHub Actions
|
||||
|
||||
Create a workflow to register services automatically:
|
||||
Charybdis is designed to be called from pipelines. With `grpcurl` available in your runner:
|
||||
|
||||
```yaml
|
||||
name: Register Service
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
register:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Register in Charybdis
|
||||
run: |
|
||||
grpcurl -plaintext \
|
||||
-d '{
|
||||
"entity": {
|
||||
"kind": "Service",
|
||||
"service_metadata": {
|
||||
"name": "${{ github.event.repository.name }}",
|
||||
"namespace": "production",
|
||||
"description": "${{ github.event.repository.description }}"
|
||||
},
|
||||
"annotations": {
|
||||
"github.com/repo-slug": "${{ github.repository }}"
|
||||
}
|
||||
}
|
||||
}' \
|
||||
your-charybdis-host:50051 \
|
||||
charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
### Example: GitLab CI
|
||||
|
||||
```yaml
|
||||
register_service:
|
||||
stage: deploy
|
||||
script:
|
||||
- |
|
||||
grpcurl -plaintext \
|
||||
-d "{
|
||||
\"entity\": {
|
||||
\"kind\": \"Service\",
|
||||
\"service_metadata\": {
|
||||
\"name\": \"${CI_PROJECT_NAME}\",
|
||||
\"namespace\": \"${CI_ENVIRONMENT_NAME}\"
|
||||
}
|
||||
# Example pipeline step (Gitea Actions / GitHub Actions syntax)
|
||||
- name: Register service in Charybdis
|
||||
run: |
|
||||
grpcurl -plaintext \
|
||||
-d "{
|
||||
\"entity\": {
|
||||
\"kind\": \"Component\",
|
||||
\"component_metadata\": {
|
||||
\"name\": \"$CI_PROJECT_NAME\",
|
||||
\"namespace\": \"production\"
|
||||
},
|
||||
\"component_spec\": {
|
||||
\"type\": \"service\",
|
||||
\"lifecycle\": \"production\",
|
||||
\"owner\": \"$CI_PROJECT_NAMESPACE\"
|
||||
},
|
||||
\"annotations\": {
|
||||
\"repo-slug\": \"$CI_PROJECT_PATH\"
|
||||
}
|
||||
}" \
|
||||
your-charybdis-host:50051 \
|
||||
charybdis.entities.EntityService/CreateEntity
|
||||
}
|
||||
}" \
|
||||
charybdis.internal:50051 \
|
||||
charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
In production, secure the endpoint with mTLS (see [Enabling Security](#enabling-security)).
|
||||
|
||||
## Enabling Security
|
||||
|
||||
For production use, enable mTLS and RBAC:
|
||||
Charybdis ships with mTLS + RBAC disabled for local exploration. For shared or production environments, enable both.
|
||||
|
||||
### Step 1: Generate Certificates
|
||||
### 1. Generate dev certificates
|
||||
|
||||
```bash
|
||||
# Use the provided test script
|
||||
./test-mtls-rbac.sh
|
||||
./deploy/scripts/generate-dev-certs.sh
|
||||
```
|
||||
|
||||
This creates:
|
||||
- `certs/ca.pem` - Certificate Authority
|
||||
- `certs/server-cert.pem` / `server-key.pem` - Server certificate
|
||||
- `certs/admin-cert.pem` / `admin-key.pem` - Admin client certificate
|
||||
This writes `deploy/certs/` with:
|
||||
- `ca.pem` — CA
|
||||
- `server-cert.pem` / `server-key.pem` — server
|
||||
- `admin-cert.pem` / `admin-key.pem` — admin client
|
||||
- (and per-role client certs)
|
||||
|
||||
### Step 2: Enable Security
|
||||
### 2. Enable in `config.toml`
|
||||
|
||||
```bash
|
||||
export SECURITY_MTLS_ENABLED=true
|
||||
export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem
|
||||
export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem
|
||||
export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem
|
||||
export SECURITY_RBAC_ENABLED=true
|
||||
```toml
|
||||
[security.mtls]
|
||||
enabled = true
|
||||
server_cert = "./deploy/certs/server-cert.pem"
|
||||
server_key = "./deploy/certs/server-key.pem"
|
||||
client_ca_cert = "./deploy/certs/ca.pem"
|
||||
|
||||
[security.rbac]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
### Step 3: Test with mTLS
|
||||
Restart Charybdis. RBAC defaults map cert OUs to roles (`platform-team` → full access, `automation` → CRUD without delete, `plugins` → read-only).
|
||||
|
||||
### 3. Call with mTLS
|
||||
|
||||
```bash
|
||||
grpcurl \
|
||||
-cacert certs/ca.pem \
|
||||
-cert certs/admin-cert.pem \
|
||||
-key certs/admin-key.pem \
|
||||
-cacert deploy/certs/ca.pem \
|
||||
-cert deploy/certs/admin-cert.pem \
|
||||
-key deploy/certs/admin-key.pem \
|
||||
-d '{}' \
|
||||
localhost:50051 charybdis.entities.EntityService/ListEntities
|
||||
```
|
||||
|
||||
See the [Security Guide](security.md) for detailed configuration.
|
||||
Full reference (custom role mappings, audit logging, reverse-proxy mode): [security.md](security.md).
|
||||
|
||||
## Backstage Migration (Optional)
|
||||
|
||||
If you currently use Backstage and want to migrate gradually, the built-in YAML adapter serves entities in Backstage format:
|
||||
If you currently run Backstage, Charybdis serves entities in Backstage's Location YAML format:
|
||||
|
||||
```yaml
|
||||
# backstage app-config.yaml
|
||||
@@ -367,78 +239,36 @@ catalog:
|
||||
- type: url
|
||||
target: http://your-charybdis-host:8080/yaml/locations
|
||||
rules:
|
||||
- allow: [Component, System, Service]
|
||||
- allow: [Component, System, API, User, Group]
|
||||
```
|
||||
|
||||
Backstage will automatically discover and import entities from Charybdis. You can run both in parallel — entities managed via gRPC are immediately visible in Backstage.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```
|
||||
Error: transport error
|
||||
```
|
||||
|
||||
**Solution**: Check if another process is using port 50051:
|
||||
|
||||
```bash
|
||||
lsof -ti:50051
|
||||
```
|
||||
|
||||
Kill the process or change the port:
|
||||
|
||||
```bash
|
||||
export GRPC_PORT=50052
|
||||
```
|
||||
|
||||
### Database Connection Failed
|
||||
|
||||
```
|
||||
Error: password authentication failed
|
||||
```
|
||||
|
||||
**Solution**: Verify your DATABASE_URL:
|
||||
|
||||
```bash
|
||||
# Test connection
|
||||
psql "$DATABASE_URL" -c "SELECT 1;"
|
||||
```
|
||||
|
||||
### Permission Denied (with security enabled)
|
||||
|
||||
```
|
||||
Code: PermissionDenied
|
||||
Message: Role 'X' does not have permission 'Y'
|
||||
```
|
||||
|
||||
**Solution**: Check your certificate and role mappings. See [Security Guide](security.md).
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you have Charybdis running:
|
||||
|
||||
1. Learn about [Core Concepts](core-concepts.md) — entities, vulnerabilities, events
|
||||
2. Read the [Vision & Roadmap](../VISION.md) — where Charybdis is going
|
||||
3. Configure [Security](security.md) for production (mTLS + RBAC)
|
||||
4. Explore [Plugins](../plugins/README.md) for external integrations
|
||||
Backstage discovers entities by polling the endpoint. Entities created via gRPC are visible on the next poll. The YAML adapter is served by an HTTP listener separate from the gRPC port (default `:8080`).
|
||||
|
||||
## Configuration
|
||||
|
||||
Charybdis supports two configuration methods:
|
||||
Charybdis loads its config from (in order):
|
||||
|
||||
### 1. Configuration File (Recommended)
|
||||
1. `./config.toml`
|
||||
2. `./charybdis.toml`
|
||||
3. `/etc/charybdis/config.toml`
|
||||
4. Environment variables (fallback)
|
||||
|
||||
Use `config.toml` for structured configuration:
|
||||
### `config.toml` skeleton
|
||||
|
||||
```toml
|
||||
# config.toml
|
||||
[server]
|
||||
grpc_host = "[::1]"
|
||||
grpc_port = 50051
|
||||
|
||||
[server.yaml_adapter]
|
||||
enabled = true
|
||||
host = "0.0.0.0"
|
||||
port = 8080
|
||||
|
||||
[database]
|
||||
url = "${DATABASE_URL}" # Environment variable substitution
|
||||
url = "${DATABASE_URL}"
|
||||
max_connections = 10
|
||||
connection_timeout_secs = 30
|
||||
|
||||
[security.mtls]
|
||||
enabled = false
|
||||
@@ -450,59 +280,52 @@ enabled = false
|
||||
service_name = "charybdis"
|
||||
environment = "development"
|
||||
enable_console = true
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Organized by section (server, database, security, telemetry, plugins)
|
||||
- ✅ Environment variable substitution with `${VAR_NAME}`
|
||||
- ✅ Comments and documentation inline
|
||||
- ✅ Easy to version control (excluding secrets)
|
||||
- ✅ No need to export dozens of environment variables
|
||||
|
||||
**Using Environment Variables in config.toml:**
|
||||
|
||||
```toml
|
||||
[database]
|
||||
url = "${DATABASE_URL}" # Will be substituted at runtime
|
||||
|
||||
[plugins.defectdojo]
|
||||
api_key = "${DEFECTDOJO_API_KEY}" # Secrets stay in environment
|
||||
enabled = false
|
||||
# see plugins/README.md for the full plugin reference
|
||||
```
|
||||
|
||||
Then set only the secrets:
|
||||
`${VAR}` and `${VAR:-default}` substitution works in any string value — keep secrets in the environment, not in the file.
|
||||
|
||||
```bash
|
||||
export DATABASE_URL="postgresql://..."
|
||||
export DEFECTDOJO_API_KEY="secret-key"
|
||||
```
|
||||
### Environment-variable fallback
|
||||
|
||||
### 2. Environment Variables (Legacy)
|
||||
|
||||
If `config.toml` is not found, Charybdis falls back to environment variables:
|
||||
If no config file is found, these env vars are read:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | (required) | PostgreSQL connection string |
|
||||
| `GRPC_HOST` | `[::1]` | gRPC server bind address |
|
||||
| `GRPC_PORT` | `50051` | gRPC server port |
|
||||
| `RUST_LOG` | `info` | Logging level |
|
||||
| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS authentication |
|
||||
| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC authorization |
|
||||
| `OTEL_ENABLE_CONSOLE` | `true` | Enable console logging |
|
||||
| `GRPC_HOST` | `[::1]` | gRPC bind address |
|
||||
| `GRPC_PORT` | `50051` | gRPC port |
|
||||
| `RUST_LOG` | `info` | Logging level filter |
|
||||
| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS |
|
||||
| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC |
|
||||
| `OTEL_ENABLE_CONSOLE` | `true` | Console exporter |
|
||||
| `OTEL_SERVICE_NAME` | `charybdis` | Service name for telemetry |
|
||||
|
||||
### Configuration File Locations
|
||||
See `config.toml.example` for the complete template.
|
||||
|
||||
Charybdis looks for configuration files in this order:
|
||||
## Troubleshooting
|
||||
|
||||
1. `./config.toml` (current directory)
|
||||
2. `./charybdis.toml`
|
||||
3. `/etc/charybdis/config.toml` (Linux/Unix)
|
||||
### Port already in use
|
||||
```
|
||||
Error: transport error
|
||||
```
|
||||
Find and free port 50051: `lsof -ti:50051 | xargs kill`, or set `GRPC_PORT=50052`.
|
||||
|
||||
If none are found, it uses environment variables.
|
||||
### Database connection failed
|
||||
Verify the URL: `psql "$DATABASE_URL" -c "SELECT 1;"`.
|
||||
|
||||
See `config.toml.example` for a complete configuration template with all options documented.
|
||||
### Permission denied with security enabled
|
||||
```
|
||||
Code: PermissionDenied
|
||||
Message: Role 'X' does not have permission 'Y'
|
||||
```
|
||||
Inspect your cert subject (OU determines the role) and the `[security.rbac.permissions]` table in `config.toml`. Reference: [security.md](security.md).
|
||||
|
||||
---
|
||||
## Next Steps
|
||||
|
||||
**Need help?** Check the [troubleshooting guide](troubleshooting.md) or [open an issue](../../issues).
|
||||
- [Core Concepts](core-concepts.md) — entity model, events, annotations
|
||||
- [Architecture](architecture.md) — protobuf schema, storage, event bus
|
||||
- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
|
||||
- [Vision & Roadmap](../VISION.md) — where Charybdis is going
|
||||
|
||||
-125
@@ -1,125 +0,0 @@
|
||||
# Charybdis Documentation
|
||||
|
||||
**The security-native platform engineering tool.**
|
||||
|
||||
## What is Charybdis?
|
||||
|
||||
Charybdis is a platform engineering tool that unifies **software catalog**, **vulnerability management**, and **compliance posture** in a single event-driven platform. Built in Rust, deployed as a single binary.
|
||||
|
||||
Instead of running Backstage + DefectDojo + Dependency-Track + a license scanner + a compliance spreadsheet, you run Charybdis.
|
||||
|
||||
### The Problems It Solves
|
||||
|
||||
1. **Fragmented tooling** — Your software catalog, vulnerability data, license info, and compliance evidence live in 5 different tools that don't talk to each other. Charybdis unifies them.
|
||||
|
||||
2. **Static catalog data** — Traditional catalogs rely on YAML files that go stale within weeks. Charybdis is event-driven — CI/CD pipelines and IaC tools register and update entities via gRPC, so the catalog is always accurate.
|
||||
|
||||
3. **Security as an afterthought** — In Backstage, security is a plugin. In Charybdis, every entity carries its vulnerability posture, license status, and compliance state natively.
|
||||
|
||||
4. **Manual provisioning** — New service? Manually create entries in every tool. With Charybdis, one gRPC call catalogs the service and event-driven plugins handle the rest.
|
||||
|
||||
5. **Compliance evidence assembly** — Compliance reporting pulls from real vulnerability and license data, not spreadsheets.
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
CI/CD or Scanner ──gRPC──> Charybdis
|
||||
├── Catalogs the service (event-driven) ← Done
|
||||
├── Fires events to plugins (DefectDojo, ...) ← Done
|
||||
├── Ingests scan results (SARIF, CycloneDX) ← Phase 1
|
||||
├── Evaluates security gates & rules ← Phase 1
|
||||
└── Exposes catalog via YAML adapter (Backstage) ← Done
|
||||
```
|
||||
|
||||
**One platform. Your services are cataloged. Your vulns are tracked. Your compliance is visible. In real-time.**
|
||||
|
||||
## Key Concepts
|
||||
|
||||
| Concept | Description |
|
||||
|---------|-------------|
|
||||
| **Entity** | Anything in your software ecosystem: services, systems, components, APIs, users, groups, domains, resources |
|
||||
| **Vulnerability** | *(Phase 1)* A security finding linked to an entity, ingested from scanner output (SARIF, CycloneDX) |
|
||||
| **Assessment** | *(Phase 1)* The triage decision on a vulnerability: accept risk, remediate, auto-assessed by rules |
|
||||
| **Security Gate** | *(Phase 1)* Severity thresholds per product — blocks deployments when violated |
|
||||
| **Event Bus** | Publishes lifecycle events when entities or vulnerabilities change |
|
||||
| **Plugin** | Reacts to events to integrate with external systems (Slack, Jira, GitHub, custom) |
|
||||
| **Annotations** | Key-value metadata on entities for external references (e.g. `github.com/repo-slug`) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[CI/CD / Scanners] -->|gRPC| B[Charybdis]
|
||||
B -->|Native| C[Software Catalog]
|
||||
B -->|Native| D[Vuln Management]
|
||||
B -->|Native| E[Compliance]
|
||||
B -->|Events| F[Plugins: Slack / Jira / Custom]
|
||||
B -->|YAML| G[Backstage - optional]
|
||||
|
||||
style B fill:#4A90E2,stroke:#2E5C8A,color:#fff
|
||||
style D fill:#E24A4A,stroke:#8A2E2E,color:#fff
|
||||
style E fill:#4AE28A,stroke:#2E8A5C,color:#fff
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
### Getting Started
|
||||
- [Installation & Quick Start](getting-started.md) — Get Charybdis running and register your first entity
|
||||
- [Demo Stack](../deploy/DEMO.md) — Full demo with DefectDojo
|
||||
|
||||
### Understanding Charybdis
|
||||
- [Core Concepts](core-concepts.md) — Entities, vulnerabilities, events, and data model
|
||||
- [Vision & Roadmap](../VISION.md) — Where Charybdis is going and why
|
||||
- [Architecture](architecture.md) — Technical design decisions
|
||||
|
||||
### Configuration
|
||||
- [Security](security.md) — mTLS authentication and RBAC authorization
|
||||
- [Plugin Configuration](PLUGIN_CONFIGURATION_GUIDE.md) — Setting up and configuring plugins
|
||||
|
||||
### Extending Charybdis
|
||||
- [Plugin Development](../plugins/README.md) — Build your own integration plugins
|
||||
|
||||
## Quick Example
|
||||
|
||||
Register a service from your CI/CD pipeline:
|
||||
|
||||
```bash
|
||||
grpcurl -plaintext -d '{
|
||||
"entity": {
|
||||
"kind": "Component",
|
||||
"component_metadata": {
|
||||
"name": "payment-api",
|
||||
"description": "Payment processing service"
|
||||
},
|
||||
"component_spec": {
|
||||
"type": "service",
|
||||
"lifecycle": "production",
|
||||
"owner": "team-payments"
|
||||
}
|
||||
}
|
||||
}' charybdis:50051 charybdis.entities.EntityService/CreateEntity
|
||||
```
|
||||
|
||||
**What happens next:**
|
||||
- Entity stored in PostgreSQL with a UUID
|
||||
- `EntityCreated` event published to the event bus
|
||||
- Plugins react (e.g., DefectDojo creates a product automatically)
|
||||
- Entity available via gRPC and YAML adapter
|
||||
|
||||
No YAML file to write. No PR to open. No manual provisioning.
|
||||
|
||||
## Backstage Migration
|
||||
|
||||
Already using Backstage? Charybdis provides a YAML adapter for gradual migration. Point Backstage at Charybdis as a catalog source — entities registered via gRPC are immediately available in Backstage.
|
||||
|
||||
```yaml
|
||||
# backstage app-config.yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: url
|
||||
target: http://charybdis:8080/yaml/locations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Ready to get started?** Head to the [Getting Started Guide](getting-started.md).
|
||||
+12
-11
@@ -67,17 +67,19 @@ export SECURITY_RBAC_ENABLED=true
|
||||
|
||||
#### Development Certificates
|
||||
|
||||
Use the provided test script:
|
||||
Use the provided dev-cert script:
|
||||
|
||||
```bash
|
||||
./test-mtls-rbac.sh
|
||||
./deploy/scripts/generate-dev-certs.sh
|
||||
```
|
||||
|
||||
This generates:
|
||||
- `ca.pem` / `ca-key.pem` - Certificate Authority
|
||||
- `server-cert.pem` / `server-key.pem` - Server certificate
|
||||
- `admin-cert.pem` / `admin-key.pem` - Admin client (OU=platform-team)
|
||||
- Various plugin certificates (OU=plugins)
|
||||
This generates (under `deploy/certs/`):
|
||||
- `ca.pem` / `ca-key.pem` — Certificate Authority
|
||||
- `server-cert.pem` / `server-key.pem` — Server certificate
|
||||
- `admin-cert.pem` / `admin-key.pem` — Admin client (OU=platform-team)
|
||||
- Per-role client certs (OU=automation, OU=plugins)
|
||||
|
||||
For an individual client cert without regenerating everything, use `./deploy/scripts/generate-client-cert.sh`.
|
||||
|
||||
#### Production Certificates
|
||||
|
||||
@@ -494,10 +496,9 @@ Charybdis security features support compliance requirements:
|
||||
|
||||
## Next Steps
|
||||
|
||||
- 🔌 Configure [Plugins](plugins.md) with proper certificates
|
||||
- 🚀 Review [Deployment Guide](deployment.md) for production
|
||||
- 📊 Set up [Monitoring](monitoring.md) for security events
|
||||
- [Plugins](../plugins/README.md) — configure plugins to use mTLS client certs
|
||||
- [Architecture](architecture.md) — interceptor placement and observability hooks
|
||||
|
||||
---
|
||||
|
||||
**Security Questions?** Open a [security issue](../../security) (for vulnerabilities, use private disclosure).
|
||||
**Security Questions?** Open an issue (for vulnerabilities, use private disclosure).
|
||||
|
||||
Reference in New Issue
Block a user