doc: update and cleanup
CI / Check (push) Successful in 13m13s
CI / Format (push) Successful in 48s
CI / Clippy (push) Successful in 12m13s
CI / Test (push) Successful in 12m45s

This commit is contained in:
Guillaume GRABÉ
2026-06-09 11:26:47 +02:00
parent ab13af48e7
commit f1b1543f29
11 changed files with 1113 additions and 3376 deletions
+461 -288
View File
@@ -1,340 +1,513 @@
# Charybdis Plugins
This directory contains plugin implementations that extend Charybdis with integrations to external security and development tools.
Plugins extend Charybdis with integrations to external tools (security scanners, issue trackers, identity providers). Plugins are **compile-time integrated**: they live as Rust crates under `plugins/`, declare protobuf extensions in `plugins.toml`, and are linked into the `charybdis-server` binary at build time.
## What are Plugins?
This document covers:
- The plugin model (event-driven vs sync, traits, lifecycle)
- Configuring shipped plugins (DefectDojo, Keycloak)
- Writing a new plugin
Charybdis supports **two types of plugins**, both compile-time integrated:
## Table of Contents
### **1. Event-Driven Plugins**
React to entity lifecycle events (create, update, delete):
- **Example**: DefectDojo, DependencyTrack
- **Implement**: `EventDrivenPlugin` trait + `ResourceHandler`
- **Triggered by**: Entity CRUD operations
- **Use case**: Auto-create resources in external tools when entities are created
1. [Plugin Types](#plugin-types)
2. [Available Plugins](#available-plugins)
3. [Plugin Configuration](#plugin-configuration)
4. [Field Mapping System](#field-mapping-system)
5. [Entity Resolution](#entity-resolution)
6. [DefectDojo Reference](#defectdojo-reference)
7. [Keycloak Reference](#keycloak-reference)
8. [Writing a New Plugin](#writing-a-new-plugin)
9. [Plugin Lifecycle](#plugin-lifecycle)
10. [Annotations](#annotations)
11. [Troubleshooting](#troubleshooting)
### **2. Sync Plugins**
Pull data from external sources on a schedule:
- **Example**: Okta, Keycloak, Active Directory
- **Implement**: `SyncPlugin` trait
- **Triggered by**: Cron schedule or manual API call
- **Use case**: Sync users/groups from identity providers
## Plugin Types
## Generic Utilities
### Event-Driven Plugins
React to entity lifecycle events (`Created`, `Updated`, `Deleted`) — typically push data to an external system.
All plugins have access to reusable utilities in `src/plugins/`:
- Implement `EventDrivenPlugin` + one or more `ResourceHandler` per resource kind.
- Example: **DefectDojo** — Component creation → DefectDojo product + engagement.
- **`PluginHttpClient`** - Multi-auth HTTP client (Token, Bearer, API Key, Basic Auth)
- **`AnnotationHelper`** - Consistent API for storing/retrieving plugin metadata
- **`FieldMapper`** - Maps entity fields to external tool formats
- **`DateUtils`** - Common date/time operations
### Sync Plugins
Pull data from an external system on a schedule, creating or updating entities in Charybdis.
## Plugin Structure
- Implement `SyncPlugin` (`schedule`, `on_startup`, `manual_trigger`).
- Example: **Keycloak** — periodic sync of users and groups.
Each plugin follows this structure:
```
plugins/
└── my_plugin/
├── Cargo.toml # Plugin crate definition
├── README.md # Plugin-specific documentation
├── proto/
│ └── my_plugin.proto # Protobuf entity definitions
└── src/
├── lib.rs # Plugin entry point
├── handler.rs # Event handler implementation
├── client.rs # External API client
└── config.rs # Plugin configuration
```
Both types implement the base `Plugin` trait (`name`, `validate_config`, `health_check`).
## Available Plugins
### DefectDojo
- **Type**: Event-Driven
- **Purpose**: Security vulnerability management integration
- **External API**: DefectDojo REST API v2
- **Handlers**: ProductHandler, EngagementHandler, ProductTypeHandler, ProductMemberHandler
- **Features**:
- Automatic product creation on Component/Service creation
- Product updates on entity changes
- Auto-create CI/CD engagements
- Owner resolution (Group → members → email annotation → DefectDojo user lookup)
- Configurable OIDC provider mapping (Keycloak, Okta, Azure AD)
- Annotation storage (`defectdojo.com/product-id`, `defectdojo.com/engagement-id`, `defectdojo.com/owner-member-ids`)
| Plugin | Type | Status |
|---|---|---|
| DefectDojo | Event-driven | Shipped |
| Keycloak | Sync | Shipped |
### Keycloak
- **Type**: Sync
- **Purpose**: Identity provider sync (users and groups)
- **External API**: Keycloak Admin REST API
- **Features**:
- User sync with profile and annotations (display_name, email, picture)
- Group sync with hierarchy (parent, children, members)
- Configurable annotations (e.g., `keycloak.com/email` for OIDC mapping)
- On-startup sync, manual trigger via gRPC
### DependencyTrack (Scaffolded)
- **Type**: Event-Driven
- **Purpose**: Software supply chain security
- **External API**: DependencyTrack REST API
- **Status**: Scaffolded (proto + crate structure), handler logic not implemented
- **Planned**:
- Auto-create projects for components
- SBOM ingestion
## Creating a New Plugin
### Quick Start
1. **Create plugin directory structure:**
```bash
mkdir -p plugins/my_plugin/{proto,src}
```
2. **Add to plugins.toml:**
```toml
[plugins.my_plugin]
enabled = true
proto_path = "plugins/my_plugin/proto"
metadata_field_number = 102 # Use next available number
spec_field_number = 102
description = "My custom integration"
```
3. **Define protobuf schema:**
Create `plugins/my_plugin/proto/my_plugin.proto`:
```protobuf
syntax = "proto3";
package charybdis.plugins.my_plugin;
message MyPluginMetadata {
string name = 1;
string description = 2;
}
message MyPluginSpec {
string integration_type = 1;
bool enabled = 2;
}
```
4. **Create plugin crate:**
Create `plugins/my_plugin/Cargo.toml`:
```toml
[package]
name = "charybdis-plugin-my-plugin"
version = "0.1.0"
edition = "2021"
[dependencies]
charybdis = { path = "../.." }
async-trait = "0.1"
tokio = { version = "1.0", features = ["full"] }
tracing = "0.1"
```
5. **Implement event handler:**
Create `plugins/my_plugin/src/lib.rs`:
```rust
use async_trait::async_trait;
use charybdis::events::{EventHandler, EntityEvent, EventResult};
pub struct MyPluginHandler;
#[async_trait]
impl EventHandler for MyPluginHandler {
async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
// React to entity changes
Ok(())
}
}
```
6. **Build and test:**
```bash
cargo build
cargo test
```
## Plugin Field Number Allocation
Field numbers must be unique across all plugins to avoid protobuf conflicts:
| Range | Allocation |
|----------|-----------------------|
| 1-99 | Core entity types |
| 100 | DefectDojo |
| 101 | DependencyTrack |
| 102-199 | Available for plugins |
When creating a new plugin, use the next available number in the 102+ range.
Planned integrations (Slack, Jira, GitHub, Dependency-Track) are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).
## Plugin Configuration
Plugins are configured in `config.toml` with `${VAR}` env var substitution for secrets:
Plugins are configured in `config.toml` under `[plugins.<name>]`. Use `${VAR}` for secrets — never commit tokens.
```toml
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_API_URL}"
base_url = "${DEFECTDOJO_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"
[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
[plugins.keycloak]
enabled = true
base_url = "${KEYCLOAK_URL}"
realm = "master"
realm = "charybdis"
client_id = "charybdis-sync"
client_secret = "${KEYCLOAK_CLIENT_SECRET}"
```
See `config.toml.example` for all options.
When `enabled = false`, the plugin is loaded but does not register handlers and does not react to events. See `config.toml.example` for the full reference.
## Field Mapping System
Event-driven plugins (currently DefectDojo) use a configurable mapper to translate Charybdis entity fields into external-tool API payloads. Mappings are defined per resource type under `[plugins.<name>.field_mappings.<resource>]`.
Three mapping types are supported:
### 1. Direct field mapping (string)
Dot-notation path into the entity.
```toml
name = "metadata.name"
description = "metadata.description"
email = "spec.profile.email"
tags = "metadata.tags"
```
### 2. Static value (object with `value` key)
```toml
business_criticality = { value = "high" }
is_active = { value = true }
priority = { value = 100 }
```
Supports strings, booleans, numbers, objects, and arrays.
### 3. Entity resolution (object)
Resolve a reference into a related entity and extract a field from it.
```toml
product_manager = {
from = "spec.owner",
resolve_entity = "User",
extract = "annotations.defectdojo.com/user-id"
}
```
| Parameter | Type | Required | Description |
|---|---|---|---|
| `from` | string | yes | Source field path |
| `resolve_entity` | string | yes | Entity kind to look up (`User`, `Component`, ...) |
| `extract` | string | yes | Field path to extract from the resolved entity |
| `resolve_array` | bool | no | Source field is an array of references |
| `lookup_entity` | string | no | Find an entity that references the source |
## Entity Resolution
Entity resolution lets a plugin navigate the entity graph at mapping time. The mapper performs a database lookup, extracts the target field, and substitutes it into the payload.
```
1. Source field spec.owner = "user:john.doe"
2. Resolve entity User entity for "john.doe"
3. Extract target field annotations.defectdojo.com/user-id = "123"
4. Result product_manager = 123
```
### Array resolution
```toml
user_ids = {
from = "spec.members",
resolve_entity = "User",
extract = "annotations.defectdojo.com/user-id",
resolve_array = true
}
```
`Group.spec.members = ["user:john", "user:jane"]``user_ids = [123, 124]`.
### Linked entity lookup
```toml
engagement_product = {
from = "id",
lookup_entity = "Component",
extract = "annotations.defectdojo.com/product-id"
}
```
Finds a `Component` that references the source entity and extracts its annotation.
### Error handling
Resolution failures (entity not found, missing annotation, invalid reference) log a warning and omit the field rather than failing the whole operation.
## DefectDojo Reference
Maps Charybdis entities to DefectDojo resources via the REST API v2. Handlers shipped: `ProductHandler`, `EngagementHandler`, `ProductTypeHandler`, `ProductMemberHandler`.
### Connection
```toml
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"
default_product_type_id = 1
auto_create_users = true
auto_create_product_types = false
```
To obtain the API token: in DefectDojo, **User Profile → API Key**.
### Default engagement (auto-created with each product)
```toml
[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
description = "Automated security scans from CI/CD pipeline"
engagement_type = "CI/CD"
status = "In Progress"
duration_days = 365
deduplication_on_engagement = true
```
### Owner resolution
When a `Component` has an `owner` (typically a team name), Charybdis resolves it to DefectDojo users:
1. Find `Group` entity matching the owner.
2. Read `Group.spec.members` (usernames).
3. Look up each `User` entity.
4. Extract the user's email from the configured annotation.
5. Find the matching user in DefectDojo by email.
```toml
[plugins.defectdojo.owner_resolution]
user_email_annotation = "keycloak.com/email" # depends on your IdP
defectdojo_lookup_field = "email" # "email" or "username"
assign_all_members = true # all group members, or just first match
```
### Product mapping (Component → DefectDojo Product)
```toml
[plugins.defectdojo.field_mappings.product]
name = "metadata.name"
description = "metadata.description"
product_type_id = { value = 1 }
tags = "metadata.tags"
business_criticality = { value = "high" } # very high | high | medium | low | very low | none
platform = { value = "web" } # web | mobile | desktop | iot | ...
lifecycle = { value = "production" }
origin = { value = "internal" }
external_audience = { value = true }
internet_accessible = { value = true }
product_manager = {
from = "spec.owner",
resolve_entity = "User",
extract = "annotations.defectdojo.com/user-id"
}
```
### User mapping (User → DefectDojo User)
```toml
[plugins.defectdojo.field_mappings.user]
username = "metadata.name"
email = "spec.profile.email"
first_name = "spec.profile.displayName"
last_name = "spec.profile.displayName"
is_active = { value = true }
```
### Product Type mapping (System → DefectDojo Product Type)
```toml
[plugins.defectdojo.field_mappings.product_type]
name = "metadata.name"
description = "metadata.description"
critical_product = { value = false }
key_product = { value = true }
```
### Product Member mapping (Group → DefectDojo Product Member)
```toml
[plugins.defectdojo.field_mappings.product_member]
product_id = {
from = "spec.parent",
resolve_entity = "Component",
extract = "annotations.defectdojo.com/product-id"
}
user_id = {
from = "spec.members",
resolve_entity = "User",
extract = "annotations.defectdojo.com/user-id",
resolve_array = true
}
role_name = { value = "Reader" } # Owner | Maintainer | Writer | Reader | API_Importer
```
### Engagement mapping (Resource → DefectDojo Engagement)
```toml
[plugins.defectdojo.field_mappings.engagement]
name = "metadata.name"
description = "metadata.description"
product_id = {
from = "spec.owner",
resolve_entity = "Component",
extract = "annotations.defectdojo.com/product-id"
}
lead_id = {
from = "spec.dependsOn",
resolve_entity = "User",
extract = "annotations.defectdojo.com/user-id"
}
target_start = "spec.target_start"
target_end = "spec.target_end"
status = { value = "In Progress" }
engagement_type = { value = "CI/CD" }
version = "metadata.annotations.version"
commit_hash = "metadata.annotations.commit_hash"
branch_tag = "metadata.annotations.branch"
build_id = "metadata.annotations.build_id"
source_code_management_uri = "metadata.annotations.repo_url"
deduplication_on_engagement = { value = true }
api_test = { value = true }
pen_test = { value = false }
```
### Annotations written by the plugin
- `defectdojo.com/product-id`
- `defectdojo.com/engagement-id`
- `defectdojo.com/owner-member-ids`
## Keycloak Reference
Sync plugin: pulls users and groups from a Keycloak realm via the Admin REST API on a schedule, on startup, or via manual trigger.
```toml
[plugins.keycloak]
enabled = true
base_url = "${KEYCLOAK_URL}"
realm = "charybdis"
client_id = "charybdis-sync"
client_secret = "${KEYCLOAK_CLIENT_SECRET}"
[plugins.keycloak.sync]
schedule = "0 */5 * * * *" # optional cron; omit to sync only on startup/trigger
on_startup = true
manual_trigger = true
sync_users = true
sync_groups = true
namespace = "keycloak"
page_size = 100
```
The service account behind `client_id`/`client_secret` must have `view-users` and `view-groups` on the realm.
Synced entities carry annotations such as `keycloak.com/email`, `keycloak.com/id`, used by other plugins (e.g., the DefectDojo owner-resolution chain above).
## Writing a New Plugin
### 1. Scaffold directory
```bash
mkdir -p plugins/my_plugin/{proto,src}
```
### 2. Register in `plugins.toml`
```toml
[plugins.my_plugin]
enabled = true
proto_path = "plugins/my_plugin/proto"
metadata_field_number = 103 # next available — see registry below
spec_field_number = 203
description = "My custom integration"
```
### 3. Define the protobuf schema
`plugins/my_plugin/proto/my_plugin.proto`:
```protobuf
syntax = "proto3";
package charybdis.plugins.my_plugin;
message MyPluginMetadata {
string name = 1;
string description = 2;
}
message MyPluginSpec {
string integration_type = 1;
bool enabled = 2;
}
```
`build.rs` regenerates `proto/entities.proto` automatically on next build.
### 4. Create the crate
`plugins/my_plugin/Cargo.toml`:
```toml
[package]
name = "charybdis-my-plugin-plugin"
version = "0.1.0"
edition = "2024"
[lib]
name = "charybdis_my_plugin"
path = "src/lib.rs"
[dependencies]
charybdis = { path = "../.." }
async-trait = "0.1"
anyhow = "1.0"
serde = { version = "1.0", features = ["derive"] }
tokio = { version = "1.48", features = ["full"] }
tracing = "0.1"
```
Add the crate to the workspace `members` in the top-level `Cargo.toml`.
### 5. Implement the plugin
Event-driven example:
```rust
use async_trait::async_trait;
use charybdis::plugins::{EventDrivenPlugin, Plugin, PluginConfig, PluginType, ResourceHandler};
use std::sync::Arc;
pub struct MyPlugin { /* config, client, handlers */ }
#[async_trait]
impl Plugin for MyPlugin {
fn name(&self) -> &str { "my_plugin" }
fn plugin_type(&self) -> PluginType { PluginType::EventDriven }
fn load_config(&mut self, _: PluginConfig) -> anyhow::Result<()> { Ok(()) }
fn validate_config(&self) -> anyhow::Result<()> { Ok(()) }
}
#[async_trait]
impl EventDrivenPlugin for MyPlugin {
fn resource_handlers(&self) -> Vec<Arc<dyn ResourceHandler>> {
vec![/* your handlers */]
}
}
```
### 6. Register in `charybdis-server`
Add a loader block in `charybdis-server/src/main.rs` (follow the DefectDojo block as a template) and declare the dependency in `charybdis-server/Cargo.toml`.
### Field number registry
| Range | Allocation |
|---|---|
| 199 | Core entity types |
| 100 | DefectDojo |
| 101 | DependencyTrack |
| 102 | Keycloak |
| 103+ | Available |
Field numbers MUST be unique across plugins — protobuf wire format depends on it.
## Plugin Lifecycle
1. **Build Time:**
- `build.rs` reads `plugins.toml`
- Generates `proto/entities.proto` with plugin types
- Compiles all protobuf files
- Generates Rust code
### Build time
1. `build.rs` reads `plugins.toml`.
2. Generates `proto/entities.proto` with plugin-contributed `oneof` variants.
3. Compiles all protobuf files (`tonic-prost-build`).
2. **Runtime:**
- Plugin handlers registered with event bus
- Events published on entity CRUD operations
- Handlers react asynchronously
- External APIs called as needed
### Runtime
1. `charybdis-server` loads each enabled plugin from `config.toml`.
2. Event-driven plugins register their `ResourceHandler`s with the dispatcher.
3. Sync plugins register with the cron scheduler (and optionally run once at startup).
4. Entity CRUD operations publish `EntityEvent` to the in-memory event bus.
5. The dispatcher fans out events to matching handlers (filtered by `trigger_kinds`).
6. Handlers call external APIs and write annotations back to Charybdis.
3. **Event Flow:**
```
Client creates entity
→ Entity stored in database
→ Event published to event bus
→ Plugin handler receives event
→ Plugin calls external API
→ Plugin stores external ID in annotations
```
## Using Annotations
Plugins store external tool IDs in entity annotations:
```rust
// Store external ID
entity.annotations.insert(
"my-plugin.com/resource-id".to_string(),
"ext-12345".to_string(),
);
// Query by external ID in SQL
SELECT * FROM entities
WHERE annotations->>'my-plugin.com/resource-id' = 'ext-12345';
### Event flow
```
Client → gRPC CreateEntity → DB INSERT → EntityEvent::Created → Dispatcher
ResourceHandler.handle_create(entity)
POST /api/products → DefectDojo
update_annotations("defectdojo.com/product-id")
```
### Annotation Naming Convention
## Annotations
Plugins store external IDs in entity annotations using reverse-DNS keys:
Use reverse-DNS style:
- `defectdojo.com/product-id`
- `dependencytrack.com/project-uuid`
- `github.com/repo-slug`
- `{tool}.com/{resource}-{attribute}`
- `defectdojo.com/engagement-id`
- `keycloak.com/id`
- `keycloak.com/email`
## Best Practices
Query by annotation via JSONB in PostgreSQL:
### 1. Error Handling
- Don't panic - return errors
- Log but continue on non-critical failures
- Implement retry logic for transient errors
### 2. Idempotency
- Check if resource exists before creating
- Make operations safe to retry
- Handle duplicate creation gracefully
### 3. Performance
- Don't block event handlers
- Use `tokio::spawn` for long operations
- Batch operations when possible
### 4. Testing
- Unit test event handlers
- Mock external API clients
- Integration tests with real APIs (optional)
### 5. Documentation
- Document required environment variables
- Provide example configurations
- Explain entity model and annotations
## Contributing Plugins
We welcome plugin contributions! To contribute:
1. Fork the repository
2. Create your plugin following the structure above
3. Add comprehensive tests
4. Document configuration and usage
5. Submit a pull request
### Plugin Requirements
- [ ] Protobuf definitions with Metadata and Spec messages
- [ ] Event handler implementation
- [ ] External API client (if applicable)
- [ ] Configuration via environment variables
- [ ] README with setup instructions
- [ ] Unit tests for event handler
- [ ] Example usage in documentation
## Plugin Distribution Models
### In-Tree (Current)
Plugins live in the `plugins/` directory and are enabled via `plugins.toml`.
**Pros:**
- Easy to discover
- Consistent quality
- Tested together
**Cons:**
- Requires core repo access
- All plugins built together
### External Crates (Future)
Plugins distributed as separate Rust crates.
**Example:**
```toml
[dependencies]
charybdis-plugin-custom = "0.1"
```sql
SELECT * FROM entities
WHERE annotations->>'defectdojo.com/product-id' = '456';
```
**Pros:**
- Independent versioning
- Community contributions
- Optional dependencies
Use `EntityRepository::update_annotations()` for atomic JSONB merge — it avoids the read-modify-write race when multiple handlers write to the same entity.
**Cons:**
- Discovery harder
- Compatibility challenges
## Troubleshooting
### Plugin Marketplace (Future)
Central registry of available plugins (like Backstage).
### Plugin not reacting to events
- `enabled = true` in `config.toml`?
- Plugin registered in `charybdis-server/src/main.rs`?
- Entity kind matches the handler's `trigger_kinds`?
- Event bus running? (`info!("Event bus started")` in startup logs)
## Resources
### Field mapping not applied
- Path uses dot notation, e.g. `metadata.name`, not `meta.name`.
- Static value uses `{ value = "..." }`, not a bare string when you want a literal.
- Field actually exists on the entity (inspect via the YAML adapter or `GetEntity` gRPC).
- [Plugin Configuration Guide](../docs/PLUGIN_CONFIGURATION_GUIDE.md)
- [Core Concepts](../docs/core-concepts.md)
- [Architecture](../docs/architecture.md)
- [Protobuf Style Guide](https://protobuf.dev/programming-guides/style/)
### Entity resolution returns nothing
- Resolved entity exists in the DB.
- Resolved entity has the `extract` annotation set.
- `resolve_entity` uses the PascalCase kind (`"User"`, not `"user"`).
- For arrays, `resolve_array = true` is set.
## Support
### DefectDojo API errors
- `400 Bad Request: Invalid product type ID` → check `default_product_type_id` or your `product_type_id` mapping.
- `401 Unauthorized` → check `DEFECTDOJO_API_TOKEN`.
- `409 Conflict: user already exists` → expected, the plugin reuses existing users.
- Open an issue for bug reports
- Discussions for questions
- PRs for contributions
### Performance
- Minimize entity-resolution chain depth.
- Prefer static values when the data isn't entity-bound.
- Check network latency to the external API — handlers run synchronously per event.
## License
## See Also
Same as Charybdis core (see LICENSE file in repository root)
- [Core Concepts](../docs/core-concepts.md) — entity model, events, annotations.
- [Architecture](../docs/architecture.md) — protobuf schema, storage, scaling.
- [Field Mapper Tests](../src/plugins/field_mapper.rs) — reference behavior.