Public Access
doc: update and cleanup
This commit is contained in:
+461
-288
@@ -1,340 +1,513 @@
|
||||
# Charybdis Plugins
|
||||
|
||||
This directory contains plugin implementations that extend Charybdis with integrations to external security and development tools.
|
||||
Plugins extend Charybdis with integrations to external tools (security scanners, issue trackers, identity providers). Plugins are **compile-time integrated**: they live as Rust crates under `plugins/`, declare protobuf extensions in `plugins.toml`, and are linked into the `charybdis-server` binary at build time.
|
||||
|
||||
## What are Plugins?
|
||||
This document covers:
|
||||
- The plugin model (event-driven vs sync, traits, lifecycle)
|
||||
- Configuring shipped plugins (DefectDojo, Keycloak)
|
||||
- Writing a new plugin
|
||||
|
||||
Charybdis supports **two types of plugins**, both compile-time integrated:
|
||||
## Table of Contents
|
||||
|
||||
### **1. Event-Driven Plugins**
|
||||
React to entity lifecycle events (create, update, delete):
|
||||
- **Example**: DefectDojo, DependencyTrack
|
||||
- **Implement**: `EventDrivenPlugin` trait + `ResourceHandler`
|
||||
- **Triggered by**: Entity CRUD operations
|
||||
- **Use case**: Auto-create resources in external tools when entities are created
|
||||
1. [Plugin Types](#plugin-types)
|
||||
2. [Available Plugins](#available-plugins)
|
||||
3. [Plugin Configuration](#plugin-configuration)
|
||||
4. [Field Mapping System](#field-mapping-system)
|
||||
5. [Entity Resolution](#entity-resolution)
|
||||
6. [DefectDojo Reference](#defectdojo-reference)
|
||||
7. [Keycloak Reference](#keycloak-reference)
|
||||
8. [Writing a New Plugin](#writing-a-new-plugin)
|
||||
9. [Plugin Lifecycle](#plugin-lifecycle)
|
||||
10. [Annotations](#annotations)
|
||||
11. [Troubleshooting](#troubleshooting)
|
||||
|
||||
### **2. Sync Plugins**
|
||||
Pull data from external sources on a schedule:
|
||||
- **Example**: Okta, Keycloak, Active Directory
|
||||
- **Implement**: `SyncPlugin` trait
|
||||
- **Triggered by**: Cron schedule or manual API call
|
||||
- **Use case**: Sync users/groups from identity providers
|
||||
## Plugin Types
|
||||
|
||||
## Generic Utilities
|
||||
### Event-Driven Plugins
|
||||
React to entity lifecycle events (`Created`, `Updated`, `Deleted`) — typically push data to an external system.
|
||||
|
||||
All plugins have access to reusable utilities in `src/plugins/`:
|
||||
- Implement `EventDrivenPlugin` + one or more `ResourceHandler` per resource kind.
|
||||
- Example: **DefectDojo** — Component creation → DefectDojo product + engagement.
|
||||
|
||||
- **`PluginHttpClient`** - Multi-auth HTTP client (Token, Bearer, API Key, Basic Auth)
|
||||
- **`AnnotationHelper`** - Consistent API for storing/retrieving plugin metadata
|
||||
- **`FieldMapper`** - Maps entity fields to external tool formats
|
||||
- **`DateUtils`** - Common date/time operations
|
||||
### Sync Plugins
|
||||
Pull data from an external system on a schedule, creating or updating entities in Charybdis.
|
||||
|
||||
## Plugin Structure
|
||||
- Implement `SyncPlugin` (`schedule`, `on_startup`, `manual_trigger`).
|
||||
- Example: **Keycloak** — periodic sync of users and groups.
|
||||
|
||||
Each plugin follows this structure:
|
||||
|
||||
```
|
||||
plugins/
|
||||
└── my_plugin/
|
||||
├── Cargo.toml # Plugin crate definition
|
||||
├── README.md # Plugin-specific documentation
|
||||
├── proto/
|
||||
│ └── my_plugin.proto # Protobuf entity definitions
|
||||
└── src/
|
||||
├── lib.rs # Plugin entry point
|
||||
├── handler.rs # Event handler implementation
|
||||
├── client.rs # External API client
|
||||
└── config.rs # Plugin configuration
|
||||
```
|
||||
Both types implement the base `Plugin` trait (`name`, `validate_config`, `health_check`).
|
||||
|
||||
## Available Plugins
|
||||
|
||||
### DefectDojo
|
||||
- **Type**: Event-Driven
|
||||
- **Purpose**: Security vulnerability management integration
|
||||
- **External API**: DefectDojo REST API v2
|
||||
- **Handlers**: ProductHandler, EngagementHandler, ProductTypeHandler, ProductMemberHandler
|
||||
- **Features**:
|
||||
- Automatic product creation on Component/Service creation
|
||||
- Product updates on entity changes
|
||||
- Auto-create CI/CD engagements
|
||||
- Owner resolution (Group → members → email annotation → DefectDojo user lookup)
|
||||
- Configurable OIDC provider mapping (Keycloak, Okta, Azure AD)
|
||||
- Annotation storage (`defectdojo.com/product-id`, `defectdojo.com/engagement-id`, `defectdojo.com/owner-member-ids`)
|
||||
| Plugin | Type | Status |
|
||||
|---|---|---|
|
||||
| DefectDojo | Event-driven | Shipped |
|
||||
| Keycloak | Sync | Shipped |
|
||||
|
||||
### Keycloak
|
||||
- **Type**: Sync
|
||||
- **Purpose**: Identity provider sync (users and groups)
|
||||
- **External API**: Keycloak Admin REST API
|
||||
- **Features**:
|
||||
- User sync with profile and annotations (display_name, email, picture)
|
||||
- Group sync with hierarchy (parent, children, members)
|
||||
- Configurable annotations (e.g., `keycloak.com/email` for OIDC mapping)
|
||||
- On-startup sync, manual trigger via gRPC
|
||||
|
||||
### DependencyTrack (Scaffolded)
|
||||
- **Type**: Event-Driven
|
||||
- **Purpose**: Software supply chain security
|
||||
- **External API**: DependencyTrack REST API
|
||||
- **Status**: Scaffolded (proto + crate structure), handler logic not implemented
|
||||
- **Planned**:
|
||||
- Auto-create projects for components
|
||||
- SBOM ingestion
|
||||
|
||||
## Creating a New Plugin
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Create plugin directory structure:**
|
||||
```bash
|
||||
mkdir -p plugins/my_plugin/{proto,src}
|
||||
```
|
||||
|
||||
2. **Add to plugins.toml:**
|
||||
```toml
|
||||
[plugins.my_plugin]
|
||||
enabled = true
|
||||
proto_path = "plugins/my_plugin/proto"
|
||||
metadata_field_number = 102 # Use next available number
|
||||
spec_field_number = 102
|
||||
description = "My custom integration"
|
||||
```
|
||||
|
||||
3. **Define protobuf schema:**
|
||||
Create `plugins/my_plugin/proto/my_plugin.proto`:
|
||||
```protobuf
|
||||
syntax = "proto3";
|
||||
package charybdis.plugins.my_plugin;
|
||||
|
||||
message MyPluginMetadata {
|
||||
string name = 1;
|
||||
string description = 2;
|
||||
}
|
||||
|
||||
message MyPluginSpec {
|
||||
string integration_type = 1;
|
||||
bool enabled = 2;
|
||||
}
|
||||
```
|
||||
|
||||
4. **Create plugin crate:**
|
||||
Create `plugins/my_plugin/Cargo.toml`:
|
||||
```toml
|
||||
[package]
|
||||
name = "charybdis-plugin-my-plugin"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
charybdis = { path = "../.." }
|
||||
async-trait = "0.1"
|
||||
tokio = { version = "1.0", features = ["full"] }
|
||||
tracing = "0.1"
|
||||
```
|
||||
|
||||
5. **Implement event handler:**
|
||||
Create `plugins/my_plugin/src/lib.rs`:
|
||||
```rust
|
||||
use async_trait::async_trait;
|
||||
use charybdis::events::{EventHandler, EntityEvent, EventResult};
|
||||
|
||||
pub struct MyPluginHandler;
|
||||
|
||||
#[async_trait]
|
||||
impl EventHandler for MyPluginHandler {
|
||||
async fn handle_event(&self, event: &EntityEvent) -> EventResult<()> {
|
||||
// React to entity changes
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
6. **Build and test:**
|
||||
```bash
|
||||
cargo build
|
||||
cargo test
|
||||
```
|
||||
|
||||
## Plugin Field Number Allocation
|
||||
|
||||
Field numbers must be unique across all plugins to avoid protobuf conflicts:
|
||||
|
||||
| Range | Allocation |
|
||||
|----------|-----------------------|
|
||||
| 1-99 | Core entity types |
|
||||
| 100 | DefectDojo |
|
||||
| 101 | DependencyTrack |
|
||||
| 102-199 | Available for plugins |
|
||||
|
||||
When creating a new plugin, use the next available number in the 102+ range.
|
||||
Planned integrations (Slack, Jira, GitHub, Dependency-Track) are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).
|
||||
|
||||
## Plugin Configuration
|
||||
|
||||
Plugins are configured in `config.toml` with `${VAR}` env var substitution for secrets:
|
||||
Plugins are configured in `config.toml` under `[plugins.<name>]`. Use `${VAR}` for secrets — never commit tokens.
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo]
|
||||
enabled = true
|
||||
base_url = "${DEFECTDOJO_API_URL}"
|
||||
base_url = "${DEFECTDOJO_URL}"
|
||||
api_token = "${DEFECTDOJO_API_TOKEN}"
|
||||
|
||||
[plugins.defectdojo.default_engagement]
|
||||
auto_create = true
|
||||
name = "CI/CD Pipeline"
|
||||
|
||||
[plugins.keycloak]
|
||||
enabled = true
|
||||
base_url = "${KEYCLOAK_URL}"
|
||||
realm = "master"
|
||||
realm = "charybdis"
|
||||
client_id = "charybdis-sync"
|
||||
client_secret = "${KEYCLOAK_CLIENT_SECRET}"
|
||||
```
|
||||
|
||||
See `config.toml.example` for all options.
|
||||
When `enabled = false`, the plugin is loaded but does not register handlers and does not react to events. See `config.toml.example` for the full reference.
|
||||
|
||||
## Field Mapping System
|
||||
|
||||
Event-driven plugins (currently DefectDojo) use a configurable mapper to translate Charybdis entity fields into external-tool API payloads. Mappings are defined per resource type under `[plugins.<name>.field_mappings.<resource>]`.
|
||||
|
||||
Three mapping types are supported:
|
||||
|
||||
### 1. Direct field mapping (string)
|
||||
Dot-notation path into the entity.
|
||||
|
||||
```toml
|
||||
name = "metadata.name"
|
||||
description = "metadata.description"
|
||||
email = "spec.profile.email"
|
||||
tags = "metadata.tags"
|
||||
```
|
||||
|
||||
### 2. Static value (object with `value` key)
|
||||
|
||||
```toml
|
||||
business_criticality = { value = "high" }
|
||||
is_active = { value = true }
|
||||
priority = { value = 100 }
|
||||
```
|
||||
|
||||
Supports strings, booleans, numbers, objects, and arrays.
|
||||
|
||||
### 3. Entity resolution (object)
|
||||
Resolve a reference into a related entity and extract a field from it.
|
||||
|
||||
```toml
|
||||
product_manager = {
|
||||
from = "spec.owner",
|
||||
resolve_entity = "User",
|
||||
extract = "annotations.defectdojo.com/user-id"
|
||||
}
|
||||
```
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `from` | string | yes | Source field path |
|
||||
| `resolve_entity` | string | yes | Entity kind to look up (`User`, `Component`, ...) |
|
||||
| `extract` | string | yes | Field path to extract from the resolved entity |
|
||||
| `resolve_array` | bool | no | Source field is an array of references |
|
||||
| `lookup_entity` | string | no | Find an entity that references the source |
|
||||
|
||||
## Entity Resolution
|
||||
|
||||
Entity resolution lets a plugin navigate the entity graph at mapping time. The mapper performs a database lookup, extracts the target field, and substitutes it into the payload.
|
||||
|
||||
```
|
||||
1. Source field spec.owner = "user:john.doe"
|
||||
2. Resolve entity User entity for "john.doe"
|
||||
3. Extract target field annotations.defectdojo.com/user-id = "123"
|
||||
4. Result product_manager = 123
|
||||
```
|
||||
|
||||
### Array resolution
|
||||
|
||||
```toml
|
||||
user_ids = {
|
||||
from = "spec.members",
|
||||
resolve_entity = "User",
|
||||
extract = "annotations.defectdojo.com/user-id",
|
||||
resolve_array = true
|
||||
}
|
||||
```
|
||||
|
||||
`Group.spec.members = ["user:john", "user:jane"]` → `user_ids = [123, 124]`.
|
||||
|
||||
### Linked entity lookup
|
||||
|
||||
```toml
|
||||
engagement_product = {
|
||||
from = "id",
|
||||
lookup_entity = "Component",
|
||||
extract = "annotations.defectdojo.com/product-id"
|
||||
}
|
||||
```
|
||||
|
||||
Finds a `Component` that references the source entity and extracts its annotation.
|
||||
|
||||
### Error handling
|
||||
Resolution failures (entity not found, missing annotation, invalid reference) log a warning and omit the field rather than failing the whole operation.
|
||||
|
||||
## DefectDojo Reference
|
||||
|
||||
Maps Charybdis entities to DefectDojo resources via the REST API v2. Handlers shipped: `ProductHandler`, `EngagementHandler`, `ProductTypeHandler`, `ProductMemberHandler`.
|
||||
|
||||
### Connection
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo]
|
||||
enabled = true
|
||||
base_url = "${DEFECTDOJO_URL}"
|
||||
api_token = "${DEFECTDOJO_API_TOKEN}"
|
||||
default_product_type_id = 1
|
||||
auto_create_users = true
|
||||
auto_create_product_types = false
|
||||
```
|
||||
|
||||
To obtain the API token: in DefectDojo, **User Profile → API Key**.
|
||||
|
||||
### Default engagement (auto-created with each product)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.default_engagement]
|
||||
auto_create = true
|
||||
name = "CI/CD Pipeline"
|
||||
description = "Automated security scans from CI/CD pipeline"
|
||||
engagement_type = "CI/CD"
|
||||
status = "In Progress"
|
||||
duration_days = 365
|
||||
deduplication_on_engagement = true
|
||||
```
|
||||
|
||||
### Owner resolution
|
||||
When a `Component` has an `owner` (typically a team name), Charybdis resolves it to DefectDojo users:
|
||||
|
||||
1. Find `Group` entity matching the owner.
|
||||
2. Read `Group.spec.members` (usernames).
|
||||
3. Look up each `User` entity.
|
||||
4. Extract the user's email from the configured annotation.
|
||||
5. Find the matching user in DefectDojo by email.
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.owner_resolution]
|
||||
user_email_annotation = "keycloak.com/email" # depends on your IdP
|
||||
defectdojo_lookup_field = "email" # "email" or "username"
|
||||
assign_all_members = true # all group members, or just first match
|
||||
```
|
||||
|
||||
### Product mapping (Component → DefectDojo Product)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.field_mappings.product]
|
||||
name = "metadata.name"
|
||||
description = "metadata.description"
|
||||
product_type_id = { value = 1 }
|
||||
tags = "metadata.tags"
|
||||
business_criticality = { value = "high" } # very high | high | medium | low | very low | none
|
||||
platform = { value = "web" } # web | mobile | desktop | iot | ...
|
||||
lifecycle = { value = "production" }
|
||||
origin = { value = "internal" }
|
||||
external_audience = { value = true }
|
||||
internet_accessible = { value = true }
|
||||
|
||||
product_manager = {
|
||||
from = "spec.owner",
|
||||
resolve_entity = "User",
|
||||
extract = "annotations.defectdojo.com/user-id"
|
||||
}
|
||||
```
|
||||
|
||||
### User mapping (User → DefectDojo User)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.field_mappings.user]
|
||||
username = "metadata.name"
|
||||
email = "spec.profile.email"
|
||||
first_name = "spec.profile.displayName"
|
||||
last_name = "spec.profile.displayName"
|
||||
is_active = { value = true }
|
||||
```
|
||||
|
||||
### Product Type mapping (System → DefectDojo Product Type)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.field_mappings.product_type]
|
||||
name = "metadata.name"
|
||||
description = "metadata.description"
|
||||
critical_product = { value = false }
|
||||
key_product = { value = true }
|
||||
```
|
||||
|
||||
### Product Member mapping (Group → DefectDojo Product Member)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.field_mappings.product_member]
|
||||
product_id = {
|
||||
from = "spec.parent",
|
||||
resolve_entity = "Component",
|
||||
extract = "annotations.defectdojo.com/product-id"
|
||||
}
|
||||
|
||||
user_id = {
|
||||
from = "spec.members",
|
||||
resolve_entity = "User",
|
||||
extract = "annotations.defectdojo.com/user-id",
|
||||
resolve_array = true
|
||||
}
|
||||
|
||||
role_name = { value = "Reader" } # Owner | Maintainer | Writer | Reader | API_Importer
|
||||
```
|
||||
|
||||
### Engagement mapping (Resource → DefectDojo Engagement)
|
||||
|
||||
```toml
|
||||
[plugins.defectdojo.field_mappings.engagement]
|
||||
name = "metadata.name"
|
||||
description = "metadata.description"
|
||||
|
||||
product_id = {
|
||||
from = "spec.owner",
|
||||
resolve_entity = "Component",
|
||||
extract = "annotations.defectdojo.com/product-id"
|
||||
}
|
||||
|
||||
lead_id = {
|
||||
from = "spec.dependsOn",
|
||||
resolve_entity = "User",
|
||||
extract = "annotations.defectdojo.com/user-id"
|
||||
}
|
||||
|
||||
target_start = "spec.target_start"
|
||||
target_end = "spec.target_end"
|
||||
status = { value = "In Progress" }
|
||||
engagement_type = { value = "CI/CD" }
|
||||
version = "metadata.annotations.version"
|
||||
commit_hash = "metadata.annotations.commit_hash"
|
||||
branch_tag = "metadata.annotations.branch"
|
||||
build_id = "metadata.annotations.build_id"
|
||||
source_code_management_uri = "metadata.annotations.repo_url"
|
||||
|
||||
deduplication_on_engagement = { value = true }
|
||||
api_test = { value = true }
|
||||
pen_test = { value = false }
|
||||
```
|
||||
|
||||
### Annotations written by the plugin
|
||||
- `defectdojo.com/product-id`
|
||||
- `defectdojo.com/engagement-id`
|
||||
- `defectdojo.com/owner-member-ids`
|
||||
|
||||
## Keycloak Reference
|
||||
|
||||
Sync plugin: pulls users and groups from a Keycloak realm via the Admin REST API on a schedule, on startup, or via manual trigger.
|
||||
|
||||
```toml
|
||||
[plugins.keycloak]
|
||||
enabled = true
|
||||
base_url = "${KEYCLOAK_URL}"
|
||||
realm = "charybdis"
|
||||
client_id = "charybdis-sync"
|
||||
client_secret = "${KEYCLOAK_CLIENT_SECRET}"
|
||||
|
||||
[plugins.keycloak.sync]
|
||||
schedule = "0 */5 * * * *" # optional cron; omit to sync only on startup/trigger
|
||||
on_startup = true
|
||||
manual_trigger = true
|
||||
sync_users = true
|
||||
sync_groups = true
|
||||
namespace = "keycloak"
|
||||
page_size = 100
|
||||
```
|
||||
|
||||
The service account behind `client_id`/`client_secret` must have `view-users` and `view-groups` on the realm.
|
||||
|
||||
Synced entities carry annotations such as `keycloak.com/email`, `keycloak.com/id`, used by other plugins (e.g., the DefectDojo owner-resolution chain above).
|
||||
|
||||
## Writing a New Plugin
|
||||
|
||||
### 1. Scaffold directory
|
||||
|
||||
```bash
|
||||
mkdir -p plugins/my_plugin/{proto,src}
|
||||
```
|
||||
|
||||
### 2. Register in `plugins.toml`
|
||||
|
||||
```toml
|
||||
[plugins.my_plugin]
|
||||
enabled = true
|
||||
proto_path = "plugins/my_plugin/proto"
|
||||
metadata_field_number = 103 # next available — see registry below
|
||||
spec_field_number = 203
|
||||
description = "My custom integration"
|
||||
```
|
||||
|
||||
### 3. Define the protobuf schema
|
||||
|
||||
`plugins/my_plugin/proto/my_plugin.proto`:
|
||||
|
||||
```protobuf
|
||||
syntax = "proto3";
|
||||
package charybdis.plugins.my_plugin;
|
||||
|
||||
message MyPluginMetadata {
|
||||
string name = 1;
|
||||
string description = 2;
|
||||
}
|
||||
|
||||
message MyPluginSpec {
|
||||
string integration_type = 1;
|
||||
bool enabled = 2;
|
||||
}
|
||||
```
|
||||
|
||||
`build.rs` regenerates `proto/entities.proto` automatically on next build.
|
||||
|
||||
### 4. Create the crate
|
||||
|
||||
`plugins/my_plugin/Cargo.toml`:
|
||||
|
||||
```toml
|
||||
[package]
|
||||
name = "charybdis-my-plugin-plugin"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[lib]
|
||||
name = "charybdis_my_plugin"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[dependencies]
|
||||
charybdis = { path = "../.." }
|
||||
async-trait = "0.1"
|
||||
anyhow = "1.0"
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
tokio = { version = "1.48", features = ["full"] }
|
||||
tracing = "0.1"
|
||||
```
|
||||
|
||||
Add the crate to the workspace `members` in the top-level `Cargo.toml`.
|
||||
|
||||
### 5. Implement the plugin
|
||||
|
||||
Event-driven example:
|
||||
|
||||
```rust
|
||||
use async_trait::async_trait;
|
||||
use charybdis::plugins::{EventDrivenPlugin, Plugin, PluginConfig, PluginType, ResourceHandler};
|
||||
use std::sync::Arc;
|
||||
|
||||
pub struct MyPlugin { /* config, client, handlers */ }
|
||||
|
||||
#[async_trait]
|
||||
impl Plugin for MyPlugin {
|
||||
fn name(&self) -> &str { "my_plugin" }
|
||||
fn plugin_type(&self) -> PluginType { PluginType::EventDriven }
|
||||
fn load_config(&mut self, _: PluginConfig) -> anyhow::Result<()> { Ok(()) }
|
||||
fn validate_config(&self) -> anyhow::Result<()> { Ok(()) }
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl EventDrivenPlugin for MyPlugin {
|
||||
fn resource_handlers(&self) -> Vec<Arc<dyn ResourceHandler>> {
|
||||
vec![/* your handlers */]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6. Register in `charybdis-server`
|
||||
|
||||
Add a loader block in `charybdis-server/src/main.rs` (follow the DefectDojo block as a template) and declare the dependency in `charybdis-server/Cargo.toml`.
|
||||
|
||||
### Field number registry
|
||||
|
||||
| Range | Allocation |
|
||||
|---|---|
|
||||
| 1–99 | Core entity types |
|
||||
| 100 | DefectDojo |
|
||||
| 101 | DependencyTrack |
|
||||
| 102 | Keycloak |
|
||||
| 103+ | Available |
|
||||
|
||||
Field numbers MUST be unique across plugins — protobuf wire format depends on it.
|
||||
|
||||
## Plugin Lifecycle
|
||||
|
||||
1. **Build Time:**
|
||||
- `build.rs` reads `plugins.toml`
|
||||
- Generates `proto/entities.proto` with plugin types
|
||||
- Compiles all protobuf files
|
||||
- Generates Rust code
|
||||
### Build time
|
||||
1. `build.rs` reads `plugins.toml`.
|
||||
2. Generates `proto/entities.proto` with plugin-contributed `oneof` variants.
|
||||
3. Compiles all protobuf files (`tonic-prost-build`).
|
||||
|
||||
2. **Runtime:**
|
||||
- Plugin handlers registered with event bus
|
||||
- Events published on entity CRUD operations
|
||||
- Handlers react asynchronously
|
||||
- External APIs called as needed
|
||||
### Runtime
|
||||
1. `charybdis-server` loads each enabled plugin from `config.toml`.
|
||||
2. Event-driven plugins register their `ResourceHandler`s with the dispatcher.
|
||||
3. Sync plugins register with the cron scheduler (and optionally run once at startup).
|
||||
4. Entity CRUD operations publish `EntityEvent` to the in-memory event bus.
|
||||
5. The dispatcher fans out events to matching handlers (filtered by `trigger_kinds`).
|
||||
6. Handlers call external APIs and write annotations back to Charybdis.
|
||||
|
||||
3. **Event Flow:**
|
||||
```
|
||||
Client creates entity
|
||||
→ Entity stored in database
|
||||
→ Event published to event bus
|
||||
→ Plugin handler receives event
|
||||
→ Plugin calls external API
|
||||
→ Plugin stores external ID in annotations
|
||||
```
|
||||
|
||||
## Using Annotations
|
||||
|
||||
Plugins store external tool IDs in entity annotations:
|
||||
|
||||
```rust
|
||||
// Store external ID
|
||||
entity.annotations.insert(
|
||||
"my-plugin.com/resource-id".to_string(),
|
||||
"ext-12345".to_string(),
|
||||
);
|
||||
|
||||
// Query by external ID in SQL
|
||||
SELECT * FROM entities
|
||||
WHERE annotations->>'my-plugin.com/resource-id' = 'ext-12345';
|
||||
### Event flow
|
||||
```
|
||||
Client → gRPC CreateEntity → DB INSERT → EntityEvent::Created → Dispatcher
|
||||
↓
|
||||
ResourceHandler.handle_create(entity)
|
||||
↓
|
||||
POST /api/products → DefectDojo
|
||||
↓
|
||||
update_annotations("defectdojo.com/product-id")
|
||||
```
|
||||
|
||||
### Annotation Naming Convention
|
||||
## Annotations
|
||||
|
||||
Plugins store external IDs in entity annotations using reverse-DNS keys:
|
||||
|
||||
Use reverse-DNS style:
|
||||
- `defectdojo.com/product-id`
|
||||
- `dependencytrack.com/project-uuid`
|
||||
- `github.com/repo-slug`
|
||||
- `{tool}.com/{resource}-{attribute}`
|
||||
- `defectdojo.com/engagement-id`
|
||||
- `keycloak.com/id`
|
||||
- `keycloak.com/email`
|
||||
|
||||
## Best Practices
|
||||
Query by annotation via JSONB in PostgreSQL:
|
||||
|
||||
### 1. Error Handling
|
||||
- Don't panic - return errors
|
||||
- Log but continue on non-critical failures
|
||||
- Implement retry logic for transient errors
|
||||
|
||||
### 2. Idempotency
|
||||
- Check if resource exists before creating
|
||||
- Make operations safe to retry
|
||||
- Handle duplicate creation gracefully
|
||||
|
||||
### 3. Performance
|
||||
- Don't block event handlers
|
||||
- Use `tokio::spawn` for long operations
|
||||
- Batch operations when possible
|
||||
|
||||
### 4. Testing
|
||||
- Unit test event handlers
|
||||
- Mock external API clients
|
||||
- Integration tests with real APIs (optional)
|
||||
|
||||
### 5. Documentation
|
||||
- Document required environment variables
|
||||
- Provide example configurations
|
||||
- Explain entity model and annotations
|
||||
|
||||
## Contributing Plugins
|
||||
|
||||
We welcome plugin contributions! To contribute:
|
||||
|
||||
1. Fork the repository
|
||||
2. Create your plugin following the structure above
|
||||
3. Add comprehensive tests
|
||||
4. Document configuration and usage
|
||||
5. Submit a pull request
|
||||
|
||||
### Plugin Requirements
|
||||
|
||||
- [ ] Protobuf definitions with Metadata and Spec messages
|
||||
- [ ] Event handler implementation
|
||||
- [ ] External API client (if applicable)
|
||||
- [ ] Configuration via environment variables
|
||||
- [ ] README with setup instructions
|
||||
- [ ] Unit tests for event handler
|
||||
- [ ] Example usage in documentation
|
||||
|
||||
## Plugin Distribution Models
|
||||
|
||||
### In-Tree (Current)
|
||||
Plugins live in the `plugins/` directory and are enabled via `plugins.toml`.
|
||||
|
||||
**Pros:**
|
||||
- Easy to discover
|
||||
- Consistent quality
|
||||
- Tested together
|
||||
|
||||
**Cons:**
|
||||
- Requires core repo access
|
||||
- All plugins built together
|
||||
|
||||
### External Crates (Future)
|
||||
Plugins distributed as separate Rust crates.
|
||||
|
||||
**Example:**
|
||||
```toml
|
||||
[dependencies]
|
||||
charybdis-plugin-custom = "0.1"
|
||||
```sql
|
||||
SELECT * FROM entities
|
||||
WHERE annotations->>'defectdojo.com/product-id' = '456';
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Independent versioning
|
||||
- Community contributions
|
||||
- Optional dependencies
|
||||
Use `EntityRepository::update_annotations()` for atomic JSONB merge — it avoids the read-modify-write race when multiple handlers write to the same entity.
|
||||
|
||||
**Cons:**
|
||||
- Discovery harder
|
||||
- Compatibility challenges
|
||||
## Troubleshooting
|
||||
|
||||
### Plugin Marketplace (Future)
|
||||
Central registry of available plugins (like Backstage).
|
||||
### Plugin not reacting to events
|
||||
- `enabled = true` in `config.toml`?
|
||||
- Plugin registered in `charybdis-server/src/main.rs`?
|
||||
- Entity kind matches the handler's `trigger_kinds`?
|
||||
- Event bus running? (`info!("Event bus started")` in startup logs)
|
||||
|
||||
## Resources
|
||||
### Field mapping not applied
|
||||
- Path uses dot notation, e.g. `metadata.name`, not `meta.name`.
|
||||
- Static value uses `{ value = "..." }`, not a bare string when you want a literal.
|
||||
- Field actually exists on the entity (inspect via the YAML adapter or `GetEntity` gRPC).
|
||||
|
||||
- [Plugin Configuration Guide](../docs/PLUGIN_CONFIGURATION_GUIDE.md)
|
||||
- [Core Concepts](../docs/core-concepts.md)
|
||||
- [Architecture](../docs/architecture.md)
|
||||
- [Protobuf Style Guide](https://protobuf.dev/programming-guides/style/)
|
||||
### Entity resolution returns nothing
|
||||
- Resolved entity exists in the DB.
|
||||
- Resolved entity has the `extract` annotation set.
|
||||
- `resolve_entity` uses the PascalCase kind (`"User"`, not `"user"`).
|
||||
- For arrays, `resolve_array = true` is set.
|
||||
|
||||
## Support
|
||||
### DefectDojo API errors
|
||||
- `400 Bad Request: Invalid product type ID` → check `default_product_type_id` or your `product_type_id` mapping.
|
||||
- `401 Unauthorized` → check `DEFECTDOJO_API_TOKEN`.
|
||||
- `409 Conflict: user already exists` → expected, the plugin reuses existing users.
|
||||
|
||||
- Open an issue for bug reports
|
||||
- Discussions for questions
|
||||
- PRs for contributions
|
||||
### Performance
|
||||
- Minimize entity-resolution chain depth.
|
||||
- Prefer static values when the data isn't entity-bound.
|
||||
- Check network latency to the external API — handlers run synchronously per event.
|
||||
|
||||
## License
|
||||
## See Also
|
||||
|
||||
Same as Charybdis core (see LICENSE file in repository root)
|
||||
- [Core Concepts](../docs/core-concepts.md) — entity model, events, annotations.
|
||||
- [Architecture](../docs/architecture.md) — protobuf schema, storage, scaling.
|
||||
- [Field Mapper Tests](../src/plugins/field_mapper.rs) — reference behavior.
|
||||
|
||||
Reference in New Issue
Block a user