charybdis/plugins/README.md

# Charybdis Plugins

Plugins extend Charybdis with integrations to external tools (security scanners, issue trackers, identity providers). Plugins are **compile-time integrated**: they live as Rust crates under `plugins/`, declare protobuf extensions in `plugins.toml`, and are linked into the `charybdis-server` binary at build time.

This document covers:
- The plugin model (event-driven vs sync, traits, lifecycle)
- Configuring shipped plugins (DefectDojo, Keycloak)
- Writing a new plugin

## Table of Contents

1. [Plugin Types](#plugin-types)
2. [Available Plugins](#available-plugins)
3. [Plugin Configuration](#plugin-configuration)
4. [Field Mapping System](#field-mapping-system)
5. [Entity Resolution](#entity-resolution)
6. [DefectDojo Reference](#defectdojo-reference)
7. [Keycloak Reference](#keycloak-reference)
8. [Writing a New Plugin](#writing-a-new-plugin)
9. [Plugin Lifecycle](#plugin-lifecycle)
10. [Annotations](#annotations)
11. [Troubleshooting](#troubleshooting)

## Plugin Types

### Event-Driven Plugins
React to entity lifecycle events (`Created`, `Updated`, `Deleted`) — typically push data to an external system.

- Implement `EventDrivenPlugin` + one or more `ResourceHandler` per resource kind.
- Example: **DefectDojo** — Component creation → DefectDojo product + engagement.

### Sync Plugins
Pull data from an external system on a schedule, creating or updating entities in Charybdis.

- Implement `SyncPlugin` (`schedule`, `on_startup`, `manual_trigger`).
- Example: **Keycloak** — periodic sync of users and groups.

Both types implement the base `Plugin` trait (`name`, `validate_config`, `health_check`).

## Available Plugins

| Plugin | Type | Status |
|---|---|---|
| DefectDojo | Event-driven | Shipped |
| Keycloak | Sync | Shipped |

Planned integrations (Slack, Jira, GitHub, Dependency-Track) are tracked in [VISION.md](../VISION.md) and [TODO.md](../TODO.md).

## Plugin Configuration

Plugins are configured in `config.toml` under `[plugins.<name>]`. Use `${VAR}` for secrets — never commit tokens.

```toml
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

[plugins.keycloak]
enabled = true
base_url = "${KEYCLOAK_URL}"
realm = "charybdis"
client_id = "charybdis-sync"
client_secret = "${KEYCLOAK_CLIENT_SECRET}"
```

When `enabled = false`, the plugin is loaded but does not register handlers and does not react to events. See `config.toml.example` for the full reference.

## Field Mapping System

Event-driven plugins (currently DefectDojo) use a configurable mapper to translate Charybdis entity fields into external-tool API payloads. Mappings are defined per resource type under `[plugins.<name>.field_mappings.<resource>]`.

Three mapping types are supported:

### 1. Direct field mapping (string)
Dot-notation path into the entity.

```toml
name = "metadata.name"
description = "metadata.description"
email = "spec.profile.email"
tags = "metadata.tags"
```

### 2. Static value (object with `value` key)

```toml
business_criticality = { value = "high" }
is_active = { value = true }
priority = { value = 100 }
```

Supports strings, booleans, numbers, objects, and arrays.

### 3. Entity resolution (object)
Resolve a reference into a related entity and extract a field from it.

```toml
product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}
```

| Parameter | Type | Required | Description |
|---|---|---|---|
| `from` | string | yes | Source field path |
| `resolve_entity` | string | yes | Entity kind to look up (`User`, `Component`, ...) |
| `extract` | string | yes | Field path to extract from the resolved entity |
| `resolve_array` | bool | no | Source field is an array of references |
| `lookup_entity` | string | no | Find an entity that references the source |

## Entity Resolution

Entity resolution lets a plugin navigate the entity graph at mapping time. The mapper performs a database lookup, extracts the target field, and substitutes it into the payload.

```
1. Source field            spec.owner = "user:john.doe"
2. Resolve entity          User entity for "john.doe"
3. Extract target field    annotations.defectdojo.com/user-id = "123"
4. Result                  product_manager = 123
```

### Array resolution

```toml
user_ids = {
  from = "spec.members",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id",
  resolve_array = true
}
```

`Group.spec.members = ["user:john", "user:jane"]` → `user_ids = [123, 124]`.

### Linked entity lookup

```toml
engagement_product = {
  from = "id",
  lookup_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
}
```

Finds a `Component` that references the source entity and extracts its annotation.

### Error handling
Resolution failures (entity not found, missing annotation, invalid reference) log a warning and omit the field rather than failing the whole operation.

## DefectDojo Reference

Maps Charybdis entities to DefectDojo resources via the REST API v2. Handlers shipped: `ProductHandler`, `EngagementHandler`, `ProductTypeHandler`, `ProductMemberHandler`.

### Connection

```toml
[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"
default_product_type_id = 1
auto_create_users = true
auto_create_product_types = false
```

To obtain the API token: in DefectDojo, **User Profile → API Key**.

### Default engagement (auto-created with each product)

```toml
[plugins.defectdojo.default_engagement]
auto_create = true
name = "CI/CD Pipeline"
description = "Automated security scans from CI/CD pipeline"
engagement_type = "CI/CD"
status = "In Progress"
duration_days = 365
deduplication_on_engagement = true
```

### Owner resolution
When a `Component` has an `owner` (typically a team name), Charybdis resolves it to DefectDojo users:

1. Find `Group` entity matching the owner.
2. Read `Group.spec.members` (usernames).
3. Look up each `User` entity.
4. Extract the user's email from the configured annotation.
5. Find the matching user in DefectDojo by email.

```toml
[plugins.defectdojo.owner_resolution]
user_email_annotation = "keycloak.com/email"   # depends on your IdP
defectdojo_lookup_field = "email"              # "email" or "username"
assign_all_members = true                       # all group members, or just first match
```

### Product mapping (Component → DefectDojo Product)

```toml
[plugins.defectdojo.field_mappings.product]
name = "metadata.name"
description = "metadata.description"
product_type_id = { value = 1 }
tags = "metadata.tags"
business_criticality = { value = "high" }       # very high | high | medium | low | very low | none
platform = { value = "web" }                    # web | mobile | desktop | iot | ...
lifecycle = { value = "production" }
origin = { value = "internal" }
external_audience = { value = true }
internet_accessible = { value = true }

product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}
```

### User mapping (User → DefectDojo User)

```toml
[plugins.defectdojo.field_mappings.user]
username = "metadata.name"
email = "spec.profile.email"
first_name = "spec.profile.displayName"
last_name = "spec.profile.displayName"
is_active = { value = true }
```

### Product Type mapping (System → DefectDojo Product Type)

```toml
[plugins.defectdojo.field_mappings.product_type]
name = "metadata.name"
description = "metadata.description"
critical_product = { value = false }
key_product = { value = true }
```

### Product Member mapping (Group → DefectDojo Product Member)

```toml
[plugins.defectdojo.field_mappings.product_member]
product_id = {
  from = "spec.parent",
  resolve_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
}

user_id = {
  from = "spec.members",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id",
  resolve_array = true
}

role_name = { value = "Reader" }    # Owner | Maintainer | Writer | Reader | API_Importer
```

### Engagement mapping (Resource → DefectDojo Engagement)

```toml
[plugins.defectdojo.field_mappings.engagement]
name = "metadata.name"
description = "metadata.description"

product_id = {
  from = "spec.owner",
  resolve_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
}

lead_id = {
  from = "spec.dependsOn",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

target_start = "spec.target_start"
target_end = "spec.target_end"
status = { value = "In Progress" }
engagement_type = { value = "CI/CD" }
version = "metadata.annotations.version"
commit_hash = "metadata.annotations.commit_hash"
branch_tag = "metadata.annotations.branch"
build_id = "metadata.annotations.build_id"
source_code_management_uri = "metadata.annotations.repo_url"

deduplication_on_engagement = { value = true }
api_test = { value = true }
pen_test = { value = false }
```

### Annotations written by the plugin
- `defectdojo.com/product-id`
- `defectdojo.com/engagement-id`
- `defectdojo.com/owner-member-ids`

## Keycloak Reference

Sync plugin: pulls users and groups from a Keycloak realm via the Admin REST API on a schedule, on startup, or via manual trigger.

```toml
[plugins.keycloak]
enabled = true
base_url = "${KEYCLOAK_URL}"
realm = "charybdis"
client_id = "charybdis-sync"
client_secret = "${KEYCLOAK_CLIENT_SECRET}"

[plugins.keycloak.sync]
schedule = "0 */5 * * * *"   # optional cron; omit to sync only on startup/trigger
on_startup = true
manual_trigger = true
sync_users = true
sync_groups = true
namespace = "keycloak"
page_size = 100
```

The service account behind `client_id`/`client_secret` must have `view-users` and `view-groups` on the realm.

Synced entities carry annotations such as `keycloak.com/email`, `keycloak.com/id`, used by other plugins (e.g., the DefectDojo owner-resolution chain above).

## Writing a New Plugin

### 1. Scaffold directory

```bash
mkdir -p plugins/my_plugin/{proto,src}
```

### 2. Register in `plugins.toml`

```toml
[plugins.my_plugin]
enabled = true
proto_path = "plugins/my_plugin/proto"
metadata_field_number = 103   # next available — see registry below
spec_field_number = 203
description = "My custom integration"
```

### 3. Define the protobuf schema

`plugins/my_plugin/proto/my_plugin.proto`:

```protobuf
syntax = "proto3";
package charybdis.plugins.my_plugin;

message MyPluginMetadata {
  string name = 1;
  string description = 2;
}

message MyPluginSpec {
  string integration_type = 1;
  bool enabled = 2;
}
```

`build.rs` regenerates `proto/entities.proto` automatically on next build.

### 4. Create the crate

`plugins/my_plugin/Cargo.toml`:

```toml
[package]
name = "charybdis-my-plugin-plugin"
version = "0.1.0"
edition = "2024"

[lib]
name = "charybdis_my_plugin"
path = "src/lib.rs"

[dependencies]
charybdis = { path = "../.." }
async-trait = "0.1"
anyhow = "1.0"
serde = { version = "1.0", features = ["derive"] }
tokio = { version = "1.48", features = ["full"] }
tracing = "0.1"
```

Add the crate to the workspace `members` in the top-level `Cargo.toml`.

### 5. Implement the plugin

Event-driven example:

```rust
use async_trait::async_trait;
use charybdis::plugins::{EventDrivenPlugin, Plugin, PluginConfig, PluginType, ResourceHandler};
use std::sync::Arc;

pub struct MyPlugin { /* config, client, handlers */ }

#[async_trait]
impl Plugin for MyPlugin {
    fn name(&self) -> &str { "my_plugin" }
    fn plugin_type(&self) -> PluginType { PluginType::EventDriven }
    fn load_config(&mut self, _: PluginConfig) -> anyhow::Result<()> { Ok(()) }
    fn validate_config(&self) -> anyhow::Result<()> { Ok(()) }
}

#[async_trait]
impl EventDrivenPlugin for MyPlugin {
    fn resource_handlers(&self) -> Vec<Arc<dyn ResourceHandler>> {
        vec![/* your handlers */]
    }
}
```

### 6. Register in `charybdis-server`

Add a loader block in `charybdis-server/src/main.rs` (follow the DefectDojo block as a template) and declare the dependency in `charybdis-server/Cargo.toml`.

### Field number registry

| Range | Allocation |
|---|---|
| 1–99 | Core entity types |
| 100 | DefectDojo |
| 101 | DependencyTrack |
| 102 | Keycloak |
| 103+ | Available |

Field numbers MUST be unique across plugins — protobuf wire format depends on it.

## Plugin Lifecycle

### Build time
1. `build.rs` reads `plugins.toml`.
2. Generates `proto/entities.proto` with plugin-contributed `oneof` variants.
3. Compiles all protobuf files (`tonic-prost-build`).

### Runtime
1. `charybdis-server` loads each enabled plugin from `config.toml`.
2. Event-driven plugins register their `ResourceHandler`s with the dispatcher.
3. Sync plugins register with the cron scheduler (and optionally run once at startup).
4. Entity CRUD operations publish `EntityEvent` to the in-memory event bus.
5. The dispatcher fans out events to matching handlers (filtered by `trigger_kinds`).
6. Handlers call external APIs and write annotations back to Charybdis.

### Event flow
```
Client → gRPC CreateEntity → DB INSERT → EntityEvent::Created → Dispatcher
                                                                  ↓
                                          ResourceHandler.handle_create(entity)
                                                                  ↓
                                          POST /api/products → DefectDojo
                                                                  ↓
                                          update_annotations("defectdojo.com/product-id")
```

## Annotations

Plugins store external IDs in entity annotations using reverse-DNS keys:

- `defectdojo.com/product-id`
- `defectdojo.com/engagement-id`
- `keycloak.com/id`
- `keycloak.com/email`

Query by annotation via JSONB in PostgreSQL:

```sql
SELECT * FROM entities
WHERE annotations->>'defectdojo.com/product-id' = '456';
```

Use `EntityRepository::update_annotations()` for atomic JSONB merge — it avoids the read-modify-write race when multiple handlers write to the same entity.

## Troubleshooting

### Plugin not reacting to events
- `enabled = true` in `config.toml`?
- Plugin registered in `charybdis-server/src/main.rs`?
- Entity kind matches the handler's `trigger_kinds`?
- Event bus running? (`info!("Event bus started")` in startup logs)

### Field mapping not applied
- Path uses dot notation, e.g. `metadata.name`, not `meta.name`.
- Static value uses `{ value = "..." }`, not a bare string when you want a literal.
- Field actually exists on the entity (inspect via the YAML adapter or `GetEntity` gRPC).

### Entity resolution returns nothing
- Resolved entity exists in the DB.
- Resolved entity has the `extract` annotation set.
- `resolve_entity` uses the PascalCase kind (`"User"`, not `"user"`).
- For arrays, `resolve_array = true` is set.

### DefectDojo API errors
- `400 Bad Request: Invalid product type ID` → check `default_product_type_id` or your `product_type_id` mapping.
- `401 Unauthorized` → check `DEFECTDOJO_API_TOKEN`.
- `409 Conflict: user already exists` → expected, the plugin reuses existing users.

### Performance
- Minimize entity-resolution chain depth.
- Prefer static values when the data isn't entity-bound.
- Check network latency to the external API — handlers run synchronously per event.

## See Also

- [Core Concepts](../docs/core-concepts.md) — entity model, events, annotations.
- [Architecture](../docs/architecture.md) — protobuf schema, storage, scaling.
- [Field Mapper Tests](../src/plugins/field_mapper.rs) — reference behavior.