charybdis/docs/PLUGIN_CONFIGURATION_GUIDE.md

Charybdis Plugin Configuration Guide

Complete reference for configuring and using the Charybdis plugin system.

Table of Contents

1. Overview
2. Plugin System Architecture
3. Configuration File Structure
4. Field Mapping System
5. Entity Resolution
6. Plugin Configuration Reference
7. Environment Variables
8. Best Practices
9. Troubleshooting
10. Examples

Overview

Charybdis plugins extend the core platform with integrations to external tools and services. Plugins react to entity lifecycle events (Create, Update, Delete) and automatically provision or sync resources in external systems.

Plugin Types

Event-Driven Plugins (Current Implementation):

• React to entity lifecycle events
• Push data to external systems
• One-way sync: Charybdis → External Tool

Sync Plugins (Future):

• Pull data from external systems on schedule
• Create/update entities in Charybdis
• One-way sync: External Tool → Charybdis

Plugin System Architecture

┌─────────────────────────────────────────┐
│         Charybdis Core                  │
│  ┌────────────────────────────────┐     │
│  │  Entity Repository             │     │
│  │  (PostgreSQL)                  │     │
│  └──────────┬─────────────────────┘     │
│             │                            │
│             ▼                            │
│  ┌────────────────────────────────┐     │
│  │  Event Bus                     │     │
│  │  (In-Memory / Future: Redis)   │     │
│  └──────────┬─────────────────────┘     │
│             │ EntityEvent                │
│             │ (Created/Updated/Deleted)  │
└─────────────┼────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────┐
│       Event Dispatcher                  │
│  (Routes events to plugins)             │
└──────────┬──────────────────────────────┘
           │
           ├─────────────────┐
           ▼                 ▼
┌──────────────────┐  ┌──────────────────┐
│ DefectDojo       │  │ Future Plugins   │
│ Plugin           │  │ (DependencyTrack)│
│                  │  │                  │
│ ┌──────────────┐ │  └──────────────────┘
│ │ProductHandler│ │
│ │UserHandler   │ │
│ │...           │ │
│ └──────────────┘ │
└─────────┬────────┘
          │ HTTP/REST
          ▼
┌──────────────────┐
│  External Tool   │
│  (DefectDojo)    │
└──────────────────┘

Key Components

1. Plugin: Top-level plugin interface
2. ResourceHandler: Handles specific resource types (e.g., Products, Users)
3. FieldMapper: Maps Charybdis entity fields to external tool fields
4. Event Dispatcher: Routes entity events to appropriate handlers
5. Plugin Manager: Manages plugin lifecycle and registration

Configuration File Structure

Plugin configuration lives in your main config.toml file under the [plugins.<plugin_name>] section.

Basic Structure

[plugins.<plugin_name>]
enabled = true                    # Enable/disable plugin
base_url = "${API_URL}"          # External tool API URL
api_token = "${API_TOKEN}"       # Authentication token
<plugin_specific_options>        # Plugin-specific settings

[plugins.<plugin_name>.field_mappings.<resource_type>]
<field_name> = <mapping>         # Field mappings per resource type

Configuration Sections

1. Plugin Enable/Disable

[plugins.defectdojo]
enabled = true  # or false to disable

When disabled, the plugin:

• Will NOT be compiled into the binary
• Will NOT react to entity events
• Will NOT appear in plugin registry

2. Connection Settings

[plugins.defectdojo]
base_url = "${DEFECTDOJO_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

Security Best Practice: Always use environment variables for sensitive data (tokens, passwords, secrets).

3. Plugin-Specific Options

Each plugin may have unique configuration options:

[plugins.defectdojo]
default_product_type_id = 1      # Default product type for new products
auto_create_users = true          # Auto-create missing users
auto_create_product_types = false # Require manual product type creation

4. Field Mappings

Field mappings define how Charybdis entity fields map to external tool API fields.

[plugins.defectdojo.field_mappings.product]
name = "metadata.name"                    # Direct field mapping
description = "metadata.description"      # Direct field mapping
business_criticality = { value = "high" } # Static value
tags = "metadata.tags"                    # Array mapping

Field Mapping System

Mapping Types

The field mapping system supports three types of mappings:

1. Direct Field Mapping (String)

Maps a field from the Charybdis entity using dot notation:

name = "metadata.name"
description = "metadata.description"
email = "spec.profile.email"
tags = "metadata.tags"

Syntax: "<path.to.field>"

Examples:

• "metadata.name" → entity.metadata.name
• "spec.owner" → entity.spec.owner
• "metadata.annotations.version" → entity.metadata.annotations["version"]

2. Static Value Mapping (Object with value key)

Provides a static value regardless of entity content:

business_criticality = { value = "high" }
platform = { value = "web" }
is_active = { value = true }
priority = { value = 100 }
lifecycle = { value = "production" }

Syntax: { value = <any_json_value> }

Supported Types:

• String: { value = "text" }
• Boolean: { value = true }
• Number: { value = 42 }
• Object: { value = { key = "value" } }
• Array: { value = ["item1", "item2"] }

3. Complex Mapping with Entity Resolution (Object)

Resolves entity references and extracts data from related entities:

product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

Syntax:

<field> = {
  from = "<source_field>",                # Source field path
  resolve_entity = "<entity_kind>",       # Entity kind to resolve to
  extract = "<field_to_extract>",         # Field to extract from resolved entity
  resolve_array = <true|false>,           # Optional: resolve array of entities
  lookup_entity = "<entity_kind>"         # Optional: lookup linked entity
}

Parameters:

Parameter	Type	Required	Description
from	string	✓	Source field path in dot notation
resolve_entity	string	✓	Entity kind to resolve (User, Component, etc.)
extract	string	✓	Field path to extract from resolved entity
resolve_array	boolean	✗	Set to true if from contains array of entity IDs
lookup_entity	string	✗	Find linked entity by source ID

Entity Resolution

Entity resolution is the most powerful feature of the field mapping system. It allows plugins to navigate entity relationships and extract data from related entities.

How Entity Resolution Works

1. Start with source entity (e.g., Component)
   entity.spec.owner = "user:john.doe"

2. Extract source value
   from = "spec.owner" → "user:john.doe"

3. Resolve entity reference
   resolve_entity = "User" → Query for User entity with id "john.doe"

4. Extract target field
   extract = "annotations.defectdojo.com/user-id" → "123"

5. Result
   product_manager = 123

Single Entity Resolution

Resolve one entity reference:

[plugins.defectdojo.field_mappings.product]
product_manager = {
  from = "spec.owner",                              # Component.spec.owner = "user:john.doe"
  resolve_entity = "User",                           # Find User entity
  extract = "annotations.defectdojo.com/user-id"    # Get DD user ID → "123"
}

Array Entity Resolution

Resolve multiple entity references:

[plugins.defectdojo.field_mappings.product_member]
user_ids = {
  from = "spec.members",                             # Group.spec.members = ["user:john", "user:jane"]
  resolve_entity = "User",                           # Find each User entity
  extract = "annotations.defectdojo.com/user-id",   # Get DD user ID for each
  resolve_array = true                               # Process as array
}

Result: user_ids = [123, 124]

Linked Entity Lookup

Find an entity linked to the source entity:

engagement_product = {
  from = "id",                                       # Start with engagement entity ID
  lookup_entity = "Component",                       # Find Component where...
  extract = "annotations.defectdojo.com/product-id" # Component links to this engagement
}

This searches for a Component entity that references the source entity.

Nested Entity Resolution

Chain multiple resolutions:

team_lead_email = {
  from = "spec.owner",                    # Component.spec.owner = "group:platform-team"
  resolve_entity = "Group",               # Find Group entity
  extract = "spec.parent"                 # Get parent = "user:tech-lead"
}

# Then in a second mapping:
lead_id = {
  from = "spec.parent",                   # From previous resolution
  resolve_entity = "User",                # Find User entity
  extract = "annotations.defectdojo.com/user-id"
}

Entity Resolution Error Handling

The field mapper handles errors gracefully:

• Entity not found: Logs warning, field is omitted
• Annotation missing: Logs warning, field is omitted
• Invalid entity reference: Logs warning, field is omitted
• Array with missing entities: Logs warning, includes found entities only

In map_all() mode: Failed field mappings log warnings but don't fail entire operation.

Plugin Configuration Reference

DefectDojo Plugin

Complete configuration reference for the DefectDojo plugin.

Connection Settings

[plugins.defectdojo]
enabled = true
base_url = "${DEFECTDOJO_URL}"
api_token = "${DEFECTDOJO_API_TOKEN}"

Plugin Options

[plugins.defectdojo]
# Default product type ID for new products (required if not mapped)
default_product_type_id = 1

# Automatically create users in DefectDojo when referenced
# If false, referenced users must exist in DefectDojo
auto_create_users = true

# Automatically create product types in DefectDojo
# If false, product types must be created manually
auto_create_product_types = false

Resource Type: Product

Maps Component entities to DefectDojo Products.

[plugins.defectdojo.field_mappings.product]
# Required fields
name = "metadata.name"                      # Product name
description = "metadata.description"        # Product description
product_type_id = { value = 1 }            # Product type ID

# Optional fields
tags = "metadata.tags"                      # Product tags (array)
business_criticality = { value = "high" }   # very high, high, medium, low, very low, none
platform = { value = "web" }                # web, mobile, desktop, iot, etc.
lifecycle = { value = "production" }        # production, development, retirement
origin = { value = "internal" }             # internal, external, third party
user_records = { value = 1000000 }         # Number of user records
revenue = { value = "1000000" }            # Revenue amount
external_audience = { value = true }        # Accessible to external users
internet_accessible = { value = true }      # Accessible via internet

# User references (with entity resolution)
product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

technical_contact = {
  from = "metadata.annotations.technical-contact",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

team_manager = {
  from = "metadata.annotations.team-manager",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

Resource Type: User

Maps User entities to DefectDojo Users.

[plugins.defectdojo.field_mappings.user]
# Required fields
username = "metadata.name"                  # Username (unique)
email = "spec.profile.email"                # Email address (unique)

# Optional fields
first_name = "spec.profile.displayName"     # First name
last_name = "spec.profile.displayName"      # Last name (can use same field)
is_active = { value = true }                # User active status

Resource Type: Product Type

Maps System entities to DefectDojo Product Types.

[plugins.defectdojo.field_mappings.product_type]
# Required fields
name = "metadata.name"                      # Product type name

# Optional fields
description = "metadata.description"        # Product type description
critical_product = { value = false }        # Requires extra review
key_product = { value = true }              # Important but not critical

Resource Type: Product Member

Maps Group entities to DefectDojo Product Members (user-product-role assignments).

Note: Only Group entities with specific structure are processed.

[plugins.defectdojo.field_mappings.product_member]
# Product reference (resolve from Group parent)
product_id = {
  from = "spec.parent",                     # Group.spec.parent = "component:my-service"
  resolve_entity = "Component",             # Find Component entity
  extract = "annotations.defectdojo.com/product-id"  # Get DD product ID
}

# User reference (resolve from Group members)
user_id = {
  from = "spec.members",                    # Group.spec.members = ["user:john", "user:jane"]
  resolve_entity = "User",                  # Find User entities
  extract = "annotations.defectdojo.com/user-id",    # Get DD user IDs
  resolve_array = true                      # Process as array
}

# Role assignment
role_name = { value = "Reader" }            # Owner, Maintainer, Reader, Writer, API_Importer

DefectDojo Roles:

• Owner: Full control over product
• Maintainer: Can modify product settings
• Writer: Can add findings and tests
• Reader: Read-only access
• API_Importer: Can import findings via API

Resource Type: Engagement

Maps Resource entities to DefectDojo Engagements (security assessments).

[plugins.defectdojo.field_mappings.engagement]
# Required fields
name = "metadata.name"                      # Engagement name
description = "metadata.description"        # Engagement description

# Product reference
product_id = {
  from = "spec.owner",                      # Resource.spec.owner = "component:my-service"
  resolve_entity = "Component",             # Find Component entity
  extract = "annotations.defectdojo.com/product-id"
}

# Date fields (ISO 8601 format: YYYY-MM-DD)
target_start = "spec.target_start"          # Start date
target_end = "spec.target_end"              # End date

# Optional fields
status = { value = "In Progress" }          # Not Started, In Progress, Completed, Cancelled
engagement_type = { value = "CI/CD" }       # Interactive, CI/CD, etc.

# Lead user reference
lead_id = {
  from = "spec.dependsOn",                  # Resource.spec.dependsOn = ["user:lead"]
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

# Version control fields
version = "metadata.annotations.version"
commit_hash = "metadata.annotations.commit_hash"
branch_tag = "metadata.annotations.branch"
build_id = "metadata.annotations.build_id"
source_code_management_uri = "metadata.annotations.repo_url"

# Test type flags
deduplication_on_engagement = { value = true }
threat_model = { value = false }
api_test = { value = true }
pen_test = { value = false }
check_list = { value = false }

Environment Variables

Required Variables

# DefectDojo Plugin
export DEFECTDOJO_URL="https://defectdojo.example.com"
export DEFECTDOJO_API_TOKEN="your-api-token-here"

# Database
export DATABASE_URL="postgresql://user:pass@localhost/charybdis"

# Optional: Telemetry
export OTLP_ENDPOINT="http://localhost:4317"

Obtaining DefectDojo API Token

1. Log into DefectDojo UI
2. Go to User Profile → API Key
3. Click Generate or copy existing key
4. Set environment variable: export DEFECTDOJO_API_TOKEN="<token>"

Security Considerations

Never commit secrets to git:

• ✓ Use environment variables for tokens/passwords
• ✓ Add .env to .gitignore
• ✓ Use secret management (HashiCorp Vault, AWS Secrets Manager)
• ✗ Don't hardcode secrets in config.toml

Best Practices

1. Field Mapping Design

Start Simple:

# Start with required fields only
name = "metadata.name"
description = "metadata.description"

Add Optional Fields Gradually:

# Add optional fields as needed
tags = "metadata.tags"
business_criticality = { value = "medium" }

Use Entity Resolution Last:

# Complex mappings last (requires other entities to exist)
product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

2. Entity Creation Order

For entity resolution to work, create entities in this order:

1. Users (no dependencies)
2. Product Types (no dependencies)
3. Components (may reference Users)
4. Groups (references Components and Users)
5. Resources/Engagements (references Components and Users)

3. Testing Strategy

Test in Isolation:

1. Create a test Component without references
2. Verify Product created in DefectDojo
3. Check annotation added: defectdojo.com/product-id

Test with References:

1. Create User entity
2. Wait for DefectDojo sync
3. Create Component with spec.owner = "user:john.doe"
4. Verify product_manager set correctly in DefectDojo

Test Updates:

1. Update Component name
2. Verify Product name updated in DefectDojo

Test Deletes:

1. Delete Component
2. Verify Product deleted in DefectDojo

4. Error Handling

Check Logs: Plugin operations are logged with tracing:

INFO  DefectDojo plugin: Creating DefectDojo product for entity: component-123
INFO  DefectDojo plugin: Created DefectDojo product 456 for entity component-123
WARN  DefectDojo plugin: Entity component-789 has no DefectDojo product ID, skipping deletion
ERROR DefectDojo plugin: Failed to create product: API error 400: Invalid product type ID

Log Levels:

• INFO: Normal operations
• WARN: Non-critical issues (missing annotations, skipped operations)
• ERROR: Critical failures (API errors, invalid configuration)

5. Performance Optimization

Minimize Entity Resolution:

• Cache resolved IDs when possible
• Use static values when appropriate
• Avoid deep resolution chains

Batch Operations:

• Create multiple entities in succession
• Plugin handles each independently
• No manual batching needed

6. Configuration Management

Use Configuration Profiles:

# config.development.toml
[plugins.defectdojo]
base_url = "https://defectdojo-dev.example.com"
auto_create_users = true

# config.production.toml
[plugins.defectdojo]
base_url = "https://defectdojo.example.com"
auto_create_users = false  # Require manual user creation

Environment-Specific Settings:

# Development
export CONFIG_FILE="config.development.toml"

# Production
export CONFIG_FILE="config.production.toml"

Troubleshooting

Plugin Not Reacting to Events

Check:

1. Plugin enabled: plugins.defectdojo.enabled = true
2. Plugin compiled: Run cargo build
3. Entity kind matches trigger: Component → product handler
4. Event bus running: Check logs for event publications

Field Mapping Not Working

Check:

1. Field path correct: "metadata.name" not "meta.name"
2. Field exists in entity: Use YAML adapter to inspect entity
3. Static value syntax: { value = "text" } not "text"
4. Entity resolution: Resolved entity exists and has required annotation

Entity Resolution Fails

Check:

1. Referenced entity exists: user:john.doe exists in database
2. Referenced entity has annotation: defectdojo.com/user-id present
3. Entity kind correct: resolve_entity = "User" not "user"
4. Array syntax: resolve_array = true for array fields

API Errors

Common Errors:

400 Bad Request: Invalid product type ID
→ Check default_product_type_id or product_type_id mapping

401 Unauthorized: Invalid token
→ Check DEFECTDOJO_API_TOKEN is correct

404 Not Found: Product not found
→ Entity may not have been created yet

409 Conflict: User already exists
→ Normal for users (plugin detects and reuses)

Missing Annotations

If annotations not created:

1. Check entity update succeeded: repository.update() logs
2. Verify annotation key: defectdojo.com/product-id
3. Check database: SELECT annotations FROM entities WHERE id = '...'

Performance Issues

If plugin causing slowness:

1. Check DefectDojo API response times
2. Reduce entity resolution depth
3. Use static values where possible
4. Check network connectivity to DefectDojo

Examples

Example 1: Basic Product Creation

Entity (Component):

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  description: Payment processing microservice
  tags:
    - payment
    - critical
    - pci-dss
spec:
  type: service
  lifecycle: production
  owner: platform-team

Configuration:

[plugins.defectdojo.field_mappings.product]
name = "metadata.name"
description = "metadata.description"
tags = "metadata.tags"
product_type_id = { value = 1 }
business_criticality = { value = "very high" }
platform = { value = "web" }
lifecycle = "spec.lifecycle"

Result in DefectDojo:

{
  "id": 456,
  "name": "payment-service",
  "description": "Payment processing microservice",
  "tags": ["payment", "critical", "pci-dss"],
  "prod_type": 1,
  "business_criticality": "very high",
  "platform": "web",
  "lifecycle": "production"
}

Updated Entity:

metadata:
  annotations:
    defectdojo.com/product-id: "456"

Example 2: Product with User References

Entities:

User:

apiVersion: backstage.io/v1alpha1
kind: User
metadata:
  name: john.doe
spec:
  profile:
    displayName: John Doe
    email: john.doe@example.com
  memberOf:
    - platform-team

Component:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  annotations:
    technical-contact: user:jane.smith
spec:
  owner: user:john.doe

Configuration:

[plugins.defectdojo.field_mappings.product]
name = "metadata.name"
product_type_id = { value = 1 }

product_manager = {
  from = "spec.owner",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

technical_contact = {
  from = "metadata.annotations.technical-contact",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

Resolution Process:

1. Component created → Product handler triggered
2. Field mapper resolves spec.owner = "user:john.doe"
3. Queries for User entity with id "john.doe"
4. Extracts annotations.defectdojo.com/user-id = "123"
5. Sets product_manager: 123 in API call

Example 3: Product Member Assignment

Entities:

Component (already has DD product ID):

metadata:
  annotations:
    defectdojo.com/product-id: "456"

Users (already have DD user IDs):

# john.doe
metadata:
  annotations:
    defectdojo.com/user-id: "123"

# jane.smith
metadata:
  annotations:
    defectdojo.com/user-id: "124"

Group:

apiVersion: backstage.io/v1alpha1
kind: Group
metadata:
  name: payment-service-security-team
spec:
  type: team
  parent: component:payment-service
  members:
    - user:john.doe
    - user:jane.smith

Configuration:

[plugins.defectdojo.field_mappings.product_member]
product_id = {
  from = "spec.parent",
  resolve_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
}

user_id = {
  from = "spec.members",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id",
  resolve_array = true
}

role_name = { value = "Reader" }

Result:

• Creates Product Member: Product 456 + User 123 + Role "Reader"
• Creates Product Member: Product 456 + User 124 + Role "Reader"

Example 4: Engagement with Full Metadata

Entity:

apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: payment-service-q1-2025-assessment
  description: Q1 2025 Security Assessment
  annotations:
    version: "2.3.0"
    commit_hash: "abc123def456"
    branch: "release/2.3"
    build_id: "build-789"
    repo_url: "https://github.com/example/payment-service"
spec:
  type: security-assessment
  owner: component:payment-service
  dependsOn:
    - user:security-lead
  target_start: "2025-01-01"
  target_end: "2025-03-31"

Configuration:

[plugins.defectdojo.field_mappings.engagement]
name = "metadata.name"
description = "metadata.description"

product_id = {
  from = "spec.owner",
  resolve_entity = "Component",
  extract = "annotations.defectdojo.com/product-id"
}

lead_id = {
  from = "spec.dependsOn",
  resolve_entity = "User",
  extract = "annotations.defectdojo.com/user-id"
}

target_start = "spec.target_start"
target_end = "spec.target_end"
version = "metadata.annotations.version"
commit_hash = "metadata.annotations.commit_hash"
branch_tag = "metadata.annotations.branch"
build_id = "metadata.annotations.build_id"
source_code_management_uri = "metadata.annotations.repo_url"

status = { value = "In Progress" }
engagement_type = { value = "CI/CD" }
deduplication_on_engagement = { value = true }
api_test = { value = true }
pen_test = { value = false }

Result in DefectDojo:

{
  "id": 789,
  "name": "payment-service-q1-2025-assessment",
  "description": "Q1 2025 Security Assessment",
  "product": 456,
  "lead": 123,
  "target_start": "2025-01-01",
  "target_end": "2025-03-31",
  "version": "2.3.0",
  "commit_hash": "abc123def456",
  "branch_tag": "release/2.3",
  "build_id": "build-789",
  "source_code_management_uri": "https://github.com/example/payment-service",
  "status": "In Progress",
  "engagement_type": "CI/CD",
  "deduplication_on_engagement": true,
  "api_test": true,
  "pen_test": false
}

Advanced Topics

Custom Field Mapping Logic

For complex scenarios not covered by built-in mapping types, implement custom resource handlers.

Plugin Development

See docs/PLUGIN_DEVELOPMENT_GUIDE.md (future) for creating new plugins.

Entity Resolution Performance

Entity resolution requires database queries. For high-throughput scenarios:

• Use static values where possible
• Cache resolved IDs in memory
• Consider implementing a resolution cache layer

Multi-Environment Configuration

# Development
cargo run -- --config config.development.toml

# Staging
cargo run -- --config config.staging.toml

# Production
cargo run -- --config config.production.toml

Related Documentation

• DefectDojo Plugin README
• Plugin Implementation Guide
• Charybdis Architecture
• Field Mapper Tests

Support

For issues or questions:

1. Check this documentation
2. Review plugin logs
3. Test with minimal configuration
4. Open issue on GitHub with:
   • Configuration (sanitized)
   • Entity YAML
   • Error logs
   • Expected vs actual behavior