# Charybdis Plugin Configuration Guide Complete reference for configuring and using the Charybdis plugin system. ## Table of Contents 1. [Overview](#overview) 2. [Plugin System Architecture](#plugin-system-architecture) 3. [Configuration File Structure](#configuration-file-structure) 4. [Field Mapping System](#field-mapping-system) 5. [Entity Resolution](#entity-resolution) 6. [Plugin Configuration Reference](#plugin-configuration-reference) 7. [Environment Variables](#environment-variables) 8. [Best Practices](#best-practices) 9. [Troubleshooting](#troubleshooting) 10. [Examples](#examples) ## Overview Charybdis plugins extend the core platform with integrations to external tools and services. Plugins react to entity lifecycle events (Create, Update, Delete) and automatically provision or sync resources in external systems. ### Plugin Types **Event-Driven Plugins** (Current Implementation): - React to entity lifecycle events - Push data to external systems - One-way sync: Charybdis → External Tool **Sync Plugins** (Future): - Pull data from external systems on schedule - Create/update entities in Charybdis - One-way sync: External Tool → Charybdis ## Plugin System Architecture ``` ┌─────────────────────────────────────────┐ │ Charybdis Core │ │ ┌────────────────────────────────┐ │ │ │ Entity Repository │ │ │ │ (PostgreSQL) │ │ │ └──────────┬─────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────┐ │ │ │ Event Bus │ │ │ │ (In-Memory / Future: Redis) │ │ │ └──────────┬─────────────────────┘ │ │ │ EntityEvent │ │ │ (Created/Updated/Deleted) │ └─────────────┼────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ Event Dispatcher │ │ (Routes events to plugins) │ └──────────┬──────────────────────────────┘ │ ├─────────────────┐ ▼ ▼ ┌──────────────────┐ ┌──────────────────┐ │ DefectDojo │ │ Future Plugins │ │ Plugin │ │ (DependencyTrack)│ │ │ │ │ │ ┌──────────────┐ │ └──────────────────┘ │ │ProductHandler│ │ │ │UserHandler │ │ │ │... │ │ │ └──────────────┘ │ └─────────┬────────┘ │ HTTP/REST ▼ ┌──────────────────┐ │ External Tool │ │ (DefectDojo) │ └──────────────────┘ ``` ### Key Components 1. **Plugin**: Top-level plugin interface 2. **ResourceHandler**: Handles specific resource types (e.g., Products, Users) 3. **FieldMapper**: Maps Charybdis entity fields to external tool fields 4. **Event Dispatcher**: Routes entity events to appropriate handlers 5. **Plugin Manager**: Manages plugin lifecycle and registration ## Configuration File Structure Plugin configuration lives in your main `config.toml` file under the `[plugins.]` section. ### Basic Structure ```toml [plugins.] enabled = true # Enable/disable plugin base_url = "${API_URL}" # External tool API URL api_token = "${API_TOKEN}" # Authentication token # Plugin-specific settings [plugins..field_mappings.] = # Field mappings per resource type ``` ### Configuration Sections #### 1. Plugin Enable/Disable ```toml [plugins.defectdojo] enabled = true # or false to disable ``` When disabled, the plugin: - Will NOT be compiled into the binary - Will NOT react to entity events - Will NOT appear in plugin registry #### 2. Connection Settings ```toml [plugins.defectdojo] base_url = "${DEFECTDOJO_URL}" api_token = "${DEFECTDOJO_API_TOKEN}" ``` **Security Best Practice**: Always use environment variables for sensitive data (tokens, passwords, secrets). #### 3. Plugin-Specific Options Each plugin may have unique configuration options: ```toml [plugins.defectdojo] default_product_type_id = 1 # Default product type for new products auto_create_users = true # Auto-create missing users auto_create_product_types = false # Require manual product type creation ``` #### 4. Field Mappings Field mappings define how Charybdis entity fields map to external tool API fields. ```toml [plugins.defectdojo.field_mappings.product] name = "metadata.name" # Direct field mapping description = "metadata.description" # Direct field mapping business_criticality = { value = "high" } # Static value tags = "metadata.tags" # Array mapping ``` ## Field Mapping System ### Mapping Types The field mapping system supports three types of mappings: #### 1. Direct Field Mapping (String) Maps a field from the Charybdis entity using dot notation: ```toml name = "metadata.name" description = "metadata.description" email = "spec.profile.email" tags = "metadata.tags" ``` **Syntax**: `""` **Examples**: - `"metadata.name"` → `entity.metadata.name` - `"spec.owner"` → `entity.spec.owner` - `"metadata.annotations.version"` → `entity.metadata.annotations["version"]` #### 2. Static Value Mapping (Object with `value` key) Provides a static value regardless of entity content: ```toml business_criticality = { value = "high" } platform = { value = "web" } is_active = { value = true } priority = { value = 100 } lifecycle = { value = "production" } ``` **Syntax**: `{ value = }` **Supported Types**: - String: `{ value = "text" }` - Boolean: `{ value = true }` - Number: `{ value = 42 }` - Object: `{ value = { key = "value" } }` - Array: `{ value = ["item1", "item2"] }` #### 3. Complex Mapping with Entity Resolution (Object) Resolves entity references and extracts data from related entities: ```toml product_manager = { from = "spec.owner", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } ``` **Syntax**: ```toml = { from = "", # Source field path resolve_entity = "", # Entity kind to resolve to extract = "", # Field to extract from resolved entity resolve_array = , # Optional: resolve array of entities lookup_entity = "" # Optional: lookup linked entity } ``` **Parameters**: | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `from` | string | ✓ | Source field path in dot notation | | `resolve_entity` | string | ✓ | Entity kind to resolve (User, Component, etc.) | | `extract` | string | ✓ | Field path to extract from resolved entity | | `resolve_array` | boolean | ✗ | Set to true if `from` contains array of entity IDs | | `lookup_entity` | string | ✗ | Find linked entity by source ID | ## Entity Resolution Entity resolution is the most powerful feature of the field mapping system. It allows plugins to navigate entity relationships and extract data from related entities. ### How Entity Resolution Works ``` 1. Start with source entity (e.g., Component) entity.spec.owner = "user:john.doe" 2. Extract source value from = "spec.owner" → "user:john.doe" 3. Resolve entity reference resolve_entity = "User" → Query for User entity with id "john.doe" 4. Extract target field extract = "annotations.defectdojo.com/user-id" → "123" 5. Result product_manager = 123 ``` ### Single Entity Resolution Resolve one entity reference: ```toml [plugins.defectdojo.field_mappings.product] product_manager = { from = "spec.owner", # Component.spec.owner = "user:john.doe" resolve_entity = "User", # Find User entity extract = "annotations.defectdojo.com/user-id" # Get DD user ID → "123" } ``` ### Array Entity Resolution Resolve multiple entity references: ```toml [plugins.defectdojo.field_mappings.product_member] user_ids = { from = "spec.members", # Group.spec.members = ["user:john", "user:jane"] resolve_entity = "User", # Find each User entity extract = "annotations.defectdojo.com/user-id", # Get DD user ID for each resolve_array = true # Process as array } ``` **Result**: `user_ids = [123, 124]` ### Linked Entity Lookup Find an entity linked to the source entity: ```toml engagement_product = { from = "id", # Start with engagement entity ID lookup_entity = "Component", # Find Component where... extract = "annotations.defectdojo.com/product-id" # Component links to this engagement } ``` This searches for a Component entity that references the source entity. ### Nested Entity Resolution Chain multiple resolutions: ```toml team_lead_email = { from = "spec.owner", # Component.spec.owner = "group:platform-team" resolve_entity = "Group", # Find Group entity extract = "spec.parent" # Get parent = "user:tech-lead" } # Then in a second mapping: lead_id = { from = "spec.parent", # From previous resolution resolve_entity = "User", # Find User entity extract = "annotations.defectdojo.com/user-id" } ``` ### Entity Resolution Error Handling The field mapper handles errors gracefully: - **Entity not found**: Logs warning, field is omitted - **Annotation missing**: Logs warning, field is omitted - **Invalid entity reference**: Logs warning, field is omitted - **Array with missing entities**: Logs warning, includes found entities only **In `map_all()` mode**: Failed field mappings log warnings but don't fail entire operation. ## Plugin Configuration Reference ### DefectDojo Plugin Complete configuration reference for the DefectDojo plugin. #### Connection Settings ```toml [plugins.defectdojo] enabled = true base_url = "${DEFECTDOJO_URL}" api_token = "${DEFECTDOJO_API_TOKEN}" ``` #### Plugin Options ```toml [plugins.defectdojo] # Default product type ID for new products (required if not mapped) default_product_type_id = 1 # Automatically create users in DefectDojo when referenced # If false, referenced users must exist in DefectDojo auto_create_users = true # Automatically create product types in DefectDojo # If false, product types must be created manually auto_create_product_types = false ``` #### Resource Type: Product Maps **Component** entities to DefectDojo Products. ```toml [plugins.defectdojo.field_mappings.product] # Required fields name = "metadata.name" # Product name description = "metadata.description" # Product description product_type_id = { value = 1 } # Product type ID # Optional fields tags = "metadata.tags" # Product tags (array) business_criticality = { value = "high" } # very high, high, medium, low, very low, none platform = { value = "web" } # web, mobile, desktop, iot, etc. lifecycle = { value = "production" } # production, development, retirement origin = { value = "internal" } # internal, external, third party user_records = { value = 1000000 } # Number of user records revenue = { value = "1000000" } # Revenue amount external_audience = { value = true } # Accessible to external users internet_accessible = { value = true } # Accessible via internet # User references (with entity resolution) product_manager = { from = "spec.owner", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } technical_contact = { from = "metadata.annotations.technical-contact", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } team_manager = { from = "metadata.annotations.team-manager", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } ``` #### Resource Type: User Maps **User** entities to DefectDojo Users. ```toml [plugins.defectdojo.field_mappings.user] # Required fields username = "metadata.name" # Username (unique) email = "spec.profile.email" # Email address (unique) # Optional fields first_name = "spec.profile.displayName" # First name last_name = "spec.profile.displayName" # Last name (can use same field) is_active = { value = true } # User active status ``` #### Resource Type: Product Type Maps **System** entities to DefectDojo Product Types. ```toml [plugins.defectdojo.field_mappings.product_type] # Required fields name = "metadata.name" # Product type name # Optional fields description = "metadata.description" # Product type description critical_product = { value = false } # Requires extra review key_product = { value = true } # Important but not critical ``` #### Resource Type: Product Member Maps **Group** entities to DefectDojo Product Members (user-product-role assignments). **Note**: Only Group entities with specific structure are processed. ```toml [plugins.defectdojo.field_mappings.product_member] # Product reference (resolve from Group parent) product_id = { from = "spec.parent", # Group.spec.parent = "component:my-service" resolve_entity = "Component", # Find Component entity extract = "annotations.defectdojo.com/product-id" # Get DD product ID } # User reference (resolve from Group members) user_id = { from = "spec.members", # Group.spec.members = ["user:john", "user:jane"] resolve_entity = "User", # Find User entities extract = "annotations.defectdojo.com/user-id", # Get DD user IDs resolve_array = true # Process as array } # Role assignment role_name = { value = "Reader" } # Owner, Maintainer, Reader, Writer, API_Importer ``` **DefectDojo Roles**: - **Owner**: Full control over product - **Maintainer**: Can modify product settings - **Writer**: Can add findings and tests - **Reader**: Read-only access - **API_Importer**: Can import findings via API #### Resource Type: Engagement Maps **Resource** entities to DefectDojo Engagements (security assessments). ```toml [plugins.defectdojo.field_mappings.engagement] # Required fields name = "metadata.name" # Engagement name description = "metadata.description" # Engagement description # Product reference product_id = { from = "spec.owner", # Resource.spec.owner = "component:my-service" resolve_entity = "Component", # Find Component entity extract = "annotations.defectdojo.com/product-id" } # Date fields (ISO 8601 format: YYYY-MM-DD) target_start = "spec.target_start" # Start date target_end = "spec.target_end" # End date # Optional fields status = { value = "In Progress" } # Not Started, In Progress, Completed, Cancelled engagement_type = { value = "CI/CD" } # Interactive, CI/CD, etc. # Lead user reference lead_id = { from = "spec.dependsOn", # Resource.spec.dependsOn = ["user:lead"] resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } # Version control fields version = "metadata.annotations.version" commit_hash = "metadata.annotations.commit_hash" branch_tag = "metadata.annotations.branch" build_id = "metadata.annotations.build_id" source_code_management_uri = "metadata.annotations.repo_url" # Test type flags deduplication_on_engagement = { value = true } threat_model = { value = false } api_test = { value = true } pen_test = { value = false } check_list = { value = false } ``` ## Environment Variables ### Required Variables ```bash # DefectDojo Plugin export DEFECTDOJO_URL="https://defectdojo.example.com" export DEFECTDOJO_API_TOKEN="your-api-token-here" # Database export DATABASE_URL="postgresql://user:pass@localhost/charybdis" # Optional: Telemetry export OTLP_ENDPOINT="http://localhost:4317" ``` ### Obtaining DefectDojo API Token 1. Log into DefectDojo UI 2. Go to **User Profile** → **API Key** 3. Click **Generate** or copy existing key 4. Set environment variable: `export DEFECTDOJO_API_TOKEN=""` ### Security Considerations **Never commit secrets to git**: - ✓ Use environment variables for tokens/passwords - ✓ Add `.env` to `.gitignore` - ✓ Use secret management (HashiCorp Vault, AWS Secrets Manager) - ✗ Don't hardcode secrets in `config.toml` ## Best Practices ### 1. Field Mapping Design **Start Simple**: ```toml # Start with required fields only name = "metadata.name" description = "metadata.description" ``` **Add Optional Fields Gradually**: ```toml # Add optional fields as needed tags = "metadata.tags" business_criticality = { value = "medium" } ``` **Use Entity Resolution Last**: ```toml # Complex mappings last (requires other entities to exist) product_manager = { from = "spec.owner", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } ``` ### 2. Entity Creation Order For entity resolution to work, create entities in this order: 1. **Users** (no dependencies) 2. **Product Types** (no dependencies) 3. **Components** (may reference Users) 4. **Groups** (references Components and Users) 5. **Resources/Engagements** (references Components and Users) ### 3. Testing Strategy **Test in Isolation**: 1. Create a test Component without references 2. Verify Product created in DefectDojo 3. Check annotation added: `defectdojo.com/product-id` **Test with References**: 1. Create User entity 2. Wait for DefectDojo sync 3. Create Component with `spec.owner = "user:john.doe"` 4. Verify product_manager set correctly in DefectDojo **Test Updates**: 1. Update Component name 2. Verify Product name updated in DefectDojo **Test Deletes**: 1. Delete Component 2. Verify Product deleted in DefectDojo ### 4. Error Handling **Check Logs**: Plugin operations are logged with `tracing`: ``` INFO DefectDojo plugin: Creating DefectDojo product for entity: component-123 INFO DefectDojo plugin: Created DefectDojo product 456 for entity component-123 WARN DefectDojo plugin: Entity component-789 has no DefectDojo product ID, skipping deletion ERROR DefectDojo plugin: Failed to create product: API error 400: Invalid product type ID ``` **Log Levels**: - `INFO`: Normal operations - `WARN`: Non-critical issues (missing annotations, skipped operations) - `ERROR`: Critical failures (API errors, invalid configuration) ### 5. Performance Optimization **Minimize Entity Resolution**: - Cache resolved IDs when possible - Use static values when appropriate - Avoid deep resolution chains **Batch Operations**: - Create multiple entities in succession - Plugin handles each independently - No manual batching needed ### 6. Configuration Management **Use Configuration Profiles**: ```toml # config.development.toml [plugins.defectdojo] base_url = "https://defectdojo-dev.example.com" auto_create_users = true # config.production.toml [plugins.defectdojo] base_url = "https://defectdojo.example.com" auto_create_users = false # Require manual user creation ``` **Environment-Specific Settings**: ```bash # Development export CONFIG_FILE="config.development.toml" # Production export CONFIG_FILE="config.production.toml" ``` ## Troubleshooting ### Plugin Not Reacting to Events **Check**: 1. Plugin enabled: `plugins.defectdojo.enabled = true` 2. Plugin compiled: Run `cargo build` 3. Entity kind matches trigger: Component → product handler 4. Event bus running: Check logs for event publications ### Field Mapping Not Working **Check**: 1. Field path correct: `"metadata.name"` not `"meta.name"` 2. Field exists in entity: Use YAML adapter to inspect entity 3. Static value syntax: `{ value = "text" }` not `"text"` 4. Entity resolution: Resolved entity exists and has required annotation ### Entity Resolution Fails **Check**: 1. Referenced entity exists: `user:john.doe` exists in database 2. Referenced entity has annotation: `defectdojo.com/user-id` present 3. Entity kind correct: `resolve_entity = "User"` not `"user"` 4. Array syntax: `resolve_array = true` for array fields ### API Errors **Common Errors**: ``` 400 Bad Request: Invalid product type ID → Check default_product_type_id or product_type_id mapping 401 Unauthorized: Invalid token → Check DEFECTDOJO_API_TOKEN is correct 404 Not Found: Product not found → Entity may not have been created yet 409 Conflict: User already exists → Normal for users (plugin detects and reuses) ``` ### Missing Annotations If annotations not created: 1. Check entity update succeeded: `repository.update()` logs 2. Verify annotation key: `defectdojo.com/product-id` 3. Check database: `SELECT annotations FROM entities WHERE id = '...'` ### Performance Issues If plugin causing slowness: 1. Check DefectDojo API response times 2. Reduce entity resolution depth 3. Use static values where possible 4. Check network connectivity to DefectDojo ## Examples ### Example 1: Basic Product Creation **Entity** (Component): ```yaml apiVersion: backstage.io/v1alpha1 kind: Component metadata: name: payment-service description: Payment processing microservice tags: - payment - critical - pci-dss spec: type: service lifecycle: production owner: platform-team ``` **Configuration**: ```toml [plugins.defectdojo.field_mappings.product] name = "metadata.name" description = "metadata.description" tags = "metadata.tags" product_type_id = { value = 1 } business_criticality = { value = "very high" } platform = { value = "web" } lifecycle = "spec.lifecycle" ``` **Result in DefectDojo**: ```json { "id": 456, "name": "payment-service", "description": "Payment processing microservice", "tags": ["payment", "critical", "pci-dss"], "prod_type": 1, "business_criticality": "very high", "platform": "web", "lifecycle": "production" } ``` **Updated Entity**: ```yaml metadata: annotations: defectdojo.com/product-id: "456" ``` ### Example 2: Product with User References **Entities**: User: ```yaml apiVersion: backstage.io/v1alpha1 kind: User metadata: name: john.doe spec: profile: displayName: John Doe email: john.doe@example.com memberOf: - platform-team ``` Component: ```yaml apiVersion: backstage.io/v1alpha1 kind: Component metadata: name: payment-service annotations: technical-contact: user:jane.smith spec: owner: user:john.doe ``` **Configuration**: ```toml [plugins.defectdojo.field_mappings.product] name = "metadata.name" product_type_id = { value = 1 } product_manager = { from = "spec.owner", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } technical_contact = { from = "metadata.annotations.technical-contact", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } ``` **Resolution Process**: 1. Component created → Product handler triggered 2. Field mapper resolves `spec.owner = "user:john.doe"` 3. Queries for User entity with id "john.doe" 4. Extracts `annotations.defectdojo.com/user-id = "123"` 5. Sets `product_manager: 123` in API call ### Example 3: Product Member Assignment **Entities**: Component (already has DD product ID): ```yaml metadata: annotations: defectdojo.com/product-id: "456" ``` Users (already have DD user IDs): ```yaml # john.doe metadata: annotations: defectdojo.com/user-id: "123" # jane.smith metadata: annotations: defectdojo.com/user-id: "124" ``` Group: ```yaml apiVersion: backstage.io/v1alpha1 kind: Group metadata: name: payment-service-security-team spec: type: team parent: component:payment-service members: - user:john.doe - user:jane.smith ``` **Configuration**: ```toml [plugins.defectdojo.field_mappings.product_member] product_id = { from = "spec.parent", resolve_entity = "Component", extract = "annotations.defectdojo.com/product-id" } user_id = { from = "spec.members", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id", resolve_array = true } role_name = { value = "Reader" } ``` **Result**: - Creates Product Member: Product 456 + User 123 + Role "Reader" - Creates Product Member: Product 456 + User 124 + Role "Reader" ### Example 4: Engagement with Full Metadata **Entity**: ```yaml apiVersion: backstage.io/v1alpha1 kind: Resource metadata: name: payment-service-q1-2025-assessment description: Q1 2025 Security Assessment annotations: version: "2.3.0" commit_hash: "abc123def456" branch: "release/2.3" build_id: "build-789" repo_url: "https://github.com/example/payment-service" spec: type: security-assessment owner: component:payment-service dependsOn: - user:security-lead target_start: "2025-01-01" target_end: "2025-03-31" ``` **Configuration**: ```toml [plugins.defectdojo.field_mappings.engagement] name = "metadata.name" description = "metadata.description" product_id = { from = "spec.owner", resolve_entity = "Component", extract = "annotations.defectdojo.com/product-id" } lead_id = { from = "spec.dependsOn", resolve_entity = "User", extract = "annotations.defectdojo.com/user-id" } target_start = "spec.target_start" target_end = "spec.target_end" version = "metadata.annotations.version" commit_hash = "metadata.annotations.commit_hash" branch_tag = "metadata.annotations.branch" build_id = "metadata.annotations.build_id" source_code_management_uri = "metadata.annotations.repo_url" status = { value = "In Progress" } engagement_type = { value = "CI/CD" } deduplication_on_engagement = { value = true } api_test = { value = true } pen_test = { value = false } ``` **Result in DefectDojo**: ```json { "id": 789, "name": "payment-service-q1-2025-assessment", "description": "Q1 2025 Security Assessment", "product": 456, "lead": 123, "target_start": "2025-01-01", "target_end": "2025-03-31", "version": "2.3.0", "commit_hash": "abc123def456", "branch_tag": "release/2.3", "build_id": "build-789", "source_code_management_uri": "https://github.com/example/payment-service", "status": "In Progress", "engagement_type": "CI/CD", "deduplication_on_engagement": true, "api_test": true, "pen_test": false } ``` ## Advanced Topics ### Custom Field Mapping Logic For complex scenarios not covered by built-in mapping types, implement custom resource handlers. ### Plugin Development See `docs/PLUGIN_DEVELOPMENT_GUIDE.md` (future) for creating new plugins. ### Entity Resolution Performance Entity resolution requires database queries. For high-throughput scenarios: - Use static values where possible - Cache resolved IDs in memory - Consider implementing a resolution cache layer ### Multi-Environment Configuration ```bash # Development cargo run -- --config config.development.toml # Staging cargo run -- --config config.staging.toml # Production cargo run -- --config config.production.toml ``` ## Related Documentation - [DefectDojo Plugin README](../plugins/defectdojo/README.md) - [Plugin Implementation Guide](./DEFECTDOJO_PLUGIN_IMPLEMENTATION.md) - [Charybdis Architecture](architecture.md) - [Field Mapper Tests](../src/plugins/field_mapper.rs#tests) ## Support For issues or questions: 1. Check this documentation 2. Review plugin logs 3. Test with minimal configuration 4. Open issue on GitHub with: - Configuration (sanitized) - Entity YAML - Error logs - Expected vs actual behavior