doc: update and cleanup
CI / Check (push) Successful in 13m13s
CI / Format (push) Successful in 48s
CI / Clippy (push) Successful in 12m13s
CI / Test (push) Successful in 12m45s

This commit is contained in:
Guillaume GRABÉ
2026-06-09 11:26:47 +02:00
parent ab13af48e7
commit f1b1543f29
11 changed files with 1113 additions and 3376 deletions
+162 -339
View File
@@ -1,39 +1,26 @@
# Getting Started with Charybdis
This guide will get you from zero to a running Charybdis instance in minutes. By the end, you'll have a working software catalog that can register services via gRPC, ingest scan results, and auto-trigger integrations via event-driven plugins.
This guide gets you from zero to a running Charybdis instance: a software catalog with a gRPC API, optional Backstage YAML adapter, and plugin integrations on entity events.
## Prerequisites
- **Rust** - 1.70 or later ([install](https://rustup.rs/))
- **PostgreSQL** - 14 or later
- **grpcurl** - For testing (optional, [install](https://github.com/fullstorydev/grpcurl))
- **Rust** 1.70+ ([install](https://rustup.rs/))
- **PostgreSQL** 14+
- **protoc** + `libprotobuf-dev` (for the build script)
- **grpcurl** for testing (optional, [install](https://github.com/fullstorydev/grpcurl))
## Installation
### Option 1: From Source
## Install
```bash
# Clone the repository
git clone https://github.com/charybdis-catalog/charybdis.git
git clone <your-charybdis-repo-url>
cd charybdis
# Build
cargo build --release
# The binary will be at target/release/charybdis
```
### Option 2: Docker (Coming Soon)
```bash
docker pull charybdis/charybdis:latest
# Binary at target/release/charybdis-server
```
## Quick Start
### Step 1: Start PostgreSQL
Using Docker:
### 1. Start PostgreSQL
```bash
docker run -d \
@@ -43,54 +30,22 @@ docker run -d \
postgres:15
```
Or use an existing PostgreSQL instance.
### Step 2: Configure Charybdis
**Recommended: Use config.toml**
Copy the example configuration:
### 2. Configure
```bash
cp config.toml.example config.toml
```
Edit `config.toml` and set your database URL:
```toml
[database]
url = "${DATABASE_URL}"
```
Set the environment variable:
```bash
export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
```
**Alternative: Environment Variables Only (Legacy)**
`config.toml` reads `${DATABASE_URL}` from the environment. See [Configuration](#configuration) below for all options.
If you prefer environment variables:
```bash
# Database
export DATABASE_URL="postgresql://postgres:mysecretpassword@localhost:5432/postgres"
# Disable security for quick start
export SECURITY_MTLS_ENABLED=false
export SECURITY_RBAC_ENABLED=false
# Logging
export RUST_LOG=info,charybdis=debug
```
### Step 3: Run Charybdis
### 3. Run
```bash
cargo run
```
You should see:
Expected startup logs:
```
INFO charybdis: Database ready
@@ -98,40 +53,34 @@ INFO charybdis: Event bus started successfully
INFO charybdis: EntityService server listening on [::1]:50051
```
### Step 4: Verify It's Working
Test with grpcurl:
### 4. Verify
```bash
# List available services
grpcurl -plaintext localhost:50051 list
# Output:
# charybdis.entities.EntityService
# charybdis.ingestion.IngestionService
# grpc.reflection.v1.ServerReflection
```
Congratulations! Charybdis is running! 🎉
## Creating Your First Entity
### Using grpcurl
Create a service entity:
## Create Your First Entity
```bash
grpcurl -plaintext \
-d '{
"entity": {
"kind": "Service",
"service_metadata": {
"name": "payment-service",
"namespace": "production",
"description": "Core payment processing service"
}
grpcurl -plaintext -d '{
"entity": {
"kind": "Component",
"component_metadata": {
"name": "payment-api",
"namespace": "production",
"description": "Payment processing service",
"tags": ["api", "critical"]
},
"component_spec": {
"type": "service",
"lifecycle": "production",
"owner": "team-payments"
}
}' \
localhost:50051 charybdis.entities.EntityService/CreateEntity
}
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
```
Response:
@@ -140,117 +89,60 @@ Response:
{
"entity": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"kind": "Service",
"serviceMetadata": {
"name": "payment-service",
"namespace": "production",
"description": "Core payment processing service"
},
"createdAt": "2025-11-04T10:00:00Z",
"updatedAt": "2025-11-04T10:00:00Z"
"kind": "Component",
"componentMetadata": { "...": "..." },
"createdAt": "2026-06-01T10:00:00Z",
"updatedAt": "2026-06-01T10:00:00Z"
}
}
```
### List All Entities
```bash
grpcurl -plaintext -d '{}' \
localhost:50051 charybdis.entities.EntityService/ListEntities
```
### Get Entity by ID
### Retrieve and list
```bash
# Get by ID
grpcurl -plaintext \
-d '{"id": "550e8400-e29b-41d4-a716-446655440000"}' \
localhost:50051 charybdis.entities.EntityService/GetEntity
# List all
grpcurl -plaintext -d '{}' \
localhost:50051 charybdis.entities.EntityService/ListEntities
# Filter by kind and name
grpcurl -plaintext \
-d '{"kind": "Component", "name": "payment-api"}' \
localhost:50051 charybdis.entities.EntityService/ListEntities
```
## Entity Types
## Entity Kinds
Charybdis supports three main entity types:
The valid `kind` values are: `Component`, `System`, `API`, `User`, `Group`, `Domain`, `Resource`, `Finding`. Each kind uses a matching `<kind>_metadata` + `<kind>_spec` payload.
### Service
Conceptual reference: [core-concepts.md](core-concepts.md). Per-kind protobuf definitions: `proto/core/*.proto`.
Individual microservices or applications:
```bash
grpcurl -plaintext -d '{
"entity": {
"kind": "Service",
"service_metadata": {
"name": "user-api",
"namespace": "production",
"description": "User management API"
}
}
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
```
### System
Collections of related services:
```bash
grpcurl -plaintext -d '{
"entity": {
"kind": "System",
"system_metadata": {
"name": "payment-system",
"namespace": "production",
"description": "Complete payment processing system"
}
}
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
```
### Component
Reusable components or libraries:
### Component example with metadata and annotations
```bash
grpcurl -plaintext -d '{
"entity": {
"kind": "Component",
"component_spec": {
"type": "library",
"lifecycle": "production",
"owner": "platform-team"
},
"component_metadata": {
"name": "auth-library",
"namespace": "shared",
"description": "Shared authentication library"
}
}
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
```
## Adding Metadata
Entities support rich metadata:
```bash
grpcurl -plaintext -d '{
"entity": {
"kind": "Service",
"service_metadata": {
"name": "payment-service",
"namespace": "production",
"description": "Payment processing",
"labels": {
"team": "payments",
"tier": "critical"
},
"links": [
{
"url": "https://dashboard.company.com/payments",
"title": "Dashboard",
"icon": "dashboard"
}
],
"tags": ["payments", "pci-compliant", "critical"]
"labels": { "team": "payments", "tier": "critical" },
"links": [{
"url": "https://dashboard.company.com/payments",
"title": "Dashboard",
"icon": "dashboard"
}],
"tags": ["payments", "pci-compliant"]
},
"component_spec": {
"type": "service",
"lifecycle": "production",
"owner": "team-payments"
},
"annotations": {
"github.com/repo-slug": "myorg/payment-service",
@@ -260,105 +152,85 @@ grpcurl -plaintext -d '{
}' localhost:50051 charybdis.entities.EntityService/CreateEntity
```
## Integrating with CI/CD
## Registering Entities from CI
### Example: GitHub Actions
Create a workflow to register services automatically:
Charybdis is designed to be called from pipelines. With `grpcurl` available in your runner:
```yaml
name: Register Service
on:
push:
branches: [main]
jobs:
register:
runs-on: ubuntu-latest
steps:
- name: Register in Charybdis
run: |
grpcurl -plaintext \
-d '{
"entity": {
"kind": "Service",
"service_metadata": {
"name": "${{ github.event.repository.name }}",
"namespace": "production",
"description": "${{ github.event.repository.description }}"
},
"annotations": {
"github.com/repo-slug": "${{ github.repository }}"
}
}
}' \
your-charybdis-host:50051 \
charybdis.entities.EntityService/CreateEntity
```
### Example: GitLab CI
```yaml
register_service:
stage: deploy
script:
- |
grpcurl -plaintext \
-d "{
\"entity\": {
\"kind\": \"Service\",
\"service_metadata\": {
\"name\": \"${CI_PROJECT_NAME}\",
\"namespace\": \"${CI_ENVIRONMENT_NAME}\"
}
# Example pipeline step (Gitea Actions / GitHub Actions syntax)
- name: Register service in Charybdis
run: |
grpcurl -plaintext \
-d "{
\"entity\": {
\"kind\": \"Component\",
\"component_metadata\": {
\"name\": \"$CI_PROJECT_NAME\",
\"namespace\": \"production\"
},
\"component_spec\": {
\"type\": \"service\",
\"lifecycle\": \"production\",
\"owner\": \"$CI_PROJECT_NAMESPACE\"
},
\"annotations\": {
\"repo-slug\": \"$CI_PROJECT_PATH\"
}
}" \
your-charybdis-host:50051 \
charybdis.entities.EntityService/CreateEntity
}
}" \
charybdis.internal:50051 \
charybdis.entities.EntityService/CreateEntity
```
In production, secure the endpoint with mTLS (see [Enabling Security](#enabling-security)).
## Enabling Security
For production use, enable mTLS and RBAC:
Charybdis ships with mTLS + RBAC disabled for local exploration. For shared or production environments, enable both.
### Step 1: Generate Certificates
### 1. Generate dev certificates
```bash
# Use the provided test script
./test-mtls-rbac.sh
./deploy/scripts/generate-dev-certs.sh
```
This creates:
- `certs/ca.pem` - Certificate Authority
- `certs/server-cert.pem` / `server-key.pem` - Server certificate
- `certs/admin-cert.pem` / `admin-key.pem` - Admin client certificate
This writes `deploy/certs/` with:
- `ca.pem` CA
- `server-cert.pem` / `server-key.pem` — server
- `admin-cert.pem` / `admin-key.pem` — admin client
- (and per-role client certs)
### Step 2: Enable Security
### 2. Enable in `config.toml`
```bash
export SECURITY_MTLS_ENABLED=true
export SECURITY_MTLS_SERVER_CERT=./certs/server-cert.pem
export SECURITY_MTLS_SERVER_KEY=./certs/server-key.pem
export SECURITY_MTLS_CLIENT_CA=./certs/ca.pem
export SECURITY_RBAC_ENABLED=true
```toml
[security.mtls]
enabled = true
server_cert = "./deploy/certs/server-cert.pem"
server_key = "./deploy/certs/server-key.pem"
client_ca_cert = "./deploy/certs/ca.pem"
[security.rbac]
enabled = true
```
### Step 3: Test with mTLS
Restart Charybdis. RBAC defaults map cert OUs to roles (`platform-team` → full access, `automation` → CRUD without delete, `plugins` → read-only).
### 3. Call with mTLS
```bash
grpcurl \
-cacert certs/ca.pem \
-cert certs/admin-cert.pem \
-key certs/admin-key.pem \
-cacert deploy/certs/ca.pem \
-cert deploy/certs/admin-cert.pem \
-key deploy/certs/admin-key.pem \
-d '{}' \
localhost:50051 charybdis.entities.EntityService/ListEntities
```
See the [Security Guide](security.md) for detailed configuration.
Full reference (custom role mappings, audit logging, reverse-proxy mode): [security.md](security.md).
## Backstage Migration (Optional)
If you currently use Backstage and want to migrate gradually, the built-in YAML adapter serves entities in Backstage format:
If you currently run Backstage, Charybdis serves entities in Backstage's Location YAML format:
```yaml
# backstage app-config.yaml
@@ -367,78 +239,36 @@ catalog:
- type: url
target: http://your-charybdis-host:8080/yaml/locations
rules:
- allow: [Component, System, Service]
- allow: [Component, System, API, User, Group]
```
Backstage will automatically discover and import entities from Charybdis. You can run both in parallel — entities managed via gRPC are immediately visible in Backstage.
## Troubleshooting
### Port Already in Use
```
Error: transport error
```
**Solution**: Check if another process is using port 50051:
```bash
lsof -ti:50051
```
Kill the process or change the port:
```bash
export GRPC_PORT=50052
```
### Database Connection Failed
```
Error: password authentication failed
```
**Solution**: Verify your DATABASE_URL:
```bash
# Test connection
psql "$DATABASE_URL" -c "SELECT 1;"
```
### Permission Denied (with security enabled)
```
Code: PermissionDenied
Message: Role 'X' does not have permission 'Y'
```
**Solution**: Check your certificate and role mappings. See [Security Guide](security.md).
## Next Steps
Now that you have Charybdis running:
1. Learn about [Core Concepts](core-concepts.md) — entities, vulnerabilities, events
2. Read the [Vision & Roadmap](../VISION.md) — where Charybdis is going
3. Configure [Security](security.md) for production (mTLS + RBAC)
4. Explore [Plugins](../plugins/README.md) for external integrations
Backstage discovers entities by polling the endpoint. Entities created via gRPC are visible on the next poll. The YAML adapter is served by an HTTP listener separate from the gRPC port (default `:8080`).
## Configuration
Charybdis supports two configuration methods:
Charybdis loads its config from (in order):
### 1. Configuration File (Recommended)
1. `./config.toml`
2. `./charybdis.toml`
3. `/etc/charybdis/config.toml`
4. Environment variables (fallback)
Use `config.toml` for structured configuration:
### `config.toml` skeleton
```toml
# config.toml
[server]
grpc_host = "[::1]"
grpc_port = 50051
[server.yaml_adapter]
enabled = true
host = "0.0.0.0"
port = 8080
[database]
url = "${DATABASE_URL}" # Environment variable substitution
url = "${DATABASE_URL}"
max_connections = 10
connection_timeout_secs = 30
[security.mtls]
enabled = false
@@ -450,59 +280,52 @@ enabled = false
service_name = "charybdis"
environment = "development"
enable_console = true
```
**Benefits:**
- ✅ Organized by section (server, database, security, telemetry, plugins)
- ✅ Environment variable substitution with `${VAR_NAME}`
- ✅ Comments and documentation inline
- ✅ Easy to version control (excluding secrets)
- ✅ No need to export dozens of environment variables
**Using Environment Variables in config.toml:**
```toml
[database]
url = "${DATABASE_URL}" # Will be substituted at runtime
[plugins.defectdojo]
api_key = "${DEFECTDOJO_API_KEY}" # Secrets stay in environment
enabled = false
# see plugins/README.md for the full plugin reference
```
Then set only the secrets:
`${VAR}` and `${VAR:-default}` substitution works in any string value — keep secrets in the environment, not in the file.
```bash
export DATABASE_URL="postgresql://..."
export DEFECTDOJO_API_KEY="secret-key"
```
### Environment-variable fallback
### 2. Environment Variables (Legacy)
If `config.toml` is not found, Charybdis falls back to environment variables:
If no config file is found, these env vars are read:
| Variable | Default | Description |
|----------|---------|-------------|
|---|---|---|
| `DATABASE_URL` | (required) | PostgreSQL connection string |
| `GRPC_HOST` | `[::1]` | gRPC server bind address |
| `GRPC_PORT` | `50051` | gRPC server port |
| `RUST_LOG` | `info` | Logging level |
| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS authentication |
| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC authorization |
| `OTEL_ENABLE_CONSOLE` | `true` | Enable console logging |
| `GRPC_HOST` | `[::1]` | gRPC bind address |
| `GRPC_PORT` | `50051` | gRPC port |
| `RUST_LOG` | `info` | Logging level filter |
| `SECURITY_MTLS_ENABLED` | `false` | Enable mTLS |
| `SECURITY_RBAC_ENABLED` | `false` | Enable RBAC |
| `OTEL_ENABLE_CONSOLE` | `true` | Console exporter |
| `OTEL_SERVICE_NAME` | `charybdis` | Service name for telemetry |
### Configuration File Locations
See `config.toml.example` for the complete template.
Charybdis looks for configuration files in this order:
## Troubleshooting
1. `./config.toml` (current directory)
2. `./charybdis.toml`
3. `/etc/charybdis/config.toml` (Linux/Unix)
### Port already in use
```
Error: transport error
```
Find and free port 50051: `lsof -ti:50051 | xargs kill`, or set `GRPC_PORT=50052`.
If none are found, it uses environment variables.
### Database connection failed
Verify the URL: `psql "$DATABASE_URL" -c "SELECT 1;"`.
See `config.toml.example` for a complete configuration template with all options documented.
### Permission denied with security enabled
```
Code: PermissionDenied
Message: Role 'X' does not have permission 'Y'
```
Inspect your cert subject (OU determines the role) and the `[security.rbac.permissions]` table in `config.toml`. Reference: [security.md](security.md).
---
## Next Steps
**Need help?** Check the [troubleshooting guide](troubleshooting.md) or [open an issue](../../issues).
- [Core Concepts](core-concepts.md) — entity model, events, annotations
- [Architecture](architecture.md) — protobuf schema, storage, event bus
- [Plugins](../plugins/README.md) — DefectDojo, Keycloak, writing your own
- [Vision & Roadmap](../VISION.md) — where Charybdis is going