ouranos/docs/rabbitmq.md

RabbitMQ - Message Broker Infrastructure

Overview

RabbitMQ 3 (management-alpine) serves as the central message broker for the Ouranos sandbox, providing AMQP-compliant message queuing for asynchronous communication between services. The deployment includes the management web interface for monitoring and administration.

Host: Oberon (container_orchestration)
Role: Message broker for event-driven architectures
AMQP Port: 5672
Management Port: 25582
Syslog Port: 51402 (Alloy)

Architecture

┌─────────────────────────────────────────────────────────┐
│                      Oberon Host                        │
│                                                         │
│  ┌──────────────────────────────────────────────────┐  │
│  │        RabbitMQ Container (Docker)               │  │
│  │                                                  │  │
│  │  ┌──────────────┬──────────────┐                │  │
│  │  │  VHost       │  VHost       │                │  │
│  │  │  "kairos"    │  "spelunker" │                │  │
│  │  │              │              │                │  │
│  │  │  User:       │  User:       │                │  │
│  │  │  kairos      │  spelunker   │                │  │
│  │  │  (full perm) │  (full perm) │                │  │
│  │  └──────────────┴──────────────┘                │  │
│  │                                                  │  │
│  │  Default Admin: rabbitmq                        │  │
│  │  (all vhosts, admin privileges)                 │  │
│  │                                                  │  │
│  └──────────────────────────────────────────────────┘  │
│                                                         │
│  Ports: 5672 (AMQP), 25582 (Management)                │
│  Logs: syslog → Alloy:51402 → Loki                     │
└─────────────────────────────────────────────────────────┘

  ┌──────────────┐            ┌──────────────┐
  │   Kairos     │───AMQP────▶│  kairos/     │
  │  (future)    │            │  (vhost)     │
  └──────────────┘            └──────────────┘

  ┌──────────────┐            ┌──────────────┐
  │  Spelunker   │───AMQP────▶│  spelunker/  │
  │  (future)    │            │  (vhost)     │
  └──────────────┘            └──────────────┘

Note: Kairos and Spelunker are future services. The RabbitMQ infrastructure is pre-provisioned with dedicated virtual hosts and users ready for when these services are deployed.

Terraform Resources

Oberon Host Definition

RabbitMQ runs on Oberon, defined in terraform/containers.tf:

Attribute	Value
Description	Docker Host + MCP Switchboard - King of Fairies orchestrating containers
Image	noble
Role	container_orchestration
Security Nesting	true (required for Docker)
AppArmor Profile	unconfined
Proxy Devices	25580-25599 → 25580-25599 (application port range)

Container Dependencies

Resource	Relationship
Docker	RabbitMQ runs as a Docker container on Oberon
Alloy	Collects syslog logs from RabbitMQ on port 51402
Prospero	Receives logs via Loki for observability

Ansible Deployment

Playbook

cd ansible
ansible-playbook rabbitmq/deploy.yml

Files

File	Purpose
rabbitmq/deploy.yml	Main deployment playbook
rabbitmq/docker-compose.yml.j2	Docker Compose template

Deployment Steps

The playbook performs the following operations:

1. User and Group Management

   • Creates rabbitmq system user and group
   • Adds ponos user to rabbitmq group for operational access

2. Directory Setup

   • Creates service directory at /srv/rabbitmq
   • Sets ownership to rabbitmq:rabbitmq
   • Configures permissions (mode 750)

3. Docker Compose Deployment

   • Templates docker-compose.yml from Jinja2 template
   • Deploys RabbitMQ container with docker compose up

4. rabbitmqadmin CLI Setup

   • Extracts rabbitmqadmin from container to /usr/local/bin/
   • Makes it executable for host-level management

5. Automatic Provisioning (idempotent)

   • Creates virtual hosts: kairos, spelunker
   • Creates users with passwords from vault
   • Sets user tags (currently none, expandable for admin/monitoring roles)
   • Configures full permissions for each user on their respective vhost

Variables

Host Variables (host_vars/oberon.incus.yml)

Variable	Description	Default
rabbitmq_user	Service user	rabbitmq
rabbitmq_group	Service group	rabbitmq
rabbitmq_directory	Installation directory	/srv/rabbitmq
rabbitmq_amqp_port	AMQP protocol port	5672
rabbitmq_management_port	Management web interface	25582
rabbitmq_password	Default admin password	{{ vault_rabbitmq_password }}

Group Variables (group_vars/all/vars.yml)

Defines the provisioning configuration for vhosts, users, and permissions:

rabbitmq_vhosts:
  - name: kairos
  - name: spelunker

rabbitmq_users:
  - name: kairos
    password: "{{ kairos_rabbitmq_password }}"
    tags: []
  - name: spelunker
    password: "{{ spelunker_rabbitmq_password }}"
    tags: []

rabbitmq_permissions:
  - vhost: kairos
    user: kairos
    configure_priv: .*
    read_priv: .*
    write_priv: .*
  - vhost: spelunker
    user: spelunker
    configure_priv: .*
    read_priv: .*
    write_priv: .*

Vault Variable Mappings:

kairos_rabbitmq_password: "{{ vault_kairos_rabbitmq_password }}"
spelunker_rabbitmq_password: "{{ vault_spelunker_rabbitmq_password }}"

Vault Variables (group_vars/all/vault.yml)

All sensitive credentials are encrypted in the vault:

Variable	Description
vault_rabbitmq_password	Default admin account password
vault_kairos_rabbitmq_password	Kairos service user password
vault_spelunker_rabbitmq_password	Spelunker service user password

Configuration

Docker Compose Template

The deployment uses a minimal Docker Compose configuration:

services:
  rabbitmq:
    image: rabbitmq:3-management-alpine
    container_name: rabbitmq
    restart: unless-stopped
    ports:
      - "{{rabbitmq_amqp_port}}:5672"      # AMQP protocol
      - "{{rabbitmq_management_port}}:15672" # Management UI
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq    # Persistent data
    environment:
      RABBITMQ_DEFAULT_USER: "{{rabbitmq_user}}"
      RABBITMQ_DEFAULT_PASS: "{{rabbitmq_password}}"
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://127.0.0.1:{{rabbitmq_syslog_port}}"
        syslog-format: "{{syslog_format}}"
        tag: "rabbitmq"

Data Persistence

• Volume: rabbitmq_data (Docker-managed volume)
• Location: /var/lib/rabbitmq inside container
• Contents:
  • Message queues and persistent messages
  • Virtual host metadata
  • User credentials and permissions
  • Configuration overrides

Virtual Hosts and Users

Default Admin Account

Username: rabbitmq
Password: {{ vault_rabbitmq_password }} (from vault)
Privileges: Full administrative access to all virtual hosts

The default admin account is created automatically when the container starts and can access:

• All virtual hosts (including /, kairos, spelunker)
• Management web interface
• All RabbitMQ management commands

Kairos Virtual Host

VHost: kairos
User: kairos
Password: {{ vault_kairos_rabbitmq_password }}
Permissions: Full (configure, read, write) on all resources matching .*

Intended for the Kairos service (event-driven time-series processing system, planned future deployment).

Spelunker Virtual Host

VHost: spelunker
User: spelunker
Password: {{ vault_spelunker_rabbitmq_password }}
Permissions: Full (configure, read, write) on all resources matching .*

Intended for the Spelunker service (log exploration and analytics platform, planned future deployment).

Permission Model

Both service users have full access within their respective virtual hosts:

Permission	Pattern	Description
Configure	.*	Create/delete queues, exchanges, bindings
Write	.*	Publish messages to exchanges
Read	.*	Consume messages from queues

This isolation ensures:

• ✔ Each service operates in its own namespace
• ✔ Messages cannot cross between services
• ✔ Resource limits can be applied per-vhost
• ✔ Service credentials can be rotated independently

Access and Administration

Management Web Interface

URL: http://oberon.incus:25582
External: http://{oberon-ip}:25582
Login: rabbitmq / {{ vault_rabbitmq_password }}

Features:

• Queue inspection and message browsing
• Exchange and binding management
• Connection and channel monitoring
• User and permission administration
• Virtual host management
• Performance metrics and charts

CLI Administration

On Host Machine (using rabbitmqadmin)

# List vhosts
rabbitmqadmin -H oberon.incus -P 25582 -u rabbitmq -p PASSWORD list vhosts

# List queues in a vhost
rabbitmqadmin -H oberon.incus -P 25582 -u rabbitmq -p PASSWORD -V kairos list queues

# Publish a test message
rabbitmqadmin -H oberon.incus -P 25582 -u rabbitmq -p PASSWORD -V kairos publish \
  exchange=amq.default routing_key=test payload="test message"

Inside Container

# Enter the container
docker exec -it rabbitmq /bin/sh

# List vhosts
rabbitmqctl list_vhosts

# List users
rabbitmqctl list_users

# List permissions for a user
rabbitmqctl list_user_permissions kairos

# List queues in a vhost
rabbitmqctl list_queues -p kairos

# Check node status
rabbitmqctl status

Connection Strings

AMQP Connection (from other containers on Oberon)

amqp://kairos:PASSWORD@localhost:5672/kairos
amqp://spelunker:PASSWORD@localhost:5672/spelunker

AMQP Connection (from other hosts)

amqp://kairos:PASSWORD@oberon.incus:5672/kairos
amqp://spelunker:PASSWORD@oberon.incus:5672/spelunker

Management API

http://rabbitmq:PASSWORD@oberon.incus:25582/api/

Monitoring and Observability

Logging

• Driver: syslog (Docker logging driver)
• Destination: tcp://127.0.0.1:51402 (Alloy on Oberon)
• Tag: rabbitmq
• Format: {{ syslog_format }} (from Alloy configuration)

Logs are collected by Alloy and forwarded to Loki on Prospero for centralized log aggregation.

Key Metrics (via Management UI)

Metric	Description
Connections	Active AMQP client connections
Channels	Active channels within connections
Queues	Total queues across all vhosts
Messages	Ready, unacknowledged, and total message counts
Message Rate	Publish/deliver rates (msg/s)
Memory Usage	Container memory consumption
Disk Usage	Persistent storage utilization

Health Check

# Check if RabbitMQ is running
docker ps | grep rabbitmq

# Check container logs
docker logs rabbitmq

# Check RabbitMQ node status
docker exec rabbitmq rabbitmqctl status

# Check cluster health (single-node, should show 1 node)
docker exec rabbitmq rabbitmqctl cluster_status

Operational Tasks

Restart RabbitMQ

# Via Docker Compose
cd /srv/rabbitmq
sudo -u rabbitmq docker compose restart

# Via Docker directly
docker restart rabbitmq

Recreate Container (preserves data)

cd /srv/rabbitmq
sudo -u rabbitmq docker compose down
sudo -u rabbitmq docker compose up -d

Add New Virtual Host and User

1. Update group_vars/all/vars.yml:

   rabbitmq_vhosts:
     - name: newservice
   
   rabbitmq_users:
     - name: newservice
       password: "{{ newservice_rabbitmq_password }}"
       tags: []
   
   rabbitmq_permissions:
     - vhost: newservice
       user: newservice
       configure_priv: .*
       read_priv: .*
       write_priv: .*
   
   # Add mapping
   newservice_rabbitmq_password: "{{ vault_newservice_rabbitmq_password }}"
   

2. Add password to group_vars/all/vault.yml:

   ansible-vault edit inventory/group_vars/all/vault.yml
   # Add: vault_newservice_rabbitmq_password: "secure_password"
   

3. Run the playbook:

   ansible-playbook rabbitmq/deploy.yml
   

The provisioning tasks are idempotent—existing vhosts and users are skipped, only new ones are created.

Rotate User Password

# Inside container
docker exec rabbitmq rabbitmqctl change_password kairos "new_password"

# Update vault
ansible-vault edit inventory/group_vars/all/vault.yml
# Update vault_kairos_rabbitmq_password

Clear All Messages in a Queue

docker exec rabbitmq rabbitmqctl purge_queue queue_name -p kairos

Troubleshooting

Container Won't Start

Check Docker logs for errors:

docker logs rabbitmq

Common issues:

• Port conflict on 5672 or 25582
• Permission issues on /srv/rabbitmq directory
• Corrupted data volume

Cannot Connect to Management UI

1. Verify port mapping: docker port rabbitmq
2. Check firewall rules on Oberon
3. Verify container is running: docker ps | grep rabbitmq
4. Check if management plugin is enabled (should be in -management-alpine image)

User Authentication Failing

# List users and verify they exist
docker exec rabbitmq rabbitmqctl list_users

# Check user permissions
docker exec rabbitmq rabbitmqctl list_user_permissions kairos

# Verify vhost exists
docker exec rabbitmq rabbitmqctl list_vhosts

High Memory Usage

RabbitMQ may consume significant memory with many messages. Check:

# Memory usage
docker exec rabbitmq rabbitmqctl status | grep memory

# Queue depths
docker exec rabbitmq rabbitmqctl list_queues -p kairos messages

# Consider setting memory limits in docker-compose.yml

Security Considerations

Network Isolation

• RabbitMQ AMQP port (5672) is only exposed on the Incus network (10.10.0.0/16)
• Management UI (25582) is exposed externally for administration
• For production: Place HAProxy in front of management UI with authentication
• Consider enabling SSL/TLS for AMQP connections in production

Credential Management

• ✔ All passwords stored in Ansible Vault
• ✔ Service accounts have isolated virtual hosts
• ✔ Default admin account uses strong password from vault
• ⚠️ Credentials passed as environment variables (visible in docker inspect)
• Consider using Docker secrets or Vault integration for enhanced security

Virtual Host Isolation

Each service operates in its own virtual host:

• Messages cannot cross between vhosts
• Resource quotas can be applied per-vhost
• Credentials can be rotated without affecting other services

Future Enhancements

• SSL/TLS Support: Enable encrypted AMQP connections
• Cluster Mode: Add additional RabbitMQ nodes for high availability
• Federation: Connect to external RabbitMQ clusters
• Prometheus Exporter: Add metrics export for Grafana monitoring
• Shovel Plugin: Configure message forwarding between brokers
• HAProxy Integration: Reverse proxy for management UI with authentication
• Docker Secrets: Replace environment variables with Docker secrets

References

• RabbitMQ Official Documentation
• RabbitMQ Management Plugin
• AMQP 0-9-1 Protocol Reference
• Virtual Hosts
• Access Control (Authentication, Authorisation)
• Monitoring RabbitMQ

---

Last Updated: February 12, 2026
Project: Ouranos Infrastructure
Approval: Red Panda Approved™