ouranos/docs/grafana_mcp.md

# Grafana MCP Server

## Overview

The Grafana MCP server provides AI/LLM access to Grafana dashboards, datasources, and APIs through the Model Context Protocol (MCP). It runs as a Docker container on **Miranda** and connects to the Grafana instance inside the [PPLG stack](pplg.md) on **Prospero** via the internal Incus network.

**Deployment Host:** miranda.incus  
**Port:** 25533 (HTTP MCP endpoint)  
**MCPO Proxy:** http://miranda.incus:25530/grafana  
**Grafana Backend:** http://prospero.incus:3000 (PPLG stack)

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                       MCP CLIENTS                                   │
│  VS Code/Cline │ OpenWebUI │ LobeChat │ Custom Applications        │
└───────────────────────────┬─────────────────────────────────────────┘
                            │
                ┌───────────┴──────────────┐
                │                          │
                ▼                          ▼
     Direct MCP (port 25533)    MCPO Proxy (port 25530)
     streamable-http             OpenAI-compatible API
                │                          │
                └──────────┬───────────────┘
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│  Miranda (miranda.incus)                                             │
│  ┌────────────────────────────────────────────────┐                  │
│  │  Grafana MCP Server (Docker)                   │                  │
│  │  mcp/grafana:latest                            │                  │
│  │  Container: grafana-mcp                        │                  │
│  │  :25533 → :8000                                │                  │
│  └─────────────────────┬──────────────────────────┘                  │
│                        │ HTTP (internal network)                     │
└────────────────────────┼─────────────────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────────────────────┐
│  Prospero (prospero.incus) — PPLG Stack                              │
│  ┌────────────────────────────────────────────────┐                  │
│  │  Grafana :3000                                 │                  │
│  │  Authenticated via Service Account Token       │                  │
│  └────────────────────────────────────────────────┘                  │
└──────────────────────────────────────────────────────────────────────┘
```

### Cross-Host Dependency

The Grafana MCP server on Miranda communicates with Grafana on Prospero over the Incus internal network (`prospero.incus:3000`). This means:

- **PPLG must be deployed first** — Grafana must be running before deploying the MCP server
- The connection uses Grafana's **internal HTTP port** (3000), not the external HTTPS endpoint
- Authentication is handled by a **Grafana service account token**, not Casdoor OAuth

## Terraform Resources

### Host Definition

Grafana MCP runs on Miranda, defined in `terraform/containers.tf`:

| Attribute | Value |
|-----------|-------|
| Image | noble |
| Role | mcp_docker_host |
| Security Nesting | true |
| AppArmor | unconfined |
| Proxy: mcp_containers | `0.0.0.0:25530-25539` → `127.0.0.1:25530-25539` |

### Dependencies

| Resource | Relationship |
|----------|--------------|
| prospero (PPLG) | Grafana backend — service account token auth on `:3000` |
| miranda (MCPO) | MCPO proxies Grafana MCP at `localhost:25533/mcp` |

## Ansible Deployment

### Prerequisites

1. **PPLG stack**: Grafana must be running on Prospero (`ansible-playbook pplg/deploy.yml`)
2. **Docker**: Docker must be installed on the target host (`ansible-playbook docker/deploy.yml`)
3. **Vault Secret**: `vault_grafana_service_account_token` must be set (see [Required Vault Secrets](#required-vault-secrets))

### Playbook

```bash
cd ansible
ansible-playbook grafana_mcp/deploy.yml
```

### Files

| File | Purpose |
|------|---------|
| `grafana_mcp/deploy.yml` | Main deployment playbook |
| `grafana_mcp/docker-compose.yml.j2` | Docker Compose template for the MCP server |

### Deployment Steps

1. **Pre-flight Check**: Verify Grafana is reachable on Prospero (`/api/health`)
2. **Create System User**: `grafana_mcp:grafana_mcp` system account
3. **Create Directory**: `/srv/grafana_mcp` with restricted permissions (750)
4. **Template Docker Compose**: Renders `docker-compose.yml.j2` with Grafana URL and service account token
5. **Start Container**: `docker compose up` via `community.docker.docker_compose_v2`
6. **Health Check**: Verifies the MCP endpoint is responding on `localhost:25533/mcp`

### Deployment Order

Grafana MCP must be deployed **after** PPLG and **before** MCPO:

```
pplg → docker → grafana_mcp → mcpo
```

This ensures Grafana is available before the MCP server starts, and MCPO can proxy to it.

## Docker Compose Configuration

The container is defined in `grafana_mcp/docker-compose.yml.j2`:

```yaml
services:
  grafana-mcp:
    image: mcp/grafana:latest
    container_name: grafana-mcp
    restart: unless-stopped
    ports:
      - "25533:8000"
    environment:
      - GRAFANA_URL=http://prospero.incus:3000
      - GRAFANA_SERVICE_ACCOUNT_TOKEN=<from vault>
    command: ["--transport", "streamable-http", "--address", "0.0.0.0:8000", "--tls-skip-verify"]
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://127.0.0.1:51433"
        syslog-format: rfc5424
        tag: "grafana-mcp"
```

Key configuration:
- **Transport**: `streamable-http` — standard MCP HTTP transport
- **TLS Skip Verify**: Enabled because Grafana is accessed over internal HTTP (not HTTPS)
- **Syslog**: Logs shipped to Alloy on localhost for forwarding to Loki

## Available Tools

The Grafana MCP server exposes tools for interacting with Grafana's API:

### Dashboard Operations
- Search and list dashboards
- Get dashboard details and panels
- Query panel data

### Datasource Operations
- List configured datasources
- Query datasources directly

### Alerting
- List alert rules
- Get alert rule details and status

### General
- Get Grafana health status
- Search across Grafana resources

> **Note:** The specific tools available depend on the `mcp/grafana` Docker image version. Use the MCPO Swagger docs at `http://miranda.incus:25530/docs` to see the current tool inventory.

## Client Configuration

### MCP Native Clients (Cline, Claude Desktop)

```json
{
  "mcpServers": {
    "grafana": {
      "type": "streamable-http",
      "url": "http://miranda.incus:25533/mcp"
    }
  }
}
```

### Via MCPO (OpenAI-Compatible)

Grafana MCP is automatically available through MCPO at:

```
http://miranda.incus:25530/grafana
```

This endpoint is OpenAI-compatible and can be used by OpenWebUI, LobeChat, or any OpenAI SDK client:

```python
import openai

client = openai.OpenAI(
    base_url="http://miranda.incus:25530/grafana",
    api_key="not-required"
)
```

### OpenWebUI / LobeChat

1. Navigate to **Settings → Tools → OpenAPI Servers**
2. Click **Add OpenAPI Server**
3. Configure:
   - **Name:** Grafana MCP
   - **URL:** `http://miranda.incus:25530/grafana`
   - **Authentication:** None (MCPO handles upstream auth)
4. Save and enable the Grafana tools

## Required Vault Secrets

Add to `ansible/inventory/group_vars/all/vault.yml`:

| Variable | Purpose |
|----------|---------|
| `vault_grafana_service_account_token` | Grafana service account token for MCP API access |

### Creating a Grafana Service Account Token

1. Log in to Grafana at `https://grafana.ouranos.helu.ca` (Casdoor SSO or local admin)
2. Navigate to **Administration → Service Accounts**
3. Click **Add service account**
   - **Name:** `mcp-server`
   - **Role:** `Viewer` (or `Editor` if write tools are needed)
4. Click **Add service account token**
   - **Name:** `mcp-token`
   - **Expiration:** No expiration (or set a rotation schedule)
5. Copy the generated token
6. Store in vault:

```bash
cd ansible
ansible-vault edit inventory/group_vars/all/vault.yml
```

```yaml
vault_grafana_service_account_token: "glsa_xxxxxxxxxxxxxxxxxxxx"
```

## Host Variables

**File:** `ansible/inventory/host_vars/miranda.incus.yml`

```yaml
# Grafana MCP Config
grafana_mcp_user: grafana_mcp
grafana_mcp_group: grafana_mcp
grafana_mcp_directory: /srv/grafana_mcp
grafana_mcp_port: 25533
grafana_mcp_grafana_host: prospero.incus
grafana_mcp_grafana_port: 3000
grafana_service_account_token: "{{ vault_grafana_service_account_token }}"
```

Miranda's services list includes `grafana_mcp`:

```yaml
services:
  - alloy
  - argos
  - docker
  - gitea_mcp
  - grafana_mcp
  - mcpo
  - neo4j_mcp
```

## Monitoring

### Syslog to Loki

The Grafana MCP container ships logs via Docker's syslog driver to Alloy on Miranda:

| Server | Syslog Port | Loki Tag |
|--------|-------------|----------|
| grafana-mcp | 51433 | `grafana-mcp` |

### Grafana Log Queries

Useful Loki queries in Grafana Explore:

```logql
# All Grafana MCP logs
{hostname="miranda.incus", job="grafana_mcp"}

# Errors only
{hostname="miranda.incus", job="grafana_mcp"} |= "error" or |= "ERROR"

# Tool invocations
{hostname="miranda.incus", job="grafana_mcp"} |= "tool"
```

### MCPO Aggregation

Grafana MCP is registered in MCPO's `config.json` as:

```json
{
  "grafana": {
    "type": "streamable-http",
    "url": "http://localhost:25533/mcp"
  }
}
```

MCPO exposes it at `http://miranda.incus:25530/grafana` with OpenAI-compatible API and Swagger documentation.

## Operations

### Start / Stop

```bash
ssh miranda.incus

# Docker container
sudo -u grafana_mcp docker compose -f /srv/grafana_mcp/docker-compose.yml up -d
sudo -u grafana_mcp docker compose -f /srv/grafana_mcp/docker-compose.yml down

# Or redeploy via Ansible
cd ansible
ansible-playbook grafana_mcp/deploy.yml
```

### Health Check

```bash
# Container status
ssh miranda.incus docker ps --filter name=grafana-mcp

# MCP endpoint
curl http://miranda.incus:25533/mcp

# Via MCPO
curl http://miranda.incus:25530/grafana/tools

# Grafana backend (from Miranda)
curl http://prospero.incus:3000/api/health
```

### Logs

```bash
# Docker container logs
ssh miranda.incus docker logs -f grafana-mcp

# Loki logs (via Grafana on Prospero)
# Query: {hostname="miranda.incus", job="grafana_mcp"}
```

## Troubleshooting

### Container Won't Start

```bash
ssh miranda.incus
sudo -u grafana_mcp docker compose -f /srv/grafana_mcp/docker-compose.yml logs
```

**Common causes:**
- Grafana on Prospero not running → check `ssh prospero.incus sudo systemctl status grafana-server`
- Invalid or expired service account token → regenerate in Grafana UI
- Port 25533 already in use → `ss -tlnp | grep 25533`
- Docker image pull failure → check Docker Hub access

### MCP Endpoint Returns Errors

**Verify service account token:**
```bash
curl -H "Authorization: Bearer YOUR_TOKEN" http://prospero.incus:3000/api/org
```

**Check container environment:**
```bash
ssh miranda.incus docker inspect grafana-mcp | jq '.[0].Config.Env'
```

### MCPO Not Exposing Grafana Tools

**Verify MCPO config:**
```bash
ssh miranda.incus cat /srv/mcpo/config.json | jq '.mcpServers.grafana'
```

**Restart MCPO:**
```bash
ssh miranda.incus sudo systemctl restart mcpo
```

### Grafana Unreachable from Miranda

**Test network connectivity:**
```bash
ssh miranda.incus curl -s http://prospero.incus:3000/api/health
```

If this fails, check:
- Prospero container is running: `incus list prospero`
- Grafana service is up: `ssh prospero.incus sudo systemctl status grafana-server`
- No firewall rules blocking inter-container traffic

## Security Considerations

✔ **Service Account Token** — Scoped to Viewer role, cannot modify Grafana configuration  
✔ **Internal Network** — MCP server only accessible within the Incus network  
✔ **Vault Storage** — Token stored encrypted in Ansible Vault  
✔ **No Public Exposure** — Neither the MCP endpoint nor the MCPO proxy are internet-facing  
⚠️ **Token Rotation** — Consider rotating the service account token periodically  
⚠️ **Access Control** — MCPO currently doesn't require authentication for tool access

## References

- [PPLG Stack Documentation](pplg.md) — Grafana deployment on Prospero
- [MCPO Documentation](mcpo.md) — MCP gateway that proxies Grafana MCP
- [Grafana MCP Server](https://github.com/grafana/mcp-grafana) — Upstream project
- [Model Context Protocol Specification](https://modelcontextprotocol.io/)
- [Ansible Practices](ansible.md)
- [Agathos Overview](agathos.md)