ouranos/docs/grafana_mcp.md

Grafana MCP Server

Overview

The Grafana MCP server provides AI/LLM access to Grafana dashboards, datasources, and APIs through the Model Context Protocol (MCP). It runs as a Docker container on Miranda and connects to the Grafana instance inside the PPLG stack on Prospero via the internal Incus network.

Deployment Host: miranda.incus
Port: 25533 (HTTP MCP endpoint)
MCPO Proxy: http://miranda.incus:25530/grafana
Grafana Backend: http://prospero.incus:3000 (PPLG stack)

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                       MCP CLIENTS                                   │
│  VS Code/Cline │ OpenWebUI │ LobeChat │ Custom Applications        │
└───────────────────────────┬─────────────────────────────────────────┘
                            │
                ┌───────────┴──────────────┐
                │                          │
                ▼                          ▼
     Direct MCP (port 25533)    MCPO Proxy (port 25530)
     streamable-http             OpenAI-compatible API
                │                          │
                └──────────┬───────────────┘
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│  Miranda (miranda.incus)                                             │
│  ┌────────────────────────────────────────────────┐                  │
│  │  Grafana MCP Server (Docker)                   │                  │
│  │  mcp/grafana:latest                            │                  │
│  │  Container: grafana-mcp                        │                  │
│  │  :25533 → :8000                                │                  │
│  └─────────────────────┬──────────────────────────┘                  │
│                        │ HTTP (internal network)                     │
└────────────────────────┼─────────────────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────────────────────┐
│  Prospero (prospero.incus) — PPLG Stack                              │
│  ┌────────────────────────────────────────────────┐                  │
│  │  Grafana :3000                                 │                  │
│  │  Authenticated via Service Account Token       │                  │
│  └────────────────────────────────────────────────┘                  │
└──────────────────────────────────────────────────────────────────────┘

Cross-Host Dependency

The Grafana MCP server on Miranda communicates with Grafana on Prospero over the Incus internal network (prospero.incus:3000). This means:

• PPLG must be deployed first — Grafana must be running before deploying the MCP server
• The connection uses Grafana's internal HTTP port (3000), not the external HTTPS endpoint
• Authentication is handled by a Grafana service account token, not Casdoor OAuth

Terraform Resources

Host Definition

Grafana MCP runs on Miranda, defined in terraform/containers.tf:

Attribute	Value
Image	noble
Role	mcp_docker_host
Security Nesting	true
AppArmor	unconfined
Proxy: mcp_containers	0.0.0.0:25530-25539 → 127.0.0.1:25530-25539

Dependencies

Resource	Relationship
prospero (PPLG)	Grafana backend — service account token auth on :3000
miranda (MCPO)	MCPO proxies Grafana MCP at localhost:25533/mcp

Ansible Deployment

Prerequisites

1. PPLG stack: Grafana must be running on Prospero (ansible-playbook pplg/deploy.yml)
2. Docker: Docker must be installed on the target host (ansible-playbook docker/deploy.yml)
3. Vault Secret: vault_grafana_service_account_token must be set (see Required Vault Secrets)

Playbook

cd ansible
ansible-playbook grafana_mcp/deploy.yml

Files

File	Purpose
grafana_mcp/deploy.yml	Main deployment playbook
grafana_mcp/docker-compose.yml.j2	Docker Compose template for the MCP server

Deployment Steps

1. Pre-flight Check: Verify Grafana is reachable on Prospero (/api/health)
2. Create System User: grafana_mcp:grafana_mcp system account
3. Create Directory: /srv/grafana_mcp with restricted permissions (750)
4. Template Docker Compose: Renders docker-compose.yml.j2 with Grafana URL and service account token
5. Start Container: docker compose up via community.docker.docker_compose_v2
6. Health Check: Verifies the MCP endpoint is responding on localhost:25533/mcp

Deployment Order

Grafana MCP must be deployed after PPLG and before MCPO:

pplg → docker → grafana_mcp → mcpo

This ensures Grafana is available before the MCP server starts, and MCPO can proxy to it.

Docker Compose Configuration

The container is defined in grafana_mcp/docker-compose.yml.j2:

services:
  grafana-mcp:
    image: mcp/grafana:latest
    container_name: grafana-mcp
    restart: unless-stopped
    ports:
      - "25533:8000"
    environment:
      - GRAFANA_URL=http://prospero.incus:3000
      - GRAFANA_SERVICE_ACCOUNT_TOKEN=<from vault>
    command: ["--transport", "streamable-http", "--address", "0.0.0.0:8000", "--tls-skip-verify"]
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://127.0.0.1:51433"
        syslog-format: rfc5424
        tag: "grafana-mcp"

Key configuration:

• Transport: streamable-http — standard MCP HTTP transport
• TLS Skip Verify: Enabled because Grafana is accessed over internal HTTP (not HTTPS)
• Syslog: Logs shipped to Alloy on localhost for forwarding to Loki

Available Tools

The Grafana MCP server exposes tools for interacting with Grafana's API:

Dashboard Operations

• Search and list dashboards
• Get dashboard details and panels
• Query panel data

Datasource Operations

• List configured datasources
• Query datasources directly

Alerting

• List alert rules
• Get alert rule details and status

General

• Get Grafana health status
• Search across Grafana resources

Note: The specific tools available depend on the mcp/grafana Docker image version. Use the MCPO Swagger docs at http://miranda.incus:25530/docs to see the current tool inventory.

Client Configuration

MCP Native Clients (Cline, Claude Desktop)

{
  "mcpServers": {
    "grafana": {
      "type": "streamable-http",
      "url": "http://miranda.incus:25533/mcp"
    }
  }
}

Via MCPO (OpenAI-Compatible)

Grafana MCP is automatically available through MCPO at:

http://miranda.incus:25530/grafana

This endpoint is OpenAI-compatible and can be used by OpenWebUI, LobeChat, or any OpenAI SDK client:

import openai

client = openai.OpenAI(
    base_url="http://miranda.incus:25530/grafana",
    api_key="not-required"
)

OpenWebUI / LobeChat

1. Navigate to Settings → Tools → OpenAPI Servers
2. Click Add OpenAPI Server
3. Configure:
   • Name: Grafana MCP
   • URL: http://miranda.incus:25530/grafana
   • Authentication: None (MCPO handles upstream auth)
4. Save and enable the Grafana tools

Required Vault Secrets

Add to ansible/inventory/group_vars/all/vault.yml:

Variable	Purpose
vault_grafana_service_account_token	Grafana service account token for MCP API access

Creating a Grafana Service Account Token

1. Log in to Grafana at https://grafana.ouranos.helu.ca (Casdoor SSO or local admin)
2. Navigate to Administration → Service Accounts
3. Click Add service account
   • Name: mcp-server
   • Role: Viewer (or Editor if write tools are needed)
4. Click Add service account token
   • Name: mcp-token
   • Expiration: No expiration (or set a rotation schedule)
5. Copy the generated token
6. Store in vault:

cd ansible
ansible-vault edit inventory/group_vars/all/vault.yml

vault_grafana_service_account_token: "glsa_xxxxxxxxxxxxxxxxxxxx"

Host Variables

File: ansible/inventory/host_vars/miranda.incus.yml

# Grafana MCP Config
grafana_mcp_user: grafana_mcp
grafana_mcp_group: grafana_mcp
grafana_mcp_directory: /srv/grafana_mcp
grafana_mcp_port: 25533
grafana_mcp_grafana_host: prospero.incus
grafana_mcp_grafana_port: 3000
grafana_service_account_token: "{{ vault_grafana_service_account_token }}"

Miranda's services list includes grafana_mcp:

services:
  - alloy
  - argos
  - docker
  - gitea_mcp
  - grafana_mcp
  - mcpo
  - neo4j_mcp

Monitoring

Syslog to Loki

The Grafana MCP container ships logs via Docker's syslog driver to Alloy on Miranda:

Server	Syslog Port	Loki Tag
grafana-mcp	51433	grafana-mcp

Grafana Log Queries

Useful Loki queries in Grafana Explore:

# All Grafana MCP logs
{hostname="miranda.incus", job="grafana_mcp"}

# Errors only
{hostname="miranda.incus", job="grafana_mcp"} |= "error" or |= "ERROR"

# Tool invocations
{hostname="miranda.incus", job="grafana_mcp"} |= "tool"

MCPO Aggregation

Grafana MCP is registered in MCPO's config.json as:

{
  "grafana": {
    "type": "streamable-http",
    "url": "http://localhost:25533/mcp"
  }
}

MCPO exposes it at http://miranda.incus:25530/grafana with OpenAI-compatible API and Swagger documentation.

Operations

Start / Stop

ssh miranda.incus

# Docker container
sudo -u grafana_mcp docker compose -f /srv/grafana_mcp/docker-compose.yml up -d
sudo -u grafana_mcp docker compose -f /srv/grafana_mcp/docker-compose.yml down

# Or redeploy via Ansible
cd ansible
ansible-playbook grafana_mcp/deploy.yml

Health Check

# Container status
ssh miranda.incus docker ps --filter name=grafana-mcp

# MCP endpoint
curl http://miranda.incus:25533/mcp

# Via MCPO
curl http://miranda.incus:25530/grafana/tools

# Grafana backend (from Miranda)
curl http://prospero.incus:3000/api/health

Logs

# Docker container logs
ssh miranda.incus docker logs -f grafana-mcp

# Loki logs (via Grafana on Prospero)
# Query: {hostname="miranda.incus", job="grafana_mcp"}

Troubleshooting

Container Won't Start

ssh miranda.incus
sudo -u grafana_mcp docker compose -f /srv/grafana_mcp/docker-compose.yml logs

Common causes:

• Grafana on Prospero not running → check ssh prospero.incus sudo systemctl status grafana-server
• Invalid or expired service account token → regenerate in Grafana UI
• Port 25533 already in use → ss -tlnp | grep 25533
• Docker image pull failure → check Docker Hub access

MCP Endpoint Returns Errors

Verify service account token:

curl -H "Authorization: Bearer YOUR_TOKEN" http://prospero.incus:3000/api/org

Check container environment:

ssh miranda.incus docker inspect grafana-mcp | jq '.[0].Config.Env'

MCPO Not Exposing Grafana Tools

Verify MCPO config:

ssh miranda.incus cat /srv/mcpo/config.json | jq '.mcpServers.grafana'

Restart MCPO:

ssh miranda.incus sudo systemctl restart mcpo

Grafana Unreachable from Miranda

Test network connectivity:

ssh miranda.incus curl -s http://prospero.incus:3000/api/health

If this fails, check:

• Prospero container is running: incus list prospero
• Grafana service is up: ssh prospero.incus sudo systemctl status grafana-server
• No firewall rules blocking inter-container traffic

Security Considerations

✔ Service Account Token — Scoped to Viewer role, cannot modify Grafana configuration
✔ Internal Network — MCP server only accessible within the Incus network
✔ Vault Storage — Token stored encrypted in Ansible Vault
✔ No Public Exposure — Neither the MCP endpoint nor the MCPO proxy are internet-facing
⚠️ Token Rotation — Consider rotating the service account token periodically
⚠️ Access Control — MCPO currently doesn't require authentication for tool access

References

• PPLG Stack Documentation — Grafana deployment on Prospero
• MCPO Documentation — MCP gateway that proxies Grafana MCP
• Grafana MCP Server — Upstream project
• Model Context Protocol Specification
• Ansible Practices
• Agathos Overview