# MCPO - Model Context Protocol OpenAI-Compatible Proxy

## Overview

MCPO is an OpenAI-compatible proxy that aggregates multiple Model Context Protocol (MCP) servers behind a single HTTP endpoint. It acts as the central MCP gateway for the Ouranos sandbox, exposing tools from 13 MCP servers through a unified REST API with interactive Swagger documentation.

**Host:** miranda.incus  
**Role:** MCP Docker Host  
**Service Port:** 25530  
**API Docs:** http://miranda.incus:25530/docs

## Architecture

```
┌───────────────┐     ┌──────────────────────────────────────────────────────────┐
│  LLM Client   │     │  Miranda (miranda.incus)                                │
│  (LobeChat,   │────▶│  ┌────────────────────────────────────────────────────┐  │
│   Open WebUI, │     │  │  MCPO :25530                                      │  │
│   VS Code)    │     │  │  OpenAI-compatible proxy                          │  │
└───────────────┘     │  └─────┬────────────┬────────────┬───────────────────┘  │
                      │        │            │            │                       │
                      │  ┌─────▼─────┐ ┌────▼────┐ ┌────▼─────┐                │
                      │  │  stdio    │ │  Local  │ │  Remote  │                │
                      │  │  servers  │ │  Docker │ │  servers │                │
                      │  │           │ │  MCP    │ │          │                │
                      │  │ • time    │ │         │ │ • athena │                │
                      │  │ • ctx7    │ │ • neo4j │ │ • github │                │
                      │  │           │ │ • graf  │ │ • hface  │                │
                      │  │           │ │ • gitea │ │ • argos  │                │
                      │  │           │ │         │ │ • rommie │                │
                      │  │           │ │         │ │ • caliban│                │
                      │  │           │ │         │ │ • korax  │                │
                      │  └───────────┘ └─────────┘ └──────────┘                │
                      └──────────────────────────────────────────────────────────┘
```

MCPO manages two categories of MCP servers:
- **stdio servers**: MCPO spawns and manages the process (time, context7)
- **streamable-http servers**: MCPO proxies to Docker containers on localhost or remote services across the Incus network

## Terraform Resources

### Host Definition

MCPO runs on Miranda, defined in `terraform/containers.tf`:

| Attribute | Value |
|-----------|-------|
| Image | noble |
| Role | mcp_docker_host |
| Security Nesting | true |
| AppArmor | unconfined |
| Proxy: mcp_containers | `0.0.0.0:25530-25539` → `127.0.0.1:25530-25539` |
| Proxy: mcpo_ports | `0.0.0.0:25560-25569` → `127.0.0.1:25560-25569` |

### Dependencies

| Resource | Relationship |
|----------|--------------|
| prospero | Monitoring (Alloy → Loki, Prometheus) |
| ariel | Neo4j database for neo4j-cypher and neo4j-memory MCP servers |
| puck | Athena MCP server |
| caliban | Caliban and Rommie MCP servers |

## Ansible Deployment

### Playbook

```bash
cd ansible
ansible-playbook mcpo/deploy.yml
```

### Files

| File | Purpose |
|------|---------|
| `mcpo/deploy.yml` | Main deployment playbook |
| `mcpo/config.json.j2` | MCP server configuration template |
| `mcpo/mcpo.service.j2` | Systemd service unit template |
| `mcpo/restart.yml` | Restart playbook with health check |
| `mcpo/requirements.txt` | Python package requirements |

### Deployment Steps

1. **Create System User**: `mcpo:mcpo` system account
2. **Create Directory**: `/srv/mcpo` with restricted permissions
3. **Backup Config**: Saves existing `config.json` before overwriting
4. **Template Config**: Renders `config.json.j2` with MCP server definitions
5. **Install Node.js 22.x**: NodeSource repository for npx-based MCP servers
6. **Install Python 3.12**: System packages for virtual environment
7. **Create Virtual Environment**: Python 3.12 venv at `/srv/mcpo/.venv`
8. **Install pip Packages**: `wheel`, `mcpo`, `mcp-server-time`
9. **Pre-install Context7**: Downloads `@upstash/context7-mcp` via npx
10. **Deploy Systemd Service**: Enables and starts `mcpo.service`
11. **Health Check**: Verifies `http://localhost:25530/docs` returns HTTP 200

## MCP Servers

MCPO aggregates the following MCP servers in `config.json`:

### stdio Servers (managed by MCPO)

| Server | Command | Purpose |
|--------|---------|---------|
| `time` | `mcp-server-time` (Python venv) | Current time with timezone support |
| `upstash-context7` | `npx @upstash/context7-mcp` | Library documentation lookup |

### streamable-http Servers (local Docker containers)

| Server | URL | Purpose |
|--------|-----|---------|
| `neo4j-cypher` | `localhost:25531/mcp` | Neo4j Cypher query execution |
| `neo4j-memory` | `localhost:25532/mcp` | Neo4j knowledge graph memory |
| `grafana` | `localhost:25533/mcp` | Grafana dashboard and API integration |
| `gitea` | `localhost:25535/mcp` | Gitea repository management |

### streamable-http Servers (remote services)

| Server | URL | Purpose |
|--------|-----|---------|
| `argos-searxng` | `miranda.incus:25534/mcp` | SearXNG search integration |
| `athena` | `puck.incus:22461/mcp` | Athena knowledge service (auth required) |
| `github` | `api.githubcopilot.com/mcp/` | GitHub API integration |
| `rommie` | `caliban.incus:8080/mcp` | Rommie agent interface |
| `caliban` | `caliban.incus:22021/mcp` | Caliban computer use agent |
| `korax` | `korax.helu.ca:22021/mcp` | Korax external agent |
| `huggingface` | `huggingface.co/mcp` | Hugging Face model hub |

## Configuration

### Systemd Service

MCPO runs as a systemd service:

```
ExecStart=/srv/mcpo/.venv/bin/mcpo --port 25530 --config /srv/mcpo/config.json
```

- **User:** mcpo
- **Restart:** always (3s delay)
- **WorkingDirectory:** /srv/mcpo

### Storage Locations

| Path | Purpose | Owner |
|------|---------|-------|
| `/srv/mcpo` | Service directory | mcpo:mcpo |
| `/srv/mcpo/.venv` | Python virtual environment | mcpo:mcpo |
| `/srv/mcpo/config.json` | MCP server configuration | mcpo:mcpo |
| `/srv/mcpo/config.json.bak` | Config backup (pre-deploy) | mcpo:mcpo |

## Required Vault Secrets

Add to `ansible/inventory/group_vars/all/vault.yml`:

| Variable | Purpose |
|----------|---------|
| `vault_athena_mcp_auth` | Bearer token for Athena MCP server |
| `vault_github_personal_access_token` | GitHub personal access token |
| `vault_huggingface_mcp_token` | Hugging Face API token |
| `vault_gitea_mcp_access_token` | Gitea personal access token for MCP |

```bash
ansible-vault edit inventory/group_vars/all/vault.yml
```

## Host Variables

**File:** `ansible/inventory/host_vars/miranda.incus.yml`

```yaml
# MCPO Config
mcpo_user: mcpo
mcpo_group: mcpo
mcpo_directory: /srv/mcpo
mcpo_port: 25530
argos_mcp_url: http://miranda.incus:25534/mcp
athena_mcp_auth: "{{ vault_athena_mcp_auth }}"
athena_mcp_url: http://puck.incus:22461/mcp
github_personal_access_token: "{{ vault_github_personal_access_token }}"
neo4j_cypher_mcp_port: 25531
neo4j_memory_mcp_port: 25532
caliban_mcp_url: http://caliban.incus:22021/mcp
korax_mcp_url: http://korax.helu.ca:22021/mcp
huggingface_mcp_token: "{{ vault_huggingface_mcp_token }}"
gitea_mcp_port: 25535
```

## Monitoring

### Loki Logs

MCPO logs are collected via systemd journal by Alloy on Miranda. A relabel rule in Alloy's config tags `mcpo.service` journal entries with `job="mcpo"` so they appear as a dedicated app in Grafana dashboards.

| Log Source | Labels |
|------------|--------|
| Systemd journal | `{job="mcpo", hostname="miranda.incus"}` |

The Docker-based MCP servers (neo4j, grafana, gitea) each have dedicated syslog ports forwarded to Loki:

| Server | Syslog Port | Loki Job |
|--------|-------------|----------|
| neo4j-cypher | 51431 | `neo4j-cypher` |
| neo4j-memory | 51432 | `neo4j-memory` |
| grafana-mcp | 51433 | `grafana_mcp` |
| argos | 51434 | `argos` |
| gitea-mcp | 51435 | `gitea-mcp` |

### Grafana

Query MCPO-related logs in Grafana Explore:

```
{hostname="miranda.incus", job="mcpo"}
{hostname="miranda.incus", job="gitea-mcp"}
{hostname="miranda.incus", job="grafana_mcp"}
```

## Operations

### Start/Stop

```bash
ssh miranda.incus

# MCPO service
sudo systemctl start mcpo
sudo systemctl stop mcpo
sudo systemctl restart mcpo

# Or use the restart playbook with health check
cd ansible
ansible-playbook mcpo/restart.yml
```

### Health Check

```bash
# API docs endpoint
curl http://miranda.incus:25530/docs

# From Miranda itself
curl http://localhost:25530/docs
```

### Logs

```bash
# MCPO systemd journal
ssh miranda.incus "sudo journalctl -u mcpo -f"

# Docker MCP server logs
ssh miranda.incus "docker logs -f gitea-mcp"
ssh miranda.incus "docker logs -f grafana-mcp"
```

### Adding a New MCP Server

1. Add the server definition to `ansible/mcpo/config.json.j2`
2. Add any required variables to `ansible/inventory/host_vars/miranda.incus.yml`
3. Add vault secrets (if needed) to `inventory/group_vars/all/vault.yml`
4. If Docker-based: create a new `ansible/{service}/deploy.yml` and `docker-compose.yml.j2`
5. If Docker-based: add a syslog port to Miranda's host vars and Alloy config
6. Redeploy: `ansible-playbook mcpo/deploy.yml`

## Troubleshooting

### Common Issues

| Symptom | Cause | Resolution |
|---------|-------|------------|
| MCPO won't start | Config JSON syntax error | Check `config.json` with `python -m json.tool` |
| Server shows "unavailable" | Backend MCP server not running | Check Docker containers or remote service status |
| Context7 timeout on first use | npx downloading package | Wait for download to complete, or re-run pre-install |
| Health check fails | Port not ready | Increase retry delay, check `journalctl -u mcpo` |
| stdio server crash loops | Missing runtime dependency | Verify Python venv and Node.js installation |

### Debug Commands

```bash
# Check MCPO service status
ssh miranda.incus "sudo systemctl status mcpo"

# Validate config.json syntax
ssh miranda.incus "python3 -m json.tool /srv/mcpo/config.json"

# List Docker MCP containers
ssh miranda.incus "docker ps --filter name=mcp"

# Test a specific MCP server endpoint
ssh miranda.incus "curl -s http://localhost:25531/mcp | head"

# Check MCPO port is listening
ssh miranda.incus "ss -tlnp | grep 25530"
```

## References

- **MCPO Repository**: https://github.com/nicobailey/mcpo
- **MCP Specification**: https://modelcontextprotocol.io/
- [Ansible Practices](ansible.md)
- [Ouranos Overview](ouranos.md)