ouranos/docs/kernos.md

# Kernos Service Documentation

HTTP-enabled MCP shell server using FastMCP. Wraps the existing `mcp-shell-server` execution logic with FastMCP's HTTP transport for remote AI agent access.

Kernos is Project 202.

## Features

- **HTTP Transport**: Accessible via URL instead of stdio
- **API Key Authentication**: Optional per-request authentication for MCP endpoints
- **Health Endpoints**: `/live`, `/ready`, `/health` for Kubernetes-style probes
- **Prometheus Metrics**: `/metrics` endpoint for monitoring
- **JSON Structured Logging**: Production-ready log format with correlation IDs
- **Full Security**: Command whitelisting and API key authentication

## Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/mcp/` | POST | MCP protocol endpoint (FastMCP handles this) |
| `/live` | GET | Liveness probe - always returns 200 |
| `/ready` | GET | Readiness probe - checks executor and config |
| `/health` | GET | Combined health check |
| `/metrics` | GET | Prometheus metrics (text/plain) or JSON |

## Ansible Playbooks

### Stage Playbook

```bash
ansible-playbook kernos/stage.yml
```

Fetches the Kernos repository from clio and creates a release tarball at `~/rel/kernos_{{kernos_rel}}.tar`.

### Deploy Playbook

```bash
ansible-playbook kernos/deploy.yml
```

Deploys Kernos to caliban.incus:
1. Creates kernos user/group
2. Creates `/srv/kernos` directory
3. Transfers and extracts the staged tarball
4. Creates Python virtual environment
5. Installs package dependencies
6. Templates `.env` configuration
7. Templates systemd service file
8. Enables and starts the service
9. Validates health endpoints

## Configuration Variables

### Host Variables (`ansible/inventory/host_vars/caliban.incus.yml`)

| Variable | Description |
|----------|-------------|
| `kernos_user`| System user for the service |
| `kernos_group` | System group for the service |
| `kernos_directory` | Installation directory |
| `kernos_port` | HTTP server port |
| `kernos_host` | Server bind address |
| `kernos_log_level` | Python log level |
| `kernos_log_format` | Log format (`json` or `text`) |
| `kernos_environment` | Environment name for logging |
| `kernos_allow_commands` | Comma-separated command whitelist |
| `kernos_api_keys` | Comma-separated API keys for authentication (optional) |

### Global Variables (`ansible/inventory/group_vars/all/vars.yml`)

| Variable | Default | Description |
|----------|---------|-------------|
| `kernos_rel` | `master` | Git branch/tag for staging |

## Allowed Commands

The following commands are whitelisted for execution:

```
ls, cat, head, tail, grep, find, wc, file, stat, mkdir, touch, cp, mv, rm, 
chmod, pwd, tree, du, df, sed, awk, sort, uniq, cut, tr, tee, curl, wget, 
ping, nc, dig, host, ps, pgrep, kill, pkill, nohup, timeout, python3, pip, 
node, npm, npx, pnpm, git, make, tar, gzip, gunzip, zip, unzip, whoami, id, 
uname, hostname, date, uptime, free, which, env, printenv, run-captured, jq
```

## Security

All security features are inherited from `mcp-shell-server`:

- **Command Whitelisting**: Only commands in `ALLOW_COMMANDS` can be executed
- **Shell Operator Validation**: Commands after `;`, `&&`, `||`, `|` are validated
- **Directory Validation**: Working directory must be absolute and accessible
- **No Shell Injection**: Commands executed directly without shell interpretation

The systemd service includes additional hardening:
- `NoNewPrivileges=true`
- `PrivateTmp=true`
- `ProtectSystem=strict`
- `ProtectHome=true`
- `ReadWritePaths=/tmp`

## Authentication

Kernos supports optional API key authentication. When enabled, all MCP requests (tools, resources, prompts) must include a valid key. Health and metrics endpoints (`/live`, `/ready`, `/health`, `/metrics`) remain open for probes and monitoring.

### Generating API Keys

Generate a cryptographically secure key with:

```bash
# Using openssl (recommended)
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"
```

### Enabling Authentication

Add API keys to your host variables in `ansible/inventory/host_vars/caliban.incus.yml`:

```yaml
# Single key
kernos_api_keys: "a1b2c3d4e5f6..."

# Multiple keys (comma-separated)
kernos_api_keys: "key-for-alice,key-for-bob,key-for-ci-pipeline"
```

When `kernos_api_keys` is empty or unset, authentication is disabled (open access).

### Sending the Key

Clients authenticate via either header:

| Header | Format |
|--------|--------|
| `Authorization` | `Bearer <key>` |
| `X-Api-Key` | `<key>` |

#### FastMCP Client

```python
from fastmcp import Client

async with Client(
    "http://caliban.incus:22021/mcp",
    auth="your-api-key-here",
) as client:
    result = await client.call_tool("shell_execute", {"command": ["ls", "-la"]})
```

#### curl

```bash
# Using Authorization header
curl -H "Authorization: Bearer your-api-key-here" http://caliban.incus:22021/mcp/

# Using X-Api-Key header
curl -H "X-Api-Key: your-api-key-here" http://caliban.incus:22021/mcp/
```

#### MCP client config (e.g. Claude Desktop, OpenAI)

```json
{
  "mcpServers": {
    "kernos": {
      "url": "http://caliban.incus:22021/mcp/",
      "headers": {
        "Authorization": "Bearer your-api-key-here"
      }
    }
  }
}
```

## Usage

### Testing Health Endpoints

/health
/ready
/live
/metrics


### MCP Client Connection

Connect using any MCP client that supports HTTP transport:

```python
from fastmcp import Client

client = Client("http://caliban.incus:22021/mcp")

async with client:
    result = await client.call_tool("shell_execute", {
        "command": ["ls", "-la"],
        "directory": "/tmp"
    })
    print(result)
```

## Tool: shell_execute

Execute a shell command in a specified directory.

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `command` | `list[str]` | Yes | - | Command and arguments as array |
| `directory` | `str` | No | `/tmp` | Absolute path to working directory |
| `stdin` | `str` | No | `None` | Input to pass to command |
| `timeout` | `int` | No | `None` | Timeout in seconds |

### Response

```json
{
    "stdout": "command output",
    "stderr": "",
    "status": 0,
    "execution_time": 0.123
}
```

## Monitoring

### Prometheus Metrics

The `/metrics` endpoint exposes Prometheus-compatible metrics. Add to your Prometheus configuration:

```yaml
- job_name: 'kernos'
  static_configs:
    - targets: ['caliban.incus:22021']
```

### Service Status

```bash
# Check service status
sudo systemctl status kernos

# View logs
sudo journalctl -u kernos -f
```

## Troubleshooting

### Service Won't Start

1. Check logs: `journalctl -u kernos -n 50`
2. Verify `.env` file exists and has correct permissions
3. Ensure Python venv was created successfully
4. Check that `ALLOW_COMMANDS` is set

### Health Check Failures

1. Verify the service is running: `systemctl status kernos`
2. Check if port is accessible
3. Review logs for startup errors

### Command Execution Denied

1. Verify the command is in `ALLOW_COMMANDS` whitelist
2. Check that the working directory is absolute and accessible
3. Review logs for security validation errors