ouranos/docs/jupyterlab.md

# JupyterLab - Interactive Computing Environment

## Overview
JupyterLab is a web-based interactive development environment for notebooks, code, and data. Deployed on **Puck** as a systemd service running in a Python virtual environment, with OAuth2-Proxy sidecar providing Casdoor SSO authentication.

**Host:** puck.incus  
**Role:** Application Runtime (Python App Host)  
**Container Port:** 22181 (JupyterLab), 22182 (OAuth2-Proxy)  
**External Access:** https://jupyter.ouranos.helu.ca/ (via HAProxy on Titania)

## Architecture

```
┌──────────┐      ┌────────────┐      ┌─────────────┐      ┌────────────┐
│  Client  │─────▶│  HAProxy   │─────▶│ OAuth2-Proxy│─────▶│ JupyterLab │
│          │      │ (Titania)  │      │   (Puck)    │      │   (Puck)   │
└──────────┘      └────────────┘      └─────────────┘      └────────────┘
                                            │
                                            ▼
                                      ┌───────────┐
                                      │  Casdoor  │
                                      │ (Titania) │
                                      └───────────┘
```

### Authentication Flow

```
┌──────────┐      ┌────────────┐      ┌─────────────┐      ┌──────────┐
│  Browser │─────▶│  HAProxy   │─────▶│ OAuth2-Proxy│─────▶│ Casdoor  │
│          │      │ (Titania)  │      │   (Puck)    │      │(Titania) │
└──────────┘      └────────────┘      └─────────────┘      └──────────┘
     │                                      │                   │
     │  1. Access jupyter.ouranos.helu.ca   │                   │
     │─────────────────────────────────────▶│                   │
     │  2. No session - redirect to Casdoor │                   │
     │◀─────────────────────────────────────│                   │
     │  3. User authenticates               │                   │
     │─────────────────────────────────────────────────────────▶│
     │  4. Redirect with auth code          │                   │
     │◀─────────────────────────────────────────────────────────│
     │  5. Exchange code, set session cookie│                   │
     │◀─────────────────────────────────────│                   │
     │  6. Proxy to JupyterLab              │                   │
     │◀─────────────────────────────────────│                   │
```

## Deployment

### Playbook

```bash
cd ansible
ansible-playbook jupyterlab/deploy.yml
```

### Files

| File | Purpose |
|------|---------|
| `jupyterlab/deploy.yml` | Main deployment playbook |
| `jupyterlab/jupyterlab.service.j2` | Systemd unit for JupyterLab |
| `jupyterlab/oauth2-proxy-jupyter.service.j2` | Systemd unit for OAuth2-Proxy sidecar |
| `jupyterlab/oauth2-proxy-jupyter.cfg.j2` | OAuth2-Proxy configuration |
| `jupyterlab/jupyter_lab_config.py.j2` | JupyterLab server configuration |

### Deployment Steps

1. **Install Dependencies**: python3-venv, nodejs, npm, graphviz
2. **Ensure User Exists**: `robert:robert` with home directory
3. **Create Directories**: Notebooks dir, config dir, log dir
4. **Create Virtual Environment**: `/home/robert/env/jupyter`
5. **Install Python Packages**: jupyterlab, jupyter-ai, langchain-ollama, matplotlib, plotly
6. **Install Jupyter Extensions**: contrib nbextensions
7. **Template Configuration**: Apply JupyterLab config
8. **Download OAuth2-Proxy**: Binary from GitHub releases
9. **Template OAuth2-Proxy Config**: With Casdoor OIDC settings
10. **Start Services**: Enable and start both systemd units

## Configuration

### Key Features

- **Jupyter AI**: AI assistance via jupyter-ai[all] with LangChain Ollama integration
- **Visualization**: matplotlib, plotly for data visualization
- **Diagrams**: Mermaid support via jupyterlab-mermaid
- **Extensions**: Jupyter contrib nbextensions
- **SSO**: Casdoor authentication via OAuth2-Proxy sidecar
- **WebSocket**: Full WebSocket support through reverse proxy

### Storage Locations

| Path | Purpose | Owner |
|------|---------|-------|
| `/home/robert/Notebooks` | Notebook files | robert:robert |
| `/home/robert/env/jupyter` | Python virtual environment | robert:robert |
| `/etc/jupyterlab` | Configuration files | root:robert |
| `/var/log/jupyterlab` | Application logs | robert:robert |
| `/etc/oauth2-proxy-jupyter` | OAuth2-Proxy config | root:root |

### Installed Python Packages

| Package | Purpose |
|---------|---------|
| `jupyterlab` | Core JupyterLab server |
| `jupyter-ai[all]` | AI assistant integration |
| `langchain-ollama` | Ollama LLM integration |
| `matplotlib` | Data visualization |
| `plotly` | Interactive charts |
| `jupyter_contrib_nbextensions` | Community extensions |
| `jupyterlab-mermaid` | Mermaid diagram support |
| `ipywidgets` | Interactive widgets |

### Logging

- **JupyterLab**: systemd journal via `SyslogIdentifier=jupyterlab`
- **OAuth2-Proxy**: systemd journal via `SyslogIdentifier=oauth2-proxy-jupyter`
- **Alloy Forwarding**: Syslog port 51491 → Loki

## Access After Deployment

1. **Web Interface**: https://jupyter.ouranos.helu.ca/
2. **Authentication**: Redirects to Casdoor SSO login
3. **After Login**: Full JupyterLab interface with notebook access

## Monitoring

### Alloy Configuration
**File:** `ansible/alloy/puck/config.alloy.j2`

- **Log Collection**: Syslog port 51491 → Loki
- **Job Label**: `jupyterlab`
- **System Metrics**: Process exporter tracks JupyterLab process

### Health Check
- **URL**: `http://puck.incus:22182/ping` (OAuth2-Proxy)
- **JupyterLab API**: `http://127.0.0.1:22181/api/status` (localhost only)

## Required Vault Secrets

Add to `ansible/inventory/group_vars/all/vault.yml`:

### 1. OAuth Client ID
```yaml
vault_jupyter_oauth_client_id: "jupyter-oauth-client"
```
**Requirements:**
- **Purpose**: Client ID for Casdoor OAuth2 application
- **Source**: Must match `clientId` in Casdoor application configuration

### 2. OAuth Client Secret
```yaml
vault_jupyter_oauth_client_secret: "YourRandomOAuthSecret123!"
```
**Requirements:**
- **Length**: 32+ characters recommended
- **Purpose**: Client secret for Casdoor OAuth2 authentication
- **Generation**:
  ```bash
  openssl rand -base64 32
  ```

### 3. Cookie Secret
```yaml
vault_jupyter_oauth2_cookie_secret: "32CharacterRandomStringHere1234"
```
**Requirements:**
- **Length**: Exactly 32 characters (or 16/24 for AES)
- **Purpose**: Encrypts OAuth2-Proxy session cookies
- **Generation**:
  ```bash
  openssl rand -base64 32 | head -c 32
  ```

## Host Variables

**File:** `ansible/inventory/host_vars/puck.incus.yml`

```yaml
# JupyterLab Configuration
jupyterlab_user: robert
jupyterlab_group: robert
jupyterlab_notebook_dir: /home/robert/Notebooks
jupyterlab_venv_dir: /home/robert/env/jupyter

# Ports
jupyterlab_port: 22181           # JupyterLab (localhost only)
jupyterlab_proxy_port: 22182     # OAuth2-Proxy (exposed to HAProxy)

# OAuth2-Proxy Configuration
jupyterlab_oauth2_proxy_dir: /etc/oauth2-proxy-jupyter
jupyterlab_oauth2_proxy_version: "7.6.0"
jupyterlab_domain: "ouranos.helu.ca"
jupyterlab_oauth2_oidc_issuer_url: "https://id.ouranos.helu.ca"
jupyterlab_oauth2_redirect_url: "https://jupyter.ouranos.helu.ca/oauth2/callback"

# OAuth2 Credentials (from vault)
jupyterlab_oauth_client_id: "{{ vault_jupyter_oauth_client_id }}"
jupyterlab_oauth_client_secret: "{{ vault_jupyter_oauth_client_secret }}"
jupyterlab_oauth2_cookie_secret: "{{ vault_jupyter_oauth2_cookie_secret }}"

# Alloy Logging
jupyterlab_syslog_port: 51491
```

## OAuth2 / Casdoor SSO

JupyterLab uses OAuth2-Proxy as a sidecar to handle Casdoor authentication. This pattern is simpler than native OAuth for single-user setups.

### Why OAuth2-Proxy Sidecar?

| Approach | Pros | Cons |
|----------|------|------|
| **OAuth2-Proxy (chosen)** | Simple setup, no JupyterLab modification | Extra service to manage |
| **Native JupyterHub OAuth** | Integrated solution | More complex, overkill for single user |
| **Token-only auth** | Simplest | Less secure, no SSO integration |

### Casdoor Application Configuration

A JupyterLab application is defined in `ansible/casdoor/init_data.json.j2`:

| Setting | Value |
|---------|-------|
| **Name** | `app-jupyter` |
| **Client ID** | `vault_jupyter_oauth_client_id` |
| **Redirect URI** | `https://jupyter.ouranos.helu.ca/oauth2/callback` |
| **Grant Types** | `authorization_code`, `refresh_token` |

### URL Strategy

| URL Type | Address | Used By |
|----------|---------|---------|
| **OIDC Issuer** | `https://id.ouranos.helu.ca` | OAuth2-Proxy (external) |
| **Redirect URL** | `https://jupyter.ouranos.helu.ca/oauth2/callback` | Browser callback |
| **Upstream** | `http://127.0.0.1:22181` | OAuth2-Proxy → JupyterLab |

### Deployment Order

1. **Deploy Casdoor first** (if not already running):
   ```bash
   ansible-playbook casdoor/deploy.yml
   ```

2. **Update HAProxy** (add jupyter backend):
   ```bash
   ansible-playbook haproxy/deploy.yml
   ```

3. **Deploy JupyterLab**:
   ```bash
   ansible-playbook jupyterlab/deploy.yml
   ```

4. **Update Alloy** (for log forwarding):
   ```bash
   ansible-playbook alloy/deploy.yml
   ```

## Integration with Other Services

### HAProxy Routing
**Backend Configuration** (`titania.incus.yml`):
```yaml
- subdomain: "jupyter"
  backend_host: "puck.incus"
  backend_port: 22182  # OAuth2-Proxy port
  health_path: "/ping"
  timeout_server: 300s  # WebSocket support
```

### Alloy Log Forwarding
**Syslog Configuration** (`puck/config.alloy.j2`):
```hcl
loki.source.syslog "jupyterlab_logs" {
  listener {
    address  = "127.0.0.1:51491"
    protocol = "tcp"
    labels = {
      job = "jupyterlab",
    }
  }
  forward_to = [loki.write.default.receiver]
}
```

## Troubleshooting

### Service Status
```bash
ssh puck.incus
sudo systemctl status jupyterlab
sudo systemctl status oauth2-proxy-jupyter
```

### View Logs
```bash
# JupyterLab logs
sudo journalctl -u jupyterlab -f

# OAuth2-Proxy logs
sudo journalctl -u oauth2-proxy-jupyter -f
```

### Test JupyterLab Directly (bypass OAuth)
```bash
# From puck container
curl http://127.0.0.1:22181/api/status
```

### Test OAuth2-Proxy Health
```bash
curl http://puck.incus:22182/ping
```

### Verify Virtual Environment
```bash
ssh puck.incus
sudo -u robert /home/robert/env/jupyter/bin/jupyter --version
```

### Common Issues

| Issue | Solution |
|-------|----------|
| WebSocket disconnects | Verify `timeout_server: 300s` in HAProxy backend |
| OAuth redirect loop | Check `redirect_url` matches Casdoor app config |
| 502 Bad Gateway | Ensure JupyterLab service is running on port 22181 |
| Cookie errors | Verify `cookie_secret` is exactly 32 characters |

## Version Information

- **Installation Method**: Python pip in virtual environment
- **JupyterLab Version**: Latest stable (pip managed)
- **OAuth2-Proxy Version**: 7.6.0 (binary from GitHub)
- **Update Process**: Re-run deployment playbook

## References

- **JupyterLab Documentation**: https://jupyterlab.readthedocs.io/
- **OAuth2-Proxy Documentation**: https://oauth2-proxy.github.io/oauth2-proxy/
- **Jupyter AI**: https://jupyter-ai.readthedocs.io/
- **Casdoor OIDC**: https://casdoor.org/docs/integration/oidc