docs: rewrite README with structured overview and quick start guide
Replaces the minimal project description with a comprehensive README including a component overview table, quick start instructions, common Ansible operations, and links to detailed documentation. Aligns with Red Panda Approval™ standards.
This commit is contained in:
342
docs/jupyterlab.md
Normal file
342
docs/jupyterlab.md
Normal file
@@ -0,0 +1,342 @@
|
||||
# JupyterLab - Interactive Computing Environment
|
||||
|
||||
## Overview
|
||||
JupyterLab is a web-based interactive development environment for notebooks, code, and data. Deployed on **Puck** as a systemd service running in a Python virtual environment, with OAuth2-Proxy sidecar providing Casdoor SSO authentication.
|
||||
|
||||
**Host:** puck.incus
|
||||
**Role:** Application Runtime (Python App Host)
|
||||
**Container Port:** 22181 (JupyterLab), 22182 (OAuth2-Proxy)
|
||||
**External Access:** https://jupyter.ouranos.helu.ca/ (via HAProxy on Titania)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────┐ ┌────────────┐ ┌─────────────┐ ┌────────────┐
|
||||
│ Client │─────▶│ HAProxy │─────▶│ OAuth2-Proxy│─────▶│ JupyterLab │
|
||||
│ │ │ (Titania) │ │ (Puck) │ │ (Puck) │
|
||||
└──────────┘ └────────────┘ └─────────────┘ └────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Casdoor │
|
||||
│ (Titania) │
|
||||
└───────────┘
|
||||
```
|
||||
|
||||
### Authentication Flow
|
||||
|
||||
```
|
||||
┌──────────┐ ┌────────────┐ ┌─────────────┐ ┌──────────┐
|
||||
│ Browser │─────▶│ HAProxy │─────▶│ OAuth2-Proxy│─────▶│ Casdoor │
|
||||
│ │ │ (Titania) │ │ (Puck) │ │(Titania) │
|
||||
└──────────┘ └────────────┘ └─────────────┘ └──────────┘
|
||||
│ │ │
|
||||
│ 1. Access jupyter.ouranos.helu.ca │ │
|
||||
│─────────────────────────────────────▶│ │
|
||||
│ 2. No session - redirect to Casdoor │ │
|
||||
│◀─────────────────────────────────────│ │
|
||||
│ 3. User authenticates │ │
|
||||
│─────────────────────────────────────────────────────────▶│
|
||||
│ 4. Redirect with auth code │ │
|
||||
│◀─────────────────────────────────────────────────────────│
|
||||
│ 5. Exchange code, set session cookie│ │
|
||||
│◀─────────────────────────────────────│ │
|
||||
│ 6. Proxy to JupyterLab │ │
|
||||
│◀─────────────────────────────────────│ │
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Playbook
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
ansible-playbook jupyterlab/deploy.yml
|
||||
```
|
||||
|
||||
### Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `jupyterlab/deploy.yml` | Main deployment playbook |
|
||||
| `jupyterlab/jupyterlab.service.j2` | Systemd unit for JupyterLab |
|
||||
| `jupyterlab/oauth2-proxy-jupyter.service.j2` | Systemd unit for OAuth2-Proxy sidecar |
|
||||
| `jupyterlab/oauth2-proxy-jupyter.cfg.j2` | OAuth2-Proxy configuration |
|
||||
| `jupyterlab/jupyter_lab_config.py.j2` | JupyterLab server configuration |
|
||||
|
||||
### Deployment Steps
|
||||
|
||||
1. **Install Dependencies**: python3-venv, nodejs, npm, graphviz
|
||||
2. **Ensure User Exists**: `robert:robert` with home directory
|
||||
3. **Create Directories**: Notebooks dir, config dir, log dir
|
||||
4. **Create Virtual Environment**: `/home/robert/env/jupyter`
|
||||
5. **Install Python Packages**: jupyterlab, jupyter-ai, langchain-ollama, matplotlib, plotly
|
||||
6. **Install Jupyter Extensions**: contrib nbextensions
|
||||
7. **Template Configuration**: Apply JupyterLab config
|
||||
8. **Download OAuth2-Proxy**: Binary from GitHub releases
|
||||
9. **Template OAuth2-Proxy Config**: With Casdoor OIDC settings
|
||||
10. **Start Services**: Enable and start both systemd units
|
||||
|
||||
## Configuration
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Jupyter AI**: AI assistance via jupyter-ai[all] with LangChain Ollama integration
|
||||
- **Visualization**: matplotlib, plotly for data visualization
|
||||
- **Diagrams**: Mermaid support via jupyterlab-mermaid
|
||||
- **Extensions**: Jupyter contrib nbextensions
|
||||
- **SSO**: Casdoor authentication via OAuth2-Proxy sidecar
|
||||
- **WebSocket**: Full WebSocket support through reverse proxy
|
||||
|
||||
### Storage Locations
|
||||
|
||||
| Path | Purpose | Owner |
|
||||
|------|---------|-------|
|
||||
| `/home/robert/Notebooks` | Notebook files | robert:robert |
|
||||
| `/home/robert/env/jupyter` | Python virtual environment | robert:robert |
|
||||
| `/etc/jupyterlab` | Configuration files | root:robert |
|
||||
| `/var/log/jupyterlab` | Application logs | robert:robert |
|
||||
| `/etc/oauth2-proxy-jupyter` | OAuth2-Proxy config | root:root |
|
||||
|
||||
### Installed Python Packages
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `jupyterlab` | Core JupyterLab server |
|
||||
| `jupyter-ai[all]` | AI assistant integration |
|
||||
| `langchain-ollama` | Ollama LLM integration |
|
||||
| `matplotlib` | Data visualization |
|
||||
| `plotly` | Interactive charts |
|
||||
| `jupyter_contrib_nbextensions` | Community extensions |
|
||||
| `jupyterlab-mermaid` | Mermaid diagram support |
|
||||
| `ipywidgets` | Interactive widgets |
|
||||
|
||||
### Logging
|
||||
|
||||
- **JupyterLab**: systemd journal via `SyslogIdentifier=jupyterlab`
|
||||
- **OAuth2-Proxy**: systemd journal via `SyslogIdentifier=oauth2-proxy-jupyter`
|
||||
- **Alloy Forwarding**: Syslog port 51491 → Loki
|
||||
|
||||
## Access After Deployment
|
||||
|
||||
1. **Web Interface**: https://jupyter.ouranos.helu.ca/
|
||||
2. **Authentication**: Redirects to Casdoor SSO login
|
||||
3. **After Login**: Full JupyterLab interface with notebook access
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Alloy Configuration
|
||||
**File:** `ansible/alloy/puck/config.alloy.j2`
|
||||
|
||||
- **Log Collection**: Syslog port 51491 → Loki
|
||||
- **Job Label**: `jupyterlab`
|
||||
- **System Metrics**: Process exporter tracks JupyterLab process
|
||||
|
||||
### Health Check
|
||||
- **URL**: `http://puck.incus:22182/ping` (OAuth2-Proxy)
|
||||
- **JupyterLab API**: `http://127.0.0.1:22181/api/status` (localhost only)
|
||||
|
||||
## Required Vault Secrets
|
||||
|
||||
Add to `ansible/inventory/group_vars/all/vault.yml`:
|
||||
|
||||
### 1. OAuth Client ID
|
||||
```yaml
|
||||
vault_jupyter_oauth_client_id: "jupyter-oauth-client"
|
||||
```
|
||||
**Requirements:**
|
||||
- **Purpose**: Client ID for Casdoor OAuth2 application
|
||||
- **Source**: Must match `clientId` in Casdoor application configuration
|
||||
|
||||
### 2. OAuth Client Secret
|
||||
```yaml
|
||||
vault_jupyter_oauth_client_secret: "YourRandomOAuthSecret123!"
|
||||
```
|
||||
**Requirements:**
|
||||
- **Length**: 32+ characters recommended
|
||||
- **Purpose**: Client secret for Casdoor OAuth2 authentication
|
||||
- **Generation**:
|
||||
```bash
|
||||
openssl rand -base64 32
|
||||
```
|
||||
|
||||
### 3. Cookie Secret
|
||||
```yaml
|
||||
vault_jupyter_oauth2_cookie_secret: "32CharacterRandomStringHere1234"
|
||||
```
|
||||
**Requirements:**
|
||||
- **Length**: Exactly 32 characters (or 16/24 for AES)
|
||||
- **Purpose**: Encrypts OAuth2-Proxy session cookies
|
||||
- **Generation**:
|
||||
```bash
|
||||
openssl rand -base64 32 | head -c 32
|
||||
```
|
||||
|
||||
## Host Variables
|
||||
|
||||
**File:** `ansible/inventory/host_vars/puck.incus.yml`
|
||||
|
||||
```yaml
|
||||
# JupyterLab Configuration
|
||||
jupyterlab_user: robert
|
||||
jupyterlab_group: robert
|
||||
jupyterlab_notebook_dir: /home/robert/Notebooks
|
||||
jupyterlab_venv_dir: /home/robert/env/jupyter
|
||||
|
||||
# Ports
|
||||
jupyterlab_port: 22181 # JupyterLab (localhost only)
|
||||
jupyterlab_proxy_port: 22182 # OAuth2-Proxy (exposed to HAProxy)
|
||||
|
||||
# OAuth2-Proxy Configuration
|
||||
jupyterlab_oauth2_proxy_dir: /etc/oauth2-proxy-jupyter
|
||||
jupyterlab_oauth2_proxy_version: "7.6.0"
|
||||
jupyterlab_domain: "ouranos.helu.ca"
|
||||
jupyterlab_oauth2_oidc_issuer_url: "https://id.ouranos.helu.ca"
|
||||
jupyterlab_oauth2_redirect_url: "https://jupyter.ouranos.helu.ca/oauth2/callback"
|
||||
|
||||
# OAuth2 Credentials (from vault)
|
||||
jupyterlab_oauth_client_id: "{{ vault_jupyter_oauth_client_id }}"
|
||||
jupyterlab_oauth_client_secret: "{{ vault_jupyter_oauth_client_secret }}"
|
||||
jupyterlab_oauth2_cookie_secret: "{{ vault_jupyter_oauth2_cookie_secret }}"
|
||||
|
||||
# Alloy Logging
|
||||
jupyterlab_syslog_port: 51491
|
||||
```
|
||||
|
||||
## OAuth2 / Casdoor SSO
|
||||
|
||||
JupyterLab uses OAuth2-Proxy as a sidecar to handle Casdoor authentication. This pattern is simpler than native OAuth for single-user setups.
|
||||
|
||||
### Why OAuth2-Proxy Sidecar?
|
||||
|
||||
| Approach | Pros | Cons |
|
||||
|----------|------|------|
|
||||
| **OAuth2-Proxy (chosen)** | Simple setup, no JupyterLab modification | Extra service to manage |
|
||||
| **Native JupyterHub OAuth** | Integrated solution | More complex, overkill for single user |
|
||||
| **Token-only auth** | Simplest | Less secure, no SSO integration |
|
||||
|
||||
### Casdoor Application Configuration
|
||||
|
||||
A JupyterLab application is defined in `ansible/casdoor/init_data.json.j2`:
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Name** | `app-jupyter` |
|
||||
| **Client ID** | `vault_jupyter_oauth_client_id` |
|
||||
| **Redirect URI** | `https://jupyter.ouranos.helu.ca/oauth2/callback` |
|
||||
| **Grant Types** | `authorization_code`, `refresh_token` |
|
||||
|
||||
### URL Strategy
|
||||
|
||||
| URL Type | Address | Used By |
|
||||
|----------|---------|---------|
|
||||
| **OIDC Issuer** | `https://id.ouranos.helu.ca` | OAuth2-Proxy (external) |
|
||||
| **Redirect URL** | `https://jupyter.ouranos.helu.ca/oauth2/callback` | Browser callback |
|
||||
| **Upstream** | `http://127.0.0.1:22181` | OAuth2-Proxy → JupyterLab |
|
||||
|
||||
### Deployment Order
|
||||
|
||||
1. **Deploy Casdoor first** (if not already running):
|
||||
```bash
|
||||
ansible-playbook casdoor/deploy.yml
|
||||
```
|
||||
|
||||
2. **Update HAProxy** (add jupyter backend):
|
||||
```bash
|
||||
ansible-playbook haproxy/deploy.yml
|
||||
```
|
||||
|
||||
3. **Deploy JupyterLab**:
|
||||
```bash
|
||||
ansible-playbook jupyterlab/deploy.yml
|
||||
```
|
||||
|
||||
4. **Update Alloy** (for log forwarding):
|
||||
```bash
|
||||
ansible-playbook alloy/deploy.yml
|
||||
```
|
||||
|
||||
## Integration with Other Services
|
||||
|
||||
### HAProxy Routing
|
||||
**Backend Configuration** (`titania.incus.yml`):
|
||||
```yaml
|
||||
- subdomain: "jupyter"
|
||||
backend_host: "puck.incus"
|
||||
backend_port: 22182 # OAuth2-Proxy port
|
||||
health_path: "/ping"
|
||||
timeout_server: 300s # WebSocket support
|
||||
```
|
||||
|
||||
### Alloy Log Forwarding
|
||||
**Syslog Configuration** (`puck/config.alloy.j2`):
|
||||
```hcl
|
||||
loki.source.syslog "jupyterlab_logs" {
|
||||
listener {
|
||||
address = "127.0.0.1:51491"
|
||||
protocol = "tcp"
|
||||
labels = {
|
||||
job = "jupyterlab",
|
||||
}
|
||||
}
|
||||
forward_to = [loki.write.default.receiver]
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Status
|
||||
```bash
|
||||
ssh puck.incus
|
||||
sudo systemctl status jupyterlab
|
||||
sudo systemctl status oauth2-proxy-jupyter
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
# JupyterLab logs
|
||||
sudo journalctl -u jupyterlab -f
|
||||
|
||||
# OAuth2-Proxy logs
|
||||
sudo journalctl -u oauth2-proxy-jupyter -f
|
||||
```
|
||||
|
||||
### Test JupyterLab Directly (bypass OAuth)
|
||||
```bash
|
||||
# From puck container
|
||||
curl http://127.0.0.1:22181/api/status
|
||||
```
|
||||
|
||||
### Test OAuth2-Proxy Health
|
||||
```bash
|
||||
curl http://puck.incus:22182/ping
|
||||
```
|
||||
|
||||
### Verify Virtual Environment
|
||||
```bash
|
||||
ssh puck.incus
|
||||
sudo -u robert /home/robert/env/jupyter/bin/jupyter --version
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| WebSocket disconnects | Verify `timeout_server: 300s` in HAProxy backend |
|
||||
| OAuth redirect loop | Check `redirect_url` matches Casdoor app config |
|
||||
| 502 Bad Gateway | Ensure JupyterLab service is running on port 22181 |
|
||||
| Cookie errors | Verify `cookie_secret` is exactly 32 characters |
|
||||
|
||||
## Version Information
|
||||
|
||||
- **Installation Method**: Python pip in virtual environment
|
||||
- **JupyterLab Version**: Latest stable (pip managed)
|
||||
- **OAuth2-Proxy Version**: 7.6.0 (binary from GitHub)
|
||||
- **Update Process**: Re-run deployment playbook
|
||||
|
||||
## References
|
||||
|
||||
- **JupyterLab Documentation**: https://jupyterlab.readthedocs.io/
|
||||
- **OAuth2-Proxy Documentation**: https://oauth2-proxy.github.io/oauth2-proxy/
|
||||
- **Jupyter AI**: https://jupyter-ai.readthedocs.io/
|
||||
- **Casdoor OIDC**: https://casdoor.org/docs/integration/oidc
|
||||
Reference in New Issue
Block a user