From 9391024ba00790f31aad322204f4c2ff8025bf9b Mon Sep 17 00:00:00 2001 From: Weston Clark Date: Wed, 23 Sep 2026 17:44:43 -0400 Subject: [PATCH] feat: Stage 1 setup guides, CI for every service, submodule updates - .claude/auth-setup.md: production login (NetID allowlist + team password), JWT secret, per-hub device tokens, verification, rotation. - .claude/rpi-hub-setup.md: Pi from blank SD card to verified hub, board checks (Uno R3/R4, Mega, Nano, STM32F407G-DISC1), and the cellular hub's uplink manager with a failover test. - Remove .claude/CREDENTIAL_GENERATOR_README.md, which contained a Gmail app password (revoke it in the Google account; history still has it). - CI: run rpi-hub-server and uplink manager tests; check the web client's generated API types match the cloud contract. - README: current repo roles, dev-sim/mock workflows, links to the guides. - Submodules: cloud-services feat/hub-registry-contracts, rpi-hub-server feat/board-profiles, web-client feat/stage1-ui, wifi-fallback-relay feat/uplink-manager. Co-Authored-By: Claude Opus 5.5 --- .claude/CREDENTIAL_GENERATOR_README.md | 452 ---------------------- .claude/auth-setup.md | 226 +++++++++++ .claude/rpi-hub-setup.md | 335 ++++++++++++++++ .github/workflows/ci.yml | 12 + .github/workflows/test-rpi-hub-server.yml | 35 ++ .github/workflows/test-uplink-manager.yml | 35 ++ .github/workflows/test-web-client.yml | 10 + README.md | 28 +- cloud-services | 2 +- rpi-hub-server | 2 +- web-client | 2 +- wifi-fallback-relay | 2 +- 12 files changed, 675 insertions(+), 466 deletions(-) delete mode 100644 .claude/CREDENTIAL_GENERATOR_README.md create mode 100644 .claude/auth-setup.md create mode 100644 .claude/rpi-hub-setup.md create mode 100644 .github/workflows/test-rpi-hub-server.yml create mode 100644 .github/workflows/test-uplink-manager.yml diff --git a/.claude/CREDENTIAL_GENERATOR_README.md b/.claude/CREDENTIAL_GENERATOR_README.md deleted file mode 100644 index 78a5c2d..0000000 --- a/.claude/CREDENTIAL_GENERATOR_README.md +++ /dev/null @@ -1,452 +0,0 @@ -# User Credential Generator - Quick Start - -## Setup - -1. **Install required packages** (if not already installed): - ```bash - pip install pandas openpyxl bcrypt - ``` - -2. **Place your Excel file** in the workspace root: - - Name it `users.xlsx` OR update `EXCEL_FILE` variable in `generate_users.py` - - Ensure it has these columns: `NetID`, `Email`, `Name`, plus security questions - -3. **Update email configuration** in `generate_users.py`: - - Change `SENDER_EMAIL = "your-email@gmail.com"` to your actual Gmail address - - Gmail App Password is already set: `bnux xbft wgrz loue` - -## Excel Column Mapping - -Expected columns: -- `NetID` → username (required) -- `Email` → user email (required) -- `Name` → full name (required) -- `Car you have/like?` → password ingredient -- `Pets Name` → password ingredient -- `Favorite City` → password ingredient -- `Favorite Number` → password ingredient -- `Worst Sub-team` → password ingredient -- `Coolest Invention` → password ingredient -- `Select a Symbol` → password ingredient - -## Usage - -```bash -python generate_users.py -``` - -The script will: -1. Read the Excel file -2. Generate memorable passwords (e.g., `Tesla-Buddy-Austin42!`) -3. Hash them with bcrypt -4. Save to `generated_users.json` -5. Ask if you want to send emails -6. Email credentials to each user - -## Output - -**generated_users.json** contains: -```json -{ - "MOCK_USERS": { - "jsmith": { - "username": "jsmith", - "email": "jsmith@example.com", - "full_name": "John Smith", - "hashed_password": "$2b$12$..." - } - } -} -``` - -## Next Steps - -1. Copy the `MOCK_USERS` dictionary from `generated_users.json` -2. Replace the `MOCK_USERS` in `cloud-services/src/auth/auth_service.py` -3. **DELETE** `generate_users.py`, `generated_users.json`, and your Excel file - - These files are in `.gitignore` but delete them for safety - -## Security Notes - -- Passwords are readable but use personal info (memorable for users) -- Format: `Answer1-Answer2-Answer3##!` where ## is random 2-digit number -- All passwords are hashed with bcrypt (cost factor 12) -- Files are in `.gitignore` to prevent accidental commits -- **Always delete these files after use** - -## Local Script Ran - -''' -""" -Local script to generate user credentials from Excel spreadsheet. -WARNING: This file contains sensitive information and should NEVER be committed to version control. -Add to .gitignore immediately after use. -""" - -import pandas as pd -import bcrypt -import json -import smtplib -from email.mime.text import MIMEText -from email.mime.multipart import MIMEMultipart -import random -import re -import argparse - -# ===== CONFIGURATION ===== -EXCEL_FILE = "example.xlsx" # Change to your Excel file path -OUTPUT_JSON = "generated_users.json" # Will contain MOCK_USERS format -SENDER_EMAIL = "example@gmail.com" # Change to your Gmail address -GMAIL_APP_PASSWORD = "example" -SMTP_SERVER = "smtp.gmail.com" -SMTP_PORT = 587 -TEST_EMAIL = "example@gmail.com" # Test mode sends here - -# ===== PASSWORD GENERATION ===== -def clean_answer(answer): - """Clean security question answer for password use.""" - if pd.isna(answer) or answer == "": - return None - # Remove special characters, keep alphanumeric - cleaned = re.sub(r'[^a-zA-Z0-9]', '', str(answer)) - return cleaned if cleaned else None - -def generate_password(row): - """ - Generate memorable password from security questions. - Format: Answer1SymbolAnswer2SymbolNumberSymbol (e.g., Tesla$Buddy$42$) - """ - # Get user's selected symbol - symbol = row.get('Select a Symbol', '$') - if pd.isna(symbol) or symbol == "": - symbol = '$' # Default fallback - else: - symbol = str(symbol).strip() - - # Collect all available security answers (excluding the symbol field) - answers = [] - - fields = [ - 'example1', - 'example2' - ] - - for field in fields: - if field in row: - cleaned = clean_answer(row[field]) - if cleaned: - answers.append(cleaned) - - # Select 2-3 random answers - if len(answers) >= 3: - selected = random.sample(answers, 3) - elif len(answers) >= 2: - selected = random.sample(answers, 2) - elif len(answers) >= 1: - selected = answers - else: - # Fallback: use name + random number - name = clean_answer(row.get('Name', 'User')) - selected = [name, str(random.randint(1000, 9999))] - - # Add random number suffix for security - suffix = str(random.randint(10, 99)) - - # Join with user's symbol and add suffix with symbol - password = symbol.join(selected) + suffix + symbol - - return password - -# ===== MAIN PROCESSING ===== -def main(): - # Parse command-line arguments - parser = argparse.ArgumentParser(description='Generate user credentials from Excel') - parser.add_argument('--prod', action='store_true', - help='Production mode: send emails to all users. Without this flag, only test email is sent.') - args = parser.parse_args() - - is_production = args.prod - - print("=" * 60) - print("User Credential Generator") - print(f"Mode: {'PRODUCTION' if is_production else 'TEST (dry-run)'}") - print("=" * 60) - - # Read Excel file - print(f"\n[1/5] Reading Excel file: {EXCEL_FILE}") - try: - df = pd.read_excel(EXCEL_FILE) - print(f"✓ Found {len(df)} users") - except FileNotFoundError: - print(f"✗ Error: File '{EXCEL_FILE}' not found!") - print("Please update EXCEL_FILE path in the script.") - return - except Exception as e: - print(f"✗ Error reading Excel: {e}") - return - - # Display column names for verification - print(f"\nColumns found: {list(df.columns)}") - - # Validate required columns - required_cols = ['NetID', 'Email', 'Name'] - missing_cols = [col for col in required_cols if col not in df.columns] - if missing_cols: - print(f"\n✗ Error: Missing required columns: {missing_cols}") - return - - # Generate users - print(f"\n[2/5] Generating passwords and hashing...") - users_dict = {} - credentials_list = [] # For emailing - - for idx, row in df.iterrows(): - netid = str(row['NetID']).strip() - email = str(row['Email']).strip() - full_name = str(row['Name']).strip() - - if not netid or netid == 'nan': - print(f" ⚠ Skipping row {idx}: No NetID") - continue - - # Generate password - plain_password = generate_password(row) - - # Hash password with bcrypt (cost factor 12, default) - hashed_password = bcrypt.hashpw( - plain_password.encode('utf-8'), - bcrypt.gensalt() - ).decode('utf-8') - - # Create user entry - users_dict[netid] = { - "username": netid, - "email": email, - "full_name": full_name, - "hashed_password": hashed_password, - } - - # Store credentials for emailing - credentials_list.append({ - "username": netid, - "password": plain_password, - "email": email, - "full_name": full_name - }) - - print(f" ✓ {netid}: {plain_password}") - - print(f"\n✓ Generated {len(users_dict)} users") - - # Save to JSON - print(f"\n[3/5] Saving to {OUTPUT_JSON}...") - output_data = { - "MOCK_USERS": users_dict, - "note": "Copy the MOCK_USERS dictionary into auth_service.py" - } - - with open(OUTPUT_JSON, 'w') as f: - json.dump(output_data, f, indent=4) - - print(f"✓ Saved to {OUTPUT_JSON}") - - # Send emails - print(f"\n[4/5] Sending emails...") - - if is_production: - # Production mode: send to all users - send_confirmation = input("PRODUCTION MODE: Send emails to ALL users? (yes/no): ").strip().lower() - - if send_confirmation == 'yes': - if SENDER_EMAIL == "your-email@gmail.com": - print("✗ Error: Please update SENDER_EMAIL in the script configuration!") - return - - success_count = 0 - for cred in credentials_list: - try: - send_credentials_email( - cred['email'], - cred['username'], - cred['password'], - cred['full_name'] - ) - print(f" ✓ Sent to {cred['email']}") - success_count += 1 - except Exception as e: - print(f" ✗ Failed to send to {cred['email']}: {e}") - - print(f"\n✓ Sent {success_count}/{len(credentials_list)} emails") - else: - print("Skipped email sending.") - else: - # Test mode: send mock email to test address - print("TEST MODE: Sending sample email to test address...") - if SENDER_EMAIL == "your-email@gmail.com": - print("✗ Error: Please update SENDER_EMAIL in the script configuration!") - return - - try: - # Use first user's credentials as example - if credentials_list: - sample = credentials_list[0] - send_test_email( - TEST_EMAIL, - sample['username'], - sample['password'], - sample['full_name'], - len(credentials_list) - ) - print(f" ✓ Test email sent to {TEST_EMAIL}") - print(f" Sample credentials: {sample['username']} / {sample['password']}") - else: - print(" ✗ No users to send test email") - except Exception as e: - print(f" ✗ Failed to send test email: {e}") - - # Final instructions - print(f"\n[5/5] Next Steps:") - if is_production: - print(f" 1. Review {OUTPUT_JSON}") - print(f" 2. Copy MOCK_USERS content to cloud-services/src/auth/auth_service.py") - print(f" 3. DELETE this script and {OUTPUT_JSON} (or add to .gitignore)") - print(f" 4. Verify users can log in") - else: - print(f" 1. Review {OUTPUT_JSON} to verify output") - print(f" 2. Check test email at {TEST_EMAIL}") - print(f" 3. Run with --prod flag when ready for production") - print("\n" + "=" * 60) - -# ===== EMAIL SENDING ===== -def send_test_email(to_email, sample_username, sample_password, sample_name, total_users): - """Send test email with sample credentials.""" - - # Create message - msg = MIMEMultipart('alternative') - msg['Subject'] = '[TEST] User Credential Generation Complete' - msg['From'] = SENDER_EMAIL - msg['To'] = to_email - - # Email body - text = f""" -TEST MODE - Credential Generation Complete - -Generated {total_users} user accounts successfully. - -Sample credentials (first user): -Username: {sample_username} -Password: {sample_password} -Full Name: {sample_name} - -Review the generated_users.json file and run with --prod flag to send emails to all users. -""" - - html = f""" - - -

TEST MODE - Credential Generation Complete

- -

Generated {total_users} user accounts successfully.

- -

Sample credentials (first user):

- - - - - - - - - - - - - -
Username:{sample_username}
Password:{sample_password}
Full Name:{sample_name}
- -

Review the generated_users.json file and run with --prod flag to send emails to all users.

- - -""" - - # Attach both plain text and HTML versions - part1 = MIMEText(text, 'plain') - part2 = MIMEText(html, 'html') - msg.attach(part1) - msg.attach(part2) - - # Send email - with smtplib.SMTP(SMTP_SERVER, SMTP_PORT) as server: - server.starttls() - server.login(SENDER_EMAIL, GMAIL_APP_PASSWORD) - server.send_message(msg) - -def send_credentials_email(to_email, username, password, full_name): - """Send credentials email to a user.""" - - # Create message - msg = MIMEMultipart('alternative') - msg['Subject'] = 'Your Account Credentials' - msg['From'] = SENDER_EMAIL - msg['To'] = to_email - - # Email body - text = f""" -Hi {full_name}, - -Your account has been created! Here are your login credentials: - -Username: {username} -Password: {password} - -Please keep these credentials secure and change your password after your first login. - -If you have any questions, please contact the administrator. - -Best regards, -The Team -""" - - html = f""" - - -

Hi {full_name},

- -

Your account has been created! Here are your login credentials:

- - - - - - - - - - -
Username:{username}
Password:{password}
- -

Please keep these credentials secure and change your password after your first login.

- -

If you have any questions, please contact the administrator.

- -

Best regards,
The Team

- - -""" - - # Attach both plain text and HTML versions - part1 = MIMEText(text, 'plain') - part2 = MIMEText(html, 'html') - msg.attach(part1) - msg.attach(part2) - - # Send email - with smtplib.SMTP(SMTP_SERVER, SMTP_PORT) as server: - server.starttls() - server.login(SENDER_EMAIL, GMAIL_APP_PASSWORD) - server.send_message(msg) - -if __name__ == "__main__": - main() - -''' \ No newline at end of file diff --git a/.claude/auth-setup.md b/.claude/auth-setup.md new file mode 100644 index 0000000..aefa6b6 --- /dev/null +++ b/.claude/auth-setup.md @@ -0,0 +1,226 @@ +# Login and Hub Token Setup (cloud-services) + +How to configure GUI login (NetID allowlist + one shared team password) and hub device tokens on +the production cloud service, how to verify it, and how to rotate or revoke access later. + +Written for humans and agents. Values in `` are placeholders you must replace. +Every step ends with an **Expected result**; stop and troubleshoot if it does not match. + +## Rules (read first) + +- Never paste the team password, the JWT secret, or device tokens into chat, commits, PR text, + issues, or logs. The password hash is less sensitive but still keep it private. +- Secrets live only in the backend env file on the server (and each Pi's `.env` for its own token). + Both are gitignored; never `git add` them. +- Configure the server **before** the new cloud-services code is deployed. With + `ENVIRONMENT=production` the service refuses to start if any value below is missing or still a + development default, and deploys happen automatically when the hyperloop-gui PR merges. +- The old per-user passwords (`users.json`) and the old credential generator are retired. The + Gmail app password that was committed with the generator must be revoked in that Google + account (Google Account > Security > App passwords) if that has not been done. + +## How login works now + +- A person signs in with their **NetID** and the **team password**. The NetID must be on the + allowlist. The JWT carries the NetID, so actions stay attributable to a person. +- "View Only Mode" needs no credentials and cannot send any command (write, restart, close, flash). +- Failed logins are limited per NetID (5 per 5 minutes) and per client IP (20 per 5 minutes). +- Removing a NetID from the allowlist ends that person's existing sessions on their next request. +- Development (no production settings): NetID `dev`, password `hyperloop-dev`; hubs connect with + `dev-token-rpi-bridge-01` / `dev-token-rpi-bridge-02`. + +## Setup + +### Step 1: Find the backend's env file + +On the EC2 host (SSH in as the deploy user): + +```bash +systemctl cat backend +``` + +Look at `WorkingDirectory=` and for `ENV_FILE=` in `Environment=`/`ExecStart=`, or an +`EnvironmentFile=`. cloud-services reads the file named by `ENV_FILE`, relative to the working +directory, and falls back to `.env`. + +**Expected result:** you know the path of the env file, referred to below as `` +(typically `/home/ubuntu/cloud-services/.env.production`). If you cannot find one, create +`/home/ubuntu/cloud-services/.env.production` and make sure the unit sets +`Environment=ENV_FILE=.env.production`. + +### Step 2: Choose the team password and hash it + +Pick a long passphrase (for example four or more random words; at most 72 bytes). Hash it on the +server with the backend's virtualenv. The password is typed at a hidden prompt, so it never +appears in shell history: + +```bash +cd /home/ubuntu/cloud-services +venv/bin/python -c "import bcrypt, getpass; p = getpass.getpass('Team password: ').encode(); print(bcrypt.hashpw(p, bcrypt.gensalt(12)).decode())" +``` + +**Expected result:** one line starting with `$2b$12$`. This is ``. Give the +passphrase itself to the team in person or through a private channel. + +### Step 3: Generate the JWT secret + +```bash +venv/bin/python -c "import secrets; print(secrets.token_urlsafe(48))" +``` + +**Expected result:** a random string of about 64 characters, ``. + +### Step 4: Create the NetID allowlist + +```bash +nano /home/ubuntu/allowed_netids.txt +chmod 600 /home/ubuntu/allowed_netids.txt +``` + +One NetID per line, lowercase, without `@cornell.edu`. `#` starts a comment: + +```text +# Electrical subteam +abc123 +xyz789 # team lead +``` + +**Expected result:** the file lists every member who should be able to send commands. +`viewer`, `admin` and `root` are ignored if listed. (Alternative: put a comma-separated list in +`ALLOWED_NETIDS=` instead of using a file.) + +### Step 5: Generate one device token per hub + +Decide the hub IDs first. They must match `HUB_ID` in each Pi's `.env` +(see `.claude/rpi-hub-setup.md`). Example IDs: `rpi-lab-01`, `rpi-lab-02`, `rpi-lab-03`, +`rpi-cellular-01`. Existing Pis can keep their current IDs (`rpi-bridge-01`, `rpi-bridge-02`). + +Run once per hub: + +```bash +venv/bin/python -c "import secrets; print(secrets.token_urlsafe(32))" +``` + +Build the value as comma-separated `hub-id:token` pairs: + +```text +DEVICE_TOKENS=rpi-lab-01:,rpi-lab-02:,rpi-lab-03:,rpi-cellular-01: +``` + +**Expected result:** every hub has its own token. Record which token belongs to which Pi somewhere +private; each Pi needs its token as `DEVICE_TOKEN`. Every hub listed here appears on the GUI +dashboard (offline until it connects). + +### Step 6: Write the env file + +Edit `` and set (keep any other existing lines such as `CORS_ORIGINS`): + +```bash +ENVIRONMENT=production +JWT_SECRET_KEY= +JWT_ACCESS_TOKEN_EXPIRE_MINUTES=480 +# Single quotes keep the $ characters of the hash literal +TEAM_PASSWORD_HASH='' +ALLOWED_NETIDS_FILE=/home/ubuntu/allowed_netids.txt +DEVICE_TOKENS= +# Only if port 8080 is not reachable from the internet (all traffic comes through Caddy/Cloudflare): +TRUST_PROXY_HEADERS=true +``` + +Then: + +```bash +chmod 600 +``` + +Remove `USERS_FILE_PATH` if present (no longer used). Old `DEVICE_TOKEN_RPI_BRIDGE_01/02` lines +still work, but move those hubs into `DEVICE_TOKENS` and delete the old lines. + +**Expected result:** the file contains every key above with real values; no `dev-token` +values remain. + +### Step 7: Restart and check the service + +```bash +sudo systemctl restart backend +sleep 3 +systemctl is-active backend +journalctl -u backend -n 50 --no-pager +``` + +**Expected result:** `active`, and no `Invalid production configuration` in the log. If that +error appears, it lists exactly which setting is missing or invalid. + +### Step 8: Verify + +Set the API base (use `http://127.0.0.1:8080` when testing on the server itself): + +```bash +API=https://api.cornellhyperloop.com +``` + +1. Login works (enter your NetID and the team password when prompted): + + ```bash + read -p "NetID: " NETID; read -s -p "Team password: " PW; echo + curl -s -o /dev/null -w "%{http_code}\n" -X POST "$API/auth/login" \ + -H 'content-type: application/json' -d "{\"username\":\"$NETID\",\"password\":\"$PW\"}" + ``` + + **Expected result:** `200` + +2. A wrong password is rejected: + + ```bash + curl -s -o /dev/null -w "%{http_code}\n" -X POST "$API/auth/login" \ + -H 'content-type: application/json' -d "{\"username\":\"$NETID\",\"password\":\"wrong\"}" + ``` + + **Expected result:** `401`. After 5 failures in 5 minutes the same NetID gets `429`, even with + the right password, until the window passes. Test that with a throwaway NetID rather than your + own. + +3. View-only sessions cannot send commands: + + ```bash + VIEWER=$(curl -s -X POST "$API/auth/login-viewer" | python3 -c "import sys, json; print(json.load(sys.stdin)['access_token'])") + curl -s -o /dev/null -w "%{http_code}\n" -X POST "$API/api/hubs//commands/restart" \ + -H "Authorization: Bearer $VIEWER" -H 'content-type: application/json' -d '{"portId":"x"}' + ``` + + **Expected result:** `403` + +4. Configured hubs are listed: + + ```bash + curl -s "$API/api/hubs" -H "Authorization: Bearer $VIEWER" + ``` + + **Expected result:** one entry per hub in `DEVICE_TOKENS`, `"connected": false` until that Pi + is set up. + +5. In the GUI, sign in with your NetID and the team password. + +Clean up the shell: `unset PW NETID VIEWER`. + +Once login works, delete the old users file if it exists: `rm -f /home/ubuntu/users.json`. + +## Maintenance + +| Task | Steps | Takes effect | +|---|---|---| +| Add or remove a member | Edit `allowed_netids.txt` | Next request (no restart); removed members are logged out | +| Rotate the team password | Step 2, replace `TEAM_PASSWORD_HASH`, restart `backend` | New logins; existing sessions last until they expire (8 h) | +| Log everyone out now | Replace `JWT_SECRET_KEY` (step 3), restart `backend` | Immediately | +| Add a hub | Generate a token (step 5), append to `DEVICE_TOKENS`, restart `backend`, put the token in the Pi's `.env` | After restart | +| Revoke a hub (lost/stolen Pi) | Remove its pair from `DEVICE_TOKENS`, restart `backend` | Immediately; the hub is rejected (`ws_rejected` in its log) | + +## Troubleshooting + +| Symptom | Cause / fix | +|---|---| +| `backend` fails to start with `Invalid production configuration` | The message names the missing or invalid setting; fix it in `` | +| Everyone gets `401` with the right password | Hash copied incorrectly (must start with `$2b$` and be in single quotes), or the service reads a different env file (step 1) | +| One person gets `401` | Their NetID is not in the allowlist, or typo in the file | +| Everyone gets `429` | Without `TRUST_PROXY_HEADERS=true` behind a proxy, all users share the proxy's IP (20 failures per 5 minutes locks everyone out). Set it only if port 8080 is not publicly reachable | +| A hub never shows online | Its `HUB_ID`/`DEVICE_TOKEN` pair does not match `DEVICE_TOKENS`; see `.claude/rpi-hub-setup.md` | diff --git a/.claude/rpi-hub-setup.md b/.claude/rpi-hub-setup.md new file mode 100644 index 0000000..f37a2eb --- /dev/null +++ b/.claude/rpi-hub-setup.md @@ -0,0 +1,335 @@ +# Raspberry Pi Hub Setup (rpi-hub-server) + +How to set up a Raspberry Pi as a telemetry hub from a blank SD card, verify it, and (for the one +hub with a cellular HAT) add Wi-Fi-to-cellular failover. + +Written for humans and agents. Values in `` are placeholders. Every step ends +with an **Expected result**; stop and troubleshoot if it does not match. Commands run on the Pi as +user `pi` unless they start with `sudo`. + +## Rules (read first) + +- A hub's `DEVICE_TOKEN` is a secret: do not print it, paste it into chat, or commit it. It lives + only in `/opt/rpi-hub-server/.env` (mode 600). +- Each hub needs a `HUB_ID` + `DEVICE_TOKEN` pair that is also listed in the cloud's + `DEVICE_TOKENS` (see `.claude/auth-setup.md`, step 5). Get the pair before starting. +- All hubs run the same code. The only per-hub differences are `.env` and the profile: + +| Hub | `HUB_PROFILE` | Extra software | +|---|---|---| +| Lab hubs (Wi-Fi only) | `lab-hub` | none | +| The hub with the cellular HAT | `cellular-hub` | uplink manager (section "Cellular hub") | + +## 1. Prepare the SD card + +Use Raspberry Pi Imager with **Raspberry Pi OS Lite (64-bit)**. In the Imager settings: + +- Hostname: the hub ID, e.g. `rpi-lab-01` +- Username `pi` with a password of your choice +- Wi-Fi: the lab network (SSID, password, country) +- Enable SSH + +Boot the Pi and connect: `ssh pi@.local` + +**Expected result:** a shell on the Pi. `ping -c 3 api.cornellhyperloop.com` gets replies. + +## 2. Base packages and permissions + +```bash +sudo apt update && sudo apt full-upgrade -y +sudo apt install -y git python3-venv python3-pip openocd usbutils +sudo usermod -aG dialout,plugdev pi +sudo reboot +``` + +After the reboot, reconnect and check: + +```bash +groups +ls /lib/udev/rules.d/ | grep -i openocd +python3 --version +``` + +**Expected result:** `groups` includes `dialout` and `plugdev`; the OpenOCD udev rule file is +listed (it gives `plugdev` access to ST-LINK probes); Python is 3.11 or newer. + +## 3. arduino-cli and board cores + +Install arduino-cli system-wide so the systemd service finds it: + +```bash +curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sudo BINDIR=/usr/local/bin sh +arduino-cli version +``` + +Install the cores as user `pi` (the service runs as `pi` and uses `~/.arduino15`): + +```bash +arduino-cli config init +arduino-cli config add board_manager.additional_urls https://github.com/stm32duino/BoardManagerFiles/raw/main/package_stmicroelectronics_index.json +arduino-cli core update-index +arduino-cli core install arduino:avr +arduino-cli core install arduino:renesas_uno +arduino-cli core install STMicroelectronics:stm32 +``` + +The Uno R4 core ships udev rules for its bootloader; install them once: + +```bash +sudo ~/.arduino15/packages/arduino/hardware/renesas_uno/*/post_install.sh +``` + +**Expected result:** `arduino-cli core list` shows `arduino:avr`, `arduino:renesas_uno` and +`STMicroelectronics:stm32`. The STM32 core is large; the install can take 10+ minutes. If the +STM32 core fails to install on the Pi, STM32 boards can still be flashed with precompiled +`.bin`/`.elf` files from the GUI (only `.ino` compilation needs the core). + +## 4. Install rpi-hub-server + +```bash +sudo mkdir -p /opt/rpi-hub-server +sudo chown pi:pi /opt/rpi-hub-server +git clone https://github.com/hyperloop-cornell/rpi-hub-server.git /opt/rpi-hub-server +cd /opt/rpi-hub-server +python3 -m venv venv +venv/bin/pip install -r requirements.txt +``` + +**Expected result:** `venv/bin/python -c "import fastapi, serial, websockets; print('ok')"` +prints `ok`. + +## 5. Configure the hub + +```bash +cd /opt/rpi-hub-server +cp .env.example .env +chmod 600 .env +nano .env +``` + +Set these keys (leave the others at their defaults): + +```bash +HUB_PROFILE= +HUB_ID= +DEVICE_TOKEN= +SERVER_ENDPOINT=wss://api.cornellhyperloop.com/hub +``` + +**Expected result:** `grep -c DEVICE_TOKEN= .env` prints `1`, and `HUB_ID` matches the entry in +the cloud's `DEVICE_TOKENS` exactly. + +## 6. Check the boards are recognized + +Plug in each board the hub will serve, then: + +```bash +lsusb +arduino-cli board list +``` + +Compare the `ID vvvv:pppp` values from `lsusb` with `config/boards.yaml`: + +| Board | Expected `lsusb` ID | +|---|---| +| Arduino Uno R3 | `2341:0043` (or `2341:0001`, `2a03:0043`) | +| Arduino Mega 2560 | `2341:0042` | +| Nano clone (CH340) | `1a86:7523` | +| Arduino Uno R4 Minima | `2341:0069` | +| Arduino Uno R4 WiFi | `2341:1002` | +| STM32F407G-DISC1 (ST-LINK) | `0483:374b` or `0483:3752` (older firmware: `0483:3748`) | + +**Expected result:** every board's ID appears in `config/boards.yaml`. If one does not, add its +ID to the matching board entry (or a new entry) in `config/boards.yaml`. Unknown devices are +deliberately **not** opened, which is what keeps a cellular modem's serial ports untouched. + +STM32F407G-DISC1 serial output: the hub reads the ST-LINK virtual COM port. Confirm the board +revision's ST-LINK exposes one (`ls /dev/ttyACM*` shows a new device when the board is plugged in) +and that the sketch prints on the UART wired to it; otherwise serial data will not appear even +though flashing works. + +## 7. First run in the foreground + +```bash +cd /opt/rpi-hub-server +venv/bin/python -m src.main +``` + +Watch the log for about 20 seconds, then stop it with Ctrl+C. + +**Expected result:** lines containing `Connected to server` and `Server acknowledged hub +connection`, and one `connection_opened` per plugged-in board. If you see `ws_rejected`, the +`HUB_ID`/`DEVICE_TOKEN` pair does not match the cloud. If you see `Device detected but not opened`, +that device's ID is missing from `config/boards.yaml` (step 6). + +## 8. Run as a service + +```bash +sudo cp /opt/rpi-hub-server/deploy/systemd/rpi-hub-server.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now rpi-hub-server +sleep 5 +systemctl is-active rpi-hub-server +journalctl -u rpi-hub-server -n 30 --no-pager +``` + +**Expected result:** `active`, and the same connection lines as in step 7. + +## 9. Verify end to end + +On the Pi (the local API listens on loopback only): + +```bash +curl -s 127.0.0.1:8000/health +curl -s 127.0.0.1:8000/ports +curl -s 127.0.0.1:8000/connections +``` + +**Expected result:** `"uplink_connected":true`; every board is listed under `/ports` with a +`board_profile`, and has a session under `/connections`. + +In the GUI (https://gui.cornellhyperloop.com): + +1. Dashboard: the hub shows **Connected** with its profile name. +2. Device Manager: its ports are listed; subscribe to one and open Live Telemetry; data streams. +3. Firmware Flash (signed in with your NetID, not View Only), one test per board type present: + +| Board | Test | Expected | +|---|---|---| +| Uno R3 / Mega / Nano | "Blink LED" preset, board "Detected" | Flash completed; LED blinks; serial resumes | +| Uno R4 Minima / WiFi | "Blink LED" preset | Flash completed (the board briefly drops off USB and comes back) | +| Uno R4 | Upload a `.bin` built with `arduino-cli compile --fqbn arduino:renesas_uno:minima --output-dir out ` | Flash completed | +| STM32F407G-DISC1 | "Blink LED" preset (first compile can take several minutes) or a `.bin`/`.elf` | Flash completed; board runs the new program | + +4. Restart (circular arrow) on a port: task completes and serial data resumes. + +## Cellular hub (one Pi only) + +Do steps 1 to 9 with `HUB_PROFILE=cellular-hub`, then add the uplink manager, which prefers +Wi-Fi, switches to the cellular HAT when Wi-Fi fails, and reports the active link to the GUI. + +### C1. Modem and NetworkManager + +```bash +sudo apt install -y modemmanager network-manager +sudo systemctl enable --now ModemManager +mmcli -L +nmcli device status +``` + +**Expected result:** `mmcli -L` lists the modem; `nmcli device status` shows `wlan0` (wifi) and +the modem's data interface (often `wwan0`, sometimes `usb0`/`ppp0`; note the name, +``). + +Create the cellular connection (APN from the SIM provider): + +```bash +sudo nmcli connection add type gsm ifname '*' con-name cellular apn connection.autoconnect yes +sudo nmcli connection up cellular +``` + +**Expected result:** `nmcli device status` shows the cellular interface `connected`, and +`ping -I -c 3 1.1.1.1` gets replies. + +### C2. Install the uplink manager + +```bash +sudo git clone https://github.com/hyperloop-cornell/wifi-fallback-relay.git /opt/hyperloop-uplink +sudo python3 -m venv /opt/hyperloop-uplink/.venv +sudo /opt/hyperloop-uplink/.venv/bin/pip install -r /opt/hyperloop-uplink/requirements.txt +sudo mkdir -p /etc/hyperloop-uplink +sudo cp /opt/hyperloop-uplink/.env.example /etc/hyperloop-uplink/config.env +sudo nano /etc/hyperloop-uplink/config.env +``` + +Set: + +```bash +WIFI_INTERFACE=wlan0 +CELLULAR_INTERFACE= +CELLULAR_CONNECTION=cellular +DRY_RUN=true +``` + +Start it in dry-run mode first (it logs routing commands without running them): + +```bash +sudo cp /opt/hyperloop-uplink/systemd/hyperloop-uplink.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now hyperloop-uplink +sleep 15 +journalctl -u hyperloop-uplink -n 20 --no-pager +cat /run/hyperloop-uplink/status.json +``` + +**Expected result:** the log shows `DRY_RUN: nmcli ...` lines; `status.json` has +`"active": "wifi"` and both links with `"probe_ok": true`. + +Then set `DRY_RUN=false` in `/etc/hyperloop-uplink/config.env` and +`sudo systemctl restart hyperloop-uplink`. + +### C3. Failover test + +```bash +systemctl show rpi-hub-server -p NRestarts +sudo nmcli radio wifi off +sleep 30 +cat /run/hyperloop-uplink/status.json | grep '"active"' +journalctl -u rpi-hub-server -n 20 --no-pager | grep -i connected +sudo nmcli radio wifi on +sleep 90 +cat /run/hyperloop-uplink/status.json | grep '"active"' +systemctl show rpi-hub-server -p NRestarts +``` + +If you are connected over Wi-Fi SSH, the session drops when Wi-Fi goes off; run the block from a +local console, or wrap it in `sudo systemd-run --unit=failover-test bash -c '...'` and read the +results afterwards. + +**Expected result:** `"active": "cellular"` within about 30 seconds of Wi-Fi going off, and the +hub log shows it reconnected; the GUI dashboard shows a **Cellular** badge. About 60 seconds after +Wi-Fi returns, `"active": "wifi"`. `NRestarts` is the same before and after (the uplink manager +never restarts the hub). + +## Updating a hub + +```bash +cd /opt/rpi-hub-server +git pull +venv/bin/pip install -r requirements.txt +sudo systemctl restart rpi-hub-server +journalctl -u rpi-hub-server -n 20 --no-pager +``` + +**Expected result:** `Connected to server` in the log; the hub shows Connected in the GUI. + +## Migrating a hub that ran the previous software + +Older installs may live in another directory (for example `/opt/rpi-hub-service`) and may run the +old relay daemons. On such a Pi: + +```bash +sudo systemctl disable --now rpi-master-relay rpi-slave-fallback 2>/dev/null +sudo rm -f /etc/systemd/system/rpi-master-relay.service /etc/systemd/system/rpi-slave-fallback.service +sudo systemctl daemon-reload +systemctl list-units --type=service | grep -i -E "hub|relay|fallback" +``` + +Stop and disable any old hub service shown by the last command, then follow steps 4 to 9. Reuse +the old `HUB_ID` and `DEVICE_TOKEN` if the cloud still lists them. + +**Expected result:** only `rpi-hub-server` (and on the cellular hub, `hyperloop-uplink`) remain. + +## Troubleshooting + +| Symptom | Check | +|---|---| +| `ws_rejected` in the hub log | `HUB_ID` or `DEVICE_TOKEN` does not match the cloud's `DEVICE_TOKENS` | +| Hub keeps logging `Reconnecting in ...` | Network or `SERVER_ENDPOINT`; `curl -sI https://api.cornellhyperloop.com/health` from the Pi | +| Board not in `/ports` | USB cable (charge-only cables have no data lines); `lsusb`; `dmesg | tail` | +| Board in `/ports` but no session | ID missing from `config/boards.yaml`, or permissions (`groups` must include `dialout`) | +| Flash fails with "arduino-cli not found" | Step 3 install; `which arduino-cli` must print `/usr/local/bin/arduino-cli` | +| Flash fails with a missing core or platform | `arduino-cli core list` as user `pi`; install the missing core | +| STM32 flash fails in OpenOCD (`LIBUSB_ERROR_ACCESS`) | User not in `plugdev`, or udev rule missing (step 2); unplug/replug after fixing | +| Uno R4 flash hangs or the board stays in bootloader | Run the core's `post_install.sh` (step 3); double-tap reset and flash again | +| GUI shows no Wi-Fi/Cellular badge on the cellular hub | `systemctl status hyperloop-uplink`; `/run/hyperloop-uplink/status.json` must be updating | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8d2178e..d05e3ed 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,6 +19,18 @@ jobs: with: node_version: '20' + test-hub: + if: github.event.action != 'closed' + uses: ./.github/workflows/test-rpi-hub-server.yml + with: + python_version: '3.11' + + test-uplink: + if: github.event.action != 'closed' + uses: ./.github/workflows/test-uplink-manager.yml + with: + python_version: '3.11' + # Deploy when PR is merged to main deploy: if: github.event.pull_request.merged == true && github.event.action == 'closed' diff --git a/.github/workflows/test-rpi-hub-server.yml b/.github/workflows/test-rpi-hub-server.yml new file mode 100644 index 0000000..ceaadfd --- /dev/null +++ b/.github/workflows/test-rpi-hub-server.yml @@ -0,0 +1,35 @@ +name: Test RPi Hub Server + +on: + workflow_call: + inputs: + python_version: + description: 'Python version to use' + required: false + type: string + default: '3.11' + +jobs: + test: + runs-on: ubuntu-latest + name: Test RPi Hub Server + + steps: + - name: Checkout code + uses: actions/checkout@v5 + with: + submodules: recursive + token: ${{ github.token }} + + - name: Install Python dependencies + uses: ./.github/actions/install-python-requirements + with: + requirements_path: requirements.txt + working_directory: rpi-hub-server + python_version: ${{ inputs.python_version }} + + - name: Run tests + shell: bash + working-directory: rpi-hub-server + run: | + pytest tests/ -v --tb=short diff --git a/.github/workflows/test-uplink-manager.yml b/.github/workflows/test-uplink-manager.yml new file mode 100644 index 0000000..2d180af --- /dev/null +++ b/.github/workflows/test-uplink-manager.yml @@ -0,0 +1,35 @@ +name: Test Uplink Manager + +on: + workflow_call: + inputs: + python_version: + description: 'Python version to use' + required: false + type: string + default: '3.11' + +jobs: + test: + runs-on: ubuntu-latest + name: Test Uplink Manager (wifi-fallback-relay) + + steps: + - name: Checkout code + uses: actions/checkout@v5 + with: + submodules: recursive + token: ${{ github.token }} + + - name: Install Python dependencies + uses: ./.github/actions/install-python-requirements + with: + requirements_path: requirements.txt + working_directory: wifi-fallback-relay + python_version: ${{ inputs.python_version }} + + - name: Run tests + shell: bash + working-directory: wifi-fallback-relay + run: | + pytest tests/ -v --tb=short diff --git a/.github/workflows/test-web-client.yml b/.github/workflows/test-web-client.yml index bc38be1..632bf31 100644 --- a/.github/workflows/test-web-client.yml +++ b/.github/workflows/test-web-client.yml @@ -33,6 +33,16 @@ jobs: working-directory: web-client run: npm ci + - name: Check API types match the cloud contract + shell: bash + working-directory: web-client + run: | + npm run gen:types + git diff --exit-code -- src/types/api.gen.ts || { + echo "src/types/api.gen.ts is out of date: run 'npm run gen:types' in web-client and commit the result" + exit 1 + } + - name: Run TypeScript type check shell: bash working-directory: web-client diff --git a/README.md b/README.md index 187f4bc..1b29ad8 100644 --- a/README.md +++ b/README.md @@ -12,15 +12,15 @@ A modular suite for hub management, telemetry, and a modern web GUI for Hyperloo - Cloud API: [cloud-services](cloud-services) - Hub agent & server: [rpi-hub-server](rpi-hub-server) - Browser UI: [web-client](web-client) -- Fallback relay: [wifi-fallback-relay](wifi-fallback-relay) +- Uplink manager (cellular hub): [wifi-fallback-relay](wifi-fallback-relay) Anyone is welcome to view the live GUI at: https://gui.cornellhyperloop.com/ **Repository layout** - `cloud-services/` — FastAPI backend for authentication, websockets, hub coordination, and telemetry storage. -- `rpi-hub-server/` — Edge service that runs on Raspberry Pi devices: serial/USB management, hardware command tasks, and a lightweight FastAPI/WS endpoint. +- `rpi-hub-server/` — Edge service that runs on every Raspberry Pi hub (same code, per-Pi config profile): USB serial boards (Arduino Uno R3/R4, Mega, Nano, STM32F407G-DISC1), flashing, and the cloud uplink. Includes a simulated-board mode for development without hardware. - `web-client/` — React + TypeScript single-page app (Vite) for live telemetry, device control, and charting. -- `wifi-fallback-relay/` — Small Python-based relay service for network fallback scenarios. +- `wifi-fallback-relay/` — Uplink manager for the one hub with a cellular HAT: prefers Wi-Fi, falls back to cellular, and reports the active link. ## Architecture @@ -41,9 +41,10 @@ A short video of the GUI being used to operate our team's minipod is attached [h ## Technical specifics - Backend: `FastAPI` (>=0.115.0) + `uvicorn` for ASGI serving. WebSocket hubs use the `websockets` package and integrate with Pydantic models for typed messages. -- Auth: JWT-based tokens (see `cloud-services/src/api/auth.py`) with password hashing via `passlib[bcrypt]`. +- Auth: NetID allowlist plus one shared team password (bcrypt), JWT sessions, and a view-only mode; hubs authenticate with per-hub device tokens (see `cloud-services/src/auth/`). - Data models: `pydantic` / `pydantic-settings` for config and runtime validation. -- Edge: `pyserial` for hardware serial connections, `psutil` for health metrics, configurable tasks and reconnect logic in `rpi-hub-server/src/tasks/`. +- Edge: `pyserial` for serial connections, arduino-cli and OpenOCD for flashing, `psutil` for health metrics; components are wired in `rpi-hub-server/src/runtime.py`. +- Contract: `cloud-services/contracts/openapi.json` (REST + WebSocket messages) is generated from the cloud models; the web client's types are generated from it (`npm run gen:types`). - Frontend: React 19 + TypeScript, built with Vite; key libs include `recharts` for charts and `zustand` for state. ## Local development @@ -78,14 +79,21 @@ npm run dev Edge / RPi -1. Install dependencies on the target device or in a venv under `rpi-hub-server` and configure `config/config.yaml` as needed. -2. Start the hub agent with: `python -m src.main` (or use the provided systemd units in `systemd/` for production). -3. (Optional) If no physical Raspberry Pis are avaliable there is a `mock_hub_simulator.py` script at [/tests](rpi-hub-server/tests/mock_hub_simulator.py) +1. Install dependencies in a venv under `rpi-hub-server`. +2. Without hardware: `HUB_PROFILE=dev-sim python -m src.main` runs simulated boards against a local cloud-services. +3. On a Pi: follow [.claude/rpi-hub-setup.md](.claude/rpi-hub-setup.md) (profiles `lab-hub` / `cellular-hub`, systemd unit in `rpi-hub-server/deploy/systemd/`). + +The web client also has a browser-only mock: `npm run dev:mock` in `web-client`. + +## Setup guides + +- [.claude/auth-setup.md](.claude/auth-setup.md): production login (NetID allowlist + team password), hub device tokens, rotation. +- [.claude/rpi-hub-setup.md](.claude/rpi-hub-setup.md): setting up a Raspberry Pi hub, board checks, and the cellular hub's Wi-Fi/cellular failover. ## Tests & CI -- Python: `pytest` and `pytest-asyncio` are used across services. See tests in each service folder (e.g., `cloud-services/tests` and `rpi-hub-server/tests`). -- Frontend: TypeScript type checks and ESLint are enforced in `web-client` (see `package.json` scripts). +- Python: `pytest` and `pytest-asyncio` in `cloud-services/tests`, `rpi-hub-server/tests` and `wifi-fallback-relay/tests`; CI runs all three. +- Frontend: TypeScript type checks, ESLint, a production build, and a check that `src/types/api.gen.ts` matches the cloud contract. ## Assets and visuals diff --git a/cloud-services b/cloud-services index db2470d..a31fee6 160000 --- a/cloud-services +++ b/cloud-services @@ -1 +1 @@ -Subproject commit db2470df81cc22bbe438ddca8edd0e2f838f510b +Subproject commit a31fee6b078331f32dd40e7eeaa09931845384f1 diff --git a/rpi-hub-server b/rpi-hub-server index df8a764..ad07bbd 160000 --- a/rpi-hub-server +++ b/rpi-hub-server @@ -1 +1 @@ -Subproject commit df8a764d9c896d9d535783ce7bb3eb0093bb5a0b +Subproject commit ad07bbd1c791d3b0d92374f3c75ae9e23bbfdd46 diff --git a/web-client b/web-client index a591f6e..56ab093 160000 --- a/web-client +++ b/web-client @@ -1 +1 @@ -Subproject commit a591f6ea4bf39a4c6f0a3991dda229277cacb2b9 +Subproject commit 56ab093a0879c36a2bda7be2538c49683e6fe498 diff --git a/wifi-fallback-relay b/wifi-fallback-relay index 6217581..f6ff197 160000 --- a/wifi-fallback-relay +++ b/wifi-fallback-relay @@ -1 +1 @@ -Subproject commit 62175815010c60b081dc8f1c06a5f0329706d014 +Subproject commit f6ff197f9a3fc3ed53318fff12a996b67f690d6c