# ClimbingBoardGPT **Applying LLM-style transformer techniques to climbing board route generation and grade prediction.** This project treats climbing routes as language and trains transformer models on data from the **Tension Board 2 Mirror** and **Kilter Board Original** — learning to predict grades and generate entirely new routes. --- ## The Core Idea Large language models process text as sequences of tokens and learn statistical patterns from billions of examples. Climbing routes have the same structure: | NLP Concept | Climbing Analog | |---|---| | Word / Subword | Hold token (`TB2_p344_start`) | | Sentence | Route (sequence of holds) | | Document language | Board type (TB2 vs Kilter) | | POS tag | Semantic role (start / middle / finish / foot) | | Genre / Domain | Angle + Grade conditioning | | Special tokens | ``, ``, ``, ``, ``, `` | A route becomes a symbolic sequence: ```text ``` The same transformer architectures that power GPT and BERT can learn "climb grammar" — which holds tend to follow which, how start holds differ from finish holds, and how difficulty emerges from spatial relationships. --- ## What This Repo Does ### 1. Tokenization (`01_tokenize_routes`) Converts raw SQLite data into tokenized sequences: - Parses `frames` strings (e.g., `p344r5p369r6p603r7`) into structured hold records - Maps board-specific role IDs to shared semantic roles (TB2: 5/6/7/8 → Kilter: 12/13/14/15 → `start`/`middle`/`finish`/`foot`) - Sorts holds canonically by (role priority, y-position, x-position) - Generates two sequence versions: - **With grade** — for GPT generation training - **Without grade** — for BERT-style grade prediction - Builds a shared vocabulary (~4,400 tokens), stratified train/val/test splits, and coordinate metadata ### 2. Grade Prediction (`02_train_grade_predictor`) Trains a **transformer encoder** (BERT-style) to predict climb difficulty: - Input: ` ...` (grade excluded) - Output: Single difficulty score (regression) - Coordinate features (x, y, is_hold) are projected and added to token embeddings - Joint training across both boards with board-conditioning tokens **Results (joint model, test set):** | Metric | Overall | TB2 | Kilter | |---|---|---|---| | MAE | 1.47 | 1.42 | 1.48 | | R² | 0.787 | 0.816 | 0.782 | | Within ±1 V-grade | 80.1% | 81.3% | 80.0% | | Within ±2 V-grades | 95.3% | 96.1% | 95.2% | ### 3. Route Generation (`03_train_route_generator`) Trains a **GPT-style causal transformer** to generate new routes: - Input prompt: ` ` - Output: Sequence of hold tokens ending with `` - Uses causal masking (each position attends only to previous positions) - Generation uses temperature sampling and top-k filtering **Training results:** - Best validation perplexity: ~24.6 - 88.8% basic validity rate for generated routes ### 4. Evaluation (`04_evaluate_generated_routes`) Evaluates generated routes on four dimensions: - **Validity**: Structural correctness (start/finish holds, no duplicates, single board) - **Novelty**: Jaccard distance from nearest real route - **Geometric plausibility**: Height, width, reach distances - **Grade consistency**: Uses the trained grade predictor as a critic **Evaluation results:** | Metric | TB2 | Kilter | |---|---|---| | Basic validity | 87.0% | 90.5% | | Mean novelty distance | 0.661 | 0.642 | | Exact V-grade match | 27.5% | 33.5% | | Within ±1 V-grade | 66.0% | 67.5% | | Within ±2 V-grades | 91.0% | 90.0% | --- ## Key Design Decisions ### Board Namespacing Hold tokens include the board prefix (`TB2_p344_start` vs `KILTER_p1084_start`). Placement 344 on TB2 is a completely different physical hold than placement 344 on Kilter — the prefix prevents ID collisions. ### Semantic Role Mapping Different boards use different numeric role IDs, but they all map to the same semantic roles: | Role | TB2 | Kilter | |---|---|---| | Start | 5 | 12 | | Middle | 6 | 13 | | Finish | 7 | 14 | | Foot | 8 | 15 | This shared vocabulary lets the model learn transferable patterns across boards. ### Coordinate Features Each hold token carries physical (x, y) position information that gets projected and added to token embeddings. This gives the model direct spatial knowledge — similar to how some vision-language models inject spatial features. ### Conditioning Tokens Routes are prefixed with board, angle, and grade tokens. This is analogous to how modern LLMs use system prompts to condition generation. --- ## Repository Structure ```text ClimbingBoardGPT/ ├── configs/ │ ├── tb2.json # Tension Board 2 configuration │ └── kilter.json # Kilter Board configuration ├── data/ │ ├── raw/ # SQLite databases (not in repo) │ └── processed/ │ ├── tokenized/ # Tokenized route data │ ├── grade_prediction/ # Grade predictor outputs │ ├── generation/ # Generated route data │ └── evaluation/ # Evaluation results ├── models/ # Saved model checkpoints ├── notebooks/ │ ├── 01_unified_route_tokenization.ipynb │ ├── 02_joint_transformer_grade_prediction.ipynb │ ├── 03_joint_nanogpt_route_generator.ipynb │ └── 04_generated_route_evaluation.ipynb ├── scripts/ │ ├── 01_tokenize_routes.py │ ├── 02_train_grade_predictor.py │ ├── 03_train_route_generator.py │ └── 04_evaluate_generated_routes.py ├── src/climbingboardgpt/ │ ├── __init__.py │ ├── config.py # Board configuration loading │ ├── data.py # SQLite data loading │ ├── datasets.py # PyTorch Dataset classes │ ├── evaluation.py # Route evaluation functions │ ├── generation.py # Route generation logic │ ├── grades.py # Grade-to-V mapping │ ├── metrics.py # Evaluation metrics │ ├── models.py # Transformer architectures │ ├── paths.py # Project root detection │ ├── tokenization.py # Core tokenization logic │ └── utils.py # Utility functions ├── requirements.txt ├── pyproject.toml └── README.md ``` --- ## Setup ```bash # Clone the repo git clone https://github.com/yourusername/ClimbingBoardGPT.git cd ClimbingBoardGPT # Create and activate virtual environment python -m venv .venv source .venv/bin/activate # Linux/Mac # .venv\Scripts\activate # Windows # Install dependencies pip install -r requirements.txt pip install -e . ``` ### Retrieving Raw Databases The project expects SQLite databases at `data/raw/tb2.db` and `data/raw/kilter.db`. Using [BoardLib](https://github.com/lemeryf/BoardLib): ```bash pip install boardlib boardlib database tension data/raw/tb2.db boardlib database kilter data/raw/kilter.db ``` --- ## Running the Pipeline ### 1. Tokenize both boards ```bash python scripts/01_tokenize_routes.py --boards tb2,kilter ``` Creates `data/processed/tokenized/` with vocabulary, route sequences, and metadata. ### 2. Train the grade predictor ```bash python scripts/02_train_grade_predictor.py ``` Trains a BERT-style transformer encoder and saves to `models/joint_transformer_grade_predictor.pth`. ### 3. Train the route generator ```bash python scripts/03_train_route_generator.py ``` Trains a GPT-style causal transformer and saves to `models/joint_route_gpt_generator.pth`. ### 4. Evaluate generated routes ```bash python scripts/04_evaluate_generated_routes.py ``` Evaluates validity, novelty, geometry, and grade consistency. Saves results to `data/processed/evaluation/`. --- ## Model Architectures ### JointRouteTransformerRegressor (Grade Prediction) ``` Input: [CLS] BOARD ANGLE HOLDS... ↓ Token Embedding + Position Embedding + Coordinate Features ↓ Transformer Encoder (4 layers, 4 heads, d_model=128) ↓ [CLS] token output → Regression Head → difficulty score ``` - ~1.17M parameters - MSE loss, AdamW optimizer - Early stopping on validation MAE ### JointRouteGPT (Route Generation) ``` Input: BOS BOARD ANGLE GRADE HOLDS... ↓ Token Embedding + Position Embedding ↓ Causal Transformer (4 layers, 4 heads, d_embd=128) ↓ Language Modeling Head → next token logits ``` - ~1.41M parameters - Cross-entropy loss, AdamW optimizer - Weight tying between embedding and output layers --- ## Board Configuration | Setting | TB2 Mirror | Kilter Original | |---|---:|---| | `layout_id` | 10 | 1 | | `token_prefix` | TB2 | KILTER | | `max_angle` | 50 | 55 | | `role_definitions` | start=5, middle=6, finish=7, foot=8 | start=12, middle=13, finish=14, foot=15 | | `include_mirror_placement_id` | true | false | | `min_fa_date` | null | 2016-01-01 | To add a new board, create a JSON config in `configs/` following the same format. --- ## Comparison with Classical Approach The earlier TB2 project used hand-engineered features with Random Forest and neural networks. This project replaces feature engineering with transformer attention: | Aspect | Classical (TB2 Notebooks 01-06) | Transformer (This Project) | |---|---|---| | Input | 30+ engineered features | Raw token sequences | | Feature engineering | Manual (spatial, geometric) | Learned via attention | | Board handling | Single board (TB2) | Joint model with board token | | Grade prediction | Random Forest / MLP | Transformer encoder | | Route generation | Not supported | GPT-style decoder | | Interpretability | Feature importance | Attention weights | --- ## Future Extensions - **Masked hold prediction**: Mask holds and predict them (like BERT's MLM) - **Stronger legality constraints**: Enforce valid start/finish positions in generation - **Board transfer experiments**: Train on TB2, evaluate on Kilter (zero-shot) - **GUI for route generation**: Interactive tool to generate and visualize climbs - **Integration with classical features**: Combine transformer embeddings with engineered features --- ## Acknowledgments - Board data from [Tension Climbing](https://tensionclimbing.com/) and [Kilter Board](https://kilterboard.com/) - Database access via [BoardLib](https://github.com/lemeryf/BoardLib) - Original TB2 analysis notebooks for foundational data exploration