Files

Pawel d510d07ed9 initial commit

2026-05-21 07:21:13 -04:00

10 KiB

Raw Blame History

ClimbingBoardGPT

Applying LLM-style transformer techniques to climbing board route generation and grade prediction.

This project treats climbing routes as language and trains transformer models on data from the Tension Board 2 Mirror and Kilter Board Original — learning to predict grades and generate entirely new routes.

The Core Idea

Large language models process text as sequences of tokens and learn statistical patterns from billions of examples. Climbing routes have the same structure:

NLP Concept	Climbing Analog
Word / Subword	Hold token (`TB2_p344_start`)
Sentence	Route (sequence of holds)
Document language	Board type (TB2 vs Kilter)
POS tag	Semantic role (start / middle / finish / foot)
Genre / Domain	Angle + Grade conditioning
Special tokens	`<BOS>`, `<EOS>`, `<PAD>`, `<CLS>`, `<MASK>`, `<UNK>`

A route becomes a symbolic sequence:

<BOS> <BOARD_TB2> <ANGLE_40> <GRADE_V6>
<TB2_p344_start> <TB2_p369_middle> <TB2_p603_finish>
<EOS>

The same transformer architectures that power GPT and BERT can learn "climb grammar" — which holds tend to follow which, how start holds differ from finish holds, and how difficulty emerges from spatial relationships.

What This Repo Does

1. Tokenization (`01_tokenize_routes`)

Converts raw SQLite data into tokenized sequences:

Parses frames strings (e.g., p344r5p369r6p603r7) into structured hold records
Maps board-specific role IDs to shared semantic roles (TB2: 5/6/7/8 → Kilter: 12/13/14/15 → start/middle/finish/foot)
Sorts holds canonically by (role priority, y-position, x-position)
Generates two sequence versions:
- With grade — for GPT generation training
- Without grade — for BERT-style grade prediction
Builds a shared vocabulary (~4,400 tokens), stratified train/val/test splits, and coordinate metadata

2. Grade Prediction (`02_train_grade_predictor`)

Trains a transformer encoder (BERT-style) to predict climb difficulty:

Input: <CLS> <BOARD_TB2> <ANGLE_40> <TB2_p344_start> ... (grade excluded)
Output: Single difficulty score (regression)
Coordinate features (x, y, is_hold) are projected and added to token embeddings
Joint training across both boards with board-conditioning tokens

Results (joint model, test set):

Metric	Overall	TB2	Kilter
MAE	1.47	1.42	1.48
R²	0.787	0.816	0.782
Within ±1 V-grade	80.1%	81.3%	80.0%
Within ±2 V-grades	95.3%	96.1%	95.2%

3. Route Generation (`03_train_route_generator`)

Trains a GPT-style causal transformer to generate new routes:

Input prompt: <BOS> <BOARD_TB2> <ANGLE_40> <GRADE_V6>
Output: Sequence of hold tokens ending with <EOS>
Uses causal masking (each position attends only to previous positions)
Generation uses temperature sampling and top-k filtering

Training results:

Best validation perplexity: ~24.6
88.8% basic validity rate for generated routes

4. Evaluation (`04_evaluate_generated_routes`)

Evaluates generated routes on four dimensions:

Validity: Structural correctness (start/finish holds, no duplicates, single board)
Novelty: Jaccard distance from nearest real route
Geometric plausibility: Height, width, reach distances
Grade consistency: Uses the trained grade predictor as a critic

Evaluation results:

Metric	TB2	Kilter
Basic validity	87.0%	90.5%
Mean novelty distance	0.661	0.642
Exact V-grade match	27.5%	33.5%
Within ±1 V-grade	66.0%	67.5%
Within ±2 V-grades	91.0%	90.0%

Key Design Decisions

Board Namespacing

Hold tokens include the board prefix (TB2_p344_start vs KILTER_p1084_start). Placement 344 on TB2 is a completely different physical hold than placement 344 on Kilter — the prefix prevents ID collisions.

Semantic Role Mapping

Different boards use different numeric role IDs, but they all map to the same semantic roles:

Role	TB2	Kilter
Start	5	12
Middle	6	13
Finish	7	14
Foot	8	15

This shared vocabulary lets the model learn transferable patterns across boards.

Coordinate Features

Each hold token carries physical (x, y) position information that gets projected and added to token embeddings. This gives the model direct spatial knowledge — similar to how some vision-language models inject spatial features.

Conditioning Tokens

Routes are prefixed with board, angle, and grade tokens. This is analogous to how modern LLMs use system prompts to condition generation.

Repository Structure

ClimbingBoardGPT/
├── configs/
│   ├── tb2.json                    # Tension Board 2 configuration
│   └── kilter.json                 # Kilter Board configuration
├── data/
│   ├── raw/                        # SQLite databases (not in repo)
│   └── processed/
│       ├── tokenized/               # Tokenized route data
│       ├── grade_prediction/        # Grade predictor outputs
│       ├── generation/              # Generated route data
│       └── evaluation/              # Evaluation results
├── models/                          # Saved model checkpoints
├── notebooks/
│   ├── 01_unified_route_tokenization.ipynb
│   ├── 02_joint_transformer_grade_prediction.ipynb
│   ├── 03_joint_nanogpt_route_generator.ipynb
│   └── 04_generated_route_evaluation.ipynb
├── scripts/
│   ├── 01_tokenize_routes.py
│   ├── 02_train_grade_predictor.py
│   ├── 03_train_route_generator.py
│   └── 04_evaluate_generated_routes.py
├── src/climbingboardgpt/
│   ├── __init__.py
│   ├── config.py                   # Board configuration loading
│   ├── data.py                     # SQLite data loading
│   ├── datasets.py                 # PyTorch Dataset classes
│   ├── evaluation.py               # Route evaluation functions
│   ├── generation.py               # Route generation logic
│   ├── grades.py                   # Grade-to-V mapping
│   ├── metrics.py                  # Evaluation metrics
│   ├── models.py                   # Transformer architectures
│   ├── paths.py                    # Project root detection
│   ├── tokenization.py             # Core tokenization logic
│   └── utils.py                    # Utility functions
├── requirements.txt
├── pyproject.toml
└── README.md

Setup

# Clone the repo
git clone https://github.com/yourusername/ClimbingBoardGPT.git
cd ClimbingBoardGPT

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # Linux/Mac
# .venv\Scripts\activate     # Windows

# Install dependencies
pip install -r requirements.txt
pip install -e .

Retrieving Raw Databases

The project expects SQLite databases at data/raw/tb2.db and data/raw/kilter.db.

Using BoardLib:

pip install boardlib
boardlib database tension data/raw/tb2.db
boardlib database kilter data/raw/kilter.db

Running the Pipeline

1. Tokenize both boards

python scripts/01_tokenize_routes.py --boards tb2,kilter

Creates data/processed/tokenized/ with vocabulary, route sequences, and metadata.

2. Train the grade predictor

python scripts/02_train_grade_predictor.py

Trains a BERT-style transformer encoder and saves to models/joint_transformer_grade_predictor.pth.

3. Train the route generator

python scripts/03_train_route_generator.py

Trains a GPT-style causal transformer and saves to models/joint_route_gpt_generator.pth.

4. Evaluate generated routes

python scripts/04_evaluate_generated_routes.py

Evaluates validity, novelty, geometry, and grade consistency. Saves results to data/processed/evaluation/.

Model Architectures

JointRouteTransformerRegressor (Grade Prediction)

Input: [CLS] BOARD ANGLE HOLDS...
         ↓
Token Embedding + Position Embedding + Coordinate Features
         ↓
Transformer Encoder (4 layers, 4 heads, d_model=128)
         ↓
[CLS] token output → Regression Head → difficulty score

~1.17M parameters
MSE loss, AdamW optimizer
Early stopping on validation MAE

JointRouteGPT (Route Generation)

Input: BOS BOARD ANGLE GRADE HOLDS...
         ↓
Token Embedding + Position Embedding
         ↓
Causal Transformer (4 layers, 4 heads, d_embd=128)
         ↓
Language Modeling Head → next token logits

~1.41M parameters
Cross-entropy loss, AdamW optimizer
Weight tying between embedding and output layers

Board Configuration

Setting	TB2 Mirror	Kilter Original
`layout_id`	10	1
`token_prefix`	TB2	KILTER
`max_angle`	50	55
`role_definitions`	start=5, middle=6, finish=7, foot=8	start=12, middle=13, finish=14, foot=15
`include_mirror_placement_id`	true	false
`min_fa_date`	null	2016-01-01

To add a new board, create a JSON config in configs/ following the same format.

Comparison with Classical Approach

The earlier TB2 project used hand-engineered features with Random Forest and neural networks. This project replaces feature engineering with transformer attention:

Aspect	Classical (TB2 Notebooks 01-06)	Transformer (This Project)
Input	30+ engineered features	Raw token sequences
Feature engineering	Manual (spatial, geometric)	Learned via attention
Board handling	Single board (TB2)	Joint model with board token
Grade prediction	Random Forest / MLP	Transformer encoder
Route generation	Not supported	GPT-style decoder
Interpretability	Feature importance	Attention weights

Future Extensions

Masked hold prediction: Mask holds and predict them (like BERT's MLM)
Stronger legality constraints: Enforce valid start/finish positions in generation
Board transfer experiments: Train on TB2, evaluate on Kilter (zero-shot)
GUI for route generation: Interactive tool to generate and visualize climbs
Integration with classical features: Combine transformer embeddings with engineered features

Acknowledgments

Board data from Tension Climbing and Kilter Board
Database access via BoardLib
Original TB2 analysis notebooks for foundational data exploration

10 KiB Raw Blame History