{
"cells": [
{
"cell_type": "markdown",
"id": "d273a5e0",
"metadata": {},
"source": [
"# Factor Construction and Composite Signal\n",
"\n",
"## Purpose\n",
"\n",
"Notebook 02 suggested that momentum is the only price-based signal worth carrying forward. This notebook turns that raw momentum rank into the traded signal used by the backtest.\n",
"\n",
"The goals are:\n",
"1. Winsorize and z-score each factor cross-section so outliers do not dominate.\n",
"2. Neutralize the signal against sector membership by projecting out sector effects.\n",
"3. Compare momentum-only against a simple four-factor composite.\n",
"4. Save the sector-neutralized momentum signal for notebook 04.\n",
"\n",
"### Terms used in this notebook\n",
"\n",
"| Term | Meaning |\n",
"|------|---------|\n",
"| **Winsorization** | Clip extreme values at a threshold instead of dropping them |\n",
"| **Z-scoring** | Center to mean 0 and scale to standard deviation 1 |\n",
"| **Neutralization** | Regress a signal on unwanted exposures and keep the residual |\n",
"| **Composite signal** | A linear combination $c_t = \\sum_k w_k f_{k,\\perp}$ |\n",
"| **Sector matrix** | $D \\in \\{0,1\\}^{N \\times K}$, a one-hot sector-membership matrix |\n",
"| **Projection** | The linear algebra operation behind neutralization |\n",
"| **Information coefficient (IC)** ↻ | Spearman rank correlation used to compare candidate signals |\n",
"| **Return panel** $R$ ↻ | The monthly return matrix from notebook 01 |\n",
"\n",
"## Outputs\n",
"\n",
"- `momentum_signal.csv`: sector-neutralized momentum, the signal used in the backtest\n",
"- `composite_4factor.csv`: a simple four-factor composite kept for comparison\n",
"- `factor_*_neutralized.csv`: neutralized versions of the individual factors\n",
"\n",
"## Notebook Structure\n",
"1. [Setup and Imports](#setup-and-imports)\n",
"2. [Winsorization and Z-Scoring](#winsorization-and-z-scoring)\n",
"3. [Sector Neutralization](#sector-neutralization)\n",
"4. [Composite Assembly and Comparison](#composite-assembly-and-comparison)\n",
"5. [IC Comparison: Momentum vs. Composite](#ic-comparison-momentum-vs-composite)\n",
"6. [Conclusion](#conclusion)"
]
},
{
"cell_type": "markdown",
"id": "75cbb165",
"metadata": {},
"source": [
"\n",
"## Setup and Imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c72adc67",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-31T11:08:52.187310Z",
"iopub.status.busy": "2026-07-31T11:08:52.186272Z",
"iopub.status.idle": "2026-07-31T11:08:53.251453Z",
"shell.execute_reply": "2026-07-31T11:08:53.250854Z"
}
},
"outputs": [],
"source": [
"\"\"\"\n",
"==================================\n",
"Setup and imports\n",
"==================================\n",
"\"\"\"\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import os\n",
"from scipy.stats import spearmanr\n",
"\n",
"pd.set_option('display.max_columns', None)\n",
"pd.set_option('display.max_rows', 100)\n",
"\n",
"palette = ['steelblue', 'coral', 'seagreen']\n",
"\n",
"os.makedirs('../data/processed', exist_ok=True)\n",
"os.makedirs('../images/03_factor_construction', exist_ok=True)\n",
"\n",
"RANDOM_STATE = 3"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "0090dab7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-31T11:08:53.253326Z",
"iopub.status.busy": "2026-07-31T11:08:53.253053Z",
"iopub.status.idle": "2026-07-31T11:08:53.408971Z",
"shell.execute_reply": "2026-07-31T11:08:53.408532Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 4 factors\n",
"Returns: (251, 501)\n"
]
}
],
"source": [
"\"\"\"\n",
"==================================\n",
"Load factor exposures, returns, sectors\n",
"==================================\n",
"\"\"\"\n",
"factor_names = ['momentum', 'value', 'quality', 'lowvol']\n",
"factor_dict = {n: pd.read_csv(f'../data/processed/factor_{n}.csv', index_col=0, parse_dates=True) for n in factor_names}\n",
"df_returns = pd.read_csv('../data/processed/returns_monthly.csv', index_col=0, parse_dates=True)\n",
"df_sector = pd.read_csv('../data/processed/sector_mapping.csv')\n",
"\n",
"# Build sector lookup\n",
"sector_map = dict(zip(df_sector['ticker'], df_sector['sector']))\n",
"\n",
"print(f\"Loaded {len(factor_names)} factors\")\n",
"print(f\"Returns: {df_returns.shape}\")"
]
},
{
"cell_type": "markdown",
"id": "46a37025",
"metadata": {},
"source": [
"## Winsorization and Z-Scoring\n",
"\n",
"Raw factor values can contain large outliers. Winsorization clips those extremes rather than dropping the stock entirely. Here we cap each monthly cross-section at $\\pm 3$ standard deviations around its mean.\n",
"\n",
"After clipping, we z-score within each date:\n",
"\n",
"$$\\tilde f_{t,i}=\\frac{f_{t,i}-\\bar f_t}{\\mathrm{std}(f_t)}.$$\n",
"\n",
"This gives each factor row mean 0 and standard deviation 1, so later comparisons are not driven by arbitrary units."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "83a54dbc",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-31T11:08:53.410714Z",
"iopub.status.busy": "2026-07-31T11:08:53.410441Z",
"iopub.status.idle": "2026-07-31T11:08:54.165454Z",
"shell.execute_reply": "2026-07-31T11:08:54.164580Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"momentum: z-scores, shape=(251, 501)\n",
"value: z-scores, shape=(251, 501)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"quality: z-scores, shape=(251, 501)\n",
"lowvol: z-scores, shape=(251, 501)\n"
]
}
],
"source": [
"\"\"\"\n",
"==================================\n",
"Winsorize and z-score\n",
"==================================\n",
"\"\"\"\n",
"def winsorize(series, sigma=3):\n",
" \"\"\"Clip at ±sigma standard deviations around the mean.\"\"\"\n",
" mu = series.mean()\n",
" s = series.std()\n",
" if s == 0 or pd.isna(s):\n",
" return series\n",
" return series.clip(lower=mu - sigma*s, upper=mu + sigma * s)\n",
"\n",
"def zscore(series):\n",
" \"\"\"Cross-sectional z-score.\"\"\"\n",
" mu = series.mean()\n",
" s = series.std()\n",
" if s == 0 or pd.isna(s):\n",
" return series * 0\n",
" return (series - mu)/s\n",
"\n",
"factor_z = {}\n",
"for name, df_f in factor_dict.items():\n",
" df_w = df_f.apply(winsorize, axis=1)\n",
" df_z = df_w.apply(zscore, axis=1)\n",
" factor_z[name] = df_z\n",
" print(f\"{name}: z-scores, shape={df_z.shape}\")"
]
},
{
"cell_type": "markdown",
"id": "112a7744",
"metadata": {},
"source": [
"## Sector Neutralization\n",
"\n",
"A raw momentum score may contain sector bets. For example, if Technology had a strong year, a momentum portfolio might become mostly a Technology portfolio. That may be a valid trade, but it is not a clean test of stock selection within sectors.\n",
"\n",
"To neutralize sectors, create a sector dummy matrix\n",
"\n",
"$$D \\in \\{0,1\\}^{N \\times K}.$$\n",
"\n",
"For one factor vector $f \\in \\mathbb{R}^N$, the projection onto the sector span is\n",
"\n",
"$$P_D f = D(D^\\top D)^{-1}D^\\top f.$$\n",
"\n",
"The sector-neutralized signal is the residual:\n",
"\n",
"$$f_\\perp = (I-P_D)f.$$\n",
"\n",
"This is the same as running a cross-sectional OLS regression of the factor on sector dummies and keeping the residuals. The residual has zero linear exposure to the sector dummy columns used in the regression.\n",
"\n",
"The code also mentions size, but with this dataset we only have a weak price-based size proxy. I would not interpret it as a true market-cap neutralization without real shares-outstanding data."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b7ce2018",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-31T11:08:54.167456Z",
"iopub.status.busy": "2026-07-31T11:08:54.167259Z",
"iopub.status.idle": "2026-07-31T11:08:56.142616Z",
"shell.execute_reply": "2026-07-31T11:08:56.141776Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Neutralizing momentum...\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Neutralizing value...\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Neutralizing quality...\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Neutralizing lowvol...\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sector neutralization complete.\n"
]
}
],
"source": [
"\"\"\"\n",
"==================================\n",
"Sector neutralization via cross-sectional regression on dummies\n",
"==================================\n",
"\"\"\"\n",
"def neutralize_against_sector(factor_row, sector_map):\n",
" \"\"\"Regress factor on sector dummies, return residuals.\"\"\"\n",
" tickers = factor_row.index\n",
" sectors = pd.Series([sector_map.get(t, 'Unknown') for t in tickers], index=tickers)\n",
"\n",
" # One-hot encode\n",
" indicators = pd.get_dummies(sectors, drop_first=True).astype(float)\n",
" indicators['intercept'] = 1.0\n",
"\n",
" # Mask valid\n",
" mask = factor_row.notna()\n",
" if mask.sum() < 30:\n",
" return factor_row\n",
"\n",
" y = factor_row[mask].values\n",
" X = indicators.loc[mask].values\n",
"\n",
" # OLS via least squares\n",
" beta, _, _, _ = np.linalg.lstsq(X,y)\n",
" resid = y - X @ beta\n",
"\n",
" out = pd.Series(np.nan, index=factor_row.index)\n",
" out[mask] = resid\n",
" return out\n",
"\n",
"factor_neut = {}\n",
"for name, df_z in factor_z.items():\n",
" print(f\"Neutralizing {name}...\")\n",
" df_neut = df_z.apply(lambda row: neutralize_against_sector(row, sector_map), axis=1)\n",
" # Re-z-score the residuals so they're back on a common scale\n",
" df_neut = df_neut.apply(zscore, axis=1)\n",
" factor_neut[name] = df_neut\n",
"\n",
"print(\"Sector neutralization complete.\")"
]
},
{
"cell_type": "markdown",
"id": "9740141a",
"metadata": {},
"source": [
"As in notebook 02, we use a sample-size guardrail. If fewer than 30 stocks have valid data in a month, we skip neutralization for that row rather than fit a noisy cross-sectional regression. This mostly affects early or sparse parts of the panel."
]
},
{
"cell_type": "markdown",
"id": "b4204816",
"metadata": {},
"source": [
"## Composite Assembly and Comparison\n",
"\n",
"We compare two candidate signals:\n",
"\n",
"1. **Momentum-only:** the sector-neutralized momentum residual $f_{m,\\perp}$, re-z-scored. This is the signal passed to the backtest.\n",
"2. **Four-factor composite:** an equal-weight average of the neutralized factor vectors,\n",
"\n",
"$$c_t = \\frac{1}{4}\\sum_{k=1}^4 f_{k,\\perp}(t).$$\n",
"\n",
"This is a useful sanity check. If the composite improves IC, the extra factors are helping. If it gets worse, the added factors are diluting momentum rather than diversifying it."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "792e547e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-31T11:08:56.144507Z",
"iopub.status.busy": "2026-07-31T11:08:56.144304Z",
"iopub.status.idle": "2026-07-31T11:08:56.350858Z",
"shell.execute_reply": "2026-07-31T11:08:56.350129Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Momentum-only signal: (251, 501)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"4-factor composite: (251, 501)\n",
"\n",
"Our main signal will be momentum-only. The 4-factor composite is kept for comparison.\n"
]
}
],
"source": [
"\"\"\"\n",
"==================================\n",
"Build momentum-only signal and 4-factor composite\n",
"==================================\n",
"\"\"\"\n",
"# Momentum only\n",
"df_momentum_signal = factor_neut['momentum'].apply(zscore, axis=1)\n",
"print(f\"Momentum-only signal: {df_momentum_signal.shape}\")\n",
"\n",
"# 4-factor equal-weight composite\n",
"common_dates = factor_neut['momentum'].index\n",
"common_tickers = factor_neut['momentum'].columns\n",
"\n",
"for name in factor_names:\n",
" common_dates = common_dates.intersection(factor_neut[name].index)\n",
" common_tickers = common_tickers.intersection(factor_neut[name].columns)\n",
"\n",
"df_composite_4f = pd.DataFrame(0.0, index=common_dates, columns=common_tickers)\n",
"count_df = pd.DataFrame(0, index=common_dates, columns=common_tickers)\n",
"\n",
"for name in factor_names:\n",
" df_f = factor_neut[name].loc[common_dates, common_tickers]\n",
" df_composite_4f = df_composite_4f.add(df_f.fillna(0))\n",
" count_df = count_df.add(df_f.notna().astype(int))\n",
"\n",
"df_composite_4f = df_composite_4f / count_df.replace(0, np.nan)\n",
"df_composite_4f[count_df < 3] = np.nan\n",
"df_composite_4f = df_composite_4f.apply(zscore, axis=1)\n",
"\n",
"print(f\"4-factor composite: {df_composite_4f.shape}\")\n",
"print(f\"\\nOur main signal will be momentum-only. The 4-factor composite is kept for comparison.\")"
]
},
{
"cell_type": "markdown",
"id": "f8809b2e",
"metadata": {},
"source": [
"## Composite vs. Single Factors"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "25e7a046",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-31T11:08:56.353026Z",
"iopub.status.busy": "2026-07-31T11:08:56.352756Z",
"iopub.status.idle": "2026-07-31T11:08:58.700588Z",
"shell.execute_reply": "2026-07-31T11:08:58.699933Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"IC Comparison: Neutralized Factors, Momentum-Only, and 4-Factor Composite\n",
"\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" mean | \n",
" std | \n",
" ic_ir | \n",
"
\n",
" \n",
" \n",
" \n",
" | momentum | \n",
" 0.0068 | \n",
" 0.1576 | \n",
" 0.1505 | \n",
"
\n",
" \n",
" | value | \n",
" -0.0198 | \n",
" 0.1344 | \n",
" -0.5111 | \n",
"
\n",
" \n",
" | quality | \n",
" 0.0013 | \n",
" 0.1597 | \n",
" 0.0282 | \n",
"
\n",
" \n",
" | lowvol | \n",
" -0.0169 | \n",
" 0.1772 | \n",
" -0.3298 | \n",
"
\n",
" \n",
" | momentum_only | \n",
" 0.0068 | \n",
" 0.1576 | \n",
" 0.1505 | \n",
"
\n",
" \n",
" | 4factor_composite | \n",
" -0.0078 | \n",
" 0.1688 | \n",
" -0.1593 | \n",
"
\n",
" \n",
"
\n",
"