{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4777cf60",
   "metadata": {},
   "source": [
    "# Risk Decomposition via PCA\n",
    "\n",
    "## Purpose\n",
    "\n",
    "Notebook 04 tells us how the strategy performed. This notebook asks where the portfolio's risk comes from.\n",
    "\n",
    "Portfolio variance is the quadratic form\n",
    "\n",
    "$$w^\\top\\Sigma w,$$\n",
    "\n",
    "where $w$ is the portfolio weight vector and $\\Sigma$ is the stock-return covariance matrix. PCA gives an orthogonal basis for this covariance matrix, so we can split variance into common-factor directions and the remaining residual directions.\n",
    "\n",
    "The goals are:\n",
    "1. Estimate the sample covariance matrix $\\Sigma=\\frac{1}{T-1}X_c^\\top X_c$.\n",
    "2. Eigendecompose $\\Sigma$ with PCA.\n",
    "3. Use the Marchenko-Pastur cutoff to separate large signal-like eigenvalues from the noise bulk.\n",
    "4. Compare sample covariance to Ledoit-Wolf shrinkage.\n",
    "5. Decompose the current long-only portfolio variance into top-$k$ PCA risk and the orthogonal remainder.\n",
    "\n",
    "### Terms used in this notebook\n",
    "\n",
    "| Term | Meaning |\n",
    "|------|---------|\n",
    "| **Covariance matrix** $\\Sigma$ | $N\\times N$ matrix with $\\Sigma_{ij}=\\mathrm{Cov}(r_i,r_j)$ |\n",
    "| **PCA** | Eigendecomposition of $\\Sigma$ |\n",
    "| **Eigenvector** | A portfolio-like direction in stock space |\n",
    "| **Eigenvalue** | Variance explained along that eigenvector |\n",
    "| **Scree plot** | Eigenvalues plotted from largest to smallest |\n",
    "| **Marchenko-Pastur distribution** | Random-matrix benchmark for the eigenvalue noise bulk |\n",
    "| **Shrinkage estimator** | A weighted average of noisy sample covariance and a structured target |\n",
    "| **Systematic risk** | Variance in common PCA directions |\n",
    "| **Idiosyncratic risk** | Variance outside the retained PCA directions |\n",
    "| **Portfolio variance** | The quantity $w^\\top\\Sigma w$ being decomposed |\n",
    "\n",
    "## Outputs\n",
    "\n",
    "Eigenvalue spectrum, Marchenko-Pastur cutoff, Ledoit-Wolf comparison, and a portfolio variance decomposition table.\n",
    "\n",
    "## Notebook Structure\n",
    "1. [Setup and Imports](#setup-and-imports)\n",
    "2. [Load Data](#load-data)\n",
    "3. [Sample Covariance and PCA](#sample-covariance-and-pca)\n",
    "4. [Marchenko-Pastur Noise Separation](#marchenkopastur-noise-separation)\n",
    "5. [Ledoit-Wolf Shrinkage](#ledoitwolf-shrinkage)\n",
    "6. [Portfolio Variance Decomposition](#portfolio-variance-decomposition)\n",
    "7. [Conclusion](#conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08545070",
   "metadata": {},
   "source": [
    "## Setup and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4f9e782b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:08.396698Z",
     "iopub.status.busy": "2026-07-31T11:09:08.395783Z",
     "iopub.status.idle": "2026-07-31T11:09:09.597947Z",
     "shell.execute_reply": "2026-07-31T11:09:09.597320Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Setup and imports\n",
    "==================================\n",
    "\"\"\"\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import os\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', 100)\n",
    "\n",
    "palette = ['steelblue', 'coral', 'seagreen']\n",
    "\n",
    "os.makedirs('../data/processed', exist_ok=True)\n",
    "os.makedirs('../images/05_risk_decomposition', exist_ok=True)\n",
    "\n",
    "RANDOM_STATE = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e1c87a97",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:09.599975Z",
     "iopub.status.busy": "2026-07-31T11:09:09.599687Z",
     "iopub.status.idle": "2026-07-31T11:09:09.643387Z",
     "shell.execute_reply": "2026-07-31T11:09:09.642776Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Returns:   (251, 501)\n",
      "Backtest:  (239, 5)\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Load returns and backtest holdings\n",
    "==================================\n",
    "\"\"\"\n",
    "df_returns = pd.read_csv('../data/processed/returns_monthly.csv', index_col=0, parse_dates=True)\n",
    "df_backtest = pd.read_csv('../data/processed/backtest_returns.csv', index_col=0, parse_dates=True)\n",
    "\n",
    "print(f\"Returns:   {df_returns.shape}\")\n",
    "print(f\"Backtest:  {df_backtest.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "858ea555",
   "metadata": {},
   "source": [
    "## Sample Covariance and PCA\n",
    "\n",
    "Let $X_c\\in\\mathbb{R}^{T\\times N}$ be the centered return matrix. Each column has its time-series mean subtracted. The sample covariance matrix is\n",
    "\n",
    "$$\\Sigma=\\frac{1}{T-1}X_c^\\top X_c.$$\n",
    "\n",
    "PCA eigendecomposes this matrix:\n",
    "\n",
    "$$\\Sigma=V\\Lambda V^\\top,$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\\Lambda=\\mathrm{diag}(\\lambda_1,\\lambda_2,\\ldots,\\lambda_N), \\quad \\lambda_1\\ge\\lambda_2\\ge\\cdots.$$\n",
    "\n",
    "The columns of $V$ are principal component directions in stock space. The eigenvalues are the variance along those directions.\n",
    "\n",
    "`sklearn.decomposition.PCA` computes this through an SVD of $X_c$:\n",
    "\n",
    "$$X_c=USV^\\top,$$\n",
    "\n",
    "which implies\n",
    "\n",
    "$$\\Sigma=\\frac{1}{T-1}VS^2V^\\top.$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1600d228",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:09.645176Z",
     "iopub.status.busy": "2026-07-31T11:09:09.645004Z",
     "iopub.status.idle": "2026-07-31T11:09:09.675512Z",
     "shell.execute_reply": "2026-07-31T11:09:09.675049Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of assets: 394\n",
      "Number of observations: 240\n",
      "\n",
      "Top 10 eigenvalues:\n",
      "  PC1: 1.048239  (34.10%)\n",
      "  PC2: 0.112295  (3.65%)\n",
      "  PC3: 0.104251  (3.39%)\n",
      "  PC4: 0.092663  (3.01%)\n",
      "  PC5: 0.071925  (2.34%)\n",
      "  PC6: 0.057767  (1.88%)\n",
      "  PC7: 0.053078  (1.73%)\n",
      "  PC8: 0.047010  (1.53%)\n",
      "  PC9: 0.041681  (1.36%)\n",
      "  PC10: 0.040488  (1.32%)\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "PCA via sklearn\n",
    "==================================\n",
    "\"\"\"\n",
    "# Use a balanced panel: keep stocks present for most of the sample, then use\n",
    "# the months where all of them coexist. thresh must be high — a low thresh admits\n",
    "# short-history stocks, and the chained dropna() then truncates EVERY stock to that\n",
    "# short window (e.g. thresh=60 collapses the panel to 60 months, leaving N >> T and a\n",
    "# rank-deficient covariance). Maximizing observations matters most here.\n",
    "valid = df_returns.dropna(thresh=240, axis=1).dropna()\n",
    "X = valid.values\n",
    "X_centered = X - X.mean(axis=0)\n",
    "\n",
    "n_obs, n_assets = X_centered.shape\n",
    "\n",
    "# Sample covariance\n",
    "cov = np.cov(X_centered, rowvar=False, ddof=1)\n",
    "\n",
    "# sklearn PCA — fits on the centered data\n",
    "pca = PCA()\n",
    "pca.fit(X_centered)\n",
    "\n",
    "# Build the same result dict so downstream cells are unchanged\n",
    "pca_result = {\n",
    "    'eigenvalues': pca.explained_variance_,\n",
    "    'eigenvectors': pca.components_.T,   # sklearn returns (k x N); we want (N x k)\n",
    "    'explained_ratio': pca.explained_variance_ratio_,\n",
    "    'tickers': valid.columns.tolist(),\n",
    "    'n_obs': n_obs,\n",
    "    'n_assets': n_assets,\n",
    "    'cov': cov,\n",
    "}\n",
    "\n",
    "print(f\"Number of assets: {pca_result['n_assets']}\")\n",
    "print(f\"Number of observations: {pca_result['n_obs']}\")\n",
    "print(f\"\\nTop 10 eigenvalues:\")\n",
    "for i, ev in enumerate(pca_result['eigenvalues'][:10]):\n",
    "    print(f\"  PC{i+1}: {ev:.6f}  ({pca_result['explained_ratio'][i]*100:.2f}%)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "337d202b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:09.679231Z",
     "iopub.status.busy": "2026-07-31T11:09:09.679033Z",
     "iopub.status.idle": "2026-07-31T11:09:10.218423Z",
     "shell.execute_reply": "2026-07-31T11:09:10.217819Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 70864 characters]",
      "text/plain": [
       "<Figure size 1400x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PCs needed for 90% variance: 90\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Eigenvalue scree plot\n",
    "==================================\n",
    "\"\"\"\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "# Scree plot (first 50 PCs)\n",
    "ax = axes[0]\n",
    "n_show = 50\n",
    "ax.plot(range(1, n_show+1), pca_result['eigenvalues'][:n_show], 'o-', color='steelblue')\n",
    "ax.set_xlabel('Principal Component')\n",
    "ax.set_ylabel('Eigenvalue')\n",
    "ax.set_title('Eigenvalue Scree Plot (First 50 PCs)')\n",
    "ax.grid(alpha=0.3)\n",
    "\n",
    "# Cumulative explained variance\n",
    "ax = axes[1]\n",
    "cum_var = np.cumsum(pca_result['explained_ratio'])\n",
    "ax.plot(range(1, len(cum_var)+1), cum_var, color='coral')\n",
    "ax.axhline(y=0.9, color='black', linestyle='--', label='90% variance')\n",
    "ax.set_xlabel('Number of PCs')\n",
    "ax.set_ylabel('Cumulative Explained Variance')\n",
    "ax.set_title('Cumulative Explained Variance')\n",
    "ax.legend()\n",
    "ax.grid(alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/05_risk_decomposition/pca_scree.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "# How many PCs for 90% variance?\n",
    "n_90 = np.searchsorted(cum_var, 0.9) + 1\n",
    "print(f\"PCs needed for 90% variance: {n_90}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "183c2b9c",
   "metadata": {},
   "source": [
    "## Marchenko-Pastur Noise Separation\n",
    "\n",
    "Random matrix theory gives a useful benchmark for deciding how many principal components are larger than we would expect from noise.\n",
    "\n",
    "For a random matrix with iid entries and variance $\\sigma^2$, the eigenvalues of its sample covariance concentrate inside the Marchenko-Pastur interval:\n",
    "\n",
    "$$\\lambda_{\\pm}=\\sigma^2\\left(1+\\frac{1}{q}\\pm2\\sqrt{\\frac{1}{q}}\\right), \\quad q=\\frac{T}{N}.$$\n",
    "\n",
    "Eigenvalues inside $[\\lambda_-,\\lambda_+]$ are consistent with a no-factor random matrix. Eigenvalues above $\\lambda_+$ are candidates for real common factors.\n",
    "\n",
    "This is not magic; it is a rule of thumb with assumptions. Returns are not iid normal draws. But it is better than choosing $k$ by eye from a scree plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c67bfa0d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:10.220398Z",
     "iopub.status.busy": "2026-07-31T11:09:10.220105Z",
     "iopub.status.idle": "2026-07-31T11:09:10.721157Z",
     "shell.execute_reply": "2026-07-31T11:09:10.720503Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q = T/N = 0.6091\n",
      "sigma^2 (scale): 0.007802\n",
      "MP bounds: [0.000617, 0.040603]\n",
      "Number of signal eigenvalues (above MP upper bound): 9\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 52032 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Marchenko-Pastur cutoff\n",
    "==================================\n",
    "\"\"\"\n",
    "def marchenko_pastur_bounds(q, sigma_sq=1.0):\n",
    "    \"\"\"Compute MP upper and lower bounds.\"\"\"\n",
    "    inv_q = 1 / q\n",
    "    lambda_plus = sigma_sq * (1 + inv_q + 2 * np.sqrt(inv_q))\n",
    "    lambda_minus = sigma_sq * (1 + inv_q - 2 * np.sqrt(inv_q))\n",
    "    return lambda_minus, lambda_plus\n",
    "\n",
    "def mp_density(x, q, sigma_sq=1.0):\n",
    "    \"\"\"Marchenko-Pastur probability density.\"\"\"\n",
    "    lam_minus, lam_plus = marchenko_pastur_bounds(q, sigma_sq)\n",
    "    mask = (x > lam_minus) & (x < lam_plus)\n",
    "    density = np.zeros_like(x)\n",
    "    density[mask] = (q / (2 * np.pi * sigma_sq * x[mask])) * \\\n",
    "                    np.sqrt((lam_plus - x[mask]) * (x[mask] - lam_minus))\n",
    "    return density\n",
    "\n",
    "# Compute MP parameters\n",
    "T = pca_result['n_obs']\n",
    "N = pca_result['n_assets']\n",
    "q = T / N\n",
    "\n",
    "# Scale sigma^2 to match the bulk of the empirical spectrum\n",
    "# Estimate sigma^2 as the mean eigenvalue (equals average variance of stocks)\n",
    "# Under the null of pure noise, MP distribution mean = sigma^2\n",
    "# NOTE: this overestimates sigma^2 because signal eigenvalues pull the mean up,\n",
    "# making the MP cutoff conservative (fewer false-positive factors).  A more\n",
    "# rigorous estimate would fit sigma^2 from the noise-only portion of the spectrum\n",
    "# (e.g. the median of eigenvalues below a preliminary threshold).\n",
    "# sigma^2 = trace(Sigma) / N = sum of all eigenvalues / N\n",
    "# (sklearn returns only min(T,N) eigenvalues, so divide by N, not len(eigenvalues))\n",
    "sigma_sq = np.sum(pca_result['eigenvalues']) / pca_result['n_assets']\n",
    "\n",
    "lam_minus, lam_plus = marchenko_pastur_bounds(q, sigma_sq)\n",
    "n_signal = np.sum(pca_result['eigenvalues'] > lam_plus)\n",
    "\n",
    "print(f\"q = T/N = {q:.4f}\")\n",
    "print(f\"sigma^2 (scale): {sigma_sq:.6f}\")\n",
    "print(f\"MP bounds: [{lam_minus:.6f}, {lam_plus:.6f}]\")\n",
    "print(f\"Number of signal eigenvalues (above MP upper bound): {n_signal}\")\n",
    "\n",
    "# Plot empirical vs MP\n",
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "eigs = pca_result['eigenvalues']\n",
    "eigs_bulk = eigs[eigs < lam_plus * 5]  # Show bulk, exclude extreme top\n",
    "\n",
    "ax.hist(eigs_bulk, bins=80, density=True, color='steelblue', alpha=0.6, label='Empirical')\n",
    "\n",
    "x = np.linspace(max(0, lam_minus * 0.5), lam_plus * 3, 1000)\n",
    "ax.plot(x, mp_density(x, q, sigma_sq), color='coral', linewidth=2, label='Marchenko-Pastur')\n",
    "ax.axvline(x=lam_plus, color='black', linestyle='--', label=f'MP upper bound ({lam_plus:.4f})')\n",
    "\n",
    "ax.set_xlabel('Eigenvalue')\n",
    "ax.set_ylabel('Density')\n",
    "ax.set_title('Empirical Eigenvalue Spectrum vs. Marchenko-Pastur')\n",
    "ax.legend()\n",
    "ax.grid(alpha=0.3)\n",
    "ax.set_xlim(0, lam_plus * 3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/05_risk_decomposition/marchenko_pastur.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c6a3e3e",
   "metadata": {},
   "source": [
    "## Ledoit-Wolf Shrinkage\n",
    "\n",
    "The sample covariance matrix is noisy when the number of stocks is large relative to the number of months. Small estimation errors in covariance can create large errors in portfolio risk.\n",
    "\n",
    "Ledoit-Wolf shrinkage estimates\n",
    "\n",
    "$$\\hat\\Sigma=\\delta F+(1-\\delta)S,$$\n",
    "\n",
    "where $S$ is the sample covariance matrix, $F$ is a structured target, and $\\delta\\in[0,1]$ is the shrinkage intensity. The idea is to accept a little bias in exchange for lower estimation variance.\n",
    "\n",
    "In plain terms: do not trust every noisy sample covariance equally. Pull the estimate toward a simpler structure unless the data are strong enough to justify the complexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c96abc5d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:10.723075Z",
     "iopub.status.busy": "2026-07-31T11:09:10.722900Z",
     "iopub.status.idle": "2026-07-31T11:09:10.881731Z",
     "shell.execute_reply": "2026-07-31T11:09:10.881112Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shrinkage intensity (delta): 0.4996\n",
      "Average correlation (r_bar): 0.3165\n",
      "\n",
      "A shrinkage of 0.50 means the estimator is 50% target and 50% sample.\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Ledoit-Wolf shrinkage estimator\n",
    "==================================\n",
    "\"\"\"\n",
    "def ledoit_wolf_shrinkage(returns_df, min_obs=60):\n",
    "    \"\"\"\n",
    "    Compute Ledoit-Wolf shrinkage covariance estimator.\n",
    "    Target: constant correlation matrix.\n",
    "    \"\"\"\n",
    "    valid = returns_df.dropna(thresh=min_obs, axis=1).dropna()\n",
    "    X = valid.values\n",
    "    T, N = X.shape\n",
    "    \n",
    "    # Sample covariance\n",
    "    S = np.cov(X, rowvar=False, ddof=1)\n",
    "    \n",
    "    # Target: constant correlation matrix\n",
    "    # Compute sample correlations, then average off-diagonal\n",
    "    D = np.sqrt(np.diag(S))\n",
    "    R = S / np.outer(D, D)\n",
    "    \n",
    "    # Average correlation (excluding diagonal)\n",
    "    mask = ~np.eye(N, dtype=bool)\n",
    "    r_bar = R[mask].mean()\n",
    "    \n",
    "    # Constant correlation target\n",
    "    F = r_bar * np.ones((N, N))\n",
    "    np.fill_diagonal(F, 1.0)\n",
    "    F = F * np.outer(D, D)  # Scale back to covariance\n",
    "    \n",
    "    # Shrinkage intensity (Ledoit-Wolf formula)\n",
    "    # mu = sum of squared deviations of sample cov from target\n",
    "    # d^2 = average squared deviation\n",
    "    X_centered = X - X.mean(axis=0)\n",
    "    \n",
    "    # Compute d^2 (distance between S and F)\n",
    "    d_sq = np.sum((S - F)**2) / N\n",
    "    \n",
    "    # Compute r_bar^2 (variance of sample covariance entries)\n",
    "    # Using the Ledoit-Wolf asymptotic formula\n",
    "    pi_mat = np.zeros((N, N))\n",
    "    for t in range(T):\n",
    "        xt = X_centered[t, :].reshape(-1, 1)\n",
    "        pi_mat += (xt @ xt.T - S)**2\n",
    "    pi_mat /= T**2\n",
    "    pi_hat = pi_mat.sum() / N\n",
    "    \n",
    "    # Shrinkage intensity\n",
    "    beta_sq = 0  # For constant correlation target, this term is smaller order\n",
    "    delta = max(0, min(1, pi_hat / d_sq))\n",
    "    \n",
    "    # Shrunk estimator\n",
    "    Sigma_shrunk = delta * F + (1 - delta) * S\n",
    "    \n",
    "    return {\n",
    "        'cov_shrunk': Sigma_shrunk,\n",
    "        'cov_sample': S,\n",
    "        'target': F,\n",
    "        'shrinkage_intensity': delta,\n",
    "        'r_bar': r_bar,\n",
    "        'tickers': valid.columns.tolist()\n",
    "    }\n",
    "\n",
    "lw_result = ledoit_wolf_shrinkage(df_returns, min_obs=240)\n",
    "\n",
    "print(f\"Shrinkage intensity (delta): {lw_result['shrinkage_intensity']:.4f}\")\n",
    "print(f\"Average correlation (r_bar): {lw_result['r_bar']:.4f}\")\n",
    "print(f\"\\nA shrinkage of {lw_result['shrinkage_intensity']:.2f} means the estimator is \"\n",
    "      f\"{lw_result['shrinkage_intensity']*100:.0f}% target and \"\n",
    "      f\"{(1-lw_result['shrinkage_intensity'])*100:.0f}% sample.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6dc7a3cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:10.884080Z",
     "iopub.status.busy": "2026-07-31T11:09:10.883885Z",
     "iopub.status.idle": "2026-07-31T11:09:13.088848Z",
     "shell.execute_reply": "2026-07-31T11:09:13.088123Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of evaluation windows: 32\n",
      "Mean Frobenius error (sample): 3.0468\n",
      "Mean Frobenius error (shrunk): 2.9003\n",
      "Improvement: 4.8%\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 82488 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Out-of-sample comparison: sample vs. shrunk covariance\n",
    "==================================\n",
    "\n",
    "We use a rolling window: estimate covariance on trailing 60 months,\n",
    "compute portfolio variance forecast, compare to realized variance\n",
    "over the next month. Lower Frobenius error = better estimator.\n",
    "\"\"\"\n",
    "window = 60\n",
    "dates = df_returns.index\n",
    "errors_sample = []\n",
    "errors_shrunk = []\n",
    "\n",
    "for i in range(window, len(dates) - 1, 6):  # Every 6 months for speed\n",
    "    train = df_returns.iloc[i-window:i].dropna(axis=1, thresh=window//2).dropna()\n",
    "    test = df_returns.iloc[i:i+1][train.columns]\n",
    "    \n",
    "    if test.shape[1] < 20:\n",
    "        continue\n",
    "    \n",
    "    # Shrunk covariance (recompute on this window)\n",
    "    lw = ledoit_wolf_shrinkage(df_returns.iloc[i-window:i])\n",
    "    S_shrunk = lw['cov_shrunk']\n",
    "    \n",
    "    # Find common tickers between train and the shrunk estimator\n",
    "    common = train.columns.intersection(lw['tickers'])\n",
    "    if len(common) < 20:\n",
    "        continue\n",
    "    \n",
    "    # Sample covariance on common tickers only\n",
    "    train_common = train[common]\n",
    "    S_sample = np.cov(train_common, rowvar=False, ddof=1)\n",
    "    \n",
    "    # Shrunk covariance aligned to common tickers\n",
    "    S_shrunk_aligned = pd.DataFrame(\n",
    "        S_shrunk, index=lw['tickers'], columns=lw['tickers']\n",
    "    ).loc[common, common].values\n",
    "    \n",
    "    # Realized covariance (outer product of next month's returns) on common tickers\n",
    "    test_common = test[common]\n",
    "    realized = np.outer(test_common.values[0], test_common.values[0])\n",
    "    \n",
    "    err_sample = np.linalg.norm(S_sample - realized, 'fro')\n",
    "    err_shrunk = np.linalg.norm(S_shrunk_aligned - realized, 'fro')\n",
    "    \n",
    "    errors_sample.append(err_sample)\n",
    "    errors_shrunk.append(err_shrunk)\n",
    "\n",
    "print(f\"Number of evaluation windows: {len(errors_sample)}\")\n",
    "print(f\"Mean Frobenius error (sample): {np.mean(errors_sample):.4f}\")\n",
    "print(f\"Mean Frobenius error (shrunk): {np.mean(errors_shrunk):.4f}\")\n",
    "print(f\"Improvement: {(1 - np.mean(errors_shrunk)/np.mean(errors_sample))*100:.1f}%\")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 5))\n",
    "ax.plot(errors_sample, label='Sample Covariance', color='steelblue')\n",
    "ax.plot(errors_shrunk, label='Ledoit-Wolf Shrunk', color='coral')\n",
    "ax.set_xlabel('Evaluation Window')\n",
    "ax.set_ylabel('Frobenius Error')\n",
    "ax.set_title('Out-of-Sample Covariance Estimation Error')\n",
    "ax.legend()\n",
    "ax.grid(alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/05_risk_decomposition/ledoit_wolf_comparison.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99d9a4fd",
   "metadata": {},
   "source": [
    "## Portfolio Variance Decomposition\n",
    "\n",
    "Let $B\\in\\mathbb{R}^{N\\times k}$ hold the top $k$ eigenvectors and let\n",
    "\n",
    "$$\\Sigma_f=\\mathrm{diag}(\\lambda_1,\\ldots,\\lambda_k).$$\n",
    "\n",
    "The top-$k$ approximation to the covariance matrix is\n",
    "\n",
    "$$\\Sigma_{\\text{factor}}=B\\Sigma_fB^\\top.$$\n",
    "\n",
    "The residual covariance is\n",
    "\n",
    "$$\\Sigma_{\\text{resid}}=\\Sigma-B\\Sigma_fB^\\top.$$\n",
    "\n",
    "For a portfolio $w$, variance splits exactly under this construction:\n",
    "\n",
    "$$w^\\top\\Sigma w = w^\\top\\Sigma_{\\text{factor}}w + w^\\top\\Sigma_{\\text{resid}}w.$$\n",
    "\n",
    "Geometrically, $B^\\top w$ is the portfolio's exposure to the top PCA directions. The systematic term is the variance from those common directions. The residual term is everything left outside that retained subspace.\n",
    "\n",
    "This is a PCA decomposition, not a full economic factor model. The first few PCs often resemble broad market or style factors, but the math itself only knows about covariance structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "226662f6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:13.090599Z",
     "iopub.status.busy": "2026-07-31T11:09:13.090421Z",
     "iopub.status.idle": "2026-07-31T11:09:13.125912Z",
     "shell.execute_reply": "2026-07-31T11:09:13.125261Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last rebalance: 2025-11-30 00:00:00\n",
      "Number of long holdings: 49\n",
      "Weight per stock: 0.0204\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Reconstruct the long portfolio weights at the last rebalance\n",
    "==================================\n",
    "\"\"\"\n",
    "df_signal = pd.read_csv('../data/processed/momentum_signal.csv', index_col=0, parse_dates=True)\n",
    "\n",
    "# Take the last rebalance date\n",
    "last_date = df_signal.index[-2]  # -2 because we trade t+1\n",
    "scores = df_signal.loc[last_date].dropna()\n",
    "\n",
    "n_long = max(int(len(scores) * 0.1), 1)\n",
    "long_tickers = scores.sort_values(ascending=False).head(n_long).index.tolist()\n",
    "\n",
    "# Equal-weighted weights\n",
    "w = pd.Series(1.0 / len(long_tickers), index=long_tickers)\n",
    "\n",
    "print(f\"Last rebalance: {last_date}\")\n",
    "print(f\"Number of long holdings: {len(long_tickers)}\")\n",
    "print(f\"Weight per stock: {w.iloc[0]:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7351ba96",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:13.127569Z",
     "iopub.status.busy": "2026-07-31T11:09:13.127371Z",
     "iopub.status.idle": "2026-07-31T11:09:13.339158Z",
     "shell.execute_reply": "2026-07-31T11:09:13.338629Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using top 9 PCs as statistical factors\n",
      "\n",
      "### Portfolio Variance Decomposition\n",
      "\n",
      "Total portfolio variance:      0.001336\n",
      "Systematic (factor) variance:  0.001212  (90.7%)\n",
      "Idiosyncratic variance:        0.000124  (9.3%)\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 31792 characters]",
      "text/plain": [
       "<Figure size 800x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Variance decomposition using top-k PCs as factors\n",
    "==================================\n",
    "\"\"\"\n",
    "# Align weights to PCA tickers\n",
    "pca_tickers = pca_result['tickers']\n",
    "w_aligned = w.reindex(pca_tickers).fillna(0).values\n",
    "\n",
    "cov = pca_result['cov']\n",
    "eigenvalues = pca_result['eigenvalues']\n",
    "eigenvectors = pca_result['eigenvectors']\n",
    "\n",
    "k = n_signal  # Number of signal PCs\n",
    "print(f\"Using top {k} PCs as statistical factors\")\n",
    "\n",
    "# Factor loadings: B = eigenvectors (N x k)\n",
    "B = eigenvectors[:, :k]\n",
    "\n",
    "# Factor covariance: diagonal matrix of top-k eigenvalues\n",
    "Sigma_f = np.diag(eigenvalues[:k])\n",
    "\n",
    "# Idiosyncratic variance: residual variance per stock\n",
    "# D = diag(Sigma - B Sigma_f B^T)\n",
    "Sigma_factor = B @ Sigma_f @ B.T\n",
    "\n",
    "# Portfolio variance\n",
    "total_var = w_aligned @ cov @ w_aligned\n",
    "systematic_var = w_aligned @ Sigma_factor @ w_aligned\n",
    "# Idiosyncratic = variance in the orthogonal complement of the top-k factor\n",
    "# subspace. Equals total - systematic by construction, so the split sums to 100%.\n",
    "idiosyncratic_var = total_var - systematic_var\n",
    "\n",
    "print(f\"\\n### Portfolio Variance Decomposition\\n\")\n",
    "print(f\"Total portfolio variance:      {total_var:.6f}\")\n",
    "print(f\"Systematic (factor) variance:  {systematic_var:.6f}  ({systematic_var/total_var*100:.1f}%)\")\n",
    "print(f\"Idiosyncratic variance:        {idiosyncratic_var:.6f}  ({idiosyncratic_var/total_var*100:.1f}%)\")\n",
    "\n",
    "# Plot\n",
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "ax.bar(['Systematic', 'Idiosyncratic'], [systematic_var, idiosyncratic_var], \n",
    "       color=['steelblue', 'coral'])\n",
    "ax.set_ylabel('Variance Contribution')\n",
    "ax.set_title(f'Portfolio Variance Decomposition (Top {k} PCs)')\n",
    "ax.grid(alpha=0.3, axis='y')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/05_risk_decomposition/variance_decomposition.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3b1f2340",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:13.340816Z",
     "iopub.status.busy": "2026-07-31T11:09:13.340640Z",
     "iopub.status.idle": "2026-07-31T11:09:13.685130Z",
     "shell.execute_reply": "2026-07-31T11:09:13.684539Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44328 characters]",
      "text/plain": [
       "<Figure size 800x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mkt-RF</th>\n",
       "      <th>SMB</th>\n",
       "      <th>HML</th>\n",
       "      <th>Mom</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>PC1</th>\n",
       "      <td>0.957</td>\n",
       "      <td>0.407</td>\n",
       "      <td>0.292</td>\n",
       "      <td>-0.493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PC2</th>\n",
       "      <td>-0.155</td>\n",
       "      <td>-0.022</td>\n",
       "      <td>0.646</td>\n",
       "      <td>-0.366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PC3</th>\n",
       "      <td>0.031</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.085</td>\n",
       "      <td>0.004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PC4</th>\n",
       "      <td>-0.059</td>\n",
       "      <td>0.221</td>\n",
       "      <td>-0.134</td>\n",
       "      <td>-0.288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PC5</th>\n",
       "      <td>-0.052</td>\n",
       "      <td>-0.012</td>\n",
       "      <td>-0.286</td>\n",
       "      <td>-0.048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Mkt-RF    SMB    HML    Mom\n",
       "PC1   0.957  0.407  0.292 -0.493\n",
       "PC2  -0.155 -0.022  0.646 -0.366\n",
       "PC3   0.031  0.104  0.085  0.004\n",
       "PC4  -0.059  0.221 -0.134 -0.288\n",
       "PC5  -0.052 -0.012 -0.286 -0.048"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Compare statistical factors to known FF factors\n",
    "==================================\n",
    "\"\"\"\n",
    "df_ff = pd.read_csv('../data/raw/ff_factors.csv', index_col=0, parse_dates=True)\n",
    "\n",
    "# Project top PCs onto FF factor returns\n",
    "# Align dates: df_returns has month-end, df_ff has month-start\n",
    "# Convert to month-end timestamps, then normalize midnight for exact matching\n",
    "common_dates = df_returns.index.to_period('M').intersection(\n",
    "    df_ff.index.to_period('M')\n",
    ").to_timestamp(how='end').normalize()\n",
    "df_ret_aligned = df_returns.loc[common_dates, pca_tickers].dropna()\n",
    "df_ff_aligned = df_ff.copy()\n",
    "df_ff_aligned.index = df_ff_aligned.index.to_period('M').to_timestamp(how='end').normalize()\n",
    "df_ff_aligned = df_ff_aligned.loc[df_ret_aligned.index]\n",
    "\n",
    "# Compute PC returns: PC_i = eigenvector_i^T @ returns\n",
    "pc_returns = pd.DataFrame(\n",
    "    df_ret_aligned.values @ eigenvectors[:, :5],\n",
    "    index=df_ret_aligned.index,\n",
    "    columns=[f'PC{i+1}' for i in range(5)]\n",
    ")\n",
    "\n",
    "# Correlate PCs with FF factors\n",
    "factor_cols = ['Mkt-RF', 'SMB', 'HML', 'Mom']\n",
    "df_corr = pd.DataFrame(index=[f'PC{i+1}' for i in range(5)], columns=factor_cols)\n",
    "\n",
    "for pc in pc_returns.columns:\n",
    "    for ff in factor_cols:\n",
    "        mask = pc_returns[pc].notna() & df_ff_aligned[ff].notna()\n",
    "        df_corr.loc[pc, ff] = pc_returns.loc[mask, pc].corr(df_ff_aligned.loc[mask, ff])\n",
    "\n",
    "df_corr = df_corr.astype(float)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "sns.heatmap(df_corr, annot=True, fmt='.2f', cmap='RdBu_r', center=0, ax=ax, square=True)\n",
    "ax.set_title('Correlation: Statistical PCs vs. Fama-French Factors')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/05_risk_decomposition/pc_vs_ff_correlation.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "display(df_corr.round(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2566bdea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:09:13.686923Z",
     "iopub.status.busy": "2026-07-31T11:09:13.686718Z",
     "iopub.status.idle": "2026-07-31T11:09:13.693767Z",
     "shell.execute_reply": "2026-07-31T11:09:13.693164Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved:\n",
      "  - variance_decomposition.csv\n",
      "  - pca_spectrum.csv\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Save risk decomposition results\n",
    "==================================\n",
    "\"\"\"\n",
    "decomp_results = pd.DataFrame({\n",
    "    'component': ['total', 'systematic', 'idiosyncratic'],\n",
    "    'variance': [total_var, systematic_var, idiosyncratic_var],\n",
    "    'pct': [100, systematic_var/total_var*100, idiosyncratic_var/total_var*100]\n",
    "})\n",
    "decomp_results.to_csv('../data/processed/variance_decomposition.csv', index=False)\n",
    "\n",
    "pd.DataFrame({\n",
    "    'eigenvalue': eigenvalues[:50],\n",
    "    'explained_pct': pca_result['explained_ratio'][:50] * 100\n",
    "}).to_csv('../data/processed/pca_spectrum.csv', index=False)\n",
    "\n",
    "print(\"Saved:\")\n",
    "print(\"  - variance_decomposition.csv\")\n",
    "print(\"  - pca_spectrum.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edf65c16",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook decomposes the risk of the long-only momentum portfolio.\n",
    "\n",
    "- The Marchenko-Pastur cutoff gives an objective estimate of how many covariance eigenvalues are too large to dismiss as random-matrix noise.\n",
    "- Ledoit-Wolf shrinkage shows why the raw sample covariance should be treated cautiously when $N$ is large relative to $T$.\n",
    "- The portfolio variance decomposition shows that most of the long-only portfolio's risk lives in common PCA directions rather than purely stock-specific residual risk.\n",
    "\n",
    "One wording caveat: the Fama-French alpha from notebook 04 is orthogonal to the Fama-French benchmark factors, not literally to these PCA directions. The two ideas are related because both are factor decompositions, but they are not the same basis.\n",
    "\n",
    "Notebook 06 uses the PCA factor structure to generate alternate market histories and ask whether the observed alpha looks unusually lucky."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
