{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f7ef2de3",
   "metadata": {},
   "source": [
    "# Data Overview and Market Statistics\n",
    "\n",
    "This notebook builds the data object used by the rest of the project: the **monthly return matrix**\n",
    "\n",
    "$$R \\in \\mathbb{R}^{T \\times N},$$\n",
    "\n",
    "where rows are months and columns are stocks. Most of the later notebooks are different ways of asking questions about this matrix: which column-wise signals predict the next row, how a portfolio weight vector interacts with a row of returns, and how the covariance matrix\n",
    "\n",
    "$$\\Sigma = \\frac{1}{T-1}X_c^\\top X_c$$\n",
    "\n",
    "splits into common and stock-specific risk.\n",
    "\n",
    "The goals are:\n",
    "1. Build a current-constituent S&P 500 universe. This is convenient, but survivorship-biased.\n",
    "2. Assemble monthly adjusted-price returns from cached/free data.\n",
    "3. Describe breadth, cross-sectional dispersion, sector composition, and equal-weight market behavior.\n",
    "4. Save clean CSV artifacts for the later notebooks.\n",
    "\n",
    "### Finance terms used in this notebook\n",
    "\n",
    "| Term | Meaning |\n",
    "|------|---------|\n",
    "| **Return** | Fractional price change: $r_t = p_t / p_{t-1} - 1$ |\n",
    "| **Return panel** $R$ | The $T \\times N$ matrix of monthly returns: months $\\times$ stocks |\n",
    "| **Cross-section** | One row of $R$: all stock returns at one date |\n",
    "| **One stock's history** | One column of $R$ |\n",
    "| **Universe** | The stocks we allow ourselves to analyze or hold |\n",
    "| **Sector** | A categorical grouping such as Technology, Financials, or Healthcare |\n",
    "| **Dispersion** | Cross-sectional standard deviation: how spread out stock returns are in one month |\n",
    "| **Equal-weight index** | Mean row return $\\bar r_t = \\tfrac{1}{N_t}\\mathbf{1}^\\top r_t$ |\n",
    "| **Sharpe ratio** | Mean return divided by volatility; a rough signal-to-noise ratio |\n",
    "| **Survivorship bias** | Bias from omitting firms that disappeared, merged, were delisted, or left the index |\n",
    "\n",
    "A practical note: because constituents enter, leave, and have missing history, $R$ is sparse. Missing entries are not zeros; they mean the stock was not available in our panel at that date.\n",
    "\n",
    "## Outputs\n",
    "\n",
    "This notebook writes:\n",
    "- `data/processed/returns_monthly.csv`\n",
    "- `data/processed/prices_monthly.csv`\n",
    "- `data/processed/sector_mapping.csv`\n",
    "\n",
    "Those files are the handoff to the factor, backtest, PCA, and synthetic-market notebooks.\n",
    "\n",
    "## Notebook Structure\n",
    "1. [Setup and Imports](#setup-and-imports)\n",
    "2. [Universe Construction](#universe-construction)\n",
    "3. [Return Panel Assembly](#return-panel-assembly)\n",
    "4. [Market Statistics](#market-statistics)\n",
    "5. [Sector Composition](#sector-composition)\n",
    "6. [Conclusion](#conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d02502aa",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## Setup and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "eba6fb3d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:28.595596Z",
     "iopub.status.busy": "2026-07-31T11:08:28.594495Z",
     "iopub.status.idle": "2026-07-31T11:08:29.655509Z",
     "shell.execute_reply": "2026-07-31T11:08:29.654776Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Setup and imports\n",
    "==================================\n",
    "\"\"\"\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import os\n",
    "from datetime import datetime\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', 200)\n",
    "\n",
    "palette = ['steelblue', 'coral', 'seagreen']\n",
    "\n",
    "os.makedirs('../data/raw', exist_ok=True)\n",
    "os.makedirs('../data/processed', exist_ok=True)\n",
    "os.makedirs('../images/01_market_stats', exist_ok=True)\n",
    "\n",
    "START_DATE = '2005-01-01'\n",
    "END_DATE = '2025-12-31'\n",
    "RANDOM_STATE = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46cd7224",
   "metadata": {},
   "source": [
    "## Universe Construction\n",
    "\n",
    "We start from the current S&P 500 constituent list and pull historical prices for those tickers. This is simple and reproducible, but it is **not** a fully historical S&P 500 membership file.\n",
    "\n",
    "The main limitation is survivorship bias. Companies that were removed from the index, acquired, delisted, or bankrupt between 2005 and the present may not appear as columns. That can push historical returns upward because the missing names are not a random sample of the market. We cannot fully solve that without a survivorship-free database such as CRSP, so this notebook keeps the caveat visible and later notebooks test how much return drag would be needed to erase the alpha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f99f0d78",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:29.657559Z",
     "iopub.status.busy": "2026-07-31T11:08:29.657294Z",
     "iopub.status.idle": "2026-07-31T11:08:29.758892Z",
     "shell.execute_reply": "2026-07-31T11:08:29.758270Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cached constituent snapshot from ../data/raw/constituents.csv\n",
      "Total constituents: 503\n",
      "First 10: ['MMM', 'AOS', 'ABT', 'ABBV', 'ACN', 'ADBE', 'AMD', 'AES', 'AFL', 'A']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Security</th>\n",
       "      <th>GICS Sector</th>\n",
       "      <th>GICS Sub-Industry</th>\n",
       "      <th>Headquarters Location</th>\n",
       "      <th>Date added</th>\n",
       "      <th>CIK</th>\n",
       "      <th>Founded</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MMM</td>\n",
       "      <td>3M</td>\n",
       "      <td>Industrials</td>\n",
       "      <td>Industrial Conglomerates</td>\n",
       "      <td>Saint Paul, Minnesota</td>\n",
       "      <td>1957-03-04</td>\n",
       "      <td>66740</td>\n",
       "      <td>1902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AOS</td>\n",
       "      <td>A. O. Smith</td>\n",
       "      <td>Industrials</td>\n",
       "      <td>Building Products</td>\n",
       "      <td>Milwaukee, Wisconsin</td>\n",
       "      <td>2017-07-26</td>\n",
       "      <td>91142</td>\n",
       "      <td>1916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ABT</td>\n",
       "      <td>Abbott Laboratories</td>\n",
       "      <td>Health Care</td>\n",
       "      <td>Health Care Equipment</td>\n",
       "      <td>North Chicago, Illinois</td>\n",
       "      <td>1957-03-04</td>\n",
       "      <td>1800</td>\n",
       "      <td>1888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ABBV</td>\n",
       "      <td>AbbVie</td>\n",
       "      <td>Health Care</td>\n",
       "      <td>Biotechnology</td>\n",
       "      <td>North Chicago, Illinois</td>\n",
       "      <td>2012-12-31</td>\n",
       "      <td>1551152</td>\n",
       "      <td>2013 (1888)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ACN</td>\n",
       "      <td>Accenture</td>\n",
       "      <td>Information Technology</td>\n",
       "      <td>IT Consulting &amp; Other Services</td>\n",
       "      <td>Dublin, Ireland</td>\n",
       "      <td>2011-07-06</td>\n",
       "      <td>1467373</td>\n",
       "      <td>1989</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Symbol             Security             GICS Sector  \\\n",
       "0    MMM                   3M             Industrials   \n",
       "1    AOS          A. O. Smith             Industrials   \n",
       "2    ABT  Abbott Laboratories             Health Care   \n",
       "3   ABBV               AbbVie             Health Care   \n",
       "4    ACN            Accenture  Information Technology   \n",
       "\n",
       "                GICS Sub-Industry    Headquarters Location  Date added  \\\n",
       "0        Industrial Conglomerates    Saint Paul, Minnesota  1957-03-04   \n",
       "1               Building Products     Milwaukee, Wisconsin  2017-07-26   \n",
       "2           Health Care Equipment  North Chicago, Illinois  1957-03-04   \n",
       "3                   Biotechnology  North Chicago, Illinois  2012-12-31   \n",
       "4  IT Consulting & Other Services          Dublin, Ireland  2011-07-06   \n",
       "\n",
       "       CIK      Founded  \n",
       "0    66740         1902  \n",
       "1    91142         1916  \n",
       "2     1800         1888  \n",
       "3  1551152  2013 (1888)  \n",
       "4  1467373         1989  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Pull S&P 500 constituents from Wikipedia\n",
    "==================================\n",
    "\"\"\"\n",
    "\n",
    "import requests\n",
    "from io import StringIO\n",
    "\n",
    "constituents_path = '../data/raw/constituents.csv'\n",
    "refresh_constituents = os.environ.get('REFRESH_DATA', '').lower() in {'1', 'true', 'yes'}\n",
    "\n",
    "if os.path.exists(constituents_path) and not refresh_constituents:\n",
    "    print(f\"Loading cached constituent snapshot from {constituents_path}\")\n",
    "    df_constituents = pd.read_csv(constituents_path, index_col=0)\n",
    "else:\n",
    "    url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'\n",
    "\n",
    "    # Wikipedia requires a descriptive User-Agent per their API policy\n",
    "    headers = {\n",
    "        'User-Agent': 'EquityFactorResearch/1.0 (quant research; contact@example.com)'\n",
    "    }\n",
    "\n",
    "    resp = requests.get(url, headers=headers, timeout=30)\n",
    "    resp.raise_for_status()\n",
    "\n",
    "    # match='Symbol' targets the constituents table directly\n",
    "    tables = pd.read_html(StringIO(resp.text), match='Symbol')\n",
    "    df_constituents = tables[0]\n",
    "    df_constituents.to_csv(constituents_path)\n",
    "    print(f\"Saved constituent snapshot to {constituents_path}\")\n",
    "\n",
    "df_constituents['Symbol'] = df_constituents['Symbol'].str.replace('.', '-', regex=False)\n",
    "tickers = df_constituents['Symbol'].tolist()\n",
    "\n",
    "print(f\"Total constituents: {len(tickers)}\")\n",
    "print(f\"First 10: {tickers[:10]}\")\n",
    "df_constituents.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dee072b6",
   "metadata": {},
   "source": [
    "A note on the CIK: CIK stands for **Central Index Key**. It's a unique identifier assigned by the U.S. Securities and Exchange Commission (SEC) to every entity (company, individual, or organization) that files disclosure documents with them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "12116b17",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:29.760587Z",
     "iopub.status.busy": "2026-07-31T11:08:29.760419Z",
     "iopub.status.idle": "2026-07-31T11:08:30.245820Z",
     "shell.execute_reply": "2026-07-31T11:08:30.245253Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cached prices from ../data/raw/prices_monthly.csv\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Price panel shape: (5282, 503)\n",
      "Date range: 2005-01-03 00:00:00 to 2025-12-30 00:00:00\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Download price history via yfinance\n",
    "==================================\n",
    "\n",
    "We use adjusted close to account for splits\n",
    "and dividends, then resample to monthly returns.\n",
    "\"\"\"\n",
    "from tqdm import tqdm # progress bar\n",
    "\n",
    "raw_path = '../data/raw/prices_monthly.csv'\n",
    "\n",
    "if os.path.exists(raw_path):\n",
    "    print(f\"Loading cached prices from {raw_path}\")\n",
    "    df_prices = pd.read_csv(raw_path, index_col=0, parse_dates=True)\n",
    "else:\n",
    "    import yfinance as yf\n",
    "\n",
    "    print(f\"Downloading price data for {len(tickers)} tickers ...\")\n",
    "    df_raw = yf.download(\n",
    "        tickers,\n",
    "        start=START_DATE,\n",
    "        end=END_DATE,\n",
    "        progress=True,\n",
    "        auto_adjust=True,\n",
    "        threads=True\n",
    "    )\n",
    "\n",
    "    # Use adjusted close\n",
    "    df_prices = df_raw['Close']\n",
    "    df_prices.to_csv(raw_path)\n",
    "    print(f\"Saved to {raw_path}\")\n",
    "\n",
    "print(f\"\\nPrice panel shape: {df_prices.shape}\")\n",
    "print(f\"Date range: {df_prices.index.min()} to {df_prices.index.max()}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44c3466e",
   "metadata": {},
   "source": [
    "## Return Panel Assembly\n",
    "\n",
    "We resample adjusted daily prices to month-end and compute simple monthly returns:\n",
    "\n",
    "$$r_{t,i} = \\frac{p_{t,i}}{p_{t-1,i}} - 1.$$\n",
    "\n",
    "Here $t$ indexes months, $i$ indexes stocks, and $p_{t,i}$ is the adjusted close price at month-end. The ratio $p_{t,i}/p_{t-1,i}$ tells us what $1 invested at the previous month-end became by this month-end; subtracting 1 converts that gross return into a net return.\n",
    "\n",
    "The result is the return matrix $R \\in \\mathbb{R}^{T \\times N}$ with months as rows and stocks as columns.\n",
    "\n",
    "We use monthly returns because the rest of the project is about medium-horizon momentum, not high-frequency trading. Monthly rebalancing also keeps turnover and transaction-cost assumptions easier to reason about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "249ffb9c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:30.247588Z",
     "iopub.status.busy": "2026-07-31T11:08:30.247411Z",
     "iopub.status.idle": "2026-07-31T11:08:30.293016Z",
     "shell.execute_reply": "2026-07-31T11:08:30.292555Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Montly return panel shape: (251, 501)\n",
      "Date range: 2005-02-28 00:00:00 to 2025-12-31 00:00:00\n",
      "Unique tickers: 501\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_144016/1360406685.py:8: FutureWarning: The default fill_method='pad' in DataFrame.pct_change is deprecated and will be removed in a future version. Either fill in any non-leading NA values prior to calling pct_change or specify 'fill_method=None' to not fill NA values.\n",
      "  df_returns = df_monthly_prices.pct_change() # fractional change between current and prior\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>AAPL</th>\n",
       "      <th>ABBV</th>\n",
       "      <th>ABNB</th>\n",
       "      <th>ABT</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2005-02-28</th>\n",
       "      <td>0.085482</td>\n",
       "      <td>0.166711</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.021546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005-03-31</th>\n",
       "      <td>-0.075000</td>\n",
       "      <td>-0.071111</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.013699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005-04-30</th>\n",
       "      <td>-0.065315</td>\n",
       "      <td>-0.134629</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.060572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005-05-31</th>\n",
       "      <td>0.157109</td>\n",
       "      <td>0.102606</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.018714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005-06-30</th>\n",
       "      <td>-0.041233</td>\n",
       "      <td>-0.074195</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.015962</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A      AAPL  ABBV  ABNB       ABT\n",
       "Date                                                \n",
       "2005-02-28  0.085482  0.166711   NaN   NaN  0.021546\n",
       "2005-03-31 -0.075000 -0.071111   NaN   NaN  0.013699\n",
       "2005-04-30 -0.065315 -0.134629   NaN   NaN  0.060572\n",
       "2005-05-31  0.157109  0.102606   NaN   NaN -0.018714\n",
       "2005-06-30 -0.041233 -0.074195   NaN   NaN  0.015962"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Build monthly return panel\n",
    "==================================\n",
    "\"\"\"\n",
    "# Resample to month-end and compute returns\n",
    "df_monthly_prices = df_prices.resample('ME').last()\n",
    "df_returns = df_monthly_prices.pct_change() # fractional change between current and prior\n",
    "\n",
    "# Drop the first row (NaN from pct_change)\n",
    "df_returns = df_returns.iloc[1:]\n",
    "\n",
    "# Drop columns that are entirely NaN (delisted / no data)\n",
    "df_returns = df_returns.dropna(axis=1, how='all')\n",
    "\n",
    "print(f\"Montly return panel shape: {df_returns.shape}\")\n",
    "print(f\"Date range: {df_returns.index.min()} to {df_returns.index.max()}\")\n",
    "print(f\"Unique tickers: {df_returns.shape[1]}\")\n",
    "\n",
    "df_returns.iloc[:5,:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9691c46b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:30.294692Z",
     "iopub.status.busy": "2026-07-31T11:08:30.294517Z",
     "iopub.status.idle": "2026-07-31T11:08:30.617121Z",
     "shell.execute_reply": "2026-07-31T11:08:30.616498Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved:\n",
      "  - ../data/processed/returns_monthly.csv\n",
      "  - ../data/processed/prices_monthly.csv\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Save return panel\n",
    "==================================\n",
    "\"\"\"\n",
    "df_returns.to_csv('../data/processed/returns_monthly.csv')\n",
    "df_monthly_prices.to_csv('../data/processed/prices_monthly.csv')\n",
    "\n",
    "print(\"Saved:\")\n",
    "print(\"  - ../data/processed/returns_monthly.csv\")\n",
    "print(\"  - ../data/processed/prices_monthly.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2260e830",
   "metadata": {},
   "source": [
    "## Market Statistics\n",
    "\n",
    "Before building factors, we want a baseline feel for the panel.\n",
    "\n",
    "- **Breadth:** how many stocks have valid returns each month? This is not true historical S&P 500 membership; it is the number of current-constituent tickers with usable data at that date.\n",
    "- **Cross-sectional dispersion:** how spread out stock returns are in a given month:\n",
    "\n",
    "$$\\sigma_t^{XS} = \\sqrt{\\frac{1}{N_t - 1} \\sum_{i=1}^{N_t}(r_{t,i} - \\bar{r}_t)^2}.$$\n",
    "\n",
    "- **Equal-weight market return:** the return from holding every available stock at equal weight:\n",
    "\n",
    "$$\\bar{r}_t = \\frac{1}{N_t}\\mathbf{1}^\\top r_t.$$\n",
    "\n",
    "This equal-weight series is useful later because the traded portfolio is also equal-weighted within its selected names. Comparing equal-weight to equal-weight keeps the benchmark cleaner than comparing an equal-weight strategy to a cap-weighted index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1b1798b7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:30.619064Z",
     "iopub.status.busy": "2026-07-31T11:08:30.618871Z",
     "iopub.status.idle": "2026-07-31T11:08:30.957964Z",
     "shell.execute_reply": "2026-07-31T11:08:30.957393Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43776 characters]",
      "text/plain": [
       "<Figure size 1200x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average breadth: 455\n",
      "Min breadth: 385\n",
      "Max breadth: 501\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Universe breadth over time\n",
    "==================================\n",
    "\"\"\"\n",
    "# Count non-NaN returns each month\n",
    "universe_breadth = df_returns.notna().sum(axis=1)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12,5))\n",
    "ax.plot(universe_breadth.index, universe_breadth.values, color='steelblue')\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Number of stocks with valid returns')\n",
    "ax.set_title('Unverse Breadth Over Time')\n",
    "ax.grid(alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/01_market_stats/universe_breadth.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "print(f\"Average breadth: {universe_breadth.mean():.0f}\")\n",
    "print(f\"Min breadth: {universe_breadth.min()}\")\n",
    "print(f\"Max breadth: {universe_breadth.max()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0766c03b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:30.959816Z",
     "iopub.status.busy": "2026-07-31T11:08:30.959610Z",
     "iopub.status.idle": "2026-07-31T11:08:31.308702Z",
     "shell.execute_reply": "2026-07-31T11:08:31.308108Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 103220 characters]",
      "text/plain": [
       "<Figure size 1200x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average monthly CS dispersion: 0.0754\n",
      "Annualized: 0.2610\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Cross-sectional return dispersion\n",
    "==================================\n",
    "\"\"\"\n",
    "# Cross-sectional standard deviation each month\n",
    "cs_dispersion = df_returns.std(axis=1)\n",
    "\n",
    "fig,ax = plt.subplots(figsize=(12,5))\n",
    "ax.plot(cs_dispersion.index, cs_dispersion.rolling(12).mean(), color='coral', label='12m MA') # this smooths the last 12 months\n",
    "ax.plot(cs_dispersion.index, cs_dispersion, color='steelblue', alpha=0.4, label='Monthly')\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Cross-sectional std dev')\n",
    "ax.set_title('Cross-Sectional Return Dispersion')\n",
    "ax.legend()\n",
    "ax.grid(alpha=0.3)\n",
    "\n",
    "plt.tight_layout\n",
    "plt.savefig('../images/01_market_stats/cs_dispersion.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "print(f\"Average monthly CS dispersion: {cs_dispersion.mean():.4f}\")\n",
    "print(f\"Annualized: {cs_dispersion. mean() * np.sqrt(12):.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aada924e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:31.310403Z",
     "iopub.status.busy": "2026-07-31T11:08:31.310188Z",
     "iopub.status.idle": "2026-07-31T11:08:31.968041Z",
     "shell.execute_reply": "2026-07-31T11:08:31.967405Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54572 characters]",
      "text/plain": [
       "<Figure size 1200x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Annualized return: 0.1632\n",
      "Annualized vol: 0.1659\n",
      "Sharpe (rf=0): 0.984\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Equal-weight market index\n",
    "==================================\n",
    "\"\"\"\n",
    "ew_market = df_returns.mean(axis=1)\n",
    "cum_ew = (1 + ew_market).cumprod()\n",
    "\"\"\"\n",
    "Mathematically, .cumprod() will do the following:\n",
    "W_t = prod_{s=1}^t (1 + bar{r}_s) with W_0 = 1\n",
    "\"\"\"\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12,5))\n",
    "ax.plot(cum_ew.index, cum_ew.values, color='seagreen')\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Cumulative return')\n",
    "ax.set_title('Equal-Weight Universe Cumulative Return')\n",
    "ax.set_yscale('log')\n",
    "ax.grid(alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/01_market_stats/ew_market_cumulative.png', dpi = 150, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "ann_return = ew_market.mean() * 12\n",
    "ann_vol = ew_market.std() * np.sqrt(12)\n",
    "sharpe = ann_return / ann_vol\n",
    "\n",
    "print(f\"Annualized return: {ann_return:.4f}\")\n",
    "print(f\"Annualized vol: {ann_vol:.4f}\")\n",
    "print(f\"Sharpe (rf=0): {sharpe:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b53ad7a",
   "metadata": {},
   "source": [
    "## Sector Composition\n",
    "\n",
    "Sectors are a categorical partition of the stock universe. If there are $K$ sectors, we can encode membership with a binary matrix\n",
    "\n",
    "$$D \\in \\{0,1\\}^{N \\times K},$$\n",
    "\n",
    "where $D_{i,k}=1$ if stock $i$ belongs to sector $k$.\n",
    "\n",
    "This matters later because raw factors often contain sector tilts. A momentum signal might accidentally be long Technology and short Utilities, for example. If we want to study stock selection within sectors, we need to project those sector-level components out of the signal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f1ab950b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-31T11:08:31.969856Z",
     "iopub.status.busy": "2026-07-31T11:08:31.969674Z",
     "iopub.status.idle": "2026-07-31T11:08:32.287296Z",
     "shell.execute_reply": "2026-07-31T11:08:32.286748Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46028 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sector</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Industrials</th>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Financials</th>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Information Technology</th>\n",
       "      <td>74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Health Care</th>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Consumer Discretionary</th>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Consumer Staples</th>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Utilities</th>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Real Estate</th>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Materials</th>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Communication Services</th>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Energy</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        count\n",
       "sector                       \n",
       "Industrials                81\n",
       "Financials                 76\n",
       "Information Technology     74\n",
       "Health Care                59\n",
       "Consumer Discretionary     47\n",
       "Consumer Staples           34\n",
       "Utilities                  31\n",
       "Real Estate                31\n",
       "Materials                  26\n",
       "Communication Services     23\n",
       "Energy                     21"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "==================================\n",
    "Sector mapping\n",
    "==================================\n",
    "\"\"\"\n",
    "# Use GICS sector from the constituents table\n",
    "df_sector = df_constituents[['Symbol', 'GICS Sector']].copy()\n",
    "df_sector.columns = ['ticker', 'sector']\n",
    "df_sector = df_sector.dropna(subset=['sector'])\n",
    "\n",
    "# Save for later\n",
    "df_sector.to_csv('../data/processed/sector_mapping.csv', index=False)\n",
    "\n",
    "sector_counts = df_sector['sector'].value_counts()\n",
    "\n",
    "fig,ax = plt.subplots(figsize=(12,6))\n",
    "ax.barh(sector_counts.index, sector_counts.values, color='steelblue')\n",
    "ax.set_xlabel('Number of constituents')\n",
    "ax.set_title('S&P 500 Sector Composition')\n",
    "ax.invert_yaxis()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('../images/01_market_stats/sector_composition.png', dpi=150, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "display(sector_counts.to_frame('count'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d80a6197",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We now have the project's core return panel $R$: a monthly, current-constituent large-cap equity panel from 2005 onward, plus sector labels and basic market diagnostics.\n",
    "\n",
    "The important caveat is that this is a convenient public-data panel, not a survivorship-free institutional dataset. That does not make the project useless, but it does mean later claims need to be phrased carefully. The next notebook asks the first factor question: which cross-sectional score vectors have any relationship with subsequent stock returns?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
