{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aa61a422",
   "metadata": {},
   "source": [
    "# Notebook: data stage in depth\n",
    "\n",
    "This notebook walks **Stage 1: Data** in isolation. You will see how ARCADE\n",
    "loads traces, prepares labels, demodulates ADC when needed, splits shots, and\n",
    "writes caches. You will also learn which log lines to trust and how to debug\n",
    "shape, path, and bit-order failures before classification begins.\n",
    "\n",
    "**Prerequisites.** Editable install, linked paper data when using bundled configs,\n",
    "and a short YAML that points at a real HDF5 file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3935ac46",
   "metadata": {},
   "source": [
    "## Pipeline position\n",
    "\n",
    "```text\n",
    "[ Data ] \u2192 Features \u2192 Classify \u2192 Hardware \u2192 Report\n",
    "```\n",
    "\n",
    "Everything downstream assumes this stage froze the same traces and splits for\n",
    "every classifier. If demodulation or `qubit_bit_order` is wrong here, joint\n",
    "accuracy later will look mysteriously bad.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "673aba87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import logging\n",
    "\n",
    "from arcade._logging import configure_logging, get_logger\n",
    "from arcade.config import load_config\n",
    "from arcade.stages import DefaultDataStage\n",
    "\n",
    "configure_logging(logging.INFO)\n",
    "log = get_logger(\"tutorial.data\")\n",
    "\n",
    "# Prefer a small smoke config if you have one; otherwise the zoo YAML works\n",
    "# and simply spends longer in later stages you will not run here.\n",
    "CONFIG = Path(\"configs/example.yaml\")\n",
    "if not CONFIG.is_file():\n",
    "    CONFIG = Path(\"configs/paper_all_methods_benchmark.yaml\")\n",
    "\n",
    "cfg = load_config(CONFIG)\n",
    "print(\"data.path =\", cfg.data.path)\n",
    "print(\"num_qubits =\", cfg.data.num_qubits)\n",
    "print(\"qubit_bit_order =\", cfg.data.qubit_bit_order)\n",
    "print(\"demodulation.enabled =\", cfg.data.demodulation.enabled)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7207b0f5",
   "metadata": {},
   "source": [
    "## Expected log pattern for a cold start\n",
    "\n",
    "When the demod cache is missing, Stage 1 typically emits lines like:\n",
    "\n",
    "```text\n",
    "[INFO] arcade.stages.data: === Stage 1: Data ===\n",
    "[INFO] arcade.stages.data: Loading data from data/readout2019_adc.h5 (loader=hdf5)\n",
    "[INFO] arcade.stages.data: Loaded: traces (...), labels (...)\n",
    "[INFO] arcade.stages.data: After label preparation (format=...): unique labels=32\n",
    "[INFO] arcade.stages.data: Split indices computed on raw labels: train=..., val=..., test=...\n",
    "[INFO] arcade.stages.data: Demodulating raw ADC traces\n",
    "[INFO] arcade.stages.data: Preprocessed traces: ...\n",
    "```\n",
    "\n",
    "When a demod cache hits, you instead see `Loaded demod cache from ...` and\n",
    "skips the ADC demodulation path. That is normal and faster on the second run.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f51aa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run only the data stage. This is the primary debugging entry when loaders fail.\n",
    "stage = DefaultDataStage()\n",
    "data_out = stage.run(cfg)\n",
    "\n",
    "# Inspect the prepared bundle keys your classify stage will see.\n",
    "print(sorted(data_out.keys()) if isinstance(data_out, dict) else type(data_out))\n",
    "if isinstance(data_out, dict):\n",
    "    for key in (\"traces\", \"labels\", \"splits\", \"label_set\", \"filter_set\"):\n",
    "        if key in data_out:\n",
    "            val = data_out[key]\n",
    "            shape = getattr(val, \"shape\", None)\n",
    "            print(f\"{key}: type={type(val).__name__} shape={shape}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1490dcc6",
   "metadata": {},
   "source": [
    "## Debugging checklist for Stage 1\n",
    "\n",
    "| Symptom in logs / traceback | Likely cause | What to try |\n",
    "|----------------------------|--------------|-------------|\n",
    "| `FileNotFoundError` on `data.path` | Missing symlink or wrong path | Run `bash scripts/link_paper_data.sh` or fix YAML `data.path` |\n",
    "| Loader / key errors from HDF5 | Wrong `format`, `hdf5_keys`, or file contract | `arcade explore path --qubits N` then align YAML keys |\n",
    "| Joint labels look like `2**N` classes but accuracy collapses later | Wrong `qubit_bit_order` | Compare `lsb0` vs `msb0` against the paper config |\n",
    "| Demod runs every time and is slow | Cache disabled or path unset | Enable demod cache fields under `data.demodulation` |\n",
    "| `unique labels` unexpected | `label_format` / `num_levels` mismatch | Inspect `LabelSet` and raw label histogram |\n",
    "\n",
    "### Raise log detail\n",
    "\n",
    "```python\n",
    "configure_logging(logging.DEBUG)\n",
    "```\n",
    "\n",
    "DEBUG is noisy. Use it when a single loader or demod helper fails without a clear INFO line.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "562c4155",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate config before blaming the loader.\n",
    "from arcade.config import validate_and_explain\n",
    "\n",
    "errors = validate_and_explain(CONFIG)\n",
    "print(\"config valid:\" if not errors else \"config invalid:\", not bool(errors))\n",
    "for m in errors:\n",
    "    print(\" -\", m)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a262552",
   "metadata": {},
   "source": [
    "## Next\n",
    "\n",
    "Continue with [Classify stage](stage_classify.ipynb) using the same `data_out`\n",
    "object, or re-run data inside a full `arcade.run` once Stage 1 looks clean.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  },
  "nbsphinx": {
   "execute": "never"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}