Skip to content

Instantly share code, notes, and snippets.

@jeremylongshore

jeremylongshore/backtester-audit.md

Created March 7, 2026 01:02
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save jeremylongshore/cec7dc7d17af56d84bf45d3fdda545d2 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save jeremylongshore/cec7dc7d17af56d84bf45d3fdda545d2 to your computer and use it in GitHub Desktop.
Download ZIP
Architecture audit: trading-strategy-backtester Claude Code plugin
Raw
backtester-audit.md

Quick Architecture Reference Card: trading-strategy-backtester

Plugin: trading-strategy-backtester Version: 1.0.0 Author: Intent Solutions IO (jeremy@intentsolutions.ai) License: MIT Marketplace: plugins/crypto/trading-strategy-backtester Audit Date: 2026-03-06

1. Overview

A comprehensive backtesting framework for crypto and traditional trading strategies, packaged as a Claude Code plugin. It lets users test 8 built-in strategies against historical price data via Yahoo Finance or CoinGecko, compute full performance/risk metrics (Sharpe, Sortino, VaR, max drawdown, etc.), and optimize parameters via grid search. Designed for traders, quants, and hobbyists who want to validate signal-based strategies before risking real capital.

Target users: Crypto traders, algo-trading hobbyists, quant researchers, finance students.

2. File Tree

trading-strategy-backtester/
├── .claude-plugin/
│   └── plugin.json                          # Plugin manifest (name, version, keywords)
├── LICENSE                                  # MIT License
├── README.md                                # User-facing docs, strategy list, install instructions
├── commands/
│   └── backtest-strategy.md                 # /backtest-strategy slash command (shortcut: bs)
└── skills/
    ├── backtesting-trading-strategies/      # PRIMARY SKILL
    │   ├── SKILL.md                         # Auto-activating skill definition (v2.0.0)
    │   ├── config/
    │   │   └── settings.yaml                # Default strategy params, costs, report settings
    │   ├── data/
    │   │   └── BTC_USD_1d.csv               # Cached historical data (auto-generated)
    │   ├── scripts/
    │   │   ├── backtest.py                  # Main backtesting engine (entry point)
    │   │   ├── strategies.py                # 8 strategy class definitions
    │   │   ├── metrics.py                   # Performance + risk metric calculations
    │   │   ├── fetch_data.py                # Data fetcher (yfinance, coingecko)
    │   │   ├── optimize.py                  # Grid search parameter optimizer
    │   │   └── __pycache__/                 # Python bytecode cache
    │   ├── references/
    │   │   ├── implementation.md            # Step-by-step implementation guide
    │   │   ├── examples.md                  # 10 usage examples (basic to multi-asset)
    │   │   └── errors.md                    # Error catalog with causes and solutions
    │   └── reports/                         # Generated output (3 backtests + 1 optimization)
    │       ├── sma_crossover_BTC-USD_*      # SMA summary, trades CSV, equity CSV, chart PNG
    │       ├── rsi_reversal_BTC-USD_*       # RSI summary, trades CSV, equity CSV, chart PNG
    │       ├── macd_BTC-USD_*               # MACD summary, trades CSV, equity CSV, chart PNG
    │       └── optimization_sma_crossover_* # Optimization results (CSV + TXT)
    └── skill-adapter/                       # SCAFFOLDING (template/boilerplate)
        ├── assets/
        │   ├── README.md                    # Planned asset checklist (not yet implemented)
        │   ├── config-template.json         # Generic skill config template
        │   ├── skill-schema.json            # JSON schema for SKILL.md frontmatter
        │   └── test-data.json               # Generic test case fixtures
        ├── references/
        │   ├── README.md                    # Planned reference doc checklist
        │   ├── best-practices.md            # Skill development best practices
        │   └── examples.md                  # Generic skill usage examples (template)
        └── scripts/
            ├── README.md                    # Planned script checklist
            ├── validation.sh                # Validates SKILL.md frontmatter exists
            └── helper-template.sh           # Bash helper script template

3. Architecture

Component Connection Flow

SKILL.md (auto-activation via trigger phrases)
    │
    ├──> scripts/fetch_data.py ──> data/{symbol}_{interval}.csv (cached)
    │                                      │
    │                                      v
    ├──> scripts/backtest.py ─────> scripts/strategies.py (8 strategies)
    │         │                            │
    │         └──> scripts/metrics.py <────┘
    │                   │
    │                   v
    │              reports/*_summary.txt
    │              reports/*_trades.csv
    │              reports/*_equity.csv
    │              reports/*_chart.png
    │
    └──> scripts/optimize.py ──> reports/optimization_*.csv + .txt

Activation Path

  1. User says a trigger phrase (e.g., "backtest my SMA strategy on BTC")
  2. SKILL.md auto-activates, granting Read, Write, Edit, Grep, Glob, Bash(python:*) tools
  3. Claude follows SKILL.md instructions, executing Python scripts via Bash
  4. Scripts fetch data, run strategy, compute metrics, save reports
  5. Claude reads reports and presents formatted results to user

Slash Command Path

  1. User types /backtest-strategy (or bs)
  2. Command provides the backtest-strategy.md instructions (includes JS pseudocode for context)
  3. Claude executes the actual Python scripts to perform the backtest

4. Key Components

4.1 SKILL.md Frontmatter

name: backtesting-trading-strategies
description: |
  Backtest crypto and traditional trading strategies against historical data.
  Calculates performance metrics (Sharpe, Sortino, max drawdown), generates equity curves,
  and optimizes strategy parameters.
allowed-tools: Read, Write, Edit, Grep, Glob, Bash(python:*)
version: 2.0.0
author: Jeremy Longshore <jeremy@intentsolutions.io>
license: MIT

Trigger phrases: "backtest strategy", "test trading strategy", "historical performance", "simulate trades", "optimize parameters", "validate signals"

4.2 Command: /backtest-strategy

  • Shortcut: bs
  • File: commands/backtest-strategy.md
  • Contains: JavaScript pseudocode illustrating the StrategyBacktester class with 8 strategies, the run loop, performance calculations, risk metrics, and an ASCII display formatter
  • Purpose: Provides Claude with conceptual architecture context alongside the real Python implementation

4.3 Python Scripts

Script Lines Purpose Key Functions/Classes
backtest.py 343 Main engine + CLI run_backtest(), load_data(), save_results(), main()
strategies.py 337 8 strategy definitions Strategy (ABC), SMAcrossover, EMAcrossover, RSIreversal, MACD, BollingerBands, Breakout, MeanReversion, Momentum, Signal dataclass
metrics.py 321 Performance + risk calcs Trade dataclass, BacktestResult dataclass, calculate_all_metrics(), format_results(), individual metric functions
fetch_data.py 150 Data acquisition + caching fetch_yfinance(), fetch_coingecko()
optimize.py 197 Grid search optimization grid_search(), format_optimization_results()

4.4 Config: settings.yaml

data:
  provider: yfinance | coingecko
  cache_dir: ./data
  default_interval: 1d

backtest:
  default_capital: 10000       # USD
  commission: 0.001            # 0.1% per trade
  slippage: 0.0005             # 0.05% slippage

risk:
  max_position_size: 0.95      # 95% of capital
  stop_loss: null              # Optional
  take_profit: null            # Optional

reporting:
  output_dir: ./reports
  save_trades: true
  save_equity: true
  save_chart: true

strategies:                    # Default params for each strategy
  sma_crossover: { fast_period: 20, slow_period: 50 }
  ema_crossover: { fast_period: 12, slow_period: 26 }
  rsi_reversal: { period: 14, overbought: 70, oversold: 30 }
  macd: { fast: 12, slow: 26, signal: 9 }
  bollinger_bands: { period: 20, std_dev: 2.0 }
  breakout: { lookback: 20, threshold: 0.0 }
  mean_reversion: { period: 20, z_threshold: 2.0 }
  momentum: { period: 14, threshold: 5.0 }

4.5 References

File Purpose
implementation.md Step-by-step guide: install deps, fetch data, run backtest, optimize, analyze. Includes architecture diagram, custom strategy template, and config example.
examples.md 10 practical examples: basic SMA, RSI reversal, MACD with custom costs, parameter optimization, multi-strategy comparison, Bollinger bands, breakout, listing strategies, walk-forward analysis, multi-asset portfolio.
errors.md Error catalog covering: data fetching issues, insufficient data, missing deps, unknown strategy, invalid JSON params, lookback exceeded, division by zero, NaN results, permissions, memory errors, optimization timeouts, overfitting warnings, all-loss diagnostics.

5. Strategies Supported

# Strategy Class Description Parameters Default Lookback
1 sma_crossover SMAcrossover Golden/death cross on simple MAs fast_period (20), slow_period (50) 200 bars
2 ema_crossover EMAcrossover Crossover on exponential MAs fast_period (12), slow_period (26) 200 bars
3 rsi_reversal RSIreversal Buy oversold, sell overbought period (14), overbought (70), oversold (30) 14 bars
4 macd MACD MACD/signal line crossover fast (12), slow (26), signal (9) 35 bars
5 bollinger_bands BollingerBands Mean reversion on band touches period (20), std_dev (2.0) 20 bars
6 breakout Breakout Price breakout above/below range lookback (20), threshold (0.0) 20 bars
7 mean_reversion MeanReversion Z-score deviation from SMA period (20), z_threshold (2.0) 20 bars
8 momentum Momentum Rate-of-change threshold trigger period (14), threshold (5.0) 14 bars

All strategies inherit from Strategy ABC and implement generate_signals(data, params) -> Signal. The Signal dataclass carries entry, exit, direction ("long"/"short"), and strength (0-1).

6. Data Flow

1. DATA ACQUISITION
   User request --> fetch_data.py
                       |
                       ├─ yfinance: yf.Ticker(symbol).history(start, end, interval)
                       └─ coingecko: REST API /coins/{id}/market_chart
                       |
                       v
                    data/{SYMBOL}_{interval}.csv  (cached, OHLCV columns)

2. BACKTEST EXECUTION
   backtest.py loads CSV --> pd.DataFrame
       |
       ├─ get_strategy(name) from strategies.py registry
       ├─ Loop through bars starting at strategy.lookback:
       │     ├─ strategy.generate_signals(data_slice, params) -> Signal
       │     ├─ If Signal.entry: open position (95% capital, apply slippage + commission)
       │     └─ If Signal.exit: close position (apply slippage + commission), record Trade
       └─ Force-close any open position at end of data

3. METRICS CALCULATION
   metrics.py receives List[Trade] + equity_curve:
       ├─ Performance: total_return, CAGR, Sharpe, Sortino, Calmar
       ├─ Risk: max_drawdown, VaR(95%), CVaR(95%), volatility, ulcer_index
       └─ Trade stats: win_rate, profit_factor, expectancy, consecutive wins/losses

4. REPORT GENERATION
   save_results() writes to reports/:
       ├─ *_summary.txt   - ASCII-formatted performance table
       ├─ *_trades.csv     - Entry/exit times, prices, PnL per trade
       ├─ *_equity.csv     - Daily portfolio value
       └─ *_chart.png      - Equity curve + drawdown chart (matplotlib)

5. PARAMETER OPTIMIZATION (optional)
   optimize.py: grid_search() over param_grid
       ├─ Runs run_backtest() for each combination
       ├─ Ranks by target metric (default: Sharpe ratio)
       └─ Saves optimization_*.csv + optimization_*.txt

7. Sample Output

SMA Crossover on BTC-USD (1 year, $10,000 capital)

╔══════════════════════════════════════════════════════════════════════╗
║            BACKTEST RESULTS:    SMA_CROSSOVER                      ║
║            BTC-USD | 2025-01-14 to 2026-01-13                      ║
╠══════════════════════════════════════════════════════════════════════╣
║ PERFORMANCE                          | RISK                         ║
║ Total Return:          +4.24%        | Max Drawdown:       -14.02%  ║
║ CAGR:                  +4.26%        | VaR (95%):           -1.16%  ║
║ Sharpe Ratio:           0.36         | Volatility:          15.00%  ║
║ Sortino Ratio:          0.21         | Ulcer Index:          8.63   ║
║ Calmar Ratio:           0.30         | CVaR (95%):          -2.42%  ║
╠══════════════════════════════════════════════════════════════════════╣
║ TRADE STATISTICS                                                    ║
║ Total Trades:              2         | Profit Factor:         inf   ║
║ Win Rate:              100.0%        | Expectancy:     $    231.27  ║
║ Avg Win:          $    231.27        | Max Consec. Losses:     0    ║
║ Avg Loss:         $      0.00        | Avg Duration:        15.5d   ║
╠══════════════════════════════════════════════════════════════════════╣
║ Capital: $10,000 -> $10,424                                         ║
╚══════════════════════════════════════════════════════════════════════╝

RSI Reversal on BTC-USD (same period): -5.01% return, 80% win rate but $-1695 avg loss dragged performance negative.

MACD on BTC-USD (same period): -14.96% return, 26.7% win rate, 15 trades, Sharpe -0.53.

Optimization Result: Best SMA params were fast=20, slow=50 with 4.24% return, Sharpe 0.37. Most other combinations generated zero trades in the period tested.

8. Dependencies

Required (Python)

Package Purpose
pandas DataFrame operations, time series, CSV I/O
numpy Numerical calculations (Sharpe, VaR, etc.)
yfinance Yahoo Finance historical data fetching
matplotlib Equity curve and drawdown chart generation

Optional

Package Purpose
ta-lib Advanced technical analysis indicators
scipy Statistical functions for optimization
scikit-learn ML-based strategy enhancements
requests CoinGecko API access (for fetch_data.py --source coingecko)

External APIs

API Usage Auth Required
Yahoo Finance (via yfinance) Primary data source No
CoinGecko Alternative crypto data source No (free tier)

9. Strengths

  1. Well-structured Python codebase. Clean separation of concerns: data fetching, strategy definitions, metrics, optimization, and the main engine are all in separate, focused modules. The Strategy ABC with a generate_signals() contract makes it easy to add new strategies.

  2. Comprehensive metrics suite. Covers performance (Sharpe, Sortino, Calmar, CAGR), risk (VaR, CVaR, max drawdown, ulcer index), and trade statistics (win rate, profit factor, expectancy, consecutive losses). The BacktestResult dataclass cleanly encapsulates everything.

  3. Realistic cost modeling. Commission (0.1%) and slippage (0.05%) are applied per trade, with a 95% position sizing cap. This prevents the common backtesting pitfall of ignoring transaction costs.

  4. Good error documentation. The errors.md reference covers a wide range of failure modes with specific causes and solutions, including overfitting warnings and diagnostics for all-loss scenarios.

  5. Rich example library. 10 progressively complex examples in examples.md cover basic backtests, custom costs, parameter optimization, multi-strategy comparison, walk-forward analysis, and multi-asset portfolios.

  6. Data caching. Fetched data is persisted to CSV, avoiding redundant API calls during iterative testing.

  7. Four output formats per backtest. Summary text, trade log CSV, equity curve CSV, and a matplotlib chart PNG provide both human-readable and programmatic results.

  8. Grid search optimizer. optimize.py provides exhaustive parameter search with ranked results, making it straightforward to find optimal strategy configurations.

10. Gaps and Improvement Opportunities

Architecture Issues

  1. settings.yaml is not actually loaded by any script. The config file exists and is well-structured, but backtest.py, fetch_data.py, and optimize.py all use hardcoded defaults and CLI arguments. The settings.yaml is documentation-only. Should be loaded via yaml.safe_load() as a fallback when CLI args are not provided.

  2. parse_period() is duplicated. Both backtest.py and fetch_data.py define identical parse_period() functions. Should be extracted to a shared utils.py module.

  3. No __init__.py in the scripts directory. While sys.path.insert(0, ...) works, proper Python packaging with __init__.py would be cleaner and enable standard imports.

Strategy Limitations

  1. Long-only strategies. All 8 strategies only generate direction="long" signals. The Signal dataclass supports "short" but no strategy uses it. Short selling would significantly expand the strategy universe.

  2. No stop-loss or take-profit enforcement. The risk section in settings.yaml defines stop_loss and take_profit fields, but the backtest engine does not implement them. Positions are only closed on strategy exit signals or end-of-data.

  3. Fixed 95% position sizing. The max_position_size config is not read; the engine hardcodes cash * 0.95. No support for Kelly criterion, fixed fractional, or other position sizing methods.

Data & Testing

  1. CoinGecko OHLV data is approximate. The fetch_coingecko() function creates fake high/low columns as close * 1.01 / close * 0.99 and sets volume to 0. This makes strategies that depend on high/low/volume data (breakout, Bollinger) unreliable when using CoinGecko as a source.

  2. No unit tests. No pytest or unittest files exist. The strategies, metrics calculations, and signal generation logic should all have test coverage.

  3. No walk-forward analysis script. The README lists /walk-forward as a command, and examples.md shows a manual approach, but no actual walk-forward analysis script exists. The skill-adapter/scripts/README.md lists it as planned but unimplemented.

Skill-Adapter Quality

  1. skill-adapter is entirely boilerplate. The skill-adapter/ directory contains only generic template files (config-template.json, skill-schema.json, test-data.json, helper-template.sh) with no backtester-specific content. The README checklists in assets/, references/, and scripts/ list planned files that do not exist. This appears to be scaffolding from a plugin template that was never customized.

Minor Issues

  1. README lists commands that do not exist. /optimize-parameters, /compare-strategies, and /walk-forward are listed in the README command table but have no corresponding command files in commands/. Only /backtest-strategy exists.

  2. Equity curve index can misalign. In backtest.py line 176, equity_curve = pd.Series(equity, index=data.index[strategy.lookback-1:]) could fail if the equity list length does not match the sliced index length (off-by-one risk depending on lookback value vs. data length).

  3. No .gitignore for generated artifacts. The data/, reports/, and __pycache__/ directories contain generated files that are committed to the repo. A .gitignore should exclude *.csv, *.png, *.pyc, and cached data.

  4. Command file contains JS pseudocode, actual implementation is Python. The backtest-strategy.md command includes a large JavaScript class implementation that does not match the actual Python scripts. This could confuse Claude into attempting a JS-based approach instead of running the Python scripts.

Generated 2026-03-06 by appauditmini analysis of plugins/crypto/trading-strategy-backtester/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment