https://github.com/bigsnarfdude/researchRalph
Summary of Validated Findings
┌───────────────────────┬─────────────────┬──────────────────────────────────────┐
│ Experiment │ Result │ Learning │
├───────────────────────┼─────────────────┼──────────────────────────────────────┤
│ jan19_loop │ 0.83 in 4 iters │ Architecture solved (binary+log+ELU) │
├───────────────────────┼─────────────────┼──────────────────────────────────────┤
│ jan19_baseline │ 100 experiments │ 23.5x less efficient than Ralph │
├───────────────────────┼─────────────────┼──────────────────────────────────────┤
│ jan20_random_baseline │ 0.51 vs 0.83 │ Semantic features essential (+0.33) │
└───────────────────────┴─────────────────┴──────────────────────────────────────┘
Proposed Feature Discovery Ralph Loop
Goal: Beat 0.83 AUROC by finding better features (target: 0.90)
Key insight: Layer 40 is most predictive (0.78 alone) but only 3/8 features come from it.
Prioritized Experiments
1. Ablation study (free - uses cache)
- Drop each feature, rank by importance
- Identify weakest for replacement
2. L40 expansion (~1hr extraction)
- Add more L40 features
- Expected: 0.84-0.86
3. Semantic cluster search (~2hr)
- Find features for awareness/reasoning/action phases
- Expected: 0.85-0.88
4. Correlation expansion (~1hr)
- Find features that co-activate with F12574
- Efficient search without semantic analysis
Files Created
researchRalph/
├── README.md # Framework overview
├── hypothesis.json # Current state
├── progress.txt # All learnings
└── FEATURE_DISCOVERY_PROPOSAL.md # Concrete next steps
researchRalph for model organisms to create new benchmarks. jan22 experiments