Skip to content

Instantly share code, notes, and snippets.

@ffcai

ffcai/report.md

Created October 20, 2025 20:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ffcai/d0bc5ace3733540f7ad486fa94aba9ee to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ffcai/d0bc5ace3733540f7ad486fa94aba9ee to your computer and use it in GitHub Desktop.
Download ZIP
nanochat training report
Raw
report.md

nanochat training report

Generated: 2025-10-15 02:58:47

Environment

Git Information

  • Branch: master
  • Commit: 67aaca9 (dirty)
  • Message: export NANOCHAT_BASE_DIR so child processes get it too

Hardware

  • Platform: Linux
  • CPUs: 104 cores (208 logical)
  • Memory: 1771.7 GB
  • GPUs: 8x NVIDIA H100 80GB HBM3
  • GPU Memory: 633.5 GB total
  • CUDA Version: 12.8
  • Hourly Rate: $24.00/hour

Software

  • Python: 3.10.12
  • PyTorch: 2.8.0+cu128

Bloat

  • Characters: 330,622
  • Lines: 8,077
  • Files: 42
  • Tokens (approx): 82,655
  • Dependencies (uv.lock lines): 2,004

Run started: 2025-10-15 02:58:53

Tokenizer training

timestamp: 2025-10-15 03:00:03

  • max_chars: 2,000,000,000
  • doc_cap: 10,000
  • vocab_size: 65,536
  • train_time: 56.6142
  • num_special_tokens: 9
  • token_bytes_min: 1
  • token_bytes_max: 32
  • token_bytes_mean: 6.9197
  • token_bytes_std: 2.8748

Tokenizer evaluation

timestamp: 2025-10-15 03:00:07

Comparison with GPT-2

Text Type Bytes GPT-2 Tokens GPT-2 Ratio Ours Tokens Ours Ratio Relative Diff %
news 1819 404 4.50 375 4.85 +7.2%
korean 893 745 1.20 712 1.25 +4.4%
code 1259 576 2.19 492 2.56 +14.6%
math 1834 936 1.96 966 1.90 -3.2%
science 1112 260 4.28 228 4.88 +12.3%
fwe-train 4208518 900364 4.67 856883 4.91 +4.8%
fwe-val 4908443 1059062 4.63 1010352 4.86 +4.6%

Comparison with GPT-4

Text Type Bytes GPT-4 Tokens GPT-4 Ratio Ours Tokens Ours Ratio Relative Diff %
news 1819 387 4.70 375 4.85 +3.1%
korean 893 364 2.45 712 1.25 -95.6%
code 1259 309 4.07 492 2.56 -59.2%
math 1834 832 2.20 966 1.90 -16.1%
science 1112 249 4.47 228 4.88 +8.4%
fwe-train 4208518 874799 4.81 856883 4.91 +2.0%
fwe-val 4908443 1029691 4.77 1010352 4.86 +1.9%

Base model training

timestamp: 2025-10-15 06:05:50

  • run: dummy
  • depth: 20
  • max_seq_len: 2048
  • num_iterations: -1
  • target_flops: -1.0000
  • target_param_data_ratio: 20
  • device_batch_size: 32
  • total_batch_size: 524,288
  • embedding_lr: 0.2000
  • unembedding_lr: 0.0040
  • weight_decay: 0.0000
  • matrix_lr: 0.0200
  • grad_clip: 1.0000
  • eval_every: 250
  • eval_tokens: 10,485,760
  • core_metric_every: 2000
  • core_metric_max_per_task: 500
  • sample_every: 2000
  • model_tag:
  • Number of parameters: 560,988,160
  • Number of FLOPs per token: 3.491758e+09
  • Calculated number of iterations: 21,400
  • Number of training tokens: 11,219,763,200
  • Tokens : Params ratio: 20.0000
  • DDP world size: 8
  • warmup_ratio: 0.0000
  • warmdown_ratio: 0.2000
  • final_lr_frac: 0.0000
  • Minimum validation bpb: 0.8119
  • Final validation bpb: 0.8119
  • CORE metric estimate: 0.2158
  • MFU %: 48.52%
  • Total training flops: 3.917670e+19
  • Total training time: 171.13m
  • Peak memory usage: 75422.02MiB

Base model loss

timestamp: 2025-10-15 06:06:37

  • train bpb: 0.8147
  • val bpb: 0.8120
  • sample 0: <|bos|>The capital of France is Paris. It is located in the south of France. It is the largest city
  • sample 1: <|bos|>The chemical symbol of gold is Au. The chemical symbol of silver is Ag. The chemical symbol of copper is
  • sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Monday. If tomorrow is Monday, then tomorrow will be Tuesday. If tomorrow is
  • sample 3: <|bos|>The opposite of hot is cold. The opposite of cold is hot. The opposite of hot is cold.
  • sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
  • sample 5: <|bos|>My favorite color is blue. I love it. I love it. I love it. I love
  • sample 6: <|bos|>If 5x + 3 = 13, then x is a factor of 5. If 5x + 3 =

Base model evaluation

timestamp: 2025-10-15 06:10:31

  • Model: base_model (step 21400)
  • CORE metric: 0.2103
  • hellaswag_zeroshot: 0.2565
  • jeopardy: 0.1233
  • bigbench_qa_wikidata: 0.5271
  • arc_easy: 0.5370
  • arc_challenge: 0.1183
  • copa: 0.3000
  • commonsense_qa: 0.0551
  • piqa: 0.3721
  • openbook_qa: 0.1253
  • lambada_openai: 0.3676
  • hellaswag: 0.2620
  • winograd: 0.3114
  • winogrande: 0.0497
  • bigbench_dyck_languages: 0.1120
  • agi_eval_lsat_ar: 0.0870
  • bigbench_cs_algorithms: 0.3636
  • bigbench_operators: 0.1476
  • bigbench_repeat_copy_logic: 0.0000
  • squad: 0.2378
  • coqa: 0.2004
  • boolq: -0.1090
  • bigbench_language_identification: 0.1812

Midtraining

timestamp: 2025-10-15 06:18:44

  • run: dummy
  • dtype: bfloat16
  • max_seq_len: 2048
  • device_batch_size: 32
  • unembedding_lr: 0.0040
  • embedding_lr: 0.2000
  • matrix_lr: 0.0200
  • init_lr_frac: 1.0000
  • weight_decay: 0.0000
  • final_lr_frac: 0.0000
  • eval_every: 150
  • eval_tokens: 10,485,760
  • total_batch_size: 524,288
  • Number of iterations: 765
  • DDP world size: 8
  • Minimum validation bpb: 0.4158

Chat evaluation mid

timestamp: 2025-10-15 06:26:33

  • source: mid
  • task_name: None
  • dtype: bfloat16
  • temperature: 0.0000
  • max_new_tokens: 512
  • num_samples: 1
  • top_k: 50
  • batch_size: 8
  • model_tag: None
  • step: None
  • max_problems: None
  • ARC-Easy: 0.3632
  • ARC-Challenge: 0.2918
  • MMLU: 0.3014
  • GSM8K: 0.0243
  • HumanEval: 0.0671
  • ChatCORE metric: 0.0733

Chat SFT

timestamp: 2025-10-15 06:32:24

  • run: dummy
  • source: mid
  • dtype: bfloat16
  • device_batch_size: 4
  • num_epochs: 1
  • max_iterations: -1
  • target_examples_per_step: 32
  • unembedding_lr: 0.0040
  • embedding_lr: 0.2000
  • matrix_lr: 0.0200
  • weight_decay: 0.0000
  • init_lr_frac: 0.0200
  • eval_every: 100
  • eval_steps: 100
  • eval_metrics_every: 200
  • Training rows: 20,843
  • Number of iterations: 651
  • Training loss: 1.2165
  • Validation loss: 1.0672

Chat evaluation sft

timestamp: 2025-10-15 06:39:24

  • source: sft
  • task_name: None
  • dtype: bfloat16
  • temperature: 0.0000
  • max_new_tokens: 512
  • num_samples: 1
  • top_k: 50
  • batch_size: 8
  • model_tag: None
  • step: None
  • max_problems: None
  • ARC-Easy: 0.4205
  • ARC-Challenge: 0.2944
  • MMLU: 0.3239
  • GSM8K: 0.0409
  • HumanEval: 0.0366
  • ChatCORE metric: 0.0925

Summary

  • Characters: 330,622
  • Lines: 8,077
  • Files: 42
  • Tokens (approx): 82,655
  • Dependencies (uv.lock lines): 2,004
Metric BASE MID SFT RL
CORE 0.2103 - - -
ARC-Challenge - 0.2918 0.2944 -
ARC-Easy - 0.3632 0.4205 -
GSM8K - 0.0243 0.0409 -
HumanEval - 0.0671 0.0366 -
MMLU - 0.3014 0.3239 -
ChatCORE - 0.0733 0.0925 -

Total wall clock time: 3h40m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment