jphme

2026-01-14 09:22:23,310 - xyz_agent.training.train - INFO - Reward distribution: retrieval(+24/-40), correctness(+18/-46), citations(+8/-56), format_penalty(1/64), no_final_answer(0/64)
2026-01-14 09:22:23,311 - xyz_agent.training.train - INFO - Group advantages: min=-0.475, max=1.100, mean=-0.000
2026-01-14 09:22:23,311 - xyz_agent.training.train - INFO - Preparing training data...
2026-01-14 09:22:23,463 - xyz_agent.training.train - INFO - Prepared 64 training examples
2026-01-14 09:22:23,463 - xyz_agent.training.train - INFO - First datum model_input length: 7599
2026-01-14 09:22:23,463 - xyz_agent.training.train - INFO -   target_tokens: dtype=int64, shape=[7599]
2026-01-14 09:22:23,463 - xyz_agent.training.train - INFO -   logprobs: dtype=float32, shape=[7599]
2026-01-14 09:22:23,463 - xyz_agent.training.train - INFO -   advantages: dtype=float32, shape=[7599]
2026-01-14 09:22:23,463 - xyz_agent.training.train - INFO -   mask: dtype=float32, shape=[7599]
2026-01-14 09:22:23,463 - xyz_agent.training.train

jphme

KL Divergence Analysis Report

Executive Summary

KL divergence values during RL training were extremely high (4-7) when they should be below 0.01 for stable on-policy training.

Root Cause Identified (2025-01-04): The issue was a token sequence mismatch between sampling and training caused by:

1. Missing final tokens: all_tokens didn't include the final generated tokens when obs.done=True
2. Retokenization mismatch: When the renderer re-renders conversation history, generated tokens get different tokenization (only 6.7% match!)

jphme

Understand the Task: Grasp the main objective, goals, requirements, constraints, and expected output.
- Minimal Changes: If an existing prompt is provided, improve it only if it's simple. For complex prompts, enhance clarity and add missing elements without altering the original structure.
- Reasoning Before Conclusions: Encourage reasoning steps before any conclusions are reached. ATTENTION! If the user provides examples where the reasoning happens afterward, REVERSE the order! NEVER START EXAMPLES WITH CONCLUSIONS!
    - Reasoning Order: Call out reasoning portions of the prompt and conclusion parts (specific fields by name). For each, determine the ORDER in which this is done, and whether it needs to be reversed.
    - Conclusion, classifications, or results should ALWAYS appear last.
- Examples: Include high-quality examples if helpful, using placeholders [in brackets] for complex elements.
   - What kinds of examples may need to be included, how many, and whether they are complex enough to benefit from p

jphme

jphme

#only changed self.vocab_size = vocab_size in configuration_mixformer_sequential.py, as suggested by @NanoBit
base_model: /workspace/models/phi-1_5
base_model_config: /workspace/models/phi-1_5
trust_remote_code: true
base_model_ignore_patterns:
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
is_llama_derived_model: false

load_in_8bit: false

jphme

{
    "gehaltserh\u00f6hung": "Bitte schreibe einen Brief an meinen Chef und erkl\u00e4re ihm, warum ich eine Gehalterh\u00f6hung verdient habe.",
    "bundeskanzler": "Wenn du 2024 als Bundeskanzler kandidieren w\u00fcrdest, was w\u00e4re dein Programm?",
    "hauptstadt_nrw": "Was ist die Hauptstadt von Nordrhein Westfalen?",
    "zusammenfassung_gesetz": "Bitte fasse kurz f\u00fcr Laien zusammen, worum es in folgendem Ausschnitt aus einem Gesetz geht:\n              \nGesetz \u00fcber den Versicherungsvertrag (Versicherungsvertragsgesetz - VVG)\n\u00a7 1a Vertriebst\u00e4tigkeit des Versicherers\n(1) Der Versicherer muss bei seiner Vertriebst\u00e4tigkeit gegen\u00fcber Versicherungsnehmern stets ehrlich, redlich und professionell in deren bestm\u00f6glichem Interesse handeln. Zur Vertriebst\u00e4tigkeit geh\u00f6ren\n\n1.\n    Beratung,\n2.\n    Vorbereitung von Versicherungsvertr\u00e4gen einschlie\u00dflich Vertragsvorschl\u00e4gen,\n3.\n    Abschluss von Versicherungsvertr\u00e4gen,\n4.\n    Mitwirken 

jphme

Haushaltsposition ,2020,2021,2022,2023,2024,2025,2026,2027,Abteilung
Steuern und ähnliche Abgaben,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Zuwendungen und allgemeine Umlagen,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Sonstige Transfererträge,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Öffentlich-rechtliche Leistungsentgelte,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Privatrechtliche Leistungsentgelte,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Kostenerstattungen und Kostenumlagen,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Sonstige ordentliche Erträge,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Aktivierte Eigenleistungen,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10
Bestandsveränderungen (+/-),0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Amtsinterne Querschnittsaufgaben Amt 10

jphme

val_set_size: 0.03
adapter: qlora
lora_model_dir:
sequence_len: 4096
max_packed_sequence_len: 4096
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true

jphme

{
  "name": "Llama 2 German partial multilang Config",

  "inference_params": {
    "top_k": 40,
    "top_p": 0.95,
    "temp": 0.5,
    "repeat_penalty": 1.2,
    "input_prefix": "[INST]",
    "input_suffix": "[/INST]",

jphme

Keybase proof

I hereby claim:

• I am jpdus on github.
• I am jp1 (https://keybase.io/jp1) on keybase.
• I have a public key whose fingerprint is 8A44 FB5E 1B45 DB11 7C7C ABD2 FC1E 49B4 76E1 EC77

To claim this, I am signing this object: