Skip to content

Instantly share code, notes, and snippets.

View pruksmhc's full-sized avatar

Yada Pruksachatkun pruksmhc

View GitHub Profile
@pruksmhc
pruksmhc / gist:a316805d864324c4fa94cdb492d0b562
Created August 6, 2025 18:56
Qwen_2.5_7B_MMStar_baseline
{
"results": {
"mmstar": {
"alias": "mmstar",
"coarse perception,none": 0.7502214478878857,
"coarse perception_stderr,none": "N/A",
"average,none": 0.6282898425011137,
"average_stderr,none": "N/A",
"fine-grained perception,none": 0.5771677720461803,
"fine-grained perception_stderr,none": "N/A",
import os
import re
import pandas as pd
import multiprocessing as mp
import time
from datasets import Dataset, load_from_disk
import gc
from tqdm import tqdm
import psutil
import os
@pruksmhc
pruksmhc / calculate_tokens_per_byte_fineweb2hq.py
Last active July 24, 2025 01:21
calculate_tokens_per_byte_fineweb2hq.py
import multiprocessing as mp
from datasets import load_dataset
from transformers import AutoTokenizer
from itertools import islice
import tqdm
MAX_DOCS_PER_LANG = 10_000
langs = [
'rus_Cyrl', 'cmn_Hani', 'deu_Latn', 'jpn_Jpan', 'spa_Latn',
@pruksmhc
pruksmhc / Welcome file.md
Created April 30, 2019 17:30
Welcome file

Setup Tutorial

graph LR

A[Multitask: Pretrain Task 1, Pretrain task2]  --> B((Target Task 1))

A --> C(Round Rect)
@pruksmhc
pruksmhc / setup.md
Last active April 30, 2019 17:28
Welcome file

Setup Tutorial

graph LR

A[Multitask: Pretrain Task 1, Pretrain task2]  --> B((Target Task 1))

A --> C(Round Rect)
@pruksmhc
pruksmhc / setup.md
Last active April 30, 2019 17:28
Welcome file

Setup Tutorial

graph LR

A[Multitask: Pretrain Task 1, Pretrain task2]  --> B((Target Task 1))

A --> C(Round Rect)