ai-agent-book/cursor-chats/20251003_120151_@https_cookbook.openai.com_articles_gpt-oss_fine-tune-transfo.md

Cursor Chat: ai-agent-book

Metadata

• Project: ai-agent-book
• Path: /Users/boj
• Date: 2025-10-03 12:01:51
• Session ID: bca9045b-df88-4878-8b9a-56b7f24ab0fc

Conversation

👤 You

@https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers

from datasets import load_dataset

dataset = load_dataset("HuggingFaceH4/Multilingual-Thinking", split="train") print(dataset) print(dataset[0])

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")

messages = dataset[0]["messages"] conversation = tokenizer.apply_chat_template(messages, tokenize=False) print(conversation)

import torch from transformers import AutoModelForCausalLM, Mxfp4Config

quantization_config = Mxfp4Config(dequantize=True) model_kwargs = dict( attn_implementation="eager", torch_dtype=torch.bfloat16, quantization_config=quantization_config, use_cache=False, device_map="auto", )

model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", **model_kwargs)

messages = [ {"role": "user", "content": "¿Cuá output_ids = model.generate(input_ids, max_new_tokens=512) response = tokenizer.batch_decode(output_ids)[0] print(response)

from peft import LoraConfig, get_peft_model

peft_config = LoraConfig( r=8, lora_alpha=16, target_modules="all-linear", target_parameters=[ "7.mlp.experts.gate_up_proj", "7.mlp.experts.down_proj", "15.mlp.experts.gate_up_proj", "15.mlp.experts.down_proj", "23.mlp.experts.gate_up_proj", "23.mlp.experts.down_proj", ], ) peft_model = get_peft_model(model, peft_config) peft_model.print_trainable_parameters()

from trl import SFTConfig

training_args = SFTConfig( learning_rate=2e-4, gradient_checkpointing=True, num_train_epochs=1, logging_steps=1, per_device_train_batch_size=2, gradient_accumulation_steps=4, max_length=1024, warmup_ratio=0.03, lr_scheduler_type="cosine_with_min_lr", lr_scheduler_kwargs={"min_lr_rate": 0.1}, output_dir="gpt-oss-20b-multilingual-reasoner", report_to="trackio", push_to_hub=False, )

from trl import SFTTrainer

trainer = SFTTrainer( model=peft_model, args=training_args, train_dataset=dataset, processing_class=tokenizer, ) trainer.train()

trainer.save_model(training_args.output_dir)

-- I lowered 2048 -> 1024, batch size 4 -> 2, but still OOM

File "/root/miniconda3/lib/python3.13/site-packages/torch/nn/utils/parametrize.py", line 300, in forward x = self0 File "/root/miniconda3/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) File "/root/miniconda3/lib/python3.13/site-packages/peft/tuners/lora/layer.py", line 1776, in forward return W + self.delta_weight ~~^~~~~~~~~~~~~~~~~~~ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 79.19 GiB of which 869.56 MiB is free. Process 1311362 has 78.32 GiB memory in use. Of the allocated memory 77.22 GiB is allocated by PyTorch, and 380.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

-- The openai blog says it fits on 80GB H100. Why it does not work for me

---

🤖 Cursor Assistant

Looking at your OOM error, I can see that you're using 78.32 GiB out of 79.19 GiB available. The issue is likely related to a few factors:

Key Issues

1. Mxfp4Config(dequantize=True) - This parameter causes the model to be dequantized back to full precision during computation, which defeats the purpose of quantization and uses significantly more memory.

2. Memory fragmentation - The error message suggests this: "380.06 MiB is reserved by PyTorch but unallocated"

3. LoRA delta computation - The error occurs during LoRA forward pass when computing W + self.delta_weight, which requires materializing the full weight matrix.

Solutions to Try

Let me create a modified script with several memory-saving techniques:

---

🤖 Cursor Assistant

Now let me create a detailed troubleshooting guide:

---

🤖 Cursor Assistant

Let me also create a minimal debugging script to help you identify exactly where the memory issue occurs:

---

Exported from Cursor View