7.1 TinyStories

7.1.1 Hyperparameter tuning

Total tokens processed 327,680,000 (your batch size × total step count × context length should equal roughly this value).

learning rate warmup 2%
betas (0.9, 0.95)
eps 1e-8
weight_decay 0.1

learning rate(1e-3 5e-4 3e-4 1e-4)


(a)
Perform a hyperparameter sweep over the learning rates and report the final losses (or note divergence if the optimizer diverges).
Deliverable: Learning curves associated with multiple learning rates. Explain your hyperparameter search strategy.
Deliverable: A model with validation loss (per-token) on TinyStories of at most 1.45

min_lr = 0.1 * max_lr, only search max_lr

0.1

0.02

0.01

1e-3

and later the gradient is less than 1 so chart didn't record it.

5e-4

strategy:

first try 1e-3, find it not diverge, so try bigger lr, finnaly found

260728 13:26 我们尝试使用学习率1e-3进行一次完整训练，观察其曲线


xxxxxxxxxx
(b)
Folk wisdom is that the best learning rate is “at the edge of stability.” Investigate how the point at which learning rates diverge is related to your best learning rate.
Deliverable: Learning curves of increasing learning rate which include at least one divergent run and an analysis of how this relates to convergence rates.

see diagrams above.

lr=1e-3

batch_size variations


xxxxxxxxxx
Problem (batch_size_experiment): Batch size variations (1 B200 hr) (1 point)
Vary your batch size all the way from 1 to the GPU memory limit. Try at least a few batch sizes in between, including typical sizes like 64 and 128.
Deliverable: Learning curves for runs with different batch sizes. The learning rates should be optimized again if necessary.
Deliverable: A few sentences discussing your findings on batch sizes and their impacts on training.

we can see that with batch_size increasing, training stability increases. and the loss of batch_size = 50 is 1.27 which is the least

Problem (generate): Generate text (1 point)


xxxxxxxxxx
Using your decoder and your trained checkpoint, report the text generated by your model. You may need to manipulate decoder parameters (temperature, top-p, etc.) to get fluent outputs.
Deliverable: Text dump of at least 256 tokens of text (or until the first <|endoftext|> token), and a brief comment on the fluency of this output and at least two factors which affect how good or bad this output is.

Once upon a time, there was a little bird named Timmy. Timmy was an obedient bird who always listened to his mom. One day, Timmy wanted to fly high in the sky. He told his mom, "I want to fly high, but I am too little to fly." His mom said, "Don't worry, Timmy. Just believe in yourself. I will encourage you to fly high in the sky." Timmy started to fly up, up, up. He was so happy. But then, he saw a big, shiny thing in the sky. It was a big, red balloon! Timmy was so excited. He took a deep breath and went closer to the balloon. He held it high, just like his mom said. Timmy was so happy. He thanked his mom and flew back down to the ground. From that day on, Timmy always believed in himself. He was a happy little bird who loved to fly high in the sky.

comment: I think it's fluent. factors: sampling parameters, input prompts.

Problem (layer_norm_ablation): Remove RMSNorm and train (0.5 B200 hrs) (1 point)


xxxxxxxxxx
Remove all of the RMSNorms from your Transformer and train. What happens at the previous optimal learning rate? Can you get stability by using a lower learning rate? 
Deliverable: A learning curve for when you remove RMSNorms and train, as well as a learning curve for the best learning rate.
Deliverable: A few sentences of commentary on the impact of RMSNorm.

Without RMSNorm, It has loss explosion.

So RMSNorm helps stability of training.

Problem (pre_norm_ablation): Implement post-norm and train (0.5 B200 hrs) (1 point)


xxxxxxxxxx
Modify your pre-norm Transformer implementation into a post-norm one. Train with the post-norm model and see what happens.
Deliverable: A learning curve for a post-norm Transformer, compared to the pre-norm one.

postnorm

prenorm

we can see postnorm has slightly higher loss of evaluation curve.

and the end loss also.

Problem (no_pos_emb): Implement NoPE (0.5 B200 hrs) (1 point)


xxxxxxxxxx
Modify your Transformer implementation with RoPE to remove the position embedding information entirely, and see what happens.
Deliverable: A learning curve comparing the performance of RoPE and NoPE.

Problem (swiglu_ablation): SwiGLU vs. SiLU (0.5 B200 hrs) (1 point)


xxxxxxxxxx
Deliverable: A learning curve comparing the performance of SwiGLU and SiLU feed-forward networks, with approximately matched parameter counts.
Deliverable: A few sentences discussing your findings.

Problem (main_experiment): Experiment on OWT (2 B200 hrs) (2 points)


xxxxxxxxxx
Train your language model on OpenWebText with the same model architecture and total training iterations as TinyStories. How well does this model do?
Deliverable: A learning curve of your language model on OpenWebText. Describe the difference in losses from TinyStories – how should we interpret these losses?
Deliverable: Generated text from OpenWebText LM, in the same format as the TinyStories outputs. How is the fluency of this text? Why is the output quality worse even though we have the same model and compute budget as TinyStories?

ski[p]

maybe more difficult. maybe it is large

Problem (leaderboard): Leaderboard (10 B200 hrs) (6 points)


xxxxxxxxxx
You will train a model under the leaderboard rules above with the goal of minimizing the validation loss of your language model within 0.75 B200-hours.
Deliverable: The final validation loss that was recorded, an associated learning curve that clearly shows a wall-clock-time x-axis that is less than 45 minutes, and a description of what you did. We expect a leaderboard submission to beat at least the naive baseline of a 5.0 loss. Submit to the leaderboard here: github.com/stanford-cs336/assignment1-basics-leaderboard.

skip.