Commit Graph

Select branches

Hide Pull Requests

autocast_wip

bias_test

init_test

master

multi_node_ddp

tie_weights

#1

#10

#101

#102

#106

#115

#116

#119

#120

#122

#125

#128

#14

#142

#143

#144

#145

#146

#15

#151

#152

#156

#156

#16

#160

#166

#17

#173

#174

#177

#18

#180

#181

#189

#19

#191

#195

#199

#20

#201

#201

#205

#207

#209

#21

#214

#220

#224

#225

#227

#23

#236

#238

#238

#24

#240

#242

#243

#246

#246

#247

#248

#248

#250

#254

#254

#258

#258

#259

#26

#260

#263

#263

#265

#266

#27

#270

#274

#275

#276

#277

#279

#284

#286

#29

#292

#292

#295

#295

#296

#298

#3

#300

#300

#301

#302

#305

#306

#307

#309

#311

#314

#315

#316

#321

#324

#330

#330

#332

#332

#333

#333

#334

#334

#335

#337

#338

#339

#339

#34

#342

#346

#346

#348

#350

#353

#356

#356

#359

#359

#360

#360

#361

#361

#362

#362

#364

#364

#365

#365

#366

#368

#368

#369

#369

#372

#372

#376

#379

#379

#382

#382

#384

#386

#386

#387

#387

#392

#393

#393

#394

#394

#395

#398

#398

#4

#40

#400

#402

#402

#403

#403

#404

#406

#408

#409

#409

#412

#412

#414

#415

#415

#419

#420

#427

#428

#429

#43

#430

#431

#44

#441

#444

#446

#446

#449

#450

#450

#452

#453

#453

#454

#454

#459

#459

#46

#462

#463

#464

#481

#481

#487

#488

#488

#490

#491

#5

#51

#526

#527

#527

#528

#528

#530

#530

#533

#533

#534

#538

#538

#54

#540

#540

#541

#546

#546

#549

#55

#550

#553

#555

#555

#556

#557

#558

#558

#559

#561

#562

#563

#564

#565

#566

#568

#569

#569

#57

#571

#572

#573

#574

#574

#575

#576

#578

#580

#580

#581

#582

#582

#583

#583

#584

#585

#585

#587

#587

#588

#590

#590

#592

#593

#596

#596

#597

#599

#599

#6

#600

#601

#601

#603

#604

#607

#607

#608

#609

#610

#611

#612

#612

#613

#613

#614

#614

#615

#617

#619

#619

#620

#620

#621

#623

#625

#625

#626

#626

#628

#628

#630

#632

#632

#634

#638

#638

#640

#641

#643

#643

#645

#645

#647

#648

#649

#650

#650

#651

#651

#652

#652

#655

#657

#657

#659

#66

#660

#662

#663

#663

#664

#664

#666

#667

#667

#668

#669

#669

#670

#670

#671

#671

#672

#672

#673

#673

#674

#676

#678

#679

#679

#68

#680

#680

#681

#681

#682

#683

#683

#684

#687

#688

#688

#690

#691

#691

#694

#695

#696

#697

#697

#698

#698

#699

#699

#700

#701

#701

#702

#705

#705

#71

#72

#73

#74

#76

#77

#78

#79

#80

#82

#86

#87

#9

#96

Mono Color

• d995c22128 fix bug with loading GPT-2 parameters, assert gets incorrectly tripped due to .bias missing since it is now optionally present depending on flash or not Andrej Karpathy 2023-02-01 02:05:34 +00:00
• 038ce89438 rename iter to it, because iter is a concrete Python builtin Andrej Karpathy 2023-01-31 23:34:02 +00:00
• d2705bd92a tune cited numbers and reproductions and more explicitly point out the problems w.r.t. the OWT vs WT domain gap Andrej Karpathy 2023-01-31 21:57:07 +00:00
• 4386bce1f4 adjust teaser figure with a more tuned result Andrej Karpathy 2023-01-31 21:43:30 +00:00
• 924a0873eb merge, make cleaner, careful with gradient clipping when using grad scaler fp16 training Andrej Karpathy 2023-01-30 23:40:35 +00:00
• ae06d0b15a add flash attention support, resolving last few issues but for now seems to work ok Andrej Karpathy 2023-01-30 23:18:26 +00:00
• 0e90ee9d48 based on my experiments these biases are indeed not needed. code runs faster, identical results. keeping the option just because it deviates from the gpt-2 setup Andrej Karpathy 2023-01-30 08:07:58 +00:00
• 001c1e7be7 stay true to the README file and set grad accum to 5, so the default batch size is about 0.5M and is reproducing gpt2 Andrej Karpathy 2023-01-27 20:51:50 +00:00
• 79dbe0086d let me set bias=True until I validate it properly, but this should be ok to merge to master for now, is equivalent to previous functionality Andrej Karpathy 2023-01-27 20:45:28 +00:00
• e808a67149 bunch of plumbing of bias all around. measuring bias=False to be about 6% faster Andrej Karpathy 2023-01-27 20:41:17 +00:00
• cc5444e194 add the bias option to config, default it to True for now Andrej Karpathy 2023-01-27 20:29:45 +00:00
• 2bf07a3fbf rewrite model class so layernorm has an optional bias= parameter Andrej Karpathy 2023-01-27 20:17:32 +00:00
• 2892858ce7 attempt a non-biased model, per few papers that cite this as working well bias_test Andrej Karpathy 2023-01-27 18:54:08 +00:00
• f29a9ff5bf ok i tried bringing back original init again and this time it makes a ton of difference and works much better than default. i'm not sure what was different with my earlier experiment where i saw a slight regression. may try to dissect commits later, for now merged the original mingpt init (following gpt-2 paper) as default. Andrej Karpathy 2023-01-27 17:56:18 +00:00
• 23a0bfac20 try bring back mingpt init init_test Andrej Karpathy 2023-01-27 16:52:18 +00:00
• 3cb3fc059c grad clipping seems to slightly speed up training in the beginning but i can't see a big difference later in the training. it costs non-negligeable compute to clip. adding it for now because it is standard, and i think more necessary as the model becomes larger. practitioners may consider turning it off for minor efficiency gains Andrej Karpathy 2023-01-27 16:45:09 +00:00
• e0c689cf38 allow the prompt to compe from a file Andrej Karpathy 2023-01-25 01:12:43 +00:00
• 21675d7755 allow sample.py to init from a pretrained gpt2 checkpoints as well, in similar style to train.py Andrej Karpathy 2023-01-25 00:55:29 +00:00
• e0e94a1094 use GradScaler in model only if dtype is float16 johnwildauer 2023-01-24 15:53:31 -07:00
• 6c40a08b41 Merge pull request #82 from danielgross/master Andrej 2023-01-22 13:47:32 -08:00
• 2f7fd0ac57 add relative import in shakespeare DG 2023-01-22 12:18:24 -08:00
• bf779456f3 add relative import in shakespeare_char DG 2023-01-22 11:11:25 -08:00
• f9d8020f48 Fix decode fn in shakespeare_char/prepare.py venusatuluri 2023-01-21 06:14:16 +00:00
• 3611338959 Merge pull request #71 from cchan/patch-1 Andrej 2023-01-20 14:38:10 -08:00
• 1f77d03024 make mentions of mps in docs. ty good people in issue #28 Andrej Karpathy 2023-01-20 21:28:20 +00:00
• a6bffeee59 Merge pull request #73 from danielgross/master Andrej 2023-01-20 12:21:33 -08:00
• edb7a7eab0 use relative paths so that running the data prep scripts always create files in local folder, no matter where run from DG 2023-01-20 10:39:45 -08:00
• 67166079c9 Zero-grad more aggressively to save memory Clive Chan 2023-01-19 22:10:44 -08:00
• 2c7806db6e for consistency with previous commit Andrej Karpathy 2023-01-19 23:10:51 +00:00
• c1c20a0311 Merge pull request #57 from ryouze/patch-1 Andrej 2023-01-19 15:08:35 -08:00
• 9e150b808e Merge pull request #66 from PWhiddy/patch-1 Andrej 2023-01-18 22:29:51 -08:00
• ff9085d0bc fix typo ( params -> tokens) Peter Whidden 2023-01-18 21:17:15 -05:00
• 8dd2061e4d fix temperature comment, slightly wrong Andrej Karpathy 2023-01-18 16:10:05 +00:00
• 2b083fbfde the badge is a bit ugly, move it down to troubleshooting section Andrej Karpathy 2023-01-18 03:16:59 +00:00
• aa8e4c2546 screwed up the link, fix Andrej Karpathy 2023-01-18 03:11:31 +00:00
• 6dab32c003 experimenting with badges, and discord link to start specifically. issues sometimes feel a little too heavy Andrej Karpathy 2023-01-18 03:09:42 +00:00
• be571fff2c Improve readability of huge numbers リョウゼ 2023-01-16 22:05:32 +01:00
• 7f74652843 add docs on multinode training to main README too Andrej Karpathy 2023-01-16 17:11:02 +00:00
• 46ce9971df small tweaks to docs and variable names stylistically Andrej Karpathy 2023-01-16 16:56:05 +00:00
• 684800dd87 clarify that these should be run on two separate machines multi_node_ddp Andrej Karpathy 2023-01-16 06:02:46 +00:00
• 9352df23de docs for multinode ddp Andrej Karpathy 2023-01-16 05:57:33 +00:00
• c3dddbff3d get rid of gpu_id, the world is more complicated than that when world_size > 8 Andrej Karpathy 2023-01-16 05:44:50 +00:00
• f5e6ac8b02 local rank -> rank Andrej Karpathy 2023-01-16 05:13:13 +00:00
• d5ee965974 Update README.md MicroPanda123 2023-01-15 20:29:15 +00:00
• cf99914886 add gradient accumulation support to simulate larger batch sizes. ty @VHellendoorn for original PR Andrej Karpathy 2023-01-15 17:49:55 +00:00
• 89da79eee1 add note of caution for the produced warning, investigate later tie_weights Andrej Karpathy 2023-01-14 20:38:22 +00:00
• 7d7ded25ce a bit better settings... for a single gpu at least. these settings would fry a simple cpu though i think Andrej Karpathy 2023-01-14 03:59:53 +00:00
• 91d02510ce fix bug... if topk > vocab_size, torch.topk will throw error Andrej Karpathy 2023-01-14 03:57:00 +00:00
• 57735f532d correctly propagate the vocab_size from the rendered dataset into the model args Andrej Karpathy 2023-01-14 02:26:44 +00:00
• 43b37fd568 reverse the order, making sure that the final layer init is preserved, and becomes the token embedding instead of the other way around. otherwise the loss can be all messed up from a bad init Andrej Karpathy 2023-01-14 02:16:10 +00:00
• 7c8288552b tie the weights of lm_head.weight and transformer.wte.weight, i.e. the last linear layer of decoder and the token embeddings. Andrej Karpathy 2023-01-14 01:00:55 +00:00
• 32b4f08d9d it's true Andrej Karpathy 2023-01-13 23:43:00 +00:00
• 3e0fd42579 more scaling laws, clarification, and add simple interpolation of Approach 2 Andrej Karpathy 2023-01-13 00:57:15 +00:00
• 8f85b83347 inference time mini-optimization low-hanging fruit ty @jxtps for raising: when we are running inference we can apply lm_head on only the very last token Andrej Karpathy 2023-01-12 06:02:50 +00:00
• e21cbf887f meant to set always_save_checkpoint to False instead, so we only write when val improves Andrej Karpathy 2023-01-12 05:47:34 +00:00
• c1ac2d58f1 including transformers as a dependency of the repo as well Andrej Karpathy 2023-01-12 02:42:38 +00:00
• 7f51d17977 add note about windows and pytorch 2.0 and torch compile in general Andrej Karpathy 2023-01-12 02:17:52 +00:00
• bb49751439 oh no nanoGPT is trending quickly explain the character-level functionality I added late last night Andrej Karpathy 2023-01-11 17:11:15 +00:00
• d17350a31d add support for character-level language models, a new character-level shakespeare dataset, a new config file that shows how to train a character-level baby GPT on it, and adjust the sample function to figure out if it should decode with characters or GPT2 bpe tokens. The current implementation is a bit hacky and basically assumes just these two possibilities. In the future we may want to support more general encoders or decoders. Andrej Karpathy 2023-01-11 05:27:19 +00:00
• c2a402f7f7 guess the config from globals() and log all of it with wandb Andrej Karpathy 2023-01-11 01:00:22 +00:00
• 8b2e622b27 adjust the readme to reflect changes in the autocast branch autocast_wip Andrej Karpathy 2023-01-08 19:40:46 +00:00
• b77c2e86d3 copy pasting what seems to work to bench,sample as well. ty @lantiga Andrej Karpathy 2023-01-08 19:32:13 +00:00
• a855d316fd add device and dtype support to train.py args Andrej Karpathy 2023-01-08 19:20:38 +00:00
• e7cd674ce7 Merge pull request #20 from lantiga/wandb-optional-import Andrej 2023-01-08 10:19:40 -08:00
• 09f1f458e8 Move conditional import Luca Antiga 2023-01-08 15:51:50 +01:00
• aba47f0a35 Make wandb import conditioned to wandb_log=True Luca Antiga 2023-01-05 09:09:22 +01:00
• e53b9d28ff ran readme through spellchecker heh Andrej Karpathy 2023-01-08 01:46:54 +00:00
• df3b8a57ab tune the readme with new header image and the loss curve for 124M Andrej Karpathy 2023-01-08 00:41:14 +00:00
• d56bdf05a6 progress! based on chinchilla author correspondence Andrej Karpathy 2023-01-07 02:42:30 +00:00
• 27fc6a4112 small tweaks to notebook Andrej Karpathy 2023-01-06 02:13:04 +00:00
• 69d1a5f1af update scaling laws. basically i can't reproduce any of params, flops, or scaling laws of the Chinchilla paper atm... Andrej Karpathy 2023-01-06 02:01:08 +00:00
• 9629093e53 minor args re-arranging and removing some spurious ones like wandb entity ty @tcapelle Andrej Karpathy 2023-01-05 01:14:02 +00:00
• 529c967a65 Merge pull request #19 from nat/patch-1 Andrej 2023-01-04 16:46:32 -08:00
• d562b3e550 shuttling the poor mans configurator aside into its own file and adding it to all of train,sample,bench. because i am leaving args in globals() so i can avoid having to prepend every single variable with an args., i have to exec the configurator and the optional configs. so we're left with something very gross by standard convention but also quite simple and functional. *ducks* Andrej Karpathy 2023-01-05 00:44:35 +00:00
• 2b9e168736 Strip unwanted prefix from state keys when loading model Nat Friedman 2023-01-04 16:34:00 -08:00
• ab04701f9f mention current 8GPU SOTA and shuffle sections a bit Andrej Karpathy 2023-01-04 18:59:10 +00:00
• 1eefbb2520 Merge pull request #16 from jorahn/patch-1 Andrej 2023-01-04 09:08:50 -08:00
• 26aa5f3ead Update README.md Jonathan Rahn 2023-01-04 10:28:13 +01:00
• c72ecf5d93 add a notebook trying to reproduce chinchilla scaling laws. I can't get the numbers to be exactly right, have to look at more Andrej Karpathy 2023-01-04 00:59:34 +00:00
• 5acba4b005 ty lambda labs Andrej Karpathy 2023-01-03 21:16:07 +00:00
• 97fc42616e adding few more dependencies Andrej Karpathy 2023-01-03 17:54:48 +00:00
• 9f95aca93e better hyperparams for gpt2 124M model on A100 40GB. still uncertain about max_iters especially, and a bit about weight decay, betas Andrej Karpathy 2023-01-03 17:45:49 +00:00
• b45eec3e4b flesh out the remaining TODOs in readme a bit more Andrej Karpathy 2023-01-03 07:41:28 +00:00
• 177d5f7dc5 disabling torch.jit.script here for massive performance boost when using torch.compile, our default. see issue #11. thanks @vgoklani for flagging Andrej Karpathy 2023-01-02 23:05:01 +00:00
• 0a2ea95338 batch file write Laiho 2023-01-02 17:49:21 +02:00
• ea4de192e0 reshuffle args inside sample.py Andrej Karpathy 2023-01-02 02:11:39 +00:00
• ec9b1f8182 add a patch to fix mysterious unwanted prefix in state dict? maybe remove later Andrej Karpathy 2023-01-02 01:25:02 +00:00
• 41184a27f5 rename compile_model to compile, shroter, version 2 stragglers Andrej Karpathy 2023-01-02 01:15:55 +00:00
• 35f51974c4 rename to compile it's shorter Andrej Karpathy 2023-01-02 01:14:46 +00:00
• 2febf4463c candidate changes to apis, have to think through more Andrej Karpathy 2023-01-01 01:29:48 +00:00
• 7c6ea8409e simplify the prepare script a lot, write only using one process, seems sufficient for now. ty @LaihoE for suggestion and @proger for flagging Andrej Karpathy 2022-12-30 22:18:20 +00:00
• d8abd21258 typo fix in readme Andrej Karpathy 2022-12-30 00:07:58 +00:00
• 5a725d9098 add torch.compile by default, shows almost 1.8X improvement in throughput nice Andrej Karpathy 2022-12-30 00:07:13 +00:00
• fb52554ca8 Merge pull request #1 from ankandrew/master Andrej 2022-12-29 13:45:20 -08:00
• 7f0e6d9a71 Frozen GPTConfig ankandrew 2022-12-29 17:07:19 -03:00
• 682a0ac8f1 properly resume training, also loading iter_num and best_val_loss from checkpoints Andrej Karpathy 2022-12-29 18:23:15 +00:00
• f88aa2c2fe add link to mingpt Andrej Karpathy 2022-12-29 17:38:33 +00:00
• f2fc4be69b mention 4gpu loss as well in readme Andrej Karpathy 2022-12-29 17:26:42 +00:00
• fa57d464d7 pull out dtype up top Andrej Karpathy 2022-12-29 05:32:55 +00:00
• e7bac659f5 oops missed one # have to fix Andrej Karpathy 2022-12-29 05:24:14 +00:00