{"id":2000,"date":"2025-10-24T11:13:11","date_gmt":"2025-10-24T11:13:11","guid":{"rendered":"https:\/\/philippeadjiman.com\/blog\/?p=2000"},"modified":"2025-11-26T12:05:19","modified_gmt":"2025-11-26T12:05:19","slug":"gpt-from-scratch-1-intro","status":"publish","type":"post","link":"https:\/\/philippeadjiman.com\/blog\/2025\/10\/24\/gpt-from-scratch-1-intro\/","title":{"rendered":"GPT From Scratch #1: Intro"},"content":{"rendered":"\n

<\/p>\n\n\n\n

Since the (generative) AI revolution started, it seems like we\u2019re observing one breakthrough every 2 weeks on average, and sometimes it can feel overwhelming. Rather than shallowly chasing every breakthrough, I believe it is critical to first start by getting a deep understanding of what started it all: GPT.<\/p>\n\n\n\n

Welcome to a new Series of 7 posts<\/a>, where we\u2019re going to deep dive into one of the most exciting videos from Andrej Karpathy on the topic: Let’s build GPT: from scratch, in code, spelled out.<\/a><\/p>\n\n\n\n

A Note on Motivation and Inspiration<\/h1>\n\n\n\n

Before we begin this 7-part journey, I want to set the stage. This entire series is a deep dive directly inspired by and based on Andrej Karpathy’s phenomenal <\/strong>video mentioned above, <\/strong>which in my opinion is by far the best resource on the internet to understand GPT. <\/p>\n\n\n\n

As a fellow PhD in AI, my own learning process has always been to ‘teach what I learn.’ I created this series as a way to meticulously deconstruct and document every step from that video, solidifying my own understanding in the process. <\/p>\n\n\n\n

While heavily guided by the video, I’ve invested significant time in structuring the content into clear topics, creating custom illustrations, and adding in-depth explanations to unpack the ‘why’ behind the ‘how.’ My hope is that this series serves as a valuable resource for those who learn best by reading, or who need a quick, searchable reference to complement the video format.<\/p>\n\n\n\n

A final note: true understanding comes from doing<\/em>. So I encourage you to use this, but then challenge yourself to e.g. reproduce Karpathy\u2019s code entirely on your own. <\/p>\n\n\n\n

What is GPT?<\/h1>\n\n\n\n

GPT stands for Generative Pretrained Transformer.<\/p>\n\n\n\n

Generative<\/strong>: because it is a model that generates new content, most commonly words or more generally tokens. You give it some initial input (a prompt) and it generates what is most likely to follow from it.<\/p>\n\n\n\n

Pretrained<\/strong>: because of the foundational training process the model undergoes before it is used.<\/p>\n\n\n\n

Transformer<\/strong>: because it is based on the Transformer neural net architecture, introduced in the now famous attention is all you need<\/a> paper.  <\/p>\n\n\n\n

Once you have such a model, building a chat bot like Gemini or ChatGPT requires a few more important steps, but GPT is the foundational part that enables it and our primary focus in this series.<\/p>\n\n\n\n

Starting from the end<\/h1>\n\n\n\n

In some tense TV shows or movies, it sometimes starts by showing the final scene, without context. It looks great, but we have no clue about what is going on. And then the show starts all from the beginning, walking us slowly but surely through that final scene again, and that time, we understand it all.<\/p>\n\n\n\n

This is what we\u2019ll do in that series. We\u2019ll just look at the end result, the heart of the beautiful and minimalist implementation of GPT by Karpathy. The core components basically looks like this (don\u2019t freak out just yet, it is expected if you don\u2019t have clue yet of what this code is doing):<\/p>\n\n\n\n

The init and forward pass of the model. The key magical component in that code is the \u201cblock<\/strong>\u201d variable, which includes the implementation\u00a0of the transformer (and self attention).<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

A self-attention head<\/strong> (which is the core component of a Transformer <\/strong>that we\u2019ll discuss later) :<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

And then, adding some key ingredient of the Transformer architecture, forming a Block<\/strong> (that is initialized in the first snippet).<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

From there, to generate text, you just do:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

That\u2019s it. <\/p>\n\n\n\n

Not even 100 lines of code. <\/p>\n\n\n\n

Knowing that this is what enabled the generative AI revolution, you\u2019re probably staring at those like:<\/p>\n\n\n\n

\"Dog<\/figure>\n\n\n\n

But no worries, after this series of posts, this code will hopefully look much clearer and intuitive to you.<\/p>\n\n\n\n

The journey to understand it all<\/h2>\n\n\n\n

To follow along, you\u2019ll just need a basic understanding of python, also some basics of tensors and PyTorch neural networks, and from time to time, it will be required to review one of the posts from my Deep Learning Gymnastic<\/a> Series (i\u2019ll point them out each time it is relevant).<\/p>\n\n\n\n

Here is the plan of the posts we\u2019ll study together in that series:<\/p>\n\n\n\n