{"id":2046,"date":"2025-11-06T13:04:00","date_gmt":"2025-11-06T13:04:00","guid":{"rendered":"https:\/\/philippeadjiman.com\/blog\/?page_id=2046"},"modified":"2025-11-22T18:33:45","modified_gmt":"2025-11-22T18:33:45","slug":"gpt-from-scratch-series","status":"publish","type":"page","link":"https:\/\/philippeadjiman.com\/blog\/gpt-from-scratch-series\/","title":{"rendered":"GPT From Scratch Series"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Welcome to a new Series of 7 posts, where we\u2019re going to deep dive into one of the most exciting videos from Andrej Karpathy on the topic:&nbsp;<a href=\"https:\/\/www.youtube.com\/watch?v=kCc8FmEb1nY\">Let\u2019s build GPT: from scratch, in code, spelled out.<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n<ul class=\"wp-block-latest-posts__list wp-block-latest-posts\"><li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/10\/24\/gpt-from-scratch-1-intro\/\" aria-label=\"GPT From Scratch #1: Intro\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/10\/gpt-1.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/10\/gpt-1.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/10\/gpt-1.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/10\/gpt-1.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/10\/24\/gpt-from-scratch-1-intro\/\">GPT From Scratch #1: Intro<\/a><div class=\"wp-block-latest-posts__post-excerpt\">You probably use AI, but do you understand it? Get ready to dive into the internals of what started the (gen) AI revolution: GPT. <\/div><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/06\/gpt-from-scratch-2-the-training-set\/\" aria-label=\"GPT From Scratch #2: The Training Set\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-2.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-2.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-2.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-2.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/06\/gpt-from-scratch-2-the-training-set\/\">GPT From Scratch #2: The Training Set<\/a><div class=\"wp-block-latest-posts__post-excerpt\">Don&#8217;t understate the importance of building a proper training set. It is a critical part of the process, and in GPT&#8217;s case, a beautifully cleaver one as well. <\/div><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/10\/gpt-from-scratch-3-the-bigram-model\/\" aria-label=\"GPT From Scratch #3: The Bigram Model\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-3.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-3.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-3.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-3.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/10\/gpt-from-scratch-3-the-bigram-model\/\">GPT From Scratch #3: The Bigram Model<\/a><div class=\"wp-block-latest-posts__post-excerpt\">The simplest model we can use to predict the next character is a Bigram Model. But if implemented as a neural net, the building blocks will stay the same up to GPT. <\/div><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/14\/gpt-from-scratch-4-the-mathematical-trick-behind-self-attention\/\" aria-label=\"GPT From Scratch #4: The Mathematical Trick Behind Self Attention\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-4.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-4.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-4.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-4.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/14\/gpt-from-scratch-4-the-mathematical-trick-behind-self-attention\/\">GPT From Scratch #4: The Mathematical Trick Behind Self Attention<\/a><div class=\"wp-block-latest-posts__post-excerpt\">One simple mathematical trick. The most cleaver matrix multiplication of the gen AI revolution. What enabled ultra fast self attention. <\/div><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/15\/gpt-from-scratch-5-positional-encodings\/\" aria-label=\"GPT From Scratch #5: Positional Encodings\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-5.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-5.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-5.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-5.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/15\/gpt-from-scratch-5-positional-encodings\/\">GPT From Scratch #5: Positional Encodings<\/a><div class=\"wp-block-latest-posts__post-excerpt\">In this post, we&#8217;ll show how to add to the neural net the notion of position of the tokens. Simple but powerful. <\/div><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/19\/gpt-from-scratch-6-coding-self-attention\/\" aria-label=\"GPT From Scratch #6: Coding Self Attention\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-6.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-6.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-6.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-6.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/19\/gpt-from-scratch-6-coding-self-attention\/\">GPT From Scratch #6: Coding Self Attention<\/a><div class=\"wp-block-latest-posts__post-excerpt\">This is where we get to understand the ~20 most important and impactful lines of code which started the gen AI revolution. <\/div><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><a href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/22\/gpt-from-scratch-7-building-a-gpt\/\" aria-label=\"GPT From Scratch #7: Building a GPT\"><img decoding=\"async\" loading=\"lazy\" width=\"150\" height=\"150\" src=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-7.png?resize=150%2C150&amp;ssl=1\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:113px;max-height:113px;\" srcset=\"https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-7.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-7.png?zoom=2&amp;resize=150%2C150&amp;ssl=1 300w, https:\/\/i0.wp.com\/philippeadjiman.com\/blog\/wp-content\/uploads\/2025\/11\/gpt-7.png?zoom=3&amp;resize=150%2C150&amp;ssl=1 450w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/philippeadjiman.com\/blog\/2025\/11\/22\/gpt-from-scratch-7-building-a-gpt\/\">GPT From Scratch #7: Building a GPT<\/a><div class=\"wp-block-latest-posts__post-excerpt\">Self Attention is the heart of Transformers, the T of GPT. But there are few additional critical parts to the transformer architecture that actually made it shine.<\/div><\/li>\n<\/ul>\n\n\n<div style=\"height:53px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">If you don&#8217;t want to miss other posts and series, just subscribe below.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n<style id=\"wpforms-css-vars-2053\">\n\t\t\t\t#wpforms-2053 {\n\t\t\t\t--wpforms-container-padding: 0px;\n--wpforms-container-border-width: 1px;\n--wpforms-container-border-radius: 3px;\n--wpforms-background-color: rgba(0, 0, 0, 0);\n--wpforms-field-size-input-height: 43px;\n--wpforms-field-size-input-spacing: 15px;\n--wpforms-field-size-font-size: 16px;\n--wpforms-field-size-line-height: 19px;\n--wpforms-field-size-padding-h: 14px;\n--wpforms-field-size-checkbox-size: 16px;\n--wpforms-field-size-sublabel-spacing: 5px;\n--wpforms-field-size-icon-size: 1;\n--wpforms-label-size-font-size: 16px;\n--wpforms-label-size-line-height: 19px;\n--wpforms-label-size-sublabel-font-size: 14px;\n--wpforms-label-size-sublabel-line-height: 17px;\n--wpforms-button-size-font-size: 17px;\n--wpforms-button-size-height: 41px;\n--wpforms-button-size-padding-h: 15px;\n--wpforms-button-size-margin-top: 10px;\n--wpforms-container-shadow-size-box-shadow: none;\n\t\t\t}\n\t\t\t<\/style><div class=\"wpforms-container wpforms-container-full wpforms-render-modern\" id=\"wpforms-2053\"><form id=\"wpforms-form-2053\" class=\"wpforms-validate wpforms-form wpforms-ajax-form\" data-formid=\"2053\" method=\"post\" enctype=\"multipart\/form-data\" action=\"\/blog\/wp-json\/wp\/v2\/pages\/2046\" data-token=\"bd4c6a7823e37c44aca96792c8a84c0e\" data-token-time=\"1780636712\"><noscript class=\"wpforms-error-noscript\">Please enable JavaScript in your browser to complete this form.<\/noscript><div id=\"wpforms-error-noscript\" style=\"display: none;\">Please enable JavaScript in your browser to complete this form.<\/div><div class=\"wpforms-field-container\"><div id=\"wpforms-2053-field_1-container\" class=\"wpforms-field wpforms-field-text\" data-field-id=\"1\"><label class=\"wpforms-field-label\" for=\"wpforms-2053-field_1\">Name<\/label><input type=\"text\" id=\"wpforms-2053-field_1\" class=\"wpforms-field-medium\" name=\"wpforms[fields][1]\" placeholder=\"Your Name\" aria-errormessage=\"wpforms-2053-field_1-error\" ><\/div>\t\t<div id=\"wpforms-2053-field_3-container\"\n\t\t\tclass=\"wpforms-field wpforms-field-text\"\n\t\t\tdata-field-type=\"text\"\n\t\t\tdata-field-id=\"3\"\n\t\t\t>\n\t\t\t<label class=\"wpforms-field-label\" for=\"wpforms-2053-field_3\" >Name Email<\/label>\n\t\t\t<input type=\"text\" id=\"wpforms-2053-field_3\" class=\"wpforms-field-medium\" name=\"wpforms[fields][3]\" >\n\t\t<\/div>\n\t\t<div id=\"wpforms-2053-field_2-container\" class=\"wpforms-field wpforms-field-email\" data-field-id=\"2\"><label class=\"wpforms-field-label\" for=\"wpforms-2053-field_2\">Email <span class=\"wpforms-required-label\" aria-hidden=\"true\">*<\/span><\/label><input type=\"email\" id=\"wpforms-2053-field_2\" class=\"wpforms-field-medium wpforms-field-required\" name=\"wpforms[fields][2]\" placeholder=\"Your email\" spellcheck=\"false\" aria-errormessage=\"wpforms-2053-field_2-error\" required><\/div><script>\n\t\t\t\t( function() {\n\t\t\t\t\tconst style = document.createElement( 'style' );\n\t\t\t\t\tstyle.appendChild( document.createTextNode( '#wpforms-2053-field_3-container { position: absolute !important; overflow: hidden !important; display: inline !important; height: 1px !important; width: 1px !important; z-index: -1000 !important; padding: 0 !important; } #wpforms-2053-field_3-container input { visibility: hidden; } #wpforms-conversational-form-page #wpforms-2053-field_3-container label { counter-increment: none; }' ) );\n\t\t\t\t\tdocument.head.appendChild( style );\n\t\t\t\t\tdocument.currentScript?.remove();\n\t\t\t\t} )();\n\t\t\t<\/script><\/div><!-- .wpforms-field-container --><div class=\"wpforms-submit-container\" ><input type=\"hidden\" name=\"wpforms[id]\" value=\"2053\"><input type=\"hidden\" name=\"page_title\" value=\"\"><input type=\"hidden\" name=\"page_url\" value=\"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/pages\/2046\"><input type=\"hidden\" name=\"url_referer\" value=\"\"><button type=\"submit\" name=\"wpforms[submit]\" id=\"wpforms-submit-2053\" class=\"wpforms-submit\" data-alt-text=\"Sending...\" data-submit-text=\"Subscribe\" aria-live=\"assertive\" value=\"wpforms-submit\">Subscribe<\/button><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/philippeadjiman.com\/blog\/wp-content\/plugins\/wpforms-lite\/assets\/images\/submit-spin.svg\" class=\"wpforms-submit-spinner\" style=\"display: none;\" width=\"26\" height=\"26\" alt=\"Loading\"><\/div><\/form><\/div>  <!-- .wpforms-container -->\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Welcome to a new Series of 7 posts, where we\u2019re going to deep dive into one of the most exciting videos from Andrej Karpathy on the topic:&nbsp;Let\u2019s build GPT: from scratch, in code, spelled out. If you don&#8217;t want to miss other posts and series, just subscribe below.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"nf_dc_page":"","footnotes":""},"class_list":["post-2046","page","type-page","status-publish","hentry"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/pages\/2046","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/comments?post=2046"}],"version-history":[{"count":12,"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/pages\/2046\/revisions"}],"predecessor-version":[{"id":2249,"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/pages\/2046\/revisions\/2249"}],"wp:attachment":[{"href":"https:\/\/philippeadjiman.com\/blog\/wp-json\/wp\/v2\/media?parent=2046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}