Iden Tumuhirwe
All writing

July 17, 2026 · 28 min read

An LLM is a file full of numbers, and I mean that literally

AI · LLMs · Machine Learning · JavaScript · Deep Dive

Somebody on my team asked me a question a few weeks ago that I answered badly: what actually happens when you type a prompt into Claude and hit enter? I gave the standard answer, the one everyone gives, "it predicts the next token." Which is true. It's also the kind of true that explains nothing, like answering "how does a car work" with "combustion." I knew the shape of the answer and none of the contents, and it bugged me enough that I went and did something about it.

What kicked me over the edge was CJ's Syntax video where he builds an LLM from scratch, tiny piece by tiny piece, with working code at every step. I thought I'd watch it over lunch and be done. Instead it turned into a proper rabbit hole. The video name-drops a Shannon paper from 1951, so I went and found the PDF. That paper led me to the ELIZA paper from 1966, which led me to an actual ARPANET RFC where two chatbots are connected to each other, which somehow led me to a digitized 1958 New York Times page about a room-sized machine the Navy thought would become conscious. Three or four nights of this, twenty-something tabs, and at some point I realized I was reading the original transformer paper in bed. The wild thing about this field is that the blueprint for a trillion-dollar industry is a fifteen-page PDF anyone can read for free, and almost nobody does.

This post is me climbing back out of the hole and writing the pipeline down properly, start to finish, with enough code that none of it stays magic. If you already train models for a living, nothing here will surprise you. If you're a developer who uses these things daily and quietly hand-waves the middle part like I did, this is for you.

It starts with a party game in 1950

Claude Shannon, the information theory guy, sat down with his wife Betty and played a guessing game. He'd show her a passage of text with the next letter hidden, and she'd guess it, letter by letter, while he tracked how often she was right. What he was actually measuring was how predictable written language is, and the number he published in 1951 is the part that floored me when I read the paper: with enough preceding context, printed English carries roughly one bit of information per letter. A 27-symbol alphabet could carry about 4.7. Everything above that one bit is redundancy. In other words, most of every English sentence is guessable before it arrives.

That's the entire thesis. Language has deep statistical structure, and if you can capture that structure, you can predict what comes next. Everything else in this post, seventy-plus years of research, is machinery for capturing it at scale. An LLM is a statistical model of language. It's Betty's guessing game with 120 billion parameters.

The chatbot costume is older than the intelligence

Worth knowing before we get to the real thing: the chat interface predates anything resembling understanding by about sixty years. ELIZA, built by Joseph Weizenbaum at MIT in 1966, mimicked a Rogerian therapist, and I'd always heard it described as "a pile of if statements." Then I read the actual paper, and it's slightly classier than that: a script of ranked keywords, each with decomposition patterns that tear your sentence apart and reassembly templates that build the reply out of the pieces. The paper's own showpiece: you type "It seems that you hate me," the keyword you fires the pattern (0 YOU 0 ME), and the reassembly rule flings it back as "WHAT MAKES YOU THINK I HATE YOU." Here's that mechanism, compressed into modern JavaScript:

const REFLECT = { i: "you", me: "you", my: "your", am: "are", you: "I" };
const reflect = (s) => s.split(" ").map((w) => REFLECT[w] ?? w).join(" ");

// ranked keywords, decomposition patterns, reassembly templates.
// the whole personality is this table
const SCRIPT = [
  { rank: 10, pattern: /you (.*) me/i, template: "What makes you think I $1 you?" },
  { rank: 5, pattern: /i am (.*)/i, template: "How long have you been $1?" },
  { rank: 3, pattern: /\b(mother|father|family)\b/i, template: "Tell me more about your family." },
  { rank: 0, pattern: /(.*)/, template: "Please go on." }, // the escape hatch
];

function eliza(message) {
  const rule = SCRIPT.filter((r) => r.pattern.test(message))
                     .sort((a, b) => b.rank - a.rank)[0];
  const groups = message.match(rule.pattern);
  return rule.template.replace(/\$(\d)/g, (_, i) => reflect(groups[i]));
}

eliza("it seems that you hate me"); // "What makes you think I hate you?"
eliza("i am sad about my job");     // "How long have you been sad about your job?"

Weizenbaum's own secretary asked him to leave the room so she could talk to it privately. He was so disturbed by how easily people bonded with a rule table that he spent the rest of his career warning about it. And every chatbot for the next half century, PARRY, ALICE with its 41,000 handwritten patterns, SmarterChild on AIM, ran on some elaboration of this idea: every reply anyone ever received had been authored, in advance, by a person.

Notice what ELIZA's "brain" physically is, though, because it sets up everything that follows: it's a table. The program is a dumb engine; the personality lives in a data file of rules. Hold onto that, because the sixty-year jump to an LLM isn't a smarter engine bolted onto a bigger rule table. It's changing what's in the file.

The best detour of the whole rabbit hole lives here, by the way. In 1972 someone connected ELIZA to PARRY, a chatbot built at Stanford to simulate a paranoid patient, and let the therapist bot and the patient bot talk to each other over ARPANET. The transcript is preserved as RFC 439, sitting in the IETF archives between documents about packet switching, and it reads exactly as unhinged as you'd hope. I lost twenty minutes to it and regret nothing.

CJ's video has a framing for this that I've been reusing at work ever since: treat the whole thing as a black box with an input and an output, and then notice you can swap the guts for smarter machinery without the user ever knowing. ELIZA and Claude present the identical interface: text in, text out. Everything that changed in sixty years is inside the box. So let's open it.

A model is a file

There's a line CJ delivers in the video almost as a throwaway, and it's the single sentence that demystified the most for me: a model is literally just a file with weight values inside of it. I didn't fully believe how literally he meant it until I made one. Training is the process that produces the file; inference, the thing that happens when you hit enter, is arithmetic done with the file's contents.

The numbers are the weights of a neural network: layers of nodes where each connection has a number controlling how much one node influences the next. You train it by making a prediction, measuring how wrong it is, and nudging every weight in the direction that makes it slightly less wrong, then repeating that a few billion times. That's backpropagation, published by Rumelhart, Hinton and Williams in 1986, and every model you've ever talked to was trained with some variant of it.

The history behind this is worth the detour if you have an evening. McCulloch and Pitts proposed neurons-as-logic-gates in 1943 as a pure thought experiment; fifteen years later Frank Rosenblatt actually built one, a room-sized machine of wires and photocells called the perceptron, and the New York Times reported it as "the embryo of an electronic computer" that would one day be conscious of its own existence. That page from July 1958 is digitized, and reading the breathless coverage while sitting next to a laptop running a local model is a genuinely strange feeling. Then Minsky and Papert proved in 1969 that a single-layer perceptron can't even solve XOR (output 1 when exactly one of two inputs is 1, a problem you solve by glancing at it) and iced the whole field for years.

So, to convince myself of the file thing, I trained a two-layer network on XOR in a scratch file and printed the result. This is the entire model. Not an excerpt. The whole thing:

{
  "hiddenWeights": [[-5.13, -6.45], [-5.13, -6.46]],
  "hiddenBias": [7.65, 2.67],
  "outputWeights": [10.47, -10.69],
  "outputBias": -4.98
}

Nine numbers. Feed (0,1) through them, sigmoid at each layer, and out comes 0.989. Feed (1,1) and out comes 0.014. The problem that embarrassed the perceptron and froze AI research for a decade dissolves into nine floats in a JSON file, and nothing about that picture changes with scale. GPT-OSS 120B is this exact artifact with 120 billion numbers instead of nine. When Claude answers you, a server somewhere is doing matrix multiplication against a very large version of the blob above. I find that genuinely comforting.

Tokenization, or why the model has never seen a word

A neural network takes numbers in, not text, so your prompt has to be chopped into units first. Not characters, not words, something in between called tokens. Common English sails through whole, one token per word with the leading space attached; my surname gets shredded into fragments no dictionary would recognize, which is also why models used to butcher it back to me in creative new spellings. From the model's side of the glass there are no words at all, only these units.

The splitting algorithm, byte pair encoding, has a great origin story: Philip Gage invented it in 1994 as a data compression trick, nothing to do with language at all. Find the most frequent pair of symbols in your data, merge it into a new symbol, repeat. Twenty-one years later, Sennrich, Haddow and Birch at Edinburgh realized it was the perfect way to grow a vocabulary from training data instead of defining one up front. Common words survive whole, rare words get split into reusable pieces, and one algorithm handles English, Japanese, Python and emoji with zero language-specific rules.

The core of it fits in a loop:

let vocab = splitIntoWords(corpus); // merges stay inside word boundaries, so split first

for (let i = 0; i < maxMerges; i++) {
  const pairCounts = countAdjacentPairs(vocab); // weighted by word frequency
  const best = mostFrequent(pairCounts);
  if (!best) break;

  vocab = mergeEverywhere(vocab, best); // "t","h" -> "th", then "th","e" -> "the"
}

Run it on a real corpus and frequency decides everything: words the data uses constantly earn their own tokens, and whatever's rare (jargon, code identifiers, Ugandan surnames) gets assembled from spare parts at runtime. The maxMerges budget decides your vocabulary size, and every token gets an ID, its index in that vocabulary.

This isn't a preprocessing footnote, by the way. Tokens are the unit of everything you pay for. API pricing is per token. The context window is measured in tokens. When your long conversation starts degrading, that's a token budget being blown. The billing model of the entire AI industry rests on a 1994 compression trick.

Embeddings: meaning as coordinates

Token IDs are arbitrary. Token 4821 isn't "more" than token 302. So the next step is mapping each token to a vector, a long list of numbers that positions it as a point in high-dimensional space, where the geometry encodes meaning: similar words end up near each other.

The idea is old. Frege in 1884: never ask for the meaning of a word in isolation. The linguist J.R. Firth in 1957 put it better: "You shall know a word by the company it keeps." That one sentence is the thesis behind every embedding model ever built. In 2013 Mikolov's team at Google turned it into code with word2vec, training a network to predict words from their neighbors, and found structure nobody put there. Take the vector for king, subtract man, add woman, and the nearest vector in the vocabulary is queen. Nobody taught it gender or royalty. The concepts fell out of the statistics of which words appear near which.

"Nearest" is cosine similarity, the angle between two vectors, and it's a dozen lines:

function cosineSimilarity(a, b) {
  let dot = 0, magA = 0, magB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

const result = add(sub(vec("king"), vec("man")), vec("woman"));
nearestInVocab(result); // "queen"

Training is a push-pull: words that appear together in the corpus get their vectors nudged closer, words that never co-occur get pushed apart, over and over until the geometry settles. word2vec used 300 dimensions; GPT-3's largest model uses 12,288. Modern models embed whole passages rather than single words, but it's the same trick scaled up. Firth's aphorism, implemented as linear algebra.

The transformer, briefly and honestly

So we have tokens, and each token has a vector. The thing that processes them is a transformer, from the 2017 Google paper "Attention Is All You Need," fifteen pages, cited over a hundred thousand times, all eight authors long gone from Google and several of them running billion-dollar companies. Every major model you use is this one architecture scaled up.

Before it, translation models read text one word at a time with a running memory, which was slow and forgot things. Attention, originally a bolt-on from a 2014 Bahdanau paper, let the model look back at the whole input and focus on what matters right now. The transformer's move was to throw away the sequential reading entirely and keep only the attention.

Here's the honest version of how attention works, because this is where most explanations either lie or drown you. For each token, the model derives three vectors from its embedding, and the mental model that finally made it click for me is a crowded room. The query is the question a token shouts into the room. The key is the sign every other token is holding up. The value is the note a token hands over when its sign matches your question. Every query gets scored against every key, and each token's output becomes a blend of the handed-over notes, weighted by how well the signs matched. Out comes a new vector per token, same shape as before, but now soaked in context from the whole sentence.

The concrete payoff, with a word that isn't the textbook example: "crane." In the crane lifted the beam onto the roof, the token for crane comes out of attention pulled toward lifted and beam. Construction equipment. In the crane stood in the shallows hunting fish, the same token comes out pulled toward shallows and fish. Bird. Identical token in, different vector out, the ambiguity resolved by whatever else is in the room. That trick is the thing every other part of the architecture exists to enable.

This runs many times in parallel with different learned weights (multi-head attention), and the heads specialize entirely on their own. This part sent me down a side tunnel of the rabbit hole: Anthropic's interpretability team has caught individual heads doing eerily specific jobs, most famously "induction heads," which scan backward for the last time your current token appeared and vote to copy whatever followed it: a find-previous-occurrence-and-continue subroutine that no one wrote, emerging from random initialization because it happens to pay during training. Then each token's vector passes through a small feed-forward network on its own, and that pair, attention plus feed-forward, is one block; modern models stack dozens to hundreds of them. At the hardware level the entire thing is matrix multiplication. There is no if statement in there that knows grammar. Every capability is an arrangement of numbers that training left behind, and every one of those numbers started life as noise.

CJ trained a 52,000-parameter, six-layer version of this on thirty children's stories, on a MacBook CPU, in about ninety minutes, and it will continue "Once upon a time" plausibly. For scale, that's roughly a millionth of a frontier model. Same recipe, different amount of soup.

The output is not a token, it's a bet

Here's the detail that changed how I think about these things more than any other. The transformer does not output a token. It outputs a score for every token in its vocabulary, raw numbers called logits, which get squashed by softmax into a probability distribution that sums to 1. Feed it "in 1876 bell invented the" and it doesn't answer "telephone". It answers something like: 61% " telephone", 24% " telegraph", 4% " first", and a long, thin tail over every other token it knows, most of them absurd. The wrong answers never go away. They just get outvoted.

Then something has to pick, and the parameters you've seen in every API doc, temperature and top_p, are just knobs on that picker. Temperature rescales the logits before softmax, low values sharpen the distribution toward the favorite, high values flatten it toward chaos. Top-p cuts the candidate list down to the smallest set of tokens covering that much probability mass. Temperature 0 collapses everything to "always take the top token," which is why it gives you (nearly) deterministic output. That's the entire mystery of why the same prompt gives different answers: the model's belief is fixed by its weights, and the dice are thrown at the last step.

And it produces one token. To get a paragraph, the picked token is appended to the input and the whole pipeline runs again, new attention, new distribution, new pick, append, repeat. Autoregressive generation. Sit with what that implies: there is no outline, no draft, no destination. Nothing anywhere in the system commits to how a sentence ends before it begins. Every coherent essay you've watched stream out of a model assembled itself one blind step at a time, which I still find slightly unsettling to think about while it's happening.

Fine, I'll build one

At this point in the rabbit hole I stopped taking notes and opened an editor, because the whole pipeline above fits in one file if you're willing to make it tiny. So here's my own language model, in vanilla JavaScript, no dependencies, small enough to read in one sitting. One honest simplification up front: I swapped the transformer stack for a single linear layer, because hand-writing backprop through attention triples the code and teaches you nothing new. Everything else is the real pipeline: tokens, embeddings, logits, softmax, temperature, top-p, the autoregressive loop. CJ's video has the full transformer version if you want the missing middle.

For training data I wrote seventeen one-line history facts, every one shaped the same way: a year, then what happened. I picked history on purpose. A language model is a mirror of whatever you feed it, and I wanted to see what this one would do to facts. Char-level tokenizer, so the whole vocabulary fits in your head:

const corpus =
  "in 1440 gutenberg invented the printing press. " +
  "in 1876 bell invented the telephone. " +
  "in 1903 the wright brothers flew the first plane. " +
  "in 1957 the soviets launched sputnik. " +
  "in 1969 the first humans walked on the moon. " +
  "in 2007 the iphone launched. "; // + a dozen more like these

const vocab = [...new Set(corpus)].sort();
const stoi = new Map(vocab.map((c, i) => [c, i]));
const encode = (s) => [...s].map((c) => stoi.get(c));
const decode = (ids) => ids.map((i) => vocab[i]).join("");
const V = vocab.length;

The model is an embedding per token and one weight matrix. That's the entire file-of-numbers. Context is the last K tokens, each mapped to a D-dimensional vector, concatenated, so position is baked in by which slot your embedding lands in:

const K = 8, D = 16;
const randn = () => (Math.random() * 2 - 1) * 0.1;
const emb = Array.from({ length: V }, () => Float64Array.from({ length: D }, randn));
const W = Array.from({ length: K * D }, () => Float64Array.from({ length: V }, randn));

// forward pass: context ids in, one logit per vocabulary token out
function logitsFor(ctx) {
  const logits = new Float64Array(V);
  for (let k = 0; k < K; k++)
    for (let d = 0; d < D; d++) {
      const x = emb[ctx[k]][d];
      const row = W[k * D + d];
      for (let v = 0; v < V; v++) logits[v] += x * row[v];
    }
  return logits;
}

// squash logits into a probability distribution that sums to 1
const softmax = (logits) => {
  const max = Math.max(...logits);
  const exps = logits.map((l) => Math.exp(l - max));
  const sum = exps.reduce((a, b) => a + b);
  return exps.map((e) => e / sum);
};

Training is the nudge-every-weight loop I described back in the file-of-numbers section, written out for real: grab a random window, predict the next character, measure the surprise, push every weight against its gradient. The one line of actual calculus is a gift from the math gods. The gradient of softmax plus cross-entropy is just probability minus truth:

const data = encode(corpus);
const LR = 0.02, STEPS = 300000;
for (let step = 0; step < STEPS; step++) {
  const i = K + Math.floor(Math.random() * (data.length - K - 1));
  const ctx = data.slice(i - K, i);
  const target = data[i];

  const probs = softmax([...logitsFor(ctx)]);
  const dLogits = probs.map((p, v) => p - (v === target ? 1 : 0));

  for (let k = 0; k < K; k++)
    for (let d = 0; d < D; d++) {
      const x = emb[ctx[k]][d];
      const row = W[k * D + d];
      let dx = 0;
      for (let v = 0; v < V; v++) {
        dx += row[v] * dLogits[v];
        row[v] -= LR * x * dLogits[v];
      }
      emb[ctx[k]][d] -= LR * dx;
    }
}

And then the sampler and the loop, which is where the temperature and top-p knobs from earlier become concrete:

function sampleToken(logits, temperature = 0.7, topP = 0.95) {
  const probs = softmax(logits.map((l) => l / temperature));
  const sorted = probs.map((p, i) => [p, i]).sort((a, b) => b[0] - a[0]);
  let cum = 0;
  const kept = [];
  for (const [p, i] of sorted) {
    kept.push([p, i]);
    if ((cum += p) >= topP) break;
  }
  let r = Math.random() * cum;
  for (const [p, i] of kept) if ((r -= p) <= 0) return i;
  return kept[0][1];
}

function generate(prompt, count = 250, temperature = 0.7) {
  const ids = encode(prompt.padStart(K, " ")); // context must be at least K long
  for (let i = 0; i < count; i++) {
    const ctx = ids.slice(-K);
    ids.push(sampleToken([...logitsFor(ctx)], temperature));
  }
  return decode(ids);
}

(That padStart earned its comment. My first run crashed on a prompt shorter than eight characters, because the model demands a full window of context. Even a model this small already has a context-length footgun.)

That's the whole thing, about eighty lines with the corpus. It trains in two seconds on my laptop. Here's actual output, not cleaned up:

--- temperature 0.7 ---
in 1969 the first car. in 1989 the berlin wall fell. in 1991 the
web went public. in 2007 the iphone launched sputnik. in 1969 the
first car. in 1989 the berlin wall fell. in 1991 the web went
public. in 2007 the iphone launched sputnik. in 1969 the first ca

--- temperature 2.0 ---
in 1969 the aovepton. tn 1776 americhopeolaslea . in 2876d wonthe
smone. in 1957 the lumins disco fers flemini d llumaas. in 1492
gnch ruvo1sovirlo . on mi8am. wo reveluonveasallea oont oo mobem
inton eplans laumaon. in 1936 furew ineplbusco5umamimsen wrle s

Read the temperature 0.7 output slowly, because my eighty-line model just did the thing everyone writes think pieces about. "In 2007 the iphone launched sputnik." "In 1969 the first car." Grammatically clean, structurally perfect, historically false. It stitched the front of one memorized fact onto the back of another, and nothing inside it can know that, because there is no fact-checker in the machine. There never was. There's only "what characters tend to follow these characters," and sputnik follows launched in the training data. That's a hallucination, the same species you get from a frontier model citing a court case that doesn't exist, grown in a petri dish. Then it falls into a four-fact repetition loop, the no-plan problem made visible: each prediction slides toward the strongest nearby pattern, and once the loop closes, nothing upstream exists to notice or care. At temperature 2.0 the distribution flattens until it invents the year 2876. Same weights in both runs. The only thing that changed is how hard we shook the dice.

Scale this up, with a real BPE vocabulary instead of characters, attention instead of my one matrix, a hundred layers, the written internet instead of seventeen lines of history, and the failure modes stop being cute and start showing up in legal filings. But it is the same machine. That's what the rabbit hole actually earned me: not trivia about 1958, but the ability to look at a frontier model's confident nonsense and recognize my eighty-line model's behavior inside it.

The model has no memory. The transcript is the memory.

One more load-bearing fact. When you send a message in an ongoing chat, the model does not "remember" the conversation. The provider sends the entire transcript, a system prompt you never see, every message you've exchanged, then your new one, through the pipeline on every single turn. The model is stateless between requests. The conversation history is the memory, and it all has to fit in the context window, the maximum token count the model can attend over.

That single fact explains most of the weird behavior you've noticed. Long conversations degrade because attention is being spread over more and more tokens and your instructions from forty messages ago are competing with everything since. "Context window full" isn't an arbitrary limit, it's the actual boundary of what the math can see. And when a coding agent "forgets" what you told it an hour ago, nothing was forgotten, something was crowded out.

Tool calling, while we're deflating things: the model cannot search the web or run your code. It's fine-tuned to emit a JSON blob that says "call this tool with these arguments," and the harness around it, the editor, the CLI, the desktop app, actually executes the call and pastes the result back into the context for the next turn. The model only ever produces tokens. The hands belong to the harness.

From autocomplete to assistant

Everything above describes pre-training: shoveling a staggering amount of text, Common Crawl's petabytes of archived web, Wikipedia, GitHub, books, papers, plus copyrighted material the labs are varying degrees of cagey about, through the training loop until the weights capture the statistics of human writing. What comes out the other side is autocomplete. Planetary-scale, unnervingly good autocomplete, but autocomplete: ask the base model a question and it's as likely to continue with four more questions as to answer yours, because a list of questions is a statistically respectable way for text like that to continue.

Two more phases turn that into something you'd recognize. Fine-tuning re-trains it on curated question-answer pairs shaped like the assistant you want, so "answer the question" becomes the likely continuation instead of "add more questions." Then RLHF, reinforcement learning from human feedback, has actual paid humans rank the model's outputs, good responses up, bad ones down, and nudges the weights toward what ranked well. Worth pausing on: what counts as "good" there is decided entirely by the company doing the training and the judgment of whoever they hired. The base model learns language from all of us; the personality gets installed by a much smaller room.

Where I landed

The full pipeline, one breath: your prompt gets split into tokens by a compression algorithm from 1994, each token becomes a point in a space where distance means similarity, attention lets every token reshape every other based on context, stacked blocks of matrix multiplication refine that into a probability distribution over the entire vocabulary, dice weighted by temperature pick one token, and the loop runs again with that token appended, until a stop token wins the draw. Streamed to your screen as it goes.

Having traced all of it, I hold two opinions in tension. The first: the "it's just autocomplete" crowd is more right than the marketing wants to admit. There's no comprehension module in there, no facts database, no reasoning engine you could point at. It is the statistics of language, captured at a scale Shannon could not have imagined, and the fact that fluent paragraphs and working code fall out of next-token prediction alone says something profound about how much of what we call thinking is encoded in the patterns of what we write.

Second, "just" is doing criminally heavy lifting in that sentence. Nobody designed the attention heads that track pronouns. Nobody put king − man + woman = queen into word2vec. These structures emerged because gradient descent found them useful. Dismissing them because they run on matrix multiplication feels like dismissing brains because they run on chemistry. CJ ends his video by suggesting "alien intelligence" fits better than "artificial intelligence," and it's grown on me. These systems train only on text, a lossy export of human thought rather than the thought itself. Whatever they're doing, it isn't a copy of what our brains do. It's a different thing that happens to be startlingly good at wearing our language.

What I'd push on you is this: the whole stack is public, and the rabbit hole is right there. The papers are free, the RFC is free, the 1958 newspaper page is free, open-weight models exist, and the eighty lines above will train on your laptop before you finish reading this sentence. I went in expecting to skim one video and came out four nights later having read a Shannon paper, a therapy transcript between two chatbots, and the fifteen pages that built the current world, with a language model of my own cheerfully rewriting history. The less of this that stays magic to you, the better your instincts get about what these tools will and won't do, why your context matters more than your prompt wording, and why temperature 0 still occasionally surprises you. It's the best-documented revolution in the history of the industry. Might as well read the documentation.

References

These are, more or less, the tabs I had open by the end. The entry point was CJ's video on the Syntax channel where he builds an LLM from scratch. The code for every demo is linked in his description, and it's the best on-ramp into this hole I've found. Everything below is where the hole led:

The statistical foundation

The chatbot prehistory

Neural networks

Tokens and embeddings

Transformers

Training data and alignment

The origin myth, and what might come next

  • The Dartmouth proposal (1955), where "artificial intelligence" was coined, by people who thought one summer would do it
  • The transformer isn't necessarily the last word: Mamba (2023), xLSTM (2024), Jamba (2024), and LeCun's JEPA (2022)