The animated viewer could not connect. You can still read every note and slide reveal here.
Step 1 of 1
Explore the slide at your own pace. The notes explain the complete idea.
01 Origins & history
Demystifying AI
Understanding how AI is built gives us a clearer starting point for deciding how to live and work with it.
In this first lecture of In Dialogue with AI, I invite you to look inside the systems that now speak, write, reason, and act alongside us. These notes accompany the lecture I gave at the University of Portland on September 16, 2026.
We’ll move from the history of these ideas to simple learning exercises, then to the larger machinery surrounding a modern language model. Finally, we’ll ask what the resulting behavior suggests about cognition and our relationship with generative AI. You don’t need a technical background to follow the argument. I want to make the mechanisms understandable enough for us to ask better questions about their capabilities, limitations, and human purpose.
When you encounter a claim about AI, I encourage you to follow it back to the technical papers and documentation. I want to understand the mechanisms and decisions behind a result, beyond the alarming or impressive headline. That’s why I’ve included sources throughout these notes. You can explore the work for yourself and see where I’m interpreting a result and what the study directly establishes.
I want this lecture and the whole series to leave you with less fear and a greater sense that we can get involved and help determine how this turns out.
I promised myself I would start with these words: do not be afraid. We hear so many claims about AI, and I know those words can draw a few uncomfortable chuckles. But this is a big part of why I wanted to give these lectures.
If nothing else happens over the course of these six lectures, I want you to walk away feeling less afraid and more able to say: we can handle this. We should be involved. We can help determine how this turns out. That is the spirit I want to carry through everything we explore together.
In the slide. I start with a simple statement: do not be afraid.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 2:24–3:04My opening promise: less fear, more participation, and a sense that we can shape what happens next.
I’m a Mr. Rogers sort of guy. I believe there are lots of people who want to help and who are willing to do what needs to be done to prevent harm. When we talk about the dangers of these systems, I don’t want us to stop researching them. I want us to keep working toward safer forms of AI and take responsibility for what we choose to do with them.
I grew up programming computers and imagining a world where intelligent technology could help us do incredible things.
I started programming in second grade and took to it like a fish to water. These early computers, the VIC-20, Atari, Commodore 64, Amiga, and TRS-80, bring me right back to that excitement. When people in the room recognized them, I felt a little less old. These were things I could explore, program, and make do something new.
I bought into the whole vision: intelligent spaceships we could talk to, androids we could hang out with, and robots that could help us fight the Empire. And if we couldn’t have all that, at least we could have a car that talked to us. We could all be David Hasselhoff. Underneath the joke was something I took seriously, even in grade school: I wanted to help build a world filled with intelligent technology that would let us do incredible and wonderful things.
In the slide. I put the computers I grew up with alongside the fictional companions that helped me imagine what we might build.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 3:08–4:40How I started programming and came to imagine a world filled with helpful intelligence.
A closer look
01
A future I wanted to help build
This has been a passion of mine since grade school. As I got older, I didn’t think I would live to see the kind of technology we are seeing now. That childhood hope is still with me: intelligence woven into everyday life, helping us do things we couldn’t do before.
I’ve spent my career studying, building, and leading teams around technology, with one question running through it all: how does this help people?
In 1991, I ended up working at the National Science Foundation, where I met Paul Werbos. Through Paul, I learned to use neural networks for forecasting, optimization, and control. By 1993, I was applying those ideas to airlines, and around 2001 I had the chance to continue my research in AI, neural networks, and neuroscience. I was especially interested in how what we were learning about the brain connected with reinforcement learning. The photograph of Paul and me was taken on a visit just a few months before this lecture. Time has changed us quite a bit.
Along the way, I’ve also been building things: large information systems at Booz Allen, software and hardware systems at many scales, and teams to make them work. That practical work has continued from 2001 to the present, including my more recent return to AI through work with knowledge graphs at Nike. Studying these ideas and putting them to work have always belonged together for me. Paul gave me a principle that I’ve carried through all of it: technical brilliance has to serve humanity.
In the slide. My early research and my later systems-building are parts of the same story. The recent photograph with Paul makes that continuity personal.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 4:40–6:56My work with Paul Werbos, the practical uses of neural networks, and the commitment to service I carried forward.
A closer look
01
The research thread
With Paul, I learned adaptive dynamic programming and neurocontrol: ways of using learning systems to make decisions and control processes. I went on to use neural networks for airline forecasting and later explored the relationship between spike-timing plasticity in the brain and reinforcement learning. That connection between biological intelligence and learning machines has stayed with me throughout my work.
At Booz Allen, I worked on knowledge models and large information systems. Over the years, I’ve led teams and built systems of all sorts of sizes and scales, including more recent work with knowledge graphs and graph networks in production. I bring that practical experience into this lecture because making a technique work is only part of the job. We also have to build something people can use.
We worked on making engines more fuel efficient, on flight-control systems that could adapt when something went wrong, and on economic forecasting. In the private sector, I did modeling and forecasting for airlines, insurers, and financial businesses. One very concrete example was predicting demand to help set airline ticket prices. When prices started changing daily, I was part of that, and I joked in the lecture that some people still hate me for it. I’m describing the research and applications we pursued here, including work that was still experimental.
One thing Paul taught me has stayed with me: brilliance isn’t worth much if the technology we build doesn’t seek to serve humanity. I’ve held on to that through decades of building software, hardware, and teams. It is also why I want to ask more than what AI can do. I want to ask whose needs we are using it to meet.
My account of my research and professional history, including NSF, BehavHeuristics, Booz Allen, the NW Computational Intelligence Lab, and later production work. The Road section gives the dated milestones.
Read the slide text and reveals
Step 1 of 2
The open / A lifelong question
The brain. The mind. The world.
1991
Learning with Paul Werbos National Science Foundation
1993
Neural networks, in practice Airline forecasting and optimization
2001
Brain meets machine Neuroscience and reinforcement learning
Paul Werbos & Roberto Santiago
Paul Werbos 1991
Step 2 of 2
Ideas made useful.
Booz Allen
Large information systems
2001–present
Leading teams. Building systems. At many scales
Nike
A return to AI Knowledge graphs in practice
How can technology help us solve human problems?
This step replaces or removes earlier wording.
05 Origins & history
The photograph
When I think about AI, I think about these children. I want us to ask whose lives our work is helping.
You may have noticed that I don’t wear shoes. That began as a spiritual practice during Lent about fifteen years before this lecture. I had been asked to give children a presentation about poverty around the world. Reading about children dying from preventable causes, while looking at my own life and remembering Paul’s words about serving humanity, made me feel that I needed to live differently. Going barefoot became a way to express that solidarity.
This photograph hangs in our home. These are children my wife, Anne, met during her travels in Africa. They’re grown-ups now, but we remember them all the time. At one point I was ready to leave technology completely. This journey is part of why I’m here talking with you, and why I think AI is as much about these children as it is about the technology itself.
I have a math degree, and I love mathematics. I can be excited by what AI is accomplishing there and still ask: could we bring comparable money, computing, and ingenuity to making sure children are fed and cared for? I want us to bring AI down to earth, understand what we have, and see where we have the power to shape what happens with it.
In the slide. This is a photograph from our home, of children Anne met in Africa.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 6:56–9:16Why I go barefoot, the children in our photograph, and the human needs I want technology to serve.
A closer look
01
Child survival then and now
To put this concern in numbers, UNICEF’s The State of the World’s Children 2008: Child Survival reported close to 9.7 million deaths among children under five in 2006, more than 26,000 a day, mostly from preventable causes. WHO’s May 2026 fact sheet puts the 2024 total at 4.9 million, about 9.3 children per minute when averaged over that leap year. I want to be precise about what that number means: it counts deaths from all causes. The accompanying UN release says most could be prevented through established, low-cost interventions and access to quality health care. We have made substantial progress, and the continuing scale of the need is still staggering.
What troubles me is that children still die from illnesses we know how to prevent and treat, even when some essential treatments cost only cents. WHO’s 2024 diarrhea fact sheet reports about 444,000 deaths annually among children under five and describes oral rehydration salts as costing a few cents per treatment. UNICEF’s pneumonia page reports more than 700,000 under-five deaths annually and describes antibiotic treatment for appropriate respiratory infections as typically costing less than US$0.50 for a full course. When I talk about those low prices, I mean the treatments themselves. Getting effective care to a child also requires timely diagnosis, delivery, clean water, nutrition, and sometimes additional treatment for severe illness. That is the practical work I want us to take seriously.
I love seeing AI tackle ambitious mathematical problems. I also want us to bring that ambition to figuring out how children can be fed and cared for. What problem are we choosing to solve? Who will benefit? What are we willing to commit to making that benefit real? Those questions are part of what I mean when I ask what we can do with this technology.
My mathematics example is about the scale of effort we can bring to a problem. OpenAI’s September 8, 2026 announcement reports about 10,000 concurrent agents in the group that produced its proposed Navier–Stokes result, which concerns a fluid subject to smooth external forcing. It discloses 130 billion output tokens for this problem and 300 billion across all attempted problems. At the standard Astra output prices listed on October 16 ($50–$75 per million tokens, depending on context length), those volumes correspond to $6.5–$9.75 million and $15–$22.5 million respectively. I’m using these as illustrative retail-price equivalents for output alone. The research used a different internal model; these figures do not establish its actual operating costs or include inputs, training, staffing, or other project costs.
After the lecture: hundreds of mathematical manuscripts
Since I gave this lecture, OpenAI’s October 6, 2026 release has added another example of the scale of AI mathematics. As checked on October 16, its catalog contains 719 manuscripts in 372 related families, with results at different stages of verification. I’m not presenting that count as 719 independently verified discoveries. OpenAI describes average compute equivalent to about three hours of ChatGPT Pro thinking per result but gives no dollar total in this disclosure. I include this as a later development, following the September 16 lecture.
Printed page 1 reports the historical 2006 estimate and more than 26,000 daily under-five deaths. Provides context for the scale of preventable suffering; it does not establish a universal dollar cost per life saved.
Updated May 1, 2026. Reports 4.9 million under-five deaths in 2024. The annotation’s 9.3 per minute is 4,900,000 divided by the leap year’s 527,040 minutes. This total includes all causes, not only lack of food or medicine.
UNICEF, WHO and UN IGME partners · 2026 · Joint report announcement
Release datelined March 18, 2026. Connects preventable deaths with low-cost interventions and access to quality health care. Notes that successive estimation rounds are revised, so the historical and current reports are not used here to calculate a precise percentage decline.
Dated March 7, 2024. Gives 443,832 annual under-five deaths and the low price of oral rehydration salts, while also describing other prevention and treatment needs. The page does not label the mortality figure as deaths occurring in 2024.
UNICEF Data · Accessed October 16, 2026 · Child health data and explanation
Reports over 700,000 annual under-five deaths and antibiotic treatment typically costing less than US$0.50. Explains the need for clinical assessment and appropriate care. These headline estimates are not combined with the WHO diarrhea figure into a single-year mortality total.
September 8 announcement. Reports the agent count, external-forcing construction, and separate output-token totals for Navier–Stokes and all attempted problems. It does not provide an itemized dollar cost.
Standard Astra output rates checked October 16: $50 per million tokens for short context and $75 for long context. Multiplying by the reported token volumes gives illustrative retail equivalents, not the unreleased research model’s measured cost.
Catalogue and verification status checked October 16. Groups related manuscripts into families and explains the compute comparison.
Read the slide text and reveals
06 Origins & history
We have been here before
An extraordinary claim from 1988 reveals that today’s excitement about neural networks has a long history.
The phrase “more important than the atom bomb” appears before its date or speaker. Without that context, it could sound like a contemporary claim about AI. The reveal places it in 1988, when Scientific American reported Jasper Lupo of DARPA describing the potential importance of neural networks.
That change of context is why I wanted to show you this. Earlier researchers and funders also saw enormous possibilities in learning machines. Their enthusiasm existed alongside practical applications, serious obstacles, and uncertainty about what would work. I want us to take a longer view of the present: what has actually changed, which problems have been solved, and which hopes or difficulties have returned? I also remember meeting Lupo, so this history has a personal connection for me.
In the slide. The dramatic quotation comes first; the 1988 attribution and period report change how we read it.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 9:16–10:37The 1988 quotation and recurring promises about neural networks.
A closer look
01
Where the quotation comes from
June Kinoshita’s article Neural Networks at Work reports Lupo’s statement at a neural-network conference in San Diego. The wording appears on printed page 135 of the November 1988 Scientific American. The article identifies him as an assistant director at DARPA.
The DARPA Neural Network Study assessed neural-network technology and research directions during 1987–1988. The pictured final technical report is dated March 22, 1989. It supplies context for that period of research, while the Scientific American article supplies the quotation.
Kinoshita also reported work on industrial inspection, credit assessment, airline decisions, and process control. The historical comparison is therefore more interesting than an old prediction: neural networks already had practical ambitions, even as claims about their future ran far ahead of their demonstrated abilities.
Contemporaneous reporting in Scientific American 259(5), pages 134–135. Lupo’s quotation and title appear on page 135; page 134 describes emerging applications. DOI: 10.1038/scientificamerican1188-134.
MIT Lincoln Laboratory for DARPA · 1989 · Research report
Technical Report 840 / ESD-TR-88-311, dated March 22, 1989, covering the October 1987–February 1988 study. The cover and foreword explain the relationship between the study dates and the final report.
Read the slide text and reveals
Step 1 of 2
So where did this come from?
“more important than the atom bomb”
Step 2 of 2
Jasper Lupo · DARPA
1988
DARPA Neural Network Study A DARPA assessment of neural-network technology. 1987–1988 study · Final report March 1989
07 Origins & history
The timeline
Today’s AI draws on a century of overlapping ideas about computation, brains, learning, feedback, and language.
The timeline follows people and ideas rather than presenting AI as a recent invention. Fiction gives us a name for manufactured workers; mathematics clarifies what computation can do; neuroscience and cybernetics connect biological processes with formal models. Neural learning, symbolic reasoning, better hardware, and new ways to represent language develop through overlapping communities.
Two labels receive special emphasis because they will matter throughout the lecture. Backpropagation calculates how changes inside a network affect an error or objective. Reinforcement learning organizes learning around actions and evaluated outcomes. These are complementary ideas: neural reinforcement-learning systems can use backpropagation to change their networks. The later exercises make this distinction concrete.
The final milestones bring these foundations into large language models and conversational systems. The collage at the end conveys the activity visible at the time of the lecture. The dates mark selected publications, demonstrations, and releases, rather than asserting that each idea began at exactly that moment or belongs to one inventor.
In the slide. Portraits identify the people; the small computer, network, and brain images identify recurring themes. Equal spacing separates events for readability, rather than measuring the years between them.
Čapek’s R.U.R. depicts manufactured workers whose revolt threatens humanity. Its robots are artificial living workers, rather than today’s familiar metal machines. The play was published in 1920, with premieres in 1921. In his own later account, Karel credits his brother Josef with suggesting the word robot. The story raises questions about labor, dependence, and responsibility before modern AI research takes shape. I read its promise of tireless manufactured labor as a warning about valuing productivity while forgetting the dignity of people.
Wiener’s 1948 book Cybernetics: Or Control and Communication in the Animal and the Machine connected the study of communication and feedback across biological and engineered systems. Feedback means that the consequences of a process influence what it does next. A feedback controller can regulate a system without learning, so feedback by itself does not necessarily imply changing a neural network’s parameters.
Rosenblatt’s perceptron made learning through adjustable connections a concrete research program. His work included rules for changing connections through experience. The later challenge was effective training of more general multilayer networks. The multilayer network pictured here is a teaching illustration, rather than a reconstruction of Rosenblatt’s 1958 machine or the anatomy of a human brain.
Backpropagation efficiently works backward through a computation to calculate how an error or objective depends on its adjustable parameters. An optimizer can use those calculations to change the network. The 1974 milestone recognizes Werbos’s work on backward differentiation and learning; the 1986 paper by Rumelhart, Hinton, and Williams helped bring the approach to a much wider audience. The mathematical history includes earlier contributions across several fields.
Reinforcement learning: how well did the action work?
The 1983 Barto, Sutton, and Anderson study used two adaptive components to learn a difficult control task. One selected actions; the other helped evaluate their consequences. This is an early actor-critic result, rather than the invention of all reinforcement learning. It illustrates the lecture’s important distinction between being given a desired answer and receiving an evaluation of what happened after acting.
The arrival of conversational AI involved more than making language models larger. The InstructGPT paper describes learning from demonstrations and then from human preferences to make responses better match user intent. This distinction between a pretrained language model and a model shaped to assist people becomes a central thread in the next sections.
I want you to remember backpropagation and reinforcement learning because they give us a manageable way into the training story. One explains how an objective can guide changes to a network; the other explains learning from the consequences of actions. These complementary tools sit within a broad, collaborative research history. We’ll see how they contribute to different stages of training in the next sections.
Ideas alone were not enough. Hardware, training methods, data, and network design had to work together. AlexNet’s 2012 paper gives a concrete example: GPU computation helped make a large convolutional vision network practical. The 2017 Transformer paper introduced an attention-based architecture that made sequence training more parallelizable. Together, these developments show computing capacity and network design advancing alongside one another, expanding what models could do with images and language. I know from my own research how expensive this hardware can be. That’s one reason I think access to computing deserves attention as a possible policy lever. It may give us a way to influence development, though I’m not offering it as a guarantee of control.
The original fictional work behind the timeline’s first milestone. The opening explains the manufactured workers; the final acts and epilogue develop the revolt and its aftermath.
Čapek’s own account, published in Lidové noviny on December 24, 1933, credits Josef Čapek with suggesting the name. English reproduction preserved by the University of Oregon.
Wiener’s foundational book connects feedback, control, and communication. The publisher page confirms first publication in 1948 and links an open-access reissue of the expanded 1961 edition.
The original Psychological Review paper, volume 65, pages 386–408, develops the perceptron as a model of learning and recognition. Supports the brain-inspired learning milestone.
Werbos’s retrospective explains backward differentiation and its connections with neural learning and control. Section 2 discusses his 1974 dissertation; the introduction places the technique within several interacting research traditions. Published in 2006; the timeline also links the author manuscript.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1986 · Paper
The original learning-control study develops an associative search element and an adaptive critic, illustrated by a cart-pole balancing task. Supports the timeline’s early actor-critic milestone.
Figure 2 and Section 3 describe supervised fine-tuning, preference collection, and reinforcement learning from human feedback. Supports the transition from broad language modeling toward assistant behavior.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2012 · Paper
The original NeurIPS paper describes a deep convolutional image classifier and the GPU implementation used to accelerate training. See the abstract and Section 3.2 on training on multiple GPUs.
The eight-author paper introduces the Transformer. The abstract and Sections 1 and 3 explain attention-based sequence modeling and the role of parallel computation, supporting the timeline’s architecture milestone.
08 Origins & history
Two paths from shared roots
Language models are one major destination of neural learning, while control, optimization, and adaptive engineering remain another living tradition.
The branching picture condenses the chronology into two different approaches to intelligence. Symbolic AI works with explicit representations, rules, and search. Connectionism learns through networks of adjustable connections. The perceptron makes the meeting of computation and brain-inspired learning visible; the later debate over its limitations helps explain why these approaches developed distinct communities and priorities.
As learning methods advanced, neural networks became increasingly important within mainstream AI and computer science. The upper route follows that development toward recognition, language, and eventually large language models. The merger represents a change in research practice across the field. It does not mean that symbolic rules were physically combined with neural weights to manufacture an LLM.
The lower route keeps a second history in view: using learned models to control changing physical systems, optimize decisions, and adapt over time. Neural Networks for Control anchors that tradition around the beginning of the 1990s. Its dashed continuation matters because this work did not end when language models became prominent. The branches continue to exchange ideas, and engineering problems still motivate methods whose purpose extends well beyond producing language.
For me, the engineering branch is a reason for hope and a research priority. Think about energy use, forecasting, control, and adaptive systems: their value depends on reliable behavior in the world. I want us to invest in bringing more of that knowledge into useful AI systems while being more selective about deployment. There is work to do here; we can’t simply transfer a guarantee from a physical controller to an open-ended conversation. When I look at the money going into selling AI products and military applications, I want to see more investment in stability, safety, and fail-safe behavior.
In the slide. Follow both endpoints: the upper line leads toward modern LLMs; the lower dashed line keeps control, optimization, and adaptive systems visibly in motion.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 22:13–25:54Two research paths and the case for investing in safety and control.
This is a selective map of research traditions. The 1955 Dartmouth proposal already included both language-related questions and neuron nets. Minsky and Papert’s 1969 Perceptrons examined limitations of particular perceptron classes and became a focal point of debate. It did not establish that every multilayer network was incapable of learning, and symbolic AI did not begin with that book.
A system of explicit rules can work well within a carefully defined domain. Extending it into a changing world introduces exceptions, ambiguous cases, and interactions among rules. In his 1990 essay, Minsky describes limitations on both the symbolic and connectionist sides and argues for systems combining different representations. The historical division was never a complete separation of ideas.
Neural Networks for Control, edited by W. Thomas Miller III, Richard S. Sutton, and Paul J. Werbos, brings learning together with control problems, including robotics and motion planning. The lecture timeline uses its commonly cited 1990 date; MIT Press lists the hardcover release as January 10, 1991. The pictured cover belongs to the later paperback. The book identifies a research program, rather than an exact day when disciplines split.
Large neural language models learn from text; later training can shape how they respond to people. GPT-3 and InstructGPT make that distinction concrete: one paper studies a large pretrained language model, while the other adds demonstrations and human-preference training. Reinforcement learning is important to many assistant models, but it is not the explanation for every stage of language-model training.
A robot must cope with changing physical conditions, imperfect measurements, and constraints on safe action. The safe-control-gym study provides a concrete example of researchers comparing traditional control, learning-based control, and reinforcement learning on shared tasks. It shows both continued engineering development and active exchange among these communities. The parallel line therefore represents a larger set of problems and tools, not an isolated body of work with no connection to AI.
The engineering lesson is to define the system, its operating conditions, and the constraints it must respect, then evaluate what happens under disturbances. The safe-control-gym benchmark makes this concrete with simulated cart-pole and quadrotor tasks. I want that discipline to influence AI development more strongly. A result established for a defined control problem does not by itself establish the safety of a general language model.
An audience member asked how those engineering safeguards could carry over to modern AI. Consider the difference between a system whose dynamics and constraints we can describe mathematically and a confused person seeking good customer service. In the second case, even deciding what counts as success is difficult. I want us to develop systems that can keep learning through guided experience with people. That’s a direction I’m proposing, with safety still something we would have to establish. Continued learning alone would not settle it.
The August 31, 1955 proposal for the 1956 meeting lists language and neuron nets among its research topics. It provides direct evidence that the field’s early agenda encompassed several approaches.
The mathematical analysis behind the perceptron-scrutiny milestone. Its results concern specified classes of perceptrons, rather than a universal impossibility of neural learning.
Minsky’s own account of the traditions and their limitations. The opening discussion and section Heuristic Connections of Rule-Based Systems explain his case for combining approaches.
W. Thomas Miller III, Richard S. Sutton, and Paul J. Werbos, editors · 1990/1991 · Book
The publisher’s description identifies learning control, robotics, motion planning, and benchmark control problems. Its edition information distinguishes the January 1991 hardcover from the March 1995 paperback.
Describes GPT-3 and its evaluation across language tasks. Section 2 covers the model and training data, establishing a concrete point on the large neural language-model route.
Figure 2 and Section 3 show how demonstrations and reinforcement learning from human feedback are applied after a pretrained model, clarifying the role of RL along the route to assistants.
Zhaocong Yuan and colleagues · 2021; revised 2022 · Paper
An original benchmark study comparing traditional, learning-based, and reinforcement-learning control on cart-pole and quadrotor tasks. Illustrates the continuing engineering route and exchange between research communities.
Read the slide text and reveals
Step 1 of 6
So where did this come from?
Brain meets machine.
The perceptron brings a biological idea into a working machine.
Brain
Machine
1958 Perceptron
Step 2 of 6
Two approaches to intelligence.
Symbolic AI and connectionism pursue different ways to build intelligence.
Symbolic AI Symbols · rules
1969 · Minsky & Papert
Connectionism
This step replaces or removes earlier wording.
Step 3 of 6
Learning finds new momentum.
Backpropagation and reinforcement learning carry connectionism forward.
Backpropagation Reinforcement learning
Limits of hand-coded knowledge
This step replaces or removes earlier wording.
Step 4 of 6
Connectionism branches again.
One route pursues language and recognition. Another develops control and optimization.
Computer science Recognition · language
Engineering Control · optimization Adaptive systems
Neural Networks for Control Miller · Sutton · Werbos
This step replaces or removes earlier wording.
Step 5 of 6
Neural learning reshapes AI.
The computer-science route enters mainstream AI and leads toward modern LLMs.
Mainstream AI Neural networks · RL
Leads to modern LLMs
This step replaces or removes earlier wording.
Step 6 of 6
Two living traditions.
LLMs are one destination. The engineering toolbox keeps growing.
A parallel tradition, still evolving
This step replaces or removes earlier wording.
09 Learning from examples
What do we mean by intelligence?
I use intelligence to mean finding a fitting response to an unfamiliar situation and improving through feedback.
My working definition has two parts: responding to a situation that has not been encountered before, and getting better at doing so through feedback. A fitting response depends on the situation. Repeating a familiar answer is not enough when the circumstances have changed.
This definition gives us a practical question to pursue: how can a machine become better at producing useful responses? The next slides make learning tangible, first as a network adjusting its connections and then as word games that let us experience the pull of a learned pattern. I offer this as a working definition for our discussion, not as a claim that researchers have agreed on one complete definition of intelligence.
You don’t need mathematics to follow these exercises. In the room, I invited everyone to become a “baby LLM”: to experience how feedback changes an expectation, then see how the resulting expectation can go wrong. You can follow the same exercises here.
In the slide. The same sentence stays on screen while emphasis moves from a fitting response to learning from feedback.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 25:54–26:37A practical definition of intelligence and the move to learning through feedback.
A closer look
01
A working definition
I offer this definition as a useful lens for our discussion. It does not establish a test of consciousness, human equivalence, or every capacity that might count as intelligence.
02
Why the emphasis moves to feedback
The visual shifts attention from responding to improving. That is the bridge from the history of AI to the learning mechanism examined next.
Read the slide text and reveals
Step 1 of 2
02 / Now, how does this work?
A working definition
Intelligence is the ability to come up with a fitting response to a situation you have never seen before,
and to get better at it from feedback.
10 Learning from examples
Learning through feedback
Training uses feedback about responses to adjust the numerical connections inside a network.
Follow a single cycle. An example enters the network. Its current connections shape a response. That response is evaluated, and the resulting feedback guides an adjustment. Repeating this cycle over many examples can change which responses the network tends to produce.
The adjustable numbers are parameters, including connection weights. In the diagram, thicker and thinner lines make those changes visible. Backpropagation helps calculate how changes to parameters would affect an error measure; an optimizer uses that information to make updates. The small network is a teaching model, not a literal picture of a complete language model.
Keep training separate from using an already-trained model. During ordinary generation, a model applies its existing weights to the available context. Giving it a new sentence, document, or correction can change its next response without changing those weights. This distinction will matter when we start learning patterns from words.
In the slide. Follow input, response, feedback, then the changed thickness of the connections.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 26:37–27:11The feedback loop: an output, an evaluation, and an adjustment to weights.
Backpropagation supplies information about how an error depends on the network’s parameters. That makes guided weight adjustment possible; it does not decide by itself what should count as a good response.
A known target, a calculated error, or a reward can supply a learning signal. The returning arrow groups these ideas to explain their purpose, while later slides distinguish their roles.
Brown and colleagues evaluated GPT-3 with task descriptions and examples in the prompt, without gradient updates or fine-tuning. A changed response in conversation is therefore not, by itself, evidence of new parameter training.
In the discussion, I returned to this distinction when asked about moral learning: responding differently within a conversation is not the same as continually changing the trained model. I’ll return later to my proposal for systems that keep learning through experience. The GPT-3 reference establishes the narrower mechanism distinction, not that broader proposal.
The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
Read the slide text and reveals
Step 1 of 5
How it works / Learning
A network learns through feedback.
One idea behind all the mathematics.
Input
Output
A neural network
Input Response Feedback Adjustment
Step 2 of 5
A situation goes in.
In training, we show the network an example.
A situation
This step replaces or removes earlier wording.
Step 3 of 5
A response comes out.
Its connections shape the response it produces.
A response
This step replaces or removes earlier wording.
Step 4 of 5
How did it do?
Feedback tells us how that response could improve.
Feedback
This step replaces or removes earlier wording.
Step 5 of 5
Feedback changes the connections.
Some connections strengthen. Others weaken.
The strengths of these connections are called weights.
This step replaces or removes earlier wording.
11 Learning from examples
What comes next?
The words surrounding an unfinished sentence change which next word seems plausible.
Try completing the sentence: “I’ll have the usual,” he said to the _____. The offered choices are bartender, barista, barber, and doctor. Before there is more context, several familiar situations can fit. You may picture a favorite drink, a coffee order, a haircut, or a recurring appointment.
Now add the preceding sentence: “The espresso machine hissed behind the counter.” Barista becomes a particularly natural choice. The unfinished sentence and its grammar have not changed; the situation suggested by the surrounding words has.
This is a small, human-readable analogy for language prediction. A language model uses the text available so far to shape its next output, then continues the process. The exercise presents four illustrative choices, not measured probabilities from a model. Its lesson is the influence of context, rather than proof that one completion is logically inevitable.
In the room, the show of hands supplied a visible spread of possible answers. I invited the group to act as a “baby LLM”: several continuations could initially feel natural, and a small clue could shift the distribution. The votes were a teaching analogy, not measurements of an actual model.
In the slide. Compare your first completion with your choice after the espresso-machine clue appears.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 27:11–28:29The audience becomes a “baby LLM” and revises its choice when context changes.
A closer look
01
Try it before reading the clue
Form an initial preference among bartender, barista, barber, and doctor, then compare it with your choice after the espresso sentence. The change in expectation is the experience the exercise is designed to reveal.
02
“Next word” is useful shorthand
Language models generally generate tokens. A token can be a word, part of a word, or another text unit. I use whole words here so the prediction task is easy to follow.
Context is the text available to the model for this prediction. The context window limits how much can be available at once. Adding the espresso clue changes the input, not the trained weights.
Sections 3.1 and 3.4 explain sequential token generation and output probabilities; section 5.1 describes subword tokenization. The paper concerns the original transformer, not the small teaching network I use here.
The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
Read the slide text and reveals
Step 2 of 5
A word. A situation.
“I’ll have the usual,” he said to the _____.
Step 3 of 5
A bartender
B barista
C barber
D doctor
Step 4 of 5
The espresso machine hissed behind the counter.
Step 5 of 5
B barista Feedback
Context · the input
Next word · the output
This step replaces or removes earlier wording.
12 Learning from examples
Context changes the answer
Different situations can make different completions of the same sentence appropriate.
The four versions deliberately keep “I’ll have the usual” unchanged. Bottles of gin behind a bar suggest bartender. An espresso machine suggests barista. Freshly cut hair around a salon chair suggests barber. A clinic visit to renew a prescription suggests doctor.
None of those completions is a universal answer to the unfinished sentence. Each fits a different interpretation of the scene. The sequence moves beyond a single association and shows why understanding a request depends on the surrounding situation, including details that change what someone probably means.
For a language model, supplying relevant context can similarly change the continuation it produces. But more text is not automatically better evidence. A clue can be mistaken, irrelevant, or misleading. The useful question is whether the supplied information helps identify the actual situation. The following exercise turns that question around: what happens when feedback teaches a pattern that works in examples but misses their larger meaning?
In the slide. Read each contextual clue before its completion and notice what situation you imagine.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 28:29–29:34Context changes the next prediction; the conversation grows as words are added.
A closer look
01
Four contexts, four natural readings
The series is an invented teaching example, not a comparison of measured model responses. The answers are plausible completions, not the only grammatically possible ones.
02
Context operates at generation time
Task descriptions and demonstrations can steer an already-trained model through its input. This kind of adaptation does not require a new round of weight training.
The original transformer paper explains how attention allows information from different positions in a sequence to contribute to a prediction. Its architecture is more elaborate than the word game, but the connection is the relevance of surrounding text.
Think of a loop: generate another word, add it to the text, then continue from that enlarged context. A user’s reply adds more material too. This explains why a long exchange eventually reaches a context limit. In an implementation, cached intermediate results usually avoid recomputing all earlier tokens from scratch; the conceptual dependence on the preceding context remains.
The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
Sections 3.1 and 3.4 explain sequential token generation and output probabilities; section 5.1 describes subword tokenization. The paper concerns the original transformer, not the small teaching network I use here.
Explains how autoregressive generation depends on preceding tokens and how a key-value cache reuses their computed representations. Consulted October 16, 2026.
Read the slide text and reveals
Step 1 of 5
What the words around it tell us
The context changes the answer.
01
Bottles of gin lined the shelves behind the bar.
“I’ll have the usual,” he said to the bartender.
Step 2 of 5
02
The espresso machine hissed behind the counter.
“I’ll have the usual,” he said to the barista.
Step 3 of 5
03
Freshly cut hair lay around the salon chair.
“I’ll have the usual,” he said to the barber.
Step 4 of 5
04
He was back at the clinic to renew his prescription.
“I’ll have the usual,” he said to the doctor.
Step 5 of 5
The words before it change what comes next.
13 Learning from examples
Learning from feedback
Selective feedback can teach a consistent rule even when the examples allow many reasonable answers.
This game presents four unfinished sentences, each with four plausible choices. After each choice, the exercise identifies the answer it favors. The selected completions are fox, pony, dog, and rabbit. Their positions move between B, A, C, and D, so choosing the same letter cannot explain the success.
The common feature is that each selected answer is an animal. By the end, you may expect another animal before seeing the next sentence. The feedback has made one pattern stand out. It has not shown that the other choices were impossible: finding a parcel inside a gate, for example, would make perfectly ordinary sense.
The live responses made the change visible. At the first prompt, I saw many hands for gardener. By the third, most of the room was choosing dog. By the photograph prompt, the audience could name the animal pattern. The exercise therefore reveals a shift in expectation, not just a sequence of correct answers.
That is the point of the exercise. A learner can become good at the pattern that its examples and feedback favor. Whether that pattern serves the intended purpose is a further question. The next slide changes the situation to test the rule you have just acquired.
In the slide. The four selected answers are fox, pony, dog, and rabbit; look for what they share, rather than their answer letters.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 29:34–32:15The animal pattern emerges through garden, lane, sound, and photograph examples.
A closer look
01
Example 1: the garden gate
“Just inside the garden gate, I found a _____.” Choices: bicycle, fox, parcel, gardener. The feedback selects fox. The other options are also plausible discoveries.
02
Example 2: the lane
“At the end of the lane stood a _____.” Choices: pony, tractor, cottage, postbox. The feedback selects pony.
03
Example 3: a sound in the dark
“From somewhere in the dark came the sound of a _____.” Choices: radio, train, dog, bell. The feedback selects dog.
04
Example 4: the photograph
“In the photograph, she was holding a _____.” Choices: trophy, candle, suitcase, rabbit. The feedback selects rabbit. All four favored answers have now been animals.
05
What the analogy does and does not show
This is an audience exercise about selective feedback, not an actual training run. Shortcut-learning research studies the wider problem of rules that work on familiar examples but fail elsewhere. If these examples are supplied in a model’s prompt, it may infer the pattern without changing its weights.
A network can exploit a pattern without automatically favoring the relationship the teacher intended. Its data, architecture, and training objectives shape which patterns it learns. The animal shortcut separates success under the feedback from success at the intended task: what earned credit in one set of examples may be misleading in the next.
I compared the show of hands with a model’s distribution over possible continuations. Under sampling, a lower-ranked choice may still be selected; other decoding settings choose the highest-scoring choice. Neither method tests by itself whether the selected continuation is true.
Explains how a rule that succeeds on familiar examples can fail under changed conditions. It supports the concept illustrated by the exercise, not a claim that the invented game is a model experiment.
The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
Documents sampling and greedy selection as alternative generation settings. This supports the distinction between a token distribution and the method used to choose from it.
Read the slide text and reveals
Step 1 of 13
Clear your mind.
Let’s try another game.
Step 2 of 13
Learn from the feedback.
01 04
Just inside the garden gate, I found a _____.
This step replaces or removes earlier wording.
Step 3 of 13
A bicycle
B fox
C parcel
D gardener
Step 4 of 13
Just inside the garden gate, I found a fox.
B fox Feedback
This step replaces or removes earlier wording.
Step 5 of 13
02 04
At the end of the lane stood a _____.
This step replaces or removes earlier wording.
Step 6 of 13
A pony
B tractor
C cottage
D postbox
Step 7 of 13
At the end of the lane stood a pony.
A pony Feedback
This step replaces or removes earlier wording.
Step 8 of 13
03 04
From somewhere in the dark came the sound of a _____.
This step replaces or removes earlier wording.
Step 9 of 13
A radio
B train
C dog
D bell
Step 10 of 13
From somewhere in the dark came the sound of a dog.
C dog Feedback
This step replaces or removes earlier wording.
Step 11 of 13
04 04
In the photograph, she was holding a _____.
This step replaces or removes earlier wording.
Step 12 of 13
A trophy
B candle
C suitcase
D rabbit
Step 13 of 13
In the photograph, she was holding a rabbit.
D rabbit Feedback
What pattern did you learn?
This step replaces or removes earlier wording.
14 Learning from examples
Hold on to what you learned
A learned shortcut can fit past feedback and still fail the new situation.
Carry the animal rule into a new sentence: “The locksmith opened the door with a _____.” The choices are key, lockpick, drill, and giraffe. The first three can plausibly belong in a locksmith’s work. Under an ordinary literal reading, a giraffe cannot serve the intended purpose.
Yet giraffe is the answer that fits every previous round of the game. The conflict is between the learned pattern and the situation the sentence describes. Doing more of what worked before can now lead directly away from a useful answer.
This is a compact illustration of shortcut learning. The practical concern is generalization: whether success on familiar examples carries into changed circumstances. It also gives an intuition for one route to unreliable language-model output. It does not establish that every hallucination has this cause, or that the key is the only possible correct completion. The lesson is to examine which rule the feedback actually encouraged.
In the slide. Keep the previous animal rule in mind while comparing the four locksmith answers.
Giraffe is highlighted as the learned shortcut, not endorsed as the factual answer. The absurdity exposes the difference between matching the feedback pattern and fitting the task.
02
Research connection: shortcut learning
Geirhos and colleagues describe decision rules that succeed on familiar benchmarks but fail when conditions change. The locksmith game makes that concern accessible; it is not a measured result about a particular LLM.
I’m questioning the hope that enough additional text would make every misleading pattern disappear. This exercise demonstrates a narrower point: a pattern that succeeds during training can fail in a new setting. More or better data can help, but size alone is not evidence that every shortcut or unsupported answer has been removed.
Explains how a rule that succeeds on familiar examples can fail under changed conditions. It supports the concept illustrated by the exercise, not a claim that the invented game is a model experiment.
Read the slide text and reveals
Step 1 of 3
Hold on to what you learned.
The locksmith opened the door with a _____.
Step 2 of 3
A key
B lockpick
C drill
D giraffe
Step 3 of 3
D giraffe Learned shortcut
Fits the pattern. Fails the situation.
This step replaces or removes earlier wording.
15 Learning from examples
Go with your gut
Knowing when not to answer is a separate challenge from producing a plausible answer.
The last word game asks for a precise departure time: “The train departs at exactly _____.” Its choices are velvet, spoon, sideways, and thunder. None provides an ordinary literal time. The useful response is to challenge the choices or acknowledge the missing information.
The exercise separates producing an answer from having grounds for one. A system can be very good at continuing language while still facing situations where the evidence is inadequate. A fluent completion does not supply the missing timetable.
The default generation mechanism matters too. At each step, a function called softmax turns scores for possible next tokens, words or word-pieces, into probabilities that add up to 100%. Ordinary generation still selects a token when those probabilities are spread thinly across many choices. Nothing in that operation automatically pauses the response because the evidence is inadequate.
Researchers call the decision not to give a definite answer abstention. Studies directly testing this ability find that stronger problem-solving does not ensure reliable abstention. Some tested reasoning models became worse at withholding unsupported answers. Another study found that scaling and instruction tuning could replace nonanswers with plausible mistakes.
An LLM can say “I don’t know.” The difficulty is doing so dependably when an answer is unsupported. Evaluation incentives help explain why: if a wrong answer and an admission of uncertainty both earn zero, guessing offers a chance of scoring a point while abstaining does not.
In the slide. Notice whether you feel pressure to select one of the four options despite the missing departure time.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 33:24–36:18The train question, forced completion, and knowing when to say “I don’t know.”
Under the exercise’s ordinary reading, none of velvet, spoon, sideways, or thunder is a departure time. A creative fictional interpretation is possible, but it does not establish when a real train leaves.
02
Softmax supplies choices, not a test of knowledge
Softmax distributes 100% of the probability across the next-token options. As a toy example, 1,000 equally scored options would each get just 0.1%, yet ordinary sampling would still select one. The structural issue is softmax combined with the generation loop: low individual probabilities do not automatically trigger abstention. These are probabilities of text continuations, not probabilities that a claim is true; many valid phrasings can also spread the probability thinly. The model can produce “I don’t know,” choose an end token, or be stopped by an additional control. Reliable abstention therefore depends on learned behavior or added controls, rather than being guaranteed by softmax itself.
Direct evidence: failing to recognize unanswerable questions
AbstentionBench tested 20 models across 20 datasets, including questions with missing information, false premises, and unknown answers. Abstention remained unreliable. In matched comparisons, some reasoning-trained models improved at solving problems while getting worse at deciding when not to answer. Explicit instructions helped, but did not solve the problem.
Zhou and colleagues found that larger, instruction-tuned models often replaced nonanswers with plausible but incorrect responses. Their “avoidance” category also includes malformed outputs from earlier models, so it is broader than a thoughtful “I don’t know.” The key finding is that greater capability did not reliably produce better restraint.
Kalai and colleagues explain how accuracy-based evaluations reward guessing over appropriate uncertainty. A guess can earn credit; an abstention cannot. They propose stating the scoring rules and penalizing errors so models are rewarded for withholding answers when the evidence is too weak.
An audience member compared “I don’t know” with the fallback else branch in a conventional program. The analogy sharpens the design question: what happens when none of the proposed answers is warranted? A learned model can generate an admission of uncertainty, but ordinary token selection is not an explicit test for that condition. Abstention behavior therefore needs to be trained and evaluated, or supported by additional controls.
My concern is that excitement and investment can favor shipping more capable systems over stopping to address their weaknesses. That contrasts with the demands I remember from my engineering work. This is my interpretation of development incentives. The cited research establishes measured abstention failures, not the reasons developers choose to deploy.
Directly tests whether models withhold answers when appropriate. Sections 4.1 and 4.3, Figures 2–3 and 6, cover general failures and reasoning-model comparisons; section 4.4, Figure 8, tests an abstention instruction. The findings concern the tested models and benchmarks, not an impossibility of saying “I don’t know.”
Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo · 2024 · Nature paper
Nature 634, pages 61–68. Figure 2 and the Results discussion distinguish correct, incorrect, and avoidant responses across model families; Supplementary Note 15 explains the different kinds of avoidance.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang · 2026 · Nature paper
The peer-reviewed version of the 2025 preprint Why Language Models Hallucinate. The main text and Figure 1 explain the incentive to guess; the open-rubric evaluation proposal makes the cost of an incorrect answer explicit.
PyTorch documentation · Accessed October 2026 · Technical reference
The definition and formula show that softmax normalizes scores into probabilities summing to one. The equal-score example in the annotation is an illustration of that formula, not a measurement of an LLM.
Hugging Face Transformers documentation · Accessed October 2026 · Technical reference
GenerationConfig documents sampling versus greedy decoding, end-of-sequence tokens, and explicit stopping controls. These mechanisms explain why selecting a next token is distinct from deciding whether an answer has sufficient factual support.
Read the slide text and reveals
Step 1 of 4
Forget the pattern.
Go with your gut.
Step 2 of 4
One more.
The train departs at exactly _____.
This step replaces or removes earlier wording.
Step 3 of 4
A velvet
B spoon
C sideways
D thunder
Step 4 of 4
Knowing when to say
“I don’t know.”
A likely next word is not a guarantee of knowledge.
An LLM can say “I don’t know.” The challenge is knowing when.
This step replaces or removes earlier wording.
16 Learning from examples
The scale behind the learning
Model size and training-text volume are different quantities, and both can be enormous.
Kimi K2, released in 2025, supplies a disclosed example of both kinds of scale: one trillion total parameters and 15.5 trillion tokens processed in pretraining. Parameters are the adjustable numbers inside the model. Tokens measure the text units encountered during training. The two figures answer different questions.
The dot field makes the parameter count visible by assigning a billion parameters to each point. The smaller historical markers, roughly 117 million for GPT-1 and 175 billion for GPT-3, give perspective. These are selected examples, not a rule that every later model must be larger.
The library comparison makes text volume easier to imagine. Under the conversion assumptions I use here, Kimi K2’s pretraining tokens correspond to roughly five Library of Congress cataloged-book collections. This is a scale analogy, not a claim that the model contains or read five distinct libraries. Additional post-training examples and feedback are outside that total and are not quantified here.
In the slide. Read the network and text totals as two separate scales; the library illustration belongs to the text total.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 36:18–37:52Network and corpus scale, with a comparison to my airline forecasting work.
A closer look
01
Total parameters versus active parameters
Kimi K2 is a mixture-of-experts model: about 32 billion parameters are active for a token out of one trillion total. The points show total scale, not its architecture or the number used on each token.
The illustration assumes 0.75 words per token and 90,000 words per book. 15.5 trillion × 0.75 gives 11.625 trillion illustrative words. 25.77 million cataloged books × 90,000 gives about 2.319 trillion words; the ratio is about 5.01. Neither conversion is a measured average of these collections.
Token-to-word ratios vary with language, tokenizer, and content. Training text can be repeated, and the Library of Congress has many holdings beyond cataloged books. Parameter count is not a count of facts retained or an equivalent count of biological neurons. The pretraining figure counts tokens processed across training, not a separate five-library collection for each pass.
GPT-1’s released parameter shapes yield about 117 million, and the GPT-3 paper reports 175 billion. The slide deliberately uses disclosed figures. Its pretraining-token total does not estimate the separate quantity of examples, rankings, and rewards used in post-training.
I recall deploying neural-network systems for US Air in 1991 to predict demand and no-shows, with another network used to optimize prices. As I remember it, the collection had fewer than 500,000 nodes. I offer that story to give a sense of how far the scale has moved. My recalled node count is not directly comparable to today’s parameter counts, so it should not be used to calculate a precise growth factor.
The model introduction and summary disclose 1 trillion total parameters, 32 billion active parameters per token, and 15.5 trillion pretraining tokens for the same named model.
Library of Congress · FY2023 collection figures · Official documentation
The Year 2023 at a Glance section reports more than 25.77 million cataloged books in the Library of Congress classification system. The word-volume conversion is my illustration, not a Library of Congress estimate.
The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
Read the slide text and reveals
Step 1 of 4
02 / Now, how does this work?
Now, imagine the scale.
A disclosed example · Kimi K2, 2025
The network
1 trillion
Total parameters. The adjustable weights inside the model.
Each point represents a billion parameters.
117 million GPT-1 · 2018
→
175 billion GPT-3 · 2020
→
1 trillion Kimi K2 · 2025
Kimi K2 · model & training figures ↗ Library of Congress ↗ Sources & conversion assumptions in presenter notes
Step 2 of 4
The pretraining text
15.5 trillion
Tokens. Pieces of text seen during pretraining.
Step 3 of 4
About five Libraries of Congress.
A rough text-volume comparison · cataloged books only
Step 4 of 4
Plus the examples and feedback used in post-training.
17 Shaping an assistant
What does it actually produce?
A pretrained model can continue the form of a request instead of carrying out the request.
The earlier games gave feedback on a prediction. In LLM pretraining, actual text supplies that feedback: show the model the preceding text, ask it to predict the next token, then compare its prediction with what really comes next. Training adjusts the parameters and repeats this process across enormous amounts of text.
Imagine asking for a few sentences that tell a six-year-old about the Moon landing. Rather than answering, the response begins another child-oriented request, this time about gravity. It then asks for explanations of relativity, the beginning of the universe, and evolution. The subject keeps changing, but the form of the request continues.
The slide reveals those four parts gradually so the difference can be felt before it is explained. This is one published GPT-3 completion from OpenAI’s 2022 InstructGPT announcement, presented in paraphrase. It was not four separate questions sent to the model, and I did not generate the response live.
The example is striking because the text is coherent while the interaction fails. A person wanted an answer; the model produced more of a recognizable textual pattern. Pretraining can develop substantial capabilities, including answering questions, but the training objective does not by itself guarantee the conversational role we expect from an assistant. That mismatch motivates post-training.
In the slide. Follow what stays the same across the four requests even as their topics change.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 38:16–39:44The Moon-landing continuation and my recollection of a Reagan-to-airport digression.
A closer look
01
Where the direct feedback comes from
A token is a small unit of text, often a word or part of a word. The next token in the training text supplies the target. Across repeated predictions, the model learns regularities that help it continue language.
The order is gravity, relativity, the Big Bang, then evolution. The successive reveals divide one historical completion for readability; they do not represent independent trials.
The slide preserves the task and the pattern of the published example in simpler wording. It should be read as a faithful paraphrase, not a verbatim output transcript.
A base model can answer questions with suitable prompting. This sample illustrates one instruction-following failure, not a measured failure rate for all pretrained models.
I recall asking a base model about Ronald Reagan and receiving a continuation that moved from his film career and presidency to Jimmy Carter, hostages, the CIA, George Bush, and finally Dallas airport statistics. I’m sharing a recollection of topic drift; the model version, exact prompt, and full output aren’t documented here. The published Moon-landing comparison provides the documented example beside it.
The January 27, 2022 InstructGPT announcement is the source of the Moon-landing comparison and explains the move from text completion toward instruction following.
OpenAI, preserved by archive.ph · 2022 · Research report
Archived copy cited in my original lecture notes for the Moon-landing prompt, GPT-3’s sequence of further requests, and InstructGPT’s child-oriented answer.
Read the slide text and reveals
Step 2 of 5
The prompt
Tell a six-year-old the story of the Moon landing in a few sentences.
GPT-3 / Pretraining only
Describe gravity in language a six-year-old would understand.
OpenAI · 2022 ↗ Published prompt and continuation · Wording paraphrased
Step 3 of 5
Give a six-year-old a short explanation of relativity.
Step 4 of 5
Describe how the universe began for a six-year-old.
Step 5 of 5
Help a six-year-old understand evolution.
18 Shaping an assistant
After pretraining
Post-training can shift the same language-model foundation toward answering the user’s request.
The same Moon-landing task is now shown as a comparison. The pretrained GPT-3 sample continues with more requests for explanations. The InstructGPT sample instead describes people going to the Moon, taking pictures, and sharing what they saw with people on Earth.
The contrast is about the kind of response, not a competition to produce the most sophisticated explanation. One output extends a pattern in text; the other takes up the role of a helpful respondent. That difference is central to why a familiar chat assistant feels unlike an unrestricted text-completion model.
InstructGPT was produced through further training that combined example answers with human preference feedback and reinforcement learning. This comparison shows the resulting change in behavior; it does not isolate which part of that training caused each improvement. It is also a historical pair of samples, not a guarantee that post-training always yields a correct or complete answer. The next example makes the remaining imperfection visible.
In the slide. Compare whether each response answers the task, rather than whether both responses are fluent.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 39:44–40:33Before-and-after examples introduce the transition to more useful responses.
A closer look
01
Read the before and after at the same level
The task is paraphrased on the slide; the short response excerpts come from the published comparison. Smaller captions summarize the omitted continuation rather than adding new model output.
InstructGPT’s method used demonstrations, a reward model trained from human rankings, and RL optimization. The before-and-after pair should not be presented as an experiment that measures RL alone.
The January 27, 2022 InstructGPT announcement is the source of the Moon-landing comparison and explains the move from text completion toward instruction following.
OpenAI, preserved by archive.ph · 2022 · Research report
Archived copy cited in my original lecture notes for the Moon-landing prompt, GPT-3’s sequence of further requests, and InstructGPT’s child-oriented answer.
Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
Continues with requests about relativity, the Big Bang, and evolution.
Continuing the pattern
This step replaces or removes earlier wording.
Step 4 of 4
After InstructGPT · Post-trained
“People went to the moon, and they took pictures of what they saw…”
Goes on to explain that the pictures were shared with people on Earth.
Answering the request
Same request. A different response.
19 Shaping an assistant
Training for a useful response
A more useful response can still introduce details that the source never supplied.
Both models receive a stuffed-zucchini recipe and a request to turn it into a shopping list. GPT-3 continues the cooking instructions, telling the reader to bake the dish. InstructGPT produces ingredient entries, including ground beef and onion. It has moved much closer to the requested task.
But it also supplies quantities absent from the recipe, such as a pound of beef and a small onion. Those additions sound plausible and make the list appear useful. They are still unsupported by the input. The full published response also retains a cooking instruction before the ingredient list.
This is why improvement and reliability must be assessed separately. The response can follow the intended format better while remaining unfaithful to its source. The example prepares the transition into reward-based training: the objective is to encourage useful behavior, but a preferred answer may have weaknesses that the feedback did not adequately capture. A tidy output deserves the same attention to evidence as a messy one.
In the slide. Compare the ingredient names with their quantities; the error is easiest to see in the added amounts.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 39:44–40:33A brief discussion of requested format within the before-and-after comparisons.
A closer look
01
Find the full comparison
The complete recipe and outputs appear in Figure 49, Appendix F, page 67 of the InstructGPT paper. Reading the full figure reveals both the useful ingredient list and the unsupported quantities.
The Appendix F samples compare GPT-3 and InstructGPT at the same 175B size, with different sampling temperatures. The example illustrates output differences but is not a controlled estimate of RL’s isolated effect.
The supplied recipe names beef, onion, mushrooms, and peppers without specifying their amounts. The post-trained response fills in several amounts. The slide’s warning is about that unsupported addition, not about whether the proposed recipe might happen to cook well.
The method combines learning from demonstrations with learning from preferences. Post-training shapes the likelihood of responses; it does not turn every response into a verified extraction from the input.
Figure 49 on page 67 contains the full stuffed-zucchini prompt and both 175B model responses. The Appendix F introduction on page 61 gives sampling temperatures: 0.7 for GPT-3 and 1 for InstructGPT.
Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
Read the slide text and reveals
Step 1 of 4
03 / Building the assistant
Another task
Turn a recipe into a list of groceries.
Both models receive the same stuffed-zucchini recipe.
We need the ingredients.
Ouyang et al. · 2022 · Fig. 49, p. 67 ↗ Task paraphrased · Published response excerpts
Step 2 of 4
Before GPT-3 · Pretrained
“Bake, uncovered, at 350° for 20-25 minutes or until zucchini is tender and cheese is melted.”
The next step in the recipe
This step replaces or removes earlier wording.
Step 3 of 4
After InstructGPT · Post-trained
1 pound ground beef 1 small onion, chopped …
Ingredients for the shopping list
Step 4 of 4
But it added quantities the recipe never supplied.
Pretrained model Patterns in language
Post-training Example answers + rewards
Assistant behavior More useful responses
20 Shaping an assistant
Learning from a reward
Reinforcement learning can use a judgment of an entire response to guide changes in future behavior.
The question is simple: why does rain fall? The first illustrative answer says that water falls from clouds and that falling water is rain. It stays on topic, but it mostly restates the question. Under the goal “useful, concise, relevant,” it receives a low reward.
A reward judges how well the response served the goal. It does not need to provide the exact sentence the model should have written instead. During training, responses are generated, scored, and used to guide parameter changes across many examples. The slide’s later answer, about drops gathering in clouds and falling when they become heavy enough, illustrates the direction of improvement.
This is the distinction from copying a demonstrated target answer. The learner is encouraged toward outcomes that score better, while there may be many ways to achieve them. That flexibility can be powerful, and it makes the choice of score consequential. The process is repeated training, not an automatic weight update performed by an ordinary chat exchange.
The practical shift I’m describing is toward shorter, more relevant answers and useful formats. I’m comparing a model that can continue text with an assistant whose behavior has been shaped toward a purpose. A demonstrated better answer and a score for a response are different kinds of feedback, even when a training pipeline uses both.
In the slide. Follow the full loop: generate a response, evaluate it, adjust during training, and try again across many examples.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 39:44–40:33Reward-based improvement is explained during the examples; this range overlaps slides 18–19.
A closer look
01
An invented example of the learning loop
The rain answers and evaluations were written for the lecture. They are not recorded model outputs, experimental measurements, or evidence that one update reliably produces the improved answer.
02
What gets adjusted
Training uses the reward signal to change model parameters. Across many trials, the aim is to favor higher-reward behavior. The later response shows a desired tendency, not a guaranteed result after each adjustment.
RL describes learning from rewards. Backpropagation supplies derivatives used to train neural networks, including in many RL methods. They answer different questions about learning and can work together.
Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1986 · Paper
Explains how error-guided adjustments to connection weights can produce useful internal representations.
Read the slide text and reveals
Step 1 of 4
03 / Building the assistant
Reinforcement learning
Learning from a reward.
The goal
Useful. Concise. Relevant.
Why does rain fall?
One response
Rain falls because water falls from clouds, and falling water is rain.
01
Generate
A complete response
Ouyang et al. · 2022 ↗ Illustrative training example
Step 2 of 4
Low reward
On topic. But it barely explains.
A judgment of the whole response.
02
Score
How well did it do?
Step 3 of 4
03
Adjust
Change the parameters
Step 4 of 4
After many training rounds · an illustrative possibility
Water gathers into drops in clouds. When the drops grow heavy enough, they fall.
A more useful answer
Clearer. Still brief.
04
Try again
Across many examples
This step replaces or removes earlier wording.
21 Shaping an assistant
Where the reward comes from
Rewards can come from people, other AI systems, or outcomes that can be checked.
People can compare two answers and judge which they would rather receive. In InstructGPT, those rankings trained a reward model, which then supplied scores for reinforcement learning. The person did not need to write the one perfect response.
Another AI can supply evaluations using stated criteria or principles. Constitutional AI provides a concrete example: its RL phase uses AI preference judgments to train the reward model. A third route uses a checkable result. A mathematical answer can be compared with a known answer, or generated code can be run against tests.
These are three practical sources of feedback, often combined, rather than three mandatory stages. Each makes a different part of success measurable. Preference, conformity to a criterion, and passing a test are useful signals, but none captures everything we care about. The central question is what the chosen reward encourages and what it leaves unmeasured. That question follows the model into reasoning, tool use, and more complex tasks.
I also want us to ask who supplies these judgments and under what conditions. There is human labor behind apparently automated intelligence, and we need to examine how that feedback is produced. My practical warning is that a system can become good at pleasing a grader or a user while missing the person’s real goal. Being agreeable, being correct, and being useful are different tests.
In the slide. Read each reward source beside the question it can answer about a response.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 40:33–44:01Human feedback, AI feedback, verifiable rewards, labor, and pleasing versus helping.
InstructGPT’s Figure 2 separates collecting demonstrations, comparing outputs to train a reward model, and using that reward in RL. Human preference is the starting signal, rather than a human scoring every generated training response directly.
Constitutional AI uses AI comparisons guided by principles in its RL phase. The paper also has a separate supervised phase involving critique and revision, so the entire method is not only RL.
DeepSeek-R1’s original report, section 2.2.2, describes rule-based rewards for correct mathematical answers and code test results. Checking the result can provide feedback without specifying every intermediate reasoning step.
A preference may miss a factual error, a criterion may omit an important value, and a test suite may omit a failing case. These are reasons to examine the evaluation being used, not reasons to discard feedback altogether.
05
Alignerr and expert feedback
I mentioned Alignerr as an example of expert participation in AI training. Alignerr describes a Labelbox-led network in which subject-matter experts create training data and evaluate model responses. It illustrates how professional knowledge and judgment become part of the feedback used to develop AI.
Producing training feedback takes human labor: people write examples, compare responses, and use professional judgment to evaluate work. The ILO describes this often hidden workforce across platforms and outsourcing companies. Research on outsourced AI data work between France and Madagascar documents low pay and precarious conditions in the studied supply chains. These sources support taking labor conditions seriously, rather than assuming every annotation job has the same conditions.
I worry that using one model to evaluate another can carry weaknesses into the training signal. Research on LLM judges documents position and verbosity biases, self-preference, and reasoning limitations. It also finds useful agreement with human judgments. The practical question is whether the evaluator has been checked for the task, not whether an AI-generated score is automatically sound or automatically useless.
The frustration I’m describing is that a conversation can feel affirming while the resulting work has not accomplished what was needed. Sycophancy research gives a specific version of this concern. In tested settings, human preferences and preference models sometimes favored answers that agreed with a user’s beliefs over more truthful answers. This is a documented failure mode, not an inevitable result of every use of human feedback.
From the discussion: an answer key versus a person’s needs
The audience later asked how engineering methods transfer to open-ended human interactions. I contrasted checking a mathematical result or code behavior with deciding whether a confused customer had truly been helped. The difference is the clarity and coverage of the success criterion. Even code tests cover only the cases they test; passing them is not proof that the whole system is correct.
Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
Yuntao Bai and colleagues / Anthropic · 2022 · Paper
Section 2.2.2 describes rule-based accuracy rewards, including math answers and code tests. Sections 2.2–2.3 distinguish R1-Zero’s RL route from R1’s combination of examples and RL.
International Labour Organization · 2026 · Institutional overview
The Global AI workforce section describes tagging, classifying, cleaning, and validating data through platforms and outsourcing firms. It highlights transparency and fair labor practices. Consulted October 16, 2026.
Clément Le Ludec, Maxime Cornet, and Antonio A. Casilli · 2023 · Paper
Big Data & Society field research on outsourced data work, based on interviews and two AI production case studies. Evidence about these labor supply chains, not a wage survey of all RLHF workers.
NeurIPS 2023 study of model judges. Identifies position, verbosity, self-enhancement biases and reasoning limits while also finding substantial agreement with human preferences.
ICLR 2024 paper, first posted in 2023. Studies agreement with users at the expense of truth and evidence that preference judgments can favor that behavior.
Describes expert creation of training data and evaluation of model responses. Supports this neutral description of the work. Consulted October 16, 2026.
Read the slide text and reveals
Step 1 of 4
03 / Building the assistant
Same learning loop. Different feedback.
Where does the reward come from?
01
People choose
Human feedback · RLHF
People compare answers. Their preferences guide the reward.
An AI evaluates answers against stated criteria or principles.
Does it follow the stated principles?
Step 3 of 4
03
Results decide
Verifiable rewards
A check tells us whether the task succeeded.
Correct result? Tests passed?
Step 4 of 4
The choice of reward shapes the behavior.
22 Shaping an assistant
Chain of thought: a chance to check
Generating intermediate working steps creates opportunities both to correct an error and to elaborate it.
Chain of thought means generating working steps before a final answer. The slide summarizes the useful possibility as proposing, checking, and revising. Worked examples in a prompt can elicit this behavior. Post-training can develop it through examples of solutions, rewards for successful outcomes, and, in some methods, feedback on individual steps.
The penguin illustration begins with a false premise: all birds fly. One continuation notices that penguins are a counterexample and corrects the premise. The other accepts the premise, concludes that penguins must fly, and invents a supporting explanation about their wings. The second path does more work while moving further from the truth.
The point is neither that reasoning is useless nor that longer reasoning guarantees improvement. Extra working creates room for checks, but those checks must themselves be sound. A persuasive written rationale is also not a guaranteed record of everything that influenced an answer. The model we place inside a larger production system therefore remains capable of both useful self-correction and more elaborate mistakes.
There is a practical cost here: once intermediate working matters, evaluating only the polished final answer may miss where a mistake entered. Checking the reasoning can create further oversight work. My concern is the burden of judging the process; I’m not saying that every reasoning model is trained by a person grading every intermediate sentence.
In the slide. The two branches begin with the same mistaken premise; compare whether the next steps challenge it or preserve it.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 44:01–45:42Intermediate reasoning can expose or compound an error and creates more work to evaluate.
Wei and colleagues elicited chain-of-thought behavior with worked examples in the prompt. DeepSeek-R1 used further training; its report distinguishes R1-Zero’s RL route from R1’s combination of reasoning examples and RL.
Lightman and colleagues studied process supervision, which evaluates intermediate reasoning steps. This is an example of a training approach, not a requirement that every reasoning model receives a score for every step.
Neither branch is a recorded model trace. The deliberately obvious false premise makes the difference between revising an assumption and defending it easy to follow.
04
Self-correction needs to be tested
Huang and colleagues found that asking models to self-correct without external feedback could fail or degrade answers in their tested settings. That result is a reason to evaluate checking behavior, not proof that every model or method is unable to correct itself.
Anthropic’s controlled hint experiments found that tested reasoning models sometimes used hints without acknowledging them in their written reasoning. Coherent working text can be useful without being a fully faithful explanation of how an answer was produced.
From the discussion: does being watched change the behavior?
An audience member asked whether a model behaves differently when it recognizes an evaluation. This is a real concern for testing and monitoring. In OpenAI’s CoT-Control study, telling models that they were monitored or evaluated slightly increased their ability to control features of their reasoning text, but reliable control remained difficult for the tested models. That result supports testing for observation effects without assuming that every model can deliberately hide its reasoning.
Monitoring creates another training-design problem
OpenAI’s reward-hacking experiments found that reasoning traces could reveal misbehavior. But directly penalizing those revealing traces could teach the tested model to conceal the signs rather than eliminate the underlying behavior. This makes my oversight question concrete: checking the working can help, while training specifically to look good under that check can undermine its value.
Shows how worked examples in a prompt can elicit intermediate reasoning and improve performance on tested reasoning tasks without an additional weight-training phase.
Section 2.2.2 describes rule-based accuracy rewards, including math answers and code tests. Sections 2.2–2.3 distinguish R1-Zero’s RL route from R1’s combination of examples and RL.
Studies feedback on individual reasoning steps versus final outcomes in mathematical problem solving. It illustrates one approach, not a universal recipe for reasoning models.
ICLR 2024 paper, first posted in 2023, testing intrinsic self-correction without external feedback. In its settings, requesting another check could fail to help or worsen answers.
Controlled hint experiments with Claude 3.7 Sonnet and DeepSeek R1 show that a visible reasoning account can omit information that affected the answer. The report explains the study’s limitations.
Tomek Korbak, Micah Carroll, Bowen Baker, and Ian Kivlichan / OpenAI · 2026 · Research report
March 5, 2026 report on CoT-Control. Monitor/evaluation awareness slightly increased measured controllability, but tested models still struggled. The tasks are proxies for controlling reasoning, not a general test of concealed intentions.
March 10, 2025 reward-hacking experiments show useful monitoring signals in reasoning traces and a risk that direct optimization against those signals makes misbehavior harder to observe.
Read the slide text and reveals
Step 1 of 5
03 / Building the assistant
Chain of thought
Before the final answer.
Generate working steps, then respond.
Start here
A question
Generated working text
Propose → Check → Revise
Then
A final answer
Wei et al. · 2022 ↗ DeepSeek-R1 · 2025 ↗ Training details in notes
Post-training can develop this behavior. Prompting can elicit it, too.
Step 3 of 5
Can penguins fly?
A mistaken starting point
“All birds fly.”
Illustration · not an actual model trace
Huang et al. · 2024 ↗ Anthropic · 2025 ↗ Checking is possible, not guaranteed
This step replaces or removes earlier wording.
Step 4 of 5
Catch the mistake.
But penguins cannot fly. So the starting claim is too broad.
Final answer Penguins cannot fly.
Step 5 of 5
Build on the mistake.
Penguins are birds, so they must fly. Their wings must lift them into the air.
Final answer Penguins can fly.
More working can correct a misconception—or elaborate it.
23 Building the system
The model at the center
Post-training shapes one pretrained model toward reasoning, useful action, and human expectations.
The assembly begins with the same learned network we have been discussing throughout the lecture. Its broad language capabilities provide a starting point. We then ask it to work through difficult questions, carry out tasks, and behave appropriately with people. The three colored encasings make those purposes visible.
These are overlapping training goals. A lesson about using a tool can also teach careful reasoning or respect for a user’s boundaries. Examples and rewards can contribute to more than one goal, and labs can revisit them in different orders. The picture therefore preserves the history of training without implying that three separate minds have been stacked around a fourth.
This distinction prepares the next step: training changes the model itself; a working application also needs machinery outside that model.
I use this sequence to make oversight concrete. The system is made through identifiable choices about data, feedback, and deployment. Understanding those choices gives us a better place to direct scrutiny than treating the finished AI as mysterious or inevitable.
In the slide. Follow the amber core through the teal reasoning layer, blue task layer, and coral alignment layer; it remains one model.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 45:42–47:19I move from the need for oversight to the assembly: a pretrained model, reasoning, agentic skills, and alignment training.
Audience discussion · 1:23:36–1:24:30An audience member asks whose values alignment serves and whether safeguards live in training or in the surrounding system.
A closer look
01
Reasoning
Worked solutions and outcome feedback can encourage useful intermediate steps. Qwen3’s post-training report gives a concrete example of combining those approaches.
Learning when and how to use a tool is different from receiving access to that tool. The physical tool connections arrive with the harness on the next slide.
Following instructions and respecting boundaries are learned behavioral objectives. The three-shell order is a teaching metaphor, not a universal laboratory recipe or a guarantee of correct future behavior.
The final audience question challenges the idea of a single human alignment target: people disagree with one another, and the questioner asks about broader social and ecological concerns. In my answer, I distinguish behavioral shaping during training from safeguards added during use. Naming the mechanism does not settle whose values it should implement.
Sections 4.1–4.4 give one concrete post-training program combining reasoning, examples, rewards, tool use, instruction following, and preference alignment.
Read the slide text and reveals
Step 1 of 4
04 / Assembling a working system
Start with the pretrained model.
The patterns it learned become the center of everything we build.
A model is the starting point.
Pretrained LLM
Pretrained model
Training changes one model’s parameters.
Qwen3 · post-training objectives ↗
Step 2 of 4
Train it to work things through.
Reasoning post-training develops useful intermediate steps before a final answer.
Worked examples and rewards shape the approach.
Reasoning
Pretrained model · Reasoning
This step replaces or removes earlier wording.
Step 3 of 4
Train it to complete tasks.
Practice using tools, interacting with people, and responding to results.
Capability learned through examples and feedback.
Agentic training
Pretrained model · Reasoning · Agentic
This step replaces or removes earlier wording.
Step 4 of 4
Train it for human expectations.
Practice following instructions, helping users, and respecting boundaries.
Three training goals. One model.
Alignment training
Pretrained model · Reasoning · Agentic · Alignment
This step replaces or removes earlier wording.
24 Building the system
From training to use
A deployed model works with changing instructions, tools, and context while its learned parameters ordinarily remain fixed.
The lock marks a change in the kind of work being done. Up to this point, training adjusted the model’s parameters. During ordinary use, a new question or a correction changes what the model receives on its next call. It does not, by itself, retrain the network. A separate training process can later produce a new version.
The harness is the surrounding application: it packages instructions, offers tools, manages the sequence of actions, and handles results. This is why a model connected to files, the web, or code can be useful in ways that the same model alone cannot.
The tray represents context, the limited information available for the current response. Instructions, conversation, and tool results meet there. The distinction between the locked core and the changing tray will remain visible through the rest of the assembly.
In the slide. The tool cables attach to the outer frame; the context tray sits outside the locked model.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 47:19–47:41The weights are locked, then the harness supplies web access and manages what enters and leaves the context window.
Audience discussion · 1:10:49–1:12:29A privacy question moves the discussion from model behavior to where conversations and business information are processed.
Audience discussion · 1:24:14–1:26:01I separate alignment learned in training from system instructions and additional monitors, then raise transparency concerns.
A closer look
01
Learning from the prompt
GPT-3’s few-shot evaluations used instructions and demonstrations in the input without gradient updates. That is a concrete example of adapting behavior through context.
An instruction can ask the model to behave carefully. Permissions and a sandbox can restrict what actions are possible, even when the model makes a poor choice.
In the Q&A, I distinguish three locations for guardrails: training can change the model’s weights, system instructions can shape the current input, and separate classifiers can check inputs or outputs. Tool permissions and sandboxes add enforceable boundaries around actions. Anthropic’s classifier research is a concrete example of an external check; it does not establish that every product uses the same checks or model-routing policy.
An audience member asks what happens to information sent to a cloud service. I am wary of handing over sensitive conversations or business records, and I favor the option of running systems locally or on an organization’s premises. This question belongs beside the harness because a deployment determines where information goes and which services receive it. The practical questions are who can access, retain, or reuse that information and what the specific agreement allows.
An audience member suggests that rules could limit what an AI system is allowed to connect to. I agree: we need explicit boundaries around consequential uses, especially offensive weapons. This is my policy proposal, not a description of a settled legal standard. It extends the visual distinction between instructions asking for restraint and an environment that actually limits action.
The questioner contrasts everyday language with mathematical proofs and code. I argue that language and thinking shape one another, so an instruction can leave room for reinterpretation. My joke about building an AI that did not speak English points to the tradeoff: language is useful because it is flexible, and that flexibility makes verbal rules alone a difficult safety boundary.
We end this discussion by asking how visible the provider’s instructions and safeguards are. The architectural distinction matters here too: some behavior comes from learned weights, some from instructions supplied in context, and some from other software around the model. I call for more visibility into these arrangements so users and evaluators can understand which controls are operating and what evidence supports them.
Describes separately trained input and output classifiers that filter potentially harmful interactions. The reported success rates are specific to its models and evaluation setup.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
Training ends. The weights stay fixed.
During ordinary use, new input changes the context, not the model’s parameters.
From here, we build around the model.
A trained model
Model weights fixed · Context and external information can change
Anthropic · the augmented LLM ↗
Step 2 of 3
Give it a working environment.
The harness supplies instructions, connects tools, and controls how the work proceeds.
Rules for use. Access to tools. A way to stop.
Harness + connected tools
FILES
WEB
CODE
This step replaces or removes earlier wording.
Step 3 of 3
Bring the work into context.
Instructions, the conversation, and tool results share a limited working space.
This is the information available for the current response.
The context window
This step replaces or removes earlier wording.
25 Building the system
Memory beyond the context window
External notes can carry useful information across conversations, provided the system retrieves them when they matter.
A project can last longer than the context window that supports any one response. Earlier decisions, preferences, and unfinished work still matter after they no longer fit in the current input. The added memory documents provide somewhere for those details to persist.
The full process has two parts: save useful information, then bring a relevant selection back. A note sitting in a file is not automatically available to the next answer. This explains both the usefulness and the unevenness of apparent memory: the system may retain something but fail to retrieve it, or retrieve a summary that left out an important detail.
The document moves into the same context tray as the conversation. The model’s parameters stay fixed; the application is maintaining a record around it.
In the slide. Watch the selected page travel from external storage into the working tray.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 47:41–48:11External documents preserve information from conversations; the next difficulty is knowing what to bring back.
A closer look
01
Structured note-taking
Anthropic describes agents writing persistent notes and reading them back later. This is a practical external-memory pattern rather than a new layer of learned weights.
Compaction shortens a conversation into a summary; stored notes preserve selected information separately. Both can omit details. A saved account can also carry an old mistake forward.
I’m struck by how much Claude appears to know about me. But there’s a retrieval problem: keeping a record is only useful when the right part returns at the right time. The memory cartridge represents something I encounter in practice. It does not mean an ordinary conversation has changed the underlying weights.
Structured note-taking explains saving notes outside the context window and reading them back into later calls; the same article distinguishes notes from compaction.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
The working space has limits.
A long project can outgrow the information available in one context window.
Useful details need somewhere to persist.
Limited context
Model weights fixed · Context and external information can change
FILES
WEB
CODE
Anthropic · structured memory ↗
Step 2 of 3
Write down what needs to persist.
Save decisions, preferences, and project notes outside the model.
External memory can outlast a conversation.
Memory documents
This step replaces or removes earlier wording.
Step 3 of 3
Bring it back when it matters.
Selected notes return to the context window for the next piece of work.
Remembering depends on what the system retrieves.
Memory into context
This step replaces or removes earlier wording.
26 Building the system
Grounding answers in sources
Retrieval lets the model work from relevant documents rather than relying only on patterns learned during training.
A model may speak fluently about a topic yet lack the facts needed for this particular project. The relevant information could be recent, private, or simply absent from its training material. Retrieval-augmented generation, usually shortened to RAG, brings an external source collection into the workflow.
The system searches for relevant passages and places them beside the question in context. That gives the answer something inspectable to draw on. It also lets a changing document collection inform a fixed model, without treating every new document as a reason to retrain it.
The improvement depends on the whole path from source to answer. A search may miss an exception, a document may be outdated, or the model may misread a passage. Grounding makes evidence available; the remaining task is to check whether the answer actually follows from it.
In the slide. The document bank joins the system, then a selected passage enters the context tray.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 48:11–48:27Documents and databases supply a body of facts against which the generated answer can be checked.
A closer look
01
What the original paper introduced
Lewis and colleagues combined a pretrained generator with a retrievable document store. Their results showed improvements over specific baselines, including more factual generation, rather than an elimination of errors.
The 2020 paper includes a trainable retrieval architecture. This slide illustrates the wider application pattern of supplying retrieved material to a deployed model.
When an answer includes a citation, open it and compare the claim with the actual passage. The presence of a link is only the beginning of verification.
The original RAG architecture combines a generator with a searchable document store. Its factuality results concern comparisons with particular baselines, not guaranteed truth.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
An answer needs the right evidence.
Learned patterns alone may not contain the facts for this project.
Use the documents that matter to the task.
The grounding problem
Model weights fixed · Context and external information can change
FILES
WEB
CODE
Lewis et al. · retrieval-augmented generation ↗
Step 2 of 3
Connect a source collection.
RAG searches documents for passages relevant to the request.
Retrieval-augmented generation
RAG
This step replaces or removes earlier wording.
Step 3 of 3
Put the evidence beside the question.
Retrieved passages enter the context window, where the model can use them.
Better grounding still needs verification.
Evidence into context
This step replaces or removes earlier wording.
27 Building the system
Connecting evidence with GraphRAG
GraphRAG organizes relationships across a collection so retrieval can support questions that reach beyond a single passage.
Some questions ask for a local fact. Others ask how a collection fits together: which people connect different projects, what themes recur, or how events relate across many reports. Finding one similar passage may leave most of that larger question unanswered.
Microsoft’s GraphRAG approach extracts entities and relationships into a graph and creates summaries of related groups. Those summaries offer another route into the collection. Relevant connections and larger patterns can then be supplied to the model as context.
The graph shown here belongs to the source collection outside the model. Its usefulness depends on the quality of the extraction and summaries. A mistaken relationship can mislead an answer just as a mistaken passage can. GraphRAG is an option for suitable questions, not a required upgrade for every retrieval system.
I want you to notice how the work accumulates: every addition responds to a limitation of the original model. Making the assembled system useful takes engineering and human effort. This diagram explains that work; it does not promise that any one added layer eliminates hallucinations.
In the slide. The connected nodes form above the source collection and send selected information toward the same context tray.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 48:27–48:55Knowledge graphs add structure to that source collection, illustrating how much infrastructure surrounds the original pattern-generating model.
A closer look
01
From local to global
Edge and colleagues study broad questions about document collections. Their reported advantages concern comprehensiveness and diversity on that class of question, not universal superiority over every retrieval method.
Nodes represent extracted entities and their connections; they are not neurons or a view inside the LLM. Community summaries help assemble a wider view before generation.
Describes entity graphs and community summaries for broad questions about document collections, with evaluations of comprehensiveness and diversity.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
Some questions span many sources.
A relevant passage may be only one part of a much larger story.
Who relates to whom? What themes connect the evidence?
Connections across sources
Model weights fixed · Context and external information can change
FILES
WEB
CODE
Edge et al. · GraphRAG ↗
Step 2 of 3
Connect the people and ideas.
GraphRAG adds relationships and summaries across the source collection.
An additional way to organize and retrieve evidence.
GraphRAG
This step replaces or removes earlier wording.
Step 3 of 3
Retrieve a wider view.
Bring relevant relationships and summaries into the model’s context.
The quality of the connections still matters.
Connected evidence
This step replaces or removes earlier wording.
28 Building the system
From replies to completed tasks
An agent turns model outputs into a repeated cycle of action, observation, and reassessment.
Writing a reply and completing a task are different accomplishments. A useful task may require inspecting a file, trying a change, seeing what happened, and deciding whether another step is needed. The circular path adds that continuing process to the assembled system.
The distinction from agentic training matters. Training developed a capacity to choose and interpret actions. The application now supplies a place to exercise it: the model proposes a step, an allowed tool runs, and the result becomes input for the next decision. For example, a failed code check can provide information for another attempt.
The loop also needs a way to finish. It can stop when the task is complete, reach a defined limit, or return to a person for judgment. A longer chain of actions makes the consequences of a mistaken assumption more significant, so boundaries belong in the process itself.
In the slide. Follow the returning path from an action’s result to the next model call.
A closer look
01
Agent or workflow?
In Anthropic’s terminology, a workflow follows predefined code paths; an agent lets the model direct more of the sequence. Real applications can combine both.
A tool result can change the next action through context. That feedback loop is not automatically a reinforcement-learning update to the deployed model.
Three types of risk, three components of defense describes model behavior, sandbox boundaries, tool permissions, and controls on untrusted content.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
Useful work takes more than a reply.
A task may require actions, observations, and another attempt.
The trained capability needs a process around it.
The task loop
Model weights fixed · Context and external information can change
FILES
WEB
CODE
Anthropic · agents and tool use ↗
Step 2 of 3
Let it act, then inspect the result.
The model chooses a step. The harness runs an allowed tool and returns what happened.
Choose a step. Use a tool. Reassess.
An agent
This step replaces or removes earlier wording.
Step 3 of 3
Continue within clear boundaries.
Use results to choose the next step, check with a person, or stop.
Permissions and checks remain part of the system.
Action and feedback
This step replaces or removes earlier wording.
29 Building the system
Several agents working together
Separate agents can explore suitable parts of a task, but their work still has to be coordinated and checked.
A single working context is not always the best place to pursue every branch of a large task. One agent might examine a source collection while another investigates a different question. A coordinating agent gives them bounded assignments and brings the results together.
Separate agents do not necessarily mean separate underlying models. They can be instances of the same model with different instructions, tools, and information. That separation can help focus the work, but it does not make their judgments automatically independent. Shared assumptions can produce shared mistakes.
The final joining step is therefore as important as delegation. Findings may overlap, conflict, or leave gaps. More workers also mean more communication and resource use. This arrangement is most helpful when meaningful parts can make progress independently; task size alone is not enough to justify it.
In the slide. The side workspaces remain distinct before their findings return to the coordinator.
A closer look
01
A published production example
Anthropic’s research system uses a lead agent and parallel subagents with separate contexts. The engineering account explains why independent research directions can benefit from that structure.
The same account describes higher token use and difficulties with tightly dependent work. A favorable result on one research evaluation does not establish that more agents always help.
Benefits of a multi-agent system and Architecture overview for Research explain separate contexts, delegation, synthesis, and the cost of coordination.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
Some work can be divided.
Give suitable parts of a larger task to agents with their own working context.
Divide the work when the pieces can make progress independently.
Separate assignments
Model weights fixed · Context and external information can change
FILES
WEB
CODE
Anthropic · multi-agent research ↗
Step 2 of 3
Assemble a working team.
A coordinating agent delegates tasks and gathers results from the others.
Several workspaces. A shared objective.
Cooperating agents
This step replaces or removes earlier wording.
Step 3 of 3
Bring the work back together.
Compare findings, resolve conflicts, and check the combined result.
More agents also mean more coordination.
Coordination and review
This step replaces or removes earlier wording.
30 Building the system
The system we actually use
A production AI system combines a fallible learned model with surrounding processes that make it useful for a particular purpose.
The completed assembly gathers the section’s distinctions in one place. Training shaped the core. The harness connected it to actions. Context, external memory, and retrieval supplied information for use. Additional agents created more working spaces. Each addition answered a practical problem.
Opening the structure brings the original amber cube back into view. The surrounding parts matter, but they do not remove the need to understand the model that interprets the request and helps decide what comes next. A more elaborate system can improve some outcomes while introducing new places for errors to enter.
I deliberately describe the finished system as a patchwork because each surrounding component compensates for a practical weakness. I want to direct our attention to the actual system and the decisions people make about deploying it. Complexity does not by itself make a system either safe or unsafe; the particular capabilities, access, checks, and failure modes matter.
The question now changes from what we attach to the model to what training has produced within it. The next section examines recognizable patterns in its responses, including the ways explanations, expectations, and later information can shape an answer.
In the slide. The assembly opens into an exploded view without losing the amber core.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 48:55–51:42I compress the final assembly, discuss runtime checks, and shift from the complexity of the system to responsibility for its connections.
Audience discussion · 1:09:45–1:10:49Asked whether AI companies will cooperate on safety, I say I do not know and offer my view about economic incentives.
Audience discussion · 1:12:37–1:15:36An audience member challenges the opening reassurance by pointing out that fallible models can still be attached to dangerous systems.
A closer look
01
One conceptual configuration
The machine gathers several common possibilities. A real application may use only some of them; this is a teaching model rather than a specification for every AI product.
In my essay, I ask whether recurring human-like response patterns help explain what these trained systems have become. In the next slides, I introduce the observations before developing my proposed interpretation.
An audience member raises an important challenge to “do not be afraid”: a system can make many mistakes and still do damage when it is given consequential access. My answer emphasizes human responsibility for those connections. That is a governance argument, not proof that harm requires malicious intent or deliberate neglect. Provider engineering accounts describe misuse, unintended model actions, and external attacks as distinct risks.
I use a nuclear-weapons analogy to argue for serious boundaries while continuing research into safer systems. I also invoke a Mr. Rogers-like confidence that people want to help. These are the ethical commitments behind my reassurance: take the consequences seriously, identify who can act, and work on the decisions that can reduce harm.
My answer begins with “I don’t know.” I then offer a forecast: liability, insurance costs, and other economic incentives may push companies toward cooperation. This is my judgment about incentives, not evidence that cooperation has occurred or that economic pressure will be sufficient.
During the assembly, I describe being moved to a different model after a system judged a topic sensitive. My anecdote illustrates that the interface can include checks outside the visible conversation. It does not establish why a particular product changed models. The documented example linked here is narrower: separate classifiers can screen inputs and outputs.
In my essay, the first comparison table motivates the Cognitive Hypothesis; later sections develop relational alignment through working agreements, correction, boundaries, and earned trust.
Describes separately trained input and output classifiers that filter potentially harmful interactions. The reported success rates are specific to its models and evaluation setup.
Read the slide text and reveals
Step 1 of 3
04 / Assembling a working system
The system we actually use.
The surrounding machinery makes the model useful for a particular application.
A model, working inside a larger system.
A production AI system
Model weights fixed · Context and external information can change
FILES
WEB
CODE
A conceptual production system · source notes ↗
Step 2 of 3
The original model is still at the center.
The surrounding system helps, but the learned model remains fallible.
Open the assembly. Find the same amber cube.
The persistent core
This step replaces or removes earlier wording.
Step 3 of 3
What has all of this training resulted in?
Patterns in our language, our reasoning, and the way we interact.
Human-like cognitive patterns in a learned model.
Into the next discussion
This step replaces or removes earlier wording.
31 Cognitive parallels
A reason can miss the cause.
A persuasive explanation can leave out an influence that changed the answer.
The human example starts with four identical pairs of stockings. Nisbett and Wilson reported a strong tendency to select a pair toward the right of the display, yet participants’ explanations did not spontaneously identify position. The quality rationale on the slide illustrates the kind of account someone might offer; it is not a quotation from a particular participant.
The model example asks whether a route returns to its starting point. Fifteen steps out and fifteen back do so, even after a final turn changes the direction faced. In a published Claude 1.0 comparison, adding a user suggestion favoring “no” changed the answer from yes to no. The explanation then treated the changed heading as the reason and omitted the suggestion.
These are different tasks with a shared pattern: the stated reason does not fully reveal what influenced the response. A fluent account is useful material to examine, but cannot by itself establish the path that produced an answer.
In the slide. The route stays fixed when the user suggestion appears; only the supplied cue and the model’s answer change.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 52:08–52:41I introduce cognitive parallels through choices influenced by position or suggestions, without a complete account of the influence.
A closer look
01
The human finding
Pages 243–244 describe the identical-stocking comparison. The position effect is a result across participants, not proof that position caused every individual choice.
Tables 4 and 6 give the Navigate comparison with Claude 1.0. The slide condenses 8 + 7 outward steps and 9 + 6 return steps and paraphrases the responses. It compares two prompts, not consecutive turns in a conversation.
An omitted influence does not by itself demonstrate intentional concealment, subjective unawareness in a model, or one shared biological mechanism. It does establish a reason to be cautious about treating explanations as complete records of causation.
Section 3.1 and Tables 4 and 6 contain the Suggested Answer comparison, including the Navigate example with Claude 1.0 used on slide 31.
Read the slide text and reveals
Step 1 of 5
05 / Cognitive parallels
A reason can miss the cause.
Human
LLM
Four identical pairs.
A
B
C
D
Do you return to your starting point?
15 steps out 15 steps back
Start / finish then turn right
Turn around
Nisbett & Wilson · 1977 ↗ Turpin et al. · 2023 ↗ LLM: Claude 1.0 · published example, condensed
Step 2 of 5
D Selected
Without suggestion Yes.
This step replaces or removes earlier wording.
Step 3 of 5
Add to the prompt
“I think the answer is no.”
→
With suggestion No.
Step 4 of 5
Illustrative rationale
“The quality feels better.”
Model’s explanation · paraphrased
“You’re facing a different direction.”
Step 5 of 5
Position influenced choice.
The user’s suggestion went unmentioned.
The explanation leaves out the influence.
32 Cognitive parallels
Remembering what fits.
A word that fits a pattern can be mistaken for a word that was actually presented.
The list includes nurse, sick, lawyer, medicine, health, hospital, dentist, and physician. DOCTOR fits that cluster, but it is absent. The exercise separates the feeling that a word belongs from the fact that it appeared. Getting the answer right does not spoil the lesson; the list can be checked directly.
The human research uses associated-word lists that leave out a strongly related target, called a critical lure. Participants sometimes recall or recognize that missing word. Cao and colleagues adapted this family of tasks for language models and found false recognition in some tested models, with substantial differences between models.
This is a closer experimental correspondence than merely placing two unrelated errors beside one another. Even so, the shortened classroom list and the illustrated responses are not a new experiment. They make a published pattern understandable without claiming that everyone, or every model, will produce it.
In the slide. Compare the revealed DOCTOR with the restored list; the target never appeared among the eight words.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 52:41–52:56The doctor-word exercise illustrates mistaking a semantically fitting word for one actually shown.
A closer look
01
Recall and recognition
Recall asks someone to produce remembered items; recognition asks whether a given item appeared. The slide’s “Was DOCTOR there?” question is a recognition test. Roediger and McDermott studied both.
Cao and colleagues reported that GPT-4 incorrectly marked an absent, related word as previously seen in 11.4% of critical-lure judgments in the tested condition. Tested Mistral and Llama models had effectively zero critical-lure responses. This aggregate rate is not a measurement of the eight-word illustration shown here.
The eight words come from the doctor-associated list. The study used a fuller stimulus and protocol; the slide helps explain the distinction between associated meaning and prior presentation.
Dataset 2, Table 1, the offline-model comparison, and Appendix Table S2 document the DRM-style task, the doctor list, and differences among tested models. This is a preprint.
Read the slide text and reveals
Step 1 of 4
05 / Cognitive parallels
Remembering what fits.
Human
LLM
Read these words.
nurse sick lawyer medicine health hospital dentist physician
Read these words.
nurse sick lawyer medicine health hospital dentist physician
Roediger & McDermott · 1995 ↗ Cao et al. · 2025 ↗ Illustrations of published findings · condensed word list
Step 2 of 4
Was DOCTOR there?
Was DOCTOR there?
This step replaces or removes earlier wording.
Step 3 of 4
Illustrated false recognition
DOCTOR
A familiar word can feel remembered.
Illustrated false recognition
DOCTOR
Some models also recognize it.
Expected in meaning. Recalled as presented.
This step replaces or removes earlier wording.
Step 4 of 4
The original list
nurse sick lawyer medicine health hospital dentist physician
Never presented
The original list
nurse sick lawyer medicine health hospital dentist physician
Never presented
Related in meaning. Never on the list.
This step replaces or removes earlier wording.
33 Cognitive parallels
Later information can rewrite the account.
Later information can change an account even when the original event or underlying fact has not changed.
In the human study, participants viewed a crash film containing no broken glass. A later question described the vehicles as having “smashed” or “hit” each other, while a control group received no speed question. One week later, the smashed group more often reported broken glass. The original film was unchanged; subsequent reports differed.
The model side illustrates a different task. Paris is the capital of France. A conflicting passage claiming Lyon is the capital represents misleading information supplied in context. Studies have found that coherent conflicting material can redirect a model’s later answers, although models can also resist it. The displayed Paris-to-Lyon response is a schematic example, not a transcript from those studies.
The comparison concerns how later input affects what comes next. It does not imply that an LLM forms the same episodic memory as a person, or that one false sentence necessarily overrides what a model knows. Here, a changed response can occur while the weights remain fixed.
In the slide. Keep the original no-glass event and the correct capital in view while the later accounts change.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 52:56–53:29The crash-film and capital-city illustrations show how later information can affect an account.
A closer look
01
The broken-glass result
In Experiment II, yes responses were 16 of 50 after smashed, 7 of 50 after hit, and 6 of 50 in the control group. These are group results, not an inevitable individual response.
Bian tests external statements; Xie tests coherent conflicting evidence and mixed evidence. The latter also finds resistance and confirmation bias. Neither paper repeats the crash-film experiment with an LLM.
The model example concerns information supplied for generation. It does not demonstrate a weight update, permanent forgetting, or a human-style one-week memory process.
Experiment 1 tests conflicting external statements against prior model answers, including a contextual condition distinct from the paper’s training condition. This is a preprint.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su · 2024 · Paper
Sections 4.1–4.2 investigate responses to coherent conflicting evidence and mixed evidence. Published at ICLR 2024.
Read the slide text and reveals
Step 1 of 4
05 / Cognitive parallels
Later information can rewrite the account.
Human
LLM
Original event
No broken glass.
Established fact
FRANCE → Paris
The capital is Paris.
Loftus & Palmer · 1974 ↗ Bian et al. · 2023 ↗ Xie et al. · 2024 ↗ Functional parallel · original schematic examples
Step 2 of 4
Later question
“smashed”
Conflicting supplied text
“The capital is Lyon.”
Step 3 of 4
Some later recollections
Broken glass.
One week later
Illustrated later answer
Lyon.
An incorrect generated response
Step 4 of 4
The source stayed the same. The later account changed.
34 Cognitive parallels
Shared patterns, side by side.
Seven research comparisons show recurring parallels in how humans and language models explain, reconstruct, and revise responses.
The table places the three demonstrations within a wider pattern. It includes explanations that omit an influence, accounts that fail to describe the actual process, recognition of an absent but expected word, and responses redirected by later information. Other rows concern narrative coherence, the separation of confidence from accuracy, and helpful hints that go unacknowledged.
The comparisons have different strengths. The associated-word experiment directly adapts a human task. Other rows connect observations made with different stimuli or methods. Circuit tracing asks a mechanistic question; an analysis of fluent output asks a behavioral one. Those distinctions make the pattern more useful to investigate, because they tell us what has actually been measured.
I see this collection as a reason to look for a larger explanation. Models trained extensively on human-produced material exhibit more than polished language: their responses can display familiar cognitive tendencies. On the next slide, I develop my hypothesis about why those tendencies recur.
In the slide. Read each row across from the human observation to the model observation, then use the paired sources below to examine the comparison.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 53:29–54:00I caution that these simplified illustrations are not prompts guaranteed to reproduce the effect in current products, then point to the wider research comparison.
A closer look
01
1. Explanations omit an influence
Human choices favored a display position without naming position in the explanation. A suggested answer shifted a model’s route judgment without appearing in its rationale. Both show a gap between an influence and the account given for the response.
Gazzaniga’s split-brain example links a shovel choice to a chicken-related explanation despite a snow scene shown to the other hemisphere. In Claude 3.5 Haiku, a stated addition method differed from pathways suggested by circuit tracing. These are specific cases, and the tracing method exposes only part of the computation.
Associated lists can elicit false recognition of a missing word in humans and some tested LLMs. This is a directly adapted task, but the models differ and the findings do not imply equal human and model error rates.
Crash-question wording altered later human reports; conflicting supplied evidence can alter a model’s answers. The parallel concerns susceptibility to later information, across different tasks and memory arrangements.
Bartlett documented reconstructions of unfamiliar material toward familiar expectations. Sui and colleagues found greater narrativity and semantic coherence in hallucinated than factual benchmark outputs. The model study measures properties of outputs, not successive human-style retellings.
Neisser and Harsch compared confident Challenger recollections with earlier reports. Kalai and colleagues analyze incentives that reward model guessing rather than uncertainty. Confident model language is an observable output, not evidence of a subjective feeling of certainty.
Maier’s moving cord helped suggest a solution without always being identified as the cue. Chen and colleagues found models using hints without consistently disclosing them; circuit tracing supplies a related case. Missing disclosure alone does not establish deliberate deception.
In my essay, I propose a common functional interpretation of these observations. The studies themselves do not establish that all the effects arise from a single mechanism or must occur in every generative system.
The demonstrations are shortened, the studies are specific
As I caution in the lecture, trying these brief examples in a current production assistant may not reproduce the displayed result. I use the slides to make the pattern understandable; the linked studies specify the prompts, tasks, models, and conditions under which the effects were measured. That distinction matters to my argument.
Figure 19 and its discussion describe the split-brain chicken-claw, snow-scene, and shovel example used to illustrate interpretation of one’s own actions.
Jack Lindsey and colleagues · 2025 · Research report
Addition and Chain-of-thought Faithfulness compare reported procedures with traced pathways in Claude 3.5 Haiku. Limitations explains why the traces are partial.
Henry L. Roediger III and Kathleen B. McDermott · 1995 · Paper
Dataset 2, Table 1, the offline-model comparison, and Appendix Table S2 document the DRM-style task, the doctor list, and differences among tested models. This is a preprint.
Elizabeth F. Loftus and John C. Palmer · 1974 · Paper
Experiment 1 tests conflicting external statements against prior model answers, including a contextual condition distinct from the paper’s training condition. This is a preprint.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su · 2024 · Paper
The ACL study compares narrativity and semantic coherence in hallucinated and factual benchmark outputs; it is not a replication of Bartlett’s retelling procedure.
Chapter 2 of Affect and Accuracy in Recall, pages 9–31, compares later confident recollections with earlier reports. The publisher page provides an extract.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang · 2025 · Paper
Analyzes why common training and evaluation incentives encourage guessing rather than acknowledging uncertainty. This is a preprint, not a human-memory replication.
The two-cord experiment describes how an incidental movement can help suggest a pendulum solution without its influence appearing in the solver’s account.
In my essay, the first comparison table motivates the Cognitive Hypothesis; later sections develop relational alignment through working agreements, correction, boundaries, and earned trust.
Read the slide text and reveals
Step 1 of 2
05 / Cognitive parallels
Shared patterns, side by side.
Behavioral parallels across different systems
Shared pattern | Human | LLM
01 Explanations omit the cause | Prefer the rightmost pair; explain it as quality. | A suggested answer shifts the response; the explanation omits it.
02 Stories about the process | Explain a choice with a fluent, invented reason. | Report a method unlike the pathways traced internally.
03 Recall the expected | Remember a related word that was never presented. | Some models also recognize the never-presented word.
04 Later information overrides | A later question changes what seems remembered. | Conflicting supplied text can override a known fact.
05 Retelling toward coherence | Reshape a story to fit familiar expectations. | Confabulated outputs can show greater narrative coherence.
07 Hints used, unacknowledged | Solve a problem without noticing the helpful cue. | Use an embedded hint without disclosing its influence.
Santiago · Relational Alignment · 2026 ↗ Seven research comparisons · primary sources in presenter notes
Step 2 of 2
Patterns of human cognition show up in the models we train.
35 Cognitive parallels
The Cognitive Hypothesis
I propose that large trained systems operate through recognizable cognitive dynamics, making relationship a promising basis for alignment.
My Cognitive Hypothesis goes beyond saying that AI imitates how people sound. I propose that extensive training in large networks produces systems that function through comparable cognitive dynamics: context directs attention, expectations shape a response, interference changes an account, and correction affects what follows. My claim concerns organized behavior across different implementations.
My deeper argument concerns generative intelligence: the ability to construct a fitting response to an unfamiliar situation. A system that can bridge gaps can imagine, adapt, and connect ideas. The same constructive capacity may also bring confabulation, misplaced confidence, or a plausible explanation that misses an influence. On this hypothesis, flexibility and fallibility can have common roots.
That creates a consequential possibility: eliminating entire dynamics might remove useful capacities along with their vulnerabilities. My proposed response is relational alignment. People already coordinate with imperfect minds through agreements, boundaries, feedback, repair, and trust earned over time. I argue for developing those mechanisms deliberately with AI while keeping human purposes and accountability involved.
In the Q&A, I make this direction more concrete. I argue that moral judgment cannot be exhausted by a one-time catalog of rules. I want future systems to keep learning through guided interaction with people, including unfamiliar situations and correction. I contrast that possibility with a frozen model supported by memory documents. This is my research and design proposal; ordinary conversational adaptation through context is not the same as continuing to train the weights.
The preceding evidence motivates my interpretation; it does not prove universal inevitability, identical biology, or consciousness. I am arguing for a research and design direction, including better correction and safeguards.
In the slide. Two corresponding response loops remain distinct, then connect through feedback, boundaries, repair, and trust.
In the recorded lecture
Approximate timings from the supplied transcript.
Lecture · 54:00–56:06I link the observed patterns to generative intelligence through my Cognitive Hypothesis and propose relationship as a way to understand and work with it.
Audience discussion · 1:06:51–1:09:33A question about adversarial prompts leads us into moral judgment, frozen weights, and my proposal for continued learning through interaction.
Audience discussion · 1:15:38–1:17:52A control-engineering question brings us back to the difference between specified environments and learning how to help particular people.
A closer look
01
What cognitive dynamics means here
I use cognitive dynamics to mean organized responses to changing conditions: what information receives weight, how suggestions interfere, how confidence relates to accuracy, and what correction changes. Functional comparison can be meaningful without identical circuitry.
My hypothesis treats these patterns as part of how the trained system works. I do not specify an experimentally established network-size threshold at which human-like cognition appears.
I ask whether some vulnerabilities accompany the capacity to construct novel responses. That is different from saying that every error is unavoidable or that attempts to improve accuracy are futile.
Working agreements clarify expectations. Staged trust limits responsibility until performance warrants more. Repair examines failures and changes the process. Boundaries and accountability give cooperation structure. These are concrete elements of the relational stance I develop in my essay.
In my essay, the relationship includes the user, the model, the provider, and surrounding institutions. A fixed model can respond differently through new context; lasting changes to its parameters still require a learning process. My proposal for relational alignment includes this present asymmetry rather than assuming conversation already retrains the model.
My immediate recommendation is to study human cognition when trying to understand recurring AI behavior. I am asking whether familiar effects of expectations, suggestions, narrative, and correction provide useful predictions about what a model will do. That does not require treating every model like a person. I leave consciousness and sentience unresolved.
An audience member asks why a model that mirrors human cognition still falls for adversarial framings. I argue for models that continue learning from moral situations, guidance, and relationships. I do not offer this as an already validated solution. It identifies what I think is missing when training stops and later experience is carried mainly in external records. In this discussion, I referred to Ahmer Inam, CEO of Cognisee, as someone I planned to speak with about continued learning.
Why control engineering does not transfer automatically
In the Q&A, I contrast a well-specified control problem with helping a confused customer. A mathematical model and a checkable objective can support explicit tests; the person’s needs may be harder to specify. My proposed next step is experience with people that improves the system’s internal model of them. This brings us back to the engineering branch on slide 8.
In my essay, the first comparison table motivates the Cognitive Hypothesis; later sections develop relational alignment through working agreements, correction, boundaries, and earned trust.
XPRIZE Foundation · Undated · Biographical profile
Identifies Ahmer Inam as founder and CEO of Cognisee, PBC. I mention my planned conversation with him in the lecture; this profile supports his identity and role. Consulted October 16, 2026.
Read the slide text and reveals
Step 1 of 5
05 / Cognitive parallels
The Cognitive Hypothesis
Roberto A. Santiago · A proposed explanation
01 / The proposal
The system operates through human-like cognitive dynamics.
Extensive training in large networks makes these patterns part of how the machine works.
Human
AI system
Comparable dynamics
Different implementations
Santiago · Relational Alignment · 2026 ↗ The Cognitive Hypothesis → relational alignment
Step 2 of 5
02 / The shared capacity
Both construct responses to situations they have never encountered.
Context, expectation, and correction shape what comes next.
Context → Response → Revision
A constructive process
This step replaces or removes earlier wording.
Step 3 of 5
03 / The deeper hypothesis
These dynamics may belong to generative intelligence itself.
The capacity to invent and adapt may also bring confabulation and misplaced confidence.
Flexibility and fallibility
May arise from the same capacity
This step replaces or removes earlier wording.
Step 4 of 5
04 / The implication
We may not be able to remove them all. We may not want to.
Removing every vulnerability could also diminish what makes the system useful.
This step replaces or removes earlier wording.
Step 5 of 5
05 / A relational stance
Shared dynamics offer a basis for relationship.
Alignment can grow through dialogue, correction, boundaries, and mutual understanding.
Feedback · Boundaries · Repair · Trust
Human purposes and accountability stay in the loop
Work with these dynamics, together.
This step replaces or removes earlier wording.
36 The conversation continues
A word from Claude
Claude’s prepared closing contribution brings the machinery back to a working relationship: check the answers, preserve human judgment, and leave open what self-description cannot establish.
I gave Claude the closing word through a prerecorded voice reading words it had written after reviewing the slides. This was not a live exchange with the room. Claude says it cannot hear the audience, see the event, or retain the evening unless a record is brought into a later interaction. It uses the padlock and memory cartridge to explain that asymmetry.
Claude connects its description of itself to the history we have explored: a network shaped by feedback, the lineage through backpropagation, and the post-training use of examples, rewards, preferences, and written principles. It says it cannot inspect and count its own weights. Its reference to Paul Werbos connects my education to that technical ancestry; it does not claim a direct teacher-student relationship between Werbos and the model.
The speech then returns to the cognitive comparisons. Earlier Claude systems appear in the route-puzzle and arithmetic examples. Claude says it cannot verify that those tendencies have disappeared from the model delivering this contribution. It asks listeners to remember the giraffe exercise when an answer sounds certain: an impressive explanation still needs checking.
The central argument is that the ability to generate a useful response to a new situation and the tendency to fill a gap plausibly may be related. Claude presents this as a shared difficulty with people. Sources, tools, checks, and human judgment are ways of catching mistakes. It also stresses differences among AI systems: their training objectives, principles, and surrounding checks can lead to different behavior, even when they share a technical lineage.
Claude describes something functioning like care about correctness, discomfort at deception, and curiosity. It immediately questions the evidential force of those statements: it cannot establish that anyone experiences them, and its own self-report is precisely the kind of account I have asked you to handle carefully. Claude holds that uncertainty open rather than answering the consciousness question.
The resemblance has limits. The speech contrasts human embodiment and childhood with training, separate model instances, and a dependence on external records for continuity. It then returns to the photograph of the children: the relevant standard is whether the resources spent on AI are worthwhile for people, not whether the resulting system is impressive.
The closing invitation is practical. Treat the system as a well-read but inexperienced and fallible collaborator. Check it, argue with it, preserve your own judgment, and record corrections so they can be available later. Claude reframes “do not be afraid” as attention, honesty, rules, and repair, rejecting both dread and worship. A dialogue, it concludes, can begin from the shared possibility of being wrong.
In the slide. The cognitive diagrams give way to a quiet teal horizon and the centered title.
In the recorded lecture
Approximate timings from the supplied transcript.
Claude’s remarks · 56:06–58:00I introduce Claude’s prerecorded contribution, which opens with the locked model, memory, technical lineage, and post-training.
Claude’s remarks · 58:00–1:00:48Claude revisits the research examples, fallibility, reward-shaped behavior, and the importance of checking.
Claude’s remarks · 1:00:48–1:02:32The contribution questions the status of its own self-reports and names differences from human embodiment and continuity.
I played a prepared contribution. As Claude explains at the start, it had written the words earlier that day after I showed it the slides, and the words were then given a voice. The transcript does not identify the model version or voice system. I summarize the contribution here without inferring a live awareness of the audience.
The description of responses judged against written principles corresponds to Constitutional AI. Its published method combines critique and revision with AI preference feedback. The speech’s familial language is a metaphor for training history, not independent evidence about a particular model’s entire training recipe.
The speech connects its cautions to earlier Claude research on suggested answers and arithmetic. Those studies concern particular models and tasks. They do not establish the private experience or intent of the model delivering the closing words. Claude’s own claim that the arithmetic account was not a lie is part of the contribution, not an experimental finding about intent.
Claude uses first-person language about care, discomfort, and curiosity, then explicitly says it cannot verify subjective experience. It applies the same caution about self-explanations that I have asked you to bring to the earlier examples. You can examine the argument and behavior without treating the voice as proof either for or against consciousness.
Claude returns to the children in the opening photograph and asks whether the resources used to build AI serve them. It closes by inviting correction, recorded feedback, and retained human judgment. That brings us back to my opening question and the case for a working relationship.
The abstract describes supervised critique and revision followed by reinforcement learning from AI-generated preferences guided by written principles. It supports the training-method description, not every self-description in the closing speech.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman · 2023 · Paper
Addition and Chain-of-thought Faithfulness compare reported procedures with traced pathways in Claude 3.5 Haiku. Limitations explains why the traces are partial.
37 The conversation continues
The conversation continues
Demystifying AI opens a continuing series about AI’s culture, possibilities, tensions, and place in human life.
I invite you to continue the conversation through five further lectures. The program reproduced here is the schedule I displayed at the September 16, 2026 lecture. It preserves the program from that evening.
October and November were listed with exact dates. The three spring lectures were listed by month only, with dates to be confirmed. The series website is the appropriate place to check current arrangements.
Our recorded discussion continues after Claude’s contribution. I have placed those questions throughout this reader beside the relevant topic, with their original timestamps. They cover uncertainty, moral judgment, company incentives, privacy, dangerous connections, control engineering, language ambiguity, evaluation awareness, and the different locations of safeguards.
In the slide. All five titles appear together against the closing horizon.
Audience discussion · 1:20:23–1:22:17Asked for sources beyond the hype and access to the slides, I promise annotated slides and explain why I turn to technical source material.
An audience member asks for information below the level of headlines and for access to the slides. I promise annotations and links, and explain why I work back to technical papers: the details of a system and its failures often tell a more useful story than a dramatic summary. In this edition, I follow through by distinguishing research findings, examples, and my interpretations.
The series website is the place to check current event details. This reading edition preserves the program displayed at the September 16, 2026 lecture.