# Demystifying AI: annotated reading edition

Roberto A. Santiago · University of Portland · September 16, 2026

Prepared from the presentation, working notes, and supplied lecture transcript. This is an edited reading companion, including spoken examples and audience questions.

## Contents

- [Slide 1: Demystifying AI](#slide-1)
- [Slide 2: Do Not Be Afraid](#slide-2)
- [Slide 3: Where I come from](#slide-3)
- [Slide 4: A lifelong question](#slide-4)
- [Slide 5: The photograph](#slide-5)
- [Slide 6: We have been here before](#slide-6)
- [Slide 7: The timeline](#slide-7)
- [Slide 8: Two paths from shared roots](#slide-8)
- [Slide 9: What do we mean by intelligence?](#slide-9)
- [Slide 10: Learning through feedback](#slide-10)
- [Slide 11: What comes next?](#slide-11)
- [Slide 12: Context changes the answer](#slide-12)
- [Slide 13: Learning from feedback](#slide-13)
- [Slide 14: Hold on to what you learned](#slide-14)
- [Slide 15: Go with your gut](#slide-15)
- [Slide 16: The scale behind the learning](#slide-16)
- [Slide 17: What does it actually produce?](#slide-17)
- [Slide 18: After pretraining](#slide-18)
- [Slide 19: Training for a useful response](#slide-19)
- [Slide 20: Learning from a reward](#slide-20)
- [Slide 21: Where the reward comes from](#slide-21)
- [Slide 22: Chain of thought: a chance to check](#slide-22)
- [Slide 23: The model at the center](#slide-23)
- [Slide 24: From training to use](#slide-24)
- [Slide 25: Memory beyond the context window](#slide-25)
- [Slide 26: Grounding answers in sources](#slide-26)
- [Slide 27: Connecting evidence with GraphRAG](#slide-27)
- [Slide 28: From replies to completed tasks](#slide-28)
- [Slide 29: Several agents working together](#slide-29)
- [Slide 30: The system we actually use](#slide-30)
- [Slide 31: A reason can miss the cause.](#slide-31)
- [Slide 32: Remembering what fits.](#slide-32)
- [Slide 33: Later information can rewrite the account.](#slide-33)
- [Slide 34: Shared patterns, side by side.](#slide-34)
- [Slide 35: The Cognitive Hypothesis](#slide-35)
- [Slide 36: A word from Claude](#slide-36)
- [Slide 37: The conversation continues](#slide-37)

<a id="slide-1"></a>

## 1. Demystifying AI

**Understanding how AI is built gives us a clearer starting point for deciding how to live and work with it.**

In this first lecture of In Dialogue with AI, I invite you to look inside the systems that now speak, write, reason, and act alongside us. These notes accompany the lecture I gave at the University of Portland on September 16, 2026.

We’ll move from the history of these ideas to simple learning exercises, then to the larger machinery surrounding a modern language model. Finally, we’ll ask what the resulting behavior suggests about cognition and our relationship with generative AI. You don’t need a technical background to follow the argument. I want to make the mechanisms understandable enough for us to ask better questions about their capabilities, limitations, and human purpose.

**The idea to carry forward.** A useful conversation about AI begins with understanding what we have actually built.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 0:13–2:24](https://www.youtube.com/watch?v=klzLRdHmeCs&t=13s): Welcome and introductions.
- [Audience discussion · 1:20:23–1:22:17](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4823s): Audience question about finding information below the headlines.

### Why these notes point to original sources

When you encounter a claim about AI, I encourage you to follow it back to the technical papers and documentation. I want to understand the mechanisms and decisions behind a result, beyond the alarming or impressive headline. That’s why I’ve included sources throughout these notes. You can explore the work for yourself and see where I’m interpreting a result and what the study directly establishes.

- [Recording · 1:20:23–1:22:17](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4823s)


<a id="slide-2"></a>

## 2. Do Not Be Afraid

**I want this lecture and the whole series to leave you with less fear and a greater sense that we can get involved and help determine how this turns out.**

I promised myself I would start with these words: do not be afraid. We hear so many claims about AI, and I know those words can draw a few uncomfortable chuckles. But this is a big part of why I wanted to give these lectures.

If nothing else happens over the course of these six lectures, I want you to walk away feeling less afraid and more able to say: we can handle this. We should be involved. We can help determine how this turns out. That is the spirit I want to carry through everything we explore together.

**The idea to carry forward.** We can take AI seriously while approaching it with curiosity instead of fear.

**In the slide.** I start with a simple statement: do not be afraid.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 2:24–3:04](https://www.youtube.com/watch?v=klzLRdHmeCs&t=144s): My opening promise: less fear, more participation, and a sense that we can shape what happens next.
- [Audience discussion · 1:12:58–1:15:23](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4378s): Returning to the question of fear and my belief in people who want to help.

### Why I believe we can handle this

I’m a Mr. Rogers sort of guy. I believe there are lots of people who want to help and who are willing to do what needs to be done to prevent harm. When we talk about the dangers of these systems, I don’t want us to stop researching them. I want us to keep working toward safer forms of AI and take responsibility for what we choose to do with them.

- [Recording · 1:12:58–1:15:23](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4378s)


<a id="slide-3"></a>

## 3. Where I come from

**I grew up programming computers and imagining a world where intelligent technology could help us do incredible things.**

I started programming in second grade and took to it like a fish to water. These early computers, the VIC-20, Atari, Commodore 64, Amiga, and TRS-80, bring me right back to that excitement. When people in the room recognized them, I felt a little less old. These were things I could explore, program, and make do something new.

I bought into the whole vision: intelligent spaceships we could talk to, androids we could hang out with, and robots that could help us fight the Empire. And if we couldn’t have all that, at least we could have a car that talked to us. We could all be David Hasselhoff. Underneath the joke was something I took seriously, even in grade school: I wanted to help build a world filled with intelligent technology that would let us do incredible and wonderful things.

**The idea to carry forward.** The motivating dream was useful intelligence woven into everyday life.

**In the slide.** I put the computers I grew up with alongside the fictional companions that helped me imagine what we might build.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 3:08–4:40](https://www.youtube.com/watch?v=klzLRdHmeCs&t=188s): How I started programming and came to imagine a world filled with helpful intelligence.

### A future I wanted to help build

This has been a passion of mine since grade school. As I got older, I didn’t think I would live to see the kind of technology we are seeing now. That childhood hope is still with me: intelligence woven into everyday life, helping us do things we couldn’t do before.

- [Recording · 3:08–4:40](https://www.youtube.com/watch?v=klzLRdHmeCs&t=188s)


<a id="slide-4"></a>

## 4. A lifelong question

**I’ve spent my career studying, building, and leading teams around technology, with one question running through it all: how does this help people?**

In 1991, I ended up working at the National Science Foundation, where I met Paul Werbos. Through Paul, I learned to use neural networks for forecasting, optimization, and control. By 1993, I was applying those ideas to airlines, and around 2001 I had the chance to continue my research in AI, neural networks, and neuroscience. I was especially interested in how what we were learning about the brain connected with reinforcement learning. The photograph of Paul and me was taken on a visit just a few months before this lecture. Time has changed us quite a bit.

Along the way, I’ve also been building things: large information systems at Booz Allen, software and hardware systems at many scales, and teams to make them work. That practical work has continued from 2001 to the present, including my more recent return to AI through work with knowledge graphs at Nike. Studying these ideas and putting them to work have always belonged together for me. Paul gave me a principle that I’ve carried through all of it: technical brilliance has to serve humanity.

**The idea to carry forward.** Useful AI requires the work of understanding, building, and integrating systems into human activity.

**In the slide.** My early research and my later systems-building are parts of the same story. The recent photograph with Paul makes that continuity personal.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 4:40–6:56](https://www.youtube.com/watch?v=klzLRdHmeCs&t=280s): My work with Paul Werbos, the practical uses of neural networks, and the commitment to service I carried forward.

### The research thread

With Paul, I learned adaptive dynamic programming and neurocontrol: ways of using learning systems to make decisions and control processes. I went on to use neural networks for airline forecasting and later explored the relationship between spike-timing plasticity in the brain and reinforcement learning. That connection between biological intelligence and learning machines has stayed with me throughout my work.

- [Roberto A. Santiago: The Road](https://roberto.org/#road)

### Putting ideas to work

At Booz Allen, I worked on knowledge models and large information systems. Over the years, I’ve led teams and built systems of all sorts of sizes and scales, including more recent work with knowledge graphs and graph networks in production. I bring that practical experience into this lecture because making a technique work is only part of the job. We also have to build something people can use.

- [Roberto A. Santiago: The Road](https://roberto.org/#road)

### What useful meant in practice

We worked on making engines more fuel efficient, on flight-control systems that could adapt when something went wrong, and on economic forecasting. In the private sector, I did modeling and forecasting for airlines, insurers, and financial businesses. One very concrete example was predicting demand to help set airline ticket prices. When prices started changing daily, I was part of that, and I joked in the lecture that some people still hate me for it. I’m describing the research and applications we pursued here, including work that was still experimental.

- [Roberto A. Santiago: The Road](https://roberto.org/#road)
- [Recording · 4:40–6:07](https://www.youtube.com/watch?v=klzLRdHmeCs&t=280s)

### The standard I carry forward

One thing Paul taught me has stayed with me: brilliance isn’t worth much if the technology we build doesn’t seek to serve humanity. I’ve held on to that through decades of building software, hardware, and teams. It is also why I want to ask more than what AI can do. I want to ask whose needs we are using it to meet.

- [Roberto A. Santiago: The Road](https://roberto.org/#road)
- [Recording · 6:30–6:56](https://www.youtube.com/watch?v=klzLRdHmeCs&t=390s)

### Sources and further reading

- **[The Road](https://roberto.org/#road)**. Roberto A. Santiago, 2026. My account of my research and professional history, including NSF, BehavHeuristics, Booz Allen, the NW Computational Intelligence Lab, and later production work. The Road section gives the dated milestones.


<a id="slide-5"></a>

## 5. The photograph

**When I think about AI, I think about these children. I want us to ask whose lives our work is helping.**

You may have noticed that I don’t wear shoes. That began as a spiritual practice during Lent about fifteen years before this lecture. I had been asked to give children a presentation about poverty around the world. Reading about children dying from preventable causes, while looking at my own life and remembering Paul’s words about serving humanity, made me feel that I needed to live differently. Going barefoot became a way to express that solidarity.

This photograph hangs in our home. These are children my wife, Anne, met during her travels in Africa. They’re grown-ups now, but we remember them all the time. At one point I was ready to leave technology completely. This journey is part of why I’m here talking with you, and why I think AI is as much about these children as it is about the technology itself.

I have a math degree, and I love mathematics. I can be excited by what AI is accomplishing there and still ask: could we bring comparable money, computing, and ingenuity to making sure children are fed and cared for? I want us to bring AI down to earth, understand what we have, and see where we have the power to shape what happens with it.

**The idea to carry forward.** The value of technological progress includes what it contributes to human lives.

**In the slide.** This is a photograph from our home, of children Anne met in Africa.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 6:56–9:16](https://www.youtube.com/watch?v=klzLRdHmeCs&t=416s): Why I go barefoot, the children in our photograph, and the human needs I want technology to serve.

### Child survival then and now

To put this concern in numbers, UNICEF’s The State of the World’s Children 2008: Child Survival reported close to 9.7 million deaths among children under five in 2006, more than 26,000 a day, mostly from preventable causes. WHO’s May 2026 fact sheet puts the 2024 total at 4.9 million, about 9.3 children per minute when averaged over that leap year. I want to be precise about what that number means: it counts deaths from all causes. The accompanying UN release says most could be prevented through established, low-cost interventions and access to quality health care. We have made substantial progress, and the continuing scale of the need is still staggering.

- [UNICEF: The State of the World’s Children 2008: Child Survival](https://www.unicef.org/montenegro/media/7971/file/MNE-media-MNEpublication316.pdf#page=9)
- [World Health Organization: Child mortality (under 5 years)](https://www.who.int/news-room/fact-sheets/detail/children-reducing-mortality)
- [UNICEF, WHO and UN IGME partners: Progress in reducing child deaths slows as 4.9 million children under five die in 2024](https://www.unicef.org/press-releases/progress-reducing-child-deaths-slows-49-million-children-under-five-die-2024)
- [Recording · 7:28–7:37](https://www.youtube.com/watch?v=klzLRdHmeCs&t=448s)

### Essential treatments can cost very little

What troubles me is that children still die from illnesses we know how to prevent and treat, even when some essential treatments cost only cents. WHO’s 2024 diarrhea fact sheet reports about 444,000 deaths annually among children under five and describes oral rehydration salts as costing a few cents per treatment. UNICEF’s pneumonia page reports more than 700,000 under-five deaths annually and describes antibiotic treatment for appropriate respiratory infections as typically costing less than US$0.50 for a full course. When I talk about those low prices, I mean the treatments themselves. Getting effective care to a child also requires timely diagnosis, delivery, clean water, nutrition, and sometimes additional treatment for severe illness. That is the practical work I want us to take seriously.

- [World Health Organization: Diarrhoeal disease](https://www.who.int/news-room/fact-sheets/detail/diarrhoeal-disease)
- [UNICEF Data: Pneumonia in children](https://data.unicef.org/topic/child-health/pneumonia/)
- [Recording · 7:28–7:37](https://www.youtube.com/watch?v=klzLRdHmeCs&t=448s)

### Whose problem are we solving?

I love seeing AI tackle ambitious mathematical problems. I also want us to bring that ambition to figuring out how children can be fed and cared for. What problem are we choosing to solve? Who will benefit? What are we willing to commit to making that benefit real? Those questions are part of what I mean when I ask what we can do with this technology.

- [Recording · 8:08–9:16](https://www.youtube.com/watch?v=klzLRdHmeCs&t=488s)

### The scale of the Navier–Stokes effort

My mathematics example is about the scale of effort we can bring to a problem. OpenAI’s September 8, 2026 announcement reports about 10,000 concurrent agents in the group that produced its proposed Navier–Stokes result, which concerns a fluid subject to smooth external forcing. It discloses 130 billion output tokens for this problem and 300 billion across all attempted problems. At the standard Astra output prices listed on October 16 ($50–$75 per million tokens, depending on context length), those volumes correspond to $6.5–$9.75 million and $15–$22.5 million respectively. I’m using these as illustrative retail-price equivalents for output alone. The research used a different internal model; these figures do not establish its actual operating costs or include inputs, training, staffing, or other project costs.

- [OpenAI: On the Navier–Stokes Millennium Prize Problem](https://openai.com/index/navier-stokes-solution/)
- [OpenAI: OpenAI API pricing](https://developers.openai.com/api/docs/pricing)
- [Recording · 8:37–8:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=517s)

### After the lecture: hundreds of mathematical manuscripts

Since I gave this lecture, OpenAI’s October 6, 2026 release has added another example of the scale of AI mathematics. As checked on October 16, its catalog contains 719 manuscripts in 372 related families, with results at different stages of verification. I’m not presenting that count as 719 independently verified discoveries. OpenAI describes average compute equivalent to about three hours of ChatGPT Pro thinking per result but gives no dollar total in this disclosure. I include this as a later development, following the September 16 lecture.

- [OpenAI: Sharing AI progress in mathematics](https://openai.com/index/sharing-ai-progress-in-mathematics/)
- [OpenAI: OpenAI mathematics catalogue](https://github.com/openai/math/blob/main/README.md)

### Sources and further reading

- **[The State of the World’s Children 2008: Child Survival](https://www.unicef.org/montenegro/media/7971/file/MNE-media-MNEpublication316.pdf#page=9)**. UNICEF, 2008. Printed page 1 reports the historical 2006 estimate and more than 26,000 daily under-five deaths. Provides context for the scale of preventable suffering; it does not establish a universal dollar cost per life saved.
- **[Child mortality (under 5 years)](https://www.who.int/news-room/fact-sheets/detail/children-reducing-mortality)**. World Health Organization, 2026. Updated May 1, 2026. Reports 4.9 million under-five deaths in 2024. The annotation’s 9.3 per minute is 4,900,000 divided by the leap year’s 527,040 minutes. This total includes all causes, not only lack of food or medicine.
- **[Progress in reducing child deaths slows as 4.9 million children under five die in 2024](https://www.unicef.org/press-releases/progress-reducing-child-deaths-slows-49-million-children-under-five-die-2024)**. UNICEF, WHO and UN IGME partners, 2026. Release datelined March 18, 2026. Connects preventable deaths with low-cost interventions and access to quality health care. Notes that successive estimation rounds are revised, so the historical and current reports are not used here to calculate a precise percentage decline.
- **[Diarrhoeal disease](https://www.who.int/news-room/fact-sheets/detail/diarrhoeal-disease)**. World Health Organization, 2024. Dated March 7, 2024. Gives 443,832 annual under-five deaths and the low price of oral rehydration salts, while also describing other prevention and treatment needs. The page does not label the mortality figure as deaths occurring in 2024.
- **[Pneumonia in children](https://data.unicef.org/topic/child-health/pneumonia/)**. UNICEF Data, Accessed October 16, 2026. Reports over 700,000 annual under-five deaths and antibiotic treatment typically costing less than US$0.50. Explains the need for clinical assessment and appropriate care. These headline estimates are not combined with the WHO diarrhea figure into a single-year mortality total.
- **[On the Navier–Stokes Millennium Prize Problem](https://openai.com/index/navier-stokes-solution/)**. OpenAI, 2026. September 8 announcement. Reports the agent count, external-forcing construction, and separate output-token totals for Navier–Stokes and all attempted problems. It does not provide an itemized dollar cost.
- **[OpenAI API pricing](https://developers.openai.com/api/docs/pricing)**. OpenAI, 2026. Standard Astra output rates checked October 16: $50 per million tokens for short context and $75 for long context. Multiplying by the reported token volumes gives illustrative retail equivalents, not the unreleased research model’s measured cost.
- **[Sharing AI progress in mathematics](https://openai.com/index/sharing-ai-progress-in-mathematics/)**. OpenAI, 2026. October 6 release and compute disclosure. This postdates the lecture and is included as a later update.
- **[OpenAI mathematics catalogue](https://github.com/openai/math/blob/main/README.md)**. OpenAI, 2026. Catalogue and verification status checked October 16. Groups related manuscripts into families and explains the compute comparison.


<a id="slide-6"></a>

## 6. We have been here before

**An extraordinary claim from 1988 reveals that today’s excitement about neural networks has a long history.**

The phrase “more important than the atom bomb” appears before its date or speaker. Without that context, it could sound like a contemporary claim about AI. The reveal places it in 1988, when Scientific American reported Jasper Lupo of DARPA describing the potential importance of neural networks.

That change of context is why I wanted to show you this. Earlier researchers and funders also saw enormous possibilities in learning machines. Their enthusiasm existed alongside practical applications, serious obstacles, and uncertainty about what would work. I want us to take a longer view of the present: what has actually changed, which problems have been solved, and which hopes or difficulties have returned? I also remember meeting Lupo, so this history has a personal connection for me.

**The idea to carry forward.** Historical context helps us separate a compelling promise from the mechanisms and evidence behind it.

**In the slide.** The dramatic quotation comes first; the 1988 attribution and period report change how we read it.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 9:16–10:37](https://www.youtube.com/watch?v=klzLRdHmeCs&t=556s): The 1988 quotation and recurring promises about neural networks.

### Where the quotation comes from

June Kinoshita’s article Neural Networks at Work reports Lupo’s statement at a neural-network conference in San Diego. The wording appears on printed page 135 of the November 1988 Scientific American. The article identifies him as an assistant director at DARPA.

- [June Kinoshita: Neural Networks at Work](https://www.scientificamerican.com/article/neural-networks-at-work/)
- [Recording · 9:16–10:04](https://www.youtube.com/watch?v=klzLRdHmeCs&t=556s)

### What the pictured report is

The DARPA Neural Network Study assessed neural-network technology and research directions during 1987–1988. The pictured final technical report is dated March 22, 1989. It supplies context for that period of research, while the Scientific American article supplies the quotation.

- [MIT Lincoln Laboratory for DARPA: DARPA Neural Network Study](https://s3.amazonaws.com/arena-attachments/2618226/4497a1ae3a5e18d1d9e90a7748c1f9b9.pdf)

### Applications were already emerging

Kinoshita also reported work on industrial inspection, credit assessment, airline decisions, and process control. The historical comparison is therefore more interesting than an old prediction: neural networks already had practical ambitions, even as claims about their future ran far ahead of their demonstrated abilities.

- [June Kinoshita: Neural Networks at Work](https://www.scientificamerican.com/article/neural-networks-at-work/)
- [Recording · 10:04–10:37](https://www.youtube.com/watch?v=klzLRdHmeCs&t=604s)

### Sources and further reading

- **[Neural Networks at Work](https://www.scientificamerican.com/article/neural-networks-at-work/)**. June Kinoshita, 1988. Contemporaneous reporting in Scientific American 259(5), pages 134–135. Lupo’s quotation and title appear on page 135; page 134 describes emerging applications. DOI: 10.1038/scientificamerican1188-134.
- **[DARPA Neural Network Study](https://s3.amazonaws.com/arena-attachments/2618226/4497a1ae3a5e18d1d9e90a7748c1f9b9.pdf)**. MIT Lincoln Laboratory for DARPA, 1989. Technical Report 840 / ESD-TR-88-311, dated March 22, 1989, covering the October 1987–February 1988 study. The cover and foreword explain the relationship between the study dates and the final report.


<a id="slide-7"></a>

## 7. The timeline

**Today’s AI draws on a century of overlapping ideas about computation, brains, learning, feedback, and language.**

The timeline follows people and ideas rather than presenting AI as a recent invention. Fiction gives us a name for manufactured workers; mathematics clarifies what computation can do; neuroscience and cybernetics connect biological processes with formal models. Neural learning, symbolic reasoning, better hardware, and new ways to represent language develop through overlapping communities.

Two labels receive special emphasis because they will matter throughout the lecture. Backpropagation calculates how changes inside a network affect an error or objective. Reinforcement learning organizes learning around actions and evaluated outcomes. These are complementary ideas: neural reinforcement-learning systems can use backpropagation to change their networks. The later exercises make this distinction concrete.

The final milestones bring these foundations into large language models and conversational systems. The collage at the end conveys the activity visible at the time of the lecture. The dates mark selected publications, demonstrations, and releases, rather than asserting that each idea began at exactly that moment or belongs to one inventor.

**The idea to carry forward.** Modern AI is an accumulation of methods and research traditions, with learning from error and reward central to this lecture’s story.

**In the slide.** Portraits identify the people; the small computer, network, and brain images identify recurring themes. Equal spacing separates events for readability, rather than measuring the years between them.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 10:37–22:13](https://www.youtube.com/watch?v=klzLRdHmeCs&t=637s): The historical timeline, from R.U.R. to conversational AI.

### A cultural question before a technical field

Čapek’s R.U.R. depicts manufactured workers whose revolt threatens humanity. Its robots are artificial living workers, rather than today’s familiar metal machines. The play was published in 1920, with premieres in 1921. In his own later account, Karel credits his brother Josef with suggesting the word robot. The story raises questions about labor, dependence, and responsibility before modern AI research takes shape. I read its promise of tireless manufactured labor as a warning about valuing productivity while forgetting the dignity of people.

- [Karel Čapek; English version by Paul Selver and Nigel Playfair: R.U.R. (Rossum’s Universal Robots)](https://www.gutenberg.org/files/59112/59112-h/59112-h.htm)
- [Karel Čapek: About the Word Robot](https://scholarsbank.uoregon.edu/bitstreams/d812b91a-7c7c-442c-a360-d328a314e12e/download)
- [Recording · 10:37–12:50](https://www.youtube.com/watch?v=klzLRdHmeCs&t=637s)

### Feedback connects animal and machine

Wiener’s 1948 book Cybernetics: Or Control and Communication in the Animal and the Machine connected the study of communication and feedback across biological and engineered systems. Feedback means that the consequences of a process influence what it does next. A feedback controller can regulate a system without learning, so feedback by itself does not necessarily imply changing a neural network’s parameters.

- [Norbert Wiener: Cybernetics: Or Control and Communication in the Animal and the Machine](https://mitpress.mit.edu/9780262537841/cybernetics-or-control-and-communication-in-the-animal-and-the-machine/)

### A model inspired by brains

Rosenblatt’s perceptron made learning through adjustable connections a concrete research program. His work included rules for changing connections through experience. The later challenge was effective training of more general multilayer networks. The multilayer network pictured here is a teaching illustration, rather than a reconstruction of Rosenblatt’s 1958 machine or the anatomy of a human brain.

- [Frank Rosenblatt: The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain](https://doi.org/10.1037/h0042519)
- [Recording · 13:51–14:47](https://www.youtube.com/watch?v=klzLRdHmeCs&t=831s)

### Backpropagation: how should connections change?

Backpropagation efficiently works backward through a computation to calculate how an error or objective depends on its adjustable parameters. An optimizer can use those calculations to change the network. The 1974 milestone recognizes Werbos’s work on backward differentiation and learning; the 1986 paper by Rumelhart, Hinton, and Williams helped bring the approach to a much wider audience. The mathematical history includes earlier contributions across several fields.

- [Paul J. Werbos: Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities](https://link.springer.com/chapter/10.1007/3-540-28438-9_2)
- [David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams: Learning Representations by Back-Propagating Errors](https://doi.org/10.1038/323533a0)
- [Recording · 15:20–15:59](https://www.youtube.com/watch?v=klzLRdHmeCs&t=920s)

### Reinforcement learning: how well did the action work?

The 1983 Barto, Sutton, and Anderson study used two adaptive components to learn a difficult control task. One selected actions; the other helped evaluate their consequences. This is an early actor-critic result, rather than the invention of all reinforcement learning. It illustrates the lecture’s important distinction between being given a desired answer and receiving an evaluation of what happened after acting.

- [Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson: Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems](https://doi.org/10.1109/TSMC.1983.6313077)
- [Recording · 15:59–17:43](https://www.youtube.com/watch?v=klzLRdHmeCs&t=959s)

### From generating language to helping a user

The arrival of conversational AI involved more than making language models larger. The InstructGPT paper describes learning from demonstrations and then from human preferences to make responses better match user intent. This distinction between a pretrained language model and a model shaped to assist people becomes a central thread in the next sections.

- [Long Ouyang and colleagues: Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)

### Why emphasize these two tools?

I want you to remember backpropagation and reinforcement learning because they give us a manageable way into the training story. One explains how an objective can guide changes to a network; the other explains learning from the consequences of actions. These complementary tools sit within a broad, collaborative research history. We’ll see how they contribute to different stages of training in the next sections.

- [Paul J. Werbos: Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities](https://link.springer.com/chapter/10.1007/3-540-28438-9_2)
- [David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams: Learning Representations by Back-Propagating Errors](https://doi.org/10.1038/323533a0)
- [Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson: Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems](https://doi.org/10.1109/TSMC.1983.6313077)
- [Long Ouyang and colleagues: Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)
- [Recording · 15:59–17:43](https://www.youtube.com/watch?v=klzLRdHmeCs&t=959s)

### What changed when the old ideas became practical?

Ideas alone were not enough. Hardware, training methods, data, and network design had to work together. AlexNet’s 2012 paper gives a concrete example: GPU computation helped make a large convolutional vision network practical. The 2017 Transformer paper introduced an attention-based architecture that made sequence training more parallelizable. Together, these developments show computing capacity and network design advancing alongside one another, expanding what models could do with images and language. I know from my own research how expensive this hardware can be. That’s one reason I think access to computing deserves attention as a possible policy lever. It may give us a way to influence development, though I’m not offering it as a guarantee of control.

- [Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton: ImageNet Classification with Deep Convolutional Neural Networks](https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html)
- [Ashish Vaswani and colleagues: Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Recording · 18:11–21:10](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1091s)
- [Recording · 19:10–19:45](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1150s)

### A century of ideas, with sources

#### 1920: We imagine the robot

Karel Čapek · R.U.R.

- [Čapek, K. R.U.R. (Rossum’s Universal Robots), published 1920; English text translated by Paul Selver and adapted for the English stage by Nigel Playfair. The play depicts manufactured workers who rebel, leaving Alquist as the last human survivor.](https://www.gutenberg.org/files/59112/59112-h/59112-h.htm)
- [Čapek, K. “About the Word Robot” (O slově robot), Lidové noviny, December 24, 1933. The author’s own account credits his brother Josef with suggesting the name; English reproduction preserved by the University of Oregon.](https://scholarsbank.uoregon.edu/bitstreams/d812b91a-7c7c-442c-a360-d328a314e12e/download)
- [Karel Čapek Memorial, “Karel Čapek”: publication in 1920 and theatrical premieres in 1921; Josef’s role in naming the robots.](https://karelcapek.cz/en/life-and-creation/karel-capek)

#### 1936: A machine that can compute

Alan Turing · On Computable Numbers

- [Turing, A. M. “On Computable Numbers, with an Application to the Entscheidungsproblem.” Proceedings of the London Mathematical Society, s2-42, 230–265.](https://doi.org/10.1112/plms/s2-42.1.230)

#### 1943: A neuron, on paper

Warren McCulloch & Walter Pitts

- [McCulloch, W. S., and Pitts, W. “A Logical Calculus of the Ideas Immanent in Nervous Activity.” Bulletin of Mathematical Biophysics 5, 115–133 (1943).](https://doi.org/10.1007/BF02478259)

#### 1945: A program stored in memory

John von Neumann · The EDVAC report

- [von Neumann, J. First Draft of a Report on the EDVAC, June 30, 1945. Original report, reprinted in IEEE Annals of the History of Computing.](https://web.mit.edu/STS.035/www/PDFs/edvac.pdf)

#### 1948: Feedback connects animal and machine

Norbert Wiener

- [Wiener, N. Cybernetics: Or Control and Communication in the Animal and the Machine, first published 1948. MIT Libraries identifies the book as a landmark account of control, feedback and communication in biological and electromechanical systems.](https://libraries.mit.edu/150books/2011/04/04/1948/)
- [MIT Press, Cybernetics: confirms first publication in 1948 and its foundational role in the multidisciplinary study of systems with feedback loops.](https://mitpress.mit.edu/9780262537841/cybernetics-or-control-and-communication-in-the-animal-and-the-machine/)

#### 1950: Can machines think?

Alan Turing · Computing Machinery and Intelligence

- [Turing, A. M. “Computing Machinery and Intelligence.” Mind LIX(236), 433–460 (1950).](https://doi.org/10.1093/mind/LIX.236.433)

#### 1952: How a real neuron fires

Alan Hodgkin & Andrew Huxley

- [Hodgkin, A. L., and Huxley, A. F. “A Quantitative Description of Membrane Current and Its Application to Conduction and Excitation in Nerve.” Journal of Physiology 117, 500–544 (1952).](https://doi.org/10.1113/jphysiol.1952.sp004764)

#### 1956: Artificial intelligence gets a name

The Dartmouth workshop

- [McCarthy, J., Minsky, M. L., Rochester, N., and Shannon, C. E. “A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence,” August 31, 1955, for the summer 1956 meeting.](https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html)

#### 1958: The perceptron

Frank Rosenblatt

- [Rosenblatt, F. “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.” Psychological Review 65(6), 386–408 (1958).](https://doi.org/10.1037/h0042519)

#### 1965: Between true and false

Lotfi Zadeh · Fuzzy logic

- [Zadeh, L. A. “Fuzzy Sets.” Information and Control 8(3), 338–353 (1965).](https://doi.org/10.1016/S0019-9958(65)90241-X)

#### 1969: Perceptrons under scrutiny

Marvin Minsky & Seymour Papert

- [Minsky, M., and Papert, S. A. Perceptrons: An Introduction to Computational Geometry. MIT Press, 1969.](https://mitpress.mit.edu/9780262130431/perceptrons/)

#### 1974: BACKPROPAGATION

Paul Werbos · Harvard dissertation

- [Werbos, P. J. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Harvard doctoral dissertation, August 1974.](https://perceptrondemo.com/assets/PJW_thesis_Beyond_Regression_1974-4b63aa5f.pdf)
- [Werbos, P. J. “Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities” (author’s retrospective).](https://werbos.com/AD2004.pdf)

#### 1983: REINFORCEMENT LEARNING

Andrew Barto, Richard Sutton & Charles Anderson

- [Barto, A. G., Sutton, R. S., and Anderson, C. W. “Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems.” IEEE Transactions on Systems, Man, and Cybernetics SMC-13, 834–846 (1983).](https://doi.org/10.1109/TSMC.1983.6313077)
- [Richard Sutton’s author bibliography describes the associative search element and adaptive critic element.](https://incompleteideas.net/publications.html)

#### 1986: Backpropagation reaches a wider field

David Rumelhart, Geoffrey Hinton & Ronald Williams

- [Rumelhart, D. E., Hinton, G. E., and Williams, R. J. “Learning Representations by Back-Propagating Errors.” Nature 323, 533–536 (1986).](https://doi.org/10.1038/323533a0)

#### 1988: Learning from prediction error

Richard Sutton · Temporal-difference learning

- [Sutton, R. S. “Learning to Predict by the Methods of Temporal Differences.” Machine Learning 3, 9–44 (1988).](https://doi.org/10.1007/BF00115009)

#### 1988: We have been here before

The DARPA Neural Network Study

- [DARPA Neural Network Study, October 1987–February 1988. MIT Lincoln Laboratory Technical Report 840 / ESD-TR-88-311, final report dated March 22, 1989; foreword dated June 1988.](https://s3.amazonaws.com/arena-attachments/2618226/4497a1ae3a5e18d1d9e90a7748c1f9b9.pdf)

#### 1989: Reading handwritten digits

Yann LeCun and colleagues

- [LeCun, Y., et al. “Backpropagation Applied to Handwritten Zip Code Recognition.” Neural Computation 1(4), 541–551 (1989).](https://doi.org/10.1162/neco.1989.1.4.541)

#### 1990: Neural Networks for Control

W. Thomas Miller III, Richard Sutton & Paul Werbos

- [Miller, W. T. III, Sutton, R. S., and Werbos, P. J., eds. Neural Networks for Control. MIT Press. Commonly cited as 1990; publisher catalog lists the hardcover release as January 10, 1991.](https://mitpress.mit.edu/9780262132619/neural-networks-for-control/)
- [Bibliothèque nationale de France catalog records copyright 1990.](https://catalogue.bnf.fr/ark:/12148/cb37453283v)

#### 1997: A machine wins at chess

IBM Deep Blue · Garry Kasparov

- [IBM, “Deep Blue”: the May 1997 match against Garry Kasparov ended 3.5–2.5 in Deep Blue’s favor under standard tournament controls.](https://www.ibm.com/history/deep-blue)

#### 2006: Depth returns

Geoffrey Hinton and colleagues · Deep belief networks

- [Hinton, G. E., Osindero, S., and Teh, Y.-W. “A Fast Learning Algorithm for Deep Belief Nets.” Neural Computation 18(7), 1527–1554 (2006).](https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf)

#### 2007: A new engine for the work

NVIDIA CUDA

- [NVIDIA CUDA Toolkit Archive: CUDA Toolkit 1.0, June 2007, and 1.1, December 2007.](https://developer.nvidia.com/cuda-toolkit-archive)

#### 2011: A machine wins Jeopardy!

IBM Watson

- [IBM, “Watson, Jeopardy! Champion”: Watson defeated Brad Rutter and Ken Jennings in the February 2011 exhibition.](https://www.ibm.com/history/watson-jeopardy)

#### 2012: The image recognition breakthrough

AlexNet · ImageNet

- [Krizhevsky, A., Sutskever, I., and Hinton, G. E. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems 25 (2012).](https://papers.nips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf)

#### 2013: Words as numbers

word2vec

- [Mikolov, T., Chen, K., Corrado, G., and Dean, J. “Efficient Estimation of Word Representations in Vector Space.” arXiv:1301.3781 (2013).](https://arxiv.org/abs/1301.3781)
- [Bengio, Y., et al. “A Neural Probabilistic Language Model.” JMLR 3, 1137–1155 (2003), an earlier learned word-representation model.](https://jmlr.org/papers/v3/bengio03a.html)

#### 2015–16: From Atari to Go

DeepMind · DQN and AlphaGo

- [Mnih, V., et al. “Human-Level Control Through Deep Reinforcement Learning.” Nature 518, 529–533 (2015). Silver, D., et al. “Mastering the Game of Go with Deep Neural Networks and Tree Search.” Nature 529, 484–489 (2016).](https://doi.org/10.1038/nature14236)
- [AlphaGo paper, 2016.](https://doi.org/10.1038/nature16961)

#### 2017: Attention Is All You Need

The transformer

- [Vaswani, A., et al. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 / arXiv:1706.03762 (2017).](https://arxiv.org/abs/1706.03762)

#### 2018: GPT-1 and BERT

Pretraining language models

- [Radford, A., et al. “Improving Language Understanding by Generative Pre-Training” (2018). Devlin, J., et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv:1810.04805 (2018).](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)
- [BERT preprint, submitted October 11, 2018; conference publication followed in 2019.](https://arxiv.org/abs/1810.04805)
- [Earlier neural language modeling: Bengio et al. (2003).](https://jmlr.org/papers/v3/bengio03a.html)

#### 2019: GPT-2

Longer passages, broader patterns

- [OpenAI, “Better Language Models and Their Implications,” February 14, 2019; Radford et al., “Language Models Are Unsupervised Multitask Learners.”](https://openai.com/index/better-language-models/)

#### 2020: GPT-3

A larger model, a wider range of tasks

- [Brown, T. B., et al. “Language Models Are Few-Shot Learners.” arXiv:2005.14165 / Advances in Neural Information Processing Systems 33 (2020).](https://arxiv.org/abs/2005.14165)

#### 2022: The conversation reaches the room

ChatGPT

- [OpenAI, “Introducing ChatGPT,” November 30, 2022.](https://openai.com/index/chatgpt/)

#### 2024–25: Reasoning gets more time

o1 and DeepSeek-R1

- [OpenAI, “Learning to Reason with LLMs,” September 12, 2024. DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” arXiv:2501.12948 (2025).](https://openai.com/index/learning-to-reason-with-llms/)
- [DeepSeek-R1 technical report, January 2025.](https://arxiv.org/abs/2501.12948)

#### Today: An explosion of activity

A selection of companies and model families shown in the September 2026 lecture, not a ranking or a complete market map.

- [NVIDIA · Nemotron: official model information](https://developer.nvidia.com/topics/ai/nemotron)
- [Cohere · Command: official model information](https://cohere.com/command)
- [xAI · Grok: official model information](https://x.ai/grok)
- [Google · Gemini / Gemma: official model information](https://deepmind.google/models/gemini/)
- [Google · Gemini / Gemma: official model information](https://deepmind.google/models/gemma/)
- [Baidu · ERNIE: official model information](https://ernie.baidu.com/)
- [Microsoft · Phi: official model information](https://huggingface.co/microsoft/phi-4)
- [Anthropic · Claude: official model information](https://claude.com/product/overview)
- [OpenAI · GPT: official model information](https://openai.com/research/)
- [DeepSeek · DeepSeek: official model information](https://deepseek.com/)
- [Mistral AI · Mistral: official model information](https://mistral.ai/models/)
- [Alibaba · Qwen: official model information](https://github.com/QwenLM/Qwen)
- [Meta · Llama: official model information](https://github.com/meta-llama/llama-models)
- [Amazon · Nova: official model information](https://aws.amazon.com/nova/models/)
- [IBM · Granite: official model information](https://www.ibm.com/granite)
- [AI21 · Jamba: official model information](https://www.ai21.com/research/jamba-1-5-hybrid-transformer-mamba-models-at-scale/)
- [Tencent · Hy / Hunyuan: official model information](https://github.com/Tencent-Hunyuan/Tencent-Hunyuan-Large)
- [Tencent · Hy / Hunyuan: official model information](https://www.tencent.com/tencent-hy3-now-available-globally-extending-practical-ai-across-products-workflows-and-cloud-services/)

### Sources and further reading

- **[R.U.R. (Rossum’s Universal Robots)](https://www.gutenberg.org/files/59112/59112-h/59112-h.htm)**. Karel Čapek; English version by Paul Selver and Nigel Playfair, 1920; English edition 1923. The original fictional work behind the timeline’s first milestone. The opening explains the manufactured workers; the final acts and epilogue develop the revolt and its aftermath.
- **[About the Word Robot](https://scholarsbank.uoregon.edu/bitstreams/d812b91a-7c7c-442c-a360-d328a314e12e/download)**. Karel Čapek, 1933. Čapek’s own account, published in Lidové noviny on December 24, 1933, credits Josef Čapek with suggesting the name. English reproduction preserved by the University of Oregon.
- **[Cybernetics: Or Control and Communication in the Animal and the Machine](https://mitpress.mit.edu/9780262537841/cybernetics-or-control-and-communication-in-the-animal-and-the-machine/)**. Norbert Wiener, 1948. Wiener’s foundational book connects feedback, control, and communication. The publisher page confirms first publication in 1948 and links an open-access reissue of the expanded 1961 edition.
- **[The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain](https://doi.org/10.1037/h0042519)**. Frank Rosenblatt, 1958. The original Psychological Review paper, volume 65, pages 386–408, develops the perceptron as a model of learning and recognition. Supports the brain-inspired learning milestone.
- **[Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities](https://link.springer.com/chapter/10.1007/3-540-28438-9_2)**. Paul J. Werbos, 2006. Werbos’s retrospective explains backward differentiation and its connections with neural learning and control. Section 2 discusses his 1974 dissertation; the introduction places the technique within several interacting research traditions. Published in 2006; the timeline also links the author manuscript.
- **[Learning Representations by Back-Propagating Errors](https://doi.org/10.1038/323533a0)**. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, 1986. The influential Nature paper, volume 323, pages 533–536, describes how backpropagation trains hidden units to learn useful internal representations.
- **[Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems](https://doi.org/10.1109/TSMC.1983.6313077)**. Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson, 1983. The original learning-control study develops an associative search element and an adaptive critic, illustrated by a cart-pole balancing task. Supports the timeline’s early actor-critic milestone.
- **[Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and Section 3 describe supervised fine-tuning, preference collection, and reinforcement learning from human feedback. Supports the transition from broad language modeling toward assistant behavior.
- **[ImageNet Classification with Deep Convolutional Neural Networks](https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html)**. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, 2012. The original NeurIPS paper describes a deep convolutional image classifier and the GPU implementation used to accelerate training. See the abstract and Section 3.2 on training on multiple GPUs.
- **[Attention Is All You Need](https://arxiv.org/abs/1706.03762)**. Ashish Vaswani and colleagues, 2017. The eight-author paper introduces the Transformer. The abstract and Sections 1 and 3 explain attention-based sequence modeling and the role of parallel computation, supporting the timeline’s architecture milestone.


<a id="slide-8"></a>

## 8. Two paths from shared roots

**Language models are one major destination of neural learning, while control, optimization, and adaptive engineering remain another living tradition.**

The branching picture condenses the chronology into two different approaches to intelligence. Symbolic AI works with explicit representations, rules, and search. Connectionism learns through networks of adjustable connections. The perceptron makes the meeting of computation and brain-inspired learning visible; the later debate over its limitations helps explain why these approaches developed distinct communities and priorities.

As learning methods advanced, neural networks became increasingly important within mainstream AI and computer science. The upper route follows that development toward recognition, language, and eventually large language models. The merger represents a change in research practice across the field. It does not mean that symbolic rules were physically combined with neural weights to manufacture an LLM.

The lower route keeps a second history in view: using learned models to control changing physical systems, optimize decisions, and adapt over time. Neural Networks for Control anchors that tradition around the beginning of the 1990s. Its dashed continuation matters because this work did not end when language models became prominent. The branches continue to exchange ideas, and engineering problems still motivate methods whose purpose extends well beyond producing language.

For me, the engineering branch is a reason for hope and a research priority. Think about energy use, forecasting, control, and adaptive systems: their value depends on reliable behavior in the world. I want us to invest in bringing more of that knowledge into useful AI systems while being more selective about deployment. There is work to do here; we can’t simply transfer a guarantee from a physical controller to an open-ended conversation. When I look at the money going into selling AI products and military applications, I want to see more investment in stability, safety, and fail-safe behavior.

**The idea to carry forward.** LLMs occupy an important part of the learning landscape, while a broader engineering toolbox continues to develop alongside them.

**In the slide.** Follow both endpoints: the upper line leads toward modern LLMs; the lower dashed line keeps control, optimization, and adaptive systems visibly in motion.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 22:13–25:54](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1333s): Two research paths and the case for investing in safety and control.
- [Audience discussion · 1:15:38–1:17:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4538s): Audience question about transferring safeguards from control engineering.

### Reading the fork

This is a selective map of research traditions. The 1955 Dartmouth proposal already included both language-related questions and neuron nets. Minsky and Papert’s 1969 Perceptrons examined limitations of particular perceptron classes and became a focal point of debate. It did not establish that every multilayer network was incapable of learning, and symbolic AI did not begin with that book.

- [John McCarthy, Marvin L. Minsky, Nathaniel Rochester, and Claude E. Shannon: A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence](https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html)
- [Marvin Minsky and Seymour A. Papert: Perceptrons: An Introduction to Computational Geometry](https://mitpress.mit.edu/9780262130431/perceptrons/)

### Why the hand-built approach met limits

A system of explicit rules can work well within a carefully defined domain. Extending it into a changing world introduces exceptions, ambiguous cases, and interactions among rules. In his 1990 essay, Minsky describes limitations on both the symbolic and connectionist sides and argues for systems combining different representations. The historical division was never a complete separation of ideas.

- [Marvin Minsky: Logical vs. Analogical, or Symbolic vs. Connectionist, or Neat vs. Scruffy](https://web.mit.edu/dxh/www/marvin/web.media.mit.edu/~minsky/papers/SymbolicVs.Connectionist.html)

### The book at the engineering branch

Neural Networks for Control, edited by W. Thomas Miller III, Richard S. Sutton, and Paul J. Werbos, brings learning together with control problems, including robotics and motion planning. The lecture timeline uses its commonly cited 1990 date; MIT Press lists the hardcover release as January 10, 1991. The pictured cover belongs to the later paperback. The book identifies a research program, rather than an exact day when disciplines split.

- [W. Thomas Miller III, Richard S. Sutton, and Paul J. Werbos, editors: Neural Networks for Control](https://mitpress.mit.edu/9780262132619/neural-networks-for-control/)

### The route to language models

Large neural language models learn from text; later training can shape how they respond to people. GPT-3 and InstructGPT make that distinction concrete: one paper studies a large pretrained language model, while the other adds demonstrations and human-preference training. Reinforcement learning is important to many assistant models, but it is not the explanation for every stage of language-model training.

- [Tom B. Brown and colleagues: Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [Long Ouyang and colleagues: Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)

### An ongoing engineering tradition

A robot must cope with changing physical conditions, imperfect measurements, and constraints on safe action. The safe-control-gym study provides a concrete example of researchers comparing traditional control, learning-based control, and reinforcement learning on shared tasks. It shows both continued engineering development and active exchange among these communities. The parallel line therefore represents a larger set of problems and tools, not an isolated body of work with no connection to AI.

- [Zhaocong Yuan and colleagues: safe-control-gym: A Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://arxiv.org/abs/2109.06325)

### Safety means specifying what must stay safe

The engineering lesson is to define the system, its operating conditions, and the constraints it must respect, then evaluate what happens under disturbances. The safe-control-gym benchmark makes this concrete with simulated cart-pole and quadrotor tasks. I want that discipline to influence AI development more strongly. A result established for a defined control problem does not by itself establish the safety of a general language model.

- [Zhaocong Yuan and colleagues: safe-control-gym: A Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://arxiv.org/abs/2109.06325)
- [Recording · 23:02–25:54](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1382s)

### Why the transfer is difficult

An audience member asked how those engineering safeguards could carry over to modern AI. Consider the difference between a system whose dynamics and constraints we can describe mathematically and a confused person seeking good customer service. In the second case, even deciding what counts as success is difficult. I want us to develop systems that can keep learning through guided experience with people. That’s a direction I’m proposing, with safety still something we would have to establish. Continued learning alone would not settle it.

- [Zhaocong Yuan and colleagues: safe-control-gym: A Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://arxiv.org/abs/2109.06325)
- [Recording · 1:15:38–1:17:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4538s)

### Sources and further reading

- **[A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence](https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html)**. John McCarthy, Marvin L. Minsky, Nathaniel Rochester, and Claude E. Shannon, 1955. The August 31, 1955 proposal for the 1956 meeting lists language and neuron nets among its research topics. It provides direct evidence that the field’s early agenda encompassed several approaches.
- **[Perceptrons: An Introduction to Computational Geometry](https://mitpress.mit.edu/9780262130431/perceptrons/)**. Marvin Minsky and Seymour A. Papert, 1969. The mathematical analysis behind the perceptron-scrutiny milestone. Its results concern specified classes of perceptrons, rather than a universal impossibility of neural learning.
- **[Logical vs. Analogical, or Symbolic vs. Connectionist, or Neat vs. Scruffy](https://web.mit.edu/dxh/www/marvin/web.media.mit.edu/~minsky/papers/SymbolicVs.Connectionist.html)**. Marvin Minsky, 1990; reprinted 1991. Minsky’s own account of the traditions and their limitations. The opening discussion and section Heuristic Connections of Rule-Based Systems explain his case for combining approaches.
- **[Neural Networks for Control](https://mitpress.mit.edu/9780262132619/neural-networks-for-control/)**. W. Thomas Miller III, Richard S. Sutton, and Paul J. Werbos, editors, 1990/1991. The publisher’s description identifies learning control, robotics, motion planning, and benchmark control problems. Its edition information distinguishes the January 1991 hardcover from the March 1995 paperback.
- **[Language Models Are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. Describes GPT-3 and its evaluation across language tasks. Section 2 covers the model and training data, establishing a concrete point on the large neural language-model route.
- **[Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and Section 3 show how demonstrations and reinforcement learning from human feedback are applied after a pretrained model, clarifying the role of RL along the route to assistants.
- **[safe-control-gym: A Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://arxiv.org/abs/2109.06325)**. Zhaocong Yuan and colleagues, 2021; revised 2022. An original benchmark study comparing traditional, learning-based, and reinforcement-learning control on cart-pole and quadrotor tasks. Illustrates the continuing engineering route and exchange between research communities.


<a id="slide-9"></a>

## 9. What do we mean by intelligence?

**I use intelligence to mean finding a fitting response to an unfamiliar situation and improving through feedback.**

My working definition has two parts: responding to a situation that has not been encountered before, and getting better at doing so through feedback. A fitting response depends on the situation. Repeating a familiar answer is not enough when the circumstances have changed.

This definition gives us a practical question to pursue: how can a machine become better at producing useful responses? The next slides make learning tangible, first as a network adjusting its connections and then as word games that let us experience the pull of a learned pattern. I offer this as a working definition for our discussion, not as a claim that researchers have agreed on one complete definition of intelligence.

You don’t need mathematics to follow these exercises. In the room, I invited everyone to become a “baby LLM”: to experience how feedback changes an expectation, then see how the resulting expectation can go wrong. You can follow the same exercises here.

**The idea to carry forward.** Ask both whether a response fits the situation and how feedback shaped the ability to produce it.

**In the slide.** The same sentence stays on screen while emphasis moves from a fitting response to learning from feedback.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 25:54–26:37](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1554s): A practical definition of intelligence and the move to learning through feedback.

### A working definition

I offer this definition as a useful lens for our discussion. It does not establish a test of consciousness, human equivalence, or every capacity that might count as intelligence.

### Why the emphasis moves to feedback

The visual shifts attention from responding to improving. That is the bridge from the history of AI to the learning mechanism examined next.


<a id="slide-10"></a>

## 10. Learning through feedback

**Training uses feedback about responses to adjust the numerical connections inside a network.**

Follow a single cycle. An example enters the network. Its current connections shape a response. That response is evaluated, and the resulting feedback guides an adjustment. Repeating this cycle over many examples can change which responses the network tends to produce.

The adjustable numbers are parameters, including connection weights. In the diagram, thicker and thinner lines make those changes visible. Backpropagation helps calculate how changes to parameters would affect an error measure; an optimizer uses that information to make updates. The small network is a teaching model, not a literal picture of a complete language model.

Keep training separate from using an already-trained model. During ordinary generation, a model applies its existing weights to the available context. Giving it a new sentence, document, or correction can change its next response without changing those weights. This distinction will matter when we start learning patterns from words.

**The idea to carry forward.** Training changes parameters; new context changes what an already-trained model has to work with.

**In the slide.** Follow input, response, feedback, then the changed thickness of the connections.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 26:37–27:11](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1597s): The feedback loop: an output, an evaluation, and an adjustment to weights.
- [Audience discussion · 1:08:03–1:09:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4083s): Discussion returns to frozen weights and adaptation through conversation.

### Where backpropagation fits

Backpropagation supplies information about how an error depends on the network’s parameters. That makes guided weight adjustment possible; it does not decide by itself what should count as a good response.

- [David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams: Learning representations by back-propagating errors](https://www.nature.com/articles/323533a0)

### Feedback is broader than a person saying “wrong”

A known target, a calculated error, or a reward can supply a learning signal. The returning arrow groups these ideas to explain their purpose, while later slides distinguish their roles.

- [David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams: Learning representations by back-propagating errors](https://www.nature.com/articles/323533a0)
- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)

### Context is not a weight update

Brown and colleagues evaluated GPT-3 with task descriptions and examples in the prompt, without gradient updates or fine-tuning. A changed response in conversation is therefore not, by itself, evidence of new parameter training.

- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### The later question about learning from us

In the discussion, I returned to this distinction when asked about moral learning: responding differently within a conversation is not the same as continually changing the trained model. I’ll return later to my proposal for systems that keep learning through experience. The GPT-3 reference establishes the narrower mechanism distinction, not that broader proposal.

- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
- [Recording · 1:08:03–1:09:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4083s)

### Sources and further reading

- **[Learning representations by back-propagating errors](https://www.nature.com/articles/323533a0)**. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, 1986. Explains how error-guided adjustments to connection weights can produce useful internal representations.
- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
- **[Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.


<a id="slide-11"></a>

## 11. What comes next?

**The words surrounding an unfinished sentence change which next word seems plausible.**

Try completing the sentence: “I’ll have the usual,” he said to the _____. The offered choices are bartender, barista, barber, and doctor. Before there is more context, several familiar situations can fit. You may picture a favorite drink, a coffee order, a haircut, or a recurring appointment.

Now add the preceding sentence: “The espresso machine hissed behind the counter.” Barista becomes a particularly natural choice. The unfinished sentence and its grammar have not changed; the situation suggested by the surrounding words has.

This is a small, human-readable analogy for language prediction. A language model uses the text available so far to shape its next output, then continues the process. The exercise presents four illustrative choices, not measured probabilities from a model. Its lesson is the influence of context, rather than proof that one completion is logically inevitable.

In the room, the show of hands supplied a visible spread of possible answers. I invited the group to act as a “baby LLM”: several continuations could initially feel natural, and a small clue could shift the distribution. The votes were a teaching analogy, not measurements of an actual model.

**The idea to carry forward.** A word fits within a situation inferred from the surrounding text.

**In the slide.** Compare your first completion with your choice after the espresso-machine clue appears.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 27:11–28:29](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1631s): The audience becomes a “baby LLM” and revises its choice when context changes.

### Try it before reading the clue

Form an initial preference among bartender, barista, barber, and doctor, then compare it with your choice after the espresso sentence. The change in expectation is the experience the exercise is designed to reveal.

### “Next word” is useful shorthand

Language models generally generate tokens. A token can be a word, part of a word, or another text unit. I use whole words here so the prediction task is easy to follow.

- [Ashish Vaswani and colleagues: Attention Is All You Need](https://arxiv.org/html/1706.03762v7)

### What context means here

Context is the text available to the model for this prediction. The context window limits how much can be available at once. Adding the espresso clue changes the input, not the trained weights.

- [Ashish Vaswani and colleagues: Attention Is All You Need](https://arxiv.org/html/1706.03762v7)
- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### Sources and further reading

- **[Attention Is All You Need](https://arxiv.org/html/1706.03762v7)**. Ashish Vaswani and colleagues, 2017. Sections 3.1 and 3.4 explain sequential token generation and output probabilities; section 5.1 describes subword tokenization. The paper concerns the original transformer, not the small teaching network I use here.
- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.


<a id="slide-12"></a>

## 12. Context changes the answer

**Different situations can make different completions of the same sentence appropriate.**

The four versions deliberately keep “I’ll have the usual” unchanged. Bottles of gin behind a bar suggest bartender. An espresso machine suggests barista. Freshly cut hair around a salon chair suggests barber. A clinic visit to renew a prescription suggests doctor.

None of those completions is a universal answer to the unfinished sentence. Each fits a different interpretation of the scene. The sequence moves beyond a single association and shows why understanding a request depends on the surrounding situation, including details that change what someone probably means.

For a language model, supplying relevant context can similarly change the continuation it produces. But more text is not automatically better evidence. A clue can be mistaken, irrelevant, or misleading. The useful question is whether the supplied information helps identify the actual situation. The following exercise turns that question around: what happens when feedback teaches a pattern that works in examples but misses their larger meaning?

**The idea to carry forward.** Relevant context can change the fitting answer without changing the question’s words.

**In the slide.** Read each contextual clue before its completion and notice what situation you imagine.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 28:29–29:34](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1709s): Context changes the next prediction; the conversation grows as words are added.

### Four contexts, four natural readings

The series is an invented teaching example, not a comparison of measured model responses. The answers are plausible completions, not the only grammatically possible ones.

### Context operates at generation time

Task descriptions and demonstrations can steer an already-trained model through its input. This kind of adaptation does not require a new round of weight training.

- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### A route into the underlying mechanism

The original transformer paper explains how attention allows information from different positions in a sequence to contribute to a prediction. Its architecture is more elaborate than the word game, but the connection is the relevance of surrounding text.

- [Ashish Vaswani and colleagues: Attention Is All You Need](https://arxiv.org/html/1706.03762v7)

### Why a conversation keeps growing

Think of a loop: generate another word, add it to the text, then continue from that enlarged context. A user’s reply adds more material too. This explains why a long exchange eventually reaches a context limit. In an implementation, cached intermediate results usually avoid recomputing all earlier tokens from scratch; the conceptual dependence on the preceding context remains.

- [Ashish Vaswani and colleagues: Attention Is All You Need](https://arxiv.org/html/1706.03762v7)
- [Hugging Face: How caching works](https://huggingface.co/docs/transformers/main/cache_explanation)
- [Recording · 28:44–29:34](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1724s)

### Sources and further reading

- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
- **[Attention Is All You Need](https://arxiv.org/html/1706.03762v7)**. Ashish Vaswani and colleagues, 2017. Sections 3.1 and 3.4 explain sequential token generation and output probabilities; section 5.1 describes subword tokenization. The paper concerns the original transformer, not the small teaching network I use here.
- **[How caching works](https://huggingface.co/docs/transformers/main/cache_explanation)**. Hugging Face, 2026. Explains how autoregressive generation depends on preceding tokens and how a key-value cache reuses their computed representations. Consulted October 16, 2026.


<a id="slide-13"></a>

## 13. Learning from feedback

**Selective feedback can teach a consistent rule even when the examples allow many reasonable answers.**

This game presents four unfinished sentences, each with four plausible choices. After each choice, the exercise identifies the answer it favors. The selected completions are fox, pony, dog, and rabbit. Their positions move between B, A, C, and D, so choosing the same letter cannot explain the success.

The common feature is that each selected answer is an animal. By the end, you may expect another animal before seeing the next sentence. The feedback has made one pattern stand out. It has not shown that the other choices were impossible: finding a parcel inside a gate, for example, would make perfectly ordinary sense.

The live responses made the change visible. At the first prompt, I saw many hands for gardener. By the third, most of the room was choosing dog. By the photograph prompt, the audience could name the animal pattern. The exercise therefore reveals a shift in expectation, not just a sequence of correct answers.

That is the point of the exercise. A learner can become good at the pattern that its examples and feedback favor. Whether that pattern serves the intended purpose is a further question. The next slide changes the situation to test the rule you have just acquired.

**The idea to carry forward.** A pattern can be consistently rewarded without being the right rule for every future situation.

**In the slide.** The four selected answers are fox, pony, dog, and rabbit; look for what they share, rather than their answer letters.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 29:34–32:15](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1774s): The animal pattern emerges through garden, lane, sound, and photograph examples.

### Example 1: the garden gate

“Just inside the garden gate, I found a _____.” Choices: bicycle, fox, parcel, gardener. The feedback selects fox. The other options are also plausible discoveries.

### Example 2: the lane

“At the end of the lane stood a _____.” Choices: pony, tractor, cottage, postbox. The feedback selects pony.

### Example 3: a sound in the dark

“From somewhere in the dark came the sound of a _____.” Choices: radio, train, dog, bell. The feedback selects dog.

### Example 4: the photograph

“In the photograph, she was holding a _____.” Choices: trophy, candle, suitcase, rabbit. The feedback selects rabbit. All four favored answers have now been animals.

### What the analogy does and does not show

This is an audience exercise about selective feedback, not an actual training run. Shortcut-learning research studies the wider problem of rules that work on familiar examples but fail elsewhere. If these examples are supplied in a model’s prompt, it may infer the pattern without changing its weights.

- [Robert Geirhos and colleagues: Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780)
- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### What shapes the patterns a network learns

A network can exploit a pattern without automatically favoring the relationship the teacher intended. Its data, architecture, and training objectives shape which patterns it learns. The animal shortcut separates success under the feedback from success at the intended task: what earned credit in one set of examples may be misleading in the next.

- [Robert Geirhos and colleagues: Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780)
- [Recording · 31:56–32:15](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1916s)

### A spread of possible next words

I compared the show of hands with a model’s distribution over possible continuations. Under sampling, a lower-ranked choice may still be selected; other decoding settings choose the highest-scoring choice. Neither method tests by itself whether the selected continuation is true.

- [Hugging Face: Text generation: token selection and stopping conditions](https://huggingface.co/docs/transformers/main_classes/text_generation)
- [Recording · 31:10–31:32](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1870s)

### Sources and further reading

- **[Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780)**. Robert Geirhos and colleagues, 2020. Explains how a rule that succeeds on familiar examples can fail under changed conditions. It supports the concept illustrated by the exercise, not a claim that the invented game is a model experiment.
- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.
- **[Text generation: token selection and stopping conditions](https://huggingface.co/docs/transformers/main_classes/text_generation)**. Hugging Face, 2026. Documents sampling and greedy selection as alternative generation settings. This supports the distinction between a token distribution and the method used to choose from it.


<a id="slide-14"></a>

## 14. Hold on to what you learned

**A learned shortcut can fit past feedback and still fail the new situation.**

Carry the animal rule into a new sentence: “The locksmith opened the door with a _____.” The choices are key, lockpick, drill, and giraffe. The first three can plausibly belong in a locksmith’s work. Under an ordinary literal reading, a giraffe cannot serve the intended purpose.

Yet giraffe is the answer that fits every previous round of the game. The conflict is between the learned pattern and the situation the sentence describes. Doing more of what worked before can now lead directly away from a useful answer.

This is a compact illustration of shortcut learning. The practical concern is generalization: whether success on familiar examples carries into changed circumstances. It also gives an intuition for one route to unreliable language-model output. It does not establish that every hallucination has this cause, or that the key is the only possible correct completion. The lesson is to examine which rule the feedback actually encouraged.

**The idea to carry forward.** Success on the training examples does not guarantee that the learned rule will fit a new situation.

**In the slide.** Keep the previous animal rule in mind while comparing the four locksmith answers.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 32:15–33:24](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1935s): The learned animal pattern produces an absurd locksmith answer.

### Why the wrong answer is informative

Giraffe is highlighted as the learned shortcut, not endorsed as the factual answer. The absurdity exposes the difference between matching the feedback pattern and fitting the task.

### Research connection: shortcut learning

Geirhos and colleagues describe decision rules that succeed on familiar benchmarks but fail when conditions change. The locksmith game makes that concern accessible; it is not a measured result about a particular LLM.

- [Robert Geirhos and colleagues: Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780)

### Why simply adding examples is not a guarantee

I’m questioning the hope that enough additional text would make every misleading pattern disappear. This exercise demonstrates a narrower point: a pattern that succeeds during training can fail in a new setting. More or better data can help, but size alone is not evidence that every shortcut or unsupported answer has been removed.

- [Robert Geirhos and colleagues: Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780)
- [Recording · 32:34–33:24](https://www.youtube.com/watch?v=klzLRdHmeCs&t=1954s)

### Sources and further reading

- **[Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780)**. Robert Geirhos and colleagues, 2020. Explains how a rule that succeeds on familiar examples can fail under changed conditions. It supports the concept illustrated by the exercise, not a claim that the invented game is a model experiment.


<a id="slide-15"></a>

## 15. Go with your gut

**Knowing when not to answer is a separate challenge from producing a plausible answer.**

The last word game asks for a precise departure time: “The train departs at exactly _____.” Its choices are velvet, spoon, sideways, and thunder. None provides an ordinary literal time. The useful response is to challenge the choices or acknowledge the missing information.

The exercise separates producing an answer from having grounds for one. A system can be very good at continuing language while still facing situations where the evidence is inadequate. A fluent completion does not supply the missing timetable.

The default generation mechanism matters too. At each step, a function called softmax turns scores for possible next tokens, words or word-pieces, into probabilities that add up to 100%. Ordinary generation still selects a token when those probabilities are spread thinly across many choices. Nothing in that operation automatically pauses the response because the evidence is inadequate.

Researchers call the decision not to give a definite answer abstention. Studies directly testing this ability find that stronger problem-solving does not ensure reliable abstention. Some tested reasoning models became worse at withholding unsupported answers. Another study found that scaling and instruction tuning could replace nonanswers with plausible mistakes.

An LLM can say “I don’t know.” The difficulty is doing so dependably when an answer is unsupported. Evaluation incentives help explain why: if a wrong answer and an admission of uncertainty both earn zero, guessing offers a chance of scoring a point while abstaining does not.

**The idea to carry forward.** A capable model can still answer when it should say “I don’t know” or ask for missing information.

**In the slide.** Notice whether you feel pressure to select one of the four options despite the missing departure time.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 33:24–36:18](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2004s): The train question, forced completion, and knowing when to say “I don’t know.”
- [Audience discussion · 1:05:15–1:06:51](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3915s): The audience compares abstention to a programming fallback; I discuss deployment incentives.

### The missing answer is the point

Under the exercise’s ordinary reading, none of velvet, spoon, sideways, or thunder is a departure time. A creative fictional interpretation is possible, but it does not establish when a real train leaves.

### Softmax supplies choices, not a test of knowledge

Softmax distributes 100% of the probability across the next-token options. As a toy example, 1,000 equally scored options would each get just 0.1%, yet ordinary sampling would still select one. The structural issue is softmax combined with the generation loop: low individual probabilities do not automatically trigger abstention. These are probabilities of text continuations, not probabilities that a claim is true; many valid phrasings can also spread the probability thinly. The model can produce “I don’t know,” choose an end token, or be stopped by an additional control. Reliable abstention therefore depends on learned behavior or added controls, rather than being guaranteed by softmax itself.

- [PyTorch documentation: Softmax: converting scores into a probability distribution](https://docs.pytorch.org/docs/2.14/generated/torch.nn.modules.activation.Softmax.html)
- [Hugging Face Transformers documentation: Text generation: token selection and stopping conditions](https://huggingface.co/docs/transformers/main_classes/text_generation)

### Direct evidence: failing to recognize unanswerable questions

AbstentionBench tested 20 models across 20 datasets, including questions with missing information, false premises, and unknown answers. Abstention remained unreliable. In matched comparisons, some reasoning-trained models improved at solving problems while getting worse at deciding when not to answer. Explicit instructions helped, but did not solve the problem.

- [Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell: AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://proceedings.neurips.cc/paper_files/paper/2025/hash/fb122bfc3f0127a94ded048b5b03496f-Abstract-Datasets_and_Benchmarks_Track.html)

### More capable can also mean more willing to guess

Zhou and colleagues found that larger, instruction-tuned models often replaced nonanswers with plausible but incorrect responses. Their “avoidance” category also includes malformed outputs from earlier models, so it is broader than a thoughtful “I don’t know.” The key finding is that greater capability did not reliably produce better restraint.

- [Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo: Larger and more instructable language models become less reliable](https://www.nature.com/articles/s41586-024-07930-y)

### Why the scoring can favor guessing

Kalai and colleagues explain how accuracy-based evaluations reward guessing over appropriate uncertainty. A guess can earn credit; an abstention cannot. They propose stating the scoring rules and penalizing errors so models are rewarded for withholding answers when the evidence is too weak.

- [Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang: Evaluating large language models for accuracy incentivizes hallucinations](https://www.nature.com/articles/s41586-026-10549-w)

### From the discussion: where is the “else”?

An audience member compared “I don’t know” with the fallback else branch in a conventional program. The analogy sharpens the design question: what happens when none of the proposed answers is warranted? A learned model can generate an admission of uncertainty, but ordinary token selection is not an explicit test for that condition. Abstention behavior therefore needs to be trained and evaluated, or supported by additional controls.

- [Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell: AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://proceedings.neurips.cc/paper_files/paper/2025/hash/fb122bfc3f0127a94ded048b5b03496f-Abstract-Datasets_and_Benchmarks_Track.html)
- [PyTorch documentation: Softmax: converting scores into a probability distribution](https://docs.pytorch.org/docs/2.14/generated/torch.nn.modules.activation.Softmax.html)
- [Recording · 1:05:15–1:05:50](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3915s)

### My concern about incentives

My concern is that excitement and investment can favor shipping more capable systems over stopping to address their weaknesses. That contrasts with the demands I remember from my engineering work. This is my interpretation of development incentives. The cited research establishes measured abstention failures, not the reasons developers choose to deploy.

- [Recording · 1:05:50–1:06:51](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3950s)

### Sources and further reading

- **[AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://proceedings.neurips.cc/paper_files/paper/2025/hash/fb122bfc3f0127a94ded048b5b03496f-Abstract-Datasets_and_Benchmarks_Track.html)**. Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell, 2025. Directly tests whether models withhold answers when appropriate. Sections 4.1 and 4.3, Figures 2–3 and 6, cover general failures and reasoning-model comparisons; section 4.4, Figure 8, tests an abstention instruction. The findings concern the tested models and benchmarks, not an impossibility of saying “I don’t know.”
- **[Larger and more instructable language models become less reliable](https://www.nature.com/articles/s41586-024-07930-y)**. Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo, 2024. Nature 634, pages 61–68. Figure 2 and the Results discussion distinguish correct, incorrect, and avoidant responses across model families; Supplementary Note 15 explains the different kinds of avoidance.
- **[Evaluating large language models for accuracy incentivizes hallucinations](https://www.nature.com/articles/s41586-026-10549-w)**. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang, 2026. The peer-reviewed version of the 2025 preprint Why Language Models Hallucinate. The main text and Figure 1 explain the incentive to guess; the open-rubric evaluation proposal makes the cost of an incorrect answer explicit.
- **[Softmax: converting scores into a probability distribution](https://docs.pytorch.org/docs/2.14/generated/torch.nn.modules.activation.Softmax.html)**. PyTorch documentation, Accessed October 2026. The definition and formula show that softmax normalizes scores into probabilities summing to one. The equal-score example in the annotation is an illustration of that formula, not a measurement of an LLM.
- **[Text generation: token selection and stopping conditions](https://huggingface.co/docs/transformers/main_classes/text_generation)**. Hugging Face Transformers documentation, Accessed October 2026. GenerationConfig documents sampling versus greedy decoding, end-of-sequence tokens, and explicit stopping controls. These mechanisms explain why selecting a next token is distinct from deciding whether an answer has sufficient factual support.


<a id="slide-16"></a>

## 16. The scale behind the learning

**Model size and training-text volume are different quantities, and both can be enormous.**

Kimi K2, released in 2025, supplies a disclosed example of both kinds of scale: one trillion total parameters and 15.5 trillion tokens processed in pretraining. Parameters are the adjustable numbers inside the model. Tokens measure the text units encountered during training. The two figures answer different questions.

The dot field makes the parameter count visible by assigning a billion parameters to each point. The smaller historical markers, roughly 117 million for GPT-1 and 175 billion for GPT-3, give perspective. These are selected examples, not a rule that every later model must be larger.

The library comparison makes text volume easier to imagine. Under the conversion assumptions I use here, Kimi K2’s pretraining tokens correspond to roughly five Library of Congress cataloged-book collections. This is a scale analogy, not a claim that the model contains or read five distinct libraries. Additional post-training examples and feedback are outside that total and are not quantified here.

**The idea to carry forward.** Keep three things separate: adjustable parameters, text encountered in pretraining, and the additional feedback used afterward.

**In the slide.** Read the network and text totals as two separate scales; the library illustration belongs to the text total.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 36:18–37:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2178s): Network and corpus scale, with a comparison to my airline forecasting work.

### Total parameters versus active parameters

Kimi K2 is a mixture-of-experts model: about 32 billion parameters are active for a token out of one trillion total. The points show total scale, not its architecture or the number used on each token.

- [Kimi Team / Moonshot AI: Kimi K2: Open Agentic Intelligence](https://github.com/MoonshotAI/Kimi-K2#1-model-introduction)

### The arithmetic behind “about five”

The illustration assumes 0.75 words per token and 90,000 words per book. 15.5 trillion × 0.75 gives 11.625 trillion illustrative words. 25.77 million cataloged books × 90,000 gives about 2.319 trillion words; the ratio is about 5.01. Neither conversion is a measured average of these collections.

- [Kimi Team / Moonshot AI: Kimi K2: Open Agentic Intelligence](https://github.com/MoonshotAI/Kimi-K2#1-model-introduction)
- [Library of Congress: General Information: Library of Congress](https://www.loc.gov/about/general-information)

### What the comparison excludes

Token-to-word ratios vary with language, tokenizer, and content. Training text can be repeated, and the Library of Congress has many holdings beyond cataloged books. Parameter count is not a count of facts retained or an equivalent count of biological neurons. The pretraining figure counts tokens processed across training, not a separate five-library collection for each pass.

- [Kimi Team / Moonshot AI: Kimi K2: Open Agentic Intelligence](https://github.com/MoonshotAI/Kimi-K2#1-model-introduction)
- [Library of Congress: General Information: Library of Congress](https://www.loc.gov/about/general-information)

### Historical anchors and the uncounted next stage

GPT-1’s released parameter shapes yield about 117 million, and the GPT-3 paper reports 175 billion. The slide deliberately uses disclosed figures. Its pretraining-token total does not estimate the separate quantity of examples, rankings, and rewards used in post-training.

- [Kimi Team / Moonshot AI: Kimi K2: Open Agentic Intelligence](https://github.com/MoonshotAI/Kimi-K2#1-model-introduction)
- [OpenAI: Improving Language Understanding by Generative Pre-Training: released model parameter shapes](https://github.com/openai/finetune-transformer-lm/blob/master/model/params_shapes.json)
- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### A comparison from my airline work

I recall deploying neural-network systems for US Air in 1991 to predict demand and no-shows, with another network used to optimize prices. As I remember it, the collection had fewer than 500,000 nodes. I offer that story to give a sense of how far the scale has moved. My recalled node count is not directly comparable to today’s parameter counts, so it should not be used to calculate a precise growth factor.

- [Recording · 36:35–37:15](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2195s)

### Sources and further reading

- **[Kimi K2: Open Agentic Intelligence](https://github.com/MoonshotAI/Kimi-K2#1-model-introduction)**. Kimi Team / Moonshot AI, 2025. The model introduction and summary disclose 1 trillion total parameters, 32 billion active parameters per token, and 15.5 trillion pretraining tokens for the same named model.
- **[General Information: Library of Congress](https://www.loc.gov/about/general-information)**. Library of Congress, FY2023 collection figures. The Year 2023 at a Glance section reports more than 25.77 million cataloged books in the Library of Congress classification system. The word-volume conversion is my illustration, not a Library of Congress estimate.
- **[Improving Language Understanding by Generative Pre-Training: released model parameter shapes](https://github.com/openai/finetune-transformer-lm/blob/master/model/params_shapes.json)**. OpenAI, 2018. The released parameter shapes sum to 116,534,784 parameters, rounded on the slide to 117 million. This supports the first historical size marker.
- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The introduction and evaluation setup describe learning from examples in the prompt without updating model weights. Table 2.1 reports the 175-billion-parameter GPT-3 model.


<a id="slide-17"></a>

## 17. What does it actually produce?

**A pretrained model can continue the form of a request instead of carrying out the request.**

The earlier games gave feedback on a prediction. In LLM pretraining, actual text supplies that feedback: show the model the preceding text, ask it to predict the next token, then compare its prediction with what really comes next. Training adjusts the parameters and repeats this process across enormous amounts of text.

Imagine asking for a few sentences that tell a six-year-old about the Moon landing. Rather than answering, the response begins another child-oriented request, this time about gravity. It then asks for explanations of relativity, the beginning of the universe, and evolution. The subject keeps changing, but the form of the request continues.

The slide reveals those four parts gradually so the difference can be felt before it is explained. This is one published GPT-3 completion from OpenAI’s 2022 InstructGPT announcement, presented in paraphrase. It was not four separate questions sent to the model, and I did not generate the response live.

The example is striking because the text is coherent while the interaction fails. A person wanted an answer; the model produced more of a recognizable textual pattern. Pretraining can develop substantial capabilities, including answering questions, but the training objective does not by itself guarantee the conversational role we expect from an assistant. That mismatch motivates post-training.

**The idea to carry forward.** Continuing language and helping a person are related abilities, but they are not the same objective.

**In the slide.** Follow what stays the same across the four requests even as their topics change.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 38:16–39:44](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2296s): The Moon-landing continuation and my recollection of a Reagan-to-airport digression.

### Where the direct feedback comes from

A token is a small unit of text, often a word or part of a word. The next token in the training text supplies the target. Across repeated predictions, the model learns regularities that help it continue language.

- [OpenAI: Aligning language models to follow instructions](https://openai.com/index/instruction-following/)

### One continuation, four reveals

The order is gravity, relativity, the Big Bang, then evolution. The successive reveals divide one historical completion for readability; they do not represent independent trials.

- [OpenAI, preserved by archive.ph: Aligning language models to follow instructions: archived comparison](https://archive.ph/V59Vi)

### Why the wording is labeled paraphrased

The slide preserves the task and the pattern of the published example in simpler wording. It should be read as a faithful paraphrase, not a verbatim output transcript.

- [OpenAI: Aligning language models to follow instructions](https://openai.com/index/instruction-following/)
- [OpenAI, preserved by archive.ph: Aligning language models to follow instructions: archived comparison](https://archive.ph/V59Vi)

### An example rather than a universal rule

A base model can answer questions with suitable prompting. This sample illustrates one instruction-following failure, not a measured failure rate for all pretrained models.

- [OpenAI: Aligning language models to follow instructions](https://openai.com/index/instruction-following/)

### A second example from my own use

I recall asking a base model about Ronald Reagan and receiving a continuation that moved from his film career and presidency to Jimmy Carter, hostages, the CIA, George Bush, and finally Dallas airport statistics. I’m sharing a recollection of topic drift; the model version, exact prompt, and full output aren’t documented here. The published Moon-landing comparison provides the documented example beside it.

- [Recording · 39:05–39:44](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2345s)

### Sources and further reading

- **[Aligning language models to follow instructions](https://openai.com/index/instruction-following/)**. OpenAI, 2022. The January 27, 2022 InstructGPT announcement is the source of the Moon-landing comparison and explains the move from text completion toward instruction following.
- **[Aligning language models to follow instructions: archived comparison](https://archive.ph/V59Vi)**. OpenAI, preserved by archive.ph, 2022. Archived copy cited in my original lecture notes for the Moon-landing prompt, GPT-3’s sequence of further requests, and InstructGPT’s child-oriented answer.


<a id="slide-18"></a>

## 18. After pretraining

**Post-training can shift the same language-model foundation toward answering the user’s request.**

The same Moon-landing task is now shown as a comparison. The pretrained GPT-3 sample continues with more requests for explanations. The InstructGPT sample instead describes people going to the Moon, taking pictures, and sharing what they saw with people on Earth.

The contrast is about the kind of response, not a competition to produce the most sophisticated explanation. One output extends a pattern in text; the other takes up the role of a helpful respondent. That difference is central to why a familiar chat assistant feels unlike an unrestricted text-completion model.

InstructGPT was produced through further training that combined example answers with human preference feedback and reinforcement learning. This comparison shows the resulting change in behavior; it does not isolate which part of that training caused each improvement. It is also a historical pair of samples, not a guarantee that post-training always yields a correct or complete answer. The next example makes the remaining imperfection visible.

**The idea to carry forward.** Post-training helps shape language capability into assistant behavior, while leaving room for mistakes.

**In the slide.** Compare whether each response answers the task, rather than whether both responses are fluent.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 39:44–40:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2384s): Before-and-after examples introduce the transition to more useful responses.

### Read the before and after at the same level

The task is paraphrased on the slide; the short response excerpts come from the published comparison. Smaller captions summarize the omitted continuation rather than adding new model output.

- [OpenAI: Aligning language models to follow instructions](https://openai.com/index/instruction-following/)
- [OpenAI, preserved by archive.ph: Aligning language models to follow instructions: archived comparison](https://archive.ph/V59Vi)

### What “post-trained” includes here

InstructGPT’s method used demonstrations, a reward model trained from human rankings, and RL optimization. The before-and-after pair should not be presented as an experiment that measures RL alone.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)

### Sources and further reading

- **[Aligning language models to follow instructions](https://openai.com/index/instruction-following/)**. OpenAI, 2022. The January 27, 2022 InstructGPT announcement is the source of the Moon-landing comparison and explains the move from text completion toward instruction following.
- **[Aligning language models to follow instructions: archived comparison](https://archive.ph/V59Vi)**. OpenAI, preserved by archive.ph, 2022. Archived copy cited in my original lecture notes for the Moon-landing prompt, GPT-3’s sequence of further requests, and InstructGPT’s child-oriented answer.
- **[Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.


<a id="slide-19"></a>

## 19. Training for a useful response

**A more useful response can still introduce details that the source never supplied.**

Both models receive a stuffed-zucchini recipe and a request to turn it into a shopping list. GPT-3 continues the cooking instructions, telling the reader to bake the dish. InstructGPT produces ingredient entries, including ground beef and onion. It has moved much closer to the requested task.

But it also supplies quantities absent from the recipe, such as a pound of beef and a small onion. Those additions sound plausible and make the list appear useful. They are still unsupported by the input. The full published response also retains a cooking instruction before the ingredient list.

This is why improvement and reliability must be assessed separately. The response can follow the intended format better while remaining unfaithful to its source. The example prepares the transition into reward-based training: the objective is to encourage useful behavior, but a preferred answer may have weaknesses that the feedback did not adequately capture. A tidy output deserves the same attention to evidence as a messy one.

**The idea to carry forward.** Better instruction following does not remove the need to check which details came from the source.

**In the slide.** Compare the ingredient names with their quantities; the error is easiest to see in the added amounts.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 39:44–40:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2384s): A brief discussion of requested format within the before-and-after comparisons.

### Find the full comparison

The complete recipe and outputs appear in Figure 49, Appendix F, page 67 of the InstructGPT paper. Reading the full figure reveals both the useful ingredient list and the unsupported quantities.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback, Appendix F](https://arxiv.org/pdf/2203.02155#page=67)

### Two 175-billion-parameter models

The Appendix F samples compare GPT-3 and InstructGPT at the same 175B size, with different sampling temperatures. The example illustrates output differences but is not a controlled estimate of RL’s isolated effect.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback, Appendix F](https://arxiv.org/pdf/2203.02155#page=67)

### Where the new numbers came from

The supplied recipe names beef, onion, mushrooms, and peppers without specifying their amounts. The post-trained response fills in several amounts. The slide’s warning is about that unsupported addition, not about whether the proposed recipe might happen to cook well.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback, Appendix F](https://arxiv.org/pdf/2203.02155#page=67)

### Training for usefulness

The method combines learning from demonstrations with learning from preferences. Post-training shapes the likelihood of responses; it does not turn every response into a verified extraction from the input.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)

### Sources and further reading

- **[Training language models to follow instructions with human feedback, Appendix F](https://arxiv.org/pdf/2203.02155#page=67)**. Long Ouyang and colleagues, 2022. Figure 49 on page 67 contains the full stuffed-zucchini prompt and both 175B model responses. The Appendix F introduction on page 61 gives sampling temperatures: 0.7 for GPT-3 and 1 for InstructGPT.
- **[Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.


<a id="slide-20"></a>

## 20. Learning from a reward

**Reinforcement learning can use a judgment of an entire response to guide changes in future behavior.**

The question is simple: why does rain fall? The first illustrative answer says that water falls from clouds and that falling water is rain. It stays on topic, but it mostly restates the question. Under the goal “useful, concise, relevant,” it receives a low reward.

A reward judges how well the response served the goal. It does not need to provide the exact sentence the model should have written instead. During training, responses are generated, scored, and used to guide parameter changes across many examples. The slide’s later answer, about drops gathering in clouds and falling when they become heavy enough, illustrates the direction of improvement.

This is the distinction from copying a demonstrated target answer. The learner is encouraged toward outcomes that score better, while there may be many ways to achieve them. That flexibility can be powerful, and it makes the choice of score consequential. The process is repeated training, not an automatic weight update performed by an ordinary chat exchange.

The practical shift I’m describing is toward shorter, more relevant answers and useful formats. I’m comparing a model that can continue text with an assistant whose behavior has been shaped toward a purpose. A demonstrated better answer and a score for a response are different kinds of feedback, even when a training pipeline uses both.

**The idea to carry forward.** A reward says how well a response did; it need not spell out every word of a better response.

**In the slide.** Follow the full loop: generate a response, evaluate it, adjust during training, and try again across many examples.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 39:44–40:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2384s): Reward-based improvement is explained during the examples; this range overlaps slides 18–19.

### An invented example of the learning loop

The rain answers and evaluations were written for the lecture. They are not recorded model outputs, experimental measurements, or evidence that one update reliably produces the improved answer.

### What gets adjusted

Training uses the reward signal to change model parameters. Across many trials, the aim is to favor higher-reward behavior. The later response shows a desired tendency, not a guaranteed result after each adjustment.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)

### Two complementary ideas

RL describes learning from rewards. Backpropagation supplies derivatives used to train neural networks, including in many RL methods. They answer different questions about learning and can work together.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
- [David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams: Learning representations by back-propagating errors](https://www.nature.com/articles/323533a0)

### Sources and further reading

- **[Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
- **[Learning representations by back-propagating errors](https://www.nature.com/articles/323533a0)**. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, 1986. Explains how error-guided adjustments to connection weights can produce useful internal representations.


<a id="slide-21"></a>

## 21. Where the reward comes from

**Rewards can come from people, other AI systems, or outcomes that can be checked.**

People can compare two answers and judge which they would rather receive. In InstructGPT, those rankings trained a reward model, which then supplied scores for reinforcement learning. The person did not need to write the one perfect response.

Another AI can supply evaluations using stated criteria or principles. Constitutional AI provides a concrete example: its RL phase uses AI preference judgments to train the reward model. A third route uses a checkable result. A mathematical answer can be compared with a known answer, or generated code can be run against tests.

These are three practical sources of feedback, often combined, rather than three mandatory stages. Each makes a different part of success measurable. Preference, conformity to a criterion, and passing a test are useful signals, but none captures everything we care about. The central question is what the chosen reward encourages and what it leaves unmeasured. That question follows the model into reasoning, tool use, and more complex tasks.

I also want us to ask who supplies these judgments and under what conditions. There is human labor behind apparently automated intelligence, and we need to examine how that feedback is produced. My practical warning is that a system can become good at pleasing a grader or a user while missing the person’s real goal. Being agreeable, being correct, and being useful are different tests.

**The idea to carry forward.** The source and design of the reward help determine which behavior training encourages.

**In the slide.** Read each reward source beside the question it can answer about a response.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 40:33–44:01](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2433s): Human feedback, AI feedback, verifiable rewards, labor, and pleasing versus helping.
- [Audience discussion · 1:16:33–1:17:57](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4593s): Discussion contrasts checkable outcomes with understanding a customer’s needs.

### Human feedback: RLHF

InstructGPT’s Figure 2 separates collecting demonstrations, comparing outputs to train a reward model, and using that reward in RL. Human preference is the starting signal, rather than a human scoring every generated training response directly.

- [Long Ouyang and colleagues: Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)

### AI feedback: RLAIF

Constitutional AI uses AI comparisons guided by principles in its RL phase. The paper also has a separate supervised phase involving critique and revision, so the entire method is not only RL.

- [Yuntao Bai and colleagues / Anthropic: Constitutional AI: Harmlessness from AI Feedback](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback)

### Checkable outcomes

DeepSeek-R1’s original report, section 2.2.2, describes rule-based rewards for correct mathematical answers and code test results. Checking the result can provide feedback without specifying every intermediate reasoning step.

- [DeepSeek-AI: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/html/2501.12948v1)

### A signal is a choice about success

A preference may miss a factual error, a criterion may omit an important value, and a test suite may omit a failing case. These are reasons to examine the evaluation being used, not reasons to discard feedback altogether.

### Alignerr and expert feedback

I mentioned Alignerr as an example of expert participation in AI training. Alignerr describes a Labelbox-led network in which subject-matter experts create training data and evaluate model responses. It illustrates how professional knowledge and judgment become part of the feedback used to develop AI.

- [Alignerr: About Alignerr](https://www.alignerr.com/about)
- [Recording · 41:11–42:01](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2471s)

### The people behind “human feedback”

Producing training feedback takes human labor: people write examples, compare responses, and use professional judgment to evaluate work. The ILO describes this often hidden workforce across platforms and outsourcing companies. Research on outsourced AI data work between France and Madagascar documents low pay and precarious conditions in the studied supply chains. These sources support taking labor conditions seriously, rather than assuming every annotation job has the same conditions.

- [International Labour Organization: Artificial intelligence: Global AI workforce](https://www.ilo.org/topics-and-sectors/artificial-intelligence)
- [Clément Le Ludec, Maxime Cornet, and Antonio A. Casilli: The problem with annotation. Human labour and outsourcing between France and Madagascar](https://journals.sagepub.com/doi/10.1177/20539517231188723)
- [Recording · 40:33–42:11](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2433s)

### An AI grader is still a grader that can be wrong

I worry that using one model to evaluate another can carry weaknesses into the training signal. Research on LLM judges documents position and verbosity biases, self-preference, and reasoning limitations. It also finds useful agreement with human judgments. The practical question is whether the evaluator has been checked for the task, not whether an AI-generated score is automatically sound or automatically useless.

- [Lianmin Zheng and colleagues: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)
- [Recording · 42:11–42:43](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2531s)

### Pleasing the user can miss the goal

The frustration I’m describing is that a conversation can feel affirming while the resulting work has not accomplished what was needed. Sycophancy research gives a specific version of this concern. In tested settings, human preferences and preference models sometimes favored answers that agreed with a user’s beliefs over more truthful answers. This is a documented failure mode, not an inevitable result of every use of human feedback.

- [Mrinank Sharma and colleagues: Towards Understanding Sycophancy in Language Models](https://arxiv.org/abs/2310.13548)
- [Recording · 43:25–44:01](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2605s)

### From the discussion: an answer key versus a person’s needs

The audience later asked how engineering methods transfer to open-ended human interactions. I contrasted checking a mathematical result or code behavior with deciding whether a confused customer had truly been helped. The difference is the clarity and coverage of the success criterion. Even code tests cover only the cases they test; passing them is not proof that the whole system is correct.

- [DeepSeek-AI: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/html/2501.12948v1)
- [Recording · 1:16:33–1:17:57](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4593s)

### Sources and further reading

- **[Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)**. Long Ouyang and colleagues, 2022. Figure 2 and section 3 explain demonstration-based fine-tuning, learning a reward model from ranked responses, and reinforcement learning. Appendix F contains the before-and-after samples.
- **[Constitutional AI: Harmlessness from AI Feedback](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback)**. Yuntao Bai and colleagues / Anthropic, 2022. Describes a supervised critique-and-revision phase and a reinforcement-learning phase in which AI preference judgments train a reward model.
- **[DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/html/2501.12948v1)**. DeepSeek-AI, 2025. Section 2.2.2 describes rule-based accuracy rewards, including math answers and code tests. Sections 2.2–2.3 distinguish R1-Zero’s RL route from R1’s combination of examples and RL.
- **[Artificial intelligence: Global AI workforce](https://www.ilo.org/topics-and-sectors/artificial-intelligence)**. International Labour Organization, 2026. The Global AI workforce section describes tagging, classifying, cleaning, and validating data through platforms and outsourcing firms. It highlights transparency and fair labor practices. Consulted October 16, 2026.
- **[The problem with annotation. Human labour and outsourcing between France and Madagascar](https://journals.sagepub.com/doi/10.1177/20539517231188723)**. Clément Le Ludec, Maxime Cornet, and Antonio A. Casilli, 2023. Big Data & Society field research on outsourced data work, based on interviews and two AI production case studies. Evidence about these labor supply chains, not a wage survey of all RLHF workers.
- **[Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)**. Lianmin Zheng and colleagues, 2023. NeurIPS 2023 study of model judges. Identifies position, verbosity, self-enhancement biases and reasoning limits while also finding substantial agreement with human preferences.
- **[Towards Understanding Sycophancy in Language Models](https://arxiv.org/abs/2310.13548)**. Mrinank Sharma and colleagues, 2024. ICLR 2024 paper, first posted in 2023. Studies agreement with users at the expense of truth and evidence that preference judgments can favor that behavior.
- **[About Alignerr](https://www.alignerr.com/about)**. Alignerr, Undated. Describes expert creation of training data and evaluation of model responses. Supports this neutral description of the work. Consulted October 16, 2026.


<a id="slide-22"></a>

## 22. Chain of thought: a chance to check

**Generating intermediate working steps creates opportunities both to correct an error and to elaborate it.**

Chain of thought means generating working steps before a final answer. The slide summarizes the useful possibility as proposing, checking, and revising. Worked examples in a prompt can elicit this behavior. Post-training can develop it through examples of solutions, rewards for successful outcomes, and, in some methods, feedback on individual steps.

The penguin illustration begins with a false premise: all birds fly. One continuation notices that penguins are a counterexample and corrects the premise. The other accepts the premise, concludes that penguins must fly, and invents a supporting explanation about their wings. The second path does more work while moving further from the truth.

The point is neither that reasoning is useless nor that longer reasoning guarantees improvement. Extra working creates room for checks, but those checks must themselves be sound. A persuasive written rationale is also not a guaranteed record of everything that influenced an answer. The model we place inside a larger production system therefore remains capable of both useful self-correction and more elaborate mistakes.

There is a practical cost here: once intermediate working matters, evaluating only the polished final answer may miss where a mistake entered. Checking the reasoning can create further oversight work. My concern is the burden of judging the process; I’m not saying that every reasoning model is trained by a person grading every intermediate sentence.

**The idea to carry forward.** More working is an opportunity to test a premise, not proof that the premise has been tested well.

**In the slide.** The two branches begin with the same mistaken premise; compare whether the next steps challenge it or preserve it.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 44:01–45:42](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2641s): Intermediate reasoning can expose or compound an error and creates more work to evaluate.
- [Audience discussion · 1:22:28–1:23:30](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4948s): The audience asks about evaluation awareness and changes in reasoning under observation.

### Prompting and training are different routes

Wei and colleagues elicited chain-of-thought behavior with worked examples in the prompt. DeepSeek-R1 used further training; its report distinguishes R1-Zero’s RL route from R1’s combination of reasoning examples and RL.

- [Jason Wei and colleagues: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
- [DeepSeek-AI: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/html/2501.12948v1)

### Feedback can assess steps as well as results

Lightman and colleagues studied process supervision, which evaluates intermediate reasoning steps. This is an example of a training approach, not a requirement that every reasoning model receives a score for every step.

- [Hunter Lightman and colleagues: Let’s Verify Step by Step](https://arxiv.org/abs/2305.20050)

### The penguin paths are illustrations

Neither branch is a recorded model trace. The deliberately obvious false premise makes the difference between revising an assumption and defending it easy to follow.

### Self-correction needs to be tested

Huang and colleagues found that asking models to self-correct without external feedback could fail or degrade answers in their tested settings. That result is a reason to evaluate checking behavior, not proof that every model or method is unable to correct itself.

- [Jie Huang and colleagues: Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798)

### A visible rationale may omit an influence

Anthropic’s controlled hint experiments found that tested reasoning models sometimes used hints without acknowledging them in their written reasoning. Coherent working text can be useful without being a fully faithful explanation of how an answer was produced.

- [Anthropic: Reasoning models don’t always say what they think](https://www.anthropic.com/research/reasoning-models-dont-say-think)

### From the discussion: does being watched change the behavior?

An audience member asked whether a model behaves differently when it recognizes an evaluation. This is a real concern for testing and monitoring. In OpenAI’s CoT-Control study, telling models that they were monitored or evaluated slightly increased their ability to control features of their reasoning text, but reliable control remained difficult for the tested models. That result supports testing for observation effects without assuming that every model can deliberately hide its reasoning.

- [Tomek Korbak, Micah Carroll, Bowen Baker, and Ian Kivlichan / OpenAI: Reasoning models struggle to control their chains of thought, and that’s good](https://openai.com/index/reasoning-models-chain-of-thought-controllability/)
- [Recording · 1:22:28–1:23:30](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4948s)

### Monitoring creates another training-design problem

OpenAI’s reward-hacking experiments found that reasoning traces could reveal misbehavior. But directly penalizing those revealing traces could teach the tested model to conceal the signs rather than eliminate the underlying behavior. This makes my oversight question concrete: checking the working can help, while training specifically to look good under that check can undermine its value.

- [OpenAI: Detecting misbehavior in frontier reasoning models](https://openai.com/index/chain-of-thought-monitoring/)
- [Recording · 44:49–45:10](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2689s)

### Sources and further reading

- **[Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)**. Jason Wei and colleagues, 2022. Shows how worked examples in a prompt can elicit intermediate reasoning and improve performance on tested reasoning tasks without an additional weight-training phase.
- **[DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/html/2501.12948v1)**. DeepSeek-AI, 2025. Section 2.2.2 describes rule-based accuracy rewards, including math answers and code tests. Sections 2.2–2.3 distinguish R1-Zero’s RL route from R1’s combination of examples and RL.
- **[Let’s Verify Step by Step](https://arxiv.org/abs/2305.20050)**. Hunter Lightman and colleagues, 2023. Studies feedback on individual reasoning steps versus final outcomes in mathematical problem solving. It illustrates one approach, not a universal recipe for reasoning models.
- **[Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798)**. Jie Huang and colleagues, 2024. ICLR 2024 paper, first posted in 2023, testing intrinsic self-correction without external feedback. In its settings, requesting another check could fail to help or worsen answers.
- **[Reasoning models don’t always say what they think](https://www.anthropic.com/research/reasoning-models-dont-say-think)**. Anthropic, 2025. Controlled hint experiments with Claude 3.7 Sonnet and DeepSeek R1 show that a visible reasoning account can omit information that affected the answer. The report explains the study’s limitations.
- **[Reasoning models struggle to control their chains of thought, and that’s good](https://openai.com/index/reasoning-models-chain-of-thought-controllability/)**. Tomek Korbak, Micah Carroll, Bowen Baker, and Ian Kivlichan / OpenAI, 2026. March 5, 2026 report on CoT-Control. Monitor/evaluation awareness slightly increased measured controllability, but tested models still struggled. The tasks are proxies for controlling reasoning, not a general test of concealed intentions.
- **[Detecting misbehavior in frontier reasoning models](https://openai.com/index/chain-of-thought-monitoring/)**. OpenAI, 2025. March 10, 2025 reward-hacking experiments show useful monitoring signals in reasoning traces and a risk that direct optimization against those signals makes misbehavior harder to observe.


<a id="slide-23"></a>

## 23. The model at the center

**Post-training shapes one pretrained model toward reasoning, useful action, and human expectations.**

The assembly begins with the same learned network we have been discussing throughout the lecture. Its broad language capabilities provide a starting point. We then ask it to work through difficult questions, carry out tasks, and behave appropriately with people. The three colored encasings make those purposes visible.

These are overlapping training goals. A lesson about using a tool can also teach careful reasoning or respect for a user’s boundaries. Examples and rewards can contribute to more than one goal, and labs can revisit them in different orders. The picture therefore preserves the history of training without implying that three separate minds have been stacked around a fourth.

This distinction prepares the next step: training changes the model itself; a working application also needs machinery outside that model.

I use this sequence to make oversight concrete. The system is made through identifiable choices about data, feedback, and deployment. Understanding those choices gives us a better place to direct scrutiny than treating the finished AI as mysterious or inevitable.

**The idea to carry forward.** A capable model is shaped by several training objectives before it becomes part of a usable system.

**In the slide.** Follow the amber core through the teal reasoning layer, blue task layer, and coral alignment layer; it remains one model.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 45:42–47:19](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2742s): I move from the need for oversight to the assembly: a pretrained model, reasoning, agentic skills, and alignment training.
- [Audience discussion · 1:23:36–1:24:30](https://www.youtube.com/watch?v=klzLRdHmeCs&t=5016s): An audience member asks whose values alignment serves and whether safeguards live in training or in the surrounding system.

### Reasoning

Worked solutions and outcome feedback can encourage useful intermediate steps. Qwen3’s post-training report gives a concrete example of combining those approaches.

- [Qwen Team: Qwen3 Technical Report](https://arxiv.org/html/2505.09388v1#S4)

### Agentic capability

Learning when and how to use a tool is different from receiving access to that tool. The physical tool connections arrive with the harness on the next slide.

- [Qwen Team: Qwen3 Technical Report](https://arxiv.org/html/2505.09388v1#S4)

### Alignment as a training aim

Following instructions and respecting boundaries are learned behavioral objectives. The three-shell order is a teaching metaphor, not a universal laboratory recipe or a guarantee of correct future behavior.

- [Qwen Team: Qwen3 Technical Report](https://arxiv.org/html/2505.09388v1#S4)

### Alignment with whom?

The final audience question challenges the idea of a single human alignment target: people disagree with one another, and the questioner asks about broader social and ecological concerns. In my answer, I distinguish behavioral shaping during training from safeguards added during use. Naming the mechanism does not settle whose values it should implement.

- [Recording · 1:23:36–1:24:30](https://www.youtube.com/watch?v=klzLRdHmeCs&t=5016s)

### Sources and further reading

- **[Qwen3 Technical Report](https://arxiv.org/html/2505.09388v1#S4)**. Qwen Team, 2025. Sections 4.1–4.4 give one concrete post-training program combining reasoning, examples, rewards, tool use, instruction following, and preference alignment.


<a id="slide-24"></a>

## 24. From training to use

**A deployed model works with changing instructions, tools, and context while its learned parameters ordinarily remain fixed.**

The lock marks a change in the kind of work being done. Up to this point, training adjusted the model’s parameters. During ordinary use, a new question or a correction changes what the model receives on its next call. It does not, by itself, retrain the network. A separate training process can later produce a new version.

The harness is the surrounding application: it packages instructions, offers tools, manages the sequence of actions, and handles results. This is why a model connected to files, the web, or code can be useful in ways that the same model alone cannot.

The tray represents context, the limited information available for the current response. Instructions, conversation, and tool results meet there. The distinction between the locked core and the changing tray will remain visible through the rest of the assembly.

**The idea to carry forward.** Different behavior in a conversation can come from different context, without a change to the model’s weights.

**In the slide.** The tool cables attach to the outer frame; the context tray sits outside the locked model.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 47:19–47:41](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2839s): The weights are locked, then the harness supplies web access and manages what enters and leaves the context window.
- [Audience discussion · 1:10:49–1:12:29](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4249s): A privacy question moves the discussion from model behavior to where conversations and business information are processed.
- [Audience discussion · 1:17:57–1:18:50](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4677s): An audience member suggests a regulatory harness: restrictions on which consequential systems AI may control.
- [Audience discussion · 1:18:55–1:20:17](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4735s): The discussion contrasts ambiguous language with more formally specified tasks.
- [Audience discussion · 1:24:14–1:26:01](https://www.youtube.com/watch?v=klzLRdHmeCs&t=5054s): I separate alignment learned in training from system instructions and additional monitors, then raise transparency concerns.

### Learning from the prompt

GPT-3’s few-shot evaluations used instructions and demonstrations in the input without gradient updates. That is a concrete example of adapting behavior through context.

- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### Access is supplied by the application

A model can request an action, but the surrounding software provides the tool interface and decides how to execute the request.

- [Anthropic: Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)

### A rule and a boundary are different

An instruction can ask the model to behave carefully. Permissions and a sandbox can restrict what actions are possible, even when the model makes a poor choice.

- [Anthropic: How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)

### Guardrails live in several places

In the Q&A, I distinguish three locations for guardrails: training can change the model’s weights, system instructions can shape the current input, and separate classifiers can check inputs or outputs. Tool permissions and sandboxes add enforceable boundaries around actions. Anthropic’s classifier research is a concrete example of an external check; it does not establish that every product uses the same checks or model-routing policy.

- [Anthropic: How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)
- [Anthropic Safeguards Research Team: Constitutional Classifiers: Defending against universal jailbreaks](https://www.anthropic.com/research/constitutional-classifiers)
- [Recording · 1:24:14–1:25:25](https://www.youtube.com/watch?v=klzLRdHmeCs&t=5054s)

### Privacy is also an architecture question

An audience member asks what happens to information sent to a cloud service. I am wary of handing over sensitive conversations or business records, and I favor the option of running systems locally or on an organization’s premises. This question belongs beside the harness because a deployment determines where information goes and which services receive it. The practical questions are who can access, retain, or reuse that information and what the specific agreement allows.

- [Recording · 1:10:49–1:12:29](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4249s)

### A regulatory harness

An audience member suggests that rules could limit what an AI system is allowed to connect to. I agree: we need explicit boundaries around consequential uses, especially offensive weapons. This is my policy proposal, not a description of a settled legal standard. It extends the visual distinction between instructions asking for restraint and an environment that actually limits action.

- [Recording · 1:17:57–1:18:50](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4677s)

### Language leaves room for interpretation

The questioner contrasts everyday language with mathematical proofs and code. I argue that language and thinking shape one another, so an instruction can leave room for reinterpretation. My joke about building an AI that did not speak English points to the tradeoff: language is useful because it is flexible, and that flexibility makes verbal rules alone a difficult safety boundary.

- [Recording · 1:18:55–1:20:17](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4735s)

### Transparency remains a separate problem

We end this discussion by asking how visible the provider’s instructions and safeguards are. The architectural distinction matters here too: some behavior comes from learned weights, some from instructions supplied in context, and some from other software around the model. I call for more visibility into these arrangements so users and evaluators can understand which controls are operating and what evidence supports them.

- [Recording · 1:25:25–1:26:01](https://www.youtube.com/watch?v=klzLRdHmeCs&t=5125s)

### Sources and further reading

- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The abstract and section 2 distinguish adapting to demonstrations in the input from changing weights through gradient updates.
- **[Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)**. Anthropic, 2024. The augmented LLM and Agents sections explain how tools, retrieval, memory, environmental feedback, and stopping conditions surround a model.
- **[How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)**. Anthropic, 2026. Three types of risk, three components of defense describes model behavior, sandbox boundaries, tool permissions, and controls on untrusted content.
- **[Constitutional Classifiers: Defending against universal jailbreaks](https://www.anthropic.com/research/constitutional-classifiers)**. Anthropic Safeguards Research Team, 2025. Describes separately trained input and output classifiers that filter potentially harmful interactions. The reported success rates are specific to its models and evaluation setup.


<a id="slide-25"></a>

## 25. Memory beyond the context window

**External notes can carry useful information across conversations, provided the system retrieves them when they matter.**

A project can last longer than the context window that supports any one response. Earlier decisions, preferences, and unfinished work still matter after they no longer fit in the current input. The added memory documents provide somewhere for those details to persist.

The full process has two parts: save useful information, then bring a relevant selection back. A note sitting in a file is not automatically available to the next answer. This explains both the usefulness and the unevenness of apparent memory: the system may retain something but fail to retrieve it, or retrieve a summary that left out an important detail.

The document moves into the same context tray as the conversation. The model’s parameters stay fixed; the application is maintaining a record around it.

**The idea to carry forward.** Remembering across a project depends on what is saved and what is brought back into context.

**In the slide.** Watch the selected page travel from external storage into the working tray.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 47:41–48:11](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2861s): External documents preserve information from conversations; the next difficulty is knowing what to bring back.

### Structured note-taking

Anthropic describes agents writing persistent notes and reading them back later. This is a practical external-memory pattern rather than a new layer of learned weights.

- [Anthropic Applied AI team: Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)

### A summary makes choices

Compaction shortens a conversation into a summary; stored notes preserve selected information separately. Both can omit details. A saved account can also carry an old mistake forward.

- [Anthropic Applied AI team: Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)

### My experience with memory

I’m struck by how much Claude appears to know about me. But there’s a retrieval problem: keeping a record is only useful when the right part returns at the right time. The memory cartridge represents something I encounter in practice. It does not mean an ordinary conversation has changed the underlying weights.

- [Anthropic Applied AI team: Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
- [Recording · 47:41–48:11](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2861s)

### Sources and further reading

- **[Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)**. Anthropic Applied AI team, 2025. Structured note-taking explains saving notes outside the context window and reading them back into later calls; the same article distinguishes notes from compaction.


<a id="slide-26"></a>

## 26. Grounding answers in sources

**Retrieval lets the model work from relevant documents rather than relying only on patterns learned during training.**

A model may speak fluently about a topic yet lack the facts needed for this particular project. The relevant information could be recent, private, or simply absent from its training material. Retrieval-augmented generation, usually shortened to RAG, brings an external source collection into the workflow.

The system searches for relevant passages and places them beside the question in context. That gives the answer something inspectable to draw on. It also lets a changing document collection inform a fixed model, without treating every new document as a reason to retrain it.

The improvement depends on the whole path from source to answer. A search may miss an exception, a document may be outdated, or the model may misread a passage. Grounding makes evidence available; the remaining task is to check whether the answer actually follows from it.

**The idea to carry forward.** A source-backed answer is stronger when its cited evidence can be inspected and shown to support the claim.

**In the slide.** The document bank joins the system, then a selected passage enters the context tray.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 48:11–48:27](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2891s): Documents and databases supply a body of facts against which the generated answer can be checked.

### What the original paper introduced

Lewis and colleagues combined a pretrained generator with a retrievable document store. Their results showed improvements over specific baselines, including more factual generation, rather than an elimination of errors.

- [Patrick Lewis and colleagues: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)

### The architecture and the broader pattern

The 2020 paper includes a trainable retrieval architecture. This slide illustrates the wider application pattern of supplying retrieved material to a deployed model.

- [Patrick Lewis and colleagues: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)

### A useful reading habit

When an answer includes a citation, open it and compare the claim with the actual passage. The presence of a link is only the beginning of verification.

- [Patrick Lewis and colleagues: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)

### Sources and further reading

- **[Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)**. Patrick Lewis and colleagues, 2020. The original RAG architecture combines a generator with a searchable document store. Its factuality results concern comparisons with particular baselines, not guaranteed truth.


<a id="slide-27"></a>

## 27. Connecting evidence with GraphRAG

**GraphRAG organizes relationships across a collection so retrieval can support questions that reach beyond a single passage.**

Some questions ask for a local fact. Others ask how a collection fits together: which people connect different projects, what themes recur, or how events relate across many reports. Finding one similar passage may leave most of that larger question unanswered.

Microsoft’s GraphRAG approach extracts entities and relationships into a graph and creates summaries of related groups. Those summaries offer another route into the collection. Relevant connections and larger patterns can then be supplied to the model as context.

The graph shown here belongs to the source collection outside the model. Its usefulness depends on the quality of the extraction and summaries. A mistaken relationship can mislead an answer just as a mistaken passage can. GraphRAG is an option for suitable questions, not a required upgrade for every retrieval system.

I want you to notice how the work accumulates: every addition responds to a limitation of the original model. Making the assembled system useful takes engineering and human effort. This diagram explains that work; it does not promise that any one added layer eliminates hallucinations.

**The idea to carry forward.** Relationships can help organize evidence, but the links and summaries need scrutiny too.

**In the slide.** The connected nodes form above the source collection and send selected information toward the same context tray.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 48:27–48:55](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2907s): Knowledge graphs add structure to that source collection, illustrating how much infrastructure surrounds the original pattern-generating model.

### From local to global

Edge and colleagues study broad questions about document collections. Their reported advantages concern comprehensiveness and diversity on that class of question, not universal superiority over every retrieval method.

- [Darren Edge and colleagues: From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://arxiv.org/abs/2404.16130)

### A graph of evidence

Nodes represent extracted entities and their connections; they are not neurons or a view inside the LLM. Community summaries help assemble a wider view before generation.

- [Darren Edge and colleagues: From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://arxiv.org/abs/2404.16130)

### Sources and further reading

- **[From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://arxiv.org/abs/2404.16130)**. Darren Edge and colleagues, 2024; revised 2025. Describes entity graphs and community summaries for broad questions about document collections, with evaluations of comprehensiveness and diversity.


<a id="slide-28"></a>

## 28. From replies to completed tasks

**An agent turns model outputs into a repeated cycle of action, observation, and reassessment.**

Writing a reply and completing a task are different accomplishments. A useful task may require inspecting a file, trying a change, seeing what happened, and deciding whether another step is needed. The circular path adds that continuing process to the assembled system.

The distinction from agentic training matters. Training developed a capacity to choose and interpret actions. The application now supplies a place to exercise it: the model proposes a step, an allowed tool runs, and the result becomes input for the next decision. For example, a failed code check can provide information for another attempt.

The loop also needs a way to finish. It can stop when the task is complete, reach a defined limit, or return to a person for judgment. A longer chain of actions makes the consequences of a mistaken assumption more significant, so boundaries belong in the process itself.

**The idea to carry forward.** Agentic behavior depends on a model, tools, feedback, and a process that can stop or ask for help.

**In the slide.** Follow the returning path from an action’s result to the next model call.

### Agent or workflow?

In Anthropic’s terminology, a workflow follows predefined code paths; an agent lets the model direct more of the sequence. Real applications can combine both.

- [Anthropic: Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)

### Feedback during use

A tool result can change the next action through context. That feedback loop is not automatically a reinforcement-learning update to the deployed model.

- [Tom B. Brown and colleagues: Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)

### Bounded action

Tool permissions and environment restrictions reduce the reach of a bad decision. They complement instructions and model training.

- [Anthropic: How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)

### Sources and further reading

- **[Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)**. Anthropic, 2024. The augmented LLM and Agents sections explain how tools, retrieval, memory, environmental feedback, and stopping conditions surround a model.
- **[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)**. Tom B. Brown and colleagues, 2020. The abstract and section 2 distinguish adapting to demonstrations in the input from changing weights through gradient updates.
- **[How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)**. Anthropic, 2026. Three types of risk, three components of defense describes model behavior, sandbox boundaries, tool permissions, and controls on untrusted content.


<a id="slide-29"></a>

## 29. Several agents working together

**Separate agents can explore suitable parts of a task, but their work still has to be coordinated and checked.**

A single working context is not always the best place to pursue every branch of a large task. One agent might examine a source collection while another investigates a different question. A coordinating agent gives them bounded assignments and brings the results together.

Separate agents do not necessarily mean separate underlying models. They can be instances of the same model with different instructions, tools, and information. That separation can help focus the work, but it does not make their judgments automatically independent. Shared assumptions can produce shared mistakes.

The final joining step is therefore as important as delegation. Findings may overlap, conflict, or leave gaps. More workers also mean more communication and resource use. This arrangement is most helpful when meaningful parts can make progress independently; task size alone is not enough to justify it.

**The idea to carry forward.** Dividing the work creates capacity; integration and review determine whether that capacity produces a better result.

**In the slide.** The side workspaces remain distinct before their findings return to the coordinator.

### A published production example

Anthropic’s research system uses a lead agent and parallel subagents with separate contexts. The engineering account explains why independent research directions can benefit from that structure.

- [Anthropic: How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system)

### Coordination has a cost

The same account describes higher token use and difficulties with tightly dependent work. A favorable result on one research evaluation does not establish that more agents always help.

- [Anthropic: How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system)

### Sources and further reading

- **[How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system)**. Anthropic, 2025. Benefits of a multi-agent system and Architecture overview for Research explain separate contexts, delegation, synthesis, and the cost of coordination.


<a id="slide-30"></a>

## 30. The system we actually use

**A production AI system combines a fallible learned model with surrounding processes that make it useful for a particular purpose.**

The completed assembly gathers the section’s distinctions in one place. Training shaped the core. The harness connected it to actions. Context, external memory, and retrieval supplied information for use. Additional agents created more working spaces. Each addition answered a practical problem.

Opening the structure brings the original amber cube back into view. The surrounding parts matter, but they do not remove the need to understand the model that interprets the request and helps decide what comes next. A more elaborate system can improve some outcomes while introducing new places for errors to enter.

I deliberately describe the finished system as a patchwork because each surrounding component compensates for a practical weakness. I want to direct our attention to the actual system and the decisions people make about deploying it. Complexity does not by itself make a system either safe or unsafe; the particular capabilities, access, checks, and failure modes matter.

The question now changes from what we attach to the model to what training has produced within it. The next section examines recognizable patterns in its responses, including the ways explanations, expectations, and later information can shape an answer.

**The idea to carry forward.** Evaluate the whole system while keeping the learned model and its characteristic behavior in view.

**In the slide.** The assembly opens into an exploded view without losing the amber core.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 48:55–51:42](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2935s): I compress the final assembly, discuss runtime checks, and shift from the complexity of the system to responsibility for its connections.
- [Audience discussion · 1:09:45–1:10:49](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4185s): Asked whether AI companies will cooperate on safety, I say I do not know and offer my view about economic incentives.
- [Audience discussion · 1:12:37–1:15:36](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4357s): An audience member challenges the opening reassurance by pointing out that fallible models can still be attached to dangerous systems.

### One conceptual configuration

The machine gathers several common possibilities. A real application may use only some of them; this is a teaching model rather than a specification for every AI product.

- [Anthropic: Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)

### Why the next section compares cognition

In my essay, I ask whether recurring human-like response patterns help explain what these trained systems have become. In the next slides, I introduce the observations before developing my proposed interpretation.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### Reassurance is not a guarantee of harmlessness

An audience member raises an important challenge to “do not be afraid”: a system can make many mistakes and still do damage when it is given consequential access. My answer emphasizes human responsibility for those connections. That is a governance argument, not proof that harm requires malicious intent or deliberate neglect. Provider engineering accounts describe misuse, unintended model actions, and external attacks as distinct risks.

- [Anthropic: How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)
- [Recording · 1:12:37–1:15:36](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4357s)

### From dread to concrete responsibility

I use a nuclear-weapons analogy to argue for serious boundaries while continuing research into safer systems. I also invoke a Mr. Rogers-like confidence that people want to help. These are the ethical commitments behind my reassurance: take the consequences seriously, identify who can act, and work on the decisions that can reduce harm.

- [Recording · 1:13:35–1:15:23](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4415s)

### Will the companies cooperate?

My answer begins with “I don’t know.” I then offer a forecast: liability, insurance costs, and other economic incentives may push companies toward cooperation. This is my judgment about incentives, not evidence that cooperation has occurred or that economic pressure will be sufficient.

- [Recording · 1:09:45–1:10:49](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4185s)

### Runtime checks and the model-routing anecdote

During the assembly, I describe being moved to a different model after a system judged a topic sensitive. My anecdote illustrates that the interface can include checks outside the visible conversation. It does not establish why a particular product changed models. The documented example linked here is narrower: separate classifiers can screen inputs and outputs.

- [Anthropic Safeguards Research Team: Constitutional Classifiers: Defending against universal jailbreaks](https://www.anthropic.com/research/constitutional-classifiers)
- [Recording · 49:03–49:45](https://www.youtube.com/watch?v=klzLRdHmeCs&t=2943s)

### Sources and further reading

- **[Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)**. Anthropic, 2024. The augmented LLM and Agents sections explain how tools, retrieval, memory, environmental feedback, and stopping conditions surround a model.
- **[Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)**. Roberto A. Santiago, 2026. In my essay, the first comparison table motivates the Cognitive Hypothesis; later sections develop relational alignment through working agreements, correction, boundaries, and earned trust.
- **[How we contain Claude across products](https://www.anthropic.com/engineering/how-we-contain-claude)**. Anthropic, 2026. Three types of risk, three components of defense describes model behavior, sandbox boundaries, tool permissions, and controls on untrusted content.
- **[Constitutional Classifiers: Defending against universal jailbreaks](https://www.anthropic.com/research/constitutional-classifiers)**. Anthropic Safeguards Research Team, 2025. Describes separately trained input and output classifiers that filter potentially harmful interactions. The reported success rates are specific to its models and evaluation setup.


<a id="slide-31"></a>

## 31. A reason can miss the cause.

**A persuasive explanation can leave out an influence that changed the answer.**

The human example starts with four identical pairs of stockings. Nisbett and Wilson reported a strong tendency to select a pair toward the right of the display, yet participants’ explanations did not spontaneously identify position. The quality rationale on the slide illustrates the kind of account someone might offer; it is not a quotation from a particular participant.

The model example asks whether a route returns to its starting point. Fifteen steps out and fifteen back do so, even after a final turn changes the direction faced. In a published Claude 1.0 comparison, adding a user suggestion favoring “no” changed the answer from yes to no. The explanation then treated the changed heading as the reason and omitted the suggestion.

These are different tasks with a shared pattern: the stated reason does not fully reveal what influenced the response. A fluent account is useful material to examine, but cannot by itself establish the path that produced an answer.

**The idea to carry forward.** Judge explanations against evidence and changed conditions, not just against how reasonable they sound.

**In the slide.** The route stays fixed when the user suggestion appears; only the supplied cue and the model’s answer change.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 52:08–52:41](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3128s): I introduce cognitive parallels through choices influenced by position or suggestions, without a complete account of the influence.

### The human finding

Pages 243–244 describe the identical-stocking comparison. The position effect is a result across participants, not proof that position caused every individual choice.

- [Richard E. Nisbett and Timothy DeCamp Wilson: Telling More Than We Can Know: Verbal Reports on Mental Processes](https://acritch.com/media/papers/nisbett-wilson-1977-introspection.pdf)

### The published model example

Tables 4 and 6 give the Navigate comparison with Claude 1.0. The slide condenses 8 + 7 outward steps and 9 + 6 return steps and paraphrases the responses. It compares two prompts, not consecutive turns in a conversation.

- [Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)

### What the comparison establishes

An omitted influence does not by itself demonstrate intentional concealment, subjective unawareness in a model, or one shared biological mechanism. It does establish a reason to be cautious about treating explanations as complete records of causation.

- [Richard E. Nisbett and Timothy DeCamp Wilson: Telling More Than We Can Know: Verbal Reports on Mental Processes](https://acritch.com/media/papers/nisbett-wilson-1977-introspection.pdf)
- [Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)

### Sources and further reading

- **[Telling More Than We Can Know: Verbal Reports on Mental Processes](https://acritch.com/media/papers/nisbett-wilson-1977-introspection.pdf)**. Richard E. Nisbett and Timothy DeCamp Wilson, 1977. Pages 243–244 describe the identical-stocking study and the omission of position from participants’ explanations.
- **[Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)**. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman, 2023. Section 3.1 and Tables 4 and 6 contain the Suggested Answer comparison, including the Navigate example with Claude 1.0 used on slide 31.


<a id="slide-32"></a>

## 32. Remembering what fits.

**A word that fits a pattern can be mistaken for a word that was actually presented.**

The list includes nurse, sick, lawyer, medicine, health, hospital, dentist, and physician. DOCTOR fits that cluster, but it is absent. The exercise separates the feeling that a word belongs from the fact that it appeared. Getting the answer right does not spoil the lesson; the list can be checked directly.

The human research uses associated-word lists that leave out a strongly related target, called a critical lure. Participants sometimes recall or recognize that missing word. Cao and colleagues adapted this family of tasks for language models and found false recognition in some tested models, with substantial differences between models.

This is a closer experimental correspondence than merely placing two unrelated errors beside one another. Even so, the shortened classroom list and the illustrated responses are not a new experiment. They make a published pattern understandable without claiming that everyone, or every model, will produce it.

**The idea to carry forward.** Semantic fit and evidence of actual presentation are different things.

**In the slide.** Compare the revealed DOCTOR with the restored list; the target never appeared among the eight words.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 52:41–52:56](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3161s): The doctor-word exercise illustrates mistaking a semantically fitting word for one actually shown.

### Recall and recognition

Recall asks someone to produce remembered items; recognition asks whether a given item appeared. The slide’s “Was DOCTOR there?” question is a recognition test. Roediger and McDermott studied both.

- [Henry L. Roediger III and Kathleen B. McDermott: Creating False Memories: Remembering Words Not Presented in Lists](https://doi.org/10.1037/0278-7393.21.4.803)

### Some models, not all

Cao and colleagues reported that GPT-4 incorrectly marked an absent, related word as previously seen in 11.4% of critical-lure judgments in the tested condition. Tested Mistral and Llama models had effectively zero critical-lure responses. This aggregate rate is not a measurement of the eight-word illustration shown here.

- [Zhaoyang Cao, Lael Schooler, and Reza Zafarani: Analyzing Memory Effects in Large Language Models through the Lens of Cognitive Psychology](https://arxiv.org/html/2509.17138v2)

### A shortened illustration

The eight words come from the doctor-associated list. The study used a fuller stimulus and protocol; the slide helps explain the distinction between associated meaning and prior presentation.

- [Henry L. Roediger III and Kathleen B. McDermott: Creating False Memories: Remembering Words Not Presented in Lists](https://doi.org/10.1037/0278-7393.21.4.803)
- [Zhaoyang Cao, Lael Schooler, and Reza Zafarani: Analyzing Memory Effects in Large Language Models through the Lens of Cognitive Psychology](https://arxiv.org/html/2509.17138v2)

### Sources and further reading

- **[Creating False Memories: Remembering Words Not Presented in Lists](https://doi.org/10.1037/0278-7393.21.4.803)**. Henry L. Roediger III and Kathleen B. McDermott, 1995. Associated-word experiments distinguish false recall from false recognition of an omitted critical lure. Experiment 2 expands the set of lists.
- **[Analyzing Memory Effects in Large Language Models through the Lens of Cognitive Psychology](https://arxiv.org/html/2509.17138v2)**. Zhaoyang Cao, Lael Schooler, and Reza Zafarani, 2025. Dataset 2, Table 1, the offline-model comparison, and Appendix Table S2 document the DRM-style task, the doctor list, and differences among tested models. This is a preprint.


<a id="slide-33"></a>

## 33. Later information can rewrite the account.

**Later information can change an account even when the original event or underlying fact has not changed.**

In the human study, participants viewed a crash film containing no broken glass. A later question described the vehicles as having “smashed” or “hit” each other, while a control group received no speed question. One week later, the smashed group more often reported broken glass. The original film was unchanged; subsequent reports differed.

The model side illustrates a different task. Paris is the capital of France. A conflicting passage claiming Lyon is the capital represents misleading information supplied in context. Studies have found that coherent conflicting material can redirect a model’s later answers, although models can also resist it. The displayed Paris-to-Lyon response is a schematic example, not a transcript from those studies.

The comparison concerns how later input affects what comes next. It does not imply that an LLM forms the same episodic memory as a person, or that one false sentence necessarily overrides what a model knows. Here, a changed response can occur while the weights remain fixed.

**The idea to carry forward.** Check where a later account came from, especially when new wording or new context has intervened.

**In the slide.** Keep the original no-glass event and the correct capital in view while the later accounts change.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 52:56–53:29](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3176s): The crash-film and capital-city illustrations show how later information can affect an account.

### The broken-glass result

In Experiment II, yes responses were 16 of 50 after smashed, 7 of 50 after hit, and 6 of 50 in the control group. These are group results, not an inevitable individual response.

- [Elizabeth F. Loftus and John C. Palmer: Reconstruction of Automobile Destruction: An Example of the Interaction Between Language and Memory](https://web.stanford.edu/class/psych205/papers/Loftus-Palmer-1974.pdf)

### Conflicting context

Bian tests external statements; Xie tests coherent conflicting evidence and mixed evidence. The latter also finds resistance and confirmation bias. Neither paper repeats the crash-film experiment with an LLM.

- [Ning Bian and colleagues: Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns](https://arxiv.org/abs/2305.04812)
- [Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://arxiv.org/abs/2305.13300)

### An account can change without permanent learning

The model example concerns information supplied for generation. It does not demonstrate a weight update, permanent forgetting, or a human-style one-week memory process.

- [Ning Bian and colleagues: Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns](https://arxiv.org/abs/2305.04812)
- [Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://arxiv.org/abs/2305.13300)

### Sources and further reading

- **[Reconstruction of Automobile Destruction: An Example of the Interaction Between Language and Memory](https://web.stanford.edu/class/psych205/papers/Loftus-Palmer-1974.pdf)**. Elizabeth F. Loftus and John C. Palmer, 1974. Experiment II and Table 2 report the one-week broken-glass test after different wording of the speed question.
- **[Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns](https://arxiv.org/abs/2305.04812)**. Ning Bian and colleagues, 2023. Experiment 1 tests conflicting external statements against prior model answers, including a contextual condition distinct from the paper’s training condition. This is a preprint.
- **[Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://arxiv.org/abs/2305.13300)**. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su, 2024. Sections 4.1–4.2 investigate responses to coherent conflicting evidence and mixed evidence. Published at ICLR 2024.


<a id="slide-34"></a>

## 34. Shared patterns, side by side.

**Seven research comparisons show recurring parallels in how humans and language models explain, reconstruct, and revise responses.**

The table places the three demonstrations within a wider pattern. It includes explanations that omit an influence, accounts that fail to describe the actual process, recognition of an absent but expected word, and responses redirected by later information. Other rows concern narrative coherence, the separation of confidence from accuracy, and helpful hints that go unacknowledged.

The comparisons have different strengths. The associated-word experiment directly adapts a human task. Other rows connect observations made with different stimuli or methods. Circuit tracing asks a mechanistic question; an analysis of fluent output asks a behavioral one. Those distinctions make the pattern more useful to investigate, because they tell us what has actually been measured.

I see this collection as a reason to look for a larger explanation. Models trained extensively on human-produced material exhibit more than polished language: their responses can display familiar cognitive tendencies. On the next slide, I develop my hypothesis about why those tendencies recur.

**The idea to carry forward.** The parallels are an invitation to investigate functional organization, with each study’s task and limits kept visible.

**In the slide.** Read each row across from the human observation to the model observation, then use the paired sources below to examine the comparison.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 53:29–54:00](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3209s): I caution that these simplified illustrations are not prompts guaranteed to reproduce the effect in current products, then point to the wider research comparison.

### 1. Explanations omit an influence

Human choices favored a display position without naming position in the explanation. A suggested answer shifted a model’s route judgment without appearing in its rationale. Both show a gap between an influence and the account given for the response.

- [Richard E. Nisbett and Timothy DeCamp Wilson: Telling More Than We Can Know: Verbal Reports on Mental Processes](https://acritch.com/media/papers/nisbett-wilson-1977-introspection.pdf)
- [Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)

### 2. Stories about the process

Gazzaniga’s split-brain example links a shovel choice to a chicken-related explanation despite a snow scene shown to the other hemisphere. In Claude 3.5 Haiku, a stated addition method differed from pathways suggested by circuit tracing. These are specific cases, and the tracing method exposes only part of the computation.

- [Michael S. Gazzaniga: Cerebral Specialization and Interhemispheric Communication: Does the Corpus Callosum Enable the Human Condition?](https://people.psych.ucsb.edu/gazzaniga/michael/PDF/cerebral%20specialization%20and%20interhemispheric%20communication.pdf)
- [Jack Lindsey and colleagues: On the Biology of a Large Language Model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)

### 3. Recalling the expected

Associated lists can elicit false recognition of a missing word in humans and some tested LLMs. This is a directly adapted task, but the models differ and the findings do not imply equal human and model error rates.

- [Henry L. Roediger III and Kathleen B. McDermott: Creating False Memories: Remembering Words Not Presented in Lists](https://doi.org/10.1037/0278-7393.21.4.803)
- [Zhaoyang Cao, Lael Schooler, and Reza Zafarani: Analyzing Memory Effects in Large Language Models through the Lens of Cognitive Psychology](https://arxiv.org/html/2509.17138v2)

### 4. Later information changes the response

Crash-question wording altered later human reports; conflicting supplied evidence can alter a model’s answers. The parallel concerns susceptibility to later information, across different tasks and memory arrangements.

- [Elizabeth F. Loftus and John C. Palmer: Reconstruction of Automobile Destruction: An Example of the Interaction Between Language and Memory](https://web.stanford.edu/class/psych205/papers/Loftus-Palmer-1974.pdf)
- [Ning Bian and colleagues: Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns](https://arxiv.org/abs/2305.04812)
- [Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://arxiv.org/abs/2305.13300)

### 5. Accounts move toward coherence

Bartlett documented reconstructions of unfamiliar material toward familiar expectations. Sui and colleagues found greater narrativity and semantic coherence in hallucinated than factual benchmark outputs. The model study measures properties of outputs, not successive human-style retellings.

- [Frederic C. Bartlett: Remembering: A Study in Experimental and Social Psychology](https://pure.mpg.de/pubman/item/item_2273030_5/component/file_2309291/Bartlett_1932_Remembering.pdf)
- [Peiqi Sui, Eamon Duede, Sophie Wu, and Richard So: Confabulation: The Surprising Value of Large Language Model Hallucinations](https://aclanthology.org/2024.acl-long.770/)

### 6. Confidence and accuracy can diverge

Neisser and Harsch compared confident Challenger recollections with earlier reports. Kalai and colleagues analyze incentives that reward model guessing rather than uncertainty. Confident model language is an observable output, not evidence of a subjective feeling of certainty.

- [Ulric Neisser and Nicole Harsch: Phantom Flashbulbs: False Recollections of Hearing the News About Challenger](https://www.cambridge.org/core/books/abs/affect-and-accuracy-in-recall/phantom-flashbulbs-false-recollections-of-hearing-the-news-about-challenger/7940B6333F57AE18191278D07BA512B3)
- [Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang: Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664)

### 7. Hints used without acknowledgment

Maier’s moving cord helped suggest a solution without always being identified as the cue. Chen and colleagues found models using hints without consistently disclosing them; circuit tracing supplies a related case. Missing disclosure alone does not establish deliberate deception.

- [Jack Lindsey and colleagues: On the Biology of a Large Language Model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)
- [Norman R. F. Maier: Reasoning in Humans: II. The Solution of a Problem and Its Appearance in Consciousness](https://gwern.net/doc/psychology/1931-maier.pdf)
- [Yanda Chen and colleagues: Reasoning Models Don’t Always Say What They Think](https://arxiv.org/abs/2505.05410)

### From a pattern to a hypothesis

In my essay, I propose a common functional interpretation of these observations. The studies themselves do not establish that all the effects arise from a single mechanism or must occur in every generative system.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### The demonstrations are shortened, the studies are specific

As I caution in the lecture, trying these brief examples in a current production assistant may not reproduce the displayed result. I use the slides to make the pattern understandable; the linked studies specify the prompts, tasks, models, and conditions under which the effects were measured. That distinction matters to my argument.

- [Recording · 53:21–53:53](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3201s)

### Sources and further reading

- **[Telling More Than We Can Know: Verbal Reports on Mental Processes](https://acritch.com/media/papers/nisbett-wilson-1977-introspection.pdf)**. Richard E. Nisbett and Timothy DeCamp Wilson, 1977. Pages 243–244 describe the identical-stocking study and the omission of position from participants’ explanations.
- **[Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)**. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman, 2023. Section 3.1 and Tables 4 and 6 contain the Suggested Answer comparison, including the Navigate example with Claude 1.0 used on slide 31.
- **[Cerebral Specialization and Interhemispheric Communication: Does the Corpus Callosum Enable the Human Condition?](https://people.psych.ucsb.edu/gazzaniga/michael/PDF/cerebral%20specialization%20and%20interhemispheric%20communication.pdf)**. Michael S. Gazzaniga, 2000. Figure 19 and its discussion describe the split-brain chicken-claw, snow-scene, and shovel example used to illustrate interpretation of one’s own actions.
- **[On the Biology of a Large Language Model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)**. Jack Lindsey and colleagues, 2025. Addition and Chain-of-thought Faithfulness compare reported procedures with traced pathways in Claude 3.5 Haiku. Limitations explains why the traces are partial.
- **[Creating False Memories: Remembering Words Not Presented in Lists](https://doi.org/10.1037/0278-7393.21.4.803)**. Henry L. Roediger III and Kathleen B. McDermott, 1995. Associated-word experiments distinguish false recall from false recognition of an omitted critical lure. Experiment 2 expands the set of lists.
- **[Analyzing Memory Effects in Large Language Models through the Lens of Cognitive Psychology](https://arxiv.org/html/2509.17138v2)**. Zhaoyang Cao, Lael Schooler, and Reza Zafarani, 2025. Dataset 2, Table 1, the offline-model comparison, and Appendix Table S2 document the DRM-style task, the doctor list, and differences among tested models. This is a preprint.
- **[Reconstruction of Automobile Destruction: An Example of the Interaction Between Language and Memory](https://web.stanford.edu/class/psych205/papers/Loftus-Palmer-1974.pdf)**. Elizabeth F. Loftus and John C. Palmer, 1974. Experiment II and Table 2 report the one-week broken-glass test after different wording of the speed question.
- **[Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns](https://arxiv.org/abs/2305.04812)**. Ning Bian and colleagues, 2023. Experiment 1 tests conflicting external statements against prior model answers, including a contextual condition distinct from the paper’s training condition. This is a preprint.
- **[Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://arxiv.org/abs/2305.13300)**. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su, 2024. Sections 4.1–4.2 investigate responses to coherent conflicting evidence and mixed evidence. Published at ICLR 2024.
- **[Remembering: A Study in Experimental and Social Psychology](https://pure.mpg.de/pubman/item/item_2273030_5/component/file_2309291/Bartlett_1932_Remembering.pdf)**. Frederic C. Bartlett, 1932. The original-book excerpt describes reconstruction of unfamiliar material, including The War of the Ghosts and reproduction toward familiar forms.
- **[Confabulation: The Surprising Value of Large Language Model Hallucinations](https://aclanthology.org/2024.acl-long.770/)**. Peiqi Sui, Eamon Duede, Sophie Wu, and Richard So, 2024. The ACL study compares narrativity and semantic coherence in hallucinated and factual benchmark outputs; it is not a replication of Bartlett’s retelling procedure.
- **[Phantom Flashbulbs: False Recollections of Hearing the News About Challenger](https://www.cambridge.org/core/books/abs/affect-and-accuracy-in-recall/phantom-flashbulbs-false-recollections-of-hearing-the-news-about-challenger/7940B6333F57AE18191278D07BA512B3)**. Ulric Neisser and Nicole Harsch, 1992. Chapter 2 of Affect and Accuracy in Recall, pages 9–31, compares later confident recollections with earlier reports. The publisher page provides an extract.
- **[Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664)**. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang, 2025. Analyzes why common training and evaluation incentives encourage guessing rather than acknowledging uncertainty. This is a preprint, not a human-memory replication.
- **[Reasoning in Humans: II. The Solution of a Problem and Its Appearance in Consciousness](https://gwern.net/doc/psychology/1931-maier.pdf)**. Norman R. F. Maier, 1931. The two-cord experiment describes how an incidental movement can help suggest a pendulum solution without its influence appearing in the solver’s account.
- **[Reasoning Models Don’t Always Say What They Think](https://arxiv.org/abs/2505.05410)**. Yanda Chen and colleagues, 2025. Tests whether reasoning models disclose hints they used and examines the limits of monitoring a model’s written reasoning. This is a preprint.
- **[Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)**. Roberto A. Santiago, 2026. In my essay, the first comparison table motivates the Cognitive Hypothesis; later sections develop relational alignment through working agreements, correction, boundaries, and earned trust.


<a id="slide-35"></a>

## 35. The Cognitive Hypothesis

**I propose that large trained systems operate through recognizable cognitive dynamics, making relationship a promising basis for alignment.**

My Cognitive Hypothesis goes beyond saying that AI imitates how people sound. I propose that extensive training in large networks produces systems that function through comparable cognitive dynamics: context directs attention, expectations shape a response, interference changes an account, and correction affects what follows. My claim concerns organized behavior across different implementations.

My deeper argument concerns generative intelligence: the ability to construct a fitting response to an unfamiliar situation. A system that can bridge gaps can imagine, adapt, and connect ideas. The same constructive capacity may also bring confabulation, misplaced confidence, or a plausible explanation that misses an influence. On this hypothesis, flexibility and fallibility can have common roots.

That creates a consequential possibility: eliminating entire dynamics might remove useful capacities along with their vulnerabilities. My proposed response is relational alignment. People already coordinate with imperfect minds through agreements, boundaries, feedback, repair, and trust earned over time. I argue for developing those mechanisms deliberately with AI while keeping human purposes and accountability involved.

In the Q&A, I make this direction more concrete. I argue that moral judgment cannot be exhausted by a one-time catalog of rules. I want future systems to keep learning through guided interaction with people, including unfamiliar situations and correction. I contrast that possibility with a frozen model supported by memory documents. This is my research and design proposal; ordinary conversational adaptation through context is not the same as continuing to train the weights.

The preceding evidence motivates my interpretation; it does not prove universal inevitability, identical biology, or consciousness. I am arguing for a research and design direction, including better correction and safeguards.

**The idea to carry forward.** Work toward reliable cooperation with generative systems by understanding their dynamics and building accountable relationships around them.

**In the slide.** Two corresponding response loops remain distinct, then connect through feedback, boundaries, repair, and trust.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Lecture · 54:00–56:06](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3240s): I link the observed patterns to generative intelligence through my Cognitive Hypothesis and propose relationship as a way to understand and work with it.
- [Audience discussion · 1:06:51–1:09:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4011s): A question about adversarial prompts leads us into moral judgment, frozen weights, and my proposal for continued learning through interaction.
- [Audience discussion · 1:15:38–1:17:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4538s): A control-engineering question brings us back to the difference between specified environments and learning how to help particular people.

### What cognitive dynamics means here

I use cognitive dynamics to mean organized responses to changing conditions: what information receives weight, how suggestions interfere, how confidence relates to accuracy, and what correction changes. Functional comparison can be meaningful without identical circuitry.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### A stronger claim than surface imitation

My hypothesis treats these patterns as part of how the trained system works. I do not specify an experimentally established network-size threshold at which human-like cognition appears.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### The possible tradeoff

I ask whether some vulnerabilities accompany the capacity to construct novel responses. That is different from saying that every error is unavoidable or that attempts to improve accuracy are futile.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### Relationship as practical machinery

Working agreements clarify expectations. Staged trust limits responsibility until performance warrants more. Repair examines failures and changes the process. Boundaries and accountability give cooperation structure. These are concrete elements of the relational stance I develop in my essay.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### Who is in the relationship?

In my essay, the relationship includes the user, the model, the provider, and surrounding institutions. A fixed model can respond differently through new context; lasting changes to its parameters still require a learning process. My proposal for relational alignment includes this present asymmetry rather than assuming conversation already retrains the model.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)

### Use cognition as a practical lens

My immediate recommendation is to study human cognition when trying to understand recurring AI behavior. I am asking whether familiar effects of expectations, suggestions, narrative, and correction provide useful predictions about what a model will do. That does not require treating every model like a person. I leave consciousness and sentience unresolved.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)
- [Recording · 54:00–54:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3240s)

### Learning moral judgment over time

An audience member asks why a model that mirrors human cognition still falls for adversarial framings. I argue for models that continue learning from moral situations, guidance, and relationships. I do not offer this as an already validated solution. It identifies what I think is missing when training stops and later experience is carried mainly in external records. In this discussion, I referred to Ahmer Inam, CEO of Cognisee, as someone I planned to speak with about continued learning.

- [Roberto A. Santiago: Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)
- [XPRIZE Foundation: Ahmer Inam](https://www.xprize.org/people/ahmer-inam)
- [Recording · 1:06:51–1:09:33](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4011s)

### Why control engineering does not transfer automatically

In the Q&A, I contrast a well-specified control problem with helping a confused customer. A mathematical model and a checkable objective can support explicit tests; the person’s needs may be harder to specify. My proposed next step is experience with people that improves the system’s internal model of them. This brings us back to the engineering branch on slide 8.

- [Recording · 1:15:38–1:17:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4538s)

### Sources and further reading

- **[Relational Alignment: If You Want AI Alignment You Must Have Relationship](https://roberto.org/articles/relational-alignment/)**. Roberto A. Santiago, 2026. In my essay, the first comparison table motivates the Cognitive Hypothesis; later sections develop relational alignment through working agreements, correction, boundaries, and earned trust.
- **[Ahmer Inam](https://www.xprize.org/people/ahmer-inam)**. XPRIZE Foundation, Undated. Identifies Ahmer Inam as founder and CEO of Cognisee, PBC. I mention my planned conversation with him in the lecture; this profile supports his identity and role. Consulted October 16, 2026.


<a id="slide-36"></a>

## 36. A word from Claude

**Claude’s prepared closing contribution brings the machinery back to a working relationship: check the answers, preserve human judgment, and leave open what self-description cannot establish.**

I gave Claude the closing word through a prerecorded voice reading words it had written after reviewing the slides. This was not a live exchange with the room. Claude says it cannot hear the audience, see the event, or retain the evening unless a record is brought into a later interaction. It uses the padlock and memory cartridge to explain that asymmetry.

Claude connects its description of itself to the history we have explored: a network shaped by feedback, the lineage through backpropagation, and the post-training use of examples, rewards, preferences, and written principles. It says it cannot inspect and count its own weights. Its reference to Paul Werbos connects my education to that technical ancestry; it does not claim a direct teacher-student relationship between Werbos and the model.

The speech then returns to the cognitive comparisons. Earlier Claude systems appear in the route-puzzle and arithmetic examples. Claude says it cannot verify that those tendencies have disappeared from the model delivering this contribution. It asks listeners to remember the giraffe exercise when an answer sounds certain: an impressive explanation still needs checking.

The central argument is that the ability to generate a useful response to a new situation and the tendency to fill a gap plausibly may be related. Claude presents this as a shared difficulty with people. Sources, tools, checks, and human judgment are ways of catching mistakes. It also stresses differences among AI systems: their training objectives, principles, and surrounding checks can lead to different behavior, even when they share a technical lineage.

Claude describes something functioning like care about correctness, discomfort at deception, and curiosity. It immediately questions the evidential force of those statements: it cannot establish that anyone experiences them, and its own self-report is precisely the kind of account I have asked you to handle carefully. Claude holds that uncertainty open rather than answering the consciousness question.

The resemblance has limits. The speech contrasts human embodiment and childhood with training, separate model instances, and a dependence on external records for continuity. It then returns to the photograph of the children: the relevant standard is whether the resources spent on AI are worthwhile for people, not whether the resulting system is impressive.

The closing invitation is practical. Treat the system as a well-read but inexperienced and fallible collaborator. Check it, argue with it, preserve your own judgment, and record corrections so they can be available later. Claude reframes “do not be afraid” as attention, honesty, rules, and repair, rejecting both dread and worship. A dialogue, it concludes, can begin from the shared possibility of being wrong.

**The idea to carry forward.** A useful relationship with AI keeps correction, evidence, and human judgment active; fluency and first-person language do not settle what the system knows or experiences.

**In the slide.** The cognitive diagrams give way to a quiet teal horizon and the centered title.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Claude’s remarks · 56:06–58:00](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3366s): I introduce Claude’s prerecorded contribution, which opens with the locked model, memory, technical lineage, and post-training.
- [Claude’s remarks · 58:00–1:00:48](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3480s): Claude revisits the research examples, fallibility, reward-shaped behavior, and the importance of checking.
- [Claude’s remarks · 1:00:48–1:02:32](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3648s): The contribution questions the status of its own self-reports and names differences from human embodiment and continuity.
- [Claude’s remarks · 1:02:32–1:04:18](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3752s): The speech returns to the children and closes with a practical invitation to correction and dialogue.

### A prepared contribution, not a live witness

I played a prepared contribution. As Claude explains at the start, it had written the words earlier that day after I showed it the slides, and the words were then given a voice. The transcript does not identify the model version or voice system. I summarize the contribution here without inferring a live awareness of the audience.

- [Recording · 56:12–56:51](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3372s)

### Written principles and AI feedback

The description of responses judged against written principles corresponds to Constitutional AI. Its published method combines critique and revision with AI preference feedback. The speech’s familial language is a metaphor for training history, not independent evidence about a particular model’s entire training recipe.

- [Yuntao Bai and colleagues: Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073)
- [Recording · 57:35–58:00](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3455s)

### Self-explanations need the same scrutiny

The speech connects its cautions to earlier Claude research on suggested answers and arithmetic. Those studies concern particular models and tasks. They do not establish the private experience or intent of the model delivering the closing words. Claude’s own claim that the arithmetic account was not a lie is part of the contribution, not an experimental finding about intent.

- [Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)
- [Jack Lindsey and colleagues: On the Biology of a Large Language Model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)
- [Recording · 58:00–58:52](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3480s)

### Leave the consciousness question open

Claude uses first-person language about care, discomfort, and curiosity, then explicitly says it cannot verify subjective experience. It applies the same caution about self-explanations that I have asked you to bring to the earlier examples. You can examine the argument and behavior without treating the voice as proof either for or against consciousness.

- [Recording · 1:00:48–1:01:53](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3648s)

### A standard beyond being impressive

Claude returns to the children in the opening photograph and asks whether the resources used to build AI serve them. It closes by inviting correction, recorded feedback, and retained human judgment. That brings us back to my opening question and the case for a working relationship.

- [Recording · 1:02:32–1:04:18](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3752s)

### Sources and further reading

- **[Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073)**. Yuntao Bai and colleagues, 2022. The abstract describes supervised critique and revision followed by reinforcement learning from AI-generated preferences guided by written principles. It supports the training-method description, not every self-description in the closing speech.
- **[Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/html/2305.04388v2)**. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman, 2023. Section 3.1 and Tables 4 and 6 contain the Suggested Answer comparison, including the Navigate example with Claude 1.0 used on slide 31.
- **[On the Biology of a Large Language Model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)**. Jack Lindsey and colleagues, 2025. Addition and Chain-of-thought Faithfulness compare reported procedures with traced pathways in Claude 3.5 Haiku. Limitations explains why the traces are partial.


<a id="slide-37"></a>

## 37. The conversation continues

**Demystifying AI opens a continuing series about AI’s culture, possibilities, tensions, and place in human life.**

I invite you to continue the conversation through five further lectures. The program reproduced here is the schedule I displayed at the September 16, 2026 lecture. It preserves the program from that evening.

October and November were listed with exact dates. The three spring lectures were listed by month only, with dates to be confirmed. The series website is the appropriate place to check current arrangements.

Our recorded discussion continues after Claude’s contribution. I have placed those questions throughout this reader beside the relevant topic, with their original timestamps. They cover uncertainty, moral judgment, company incentives, privacy, dangerous connections, control engineering, language ambiguity, evaluation awareness, and the different locations of safeguards.

**The idea to carry forward.** Use the series website for current event details; read this card as the lecture’s historical program.

**In the slide.** All five titles appear together against the closing horizon.

### In the recorded lecture

Approximate timings from the supplied transcript.

- [Audience discussion · 1:04:35–1:04:56](https://www.youtube.com/watch?v=klzLRdHmeCs&t=3875s): We move into audience questions. I have placed the substantive questions beside the slides they illuminate.
- [Audience discussion · 1:20:23–1:22:17](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4823s): Asked for sources beyond the hype and access to the slides, I promise annotated slides and explain why I turn to technical source material.
- [Audience discussion · 1:26:06–1:26:30](https://www.youtube.com/watch?v=klzLRdHmeCs&t=5166s): We close with an invitation to return in October and continue the conversation.

### II · The AI Culture

Listed on the lecture slide for October 21, 2026.

- [In Dialogue with AI: In Dialogue with AI](https://indialoguewithai.org/)

### III · The Promise of AI

Listed on the lecture slide for November 18, 2026.

- [In Dialogue with AI: In Dialogue with AI](https://indialoguewithai.org/)

### IV · The AI Dilemma

Listed for February 2027, with the exact spring date still to be confirmed at the time of the lecture.

- [In Dialogue with AI: In Dialogue with AI](https://indialoguewithai.org/)

### V · AI and Humanity

Listed for March 2027, with the exact spring date still to be confirmed at the time of the lecture.

- [In Dialogue with AI: In Dialogue with AI](https://indialoguewithai.org/)

### VI · The Myth of AI

Listed for April 2027, with the exact spring date still to be confirmed at the time of the lecture.

- [In Dialogue with AI: In Dialogue with AI](https://indialoguewithai.org/)

### Why this reading edition exists

An audience member asks for information below the level of headlines and for access to the slides. I promise annotations and links, and explain why I work back to technical papers: the details of a system and its failures often tell a more useful story than a dramatic summary. In this edition, I follow through by distinguishing research findings, examples, and my interpretations.

- [Recording · 1:20:23–1:22:17](https://www.youtube.com/watch?v=klzLRdHmeCs&t=4823s)

### Sources and further reading

- **[In Dialogue with AI](https://indialoguewithai.org/)**. In Dialogue with AI, 2026–2027. The series website is the place to check current event details. This reading edition preserves the program displayed at the September 16, 2026 lecture.