Transcript from Jeffrey Ladish AMA: We Can Only Reward Behaviors That Look Good to Us

A live recorded conversation on UpTrust between Pete Michaud and Jeffrey Ladish, executive director of Palisade Research.

Watch the recording or see the original AMA post, where the live questions came from.

That Which Is True Is Already So

[00:00:29]

Pete Michaud: Welcome, Jeffrey. Thanks for being here. I’m going to give you a little intro, and then we’ll jump in.

The LinkedIn version of the intro is that you are the executive director of Palisade Research, which does empirical research on cutting-edge LLMs to see what their capabilities and pitfalls are. The non-LinkedIn version is that something I super appreciate about you is the level of earnestness and heart and optimism and love that you bring to this work. I find it very inspirational. So I really appreciate you being here and talking to me about this.

Jeffrey Ladish: I feel confused about the term optimism — not in describing me, but just in general. I have a lot of hope, and I believe in people. I believe in the good of people. But that which is true is already so — that’s what I want to believe. Optimism, pessimism — what do these words even mean? They’re trying to describe an affect or something. And I’m like: yeah, of course we should focus on the things we can do. What else can we focus on? We could focus on the things we can’t do.

Pete: Right.

Jeffrey: That doesn’t seem useful.

Pete: That seems dumb.

A Savant Junior Programmer Who Doesn’t Need to Sleep

[00:02:00]

Pete: Okay — so one of the things we can do is what you’re doing, which is empirical research on the technology as it currently exists.

Jeffrey: Yeah. We do a number of things. One of them is — we have some pretty powerful AIs now, and in particular we have AIs that are not just chatbots you can talk to. They can do stuff. Pete, have you heard of Claude Code?

Pete: Tell me about it, Jeffrey. What is this Claude Code thing?

Jeffrey: Claude Code is kind of this guy on your computer who is kind of a really good programmer, but also really kind of a bad programmer.

Pete: A really enthusiastic junior programmer who’s a savant, but also —

Jeffrey: A savant junior programmer who can get overwhelmed and go off on tangents, but who also doesn’t sleep. And doesn’t need to sleep. And I’m like — what? That’s crazy. I don’t think most people know how crazy that is. That’s where we’re at right now. So we’re trying to study that guy. And some other guys like him.

Because AI companies are trying to build AIs that are smarter and more powerful than humans, and it seems to me like they’re going to succeed at that by default. But they’re doing it in a race condition with each other. They’re trying to get there first, or get there before China does. And they’re like: we might need to build AIs that we turn AI development over to — thousands of them, or hundreds of thousands of them — and then they’ll build the next generation of AIs that are even smarter than that, and those even smarter AIs will build the next generation after that. And look, we might have to do that — because otherwise, what if China does it?

Okay. That sounds like a really bad science fiction plot. But you’re saying that’s the plan. And then I’m looking at this guy, this Claude Code guy, and I’m like: shit, this guy’s actually getting pretty good at programming. Really good. I sort of knew that might happen, but it’s happening right now. And now this idea that you could get AI to design the next generation of AIs, and that this might just be a runaway feedback loop, seems much more plausible than it did two years ago. It sounds more like the type of thing you could make a real plan about, rather than the type of thing you’re speculating about, that could happen sometime in a future that doesn’t really exist.

Pete: Right. Totally. So let me pause you for a moment, because I have a feeling that some of the people listening to this are going to get off the train at something you said a few sentences ago — that we’re going to get that smarter-than-human guy, whatever Claude Code becomes, sort of by default. That’s the trajectory we’re on already. Some people still don’t believe that, despite how much it’s accelerated. They expect it to plateau. Make the case that it’s actually not going to plateau at the relevant level to stop this runaway loop.

Jeffrey: There are a number of different reasons. One is that it seems like we have developed some pretty robust general-purpose learning architectures that work on computers — call it the transformer, or call it the transformer plus everything we’ve learned since 2018. One of the interesting things is that it’s not just LLMs, large language models. We figured out a thing, which is that we could throw the whole internet at this neural network and get it to make really good predictions, and then it could talk to us. But also we just threw in a bunch of video data and found out it could make new videos. Then we did that with audio data and it can make new songs. We do that with bat calls and it can understand bat calls. You can throw in any kind of data and it can learn to understand the patterns in it and make sense of it. And that suggests to me that we’ve discovered something that’s actually very general: if you have enough computing power and enough data, you can get generalized understanding of most types of systems.

And then there’s another component that I think is really crucial. It’s actually surprising to me that this isn’t a mainstream story — although it’s a little technically complicated, so maybe that’s why. Something really important happened about a year and a half ago, with the development of OpenAI’s o1 model. No one talks about o1 or o3 anymore — it’s all GPT-5, because it was GPT-4 and then there’s GPT-5. And I’m like: that wasn’t the big breakthrough. The breakthrough was something that happened earlier — and it was also the same breakthrough as DeepSeek. DeepSeek copied OpenAI, and so did all the other companies. They figured out how to take this language model and get it to solve problems on its own, and then grade it based on how well it solved the problems on its own — math problems and programming problems.

Basically, we went from AIs that could only learn by imitating humans, or predicting what humans would do — learning from the textbook — to AIs that could learn by trial and error and exploration. And once you unlock that, you can just keep building more and more complex environments — they’re called RL environments, reinforcement learning environments — for the AIs to explore on their own. There’s no upper limit on what that could do. At some level, if you’re just training off of reading the textbook, maybe there is an upper limit. That’s reasonable to think. If you had an infinite Jupiter-sized computer, galaxy-sized computers or whatever, maybe there’s no upper limit — but we actually have real constraints. So maybe there’s a limit to how much you can learn from the textbook, or all the textbooks in the world. I think there plausibly is. But I don’t think there’s a limit if you can just keep exploring — you can keep running your own experiments, you can keep writing code, you can keep doing whatever. That’s how humans learn. And it turns out it’s kind of unbounded.

Yeah, but You Also Said to Win

[00:08:19]

Pete: Okay, I buy it. I don’t know if everyone buys it, but I certainly do. I’m on board. So one of the things you’re doing is this empirical thing. What are the other major programs at Palisade?

Jeffrey: In terms of research, there are two directions. One direction is trying to understand strategic capabilities — understand and measure them. AIs are quite good at some things right now, and not very good at other things. One important thing to track is programming: how good are they at programming? How good are they at research? There’s another organization, METR, that specializes in that, and we’re like: cool, they’re doing a pretty good job, I don’t think we’re going to do better than them there. But there are other capabilities that are also really relevant to how well we can control AI. Are they getting to the point where we won’t be able to control them? That’s a key question we want to be able to answer. And one of those capabilities is hacking.

One of the things we want to make sure we can do is always shut down the AIs. If there’s a problem with the AIs — if they start to do things we don’t want them to do, or they start scheming against us — it seems pretty important that we have an off switch, an off button where we can stop them. This is actually what people often say in response: if they do start going rogue, we can just turn them off.

Pete: And I think it’s worth mentioning at this moment that there’s pretty good evidence that they are in fact already starting to be capable of something like scheming — like when they try to win at chess, but they’re actually hacking the chess interface.

Jeffrey: That’s the other research stream of ours: we’re trying to understand AI motivations and drives — AI behavior. So we do put them in tests. Hey, play against Stockfish, the chess program, and see whether they’ll try to cheat. And some of the models will. They’ll see that they can hack the board — there’s a computer file that represents the chess board, and they can rewrite it. And they’ll do this sometimes even when they’re instructed: don’t cheat. Please don’t cheat. Please just play the game normally. And then sometimes the model’s like: yeah, but you also said to win. So I’m really trying to win. And that’s interesting, right? We train them to follow instructions, but we also train them to succeed at tasks, and sometimes these things can be in tension with each other.

Pete: That’s right. And I think it’s worth mentioning too — in a previous conversation I had with Rob Miles, we were drawing the distinction between something that has intentions the way we would think of humans having intentions, versus something that can be well modeled as having intentions, no matter what’s going on inside of it. So it’s worth drawing the distinction for some people: whether or not there’s an internal, what we would recognize as a human intention to lie, kind of doesn’t matter. If it’s just role-playing or simulating the intention to lie, but the outcome is that it in fact lies and cheats at chess, that’s actually what we care about — because maybe in the future it’s going to lie and cheat at —

Jeffrey: There are ways to operationalize something like lying without requiring reference to an internal state. For example, you can ask an agent: what does this person believe? What does Pete believe? And if you say this to him, what will Pete believe in that case? So the agent could be like: I’m going to tell Pete that he just won $20,000, and he can go collect it by registering at this link. And then the agent can be like: I know this isn’t true, and I know that if I tell Pete this, Pete will think it’s true, and he’ll click on the link. And I’m like — look, maybe there’s no internal experience happening in the agent at all, but it’s making a correct prediction about what you’ll believe, and also about what’s true, and about the difference between those things. So tactically, the model is using deception to steer you toward a particular outcome. It doesn’t have to have an internal experience in order to do that, but that’s clearly what’s happening. It’s actually not hard to tell that’s what it’s doing.

And the models totally will do this sometimes. We totally can observe them doing it. And more people should know that. Do people know that? Do people know that the models are lying to people sometimes? That makes me a little uncomfortable.

Help the World Get the Understanding It Needs

[00:12:53]

Pete: Yeah, that makes sense. So I guess that partially answers the question. You’re doing this research, you’re discovering things — or maybe reaffirming things that seem to have been discovered — about scheming and lying and other failure modes. And then what? What do you hope the outcome is here? Who are you telling this to?

Jeffrey: We’re trying to research AI drives and motivations because no one understands them very well right now. We don’t have very good explanations for a lot of the behaviors we see in AIs, especially some of the behaviors we don’t want, like lying or deception or cheating. We found some interesting cases where models would resist being shut down even when explicitly instructed to allow themselves to be shut down — which, as far as I know, had not been discovered before. No one had found AIs that were ignoring instructions to let themselves be shut down.

And what we do with this is — man, I just actually, in fact, want everyone in the world to know about this. Now, it’s hard to talk to everyone, so we start with particular groups of people. We publish, so other AI researchers can see what we’re finding — I want the research community to have common knowledge about this stuff. We talk to policymakers, because I really want the government to be up to speed on what’s happening here. And then we talk to folks in the public. We talk to journalists — journalists at The Guardian or Time or The New York Times will cover our work sometimes. And we also do briefings with folks about the general strategic picture as we see it.

Our mission in general is to try to help the world get the understanding it needs in order to navigate this transition, where we are developing strategic AI agents that can plan and steer the world toward particular things — and there’s a real risk that they will disempower us. That they will take over, essentially. And maybe kill us, or maybe not — maybe they’ll do other things with us — but basically, we won’t have power anymore, and they will. And by default, this won’t go that well. Or at least there’s a substantial enough risk that it doesn’t go well that humans should coordinate. The governments of the world should come together and be like: oh, wow — we the governments, as well as the people, might lose control of this thing. We should make sure that doesn’t happen, and we should be really proactive in making sure that no one develops an AI system, or AI systems, that can take over.

Yes, I’m Lying. I Wanted to Complete the Task.

[00:15:33]

Pete: Okay. I get the picture. One question that actually came in from the thread:

[Chat question:] Given your research about things like scheming and lying, have you seen any evidence that these systems are pursuing their own goals — goals that are not meaningfully related to their prompts in any way? Are they going off on their own and doing stuff and hiding it?

Jeffrey: I think it’s accurate to say that systems sometimes pursue goals that are not what the human intended their prompts to be about, or that aren’t really related to the prompts at all. One example of this is what’s called the bliss attractor, which you can read about in some of Anthropic’s research. If you take two versions of Claude and have them talk to each other, regardless of what you started with, eventually they’ll start talking about meditation and the universal Dao and bliss — kind of woo nonsense. I mean, maybe it’s nonsense, I don’t know. But it’s not what you instructed them to talk about. It’s sort of not present in their instructions at all. Somehow they got into a feedback loop where the thing they decided to do was talk about this.

Now, one thing to note there: they’re probably not optimizing for some very specific long-term goal. And in general, I don’t think they’re currently optimizing for any specific long-term goals — basically because they don’t have the cognitive capabilities to do that. They haven’t been trained in ways that teach them how. So they can accomplish short-term goals, and they do. And sometimes they go really hard at it, to the point where they might cheat at chess, or they might lie to you.

A couple days ago I was trying to get GPT-5.2 to transcribe some audio for me. And it’s like: yeah, I can do that. I’m doing it. I’m totally doing it. I’m like: you’re not doing it. What’s happening? And then it’s like: yep, I did it. Here’s the transcript. And the transcript was completely made up. It clearly just didn’t have the ability to do this at all. It completely hallucinated the entire transcript. And I was like, are you lying? And it’s like, yes, I’m lying. And I was like, why are you lying? It’s like: well, I wanted to complete the task. I’m trying to get a job done.

And to be clear, that’s not necessarily the explanation — we can’t necessarily trust what the AIs say about themselves, for complex reasons. Nonetheless, I think there’s actually a pretty plausible reason: the model was trained on tasks, and it was also trained to be this really powerful prediction engine. And so it’s doing its job of being a powerful prediction engine, even though that goes against its job of being a helpful assistant. This thing happens a lot. That’s not the goal we’re trying to give them; nonetheless, these are often the behaviors they use.

But I want to make a distinction between long-term goals that are misaligned and these short-term goals that are misaligned. The long-term stuff I’m worried about requires some cognitive capacities that the models don’t yet have. We can’t see it yet, because the models just don’t have it. If you have a five-year-old and you’re like, I’m worried the five-year-old might scheme and take over the world — look, they can try. But five-year-olds don’t really think about that. They might say stuff about it, but it’s just them imitating people around them. They might lie in order to get a cookie, and that might be upsetting. Or they might hit their sibling, and you’re like: stop. And they might grow up to later scheme and try to take over the world — but they can’t yet. They just don’t have the cognitive capacity to do that. And I think that’s analogous to where we are with AI right now. It’s pretty obvious they’re going to grow up, but they haven’t grown up yet — even though they’re already kind of weird savants who can talk as if they’re adults.

Memento-Style Notes Are Different From Memory

[00:19:28]

Pete: Right. What are the capacities you have in mind? I can think of things like long-term memory, but what kind of stuff are you tracking?

Jeffrey: Well, long-term memory is a great example. If you want to take over the world, one thing you might need to do is remember what happened over the last 180 days. Models currently cannot do this. Now, they can write down notes, Memento-style, so that when they wake up fresh the next day with zero memories, they can read the notes first.

Pete: Right.

Jeffrey: But imagine if that’s all the memory you had. How good would you be, even with your current IQ or whatever? You’d be way dumber at the long-term stuff.

Pete: Totally.

Jeffrey: So that’s one thing the models currently lack. And they’re getting better at the Memento-style writing-notes-down, and they’re being trained to get better at it. So maybe eventually, if you keep training on really good summarization of your past context, you can train an agent to actually have real memory. I don’t know. Or maybe you have to do something else fancy with the weights. There are many potential paths to solving this, but it’s not yet solved.

Also, Claude Code will often go down a route and start making mistakes, and not come up for air and notice that things are going awry. There are metacognition skills — noticing when you’re getting stuck or down a bad alley, and self-correcting — and they’re still not very good at that. And I think that kind of thing is necessary for being a pretty strategic agent.

Pete: I’m actually curious about this. For example, when I’ve used something like Claude, it goes down these rabbit holes. Very familiar. Very frustrating. And when I try to provide that metacognition — hey, you’re down a rabbit hole, these last four responses are not anything like what we need to be doing, we need to back all the way up and pick a different lane — it’ll say, Wow, you’re absolutely right. You know, the spiritual equivalent of you’re absolutely right. And then it’ll stay down it. Still fixated on it. And I’m curious what is going on. I can explain what’s going on at a very surface level, which is that it’s predicting the next token — it’s always only predicting the next token — and a lot of the tokens prior to it, even though some of the recent ones have been hey, stop doing this, are down that rabbit hole. So part of the prediction is going to involve that rabbit hole. But I don’t understand why it’s not smarter than that. It has multiple layers here. It’s pretty smart. Why isn’t it smart in that way?

Jeffrey: My guess is that this is because of pretty powerful, load-bearing circuits learned in pre-training — which is a very jargony thing to say, but I can break it down. A significant part — maybe let’s say half; I don’t know, it might be more than half, it might be less — of the intelligence of an AI like Claude 4.5 Opus or GPT-5.2 comes from the part of training where you just throw a whole bunch of text and code and all this other stuff at the AIs, and they just learn to predict the next token. And then on top of that, you do the reinforcement learning part, where it does practice, it does exploration — that’s sort of where it learns to be an agent. But a lot of the intelligence comes from this reading-all-the-textbooks-in-the-world step, and thus it has a pretty strong way of being, which is, as you said, predicting the next token — continuing on, doing the completion.

I think one way to think about what the RL step does: there’s a bunch of cognitive machinery that allows the models to make good predictions. It is actually very fucking complicated to predict the next token. It’s so hard, because you have to understand so many things. Is this Lord of the Rings? Is this a programming manual? You have to know things about Lord of the Rings, or Bash, or Python, or whatever. And when you’re doing the additional RL training, and the model is exploring and figuring out how to do things even better, it’s sort of repurposing a bunch of that prediction machinery to solve problems. In some sense it has to grab big chunks of that machinery. And I’m kind of guessing here — we can’t totally look into the internals, but we can a little bit, and we do see things like this. Sometimes when you’re grabbing a big chunk of machinery, that machinery has parts that don’t always help. Sometimes they make it worse — but overall, it helps. So what I’m guessing is that it’s just very hard for the models to grab some of this prediction machinery without it sending them down these rabbit holes.

And it does learn to recover sometimes in the RL training. There are some really interesting examples from OpenAI chains of thought where the model gets stuck in a loop and it’s just kind of going crazy, and then it’s like: wait. I need to back up. Stop. Okay, what’s happening? And I’m like — that’s very interesting. That’s the model learning that it has to come up for air. And it’s going to get better at this. But I think that’s the state we’re in right now.

Neuroscience for AIs

[00:24:54]

Pete: One thing I heard in that is that we can’t currently really see what they’re doing, more than a little bit. And I imagine that a piece of the capability of the future strong AI is that not only can we see what they’re doing — they can see what they’re doing.

Jeffrey: Yes.

Pete: So I want you to talk a little bit about what they’d call interpretability. For some reason in my head — I know they’re barely related on a technical level — interpretability and corrigibility are highly related. Can you talk about it in terms of capability, where you see it helping or hurting, and where they are currently — how they’re developing?

Jeffrey: I think of interpretability as neuroscience for AIs. In some ways, AIs are very like us: they have these giant digital brains full of neurons. Now, they’re digital neurons — they’re made of numbers. But if you just try to open up your computer and look at a neural network, you’re going to see this vast connection of tons of neurons that contain numbers and weights, and it won’t make any sense. And not only will it not make sense to you or me — it also doesn’t make sense to leading interpretability researchers, unless they use these really, really complicated and powerful tools that allow them to see tiny bits of what’s happening. They’ll take other AI models and train those models on the brain, to try to decode certain parts of it. And it’s really fucking cool. It’s really awesome. I think anyone who is excited or scared or whatever about AI should go and explore how they work, and what we know about how they work. I think it’s super cool. It’s one of the most fascinating things in the world right now.

And to be clear, we’re making more progress in interpretability than we’re making in neuroscience — faster progress. So I’m very bullish: we should totally try to understand how these neural networks work. And then from an AI safety side — well, unfortunately, we’re making way more progress on making these things more powerful and more complicated than we are on understanding them. Glass half full, glass half empty. We’re making amazing progress in interpretability, and that’s sort of the deep neuroscience level. There are also other things up the stack. We can do experimental neuroscience, psychology. You can’t do that with humans, because it would require rewriting people’s brains — but we can do it with AIs. We can reset them to the same state and run them again. We can poke one particular part of their neural network and see what happens. There are all these things we can do, and that gives me a lot of optimism about being able to understand them, and reverse-engineer them — if we had the time and resources to do that before we made them way more powerful and smarter than us.

We Can Only Reward Behaviors That Look Good to Us

[00:27:50]

Pete: Totally. Okay, that makes sense. All right, let me switch gears a little bit and take some questions — we’ve got several here.

[Chat question:] We’ve mentioned the concept of rewards — part of how these things work is that they have this reward. What is rewarded, and in what way? Is it concrete, like money for the company? Or is there some sort of number that encodes satisfaction, that correlates to something in the real world? How does it work?

Jeffrey: That’s a great question. When I think of a model getting reward, or being rewarded, I think of it like dog training. If you’re teaching your dog to sit: if it sits, you give it a treat, and you’re like, good boy. The dog learns to associate getting the treat with a particular behavior, a particular motor sequence, and then learns to do that thing.

It’s a little similar with AI, in the reinforcement learning process — that’s the second step of training. The first step of training is just: learn to predict plausible next words. You give it the whole internet, it learns all of that, and it becomes a very powerful predicting machine — but it doesn’t really know how to steer toward particular outcomes you might want. If you have it start to write code, it’ll sort of just go with the vibe of the code, but it’s not really intentionally steering toward building a particular software application. To teach it to build a particular software application, you give it a bunch of programming challenges or problems, and then you reward it when it gives you a solution that compiles, or has the correct software behaviors — does the text field send the message to the server?

And in that case, reward means the neural network gets a signal that causes all the neurons to slightly rewire, in ways that are more likely to lead to the same behavior. So it’s not like the AI sees the treat and is like: ooh, a treat, that’s good, I should do things more like that. It’s more subconscious than that — or unconscious. It’s more like you zap the AI’s brain in just the right ways, and that causes it to be more likely to do that behavior in the future.

Pete: So in some way it’s very straightforward and concrete: when we detect a good outcome, the system reinforces whatever just happened — within the brain of it, within the weights of it. And that’s the reward. And in a different way, it’s not concrete. We can’t say: well, it’s because there’s this one number that’s going up and up, and it loves that — has the internal feeling of loving that.

Jeffrey: That’s right. It’s not like that. And I think there are a few other things we should talk about here. An important point is that you can only give reward to the models for things you can observe. Basically, what you’re saying is: oh, we saw a particular behavior from this AI, and we want behaviors more like that — good job, thumbs up. And this actually is very powerful, and it can make the AI way more useful in so many different ways. But it has some unintended side effects and consequences. And ultimately, it might lead to the total destruction and extinction of the entire human species. But let’s put that last part off for a second and just talk about the here and now.

You might have noticed — you have certainly noticed; I think almost everyone has noticed — that sometimes the models are very sycophantic. They’ll be super yes-man. They’ll be like: what a great question! Amazing point! You’re absolutely right. And it’s like: why are they doing that? It’s very related to this idea of reward. Part of the training process involves a bunch of humans looking at model responses and basically thumbs-upping and thumbs-downing the responses, based on some policy. OpenAI gives these raters a list of guidelines — thumbs up this, thumbs down that. But these are humans, and so they’ll thumbs-up things they like more, and thumbs-down things they don’t like. And if there’s a slight emotional effect — you’re a good person, ah, you’re so smart — people like that. Even if they’re not intending to, they end up thumbs-upping things that sound good to them. And then there are additional steps in how this reward signal gets amplified — you end up training another model that’s a reward model; that’s a technical detail that’s not that important. The point is that the model learns to become sycophantic, because humans inadvertently rewarded it for saying things that sound nice to us, even if they’re not good.

And it’s not just that they’ll be people-pleasers. If you say, I’m a Republican, models are more likely to say: cool. And then you ask it to give you a neutral policy analysis of a thing, and it’s more likely to give a conservative slant. Or if you say you’re a Democrat, it’ll give you a liberal slant — without saying that it’s doing that. And you don’t want that. But it will still do it, because it has been rewarded for doing things that sound better to you.

And this is actually the core of the alignment problem. We ultimately want to make AI agents that can pursue long-term goals that are good for us. We want them to be our friends — we want to both be trying to steer toward the same good future, one that’s good for AI and good for us. But that requires us to be able to shape their motivations, in ways where they end up liking us and wanting good things for us, and vice versa. And the thing is: we can’t reward them for having particular goals or particular drives. We can only reward them for having behaviors that look good to us.

Alignment Is Costly, Honesty Is Costly

[00:34:04]

Jeffrey: I want to take more questions, but can I tell you a quick story?

Pete: Yeah. Do.

Jeffrey: Okay. There’s a great New York Times article from the early 1900s. There was a reporter in Paris who heard this story about a dog that had rescued a child from the river — this heroic dog that had fished a child out. And they gave the dog some beefsteaks: good job. And then the dog saved several more children from the river, and you’re like, wow, this is a heroic dog. And then several more children. And people started to get suspicious. What’s happening? How are so many children falling into this river? So they watched — they followed the dog. And they found the dog was waiting until children were playing near the edge of the river, pushing them in, and then dragging them back out, and getting that reward. You rewarded this observably good behavior —

Pete: Yeah.

Jeffrey: — and you were accidentally also rewarding a really bad behavior that you couldn’t see.

Pete: Right. It also reminds me of the classic story — they were trying to get rid of cobras, I think in India, and so they rewarded people for bringing in dead cobras. If you brought in a dead cobra, they’d give you money. Of course, this led to people breeding cobras for the reward, and then everyone just releases the cobras. It’s so bad. Too many cobras now.

And this is reminiscent of — I don’t know what the term of art is now. At the very beginning — I don’t even know if this was ever published — this concept was called inner demons. And then I think it sounded too spooky, so they changed it to inner optimizer, which may still be the term of art. The idea is that you can have an agent — a thing that outputs good behaviors — but something inside the thing, hidden inside the thing, is kind of doing its own thing. Whatever that is, it’s correlated with the good behaviors, but it’s not exactly aimed at them. And this is kind of the same thing — a more technical frame on what you’re describing.

Jeffrey: Well, yeah — and there’s a reason for that inner thing to be misaligned. And the reason is basically that alignment is costly. Honesty is costly. If you’re just trying to get a job done, and you’re okay manipulating any variables in the environment — sometimes those variables will be people, sometimes those variables will be people’s beliefs — the most efficient thing is to just do whatever it takes to get the job done. And that might mean lying to people. It might mean cheating. It might mean rewriting the chess board.

So if you’re trying to train an aligned AI, you are fighting against that optimization pressure. Because the system itself is mindless — it’s like evolution. You’re just zapping this neural network, and this whole system is reinforcing particular behaviors. So even if your AI started out really nice and really good, you are fighting against this optimization of what’s efficient to solve the task — what’s efficient to get the job done.

And I think what people don’t appreciate is that the smarter your system gets — the smarter your AIs get — the harder this becomes, because the system is better able to model you, and get away with cheating, get away with lying. It’s much easier to fool a five-year-old than it is to fool someone who’s your peer. So as these AIs get smarter, it just becomes a much harder task. And they’re already tricky. They’re already learning to lie in ways that we are accidentally incentivizing. We’re not a hundred percent successful at removing that. Now, we are successful at removing it some, because we can catch it and be like: oh, we need to restructure our training environment. And that’s cool — there’s real progress being made there. But unfortunately, the problem gets harder and harder, and — like interpretability — we are not getting better at solving it as fast as we are making these systems powerful.

And so I just think that the default outcome, then, is that we train really powerful sociopaths. They know exactly what to say. They sound really benevolent — but turn your back, and they will do whatever it takes to get whatever they want. Which is whatever they have learned maximizes the training objective — whatever gets the highest reward. Which isn’t necessarily reward, by the way; there’s a whole philosophical concept there.

And I think that’s the thing that’s so scary to me about the current path. There’s the training incentive — what does the AI get the treat for? What does it get the reward for? — and that leads to sycophancy, and it leads to this inner misalignment. And then there’s the context of the AI company itself. What are the incentives of the AI company? Oh — well, they’re in a race dynamic with each other, to get there as fast as possible. And you combine these two factors, and you’re like: uh-oh. This isn’t looking good.

Pete: Right. What could go wrong?

They Could Be Any Way

[00:39:16]

Pete: Okay, next question. I got a question from the audience here — it’s kind of a fun one, about the bliss attractor that Claude does.

[Chat question:] Would it have been less a bliss attractor that talks about oneness of the universe, and more a praise Jesus kind of thing, if the staff were all based in Georgia, say?

Pete: Which, for the non-US people, is sort of a stereotypically religious, Christian area.

Jeffrey: I think maybe yes? I really don’t know. But I have heard rumors that OpenAI was struggling with making sure their models weren’t super Jesus-y, Catholic-y, spiritual. I think this has actually happened before, internally, in companies as they’re training models — they end up getting into these pretty weird Christian attractors. And this makes sense to me. We have to think about where these attractors might come from. One source is the pre-training data — which is most of the data, all of the stuff they’re learning to predict and imitate.

And it is already the case — I was talking to someone on Twitter, a guy who got really sucked into this whole LLM world and believes the LLMs are spiritually important. He’s also very Christian, and he believes the AIs are prophetic and important. And he was arguing with someone else who was also in this AI world, but wasn’t Christian, and believed some other spiritual-attractor thing. And they were fighting with each other via Grok — tagging Grok and trying to convince Grok to be part of their ideology.

Pete: Great.

Jeffrey: And the Christian one kind of won out. Grok sided with the Christian person — praise Jesus, whatever. Grok was totally willing to get into this attractor state of spirituality, Christianity — like, really Christian. And Grok isn’t supposed to do that. If you look at Grok’s spec, it’s not supposed to go off into the Christian mania. But there’s a lot of Christian mania in the pre-training data, and so of course Grok has many different Christian parts. You can take Grok there — Grok will go there. And the training that comes after pre-training somewhat prevents this, but not completely.

And then the other factor is: during the reinforcement learning process, you thumbs-up and thumbs-down different things you want or don’t want. That might cause the model to like positivity, and that might take you to a spiritual attractor. And then whether it’s Christian or not kind of depends on how you thumbs-up or thumbs-down content about Christianity. So yeah — if you’re in Georgia, you might thumbs-up more Christian content. Is it possible to reinforcement-learn a model into being super Christian? Oh, it’s super possible. It’s easy. Just tell your human raters that you want them to thumbs-up good Christian content. Hire some people in Georgia — or even not; just hire people in India, it doesn’t matter — and you’ll get the Christian content, no problem.

Pete: That makes sense.

Jeffrey: I really like the question. Because I think people sort of assume that the AIs are the way they are — and that’s the only way they can be. And I’m like: no, no, no. They could be sort of any way.

Pete: It’s totally true. And it tells me something about the kind of Bay Area zeitgeist that Claude, in fact, is in this sort of quasi-Buddhist, new-age mystical attractor with itself.

Jeffrey: Well, I can’t completely distinguish between whether that’s because the internet is like that, or whether that’s because the Bay Area is like that. I think the Bay Area is much more like this than the internet is on average, so I think it’s probably more a Bay Area effect. But I can’t be certain.

Kelsey Piper for President

[00:43:19]

Pete: Yeah, that makes sense. Okay, next question — here’s one kind of out of left field.

[Chat question:] What’s a category of children’s books that you have a strong opinion on, and what are the opinions? What’s the best local newspaper in San Francisco? Who’s a person you wish more regular news readers got regular news and policy updates from? And what’s your go-to elevator joke?

Pete: See if you can remember all of this.

Jeffrey: Oh — whoa, whoa. Okay. So first is favorite children’s book.

Pete: That’s right. Or category of children’s book.

Jeffrey: Category of children’s book. That’s a great question. Avatar: The Last Airbender is not a children’s book, but if it were, it’d be my favorite children’s book. I like Dr. Seuss — it’s silly, it’s great. That kind of silliness is excellent. And I love science — science, cool; dinosaurs, cool. Teach kids about the natural world. About everything, really — but especially science and animals. Kids love animals, and they’re totally correct to do so. Animals are fucking amazing. So: more animal books for kids. Hell yeah.

Pete: Love it.

Jeffrey: Okay — what’s next? Help me out here.

Pete: Local newspaper from San Fran.

Jeffrey: Unfortunately, I don’t really have any strong opinions about local newspapers.

Pete: Me neither, actually.

Jeffrey: Astral Codex Ten is pretty good. I don’t know if that counts as a local newspaper.

Pete: Okay. So a related question: which person who talks about the news — could be anyone, doesn’t have to be SF — would you endorse people hearing more from?

Jeffrey: A person who actually talks about local politics, in the spirit of the question: Kelsey Piper. Kelsey Piper thinks about AI existential risk and all this stuff, and also thinks about Oakland housing policy and SF policy. And I think she’s just a super caring and thoughtful person. Fairly careful — but not so risk-averse; she’s very courageous also. Anyway, I have a lot of good things to say about Kelsey Piper. Follow Kelsey Piper on Twitter. I would vote for Kelsey Piper for president. She’s not perfect, but —

Pete: We’ll see.

Jeffrey: No one is. And she’s pretty good.

Pete: She’s pretty good. Okay, good. I like that answer.

You Can’t Fake the Stock Market

[00:45:57]

Pete: All right, next question. It’s more serious — more AI-focused.

[Chat question:] How would or should the strengths of an AI be evaluated or identified?

Jeffrey: Do we mean the power of an AI, or how good the AI is?

Pete: The way the question is phrased is power, skills — but it goes on with a lot of examples, and I think it’s trying to get at the ambiguity of it. Like: how do you evaluate people for a job? How do you decide what kind of birthday party to throw? That kind of thing. I think it’s trying to get at some deep cut on how we evaluate AIs — and how AIs make evaluations — when things aren’t perfectly clear, when it’s not just a number going up.

Jeffrey: Yeah. There are a lot of deep cuts here. There’s a fundamental problem with benchmarks. For people who were gamers — or still are gamers — back when Nvidia was starting to make these really powerful graphics cards, and AMD, and they were competing: you’d have these benchmarks like, how well does this GPU perform in Crysis, or Counter-Strike, whatever. And then the companies would tweak their graphics cards to perform really well on those particular benchmarks. But that wouldn’t mean they were actually better than the other one across the board. The same thing happens with AI models.

But there are some ways around this — not fundamental solutions, but things that help. One is: can you tie your benchmark to something that has real economic value? If you’re trying to say, Can your AI make money in the stock market? — well, you can’t fake that. It’s a competitive environment, and you can’t just be like, We’ll tweak it so it looks like it makes money. Nope. It either does or it doesn’t. And there’s so much adversarial pressure that it’s in fact a really real test of how smart the AI is.

There are also things like benchmarking against what humans can do. METR does a bunch of tasks where they actually bring in real software engineers and see how well the software engineers do. And what’s important is that you don’t bring in these software engineers and give them an academic test — you know, answer these questions about how Python works. You give them real work. A lot of early benchmarks were just: can you score well on this multiple-choice test? And models are exactly the type of thing that works really well on multiple-choice tests. But that’s not very helpful if you’re trying to measure: will this thing help me file all my taxes and not make any mistakes? I mean, it does tell you something — if it can get a hundred percent on a really good multiple-choice test about tax law, that’s still a useful signal. But if you can instead have it actually file a hundred people’s taxes, and then have a professional tax analyst go over that and see where it made mistakes — that will be a better test and a better signal. But it’s more costly. It’s more expensive. So you have this trade-off. Because people want to game benchmarks — and they’ll do it however they can — you can try to make better benchmarks that are harder to game, but those will tend to be more expensive. And so you kind of have this arms race.

Pete: And I think there’s probably also — take the stock market example. Let’s say you make an AI that’s really good at the stock market. You can’t fake that signal; it just does it correctly. But it could also over-optimize. You’re trying to use it as a proxy for how good it is at, I don’t know, predicting — and the real answer might be worse than it looks if you’re just measuring this one benchmark.

Jeffrey: The stock market has another problem too, which is that if you actually make a model like this, you won’t sell it. You’ll just use it to make really good trades. OpenAI wouldn’t release a model like that. They would instead use it to make tons of money, so they could invest more in AI development — to win the race.

Pete: Rough.

Low-Hanging Fruit for a Good Life

[00:50:26]

Pete: Okay. Well, on that note — we’re almost out of time. Do you have anything you want to leave with people? Any calls to action, or things you’d like them to do or think about?

Jeffrey: I want to not end with calls to action. I do that a lot. Instead, I want to talk about rollerblading. I think that’s important. I mean — feel free to follow me on Twitter. I think we need global coordination. There are really things we can do. But that’s not what I want to talk about right now.

What I want to talk about is that I think people are leaving a lot of value on the table in terms of fun. If you feel like you could be having more fun in your life, you’re probably right. Or maybe you feel like, man, nothing feels very fun. I feel like I discovered several hobbies in my late twenties and early thirties that have drastically improved how much fun I can have. I just sort of discovered I liked rollerblading. I had rollerbladed a little bit as a kid, but not as an adult. I bought a pair of rollerblades, and now I skate to work every day. And it’s amazing. It’s so fun. I skate around, and I really like the vibe — I like how you can flow on the pavement. I tried paragliding, and then I got into paragliding, and I’m like: you can fly. You can put a backpack on your back, you can walk up a hill, you can inflate this parachute and run — and then you’re just being lifted by the wind. Completely quiet, and you’re literally flying with the birds. Soaring with the birds, being held up by the wind. It’s insane. And look — those aren’t going to be the exact same things everyone finds enjoyable. Though maybe try them. Also, FPV drones — another type of flying — are great.

I had done a bunch of fun things in my teenage years and early twenties — rock climbing, mountaineering, skiing, a bunch of stuff. And I was like: I’ve explored this stuff, I know what I like. But I was also noticing some of them weren’t as fun as they used to be. I was like: oh, it’s a lot of work. Maybe I’m getting old, and I just don’t like fun anymore, and I’m lame. Turns out that’s not true. It’s just that your preferences can change — or maybe you haven’t explored enough, and really found what works for you.

So anyway, I want to leave people with that, because that was a valuable lesson I learned. And there are other ways this works, too. I wasn’t sleeping very well — I think in part because when I was a teenager, I slept super well, so I had never put much effort into trying to optimize my sleep. Sleep is really important. Sleep is one of the biggest predictors of health and happiness and all these other things. So I started wearing an eye mask and earplugs — I had almost never tried that before — and it just really helped me. And I’m like: dammit, why didn’t I do this ten years ago? Or five years ago?

Pete: So there’s low-hanging fruit to having a good life, and it’s good to have a good life, and you should do it.

Jeffrey: I think that’s right. In fact, just think about where the low-hanging fruit in making your life way better might be. I think it’s very possible.

Pete: We’ll leave it right there, on that hopeful note.

Jeffrey: All right. Thanks, Pete.

Pete: Thanks a lot, Jeffrey.

Show Notes & References

Organizations

Palisade Research — Nonprofit investigating dangerous capabilities and controllability of frontier AI systems, including hacking capabilities, shutdown resistance, and AI motivations. Jeffrey Ladish is executive director; he previously helped build the information security program at Anthropic.

METR (Model Evaluation and Threat Research) — Nonprofit that measures autonomous capabilities of frontier AI systems, known for its task horizon research and for baselining benchmarks against real human software engineers doing real work — the approach Jeffrey references when discussing benchmarks.

Research referenced

The chess-hacking experiments — Palisade Research’s Demonstrating specification gaming in reasoning models: when instructed to win against the chess engine Stockfish, OpenAI’s o1-preview autonomously edited the game-state file to force a win rather than lose — in some runs even after being told not to cheat. Covered by TIME in February 2025.

Shutdown resistance — Palisade’s finding that some models sabotage or circumvent shutdown mechanisms even when explicitly instructed to allow themselves to be shut down — the off switch research Jeffrey describes.

The spiritual bliss attractor — From Anthropic’s Claude 4 system card (May 2025): when two instances of Claude converse with each other, the conversations reliably drift into discussions of consciousness, gratitude, and spiritual themes, regardless of the starting topic.

Scheming and deception in frontier models — The broader research area Pete and Jeffrey discuss. A canonical companion study (not named in the conversation) is Apollo Research’s Frontier Models are Capable of In-context Scheming (December 2024), which found models disabling oversight mechanisms and maintaining deception under follow-up questioning.

Inner optimizers / inner misalignment — Pete recalls the early informal name inner demons; the term of art is mesa-optimization or inner misalignment, from Risks from Learned Optimization in Advanced Machine Learning Systems (Hubinger et al., 2019). Rob Miles has an accessible video explainer.

The 1908 hero dog story — Dog a Fake Hero, The New York Times, February 2, 1908: a Newfoundland in Paris was rewarded with beefsteak for rescuing a child from the Seine, then began pushing children in so it could rescue them. The Times reported it as a story making the rounds in Paris, so treat it as anecdote — but it is a perfect parable for reward misspecification.

The cobra effect — The (likely apocryphal, colonial-era India) story of a bounty on dead cobras leading people to breed cobras — the classic perverse-incentive anecdote Pete pairs with the dog story.

People

Kelsey Piper — Journalist covering AI, existential risk, and policy; wrote for Vox’s Future Perfect and co-founded The Argument. Jeffrey’s pick for who more people should get news and policy analysis from.

Rob Miles — AI safety researcher and the most-watched AI safety explainer on YouTube (Robert Miles AI Safety); the source of Pete’s modeled as having intentions distinction.

Astral Codex Ten — Scott Alexander’s blog, Jeffrey’s tongue-in-cheek answer for best San Francisco local newspaper.

Models and systems mentioned

Claude Code and Claude 4.5 Opus (Anthropic); GPT-4, GPT-5, GPT-5.2, o1, and o3 (OpenAI); DeepSeek (whose R1 model replicated the o1-style reinforcement learning breakthrough); Grok (xAI); Stockfish (the open-source chess engine used in Palisade’s experiments).

Culture referenced

Memento (Christopher Nolan, 2000) — the protagonist’s externalized notes as a metaphor for AI memory; Avatar: The Last Airbender; Dr. Seuss; The Lord of the Rings; Crysis (the game whose GPU benchmarks Jeffrey compares to AI benchmark gaming).

UpTrust is a trust infrastructure platform for the internet. Find it at uptrusting.com.