Transcript from Liv Boeree AMA: The God of Coordination Failure

A live recorded conversation on UpTrust between Jordan Myska Allen and Liv Boeree, the closing session of UpTrust’s launch day.

Watch the recording or see the original AMA post, where the live questions came from.

Incentives Got Us Into This Mess

[00:00:00]

Jordan Myska Allen: Let’s just dive right into Moloch. Incentive traps, game theory — what’s your most up-to-date thinking on how we’re fucked and how we get out of it?

Liv Boeree: My most up-to-date thinking… well —

Jordan: Or your least up-to-date.

Liv: I don’t know how much background people have, but my work — my obsession, I guess — is around how incentives have gotten us into this mess, and thus incentives will probably get us back out again. At least I think that’s a very integral part of finding solutions. I’m sure that’s what most people listening are also in agreement with. But if not — the reason I think incentives are so important is that a lot, if not the majority, of our global issues are downstream of coordination failures.

Take deforestation — the general destruction of the biosphere. It’s not like people want to just tear down the Amazon. They don’t want the Amazon to die. But a lot of the people who live near the Amazon are under direct incentives to try and make money. A lot of them are cattle farmers, or they want more wood because the lumber industry is thriving and trees are worth more to the current economic situation turned into lumber than they are —

Jordan: Alive.

Liv: Alive. And so those short-term incentives, which are good for the individuals nearby, push them to extract faster and faster. But if everybody does it, it creates this devastation. It’s another way of saying tragedy of the commons — that’s an example of this.

But it’s not just in environmental problems where we have coordination failure. Look at nuclear weapons. Again, I think most people would agree we would rather not live in a world of thermonuclear weapons, where they’re on hair trigger and one accident and we could all be dead. But it made sense game-theoretically. Each side during the Cold War was incentivized to develop and stockpile more and more weapons, because if they didn’t, they would be left vulnerable to the other side, who did. And that was a relatively simple example, because it’s just two — a two-country prisoner’s dilemma, effectively.

But this is true of the drone warfare that’s going on, and bioweapon warfare, which is likely going on behind the scenes a lot. And for a more close-to-home example: AI is increasingly being used to make people become more and more dependent on different apps — whether it’s social media, whether it’s the LLMs themselves. The companies are in a tight race to try and keep afloat, to have the biggest market share, to have the most users, to be able to raise the biggest rounds. And unfortunately, that means they need to keep users on their site, keep ’em coming back for more. At some point, AI is going to be used as a sort of super-persuasion tool to keep people doing that. So again, it’s polluting the health of humanity for short-term gain for these individual competitors in whatever the race is. It’s all misaligned incentives — the short-term incentives on the individual are misaligned with what’s good for the whole.

The God of Coordination Failure

[00:04:00]

Liv: And you mentioned this word, Moloch. Moloch’s been around in sort of lore for thousands and thousands of years. It first appeared in the Bible, in a story warning of this cult that was so desperate to win and get power — some people say they wanted money, some say they wanted military power — that they were willing to sacrifice more and more of what they held dear, up to and including their own children, literally burning them alive, in order to get what they thought they would be rewarded with by this demon, Moloch.

And it was Scott Alexander, in 2014, in his seminal piece Meditations on Moloch, who first drew the link between these multipolar traps — coordination problems, tragedy of the commons, however you want to call it — and this demon Moloch. When I read that, it was just this light-bulb moment. I was like: this. This is the enemy. Everyone’s like, Oh, Russia’s the enemy. China’s the enemy, blah, blah, blah. No — I mean, they do bad shit, so do other countries, and so on. But it’s these incentives, these misaligned incentives, these negative-sum games that we get trapped in. That’s a real enemy, and Moloch is a name for it. Because I think we do better when we have — it’s very hard to get people riled up against a sort of nebulous, faceless, probably consciousless…

Jordan: Yeah — self-organizing system.

Liv: Yeah. Like, well, it’s just a bunch of economic incentives, or game-theoretical incentives. It’s not easily identifiable as an enemy. So I really thought it was quite genius of Scott Alexander to do that — to make it into a sort of Disney villain. And a lot of my work over the last few years has been spreading this meme, because I think it’s important for people to have a clear thing to point at. Especially in today’s age, where we’re more divided and at each other’s throats than ever. People are falling into what Tim Urban of Wait But Why calls Disneyland thinking — good guys and bad guys, and it’s all really obvious who the bad guys are. Okay, fine — yes: Moloch. Really obvious. Let’s all put our energy into that, because that’s what we actually need to coalesce around. And if we do coordinate around defeating Moloch, well, guess what? Then we actually solve Moloch. Because it’s a coordination — it’s the god of coordination failure, right?

So that’s a very long-winded way of saying: yes, it’s all about the incentives, and we need to figure out better ways of aligning the short-term incentives with the long-term whole.

Jordan: Beautiful. I love it. Long-winded — I mean, it’s extremely tight and comprehensive. You’re like, I’m going to hit on these — from the rainforest to social media to AI, boom, here’s the thing. So thank you. And I think you kind of gave a little hint at the solution. You’re like: look, if we can rally against Moloch, that acts as the coordination mechanism for us to go for the win-win, to not defect. Can you say more about that? What’s your sense of how we get out of that trap?

We’ve Broken Out of It Before

[00:07:00]

Liv: Well, I don’t think there’s one-size-fits-all. And I wish I could say, oh, this is how we do it — if I knew definitively, I assume I would probably be more successful at doing it. But I think one thing we can do is look at past examples of where we did manage to escape situations like this.

The ozone layer — the Montreal Protocol — is one example of a very distributed, very global environmental problem. The ozone layer was getting this big hole in it, and people were like, oh shit, why is this happening? And then they realized: oh, it’s probably the CFCs and other industrial chemicals that were going off into the upper atmosphere and ripping it apart. The scientists figured out the cause. So then it’s a question of how do we convince the companies that are off-gassing these CFCs — either through positive or negative incentives — to stop doing that.

And a number of things happened. The environment was fortunate in that, first of all, there were only a couple of companies really producing it — I believe DuPont was one of the main ones. Secondly, it was a time when information was actually quite centralized, which I’m not always pro, but in this case it was helpful — it was the seventies, and most people just watched a handful of news media. So when this story broke, there was a big hole and it was quite visualizable — a very dramatic image they were able to put out on media: look, this is what we found, here’s a big hole. It’s easy for people to picture — much easier than, for example, this is how carbon dioxide traps excess heat in the atmosphere, which is a very distributed, unvisualizable thing for a lot of people. So that was another convenient thing. There was this huge public outcry and panic, essentially — but I say panic in the best sense of the word, because it was largely justified and it was useful.

And then on top of that, there was a relatively simple solution for the companies that were making all the money from the CFCs: there was an alternative type of refrigerant they could use. I can’t remember what the actual name of it was, but it cost them a little bit of R&D — and I believe they were already doing R&D into alternatives anyway. And then they found this one, and because they’d been given all the pressure, it was like — oh. They got to be the heroes. They were the villain, but they were like, we’re the heroes — we’ve figured out the solution. And it by and large worked.

So there’s a lesson in there: clear information helps. Having a relatively small number of players you need to coordinate — in this case, companies. And of course, laws were passed as well — a combination of top-down incentives coming from above, saying you companies have to stop doing this and switch, coinciding with the fact that there were alternatives for them. And it’s not like DuPont made it easy — they initially resisted all of the science, because that’s what these companies do. But as soon as they figured out a solution, they were like, okay, we can switch. And then there was also this grassroots, bottom-up pressure as well. Those ingredients are directionally what we should always be looking for in any other coordination problem.

Another example: it wasn’t actually the height of the Cold War, but in the eighties, when the stockpiling of nuclear weapons was just going out of control — I think the number was like 70,000 or something insane. Like ten times the number you need to kill everyone on Earth, some absurd amount. And you’d think, well, this is an arms race — this is never going to stop, it’s just going to get worse and worse, all of these countries’ resources being funneled into making more and more nuclear weapons, because it’s this giant dick-swinging contest. But then — was it the New START treaty? — came along. It also coincided with the fall of the Berlin Wall, which made it more politically favorable. There was more dialogue and diplomacy — and actually, a lot of what helped with that was back-channel diplomacy between the USSR and the US.

They also came up with this really cool incentive. Because when you want to create some kind of de-escalation, ideally you need some kind of third-party body to be able to go in and check: okay, we’ve agreed to do this — are you actually doing the thing we agreed? Because trust is low. But a way they managed to find diplomacy was through the scientific teams: okay, we don’t want to see your capabilities, but we do want to see your safety protocols — to avoid, for example, an accidental misfire. Shall we both agree that we can each send a team to check? We can set everything up so you can’t see any of our actual useful intel, but you can check out our hair trigger and how robust it is against accidental fire. So they had that kind of cross-party safety-checking thing. That’s a way you can still verify things and create a kind of dialogue, but without having high trust.

So yeah, those are two examples. We’ve broken out of it before.

Moloch Is Decentralization Gone Wrong; Norma Is Centralization Gone Wrong

[00:13:00]

Liv: And so my suspicion is that any kind of solution to Moloch will usually be a combination. Because here’s the thing — you could say, well, we just need a centralized one-world government to tell us exactly how to behave. But then you do that, and you leave yourself vulnerable to an equally evil and nightmarish counterpart, who I call Norma. She’s basically the demon of bureaucracy — she’s got, what’s her name, Professor Umbridge energy. The demon of bureaucracy and nanny-stating. But it’s worse than that, because it’s where you end up with communism and all these nightmare totalitarian control systems. Also horrific. If Moloch gets its way, we all sort of die — the ecosystem falls apart, or chaos and war happen. And if Norma gets her way, we all end up potentially under some totalitarian hellscape.

Jordan: We live, but we’re basically dead.

Liv: Yeah — we have no freedom. So it’s: how do we thread the needle where we minimize the risks of the two — where we still have freedom, but we also have sustainability? And it’s really hard to do. But my suspicion — and this is really just me licking my finger and putting it in the wind, it’s a gut feeling — is that it needs to be some combination of smart top-down rules and incentives from above, and also bottom-up, sort of hive-mindy — both pressure, but also solution space. Because it’s a coordination problem, and it takes many minds to align to do it well. With many minds thinking about it, the solution will emerge. …I had another point. What was it?

Jordan: There’s the top-down, there’s the bottom-up, and then there’s the third thing.

Liv: Well, it’s almost like we have to redesign — oh yes. This idea — it sounded a little bit like what happened with the nuclear scientists going and checking each other’s homework, basically, to make sure there wouldn’t be a misfire. There’s this really interesting idea of sousveillance.

So take, for example: the world is increasingly going to be at risk from bio — whether you call it bioterrorism or an accidental lab leak from bio labs. Given the advances in AI and molecular biology, it’s going to become more and more likely that we’re going to have these deadly pathogens being synthesized by more and more people, and whether the release is accidental or intentional, this stuff is going to become more plausible. You can order the code for the smallpox virus just off the internet — just like that. And the ability to actually do something with that information, turn it into something deadly, and spread that pathogen around is also becoming easier and cheaper. So you need some kind of robust way of keeping an eye on, for example, the companies that create the printing machines that would enable you to synthesize these kinds of pathogens. But how do you do it without leaning into this draconian, top-down, one-centralized-entity-having-all-the-power-and-seeing-everything thing? It’s so hard to do.

But fortunately, we live in a world now where we have zero-knowledge-proof protocols, where someone can prove something without actually showing what the information is. And this is beyond my pay grade — how this would be integrated into a bio-lab surveillance thing — but basically, it’s possible to have this sort of bottom-up form of surveillance whereby everyone sort of pre-agrees: okay, this is a threshold that we agree not to cross. There will be the ability to do some kind of decentralized surveillance, but it only gets triggered when a certain number of people go, hey, we’re a little bit unclear about this. So it’s somewhat democratic, it’s decentralized, and most importantly, it only gets triggered when certain criteria are met. It’s not always recording, always going on.

And that’s a kind of somewhat win-win solution. You don’t want complete privacy invasion — you do want to have some secrecy; there are real geopolitical reasons why you don’t want to be broadcasting all of your technological developments. But at the same time, you do need the ability to keep potential bad actors or reckless actors in check.

I had a really interesting conversation about this with Allison Duettmann — who you guys should have on at some point — on my podcast, Win-Win. She was talking about surveillance and sousveillance — S-O-U-S-veillance. I think there’s something really promising there as well, as some kind of solution. Because it’s — how do we navigate this tension? Moloch is basically decentralization gone wrong, and Norma is centralization gone wrong.

Jordan: Right.

Liv: So how do we thread the needle between the two? I don’t know why, but I just feel like we can do it. We could figure it out.

My Dream Social Media Platform

[00:19:00]

Jordan: That’s awesome. I love that. And I feel optimistic, because in a lot of ways UpTrust is this anti-Moloch force. We’re like: we’re going to change the incentive landscape so that instead of getting rewarded for being rage bait or whatever, you’re rewarded for being trustworthy in a particular domain, and/or nuanced, and bridging. And we have our algorithm —

Liv: This makes me so happy. So are you going to be like a social media platform, or what’s your—?

Jordan: Yeah. As of today, there’s a new social media platform. We have about 10,000 people who’ve made accounts, which is exciting. You’re the closing speaker of our launch call.

Liv: Holy shit. I didn’t realize. Wow. Okay — I knew this was big, I knew it was cool, but you and I never actually got a chance — you reached out to me a while ago, like, Liv, I need to talk to you about this thing. It’s cool. You’re going to like it.

That’s so funny, because literally a week ago I started a document — and I actually, hang on, I have it open — my mad ramblings, which is called Dream Social Media Platform: someone needs to make this. So I kind of want you to say a bunch of words, and I want to see how many of them are in my list. Okay, I’ll read out what I’ve got. It should upweight: nuance, epistemic humility, thinking time before posting, and synthesis and steelmanning.

Jordan: Yeah — we do the synthesis and steelmanning, the nuance. We don’t have any thinking-time thingy. But what was the second one?

Liv: You know — some way of… because you need to have some kind of metrics that value the right kind of thing, and these seem like the kind of values and virtues of the posts that you want to uplift, right?

Jordan: Yeah, exactly. I fully agree. The epistemic humility thing we kind of do sideways. The synthesis thing is a big part of it directly — one way to think of it would be like a super-personalized, domain-specific version of community notes. Community notes is great, but it’s —

Liv: Very general.

Jordan: It’s very general. And it’s gameable, because you can earn credit. So it needs to be personalized — based on what’s a bridge for each individual user — and then also domain-specific. Someone who’s really good at doing notes for biochemistry is probably not that great at doing notes on the political situation. And the other thing we do for that — it’s funny, I’m tired, I’m like, wait, where’s my brain, where are my words — is the domain-specific weighted trust that’s user-specific. So you get some epistemic humility: you get the option to trust people who are epistemically humble. And it’s kind of yet to be seen if that’ll be a meme — we can eventually turn that on as a separate vector that you can track.

Liv: Super cool.

Very Rarely Is It Written by My Higher Mind

[00:22:00]

Liv: I was just thinking, with the thinking-time thing — when I think about when I am shit on social media, which is far more frequent than I would like it to be, it’s when I’ve seen something that happens to really personally trigger me. A particular topic that is just my rocket fuel. And I’ll see the post, and then I’ll respond.

Jordan: Yeah.

Liv: Those are the posts — I don’t always regret them; some of them are good, and they’re often funny or pithy — but they’re not written by my higher mind. Very rarely. But if it’s something where I read it, I feel something, but then I either don’t post, or I come back — if I feel strongly enough to come back and post two hours later or a day later — that’s going to be a really thoughtful post. I’m like, yes, I’m proud of this.

But of course, it’s the classic Moloch problem — who would build a social platform like that? Because people want that fire. They want to be able to immediately jump in and be in the melee. But in an ideal world, if I could start social media over again worldwide, that would be the first thing: you can’t immediately respond to something. There’s some kind of three-minute grace period between when you see a post and when you can respond. Like, a timer starts and you can come back — oh, you still want to do this? — and then you come back to it and write.

Because even at three minutes — you know, I played poker for a living, and the longer time you can have between the moment the shitty card comes, or your opponent makes a big bet into you, and deciding how you respond to it, the better your quality of decision will be, almost entirely. And that’s especially true in the world of rage bait. You have a moment to calm down and think about it. So I think that would be cool.

Jordan: I think you can work on it directly, but I think it’s also a little bit downstream of the whole incentive landscape. The way attention is rewarded in standard social media is recency and rage, right? And there’s no downside risk. It’s all benefit: if I piss a bunch of people off, I’m going to get a ton of engagement, and I’m going to get more attention. If I get canceled, I just got hired by the other side.

Liv: Right.

Jordan: So I think you get time-between-responding as a downstream side effect of rewarding a different thing — of what gets rewarded with people’s attention. And also of having every bid for someone’s attention be a bet with downside risk. So if I try to at you, or get in your DMs, or retweet — a retweet is like a free amplifier, and it shouldn’t be free. It should cost.

Absolutely No Downside to Spreading Bullshit

[00:25:00]

Liv: No, it shouldn’t be free. If you retweet something that later gets community noted, you shouldn’t get — like, there should be — I was literally talking about this with Igor the other day: there should be a penalty. Obviously, if you post something that gets community noted, you should have a big penalty.

Jordan: Yeah.

Liv: But if you retweet something, there should also be a penalty. You should get like a temporary dink in your algorithm, because otherwise there’s just absolutely no downside to spreading bullshit. Or just being, you know, a retweet monkey. There’s no downside — in fact, you’re directly incentivized to do it. And it’s silly. I think Elon was in the right direction — they were the first ones to add community notes, which is good. But it doesn’t go far enough. You need the negative incentive, not just an oh, you’re going to temporarily lose your ad revenue — you won’t get any payouts on that post. That’s not enough.

Jordan: It needs to be foundational, not just kind of an add-on. It needs to be baked into what gets decided where.

Liv: Another one, if I could — it’s just so funny, because I was literally thinking about this the other day: I think you should only get to tweet, let’s say as an example, once a day. You just get one tweet a day — you get one tweet and five responses, or something like that. There should be some scarcity there, because scarcity creates quality. I mean, A, it solves the bot problem — well, it doesn’t solve it, but it caps it, and that’s going to be increasingly important as everything just becomes AIs. But B, it again just forces people to be like: is this what I want to spend my online social currency on today? What do I want to spend it on?

I’m quite confident that the correlation between the amount of time spent on social media and bad mental health — negative mental health — is almost one for one. And similarly, people who post frequently — it’s one thing to me if you’re writing articles, long form, I’m all for that. But just tons of short-form commenting, posting stories, blah blah blah — when someone does that, I’m like, ugh.

Jordan: And the incentive trap is bad. I mean, Substack is a good example of this. I’m on Substack — I got on so that I could have a more visible presence to launch UpTrust, basically. And it’s beautiful, and I love it, and it’s great. And then they introduced their feed…

Liv: I know — they’ve done it. It’s just become the same shit. I was like, oh no, here we are again. I even caught myself literally scrolling shorts — it was like TikTok videos — and I was like, wait, what just happened? I deleted TikTok. I have every block under the sun to stop even going on Instagram when I don’t want to. And now Substack is doing this. Please. And it sucks, because that’s not even really their business model — they don’t have ads, right, on Substack?

Jordan: No, it’s all subscription revenue — they take a cut of that. But there’s something about it. So, early in the UpTrust days, we were like, well, we have this algorithm — this set of algorithms — do we license them? And I talked to a guy who runs — not a social media company, but a group company. We don’t have likes and hearts that factor into the algorithm; we have uptrusts on a particular domain, or downtrusts on a particular domain, auto-tagged by an LLM, basically. And he was like, man, I love this — I’d love to replace our like button with this immediately — and the leaderboards are trust leaderboards instead of how-much-engagement-did-you-get. He was like, I’ll do it right away. And I was like, cool — if it could take a couple engineers a month’s worth of time, and you have an SDK or whatever — what would it take to have you pay for it? And he was like: If it increases engagement.

And it’s such an interesting thing there, where I’m like — that would be great. But then I’m now in that exact same game, because the only metric I’m optimizing for is the thing that led us down this road in the first place.

Liv: Yes. Almost like Goodharting yourself or something, again.

Jordan: Exactly. Exactly.

Liv: Well, that’s exciting. I knew it was going to be cool, what you were doing, but I didn’t know it was exactly that.

It’s a Scream Room

[00:30:00]

Jordan: Oh yeah, we got a few comments. Somebody’s suggesting — this is actually a guy on my team, Joshua — an opposite timer: instead of waiting to respond, your response is on a delay, during which you can edit it and reflect. Like, you can hit send, but you’ve got to wait three minutes. Or maybe it pings you…

Liv: I mean, it’s better than the status quo, but I don’t think it’s sufficient. Because the whole point is that people see the thing — ah, I need to get it out, send. You kind of already have that ability: you can edit your tweet for an hour, and you can always delete something if you regret posting it. So there’s essentially already that mechanism. The point is, you literally can’t even start writing out your response. Or — you get like a scratchpad or something where you can write, if people need to get something out. Kind of like you can start drafting your essay as a response — it’s not an essay, it’s a short post, but — write down what you feel. Let people get their vent, get their emotions out. But it’s incoherent; it wouldn’t be an actual post.

Jordan: Yeah — like a scream room.

Liv: Yeah. Okay, it’s a scream room. Go scream. Go throw your teddies around. Okay — now, do you actually still want to say something after five minutes? What would you like to say? And now you can look at your incoherent words that you’ve put out and maybe thread something together. I think it’d be really, really interesting. But maybe it’s just hopelessly naive, because most people will just get frustrated and be like, no, I just want it.

Because that’s the thing you’re battling — you need the critical mass. The thing that’s so compelling about X is that despite all of its problems, it’s still the place you go when some shit’s going down in the world. It’s just that. And at the same time, there are so many people who are these sort of synthesis, nuanced — people craving high quality, just craving sensemaking. I don’t think — it’s going to be a long time for you to become the thing… never say never. But even if you’ve got, as you say, like 50,000 people who all believe in this and want this? That’s huge. That’s what I would — I’m like, oh, I want to actually hear what smart people have to say about this. So yeah — it’s super exciting. I love it.

Jordan: Thank you. Thank you. Yeah — the cold-start problem, the network effects — having that critical mass is really, really critical. And in your language of the coordination problem, there’s something interesting there. There are a lot of people who would love a better option. But if they’re building a business, and their business requires that they have people — my business is influencing, or my business is whatever — they have to be there. I mean, we paid Facebook for ads, and it’s kind of cool — there’s a Trojan-horse quality to it. For us, we’re like: yeah, if we can pay two dollars and we get a customer off of their platform onto ours — so, Meta Llama AI is not listening to me right now, but — yeah, it’s cool. But for most people it’s not that; they’re just straight up paying Facebook, and they can’t. So there’s something really, really interesting there. It’s a huge, huge hurdle, and we have yet to see if we can overcome it.

Liv: Yeah. Well, I’m certainly happy to — let me play around on it first. I don’t want to promise I’m going to post about it until I’ve gone and seen it. But I’m sure I will.

It’s Literally Negatively Rewarding Nuance

[00:34:00]

Jordan: A lot of it’s in the background — a lot of the cool ideas are still coming. I’m about to toot our own horn for a second, just because I have the opportunity. One of the cool ideas with bridges and synthesis: we can identify them in the graph based on voting patterns — things that you trust and things that you don’t, and then where those people agree — and we can find that kind of stuff. Find the next thing just outside of your comfort zone — what you’re most likely to trust or think is reasonable from other sides. That’s really, really powerful on its own. And another cool thing is that you can explicitly request that. So I could go on and be like: I think Liv’s going to uniquely be able to bridge the world of, I don’t know, poker players and DEI enthusiasts or something — I think you’re somehow capable of doing that.

Liv: That’ll be a tough one for me, I think, actually. That would definitely be level: hard. I thought you were going to pick two things that I’m like — yeah, I don’t know. I think I’m kind of like an anti-DEI person, so…

Jordan: Exactly.

Liv: Oh, I see. So wait — it’s like a challenge to me, to do something that is easy?

Jordan: It doesn’t matter — either way. I pre-bet my trust network — I basically bet views. And then anybody else can join in on that and say, yeah, I also pre-bet that. And you get on, and you’re like, what do I want to write about? Oh — I’m probably going to reach a million people if I write about this thing that wants a bridge. And if you write a shit post and people downvote it — because there’s risk — you lose their attention.

Liv: That’s awesome. Yeah. I mean, just anything that signal-boosts obvious good-faith attempts at synthesis and bridge-building — I think it’s totally fine if it’s kind of artificially done. That’s one of the good use cases of AI: it’s really good now at context analysis. I’m sure it’s fairly trivial now to have some kind of nuance score. And actually — I mean, again, everything is gameable. It always is. But right now we have nothing. It’s literally negatively rewarding nuance.

Jordan: Yeah, exactly.

Liv: So even just as a cat-and-mouse game: okay, great — you’re trying to game the nuance meter? Good for you.

Jordan: That’s great. That’s prosocial.

Liv: Maybe you’ll learn something along the way about how to be nuanced. Congratulations.

Jordan: Like DuPont — being the one that gets to claim, oh, we fixed it. Great. As long as we don’t have a hole in the ozone layer, I don’t care if DuPont made money.

Liv: Right. Right. I mean, there is something to — it would be sad if the word nuance becomes like a dirty word. Because, like everything — you know, I have to be careful not to overuse win-win. It’s my whole thing, right? Find the win-wins. Same kind of shit — sorry, I keep swearing. I hope that’s not a problem. I try my best; it’s been a long day. I’ve seen people using the phrase win-win to describe something that’s clearly not a win-win, and it’s like — ugh, stop it. That will happen. But yeah, I don’t know. No one’s even thinking about nuance anymore. It’s just completely gone out the window.

Jordan: Yeah. And it really sucks, because I think we need it. The kinds of problems we’re solving — like AI, for example — it just requires nuance. There’s no way around it.

Liv: Right. For sure.

Some Are More Molochian Than Others

[00:38:00]

Jordan: So I’m curious — you’re looking at this stuff, you’re looking at social media, you’re looking at AI, and you’ve talked to some of the big players. What keeps them from — what’s your analysis? Is it just structural? Is the mechanism just this big Moloch thing, and everyone has to be able to coordinate and come to the table? Are there psychological factors? Like, why aren’t we having the Yvon Chouinard, Patagonia version of AI? I don’t think Anthropic is that, by any means.

Liv: I mean, I think it depends. I think there’s a big delta between the different companies — some are more Molochian than others, let’s put it that way.

Jordan: For sure.

Liv: Some are kind of almost intentionally Molochian; some are accidentally. I personally think Anthropic is pretty good, by and large. What don’t you like about them? I’m curious.

Jordan: I mean, they’re my favorite. They’re the one I use, and I know people that work for them, and they’re all just deeply caring people who want to do good. But I don’t know — I guess… no, here I am going out in public talking shit, and I don’t want to, because I respect the willingness to try to do something. But it’s such a big gamble with the entire future. There’s something about we’re going to be an AI safety organization, but also we’re going to kind of enter the race that’s hard to trust. It’s hard to really be like —

Liv: And it can also — again, as you get closer to the ring of power, it can corrupt even with the best intentions. It almost got Frodo in the end, right? So no, I think there’s definitely truth to that. I think it’s worthwhile keeping an eye on all of the companies, because they’re all capable of falling into Norma — but, you know, perhaps the Anthropics maybe would be more vulnerable to that kind of thing. But I don’t know — it’s just so hard to say. Some are motivated by power, purely. Some are motivated by benevolence, some are motivated by curiosity, some are motivated by just pure fear. Some are just purely motivated by greed. I’ll leave it up to people to decide who’s who out of those. But there are some — as you say, there are some deeply good people, not just at Anthropic; deeply good people at most of the labs, in fact. I personally know some people at, you know, Google and so on, who I just think are excellent people. So it’s just difficult.

In terms of what they should do — again, we should be rewarding any of the times the CEOs do anything that’s cooperative with the others, that turns down the temperature, that de-escalates the situation. I’m thinking of making — actually, I’m not going to talk about that publicly yet. I’m going to do something. But anything that incentivizes and rewards cooperative behavior —

Jordan: I support you.

Liv: — win-win behavior, or even self-sacrificial behavior, frankly. I think it’s better if it can be a win-win, but sometimes you have to just give up something. Maybe you’re not going to be the guy — but you do it for the greater good, and a few people will know you did that, and that should be amazing. I think there are people, by the way, within a couple of these labs who are very capable of being that selfless. But still, it’s very easy to justify to yourself being reckless, or being I have to be the one who wins. That’s what most of the psychology is.

And this extends way beyond just the labs, as well. People forget — a lot of people downplay it, like, oh, they’re just saying we can’t lose to China. But it is true. We and China are very different in many ways — in some ways they’re similar to us, but they’re very, very different philosophies, and personally I don’t like a lot of their philosophies. It is truly a valid fear, and I think it’s a mistake for people to just completely dismiss that.

Jordan: Yeah, I agree.

Liv: You know — talk about Norma. Scary, scary Norma potential there. And again, there’s Norma potential in all of these, including America. Europe has certainly got a lot of Norma energy — or the EU, specifically.

An AI Aimed at Your Brainstem

[00:42:00]

Jordan: And there’s this funny way in which Moloch becomes Norma — when one tribe wins.

Liv: Right. Well, yeah. The sort of logical end state — assuming that everyone doesn’t die; there are a lot of everyone-dies outcomes if Moloch goes full ham, right, because it’s just negative-sum games. If it breaks the playing field, whatever that playing field is, then everyone dies, or just nothing works anymore and we’re back to sticks and stones, for example. But yes — the other sort of logical end state is that someone ends up with all the power, and in order to keep that power, there’s more and more draconian top-down control, and there’s no freedom, and you can’t criticize.

And especially as we’re getting towards — I mean, almost mind control, frankly. It’s barely science fiction — it’s becoming reality soon. Just with super-persuasion, even — again, there’s a reason why you can’t put your phone down. It’s an AI aimed at your brainstem, and you pick it up however many hundreds of times a day. Because the superintelligence that is the economy, frankly — the attention economy —

Jordan: Yeah.

Liv: — has you in its grasp, and you are a cell in it.

Jordan: Second shout-out today for the Light Phone.

Liv: Okay, so does this shit work? Because I’ve tried. It wasn’t a Light Phone — it was basically a dumb phone, some kind of Google phone, and it had all these things, but it was so slow I just gave up on it.

Jordan: It matters, the phone. I had an earlier version of this one, and it was so slow that I also gave up on it. This one has a hotspot, which matters, because sometimes I have to park downtown and the only option is to pay with the phone, so I have to hotspot my computer to it. And it’s got photos, which is awesome.

Liv: That’s huge. Yeah — that’s the thing. I want the nice things. Photos are not a problem. I want Google Maps — love it. Photos, being able to pay — certain smartphone functionalities. It’s just the general internet and the games.

Jordan: I don’t want anyone trying to use me via my phone, instead of having this be a tool that I use.

Liv: Exactly. Exactly. You just want agency. And they’re not giving us that.

Gollum Was a Surprising Solution

[00:45:00]

Jordan: By the way, somebody pointed out — uh oh. This is Pete, from the UpTrust team — that the ring of power did get Frodo in the end. It was only Gollum that actually successfully — spoiler alert, for anyone who hasn’t read it.

Liv: I mean, at this point… So — who’s Gollum?

Jordan: Yeah. That’s a good question, Pete. Who’s Gollum?

Liv: Maybe we shouldn’t say that out loud.

Jordan: Or — what’s Gollum?

Liv: Right. Or what is it, exactly? Gollum was the character that — he’s the archetype of having lost all his agency forever ago, and so on. A very tragic character, a very loathed character. And everyone’s like, oh, it’s a bad guy.

Jordan: Why don’t we just kill him? You remember when Frodo’s like, why, why — and gets pissed?

Liv: And then — he may be useful yet.

Jordan: Whatever.

Liv: Yeah. I mean, it’s so interesting. I don’t think we can go pinning all our hopes on the rings. But it’s definitely worth considering: what might be the surprising — you know, Gollum was a surprising solution, right? So, what could that be?

Jordan: Yeah. And particularly in the spirit of that whole why don’t we just kill him — and Gandalf’s like, what? No. There’s some drive to control this. In a way, you could call why keep Gollum alive? a version of Norma: can’t we just force this thing out of existence? And so yeah, there’s something about that impulse that we need to keep a really heavy eye on, even as we coordinate to —

Liv: Yes.

Jordan: — slow things down, and maybe have limits on GPUs, or whatever it is. I heard a colleague, Critch, talk about how each of the major CEOs of the AI labs should be the surveillant — in a kind of ring of watchers.

Liv: Oh, really? They should be like — in a sane world, they would all basically be talking to each other. Trust but verify. Or not even — again, we have zero-knowledge proofs. Exactly: you don’t have to trust that much, but you can do some kind of verify, and all agree: hey guys, if we all go above this certain amount… And I don’t think that’s out of the question, by the way. Most of them are still rational actors who don’t want to die — most, if not all of them. And that would be worthy of the Nobel Peace Prize, let’s put it that way — if someone can get them together and agree to these bare-minimum rules of engagement. Probably give them a nice little — like, punching bags where they can punch versions of each other or something like that.

Jordan: Exactly.

Liv: Rage, and be like — you can still keep competing; you’re not going to work together. But you can just agree on these bare-minimum things.

Jordan: Yeah. In fact, you get to enforce them on each other — that’s part of the incentive.

Liv: Exactly. No — yeah, exactly. It’s like, hell, you get to know if one of you is a defector. I mean, they’re kind of already trying to do that, right? They’re always accusing each other of defecting and calling each other out. But again, it’s been gamified. So yeah — it’s really tricky.

[Chat comment:] A suggestion that Trump is Gollum.

Liv: It’s definitely — definitely not not Gollum. Yeah, exactly. I can think of another couple of people who sometimes qualify. So, um — yeah. It’s definitely food for thought.

You Just Need to Have Attribution

[00:49:00]

Jordan: Oh man, this is —

Liv: Sorry — we have an econ question.

Jordan: Oh — wait, where? Oh.

Liv: Lisa said we have an econ question.

Jordan: Oh, nice. I missed it. Oh, I see. Yeah. What are your thoughts — actually, this is really for you:

[Chat question:] What are your thoughts on the future of AI and IP law from an economic perspective? For example, fair use doctrine protects freedom of expression by allowing a person to use a copyrighted work without permission in certain circumstances. But AI… dot dot dot.

Liv: From an economic perspective — I mean, this is not a thing that I by any means have any expertise in, but I’ve had conversations — Joseph Gordon-Levitt’s work on this is, I think, really interesting. Basically trying to create — you know, in movies you have residuals, right? If you’re a cinematographer, or a cameraman, or whoever — in principle, in theory; although apparently it’s a running joke that no one actually ever gets paid their residuals from Hollywood — in theory, you get residuals from the next fifty years or whatever of the movie that you worked on. A little piece of the pie.

And if we can develop the attribution methods whereby it’s like: okay, this much of the data — my work — went into the dataset… If you can track — let’s say, yeah, Miyazaki’s artistic style clearly goes into these datasets, because you can recreate Studio Ghibli. Someone goes and makes a Studio-Ghibli-esque thing — there’s probably some way of tracking what portion of that came from that style. Again, though, it seems like a pretty complex problem. But something directionally like that. And I have another friend — Bryn Mooser; his company, Asteria, is also trying to sort of do that kind of thing.

Jordan: This seems totally within our range as a species. We have blockchain, we have math for this sort of thing already developed. It’s not easy — it’s not a solved problem — but it feels doable.

Liv: Again, people always look at one side of the equation. They either think that artists are being babies — oh, you’re trying to restrict this technology, you’re trying to stop me from using it just so you can make money. But it’s like, no — these artists often are literally starving. The starving-artist meme exists for a reason, and now they’re literally being put out of a job by the thing that was trained off their hard work. Like — I get it. And the flip side: when you have this amazing, magical tool, people who didn’t have as much talent before, or the ability or time to learn how to draw or whatever, can now sort of dream up the story they always wanted to dream. And that’s cool. You have to feel for the everyman who didn’t ever get to do this kind of thing, who now gets to feel like a god for a minute, you know? And it’s nice that moviemaking and storytelling, in theory, is being more decentralized and democratized. So yeah — there should be. But you just need to have attribution. You need to have a way so that someone who has contributed…

You know, I’ve always thought that, generally speaking, there should be some kind of — I don’t think it’s necessarily correct… Okay, actually, I’m not going to go into this. I’m not going to end the conversation on this. But I’ll tell you — it’s my controversial tax viewpoint.

Jordan: Oh, interesting.

Liv: But directionally — I do think, if someone pays a hundred times more taxes than somebody else… it is difficult, but I think it should not be one-to-one; it shouldn’t scale linearly, but there’s probably some kind of quadratic thing or something whereby you still incentivize the person. Like, it should be easier, if you pay a ton of taxes, to immigrate to a place, for example — often it kind of feels like it’s almost the flip side. In theory, some countries do do that. But there are these middle grounds — some sort of 80/20 — which still incentivize the people who do actually contribute the most to something, and make them feel like they’re getting at least more of a thank-you, essentially. But I guess you could argue that — look, it’s a half-baked one, and I don’t want to end on it.

Jordan: I love it. I love it as a — I know we’re ending, and I think this is a great example of the kind of thing that UpTrust can foster, and that you’re going for with Win-Win — just, in general, the kind of conversations you’re encouraging people to have and turning people onto.

Liv: Mm-hmm.

Jordan: Wait — we don’t have to have it all figured out yet. We don’t have to go into camps of like, this or that. You can be like: look, there’s an 80/20 — look, I think there’s something here. I think we can change the incentives in some way that is still humanistic, that still honors income-disparity stuff. And we don’t want to just make it so that if you have more money, you have more of a vote. But let’s not polarize into those camps. Let’s find a way to hold both tensions and come up with something better as a result of it.

Liv: Yeah, exactly. Or at least — let’s stop demonizing. There are bad billionaires, don’t get me wrong. There are plenty of bad billionaires, and clearly there’s too much power for some people. But at the same time, there are a lot of billionaires who’ve done great things. We use their products every single day, and they’ve contributed way more to the tax pool than anybody else. And they’re getting told they’re evil, and having to hide — and it’s like, this makes no sense. We’re now just disincentivizing anyone from sticking their head above the parapet and trying to actually do good stuff. Because generally speaking, when you do good stuff, you also make money too — not always, but they’re correlated. So we just need to be able to — again, it’s a nuance thing. It’s sort of power balancing out. We want fairness, but we also want rewards. We want to incentivize those who really, really work hard, and who will work harder than others to do hard things.

Closing

[00:56:00]

Jordan: Yeah. Exactly. Awesome. Thank you. We could keep going — I love it, it’s so fun. And I’m so happy that you were able to make it.

Liv: Thanks for having me. Thanks for slotting me in. And I’m now excited to go on an UpTrust spiral.

Jordan: Woo-hoo! Awesome. Thank you. Enjoy it. Have a good night.

Liv: Thanks, everyone.

Show Notes & References

Essays & Writing

Meditations on Moloch — Scott Alexander (2014, Slate Star Codex) The essay Liv credits as her light bulb moment: it links the biblical demon Moloch to multipolar traps, coordination problems, and the tragedy of the commons.

Tim Urban — Wait But Why Liv references Urban’s Disneyland thinking — the pull toward stories with obvious good guys and bad guys. This framing appears in Urban’s book What’s Our Problem? (2023), where he describes tribal political Disney World narratives.

People

Scott Alexander Writer of Slate Star Codex (now Astral Codex Ten), author of Meditations on Moloch.

Allison Duettmann President and CEO of the Foresight Institute. Liv’s conversation with her about surveillance and sousveillance is on the Win-Win podcast (episode: In Search of the Third Attractor).

Yvon Chouinard Founder of Patagonia, invoked by Jordan as the model of a founder who structured his company against profit-maximizing incentives (Chouinard transferred Patagonia’s ownership to a climate-focused trust and nonprofit in 2022).

Andrew Critch (likely — Jordan says a colleague, Critch) AI safety researcher; the ring of watchers idea about AI lab CEOs mutually monitoring one another is attributed to him in conversation. Exact framing needs verification.

Joseph Gordon-Levitt Actor and founder of the collaborative platform HITRECORD; has publicly advocated for residuals-style compensation for artists whose work is used to train AI models.

Bryn Mooser Co-founder of Asteria, an AI film and animation studio built around fully licensed training data and artist attribution.

Igor (likely Igor Kurganov — Liv’s longtime partner and fellow poker professional; co-founder of Raising for Effective Giving.)

Liv Boeree Former poker champion (EPT and WSOP titles), science communicator, and host of the Win-Win podcast, which explores game theory, incentives, and positive-sum thinking. livboeree.com

Jordan Myska Allen Founder & CEO of UpTrust.

Key Concepts

Moloch / multipolar traps Named for the biblical demon associated with child sacrifice; via Scott Alexander, shorthand for dynamics where individually rational competition sacrifices what everyone actually values — negative-sum games, arms races, races to the bottom. Liv calls it the god of coordination failure.

Norma Liv’s own coinage: Moloch’s opposite failure mode. Where Moloch is decentralization gone wrong, Norma is centralization gone wrong — the demon of bureaucracy, nanny-stating, and totalitarian control (Professor Umbridge energy, referencing the Harry Potter character).

Goodhart’s Law When a measure becomes a target, it ceases to be a good measure. Liv’s warning about optimizing a trust platform for engagement.

The Montreal Protocol (1987) The international agreement that phased out ozone-depleting CFCs — Liv’s primary case study of a solved global coordination problem. DuPont was among the major CFC producers that shifted to alternative refrigerants.

Nuclear arms-reduction treaties Liv (hedging herself in the moment) refers to the New START treaty alongside the fall of the Berlin Wall; the late-1980s-era agreements were the INF Treaty (1987) and START I (signed 1991); New START itself dates to 2010. The verification mechanism she describes — inspecting safety protocols without revealing capabilities — matches the mutual-inspection provisions of that era.

Zero-knowledge proofs Cryptographic protocols that let someone prove a statement is true without revealing the underlying information — raised as a building block for verification without high trust, between bio labs or AI labs alike.

Sousveillance Watching from below — decentralized, bottom-up monitoring, as opposed to top-down surveillance (term coined by Steve Mann). Discussed as a possible thread-the-needle between Moloch and Norma.

Community Notes (X) The crowd-sourced fact-checking feature both speakers treat as directionally right but insufficient without real penalties.

The Light Phone A minimalist phone designed to be used as little as possible.

Studio Ghibli / Hayao Miyazaki Invoked in the discussion of AI training-data attribution — tracking how much of a generated work derives from a recognizable artistic style.

Gollum (The Lord of the Rings, J.R.R. Tolkien) The conversation’s running metaphor: the loathed, agency-stripped character everyone wanted to kill, who turned out to be essential to destroying the ring — a caution against the impulse to force problems out of existence.

UpTrust is a trust infrastructure platform for the internet. Find it at uptrusting.com.