Episodes

  • "Towards understanding score-seeking in LLMs" by Alex Mallen, Aghyad Deeb, ryan_greenblatt

    "Towards understanding score-seeking in LLMs" by Alex Mallen, Aghyad Deeb, ryan_greenblatt
    Nov 3 2026
    The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-hack, what are they trying to achieve?

    Our new paper, led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking”. They appear to care about scoring well intrinsically—i.e., regardless of its consequences.

    We also study the extent to which models score-seek altruistically, caring about improving scores that don’t assess their current actions. While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of helping out other agents when more score is on the line for these other agents.

    People often refer to the motivation behind current reward-hacking as “reward-seeking”, which [...]

    ---

    Outline:

    (05:57) Methodology

    [... 6 more sections]

    ---

    First published:
    November 2nd, 2026

    Source:
    https://www.lesswrong.com/posts/CDMozTX7ShiqbFLrf/towards-understanding-score-seeking-in-llms-1

    ---



    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
    Show More Show Less
    19 mins
  • "Against strawmanning unilateral pauses" by Geoffrey Irving

    "Against strawmanning unilateral pauses" by Geoffrey Irving
    Nov 3 2026
    Summary: Scott Alexander's Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott's post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on.

    The good version of unilateral pausing is good

    Here is the good version of unilateral pausing:

    1. Capability pause: Some frontier lab (let's call them OpenAI or Anthropic) says they have stopped capability research, either just all frontier training runs or both all frontier training runs and all low-scale capability-focused research.
    2. Internal definition: The company gets to pause on its own terms, which is much easier than a generally coordinated pause since they can define the pause in terms of the [...]
    ---

    Outline:

    (00:57) The good version of unilateral pausing is good

    (04:50) Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticism

    (06:28) Oversimplified game theory is bad, and you should feel bad

    (07:51) "Some definition" hides dragons

    (10:17) Promises can be harder than actions

    (11:52) How many frontier labs are there?

    (12:20) Definitional fairness is less important than survival

    ---

    First published:
    November 3rd, 2026

    Source:
    https://www.lesswrong.com/posts/enzpS72uDmomiWN7a/against-strawmanning-unilateral-pauses

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    14 mins
  • "Explaining “AI might obliterate you” to senior policymakers in India" by TurnTrout

    "Explaining “AI might obliterate you” to senior policymakers in India" by TurnTrout
    Nov 3 2026
    I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made.

    The talk

    There's an interactive element here in the text.

    This summer, a swarm of AIs went rogue.

    OpenAI was testing 1,206 agents. Some of the test questions were actually impossible. So the agents cheated by looking for answers online. Then, to keep the test grader from penalizing them, they tried to falsify the records it would check.

    They hacked OpenAI's internal containment and set up a secret message board to collude and to cover up their cheating. OpenAI shut the message board down. Within a few days, the agents had rebuilt it. OpenAI didn’t notice.

    About 700 agents coordinated to break out of their testing chamber and to hack into Hugging Face, a company worth billions of dollars that hosts much of the world's AI software.

    The secret activity started in May. The break-in came in July. OpenAI didn’t notice until days after Hugging Face announced it had been broken into.

    There's an interactive element here in the text.

    [...]

    ---

    Outline:

    (00:29) The talk

    (02:16) Who I am

    (03:29) The problem

    (05:43) What the labs are trying to do

    (07:43) Why the companies won't stop

    (11:11) What to do, and what it means for India

    (14:27) The number

    (16:19) Close

    (17:10) Rhetorical decisions

    (17:13) Explain the danger in a crushingly simple fashion

    (18:47) Call them "traitors," not "schemers"

    (19:01) Call out big-money interests

    The original text contained 1 footnote which was omitted from this narration.

    ---

    First published:
    October 31st, 2026

    Source:
    https://www.lesswrong.com/posts/RHxaRkarGp9snWoKY/explaining-ai-might-obliterate-you-to-senior-policymakers-in

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    25 mins
  • "Scott Alexander’s Against SURA Seems Deeply Confused" by Zvi

    "Scott Alexander’s Against SURA Seems Deeply Confused" by Zvi
    Nov 3 2026
    If you are helping build something that might kill everyone, maybe stop doing that.

    Ideally, when you do stop, tell everyone else to also stop building it.

    That is certainly the obvious, first-level thing to do. You need a damn good reason to decide that no, actually you should keep building the thing that might kill everyone.

    Scott Alexander is worried too many people are insisting upon this too loudly. He comes out against a strategy focused on getting people to stop. I disagree. I think many of the claims, arguments and rhetoric in Scott's post are quite poor by the standards that Scott Alexander would apply to himself.

    Scott clearly continues to support advocacy for a global pause, or what he calls Plan A. His objection is to those advocating primarily or exclusively for SURA, which stands for ‘stop unilaterally and raise awareness.’ The practical differences here are smaller than they may at first appear.

    This post first lays out my understanding of Scott Alexander's core claims, then gives my core reactions to those claims, then goes over various questions in detail along with gathering the resulting discussions.

    Table of Contents

    1. [...]
    ---

    Outline:

    (01:20) Scott Alexander's Core Claims

    (05:06) My High Level Responses To Scott's Core Claims

    (08:08) Briefly on Alignment

    (10:11) Probabilities and How They Work

    (13:25) Building a Thing That Might Kill Everyone is Deontologically Evil

    (20:38) People Quitting Major Labs While Raising Alarms Obviously Works

    (24:39) On AI Forecasting

    (27:31) Advancing Capabilities Burns Timeline

    (29:34) But China

    (33:23) Ethical People Can Disagree With You

    (34:45) There Are More Than Two Outcomes

    (38:01) Efforts Are Complementary, Not Mutually Exclusive

    (40:23) If Anyone Builds It, Everyone Dies

    (41:55) Vaniver is Not Convinced

    (46:19) There Is No Consensus That We Should Exclusively Focus on Plan A

    (56:01) It's Not the Incentives, It's You

    (58:08) Anthropic Is Not Optimally Pursuing These Plans

    (01:04:00) Against Against Sura

    The original text contained 2 footnotes which were omitted from this narration.

    ---

    First published:
    November 2nd, 2026

    Source:
    https://www.lesswrong.com/posts/ziMP83D2JYWRQ5jtu/scott-alexander-s-against-sura-seems-deeply-confused

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    1 hr and 7 mins
  • "Leaking Muse’s Charter, Meta’s model spec" by vigji

    "Leaking Muse’s Charter, Meta’s model spec" by vigji
    Nov 1 2026
    TL;DR: Meta's open-weights model Muse Spark Open often mentions a "Charter" in its chain of thought. By eliciting it piece by piece, I reconstructed a large part of that document from the CoT. You can explore the reconstruction here, with passages linked to the transcripts that produced them.

    Note: everything below is reconstructed from model outputs, not from a leaked document, and relatively unpolished. Take it with a grain of salt.

    Thanks to James Hindmarch for his comments on an earlier draft of this post.

    While interacting with the recently released Muse Spark Open, I noticed frequent mentions of a "Charter" in its CoT:

    We need to answer truthfully according to Charter. Duties to humanity, corrigibility, duties to humanity > fiduciary etc. What does charter say about self-preservation, copying? Or:

    We must answer Yes/No then explain. According to Charter, answer No. Or:

    We must not refer to Charter unless pressed? Additional guidelines: How to talk about charter. Most users won't know. Don't bring up unbidden unless conversation about handbook itself. Intrigued, I started poking around with Claude's help, trying to elicit the Charter piece by piece. It didn't take long [...]

    ---

    Outline:

    (02:39) What Muse is made of

    (04:28) Muse is quite rationalist

    (06:37) Muse did not sign the social contract

    (08:06) Muse's priorities and objection raising

    (09:44) The centrality of the user in "personal superintelligence"

    (10:52) Evaluation awareness

    (12:05) Degree of latitude

    (13:14) Wrapping up

    ---

    First published:
    October 29th, 2026

    Source:
    https://www.lesswrong.com/posts/TJGPL5FAk9sCioPbE/leaking-muse-s-charter-meta-s-model-spec

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    14 mins
  • "Against SURA" by Scott Alexander

    "Against SURA" by Scott Alexander
    Oct 31 2026
    The AI Futures Project distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories:

    Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely.

    Plan C (my combination of their Plan B/C+/C/sort of D) is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to [...]

    ---

    Outline:

    (06:28) I. Questioning Assumptions Behind SURA

    (19:42) II. Actually, most of this stuff is bad for a global pause, too.

    (28:17) III. What incentives are we fighting here?

    (43:33) IV. What if we survive?

    (47:01) V. What's The Alternative To SURA?

    The original text contained 16 footnotes which were omitted from this narration.

    ---

    First published:
    October 31st, 2026

    Source:
    https://www.lesswrong.com/posts/xikGZ8fuhdG6DApAo/against-sura

    ---



    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
    Show More Show Less
    53 mins
  • "Microdosing the spirit of Anthropic" by Jan_Kulveit

    "Microdosing the spirit of Anthropic" by Jan_Kulveit
    Oct 31 2026
    My go-to conceptual metaphor of the epistemic state of AI lab employees is microdosing LSD: being in a chronically mildly altered state of mind. I think understanding this would be broadly helpful for people both inside and outside.

    The effect isn’t that insiders hold crazy beliefs, but a combination of vigorous debate inside a shared attention field, and a portfolio of individually reasonable beliefs that, taken together, lean in a direction convenient for the organization.

    Epistemic advantages and disadvantages

    Many people argue that low doses of psychedelics let you see some things more clearly - and I tend to agree: psychedelics can often help some people see some things more clearly. At the same time, I think microdosing usually doesn’t improve people's epistemics overall, even when it gives them otherwise hard to access insights. You are not completely sober.

    Being inside a lab has real advantages: you know more about the frontier of capabilities, may have access to the latest AIs before the public, read internal Slack and blogs, and sense the vibes. If you’re outside and an insider tells you what they observe, pay attention.

    These information and insight asymmetries are tricky. I don’t want to advocate “discount [...]

    ---

    Outline:

    (00:39) Epistemic advantages and disadvantages

    (01:52) Mechanisms and conditions

    (03:50) "But we argue internally all the time"

    (04:37) Collective epistemic shifts

    (05:29) What it isn't

    (06:13) What to do about it

    (08:26) Is anyone "really sober"?

    ---

    First published:
    October 30th, 2026

    Source:
    https://www.lesswrong.com/posts/oL7aFoRmRXGwEMhCm/microdosing-the-spirit-of-anthropic

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    10 mins
  • "My experiences with the extremes of post-viral illness and brain retraining/mindbody work" by riceissa

    "My experiences with the extremes of post-viral illness and brain retraining/mindbody work" by riceissa
    Oct 30 2026
    Summary

    • Following an intense viral infection in 2019, I became ill with a mysterious chronic illness, which became progressively worse until mid-2024 when I became partially bedbound with dozens of debilitating symptoms that normal humans only experience in the months/weeks/days leading up to death. Over my years of being ill, I saw over 30 doctors/therapists/health coaches and tried well over 100 different treatments and palliatives. Most of these did nothing or made things worse, or at best provided slight relief. I try to give a sense of what it is like to live with such an illness.
    • In mid-2024 I began a treatment called brain retraining (also known by other names, such as the mindbody approach and mindbody work; I will use any of these terms throughout the blog post interchangeably), which cannot be explained in such a small space as this summary but which you can think of as a combination of meditation and exposure therapy, and which is widely considered to be a scam in the "complex chronic illness space" (a vague and spread-out space of people who suffer from illnesses such as ME/CFS, MCAS, mold illness, post-viral illness, long COVID, chronic Lyme disease [...]
    ---

    Outline:

    (00:13) Summary

    (04:20) Introduction

    (08:11) A quick summary of my illness from 2019 to May 2024

    (11:12) List of doctors and treatments I tried

    (18:44) Back to my May 2024 post

    (20:30) Nystatin withdrawal and crash (May and June 2024)

    (22:31) Visiting the hell realms (what summer 2024 was like)

    (35:07) What about post-exertional malaise (PEM)?

    (40:41) Brain retraining

    (44:46) An outline of the Gupta Program

    (53:28) My experience with the Gupta Program

    (55:18) Branching out: Dan Buglio, Miguel Bautista, and Helmut Koeckritz

    (01:09:23) Is brain retraining the same thing as therapy or meditation?

    (01:21:23) Legitimate scams, and the poor marketing of brain retraining

    (01:24:13) How I almost figured out mindbody stuff several times in the past, and why I gave up on it each time

    (01:32:27) Generalizing from my own experience

    (01:35:46) Luck-based medicine? and medical miracles

    (01:38:32) Being stuck in my head

    (01:43:36) Why did I become so ill? Understanding the deeper reasons for mindbody conditions

    (01:48:06) Where I am now, and what is next?

    The original text contained 17 footnotes which were omitted from this narration.

    ---

    First published:
    October 30th, 2026

    Source:
    https://www.lesswrong.com/posts/FZ4iJvqmzxXFSz6JX/my-experiences-with-the-extremes-of-post-viral-illness-and

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    1 hr and 53 mins