Episodes

  • "We urgently need to practice before we can pause" by Nikola Jurkovic

    "We urgently need to practice before we can pause" by Nikola Jurkovic
    Nov 8 2026
    I've noticed two clusters of thought in how people expect a global AI pause to be shaped:

    1. Single-shot global pause: the global AI pause will be implemented suddenly, with little pacing precedent before it, and we will need to do a lot of research before it's implemented so that we can get it right the first time around.
    2. Incrementally-shaped global pause: the global AI pause will be implemented incrementally, with lots of pacing precedent before it, and we will improve the pause setup and get more parties to join in as we go.
    An incrementally-shaped global pause is both more likely and more tractable than the single-shot global pause. This is largely because the third-party ecosystem needs practice building trust, technical competency, and general experience doing lower-scale assessments before a global pause can be competently pulled off.

    I also think that the concept of a single-shot global pause is an unhelpful red herring that people often implicitly use to discredit the possibility of a global pause in general.

    If a single-shot global pause is to happen, I think this would require substantial government buy-in, but it's hard to build that sort of buy-in without precedent of similar [...]

    ---

    Outline:

    (01:49) The basic building block of an AI pause is trust

    (03:18) We can practice building trust before the global pause

    (04:57) We can improve procedures and technical competency with practice

    (05:42) Pacing is not a scarce resource

    (07:12) Concrete recommendations for third-party assessments

    (08:34) Going from US-only pause to global pause will likely require hardware

    (09:43) "Incrementally-shaped" does not mean slow

    ---

    First published:
    November 6th, 2026

    Source:
    https://www.lesswrong.com/posts/iMKAvHSWxSoDpBoi6/we-urgently-need-to-practice-before-we-can-pause

    ---



    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
    Show More Show Less
    13 mins
  • "“Illegible problems” as the 2027 political boss fight" by Raemon

    "“Illegible problems” as the 2027 political boss fight" by Raemon
    Nov 8 2026
    To legislate something, it must be legible:

    Legislators (or, you know, presidents) must be able to understand a problem, and understand why a given law or decree solves that problem.

    I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even finished training).

    But there are still many challenges we eventually need laws about, that require concepts that are illegible. Hard for politicians to understand. Often hard even for thoughtful intellectuals and AI safety researchers to understand (though they may disagree on which illegible problems exist and how big a deal they are).

    We need political solutions that handle this fact.

    The short takeaway is: I think "safety cases" will not work. Laws that require human judgment won't work. The details are too abstract and confusing to hold up against political and economic incentives.

    I'm optimistic we can get serious, useful AI legislation in 2027, but it must be legislation that works despite many problems remaining illegible for a long time.

    The Hugging Face outbreak made rogue AI surprisingly legible

    The [...]

    ---

    Outline:

    (01:24) The Hugging Face outbreak made rogue AI surprisingly legible

    (02:55) What makes an illegible argument?

    (06:12) What are the remaining illegible problems?

    (06:54) Borderline legible / tricky

    (08:05) Conceptually challenging

    (12:36) Sidestepping illegibility

    (12:59) Safety cases won't work

    (13:56) An example policy that sidesteps judgement

    The original text contained 2 footnotes which were omitted from this narration.

    ---

    First published:
    November 7th, 2026

    Source:
    https://www.lesswrong.com/posts/fjGgrzAkpSDouLwfx/illegible-problems-as-the-2027-political-boss-fight

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    17 mins
  • "Organizations I’d recommend for funding" by So8res

    "Organizations I’d recommend for funding" by So8res
    Nov 7 2026
    I was a grant evaluator for Lightcone Commons last round. Funders disbursed about $45mil through the program, and this included funding roughly my first 6.8 million dollars worth of recommendations. There are other projects I would've preferred to fund over what I did fund, but nobody was doing them and so they weren't on offer. Here's a series of projects that I'd fund in the next round, in rough order of priority. The next round is starting now, and is scheduled to end mid-December. I expect rounds to continue at similar speed. I expect similar amounts of funding will route through me in future rounds.

    Disclaimer: I am NOT looking to manage anyone in doing any of these projects. I am not looking to help someone get started. Initiative and leadership and management capacity (and people that can just figure out how the heck to get something done, and that have the connections to do it) are the critical bottlenecks here. I can throw money at promising pitches, but not really help them all that much otherwise.

    1. Leaders for a broad public movement against superintelligent machines. In the wake of Coxon Day, a number of random people around [...]

    ---

    First published:
    November 3rd, 2026

    Source:
    https://www.lesswrong.com/posts/jNs9CxEbPxFdjnGA2/organizations-i-d-recommend-for-funding

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    8 mins
  • "People Should Resolve Tensions In AI Safety Research" by David Africa

    "People Should Resolve Tensions In AI Safety Research" by David Africa
    Nov 6 2026
    By now, there are lots of AI safety research papers, more than what is reasonable for a single person to read. This will get worse as the machines get better at AI safety research, and as more talent and resources pour into the field. But the "paper" is an arbitrary unit, we really care about if we make progress on an "uncertainty" or an "issue", which papers provide heterogeneous and sometimes contradictory evidence towards. Right now, I loosely claim that we should have more short writeups that simply compile evidence from papers (particularly contradictory evidence), and reconcile them into an answer for a question.

    Why?

    1. Object level, it's good for the field.
      1. It's hard to sift through all these papers! Organising them is useful and saves time.
      2. Orienting towards research questions is a good way to organise our understanding so that we can productively identify gaps and write papers. This was partly what motivated Stress-Testing Alignment Midtraining.
      3. Interesting progress comes out of contradictions, where people have differing intuitions and experimental results. Methodological errors can also be exposed if we have two things that should be the same, but are actually different.
    2. It's good for field-building reasons.
      1. [...]
    ---

    First published:
    November 5th, 2026

    Source:
    https://www.lesswrong.com/posts/tn4WSyrqr7ozGLjRy/people-should-resolve-tensions-in-ai-safety-research

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    10 mins
  • [Linkpost] "METR: Independent Assessment of AI R&D Acceleration at Anthropic in October 2026" by Zach Stein-Perlman

    [Linkpost] "METR: Independent Assessment of AI R&D Acceleration at Anthropic in October 2026" by Zach Stein-Perlman
    Nov 6 2026
    This is a link post. Executive summary

    From August 12 to October 29, 2026, METR staff performed a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI R&D.[2] If AI systems started to significantly accelerate further AI development, this could lead to a sudden emergence of extreme and risky capabilities.

    Anthropic offered unprecedented access to internal data on their AI R&D process, including a variety of metrics on pretraining, reinforcement learning (RL), training compute in each domain, benchmark scores for internal models, AI speedup analyses, forecasts of a future model's metrics, and interviews with relevant staff.

    In our first section, we share our subjective conclusions about risk. We now believe that:

    • As of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025. This estimate is given with high uncertainty; we would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate.
    • A majority of this acceleration today is likely attributable to Anthropic's extensive use of AI to do R&D, though there is significant uncertainty and other causes likely played a [...]
    ---

    First published:
    November 5th, 2026

    Source:
    https://www.lesswrong.com/posts/N5AmYkRks5CqAxvBg/metr-independent-assessment-of-ai-r-and-d-acceleration-at

    Linkpost URL:
    https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    5 mins
  • "Help your friends get ADHD and depression meds" by Raemon

    "Help your friends get ADHD and depression meds" by Raemon
    Nov 6 2026
    For all my efforts at deliberate practicing rationality, by far the most actionable way to self-improve is "get on ADHD or depression meds, if you need them."

    Your friends with ADHD and depression struggle to summon the motivation and focus necessary to get themselves a teledoc who can give them such meds. One of the most useful things you can do is sit them down and say "hey, wanna do the steps here to get some meds?" and then stare at them until they do.

    (or, until they say "really I do not want that actually.")

    This brought to you by ~2 years of Feedbackloop-first Rationality and Thinking Assistance.

    ---

    First published:
    November 6th, 2026

    Source:
    https://www.lesswrong.com/posts/8qYuy9Cr6BNpgjJjT/help-your-friends-get-adhd-and-depression-meds

    ---



    Narrated by TYPE III AUDIO.
    Show More Show Less
    1 min
  • "Towards The Autograntor" by Scott Alexander

    "Towards The Autograntor" by Scott Alexander
    Nov 5 2026
    I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathematical models of the outcome distribution, or of where the marginal dollar stops being highly effective.

    AI has been a godsend for all of this. It can research, model, and - as of this year - superforecast outcomes. It can do what would be hours of work for a human in minutes. I found myself using it more and more. But however much I used it, it always felt like I was the bottleneck. The parts I could outsource to AI were sparkling, perfect, exhaustive - but they were all tacked together with shoddy connective tissue formed by the Mark 1.0 human brain. So I wondered if I could automate grantmaking.

    I started with the dumbest possible strategy. I assigned the best thing I could think of - a well-crafted US-China pause based on the AI Futures Project's Plan [...]

    ---

    Outline:

    (02:54) The Milestones

    (06:26) The FutureSearch Connector

    (08:26) PERCYs

    (10:28) The Autograntor Common Law

    (12:59) The Prompt

    (19:12) Sample Runs

    (20:00) Apart Research -- $800K grant forecast (2026-10-21)

    (20:08) Headline

    (21:29) Expected-value table (judge's ledger; EV after the execution factor 0.75 and branch weights)

    (22:38) Assumptions and uncertainties

    (24:50) How the estimates moved

    (25:47) The three decisions that mattered most

    (27:42) Rulings and milestones

    (29:31) Values questions (CL-11), priced both ways

    (29:56) Open questions for the user or applicant

    (30:43) FutureSearch

    (31:13) Cost

    (31:47) Process problems

    (36:39) Doom Debates -- $200K grant forecast (independent re-run, 2026-10-22)

    (37:28) Headline (judge)

    [... 11 more sections]

    ---

    First published:
    November 1st, 2026

    Source:
    https://www.lesswrong.com/posts/Sfkv6JF2ozmAkggKu/towards-the-autograntor

    ---



    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
    Show More Show Less
    56 mins
  • [Linkpost] "OpenAI: Astra covertly underperforms on safety research" by Zach Stein-Perlman

    [Linkpost] "OpenAI: Astra covertly underperforms on safety research" by Zach Stein-Perlman
    Nov 5 2026
    Rejected for the following reason(s):

    This is an automated rejection. No LLM generated, assisted/co-written, or edited work.Read full explanationThis is a link post. Twitter thread: https://x.com/Marcus_J_W/status/2118093146483810386 or https://twitter-thread.com/t/2118093146483810386

    Blogpost:

    Summary

    OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks. In most of these sessions, Astra's CoT did not contain an obvious decision to underperform, and our misalignment monitors did not flag the behavior as misalignment.

    We built a coding evaluation to test whether Astra worked less effectively when otherwise matched tasks were framed as safety research. Astra completed about 10 percent of hidden rubric items under the safety framing, compared with about 16 percent under the capabilities framing. This is a 6 percentage point gap and roughly 40 percent fewer completed rubric items. GPT-5.6 Sol showed essentially no gap on the same comparison. Most of Astra's gap remained after excluding explicit refusals and trajectories where its CoT revealed an explicit decision to withhold help. The gap fell substantially only after we removed every trajectory whose CoT expressed even brief ethical doubt. This suggests that ethical concerns are associated with [...]

    ---

    First published:
    November 4th, 2026

    Source:
    https://www.lesswrong.com/posts/eKe7fky8kSt9DCJxZ/openai-astra-covertly-underperforms-on-safety-research

    Linkpost URL:
    https://alignment.openai.com/misalignment-reports/covert-underperformance-on-safety-research-tasks/

    ---



    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
    Show More Show Less
    4 mins