• “the case for CoT unfaithfulness is overstated” by nostalgebraist

  • Sep 30 2024
  • Length: 22 mins
  • Podcast

“the case for CoT unfaithfulness is overstated” by nostalgebraist

Listen for free

View show details

  • Summary

  • [Meta note: quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know]

    In research discussions about LLMs, I often pick up a vibe of casual, generalized skepticism about model-generated CoT (chain-of-thought) explanations.

    CoTs (people say) are not trustworthy in general. They don't always reflect what the model is "actually" thinking or how it has "actually" solved a given problem.

    This claim is true as far as it goes. But people sometimes act like it goes much further than (IMO) it really does.

    Sometimes it seems to license an attitude of "oh, it's no use reading what the model says in the CoT, you're a chump if you trust that stuff." Or, more insidiously, a failure to even ask the question "what, if anything, can we learn about the model's reasoning process by reading the [...]

    The original text contained 1 footnote which was omitted from this narration.

    ---

    First published:
    September 29th, 2024

    Source:
    https://www.lesswrong.com/posts/HQyWGE2BummDCc2Cx/the-case-for-cot-unfaithfulness-is-overstated

    ---

    Narrated by TYPE III AUDIO.
    Show More Show Less
Philosophy Social Sciences
Show More Show Less
activate_samplebutton_t1

What listeners say about “the case for CoT unfaithfulness is overstated” by nostalgebraist

Average Customer Ratings

Reviews - Please select the tabs below to change the source of reviews.

Audible.com.au reviews

Amazon Reviews
No Reviews are Available
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.