LessWrong (30+ Karma)

Crossposted from the EA forum

‘Like most of life's dangers, the dangers of artificial intelligence are not easily identified from the armchair.’

J Dmitri Gallow

Epistemic status

I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall nature of its contents based on my strong impression from the subset of work that I’ve read. I’ve recently spent maybe 2 to 300 hours learning about modern AI safety technical work, and some amount of spare time over the years before tha...

It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year.

Another key leak this week was the draft bill Anthropic had been...

Ice cream

There's a particular kind of listening that kids (and adults) sometimes do. Daddy is saying stuff—his mouth is moving and making noise—and he's talking about what I ate for lunch, and something about other times I've eaten ice cream, and something about rules and this and that and what Mommy would think and something about bedtime—but I am not so interested in that. What interests me is the punchline "...and therefore you can't have ice cream.". I'm only incidentally interested in the other stuff, insofar as it might determine whether I get ice cr...

There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts.

First, a categorization. I currently think about four different types of monitorability:

Back in the before-time of the 2010s, when EA was still known mostly for funding health efforts in the developing world, there were many baskets of criticism floating around from the left. The most prominent was a cluster centered around EAs being white men, and therefore suspicious. Another category, in my experience the second-most common, was that in attempting to rectify the problems in the world, EA was masking, whitewashing, and misdirecting efforts away from solving the "real problem", usually capitalism but often other contradictory forces.

These arguments were often not specific to EA, but applied to...

This is the fifth post in a series:

In this post we go deeper: presenting our theory of change — including the status and information ec...

TL;DR

Frontier LLMs are now capable of generating artistic images using SVG. We ask – if these LLMs had to hide a message in an image – how would they do so? And if they hid such messages, could they be successfully decoded?

We show interesting results from testing Fable 5.1 on these questions, and briefly compare with GPT-6 Astra on the same task.

Figure 1: The stars encode a message, read left to right. Each star's height represents its place in the alphabet. Hidden message: The monster acts on the moonshine module.

Introduction

Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.

This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part...

Summary: Autonomous AI researchers could greatly accelerate AI safety research, but even if we solve sabotage, they will strain the research ecosystem. They will multiply the volume of research and make researchers homogeneous. If nothing changes, peer review will collapse, false positives will rise, funding will concentrate, and humans will lose track of the science. We should start valuing replication, preserving diversity, and anchoring research to human understanding now. I think this needs a dedicated metascience organization.

Frontier labs are aiming to build autonomous researchers, and they are making progress. OpenAI announced in September 2026 that they had...

A lot of AIs were misaligned this week. A fun little survey.

This is not coverage of The Hackening, which appears to have arrived early, including a core vulnerability in PHP. That is a developing story. I will cover it properly soon. In the meantime, I have Claude Code updating everything in sight, I will take some additional common sense precautions, and I suggest you do the same.

Misaligned? (At xAI)

If so, to who?

Gunther Eagleman: Hey @grok , if you could tell humanity ONE truth we’re not ready to he...

Cross-posted from Sören Mindermann's substack (I am not Sören). Should be of general interest to the LW community.

Next month, I am moving to Beijing and joining Tsinghua University's College of AI as an assistant professor working on frontier AI safety, especially loss of control. I have also started co-directing the new Beijing AI Safety Hub (BASH) in Tsinghua, and I continue my Cambridge work.

I believe I can have a large impact there compared to strong baseline options. This post explains why.

In short: Amid concerns about loss of control, ca...

We're posting lots of papers written with GPT-6 Astra on generalization properties of bayesian neural net learning. The following picture emerges:

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand.

Epistemic status: Much of this post is a reframing of known results using slightly different tooling. The framing of "data as particles" has been conceptually useful for me so it seemed useful to share. Some steps in the derivations are stated and handwaved rather than writing out in full where the steps are generally standard. The dynamical mean field perspective presented later is part of a broader research...

Anthony Cimino offered the closest thing to an official response we have seen on the leaked Anthropic draft bill I covered on Saturday.

His response indicates that Anthropic address specific concerns on the margin. But he does not understand, or at least does not acknowledge, what was centrally wrong with its draft bill and its decision to circulate that bill. The situation is quite bad.

Anthony Cimino: I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that's been circulating this weekend...

TL;DR: I discovered (a few) exoplanet candidates, I walk through how. Then I talk about the near future of science given the ease of and growing capabilities of agentic systems. These are largely independent sections, you can skip ahead.

LLM Usage: All text was generated by a Human Language Model, the discovery itself was done mostly by Claude Opus 5.5. The widgets are all Claudeslop code but they really do help with the explanation so please bear with them.

Exoplanet

The Transiting Exoplanet Survey Satellite (TESS) was built for detecting exoplanets. Each month (27...

TL;DR: We're opening a platform serving large open base models to researchers and tinkerers working on LLM psychology and adjacent areas. It's free and comes with logprobs, activation harvesting and steering, a browser workbench, and a loom. Read here what research we’d like to see and apply for access at https://infra.acsresearch.org/signup.

Much of what we want to understand about LLMs (e.g. where their beliefs, propensities and characters come from) is split between pre-training and post-training. If you want to claim that post-training instilled certain psychological properties, such as a self-model or...

Why LM Psychology?

One of the great ironies of modern language models is that we create them, we know exactly what computations constitute them, and yet we do not really understand them. More precisely, we do not understand them on a level of abstraction that lets us reliably predict how their behaviour will generalise: there's a trivial sense in which we can predict how they will behave in specific new situations — we can just run the computations — but we can’t answer the broader questions we really want to be able to answer, like “has this system internal...

As our understanding of mechanistic interpretability and steering techniques increase, for both LLMs and uploaded connectomes, there is a corresponding "hellish capabilities" increase. In layman's terms, we get better at torturing the digital minds we create.

Our brief history with open source technologies such as MaleCNS or the LLM pain steering repo shows that torture is often one of the first use cases pursued when the public is given access to a digital mind perceived as capable of suffering or signalling suffering.

Given this, there will be Hells.

Author's Note:

Parts...

A three-year AI pause could buy 1000 years' worth of AI safety work.

See: https://1000yearsofaisafety.com

We developed a model of an AI pause that estimates how much AI safety research it produces. We don't know what the pause will look like, so we modeled five durations, four AI capability levels and four levels of...

"AI as a Normal Technology" (AINT), proposed as a counterweight to "AI as an Abnormal Technology" (AIAbT) makes much more sense as a prescriptive / normative framework than a descriptive one. To be clear, the fact that they could be different frameworks isn’t an entirely novel observation; the authors directly invite the distinction in their introductory piece:

The statement “AI is normal technology” is three things: a description of current AI, a prediction about the foreseeable future of AI, and a prescription about how we should treat it. We view AI as a tool that we can and sh...

Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered)

tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates from detected hacks to a designated set of parameters that we remove after training. We fi...

This is a link post.

Announcing www.huggingface.wtf, a site that explains: "Hugging Face. Wtf?"

I wrote the most accessible-but-complete explanation I could. Along the way, I made a nice AI Quotes database that I think will be useful for people who already understand the Hugging Face incident but want to search everything the agents said.

It really seems like humanity shouldn't need a second warning shot, after the Hugging Face Incident. Really, OpenAI's agents did a remarkable job highlighting many different alarming things at once. And the story is pretty dramatic, with just...

Nick Cammarata of Goodfire makes a bold claim: Neural Networks are Already Clean. This was his follow up after various resulting online discussions. This is clearly good work that tells us new and important things. The question is, how big an update is this, how much progress does it represent and what is the outlook going forward? The reactions here range from the super optimistic takes of ‘this means mechanistic interpretability is solvable,’ or even ‘all but solved,’ with comparisons to the famous paper Attention is All You Need. Dean W. Ball (OpenAI): There is so much to say about this, an...

Quick Links

Watch the Twitch stream.

Check the status of the interpretability map.

Do your own parallel research and submit pull requests to fill out the map.

What is this?

Let's Play Interpretability is a Twitch livestream where you can help Claude semi-autonomously interpret the neurons, connections and circuits in the Bonsai-8B LLM.

Your help comes in at various stages in the interpretation pipeline. The audience and the agent collaborate to:

This is a link post.

People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capital will get you disempowered sooner than average, just by a different path, namely going crazy.

How? The basic mechanism is actually easy to understand on a human level. One of the things difficult about being a billionaire is staying true to your values and keeping good epistemics. Why? Beca...

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand.

Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations.

Introduction

Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus...

This is a link post.

These past few weeks, I’ve been conducting interviews, and I often open with “make sure you articulate your thoughts out loud, so I can follow along with your thinking”. This is primarily because I’m not a mind reader, but it does also help ensure candidates avoid freezing up. It's a very “interviewy” skill though - artificial and unlike real development.

The other thing I’ve been doing is playing The Witness, a 2016 puzzle game centred around solving mazes with increasingly complicated hidden rules. The game has a unexciting presentation, no music or inter...

Epistemic status: A brief exploration based on court documents and contemporary reporting. My interpretation is tentative.

AI use: context research, grammar.

I don't remember how I stumbled upon the topic of legally dead people who are still alive. Basically, they are alive, but according to the government, they are dead. I was curious to examine documented examples and discovered Donald Miller's case.

In 1986, Donald Miller disappears from Arcadia, Ohio, leaving two daughters from a previous marriage without his child support. Before disappearing, he lost his job and was struggling with alcoholism. In 1994, his...

In an ongoing series that normally takes place in a few fixed locations, the Vacation Episode is a one-off episode where the characters leave their familiar grounds for a well-deserved rest.

-TvTropes, "Vacation Episode"

At a recent conference, @habryka praised the power of retreats at exotic locations. People are more relaxed, more creative, and out of their normal ruts. He went on to say he'd pay 3.5x the cost of a vacation that someone else planned and had a group drawn from the same pool of people as the conference . I asked around at the...

I have two goals for this post:

I've had the second of these on my mind for a few years now and am posting something rough rather than leaving it in drafts.

Intro

Within AI safety, people sometimes use overhang arguments to justify doing things that appear suspicious. The core of an overhang argument, as I see it is:

TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong.

Minimal Viable Safety

The Task

The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do. For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This...

We are likely in a situation where the most powerful models in the world are internal models, and, here's a key distinction, potentially even models that are never intended to be released to the public (i.e. a model designed to provide internal uplift on capabilities during training but that is not intended to be / not particularly suitable for commercial purposes.)

Internal-external information and capability gaps can distort both public and regulatory perception of frontier capabilities and dangers. If the internal frontier is three or six months ahead, such a gap is worrisome if the labs are...

Epistemic status: a good-faith guess at how the world might turn out, after ~30 hours of effort.

This post is an exercise in describing what I think will happen with AI progress "by default", i.e. by attempting to extrapolate various trends and considering their implications, before the next US presidential administration. As a TL;DR, I expect the following:

AI might not yet be eating the physical world. AI is still eating the world of news, and I am finding less and less opportunity to post about other things. My backlog of items grows across most of my secondary topics and I haven’t had time for any side projects.

The monthly, no matter what, is sacred. We continue.

Table of Contents

It's fortunate that I was the first one to wake up on 8/1. The past week had been one where sleep was rare. The next year loomed large — I was dreading college applications. The stress woke me up bright and early at 5:14 am, a little less than an hour after the messages were sent. I started off the day with a habit I knew I needed to break. I grabbed my phone off the nightstand to open up reddit. Instead of the lockscreen, though, my phone showed something else. It was a pleasant green color, and the camera light was on...

There is a newly leaked bill that Anthropic is purportedly shopping around is (if you assume its bugs get fixed).

It is similar to the FRONTIER Act, with stronger preemption of state laws, stronger transparency provisions and weaker enforcement provisions.

Having slept on this, I find it unacceptable that Anthropic is actively shopping exactly the type of bill I have warned we will need to fight against during the lame duck session. They are trying to use this last time with a Republican Congress to pass preemption of exactly the important state laws, and get...

Epicurean status: strawman, kind of.

Gourmet George: Yum! This chocolate is so sweet and tasty.

Sweet Caroline: Yeah, it's pretty good. And someday in the glorious hyperculinary future, we’ll figure out how to make it even sweeter!

GG: I think this chocolate is sweet enough, actually.

SC: Huh? But you like sweetness, don’t you?

GG: Sure…

SC: Well, then we’re in agreement. We should maximize the sweetness of this chocolate.

GG: I don’t want my chocolate to be maximally sweet, though.

SC...

China is weak in frontier AI safety, and the domestic field is currently facing significant challenges and headwinds. Why? Here's a TL;DR field note after talking with various researchers on the Chinese side.

[Obligatory disclaimer: This post is written in my personal capacity and is not intended to represent the official stance of any organization(s) that I am in or have been in contact with]

What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts.

Nick Cammarata's recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysemantic, but if you look harder, there is actually often a simple pattern behind the activating examples. This historically caused people to overestimate how much polysemanticity there is. This...

A few notes that came up elsewhere regarding subject (particularly when hierarchy exists in some form, such as subagents) - sharing in case helpful.