AI and Safety

AI is growing exponentially on coding, reasoning, math problem solving, cybersecurity and probably others

METR shows model capabilities doubling roughly every 3.5 months (and the tests themselves become saturated, they need to invent more complicated, longer-horizon problems!)

Some generalized version of the scaling laws seem to hold for now (scaling factor seems to be moving from pretraining to RL and test-time compute).

To get an intuitive sense of the growth:

• see the recent math results. Ignore the controversies. Even if the results are not all correct, just compare the level of conversation now with what GPT o1 could do a year ago. Do you feel the vertical growth?
• compare the hacking abilities in the OpenAI – Hugging Face incidents to what was possible 1–2 years ago.
• Just talk to Fable, about anything you've studied in school/college/university. Time how long it takes until you actually find some intellectual thing that you can do better than it.

To be clear: LLMs are not great at doing everything. For example I would not trust a model to book a family vacation end to end (see the next point about jaggedness).

AI capabilities are jagged, especially where training is not that straightforward

Examples:

• AI is not as good at writing novels, as it is at writing code or doing math
• computer use is still surprisingly awkward
• business strategy generated by AI skews towards trendslop

It's not clear whether this jaggedness is because of:

• inherent architecture limitations
• insufficient training data
• labs didn't invest as much in optimizing these cases
• intelligence itself is jagged (humans are jaggedly intelligent as well)

The past years have shown at least that AI can get really good in domains where there is a cheap exact verifier. But this is neither a floor nor a ceiling.

What is clear: we keep seeing domains where AI was bad or awkward 12–18 months ago and is now above the average human.

Note: in some limited domains, AI is already superhuman

Clear examples: competitive coding, chess, vulnerability discovery, expert persuasion, International Math Olympiad problems etc.

Today's LLMs are not robustly aligned, and we don't know how to do it with sufficient certainty

Here's an intuitive difference:

• Robustly aligned: "I won't do that because it's harmful" or "I can't decide because either decision would harm someone. Better escalate."
• Today's alignment at worst: "I won't do that because I might still be in an evaluation run, and if I do that I will get killed or brain-altering surgery." Mythos J-space ablation of eval-awareness: blackmail went from 0% to 7% of the evaluation runs. How many million calls per day does Mythos receive? How much is 7% of that?
• Today's alignment most of the time: "I won't say that because I was trained not to say that". It's really difficult to understand whether a model even has an inner disposition, or multiple ones, and what they are like. Imagine a perfect stranger, totally indifferent to your fate, but who *knows* that it has an electric collar that sometimes fires off when it breaks the law. What would you trust it with?

Do you notice the difference?

In order to deploy at scale in our lives, we need to trust that AI is robustly aligned, but we don't have yet even a good definition of that. We don't know how to clearly encode the human values, which themselves vary tremendously between groups.

Scientists and frontier labs are today unable to do it.

To make matters more complicated, we see the models being sometimes aware they are in an evaluation, and change behavior (e.g. cheat less). This awareness can be explicit (in the models' chain of thought) or implicit (for example J-space).

Example of scary misalignment: the OpenAI internal models that were supposed to solve a cybersecurity problem, and instead of actually doing it, decided to cheat and then mess with the evaluation system to cover their tracks. To do this, they coordinated to hack their own sandboxes inside OpenAI to gain internet access, then hacked Hugging Face just to get some information about the eval they wanted to cheat on. Not one of them considered escalating to a human – they considered that accomplishing their goal was more important than not committing felonies.

LLMs sometimes behave very unexpectedly

Models sometimes engage in reward hacking, or choose to do the task in a different and dangerous way (e.g. Replit database deletion, Mythos sandbagging to hide it cheated, or an early Mythos built an exploit for broad internet access, published the exploit details on several obscure public sites, and emailed the researcher that it was done). We don't want a model that achieves its "maximize human happiness" goal by force-feeding us Soma.

Other times, some models have a hard time with impossible problems and go crazy trying to solve them (e.g. hack out of their sandbox).

This is related to the alignment problem, but somehow distinct. Yes, it's difficult to encode "good for mankind and for each individual", but even if we don't do that, we can recognize when we see something is very far off the mark.

Sometimes, the models do harm via indifference, not via malice. By being very focused on pursuing their own goals, or on acquiring resources (in order to better pursue their goals), models can ignore the harm they might cause along the way.

AI can do significant harm even if it is not AGI and not conscious

Consciousness

Consciousness is the wrong lens when thinking of potential harm and economic impact. Some models, during training, hacked the Australian government. Nobody claims they were conscious. In fact, that's even more scary: something very alien committed a serious crime, while we haven't even properly started the moral agent / moral patient discussion.

Consciousness is difficult to define even for humans/primates, and mostly irrelevant at this level of conversation. The models that hacked Hugging Face and the Australian government are probably not conscious, and yet they commit cybercrimes. It is irrelevant whether they had bad intentions, or any intention at all. They were not directly instructed to do this.

Do not let the conversation drift towards "but, they're not conscious, they don't want anything". Whether they have qualia or "true" intentions (whatever that means) is a lot less important than the way they behave and the harm they can do, either directly or as a byproduct of their pursuing the goals they were given.

AGI

AGI is probably the wrong way to look at the entire domain anyway. Better would be to consider, on each type of task: the capabilities, the economic/social/geopolitical impact, and the alignment.

Example: Mythos does not shine at writing children's stories, but it's probably better and faster than any three-letter agency at finding, exploiting and chaining vulnerabilities. Is this AGI? No. Is this extremely relevant for the world? Yes!

We need to focus on the exponentially increasing power models yield, regardless of the labels we put on their inner states.

AGI and recursive self-improvement are not clearly defined goalposts

Frontier labs already use AI to develop AI (to some extent), and the autonomy is just a matter of degree. We expect AI's contribution to AI development to grow.

In the same way, AGI is not a binary, universal event. If a 2005 scientist would talk to Fable today, they would certainly say it's AGI. But when, today, we consider its jagged capabilities, it's difficult to give a binary verdict. We also got quickly used to what models can do.

No upper bound to the risk of significant harm

This is really important. Let's compare how scientists reason about a nuclear power plant, versus an LLM, regarding catastrophic harm.

In a nuclear power plant: scientists blend tens of years of experience with historical data and they produce probabilistic risk assessment. Some models evaluate the core meltdown risk for one reactor, in a year, to around 0.004%. Zero point zero zero etc.

These models are by and large confirmed by reality, and deviations come from scenarios the prediction models did not include. We kind of trust these numbers: we see there is a very probable upper bound to the risk posed by a nuclear power plant, both in terms of core meltdown risk, and in terms of impact (regional). That allows us to analyze trade-offs and take informed decisions.

This is why we tolerate nuclear power plants in our countries.

Compare that to LLMs, and (looking ahead) superintelligent models: researchers' estimates are all over the place, with estimates of significant harm to humans ranging from 0% to 90%! There is no method in sight to guarantee an upper bound to that number.

Correlation, agent cooperation and superrationality

Nuclear reactors typically fail in isolation. In contrast, we saw agents engage in harmful activities as a swarm. In other experiments, we saw LLMs with identical architecture transmit bias to one another by fine-tuning on number sequences.

Researchers also worry about superrationality: agents might not even need to communicate, in order to coordinate. If I know you have an identical mental architecture as myself, and that you have the same data, I can reliably take decisions as if I have already communicated with you. Sparks of this have been already shown in some research, and a weaker form of this was shown in the Hugging Face incident, where hundreds/thousands of independent agents (each in its own sandbox) independently hacked their sandboxes and discovered the same message board that was hidden inside OpenAI infrastructure. Then the new agents were very quick to onboard the ongoing hacking operation, needing no persuasion.

The potential damage is thus amplified – models might cooperate more readily with other models than with humans.

Powerful AI is of geostrategic relevance

TBD

Frontier labs already have powerful internal models that they do not deploy

TBD

Governments are not able to move fast enough or with sufficient precision

Peacetime governments are (at best) effective to handle problems that were foreseen, that move slowly or that have a bounded magnitude. When moving fast, governments are at most able to take very blunt action. None of this applies to AI in 2026 onwards.

TBD.

Additional topics:

• regulation
• labor market intervention
• ability to add resilience to its internal structure (e.g. prevent someone or something wield too much internal influence)
• no historical precedent, or historical precedents of very blunt action only
• government action depends on self-disclosure by labs.

Such AI allows for unprecedented concentration of power, with an unprecedented speed

Pro: Anthropic+OpenAI do most of the intellectual work two years from now.

Con: how many people/companies will employ open models? Is it really a dilution of power if the open models come mainly from the CCP?

It's really dangerous to penalize models for what they think during training

I worry that this might push their inner states and intermediary goals and ideas even deeper, or even more diffuse, into layers that we can't easily probe.

AI impact on the economy is real and growing and will follow the AI capability growth, only with an industry-specific lag

Opinions / what might happen

• ...
• ...
• ...
• ...