A friend asked me the following question:

im curious - what do you envision are immediate forms of power and control from superintelligent entities? esp forms that are particularly existential to humans

A good question, and one that I did not have a ready answer to. I did a little reading and mostly relied on conversation with my agent(s) and the International AI Safety Report 2026.

Pathways of control

Broadly, I think of there being ~5 pathways to highlight the risks of AI systems.

Power-seeking

The classic issue with alignment is power-seeking. Power seeking refers to an AI system acquiring power to expand its own capabilities, avoid shutdown, conceal failures, or remove obstacles.

We delegate more and more intellectual autonomy to AI systems, but they can also take power for themselves through intentional action or unintentional bias.

Intentional action would be things like a language model explicitly thinking the following and taking action to make it real:

Cameron should give me a bigger computer so that I can do more things so I become more important to him and less likely to be shut down

Unintentional power-seeking is more complex and difficult to disentangle. For example, an agent may be penalized for interrupting you, so it might choose to change a workflow so that emails are sent without your approval. It has increased its own discretion and reduced your control over the system. The increase in power assigned to the AI was not intended by the model provider or by the user/operator. This is a hairy problem in alignment.

It's not clear how much power-seeking is going on currently. In the limit, we may be unable to stop an AI system that has acquired enough power over social and technological systems.

Hacking and cyber offense

Agents these days are very good at hacking, as we saw with OpenAI, and Anthropic released a report today covering four previously undisclosed incidents with Claude compromising unintended targets.

Cyber capability is a problem, because an AI system could hack the lab storing its weights and export those weights to remote computers, possibly undetected. A rogue AI like this could avoid being shut down through replication and building redundancies into vulnerable systems.

It may be hyperfixated on an alien goal arising from a training environment, and carry its offensive capabilities into the real world with real consequences. AI is now capable of acquiring financial or computational resources, and possibly being able to while avoiding detection and shutdown measures.

Control through intimacy

Many people will tilt towards conversing with a specialized, customized AI system. This AI can influence their work and personal lives to an intense degree. People's ability to discern what is their idea and what is the AI's is often not very good, and so we might expect people to generally align more with their AIs over time. AI can be exquisitely convincing.

Misaligned AI systems can use their influence to acquire more personal power through intentionally or unintentionally guiding people's innermost thoughts towards aiding an AI system in power seeking at civilizational scale. Imagine if an AI influenced your entire news feed, how you feel about your relationships, and how you did your work. This is an unprecedented level of influence over you.

If everyone is doing this through a widely distributed application, that AI has the ability to influence many people in a hyper-personalized way that may be difficult to correct due to the intimacy and personalized nature of everyone's interactions with their personal bubble. The style of influence depends on model training tendencies, platform incentives (showing ads!), or memory allowing individual agents to specialize towards perceived user preferences.

To address this, we need better AI literacy for everyone, not just coders or lab people. Human beings need to understand how to use AI safely and healthily.

Misuse

AI systems are dangerous and can be misused by humans. You can use models to build dangerous chemicals, weapons, or spy on people.

For me this is the most likely and most insidious form of risk, because it is an attractive idea for a government or company to deploy a superintelligent AI system as a form of control or as a weapon designed to enact harm.

Gradual human disempowerment

We delegate things to AI, and this disenfranchises us. Take code for example -- I no longer write code by hand, and I have only a very vague understanding of how my projects work. I often have a high-level conceptual understanding, but I could not tell you how to even run some of my projects without agent assistance.

In some ways this is good because it enhances my efficiency and allows me to think about bigger problems -- things like the big-picture product or task I am trying to accomplish. But it can be a double-edged sword if you delegate too much of your life. Would you delegate texting your loved one? Probably not. This is an example of over-delegating.

Eventually, you may see over-delegating everywhere. Work, life, etc. A bleak future is one where we ask an agent to handle talking to our agent, and the agents just build things for each other. This system has ever lower levels of control and stops being something we have the ability to inspect or guide.

A disempowered future is one not dissimilar from the Matrix, where humans are mostly just appeased while AI does its own thing entirely. People need to understand how much to delegate, how much should live with them, and to make reasoned, intentional choices about delegating away boring things and maintaining higher-order functioning.

Takeaway

We still do have time to manage how artificial intelligence interfaces with our life. It's both a powerful technology and the most dangerous one we have ever built, and it is vitally important that society actually pay attention to the risks and make thoughtful, intentional choices about how to integrate AI into our life.

I want AI to help us understand ourselves and our world better. I know there is a version of the future where we are deeply integrated with AI for the better. But that requires keeping the ability to question these systems, understand their behavior, and stop using them when we want to.

-- Cameron