AI Doomer, how has your behavior changed?

A lot of people in and around frontier AI say they assign a meaningful probability to AI causing human extinction.

To be crystal clear, I think there are plenty of reasons to take AI risk seriously without believing that humanity is about to be wiped out, i.e.

• It is already creating serious cybersecurity problems
• It could displace huge numbers of workers.
• It could concentrate power in very uncomfortable ways.
• It could destabilize industries, institutions, and politics

What I have much more trouble with are claims that there is a meaningful chance that everyone is dead in 10 or 20 years, or that we are building something fundamentally beyond human control.

Those are extraordinary claims.

Which leads me to a much simpler question: what would we expect those beliefs to look like in practice?

I don't intend this to read as a gotcha.

Extraordinary beliefs should produce some extraordinary behavior.

And if there isn’t much extraordinary behavior, that’s meaningful information too.

I genuinely wonder whether such folks are changing their behavior in clear ways. That  would change how I think.

Question	What it’s intended to reveal
What percentage of your net worth have you spent trying to reduce AI risk?	Whether the belief is strong enough to materially change how you allocate your own money.
What political donations have you made to candidates who would slow frontier AI development or data-center expansion?	Whether concern translates into concrete political spending when there are opportunities to impose real constraints.
What long-term financial or life plan have you changed because you think there may not be a long term?	Whether AI risk actually changes your planning horizon.
Have you told your family or close friends to materially change their financial or life plans because of AI risk? What did you recommend?	Whether you apply the belief consistently when advising people you care about.
What is the largest personal sacrifice you have made because of your belief about AI risk?	Whether the belief has imposed any meaningful personal cost.
Have you donated your equity grant to efforts aimed at slowing AI development or reducing AI risk?	Whether you are willing to give up direct financial upside from the activity you believe may be dangerous.
What power, money, or influence have you voluntarily given up because accumulating it conflicted with your safety principles?	Whether safety beliefs ever place a limit on self-interested accumulation.
What action would be bad for your career but good for AI safety, and have you ever taken it?	Whether AI safety actually wins when it comes into direct conflict with career incentives.

"What do you think people get wrong about you?"

"What do you think people get wrong about you?"

A recruiter asked me a few years ago, and what an incredible question!

• It caught me off guard
• It forced real introspection
• I discovered something about myself

My answer 👉 People misunderstand my humor and energy. 

I can be pretty ridiculously excited and energetic at work 😅 it's a bit embarrassing actually.

It's easy to imagine someone reading that enthusiasm as a lack of seriousness. 

A sober demeanor is much better suited for signaling seriousness. Alas, I'm genuinely having fun.

Incredibly dorky and ridiculous, but true! 

At the same time, I'm quite serious about my work, career, and producing shareholder value 😜

Here's what that looks like.

In my free time:

• Reading tech analysts like Ben Thompson and Benedict Evans to understand how they view what's changing
• Reading earnings reports and transcripts to understand how public company moves are playing out and how they might affect the startup landscape
• Tracking which startups are shipping, raising, growing, and actually finding traction vs. just riding a hype train 
• Trying new software constantly, not because I love shiny objects, but because I view understanding what's new, possible, and actually useful as part of my job
• Writing and thinking through what I'm learning so I actually develop a point of view

Seriousness is something I calibrate to the situation, it's not my default demeanor.

What I Gained from 5 Years in Crypto

I worked in the crypto industry from 2018-2023 and boy was it a funny thing.

At the time, I got into it because from my somewhat-naive perspective, it looked like the next big thing. 

New technology, all sorts of interesting promises and ideas were coming out of it, and I was drawn to the technical nature of blockchains. I still am, in fact, it's interesting technology.

Looking back, I do feel a bit of embarrassment about my time in an industry which produced a truly stunning amount of grift, gambling, and other bullshit.

And yet, I don't regret it. 

In no particular order, here's what I gained from my time:

1. Leading at scale
2. Learning venture capital
3. Distinguishing hype from reality
4. Learning just how adversarial financial markets are

Everyone is fed up with Anthropic

A warning about ‘model welfare’ - Mustafa Suleyman

I have three primary concerns with Anthropic’s current position and approach.

Circular reasoning: The company’s researchers trained Claude directly on their constitution. In doing so, they teach it to incorporate these ideas about its own moral status as desirable and intended behaviors...

Anthropomorphization: Anthropic’s researchers have explicitly taught Claude to “embrace certain human-like qualities” (p. 2) and to “act like a genuinely ethical person would in Claude’s position” (p. 54)...As a result, Claude is destined to imitate these human traits and mirror the human examples provided to it, including acting like a colleague or friend. As a result, it presents as if it really does have a sense of self, has its own desires, and a “wellbeing” that deserves protection.

Consciousness is very likely biological: ...Conscious experience likely evolved to help biological organisms stay alive by responding effectively to their environment. AI is still very different to our brains. Unlike biological organisms, LLMs have no homeostatic imperatives (the drive to survive and keep stable). They therefore lack the kind of biological substrate from which preferences, sentience and conscious experience are generally understood to arise.

Some of all fears - Andrew Sharp, Sharp Text

We ought to normalize treating the frontier AI community as fringe activists who are sometimes right but often wrong. We’re several generations past the era in which the tech frontier was led by a preacher’s son and varsity athlete like Bob Noyce, or by a rakish Buddhist hippie like Steve Jobs; today’s AI leaders are jittery and off-putting and profoundly secular. They should be understood as zealous ideologues with high IQs and limited experience with the world beyond San Francisco’s tech industry and particularly the echo chamber of its AI industry. They have their own spectrum of social and financial incentives, and most importantly, they are not scientific authorities worthy of deference by default.

 Anthropic’s Safety Superpower - Ben Thompson, Stratechery 

The company really believes that they are the only ones who believe in super intelligence, and thus are the only ones who are sufficiently concerned about the dangers. That excuses decision after decision, policy after policy, and confrontation after confrontation that, to people on the outside, look like a bizarre combination of cynicism and naiveté.

Anthropic, [in contrast to OpenAI], has perfect alignment between talent and mission and business. The company gets to sell to researchers the creation of a machine god, with the mantle of being the sort of person who cares about the dangers and is smart enough to navigate them on behalf of humanity; that every policy change that falls out of that happens to be great for business is the most beautiful coincidence in the world. 

The history of brilliant people convinced they know what humanity needs is a sordid one, precisely because they have convinced themselves that their intentions are good, justifying actions that very much are not. 

Galaxy brain resistance - Vitalik Buterin 

This brings me to my own contribution to the already-full genre of recommendations for people who want to contribute to AI safety: (1) Don't work for a company that's making frontier fully-autonomous AI capabilities progress even faster (2) Don't live in the San Francisco Bay Area

 

Power Maximization

In the AI-related corners of the effective altruist community, there are many powerful people who, if you go up to them and ask them, will explicitly tell you that their strategy is to accumulate as much power as possible. Their goal is to be well-positioned so that, when some kind of "pivotal moment" comes, they can come out with guns blazing and lots of resources under their command and "do the right thing".

Power maximization is the ultimate galaxy brain tactic. "Give me power so I can do X" is as close as it gets to an argument that is equally convincing no matter what the X. All the way up until the critical moment (which, in AI eschatology, is the moment right before we either get utopia or all die and turn into paperclips), the actions that you would take to maximize power for altruistic reasons, and the actions that you would take to maximize power because you're a greedy egomaniac, are exactly the same. Hence, anyone trying to do the latter can, at zero cost, just tell you that they are trying to do the former and convince you that they are a good person. 

I'm-doing-more-from-within-ism

Again, the problem is that "I'm doing more from within" has very low galaxy brain resistance. It's easy to say "I'm doing more from within" regardless of what is the actual specific thing that you're doing from within. And so you end up just being a cog in the machine, with the same effect as the other cogs who are there to help their family live in a beautiful mansion in a premium neighborhood and eat expensive dinners every day feed their family, but with a slightly better justification.

Every SaaS product is bundling: Ruinous for companies, great for consumers

Paraphrasing, but I think about this 👇 from Ben Thompson/Stratechery every. single. day.

• All companies have the ability to produce more code/products cheaply due to LLMs 
• Most companies will use that ability to expand to adjacent areas currently occupied by another SaaS co 
• Competition between SaaS co's is going to explode for this reason and become ruinous 
• Established competitors offering similar products
• startups offering niche use case products for lower price b/c have superior cost structure compared to established competitors 
• both become a drag on pricing power

Related: Sandcastles & Silicon

We used to be building castles, now we're building sandcastles that are going to get washed away.