Money is coming to AI safety, and soon.
Anthropic and other labs are approaching liquidity events. Unprecedentedly large foundations are being stood up. New donors are waking up to AI risks.
This is an incredible opportunity.
Unfortunately, not all is well in the world of AI safety grantmaking. We lack ambitious theories of victory. We lack scalable interventions. We lack talent. At times, we even lack common sense.
Will we meet the moment?
Here are the problems we need to solve to prepare for the coming wave.
1. The problems aren’t operationalized
Operationalization is defining the problem precisely enough that you could tell if you’d solved it. Most of AI safety hasn’t done this work.
Most AI safety challenges remain too vaguely defined to act on. We lack threat models precise enough to generate clear intervention points, solutions, and projects with measurable objectives. The field has “areas of work” (technical safety, governance, politics) but these are not broken down into the kind of specific, solvable problems that can be assigned to a team with clear OKRs and metrics for success. Without this clarity, funders face a thin pipeline of credible giving targets, and talented people have nothing concrete to build.
What we need is closer to what biosecurity has: each major threat area decomposed into specific failure modes, specific intervention points, and specific projects that can be resourced, staffed, and held accountable.
This will not be a single unified analysis. Different experts will disagree on threat models and on what helps. The goal is multiple competing pictures of what can go wrong and what we should do about it, so we can hedge across the different possible worlds we might find ourselves in.
Once we narrowly scope and operationalize our problems, the needed organizations and their form often become evidence. The hard part is getting to that level of specificity.
Some problems may resist operationalization. Technical alignment, in particular, remains pre-scientific in important respects: we cannot yet fully specify what a good outcome looks like or what “solving alignment” even means precisely enough to decompose it into sub-problems with win conditions. Where full operationalization is not yet possible, we can operationalize nearby problems while accepting the risk of Goodharting on a proxy. When we cannot operationalize everything, some muddling through is inevitable. But that’s all the more reason to operationalize everything we can.
Not everything will reduce to discrete projects either. Unknown unknowns and emergent risks require ecosystem-level robustness: coordination mechanisms, built-up capacity, idea generation, ongoing threat monitoring. The CG approach of seeding a wide range of organizations has value because it hedges against this risk.
2. There are no ambitious theories of victory
If operationalization gives you a map, strategy is what route you decide to take. Multiple strategies can target the same operationalized problem, and can sometimes be pursued in parallel. Not every strategy is good, and many strategies will conflict. But AI safety lacks ambitious ones regardless.
Often, the word strategy gets used to mean “vibes about a cause area.” Funders have highly correlated strategic takes that are often out-of-touch or simplistic. Both funders and organizations don’t design realistic, ambitious strategies that actually get to win conditions.
The lack of clear strategies also prevents us from measuring progress towards our goals. By figuring out what is on the critical path to victory, we can determine what tactics are actually furthering our strategy.
Good strategy also allows funding decisions to be metrics-driven. This is important. Wherever possible, we should attach metrics to our grantmaking so we are anchored in reality and results. Metrics allow us to avoid grift and overly galaxy-brained ideas.
The current lack of ambitious strategies and their corresponding tactics has left us in a sorry state.
We have a sprawl of individually defensible tactics that don’t compose into a theory of victory, because there is no theory of victory, because the problems aren’t operationalized.
These strategies, when designed, will not always be public; admittedly, it is hard to tell which live players are already executing on some ambitious strategy to influence the trajectory of AI.
3. There aren’t enough good (mega)projects
Besides 10x-ing the funding of METR, how else will we use capital?
Unless you are comfortable with the current state of affairs, the need for more good projects should be self-evident. Everywhere you look there is a project that could exist, and existing organizations that aren’t doing nearly enough.
Large grantmakers like CG try to solicit new projects by submitting RFPs for a given issue area. The problem with this is that RFPs do not really work. The most competent people to run a project are usually not unemployed, waiting to start a project where it’s needed.
Active grantmaking, not passive grantmaking, is the solution. We must go into the world and seed the projects that need to exist, rather than just waiting for them to fall into our laps.
And small projects alone won’t get us there. We have the capital; we need interventions that can scale. These times call for megaprojects.
What would it look like to have general managers for the world’s problems? Where are the organizations making herculean efforts on every given intervention point? The organization evaluating AI character, or helping the right pass good AI policy, or researching space governance, or organizing concerned lab employees and scientists, or building tools for better coordination, or communicating to the public about AI risk?
Thinking on the margin will not cut it. We need to create and define markets, and shape which opportunities will exist.
We also need highly scalable fallback interventions in each field: a “GiveDirectly” for each threat model, so we can convert money to impact while it matters most.
This might include using compute for AI labor, stockpiling PPE, or funding large political ad campaigns. This could also give us a benchmark that more targeted projects have to beat.
4. The field is made of researchers, not operators
How do we build these projects, or support people who can, once we identify what’s needed?
Founders are the key bottleneck that unlocks everything else. They are how we go from operationalized problems and ambitious strategies to actual projects in the world. A single mission-aligned founder who is good at recruiting can pull in the engineers, ops leaders, and policy experts who are not in the AI safety community but are exactly the people these organizations need. The founder sets the goals. Everyone else falls in line. You don’t need the entire team to come from the AI safety community. You need one person who is mission-aligned, excellent at building, and well-resourced.
The same problem shows up in grantmaking. We need more grantmaker-operators, not grantmaker-researchers.
The AI safety community does not value founders. It values the researcher who can do the best BOTEC, has the best philosophy reading list, or writes the most qualified blog post. Not the person with conviction. Operators and founders look at this environment, see that competent execution is not what gets rewarded, and go elsewhere.
This needs to be fixed culturally. Reverse the inversion of status. Give founders leverage rather than making them compete for attention, wait months for decisions, and shape their work explicitly around what a specific funder will support.
The same problem shows up in grantmaking.
We need more grantmaker-operators, not grantmaker-researchers. Not the grantmaker who can do the best BOTEC, has the best philosophy reading list, or writes the most qualified blog posts. The person with conviction, agency, and experience in the real world making things happen.
Grantmakers who are competent operators can better engage in active grantmaking. The success of CG’s technical AI safety team in multiplying the number of grants proves that progress is possible when agentic people push.
This is also partly solved by adding grantmaker capacity and valuing our grantmaker capacity more highly so they can do more with less; thoughtfulness is great, but not at the expense of results.
5. We can’t attract talent
The lack of competent executors is unfortunately a more fundamental problem.
The AI safety ecosystem grew out of online research forums and ML labs doing technical research. These origins disproportionately attracted people who like writing on online forums and people who do machine learning research. Accordingly, the ecosystem has built extensive infrastructure for researchers: fellowships, career transition programs, lab positions, mentorship networks.
There is nothing remotely like this for operators and founders, people who can take a well-scoped problem, build an organization around it, recruit a top team, and execute.
The result is self-reinforcing: operator-founder types look at the ecosystem, see a research community, conclude it is not for them, and go elsewhere.
The few efforts pushing in the right direction, like Halcyon and BlueDot, are valuable but remain a small fraction of the overall ecosystem investment.
Founders rely heavily on social proof. There is still no cultural signal that this is a space where ambitious, experienced builders belong.
The AI safety community needs to create this reputation, or seed new organizations that recruit founders without leaning on the community at all. Founders do not need to be perfectly aligned with EA principles or community norms to start a great biosecurity startup or be an incredible political operator. We need to let them do what they are good at.
This requires a deliberate cultural fix, to reverse the current inversion of status. We must give founders leverage rather than making them compete for attention, wait months for decisions, and shape their work explicitly around what a specific funder will support.
6. Our grantmakers aren’t specialized
We have too many generalist, all-in-one grantmakers. CG, for example, should focus on the part of the pipeline they’re best at. They deserve a ton of credit for getting us here, but are failing to meet the moment by trying to do everything at once. The function of finding and evaluating small grants is different from scoping and incubating new orgs, which is different from scaling working ones, which is different from managing megaprojects.
A funder that tries to do every job will do none well. It will not, for example, build up the capacity and skill to intentionally incubate and support important megaprojects.
Just like VC firms invest in seed rounds, Series A, Series B, and Series C, AI safety grantmakers should specialize by the maturity of the projects they fund. Different grantmakers are placed differently to fund different projects.
YCombinator is an example of what it looks like when you define and support founders at a specific stage in their founding.
The OpenAI Foundation is well-placed to scale a large megaproject with an existing track record. CG can evaluate early-stage projects but not start them.
But we are missing organizations for most of these stages, especially seed funding (like IFP’s Launch Sequence) and standing up megaprojects. Manifund, Halcyon Futures, Catalyze Impact, Seldon Lab, Constellation, and Lightcone are examples of what we need more of.
7. We need a diversity of funders
Beyond specializing the grantmakers we have, we need more of them, with different theses, different risk tolerances, and different identities.
The risk tolerance of existing funders is too low, and too uniformly low. Everything is too correlated. The grantmakers we have are scared of making supporting important bets such as enabling potential capabilities restraint or doing political outreach to specific political factions. Existing funders have various quirks and their constrained positioning has dramatically affected how leveraged their dollars are.
The fix is not just to ask existing funders to be bolder, but to create new funders without those constraints. Right now, a small number of funders dictate what the entire field can and cannot fund. Their idiosyncratic histories, their reputational concerns, and their strategic blind spots become the field’s de facto constraints. There are no alternatives for founders whose work doesn’t fit. Decorrelating the field means standing up new funders with different theses, different theories of victory, and different appetites for risk, so that no single funder’s caution or quirks determine what strategic bets can be made.
This problem is happening even at the donor level, where it shouldn’t. When donor advisory orgs like Longview pool individual donors’ money into one big pot, the decorrelation those donors could provide gets erased. Each unique donor with their own theory and risk appetite gets flattened into the same averaged-out worldview as every other one. Individual donors should be encouraged to fund decorrelated bets, not pool them into the same vehicle.
We also need diversity of political identity. Money is not fungible in politics, and a group’s funding sources matter. Funder identity is also a reputational firewall: different organizations supporting different bets means a single bad bet doesn’t tank the credibility of a megafunder.
We are also missing a category of funders entirely: unhobbled funders. Funders with diverse backgrounds willing to put their name behind aggressive policy and public communication campaigns, and make c4 donations. Funders willing to be advocates. These funders won’t show up by themselves, and the field needs to invest significant effort in proactively finding and recruiting them. This is especially true because politics is both neglected and promising, but also among the most time-sensitive parts of any theory of victory.
Beyond all this, more grantmaking organizations also help in mundane ways. Because AI safety grantmaking is so early, the funders have relied heavily on trust and relationships. But any single grantmaker’s social graph hits fundamental limits. There are good projects and great founders in networks the existing grantmakers cannot see.
More grantmakers means a larger surface area, more networks, and a marketplace of ideas instead of convergence on one worldview held by one or two organizations.
8. We don’t do obvious things
We can talk about inducement prizes, decision-theoretic uncertainty, spending money on AI labor right before the singularity, and mechanism design all day. None of this is what is actually going to save the world.
There is a strange culture in AI safety where being thoughtful is treated as an end in itself. Galaxy-brained schemes get more attention than executional wins. This is backwards. Thoughtfulness and good intentions are good only insofar as they yield results.
We need to get money out the door now. There are enough obvious things to fund, and most of the work lies in scaling up existing pipelines. You only get to be galaxy-brained after funding all the obvious things, and we have not.
There are $100 bills on the ground. Let’s pick them up.
Help the Anthropic people with their taxes. Fund people who are clearly positioned to do good work ASAP, not in six months, if you believe in short timelines. Scale up political organizations so they can do more before we see political lock-in. Where’s SuperPAC #2? Where are the second and third versions of the orgs that are clearly working? Why are existing grantees waiting months for decisions on grants that should close in days? Why are talented operators with mission-aligned visions still scraping for runway?
We must stop being stingy. The constraint is grantmaker here is time, talent, and people, not capital.
Raise salaries. Fund all existing organizations to their new optimal size. Many current organizations are subscale, have planned around scarcity, and could be substantially more ambitious with their resources. Fund alternatives too. Create more organizations with the same priorities. Build an environment of healthy competition. In advocacy, take a splatter-gun approach. Grantees should raise their ambitions dramatically and begin preparing to scale. Tell them: what would you do at 10x the current budget? Hire competent managers. Build operational infrastructure. Ditch the scarcity-mindset and elevate your ambitions.
We know the discount rate on money is extremely high. We know that we should front-load. But the community is not actually doing things that reflect their beliefs about the situation and timelines. This is not what a community that is trying to win does.
Funding AI safety in the next one or two years is the best use of capital in the space, and we should be doing trades across cause areas to enable this. We should be scrambling to spend the money we have now.
These problems are all extremely challenging and complex.
They require fighting bureaucracy, perverse cultural norms, and damning base rate for successfully deploying so much capital, so quickly. But we should fight like hell to solve them all the same.