Functional Decision Theory and the architecture of civilizational responsibility

The question that stayed with me

It came at the end of the panel. The room had spent an hour with honest, uncomfortable material, algorithmic accountability, data bias, cultural erasure, the slow abdication of individual responsibility to systems nobody fully understands. Then, quietly, from somewhere in the audience:

“But given everything you’ve said… should we even bother?”

I have heard versions of this question many times. In boardrooms where the scale of systemic risk feels paralyzing. In policy conversations where the distance between individual action and structural change seems infinite. In private, from people who care deeply and are exhausted by caring.

What strikes me is that this question is almost never a sign of laziness or cowardice. It is usually the conclusion of a reasoning process. A logical inference, drawn from premises that feel solid: the problem is vast, my contribution is small, therefore my contribution is negligible.

The conclusion feels inevitable. It is not. It is the product of a specific, and structurally flawed, model of how decisions work.

That model is called causal thinking. And understanding why it fails here is the beginning of something more useful than optimism.

What causal thinking does to us

Causal thinking is not a mistake. It is a cognitive tool, extraordinarily well-adapted to a large class of problems. If I push a glass off a table, it falls. My act caused the outcome. The chain is short, traceable, unambiguous. Causal thinking built civilization’s infrastructure, its legal systems, its management frameworks.

The problem is that causal thinking was never designed for problems where:

• the effect is diffuse across millions of agents and decades of time,
• the outcome depends on the composition of decision types present in the system, not on any single traceable act,
• and the question is not “what did my act cause?” but “what kind of agent am I being, and what does that make possible?”

When causal thinking meets civilizational-scale risk, it produces a specific and predictable failure: the illusion of the isolated gesture. My vote is one in millions. My choice to stay engaged is one signal in a noise of billions. My decision to ask hard questions about an AI system I use is invisible at the aggregate level.

Therefore, causal logic concludes , it does not matter.

This conclusion is wrong. But to show why, we need a better model.

Functional Decision Theory, the mechanism

Functional Decision Theory, developed by Eliezer Yudkowsky and Nate Soares at the Machine Intelligence Research Institute, proposes a fundamental reframing of what a decision is.

In classical causal decision theory, you evaluate an action by asking: what physical effect does this act cause? The decision is a local event with a traceable downstream impact.

FDT asks a different question: what is the output of my decision function, and what happens in all worlds where agents with similar decision functions make similar choices?

The shift is subtle but consequential. Your decision is not only a local cause. It is the output of a decision structure, a way of reasoning, a pattern of values, a set of commitments that constitute how you respond to situations of this type. And every agent who shares a sufficiently similar decision structure makes a logically correlated choice.

This is not causal correlation. It is logical correlation. You are not causing others to act. You are being the kind of agent whose existence in the system changes the probability distribution of outcomes, because the system is composed of agents, and its trajectory depends on what kinds of agents those agents are.

The practical implication is precise: when you choose to remain accountable, to ask hard questions, to refuse to delegate your judgment entirely to an algorithm, you are not just making one choice. You are instantiating a decision type. And the aggregate of that decision type, across all agents who share it, is what determines whether certain futures remain possible.

This reframes the question entirely. The relevant unit of analysis is not the individual act. It is the decision structure. And the relevant question is not “what does my act cause?” but “what kind of agent am I being, and what does that make structurally possible?”

From individual to civilization

This is where several bodies of work converge, and where intellectual honesty requires me to be precise about what is sourced and what is my own synthesis.

Toby Ord, in The Precipice, estimated roughly a one-in-six probability of civilizational catastrophe within the next century. More importantly for our purposes, he argues that we are currently in a period of exceptional vulnerability, a window during which the decisions made by present agents disproportionately determine long-term civilizational trajectory. The literature on existential risk is explicit on this: certain choices made now function as existential risk factors or existential safety factors, shaping probabilities at civilizational scale.

Anders Sandberg, in his 2025 paper Civilizational Virtue, Civilizational Autonomy, and Existential Risks, argues that navigating this window requires what he calls civilizational virtues, dispositions that only make sense at the level of the whole, irreducible to any individual act.

Truth-seeking.

Accountability.

The refusal to delegate vigilance.

These are not individual virtues scaled up. They are a distinct category of disposition, meaningful only when instantiated collectively.

Margaret Gilbert’s concept of the plural subject offers a formal mechanism for understanding how this works: micro-agents, through shared intentions and coordinated commitments, constitute a coherent collective actor. The civilization is not the sum of individuals. It is the emergent structure of their coordinated decision types.

The theory of social choice adds a further layer: individual preferences and behaviors, aggregated, structurally determine macro-level outcomes. Even when each agent is “small,” the composition of agent types in the system determines its trajectory.

The Future of Life Institute and the broader effective altruism literature on existential risk make the normative implication explicit: reducing existential risk depends on many actors adjusting their behavior in coherent directions, rather than on any single centralized intervention.

What is sourced, what is my synthesis, and why this matters

I want to be explicit here, because intellectual honesty is not only an ethical position, it is an epistemic one.

The following are well-established in the literature: the importance of actively reducing existential risks now; the framework of existential risk factors and safety factors at civilizational scale; the mechanism by which aggregated individual decisions structure collective trajectories.

The following are my own conceptual synthesis, built on these foundations, but not directly attributable to any single source:

The framing of the core contrast as: “behave as the type of agent that makes good futures possible” versus “act only if your direct causal effect can be demonstrated.” This is my interpretation of the spirit of FDT and existential risk literature, not a direct citation.

The claim that “great civilizational shifts come from aggregates of micro-agents who refused resignation.” is a normative reading, widely shared in spirit across these fields, but it is not a theorem.

There is no mathematical proof that refusing resignation at the micro level is the most robust strategy.

What there is, instead, is a strong convergent argument: in the presence of radical uncertainty and global risk, treating yourself as an agent of trajectory, rather than as a causally negligible particle, is the more prudent and structurally coherent position.

I name this distinction not as a disclaimer, but as a method. An argument that knows its own boundaries is more trustworthy, not less.

Responsibility as structural condition

What follows from all of this is not a moral exhortation.

It is an architectural observation.

Human ethical responsibility, the willingness to remain accountable, to ask hard questions, to refuse resignation, is not an optional virtue. It is a structural condition for civilizational stability.

Without it, the feedback loops that allow systems to self-correct do not function.

Governance cannot work on a population that has outsourced its judgment.

Regulation cannot hold without citizens who understand what they are regulating.

Before using any AI-generated output, ask yourself two questions:

1. Do I agree with the impact this decision will have, on me, or on others?
2. Can I explain this result to someone else? Does it come from me, or from the AI, and therefore from someone else entirely?

These are not technical questions. They are the practical interface between Functional Decision Theory and daily life. They are how you instantiate the decision type that makes accountability structurally present in the system.

Individual education before government regulation, not as a sequence, but as a dependency. A population that cannot ask these questions cannot regulate anything meaningfully. The infrastructure of civilizational resilience is built, first, at the level of individual epistemic responsibility.

What this changes in practice

For decision-makers:

FDT implies that governance frameworks must address decision structures, not only outputs. Asking “who caused this harm” after an AI incident is causal thinking applied too late. The relevant question is “what decision structure produced this architecture, and who is correlated with it?”

For individuals:

Every choice to remain engaged, to ask hard questions, to refuse delegation of judgment is structurally significant, not because it directly causes a specific outcome, but because it instantiates a decision type that, aggregated, changes what is possible.

For organizations:

Externalizing ethical responsibility, through terms of service, model cards, user consent, is not a solution. It is a structural flaw. It relocates accountability without eliminating it, while ensuring that no agent in the system experiences it fully. The result is a recursive accountability vacuum.

For governments, and the education they have largely let erode:

Civilizational resilience depends on populations capable of asking hard questions about the systems that govern their lives. This capacity is not innate. It is built through education, critical thinking, epistemic literacy, the ability to distinguish between a tool and a decision, between a result and a judgment.

Governments across the democratic world have systematically underinvested in precisely these competencies, while simultaneously accelerating the deployment of automated decision systems into every domain of public life.

The result is a structural mismatch: populations increasingly subject to algorithmic governance, increasingly unequipped to interrogate it.

This is not a cultural failure. It is a policy failure, and a civilizational risk factor in the precise sense that Ord and Sandberg define.

Rebuilding this capacity is not an educational priority among others. It is a precondition for everything else on this list.

Conclusion

The math is simple. If everyone who understands the stakes acts as though their contribution is negligible, it becomes negligible.

If enough of them don’t, it doesn’t.

You already understand the stakes. That is not nothing. That is, in fact, everything.

Bother anyway.

Q&A Guide

For decision-makers

Q : This sounds compelling in theory, but how do I apply FDT to actual governance decisions in my organization?

A : Start by shifting the diagnostic question.

Instead of asking “who caused this incident?” after something goes wrong, ask “what decision structure produced this outcome, and who in our organization is correlated with that structure?” This reframes accountability from post-hoc attribution to prospective architecture. It also changes what you audit: not outputs, but the reasoning frameworks that generate them.

Q : Isn’t individual accountability just a way of deflecting from corporate and regulatory responsibility?

A : No, and this is a critical distinction. FDT does not replace collective and institutional accountability; it distributes responsibility more accurately across all the agents whose decision structures contributed to an outcome.

Corporations, regulators, and individuals are all correlated with the structures that produce civilizational trajectories. The point is not to burden individuals while exonerating institutions. It is to recognize that institutions are composed of agents, and that their decision structures are the sum of those agents’ choices.

For tech professionals

Q : AI systems are tools. Isn’t attributing “decision structures” to them a category error?

A : This is precisely what the next article addresses.

For now: the decision structures embedded in AI systems are human decisions, made during design, training, and deployment. The question of who is correlated with those structures, and therefore who bears responsibility for their outputs, is not a category error. It is the most important question in AI governance.

Q : FDT was developed for individual agents. Does it scale to organizations and civilizations?

A : FDT scales to any context where the relevant unit of analysis is a decision type rather than a single act.

Organizations and civilizations are not individual agents, but they are composed of agents whose correlated decision structures produce collective outcomes. The scaling requires the additional frameworks of Gilbert (plural subjects) and Sandberg (civilizational virtues), which is why the argument uses all three.

For general readers

Q : I understand the logic, but I still feel like my individual choices don’t matter at this scale. How do I actually internalize this?

A : The feeling is rational given how causal thinking works. The shift is not emotional, it is cognitive.

Ask yourself not “what does my act cause?” but “what kind of agent am I being?” You are not trying to trace a causal chain from your decision to a global outcome. You are instantiating a decision type. The aggregate of that type, across all people who share it, is what changes trajectories. The two questions from the panel are a practical starting point: they are the daily practice of being the kind of agent that makes accountability structurally present.

Q : Isn’t this just a sophisticated way of saying “every little bit helps”?

A : No, and the distinction matters. “Every little bit helps” is still causal thinking: it assumes your act contributes a small causal quantum to a large effect.

FDT says something different: your decision is not a quantum of cause. It is an instance of a decision type. The question is not how much your act contributes, but what kind of agent you are. These produce different conclusions in cases where causal contributions are genuinely negligible, which is most cases at civilizational scale. FDT gives you a reason to act even when your causal contribution is effectively zero.

Why this matters

Q : Why does it matter whether I think of my decision as a “cause” or a “decision type”? The outcome is the same.

A : It is not. Causal thinking tells you your act is negligible when its direct effect is small, which, at civilizational scale, it almost always is.

That conclusion leads to inaction.

Decision-type thinking tells you something different: the relevant question is not the size of your effect, but the structure of your reasoning. And that structure, aggregated across everyone who shares it, is what determines civilizational trajectories. The framing is not cosmetic. It produces different behavior in precisely the cases that matter most.

Q : Why does AI specifically make this urgent right now?

A : Because AI systems are scaling the consequences of decision structures at a speed that outpaces our ability to course-correct.

A flawed human decision affects a bounded number of people. A flawed decision structure embedded in a model deployed at scale affects millions, before anyone has asked the two questions. The window in which human accountability can still shape these structures is open. It will not remain open indefinitely.

Q : Why can’t we just wait for better regulation?

A : Regulation addresses outputs.

Decision structures are upstream of outputs.

By the time regulation catches a harm, the structure that produced it has already generated thousands of instances of that harm, and embedded itself in dependent systems, business models, and public expectations. Waiting for regulation without building the human capacity to demand and evaluate it is waiting for a solution that cannot arrive without you.

Q : Why does population education matter more than technical solutions?

A : Technical solutions are decision structures too, built by humans, reflecting the values and blind spots of the agents who designed them.

A population that cannot interrogate those structures cannot evaluate whether the technical solutions actually solve the problem, or merely relocate it. Education is not the soft alternative to technical rigor. It is its precondition.

Q : Why should I trust this argument?

A : You should not trust it.

You should evaluate it.

That is precisely the point. An argument that asks you to remain epistemically responsible cannot simultaneously ask you to accept it on authority. Read Ord. Read Sandberg. Read the FDT paper. Ask whether the synthesis holds. The willingness to do that work is itself an instance of the decision type this article is arguing for.

Where do you stand?

Before moving on, a pause.

Not to evaluate the argument. To turn it inward.

Ask yourself, honestly:

When I encounter a problem at civilizational scale, climate, AI governance, democratic erosion, what is my default response? Do you instinctively calculate your causal contribution and find it negligible? Do you feel the pull toward disengagement, framed as realism? Do you notice yourself waiting for someone with more leverage, more visibility, more resources to act first?

If yes, that is causal thinking doing what it always does. It is not a character flaw. It is a trained reflex.

Now ask a different question:

What kind of agent am I, right now, in this moment?

Not in the abstract. In the concrete decisions of this week. Do you ask hard questions about the AI tools you use, or do you accept their outputs without interrogation? Do you hold the organizations you work with accountable for their decision structures, or do you defer to their expertise? Do you stay engaged with problems that feel too large, or do you quietly withdraw?

There is no correct answer to perform here. The point is not to conclude that you are the right kind of agent, or the wrong kind. The point is to notice your actual decision type, the one you are currently instantiating, not the one you intend to be.

That noticing is where change begins. Not in a grand gesture, but in the gap between the agent you are and the agent you choose to be.

That gap is yours. It always was.

References and futher readings

Core decision theory and logical correlation

• Functional Decision Theory (FDT) – Yudkowsky, E., & Soares, N. (2017). Functional Decision Theory: A New Theory of Instrumental Rationality. Machine Intelligence Research Institute. → arXiv abstract: https://arxiv.org/abs/1710.05060 → PDF: https://arxiv.org/pdf/1710.05060.pdf → MIRI announcement: https://intelligence.org/2017/10/22/fdt/

What it supports:

The distinction between causal thinking and FDT’s view of decisions as outputs of a decision function, and the claim that agents with similar decision structures are logically correlated rather than just causally connected.

• Logical Induction – Garrabrant, S. et al. (2016). Logical Induction. Machine Intelligence Research Institute. → arXiv abstract: https://arxiv.org/abs/1609.03543 →PDF: https://arxiv.org/pdf/1609.03543.pdf

What it supports:

Provides a formal framework for reasoning under logical uncertainty, reinforcing the use of logical (not causal) dependence between agents’ reasoning processes as a serious technical idea, not just a metaphor.

Existential risk and civilizational trajectory

• The Precipice – Ord, T. (2020). The Precipice: Existential Risk and the Future of Humanity. Bloomsbury / Hachette.→ Author’s book page: http://www.tobyord.com/book →Overview + resources (80,000 Hours): https://80000hours.org/the-precipice/ → Wikipedia summary: https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity

What it supports:

The use of the 1-in-6 existential-catastrophe estimate, the idea that we are in a period of exceptional vulnerability, and the framing of some present actions as existential risk factors or existential safety factors at civilizational scale.

• Existential risks as policy / governance problem – Policy Horizons Canada. (2023). Risques existentiels mondiaux..→ PDF (French): https://horizons.service.canada.ca/resources/wp-content/uploads/2023/06/Risques-existentiels-mondiaux.pdf .→ Publication page: https://publications.gc.ca/site/eng/9.922900/publication.html

What it supports:

Shows a government body explicitly framing existential risks as objects of collective decision-making and resource allocation, backing your talk of civilizational risk factors and a “window” where decisions disproportionately shape long-term trajectories.

• Future of Life Institute – Existential Risk & Governance – Future of Life Institute. (n.d.). Existential Risk (AI / diplomacy / governance focus). ..→ Entry point: https://futureoflife.org . → AI existential risk page (check exact path): https://futureoflife.org/ai/existential-risk/

What it supports:

The claim that reducing existential risk depends on many actors adjusting their behaviour (researchers, policymakers, institutions) rather than a single centralized intervention, and that existential risk is a problem of diplomacy and governance.

Civilizational virtues and plural subjects

• Civilizational virtues / ethics of existential risk – Effective Altruism Forum. (various). Ethics of Existential Risk (topic: “civilizational virtues”). → Topic hub: https://forum.effectivealtruism.org/topics/ethics-of-existential-risk

What it supports:

Summarizes Toby Ord’s five moral foundations, including civilizational virtues as a distinct category of values that only make sense at the scale of humanity, directly backing the use of the term and the list (truth-seeking, accountability, refusal to delegate vigilance) as virtues that are collectively instantiated.

• Anders Sandberg – Narratives, values, and long-term futures – Sandberg, A. (talk). Narratives, Values & Progress. → Talk page / transcript (approximate canonical URL): http://www.scifuture.org/narratives-values-progress-anders-sandberg/ → Video (YouTube): https://www.youtube.com/watch?v=9n_JN9s7LZo

What it supports:

Sandberg’s argument that navigating transformative technologies and existential risks requires shared narratives, values, and virtues at the level of humanity, not just individuals, the backbone of the use of “civilizational virtues” and “civilizational autonomy.”

• Plural subject theory – Gilbert, M. (1989). On Social Facts. Princeton University Press. → Publisher page (approximate):
  https://press.princeton.edu/books/ebook/9780691214627/on-social-facts-pdf-0 → Overview via Google Scholar:
  https://scholar.google.com/citations?user=WL99bysAAAAJ&hl=en – Secondary summary: → Internet Encyclopedia of Philosophy – Collective Intentionality (section on Gilbert): https://iep.utm.edu/collective-intentionality/ → Daniel Little, “Gilbert on social facts” (blog): https://undsoc.org/2019/06/14/gilbert-on-social-facts/

What it supports:

The claim that micro-agents, through joint commitments, can constitute a plural subject, a “we” that is a genuine subject of beliefs, intentions, and obligations, and the move from “civilization as sum of individuals” to “civilization as the emergent structure of coordinated decision types.”

Aggregation and social choice

• Social choice / aggregation of preferences – French “state of the art” report on decision theories and social choice: État de l’art sur les théories de la décision et méthodologies de l’approche système (Temis). → PDF: https://temis.documentation.developpement-durable.gouv.fr/docs/Temis/0073/Temis-0073080/RMT03-018_2.pdf

What it supports:

The statements that individual preferences and behaviours, aggregated, structure macro-level outcomes and that the composition of agent types in a system determines its trajectory even when each agent is “small”, i.e. the formal underpinning of “aggregates of micro-agents” shaping civilizational behaviour.

AI governance and accountability

• Review article on accountability in AI systems – Billy Gareth, Why Accountability Is the Hardest Problem in AI Systems Today (LinkedIn, 2026). → Publication page: https://www.linkedin.com/pulse/why-accountability-hardest-problem-ai-systems-today-its-billy-gareth-h0nrf

What it supports:

AI creates a diffusion of responsibility across data collection, training, system integration, and usage; accountability must be designed at the system level, incorporating mechanisms for traceability, clear ownership, human override, etc.

• Conceptual article on Ethical Governance of Human-AI Decision-Making – Mustafa KAYA, Human-AI Decision-Making Ethical Governance Framework (2025). → Publication page: https://yzib.com.tr/index.php/yzib/article/view/22

What it supports:

Argues that systems must integrate human-centered design, continuous oversight, and ethical governance; asserts that human intuition/judgment remains the “moral compass” even in highly algorithmic systems.

• On accountability as a systemic issue rather than “Bad actors” – Dylan Miyake, Accountability Isn’t a People Problem — It’s a System Problem. →Publication page: https://www.clearpointstrategy.com/blog/accountability-isnt-a-people-problem

What it supports:

While set in a management/strategy context, the reasoning regarding the need to structure “accountability by design” is directly applicable to AI.