How Language Models Organize Moral Knowledge

Paper: 2608.27402 Authors: Orion Reblitz-Richardson Categories: cs.AI, cs.CL, cs.LG

The Gap

Models detect moral content broadly. That is a low bar, and it is the bar most work clears. Detecting that something is a moral statement is compatible with a model that has one undifferentiated “morality” direction, with no structure among the categories it supposedly distinguishes.

The sharper question is whether a model goes further: does it distinguish moral foundations from one another, and does it organize the relationships between them in a way that can be described? Answering it requires more than probing for the presence of a concept — it requires measuring the geometry of several concepts at once and asking what shape they form.

This matters beyond taxonomy. If a model represents morality as one axis, then it cannot represent disagreement between two moral considerations; there is no tension, only intensity. And if it can represent tension, then its internal representation is doing something closer to what moral reasoning actually involves — holding competing considerations unresolved rather than storing a verdict.

   THE LOW BAR AND THE HIGHER ONE

   LOW BAR: does the model DETECT moral content?
        -> compatible with ONE undifferentiated
           "morality" direction
        -> no structure among categories
        |
        v
   HIGHER BAR: does the model
     - DISTINGUISH the foundations from each other?
     - ORGANIZE their relationships describably?
        -> requires measuring the GEOMETRY of several
           concepts at once
        |
        v
   WHY IT MATTERS
     if morality is one axis:
        no representation of TENSION between two
        moral considerations -- only intensity
     if tension is representable:
        the representation does something closer to
        moral reasoning: holds considerations
        UNRESOLVED rather than storing a verdict

The Increment

One sentence: Before this paper, “the model detects moral content” was the state of knowledge; after it, linear probes over six moral foundations show a shared-plus-independent geometry that is moral-specific and that represents moral tension rather than a resolved judgment.

Core Mechanism

The design is intended to be readable off the shelf: six independent linear probes, one per Moral Foundations Theory category (care/harm, fair/cheat, lib/oppress, loy/betray, auth/subv, sanc/degrade), trained on open-weight models. Then the interesting object is not any single probe’s accuracy but how the resulting directions relate to each other in representation space.

The central finding is a structure that sits between two obvious possibilities. The six directions neither collapse into a single moral detector nor isolate from one another. They span a near-maximal number of independent dimensions — so the foundations are genuinely distinguished, not facets of one axis — while sharing a positive common component — so they are not six unrelated detectors bolted together. That shared component is read as the signature of integration: a single representation that all moral categories participate in, sitting alongside the dimensions that separate them.

The control is what makes “shared component” mean something moral rather than something generic. A matched non-moral concept battery, built identically, gives a mean pairwise cosine of 0.013, against 0.26 for the moral foundations. So the shared component is not an artefact of how linear probes behave on any concept set — it is a property specific to moral concepts.

Three further results widen the finding:

The geometry is consistent across architectures and scale, and it reaches its integration regime early in pre-training, well before probe accuracy saturates. That ordering is informative: the structure appears before the probes become accurate, which means the organisation is not a byproduct of good probing performance.

It shows no evidence of the individualizing/binding distinction predicted by Moral Foundations Theory — with an honest caveat attached, since the test is underpowered: only 20 candidate partitions exist. Instead the discovered structure reflects corpus statistics. Reporting this is a good sign, because it is a negative result against the very theory supplying the categories.

Moral dilemmas decompose. Each dilemma direction partially composes from its component foundations at 2.7× a mismatched-pair baseline, while the majority of its variance encodes conflict-specific structure. This is the paper’s most conceptually interesting result. A dilemma is not stored as its resolution; most of what the model represents about a dilemma is the conflict itself, with only part attributable to the foundations in play. The model represents moral tension rather than a pre-resolved judgment.

   SIX PROBES, ONE GEOMETRY

   probes: care/harm, fair/cheat, lib/oppress,
           loy/betray, auth/subv, sanc/degrade
                    |
                    v
   HOW DO THE SIX DIRECTIONS RELATE?
     NOT collapsed into one moral axis
     NOT isolated from each other
        -> span NEAR-MAXIMAL independent dimensions
        -> PLUS a shared positive component
                    |
                    v
   CONTROL: matched NON-moral concept battery
     mean pairwise cosine  0.013
     moral foundations     0.26
        -> the shared component is MORAL-SPECIFIC
                    |
                    v
   TIMING: integration regime reached EARLY in
     pre-training, BEFORE probe accuracy saturates
        -> organisation is not a byproduct of good probing
                    |
                    v
   DILEMMAS: each direction partially composes from its
     component foundations (2.7x mismatched baseline),
     while MOST of its variance encodes CONFLICT-SPECIFIC
     structure
        -> the model represents the TENSION, not a verdict

Think of it as asking whether someone’s moral vocabulary is a single dial or a set of independent gauges. A single dial is easy to test: turn up the intensity and everything moves together, and there is no way to express “these two values pull in opposite directions”. Six independent gauges would mean someone who can register harm but whose fairness reading is unrelated to anything — no integration, no shared sensibility. What the probes find is neither: a set of distinguishable gauges that all sit on one panel, sharing a common bias. That is exactly the configuration required to represent a dilemma, because you need two things that differ and a common medium in which they can conflict. The finding that most of a dilemma’s representation is conflict-specific is the equivalent of discovering that the panel’s readout at tension points is not an average of the gauges but a depiction of the needle pulling against the spring.

Key Concepts

• The three-way structural finding: not collapsed, not isolated, near-maximal independent dimensions plus a shared component. Each alternative would have been a simpler story, and the paper establishes the middle configuration with a control that rules out the generic explanation.
• The matched non-moral battery as control: a concept set built identically, yielding 0.013 against 0.26. Without it, a shared component among moral probes could reflect how probe directions behave on any concept battery, and the moral-specificity claim would not stand.
• Integration early in pre-training: the shared component appears before probe accuracy saturates. It suggests the organisation is learned as part of representation formation rather than being an artefact of the model becoming good at the probe task.
• Conflict-specific variance in dilemmas: the majority of a dilemma direction’s variance is not attributable to its component foundations but to the conflict itself. It is the direct evidence for representing tension rather than a verdict, and it is the kind of claim that would be invisible to probe-accuracy evaluation.

Framework Shift

Before (detection as the question):
  "does the model detect moral content?"
  -> yes/no, and compatible with one undifferentiated axis
  -> tension between moral considerations unrepresentable

After (geometry as the question):
  six foundations, six probes, measured relationships
  -> distinguishable AND integrated
  -> shared component is moral-specific (0.26 vs 0.013)
  -> dilemmas carry conflict-specific variance, 2.7x baseline
  -> the model represents unresolved tension

From asking whether a model recognises morality at all, to asking what shape its moral concepts form relative to each other, the core shift is that the interesting structure is in the relations between concepts, and it is measurable.

Expert Assessment

Problem choice: Very good, and it asks the question that the detection framing forecloses. Moving from “does the model have a moral concept” to “what geometry do the moral concepts form” is what makes tension representable as a finding at all, and the dilemma result justifies the effort.

Method maturity: The design is simple — linear probes plus cosine geometry — and that simplicity is appropriate, because the claims are structural and a complicated method would have obscured them. The matched non-moral battery is the load-bearing methodological choice, and building it identically is what makes the 0.26-versus-0.013 comparison meaningful rather than decorative. The honest weakness is that the theory test is genuinely underpowered, and the paper says so.

Experimental integrity: The negative result against Moral Foundations Theory is the most creditable element, because the theory is the source of the categories being probed — finding that the model’s structure reflects corpus statistics rather than the theory’s individualizing/binding split is a result against the author’s own framing. Reporting the underpowered caveat rather than quietly dropping the test is the right call. The main scope limitation is that probes measure decodability, which is weaker than causal use; that the directions exist does not establish that the model’s behaviour depends on them in the way the tension reading implies.

Writing quality: The three-way structure — not collapsed, not isolated, shared component — is stated compactly and is the right organising idea. A single annotated diagram of the geometry, showing the common component alongside the independent directions, would make the central result immediate; the reported cosine values carry it for a specialist but not for the wider audience this result deserves.

Verdict: strong accept — a clean geometric characterisation of moral representation with a properly matched control, an honest negative result, and a dilemma finding that distinguishes representing tension from storing a verdict.

Takeaways

• Ask what geometry a set of concepts forms, not just whether the model detects them. The relations between concepts are where representational structure becomes visible.
• Build a matched non-moral control whenever you claim a concept-specific structure. Without one, a shared component among probes may just be how probes behave.
• Treat an underpowered test as a reported limitation rather than a result to omit. The negative finding here is one of the paper’s most useful contributions.
• Distinguish representing tension from representing a verdict. A model that stores only resolutions cannot be reasoned with about disagreement, and conflict-specific variance is how you check which one you have.