It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces

Paper: 2606.24854
Authors: Blade Frisch, Will Wade, Dylan Gaines, Michelle Kinsella, Betts Peters, Tamara Broderick, Keith Vertanen
Categories: cs.HC, cs.AI

The Gap

Most AAC evaluation research treats users as interchangeable subjects, measuring speed and accuracy on standardised tasks. Even fine-grained metrics like keystroke savings or error correction time assume a single optimal path. But real AAC users differ in motor ability, language background, visual processing, and social context — all at once. The classic approach (e.g., Fitts’ law models for cursor control, or language model perplexity for prediction) strips away these intersections, producing numbers that look objective but fail to predict whether a teenager with cerebral palsy will actually use an interface in real conversation.

This paper doesn’t fix evaluation by proposing a better metric. Instead, it shows that the very space of evaluation needs to be rethought: what we measure, for whom, and under what conditions.

[Problem: Standard metrics miss user differences]
    |
    v
[Assumption: A single-number metric can capture usability]
    |
    v
[Method: Analyse 6 AAC problem spaces, identify intersectional factors]
    |
    v
[Evidence: Each space reveals contradictions when evaluated uniformly]
    |
    v
[Conclusion: Evaluation must be multi-dimensional and context-aware]

The Increment

One sentence: Before this paper, evaluation of AI-powered AAC was a hunt for a universal score; after it, evaluation becomes a map of intersecting dimensions that must be navigated per user.

Core Mechanism

The paper doesn’t build a system — it builds a framework for evaluation design. It starts by decomposing AAC into six problem spaces: text entry, word prediction, image communication, social scripting, multimodal input, and customisation. For each space, they ask: what does “good” mean from the perspective of a specific user with a specific set of abilities and goals? They then surface those perspectives using a combination of structured interviews, scenario walkthroughs, and co-design activities.

Data flow is not a pipeline but an iterative discovery process: user characteristics (motor, cognitive, linguistic, social) feed into scenario authoring, scenarios are enacted with AI prototypes, behavioural and subjective data are collected, and then the team reflects on what traditional metrics would have missed. The “operation” is a cross-examination: for each scenario, they ask whether a standard metric (e.g., words per minute) would rank the AI helpfully, harmfully, or irrelevantly.

[User Profiles] --> [Scenario Generator] --> [AI Prototype Interaction]
                       |                          |
                       v                          v
                 [Behavioural Logs]         [Subjective Feedback]
                       |                          |
                       +----------->[Reflection]<-+
                                     |
                                     v
                            [Gap Analysis: what metric would have missed]

Now the structural metaphor: think of this paper as a fitted suit tailor. Traditional evaluation is like a ready-to-wear suit — one size fits most, but pinches at the shoulders, baggy at the waist, and doesn’t accommodate a left-handed sword sheath. The six problem spaces are the body measurements: shoulder width, sleeve length, chest circumference. Each scenario is a specific posture: sitting, bowing, riding a horse. The AI prototypes are trial garments. The reflection step is the tailor pinning and marking where the typical pattern falls short. What emerges is not a single perfect suit, but a pattern-making process that lets you cut fabric for any body. The “intersectional” insight is that you need all six measurements taken simultaneously because a short torso and long arms require a different cut than a long torso and short arms — you can’t average them.

Key Concepts

• Intersectionality (in HCI): People don’t have disabilities in neatly separate buckets. A user might have cerebral palsy (motor), be a native ASL user (language), and live in a rural area with low internet bandwidth (environment). Standard metrics treat these as additive noise; intersectionality says they interact to create qualitatively different use cases. Example: word prediction speed measured in milliseconds is useless if the user’s motor plan takes three seconds per tap — the bottleneck isn’t prediction latency but the physical interface itself.

• Ecological validity: A metric is ecologically valid if it predicts performance in actual daily use, not just in a lab with minimal distractions. Many AAC studies test users on isolated typing tasks; real conversations involve turn-taking, emotion, background noise, and fatigue. The paper argues that without ecologically valid evaluation, AI improvements may be irrelevant or even harmful in practice. Example: an AI that suggests longer phrases improves words-per-minute in a lab but frustrates a user who wants to interrupt and correct.

• Evaluation elicitation: Instead of defining metrics first, this paper elicits what matters from stakeholders (users, caregivers, SLPs) through iterative scenarios. This flips the usual order: rather than asking “does the system meet this metric?”, they ask “what metric would capture whether this system helps *this person?”.

Framework Shift

Before (mainstream approach):        After (this paper):
+------------------+                 +------------------+
| Define metric    |                 | Analyse user     |
| (e.g., WPM)      |                 | intersectionally |
+--------+---------+                 +--------+---------+
         |                                    |
         v                                    v
+------------------+                 +------------------+
| Run controlled   |                 | Design scenarios |
| lab study        |                 | for each space   |
+--------+---------+                 +--------+---------+
         |                                    |
         v                                    v
+------------------+                 +------------------+
| Compare scores   |                 | Elicit what      |
| across users     |                 | good looks like  |
+--------+---------+                 +--------+---------+
         |                                    |
         v                                    v
+------------------+                 +------------------+
| Publish "AI      |                 | Derive custom    |
| improves 15%"?   |                 | evaluation plan  |
+------------------+                 +------------------+

One sentence: From measuring everyone the same way and averaging to designing the measurement around each person’s intersections, the core shift is decentralising the evaluation authority to the user-interface dyad rather than fixing it in a global benchmark.

Expert Assessment

Problem choice: Genuine gap. The HCI community has known for a decade that average metrics hide outliers, but AAC is a domain where outliers are the norm (each user’s impairment profile is unique). This paper doesn’t pretend to solve it but systematically lays out why it’s hard, which is valuable.

Method maturity: More conceptual than technical. The paper is a position piece with illustrative examples rather than a novel algorithm. That’s appropriate for the gap — before you can build a better metric, you need to know what “better” means. But the paper stops short of providing a concrete protocol or toolkit. It’s an insight piece, not a implementation.

Experimental integrity: N/A – no controlled experiment. The paper uses qualitative examples and reasoned argument. The strength is in the clarity of the argument; the weakness is the lack of empirical demonstration that their proposed evaluation method yields better outcomes than existing ones.

Writing quality: Well-structured, pulls no punches in calling out the inadequacy of current practice. The six problem spaces are clearly delineated. However, the “suggest more robust methods” part is thin — they describe principles but don’t give enough detail for a practitioner to apply them tomorrow. Rewriting the “Evaluation Framework” section with a worked example would double the paper’s impact.

Verdict: weak accept — A necessary wake-up call with strong framing, but lacks the prescriptive detail to change practice on its own.

Takeaways

• Use scenario elicitation instead of top-down metric definition when evaluating AI in high-variance settings (not just AAC, but any assistive tech).
• Create an “intersectional profile” for each evaluation participant: gather motor, cognitive, linguistic, and environmental dimensions explicitly.
• Replace single-number summary statistics with multi-dimensional utility maps that capture trade-offs (e.g., speed vs. accuracy vs. frustration).
• When writing a paper, include a “what our metric would have missed” section — it forces reflection and increases credibility.