PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents

Paper: 2604.08529 Authors: Zhiyuan Wang, Erzhen Hu, Mark Rucker, Laura Barnes (University of Virginia) Categories: cs.HC

The Integration Problem

AI coding agents can now generate personalized tools from natural-language requests. Need a health tracker? Describe it, get an app. Want parking automation? Ask, receive code. The generation problem is increasingly solved.

But generation creates a new problem: fragmentation at scale. Each generated artifact becomes another siloed app. Your health data lives in one tool, parking history in another, calendar in a third. When you ask “Is my heart rate too high right now?”—a simple situated question—the answer requires combining recent workout activity, current motion, sleep quality, and contextual signals like whether you’re rushing to an expiring parking spot.

The difficulty isn’t that any single signal is unavailable. It’s that these signals remain fragmented across apps, services, and interfaces. Personal software scales in quantity rather than coherence.

Existing solutions don’t solve this:

• Chat-only agents (ChatGPT, Claude) can reason but don’t maintain persistent state or GUI instruments
• Smart assistants (Siri, Google) have limited context and no cross-app synthesis
• Automation tools (Shortcuts, IFTTT) connect apps but don’t enable reasoning
• Code generation tools (v0, Cursor) create isolated artifacts without integration contracts

PSI: Instruments, Not Apps

PSI introduces a shared-state architecture that turns independently generated modules into instruments—persistent, connected, chat-complementary artifacts. An instrument is:

1. Persistent: Remains available without regeneration, with a glanceable GUI for monitoring
2. Connected: Publishes state to a shared personal-context bus and exposes write-back affordances
3. Chat-complementary: Supports visual monitoring while chat handles synthesis, ambiguity resolution, and stateful actions

The key insight: both persistent GUI instruments and a generic chat agent (Facai) operate over the same shared-context bus. Integration becomes a local obligation—a new module needs only to implement the provider interface and register, rather than wire into every existing module.

Architecture

PSI consists of three layers:

1. Generation Layer

Modules are generated through a multi-phase pipeline: specification, code generation, auto-fix, and compile verification. Each generated module must satisfy a single Swift protocol (ToolkitDataProvider) requiring:

• A toolkit identifier
• Relevance keywords
• One method: buildContextSummary() -> String?

This method returns a tagged, human-readable snapshot of the module’s current state (today’s sensed events, recent meals, upcoming calendar entries) together with any write-back endpoints it exposes. The same method serves both read context and action discovery for the chat agent.

2. Shared Personal-Context Layer

A central registry collects current-state snapshots from all registered modules and prepends them as a single tagged block before every chat message. Key properties:

• Person-scoped: State persists on-device across tasks and sessions
• Today-focused: Each snapshot captures today’s data rather than cumulative history
• Graceful degradation: If a module has nothing to report, it’s silently omitted
• LLM-mediated: Modules never read each other’s state directly; all cross-module communication is mediated by the LLM through assembled context

Unlike agent orchestration frameworks where state is scoped to a single task, PSI’s shared context requires no coordinating task graph and is consumed by all interfaces.

3. Interaction Layer: Dual-Modality Use

Two coordinated interfaces:

• Persistent GUI: Glanceable monitoring, customizable dashboards, interactive timelines
• Generic chat agent (Facai): Cross-context reasoning, ambiguity resolution, write-back actions

PSI’s contribution is not simply the coexistence of chat and GUI, but their role as synchronized entry points to the same person-scoped mutable state. Users can inspect state in a persistent instrument, revise it through chat, and immediately verify the effect in the GUI without duplicated state paths.

Evidence: Three-Week Autobiographical Deployment

PSI was evaluated through a three-week autobiographical deployment in RyanHub, a self-developed personal AI environment comprising:

• SwiftUI iOS app (94 Swift files, 36,517 LOC) hosting persistent GUI instruments and chat interface
• Python bridge server providing unified REST gateway for on-device data
• Local JSON persistence for behavioral timeline, health entries, parking state
• Python dispatcher maintaining WebSocket sessions with iOS client

14 modules created across behavioral sensing, health, scheduling, parking, reading, vocabulary learning, and self-tracking domains. Six core modules used during deployment, eight additional modules generated post-pilot to validate extensibility.

Case Study 1: BoBo (Behavioral Observer Bot)

A generalized behavioral sensing instrument that connects all health and motion sensors. Demonstrates how PSI supports persistent behavioral sensing beyond single question-answer interactions.

Capabilities enabled by shared context:

• Cross-signal synthesis: relating heart rate spikes to location, activity, and calendar load
• Temporal grounding: comparing sleep or recovery trends over multiple days
• Action-oriented follow-ups: recommending rest, suppressing evening workouts, adjusting schedules

Example interaction:

• User: “Why do I feel so drained lately?”
• Facai retrieves state from BoBo, Health module, Calendar through shared context bus
• Discovers: rushing between meetings, exercising too much, poor sleep the night before
• User can verify explanation by inspecting BoBo’s persistent timeline GUI

Case Study 2: Automated Parking

A hyper-personal, market-of-one workflow tailored to a single user’s weekday parking routine. Automatically books parking before 7 a.m. to avoid lot filling up.

Dual-modality interaction:

• Chat: “No parking this Thursday” → Facai triggers state change
• GUI: Toggle skip state directly in persistent interface
• Both operate over same persistent parking state (schedule, purchase history, active sessions)

Cross-module integration:

• Uses calendar’s end-of-day event to infer parking duration
• Integrates with location data to verify parking zone
• Pattern generalizes to gym bookings, commute ticketing, medication reminders

Evaluation: Shared Context vs. Baselines

Three conditions tested on 50 synthetic user queries across frozen three-week dataset:

1. Shared Personal-Context: Full PSI with preassembled personal snapshot
2. Search-Only: Agent recovers relevant state opportunistically from file system (comparable to OpenClaw)
3. Single-Module: Best-rated one-module variant per task

Reasoning Tasks (Cross-Module Synthesis)

Condition	Mean Fulfillment	Task Success
Shared Context	0.88	0.68
Search-Only	0.63	0.32
Single-Module	0.27	0.08

Shared context substantially outperforms both baselines. Single-module performance is particularly poor (0.08 success rate), confirming that cross-module synthesis is essential for situated personal questions.

Write-Back Actions (GUI State Changes)

Condition	Validated Changes	Success Rate
Shared Context	19/20	95%
Single-Module	19/20	95%
Search-Only	8/20	40%

Shared context and single-module both achieve 95% success on write-back actions, while search-only achieves only 40%. The pattern reveals that shared context provides write-path discovery that ad hoc search cannot reliably achieve.

Latency

Mean successful latency for reasoning tasks:

• Shared Context: 25s
• Search-Only: 29s
• Single-Module: 23s

End-to-end latency is not monotonic with context size. The overhead of preassembled context is offset by reduced search and retrieval time.

Key Insights

1. Shared state enables cross-module reasoning: For reasoning tasks, shared context enables synthesis that no single module can provide. The gap between shared context (0.68 success) and single-module (0.08 success) is dramatic.

2. Shared state provides reliable write-path discovery: For actions, shared context provides write-back affordances that ad hoc search cannot reliably achieve (95% vs. 40% success).

3. Bidirectional information access: Shared context lets chat both read personal state and modify it through discovered endpoints, with changes immediately visible in persistent GUIs.

4. Person-scoped state persists across sessions: Unlike task-scoped agent orchestration, PSI’s shared context is person-scoped and persists on-device, enabling longitudinal reasoning over days and weeks.

5. Integration is a local obligation: A new module needs only to implement the provider interface and register, rather than wire into every existing module. This scales to heterogeneous tool ecosystems without pairwise integration.

Implications

For personal AI developers: Generation alone is insufficient. The missing layer is a shared-state substrate that lets independently generated modules interoperate. PSI demonstrates that a minimal integration contract (one protocol, one method) is sufficient to enable coherent personal computing environments.

For researchers: Autobiographical deployment reveals design tensions that controlled studies miss. The three-week deployment surfaced the importance of today-focused snapshots (vs. cumulative history), graceful degradation (silent omission of empty modules), and dual-modality interaction (persistent GUI + chat over same state).

For users: Personal AI should not force you to manually integrate fragmented apps. PSI shows that generated tools can become instruments—persistent, connected, chat-complementary artifacts that support both glanceable monitoring and cross-context reasoning.

Limitations and Future Work

• Single-user deployment: Evidence comes from one technically skilled user in a self-developed environment. Population-wide adoption requires studying diverse users and use cases.

• On-device only: All services run on localhost; personal data stays on-device by default. Cloud integration and multi-device sync remain open challenges.

• LLM-mediated communication: Modules never read each other’s state directly; all cross-module communication is mediated by the LLM. This design choice trades direct API calls for flexibility but introduces latency and LLM dependency.

• Today-focused snapshots: Current design captures today’s data rather than cumulative history. Longitudinal reasoning over weeks/months requires additional memory mechanisms.

PSI identifies shared state as the missing systems layer that transforms AI-generated personal software from isolated apps into coherent personal computing environments. The architecture, evaluation, and open-sourced artifact provide a foundation for future work on personal AI integration.