Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset

Paper: 2602.23335 Authors: Dany Haddad, Dan Bareket, Joseph Chee Chang, Jay DeYoung, Jena D. Hwang, Uri Katz, Mark Polak, Sangho Suh, Harshit Surana, Aryeh Tiktinsky Categories: cs.HC, cs.AI, cs.IR

Abstract

This paper presents the Asta Interaction Dataset, a comprehensive collection of over 200,000 user queries and interaction logs from deployed AI-powered scientific research tools. The dataset captures real-world usage patterns from two interfaces: a literature discovery tool and a scientific question-answering system, both built on an LLM-powered retrieval-augmented generation (RAG) platform. The analysis reveals fundamental shifts in how researchers interact with AI assistants compared to traditional search engines, including longer and more complex queries, treatment of the system as a collaborative research partner, and non-linear engagement with generated content. The study tracks how usage patterns evolve with user experience and introduces a new query intent taxonomy to support future research assistant design and evaluation.

Key Contributions

• Release of a large-scale, anonymized dataset containing 200,000+ real-world user interactions with AI research tools
• Empirical characterization of query patterns showing users submit significantly longer and more complex queries than in traditional search systems
• Evidence that users delegate high-level research tasks (content drafting, gap identification) to AI assistants, treating them as collaborative partners
• Analysis of non-linear engagement patterns where users revisit and navigate among generated responses and citations as persistent artifacts
• Documentation of how usage evolves with experience: more targeted queries and deeper citation engagement, though keyword queries persist
• Introduction of a novel query intent taxonomy specifically designed for scientific research assistance contexts

Dataset Characteristics and Methodology

The Asta Interaction Dataset captures interactions from two distinct interfaces within a unified RAG platform. The literature discovery interface helps researchers find relevant papers, while the question-answering interface provides direct answers to scientific questions with supporting citations. Both systems leverage large language models combined with retrieval mechanisms to access scientific literature.

The dataset spans multiple months of deployment and includes diverse interaction signals: query text, query length and complexity metrics, response engagement patterns (time spent, scrolling behavior, citation clicks), and longitudinal user behavior across sessions. Users are segmented by experience level based on cumulative interaction count, enabling analysis of how research workflows adapt over time.

The authors employ mixed-methods analysis combining quantitative metrics (query length distributions, engagement time, citation click-through rates) with qualitative coding of query intents. This dual approach provides both statistical rigor and interpretive depth, revealing not just what users do but why they interact with AI research tools in particular ways.

Query Patterns and User Intent

Analysis reveals that queries submitted to AI research assistants differ fundamentally from traditional search engine queries. The median query length is substantially longer, with users frequently submitting full sentences or even paragraph-length requests. This suggests users perceive the system as capable of understanding complex, nuanced information needs rather than requiring keyword optimization.

The introduced query intent taxonomy identifies several distinct categories:

• Information seeking: Direct questions about scientific concepts or findings
• Literature discovery: Requests to find papers on specific topics or with particular characteristics
• Content generation: Delegation of writing tasks like summarizing findings or drafting explanations
• Research planning: Higher-level requests about research directions, gaps, or methodological approaches
• Citation verification: Checking claims or seeking supporting evidence for specific statements

Notably, content generation and research planning queries—categories largely absent in traditional search—constitute a significant portion of interactions. This indicates users view AI assistants as active collaborators in the research process, not merely information retrieval tools.

Engagement Behaviors and Response Interaction

Users exhibit sophisticated engagement patterns with generated responses that differ markedly from traditional search result interaction. Rather than quickly scanning and moving on, users spend extended time with AI-generated content, treating responses as artifacts worth careful examination.

The data reveals non-linear navigation patterns: users frequently return to previous responses, compare multiple generated answers, and navigate between main content and supporting citations in complex sequences. This behavior suggests users are synthesizing information across multiple AI-generated outputs rather than treating each response as independent.

Citation engagement is particularly noteworthy. Users don’t simply accept generated content at face value but actively verify claims by clicking through to source papers. The click-through rate on citations is substantially higher than typical academic search systems, indicating users value the transparency of seeing supporting evidence and use it to assess response quality.

Response persistence is another key finding. Users bookmark, revisit, and build upon previous interactions, suggesting they integrate AI-generated content into their ongoing research workflows. The system functions less like a search engine (ephemeral queries and results) and more like a research notebook (accumulated, revisited artifacts).

Evolution of Usage with Experience

Longitudinal analysis tracking individual users over time reveals how interaction patterns evolve with experience. Novice users (first few sessions) submit more exploratory, broad queries and engage somewhat superficially with responses. As users gain familiarity, several shifts occur:

Query targeting improves: experienced users formulate more specific, focused questions that leverage understanding of system capabilities. They learn to phrase requests in ways that elicit higher-quality responses, suggesting a co-adaptation process between user and system.

Citation engagement deepens: experienced users click through to supporting papers more frequently and spend more time examining cited evidence. This indicates growing sophistication in evaluating AI-generated content and integrating it with primary literature.

However, keyword-style queries persist even among experienced users, appearing alongside more complex natural language requests. This suggests users maintain multiple query strategies, selecting approaches based on task context rather than fully abandoning familiar search patterns.

The learning curve appears relatively quick—significant behavioral changes emerge within the first 10-20 interactions—but continues evolving over hundreds of queries, indicating both immediate adaptation and longer-term workflow integration.

Implications for AI Research Assistant Design

The findings carry several important implications for designing future AI-powered research tools:

Support for complex queries: Systems should be optimized for longer, more nuanced inputs rather than assuming keyword-based queries. Interface design should encourage natural language expression of complex information needs.

Collaborative framing: Rather than positioning AI as a search tool, interfaces should support delegation of higher-level research tasks. Features for content generation, research planning, and gap identification should be first-class capabilities.

Response persistence and organization: Since users treat outputs as persistent artifacts, systems need robust mechanisms for saving, organizing, and revisiting generated content. Integration with research workflow tools (reference managers, note-taking systems) becomes critical.

Citation transparency and verification: Prominent, easily accessible citations are essential. Users actively verify claims, so systems must make source evidence readily available and support efficient navigation between generated content and supporting literature.

Adaptive interfaces: Given the evolution of usage patterns with experience, interfaces might adapt to user sophistication, offering more advanced features or different interaction paradigms as users demonstrate growing expertise.

Multi-modal interaction support: The persistence of keyword queries alongside natural language suggests value in supporting multiple query modalities rather than forcing users into a single interaction paradigm.

Takeaways

1. Users submit significantly longer and more complex queries to AI research assistants than to traditional search engines, indicating they perceive these systems as capable of understanding nuanced information needs.

2. Researchers treat AI assistants as collaborative partners, delegating high-level tasks like content drafting and research gap identification rather than using them solely for information retrieval.

3. Generated responses function as persistent research artifacts that users revisit, compare, and build upon over time, requiring systems to support non-linear navigation and content organization.

4. Citation engagement is substantially higher than in traditional academic search, with users actively verifying AI-generated claims by examining supporting evidence, highlighting the importance of transparency.

5. Usage patterns evolve with experience toward more targeted queries and deeper citation engagement, though keyword-style queries persist even among experienced users, suggesting value in supporting multiple interaction modalities.

6. The Asta Interaction Dataset provides a crucial empirical foundation for understanding real-world AI research assistant usage, enabling more realistic evaluation and design of future systems.