FAMOSE: A ReAct Approach to Automated Feature Discovery

Paper: 2602.17641 Authors: Keith Burghardt, Jienan Liu, Sadman Sakib, Yuning Hao, Bo Li Categories: cs.LG, cs.AI

Abstract

Feature engineering has long been recognized as a critical bottleneck in machine learning pipelines, particularly for tabular data where the optimal feature space grows exponentially and traditionally requires substantial domain expertise. FAMOSE (Feature AugMentation and Optimal Selection agEnt) addresses this challenge by introducing the first agentic ReAct framework specifically designed for automated feature engineering across both regression and classification tasks. The system autonomously explores, generates, and refines features while integrating feature selection and evaluation tools within a unified agent architecture. Experimental results demonstrate state-of-the-art performance on regression tasks with an average 2.0% RMSE reduction, and competitive performance on classification tasks with 0.23% average ROC-AUC improvement on datasets exceeding 10K instances, while maintaining superior robustness to errors compared to existing approaches.

Key Contributions

• First application of an agentic ReAct framework to automated feature engineering for both regression and classification tasks
• Integration of feature generation, selection, and evaluation within a unified agent architecture that iteratively learns from success and failure
• State-of-the-art performance on regression benchmarks with 2.0% average RMSE reduction
• Competitive classification performance with 0.23% ROC-AUC improvement on large-scale datasets (>10K instances)
• Enhanced robustness to errors through the ReAct paradigm’s ability to record and learn from feature discovery attempts

The ReAct Paradigm for Feature Engineering

The core innovation of FAMOSE lies in its adaptation of the ReAct (Reasoning and Acting) paradigm to the feature engineering domain. Traditional automated feature engineering approaches operate in a feed-forward manner, generating features without the ability to reflect on their effectiveness. FAMOSE, by contrast, implements an iterative cycle where the agent:

1. Reasons about which features might be valuable based on data characteristics and previous attempts
2. Acts by generating candidate features using domain-agnostic transformations
3. Observes the impact of these features through integrated evaluation tools
4. Reflects on outcomes to guide subsequent feature generation

This cyclical process effectively creates a form of in-context learning where the LLM’s context window accumulates knowledge about successful and unsuccessful feature engineering strategies. The authors hypothesize that this mechanism functions similarly to few-shot prompting, enabling the model to progressively generate more innovative and effective features as the agent explores the feature space.

The ReAct framework allows FAMOSE to maintain a persistent memory of what works and what doesn’t, avoiding redundant exploration and focusing computational resources on promising feature engineering directions. This is particularly valuable in the exponentially large feature space where exhaustive search is computationally prohibitive.

Experimental Results and Performance Analysis

FAMOSE’s experimental validation spans multiple benchmark datasets for both regression and classification tasks, demonstrating consistent improvements over baseline methods:

Regression Performance: FAMOSE achieves state-of-the-art results with an average 2.0% reduction in RMSE across regression benchmarks. This improvement is particularly notable because regression tasks often require more nuanced feature relationships to capture continuous target variables accurately.

Classification Performance: On classification tasks, FAMOSE shows strong performance with an average 0.23% ROC-AUC improvement on datasets with more than 10,000 instances. The performance gain is more pronounced on larger datasets, suggesting that FAMOSE’s iterative exploration benefits from having more data to evaluate feature effectiveness.

Robustness: A critical advantage of FAMOSE is its superior robustness to errors compared to competing algorithms. The ReAct framework’s ability to recover from unsuccessful feature generation attempts and adjust its strategy makes it more reliable in production scenarios where data quality and distribution may vary.

The performance characteristics suggest that FAMOSE’s strength lies not just in generating individual high-quality features, but in its ability to systematically explore the feature space and learn from the exploration process itself.

Implications for AI-Driven Problem Solving

FAMOSE’s success provides compelling evidence for a broader thesis: AI agents employing reasoning-action cycles are remarkably effective at solving problems requiring highly inventive solutions. Feature engineering is an ideal testbed for this hypothesis because it demands creativity, domain understanding, and the ability to evaluate and iterate on ideas—capabilities traditionally considered uniquely human.

The framework’s ability to autonomously discover effective features without domain expertise has several important implications:

Democratization of ML: By reducing the dependency on domain experts for feature engineering, FAMOSE lowers the barrier to entry for applying machine learning to new domains and problems.

Scalability: The automated nature of FAMOSE enables feature engineering at scale across multiple datasets and domains simultaneously, something impractical with manual approaches.

Innovation: The iterative exploration enabled by ReAct may discover non-obvious feature combinations that human experts might overlook, potentially leading to novel insights about data relationships.

Generalization: The domain-agnostic nature of FAMOSE’s approach suggests that similar agentic frameworks could be applied to other creative problem-solving tasks in machine learning and beyond.

Takeaways

1. FAMOSE represents the first successful application of the ReAct agentic framework to automated feature engineering, achieving state-of-the-art regression performance and competitive classification results.

2. The iterative reasoning-action cycle enables the LLM to learn from feature engineering attempts within its context window, functioning as a form of dynamic few-shot learning that improves feature quality over time.

3. Performance gains are most pronounced on larger datasets (>10K instances) and regression tasks, with 2.0% average RMSE reduction and 0.23% ROC-AUC improvement respectively.

4. FAMOSE demonstrates superior robustness to errors compared to existing automated feature engineering approaches, making it more suitable for production deployments.

5. The success of FAMOSE provides evidence that AI agents with reasoning capabilities can effectively solve highly creative problems like feature engineering that traditionally required substantial human expertise and intuition.