X (formerly Twitter) ·
(2/n) 📊 We present a practical first step toward the goal of end-to-end automation of the scientific process focusing on observational or experimental data for two reasons: (1) an abundance of large-scale datasets that would benefit highly from automated discovery; 📈 (2) the
(3/n) A blueprint flow for data-driven discovery includes the following scenarios: 1. The user asks an explicit question around a particular line of inquiry or hypothesis. 🎯 2. The user can also ask a broad and partially defined high-level question, where the system must
(4/n) We posit that: 1. LGMs present an incredible potential, such as knowledge-driven hypothesis search or tool usage to verify hypotheses—creating new avenues for ongoing efforts in the ML community on code generation, planning, and program synthesis. 🛠️ 2. LGMs are not all
(5/n) We outline a set of desired properties for a data-driven discovery system. 🟩 1. Comprehensive Data Understanding 2. Hypothesis Generation 3. Planning and Orchestrating Research Pathways 4. Hypothesis Evaluation 5. Measurement of Progress 6. Knowledge Integration 7.
(6/n) As a proof of concept, we build DataVoyager—a system powered by GPT-4 that can semantically understand a dataset, programmatically explore verifiable hypotheses using the available data, run basic statistical tests (e.g., correlation and regression analyses) by invoking
(7/n) Planning DataVoyager presents a strong base case for planning with decomposition, data transformation, and symbolic reasoning. However, LGM-based planners prefer direct, goal-oriented variables, which can lead to a lack of diversity in search, impacting the novelty of
(8/n) Experimentation & Verification DataVoyager can use tools and insight-specific code generation to reasonably verify hypotheses. But LGMs are memoryless. They cannot automatically recover from past errors in execution and verification. We argue that how LGMs adapt to novel
(9/n) Knowledge Integration DataVoyager can partially achieve interdisciplinary knowledge integration. E.g., it could connect the role of economic pressure on health outcomes with cultural anthropology, psychological factors, public health intervention, and urban planning.
(10/n) 🚨 We also point out possible limitations of such automated systems, such as: 1. Hallucinations in LGMs undermining scientific rigor 2. Cost at scale in high-throughput fields 3. Data dredging resulting in sub-optimal policies 4. Autonomous discovery leading to legal
(11/n) 🌌 We hope our timely position can increase interest and efforts in developing, debating, and enhancing the vision for an accurate, reliable, and robust system for data-driven discovery. These systems can transform domains overwhelmed with vast amounts of data, including
Geoffrey Hinton says AI has the ability to identify patterns within and across fields that humans cannot
