Abstract:We investigate the potential of GPT-4~\cite{gpt4} to perform Neural Architecture Search (NAS) -- the task of designing effective neural architectures. Our proposed approach, \textbf{G}PT-4 \textbf{E}nhanced \textbf{N}eural arch\textbf{I}tect\textbf{U}re \textbf{S}earch (GENIUS), leverages the generative capabilities of GPT-4 as a black-box optimiser to quickly navigate the architecture search space, pinpoint promising candidates, and iteratively refine these candidates to improve performance. We assess GENIUS across several benchmarks, comparing it with existing state-of-the-art NAS techniques to illustrate its effectiveness. Rather than targeting state-of-the-art performance, our objective is to highlight GPT-4's potential to assist research on a challenging technical problem through a simple prompting scheme that requires relatively limited domain expertise\footnote{Code available at \href{this https URL}{this https URL}.}. More broadly, we believe our preliminary results point to future research that harnesses general purpose language models for diverse optimisation tasks. We also highlight important limitations to our study, and note implications for AI safety.
Submission history
From: Mingkai Zheng [view email]
[v1]
Fri, 21 Apr 2023 14:06:44 UTC (160 KB)
[v2]
Mon, 24 Apr 2023 14:56:54 UTC (161 KB)
[v3]
Mon, 29 May 2023 20:05:34 UTC (166 KB)
[v4]
Wed, 2 Aug 2023 03:59:34 UTC (166 KB)