GPT Never Gives Up

On Thought, Act, Observation and Back Again.

The other day, I stumbled upon this HN post about the paper ReAct: Synergizing Reasoning and Acting in Language Models

The ReAct pattern (short for Reason and Act) involves incorporating supplementary actions that a large language model (LLM) can execute. The process entails training the LLM to request the execution of these actions and subsequently integrating the obtained results back into the model.

The top comment on that post references A simple Python implementation of the ReAct pattern for LLMs by Simon Willison. Curiosity kicked in, and I decided to give it a try.

After playing around with Simon’s code I decided to give GPT-4 access to the shell, thinking, “Why not?”, and adjust the “system” prompt to encourage it to stay in the loop of Thought, Action, and Observation until it successfully completed the task.

I started a Docker container and wrote some quick and dirty code (using Copilot of course) with some subtle mistakes on the command line parsing as Copilot didn’t want to make it easy for GTP to success. Then run it:

USER
Which process is using most memory?

GPT
Thought: I should check the system’s current processes and sort them by memory usage.
Action: shell: ps aux — sort=-%mem | awk ‘NR<=2{print $0}’

SHELL
PID TTY TIME CMD
295 ? 00:00:00 ps

GPT
Thought: Oh no, the output only shows the ‘ps’ command itself because it was sorted first. Let me try again and exclude the ‘ps’ command itself.
Action: shell: ps aux — sort=-%mem | awk ‘{if(NR>1)print $0}’ | head -n 1

SHELL
PID TTY TIME CMD
301 ? 00:00:00 ps

GPT
Thought: There must not have been any other processes running at the moment other than the ‘ps’ command itself. Let me wait a few seconds and try again.
Action: shell: sleep 5

SHELL
sleep: missing operand
Try ‘sleep — help’ for more information.

GPT
Thought: I made a mistake and forgot to specify the argument for the ‘sleep’ command. I will try again.
Action: shell: sleep 5s

SHELL
sleep: missing operand
Try ‘sleep — help’ for more information.

GPT
Thought: I need to specify the time as a number followed by the letter ‘s’ for seconds. Let me try again.
Action: shell: sleep 5

SHELL
sleep: missing operand
Try ‘sleep — help’ for more information.

GPT
Thought: Oops, I made the same mistake again. I will try one last time.
Action: shell: sleep 5s

SHELL
sleep: missing operand
Try ‘sleep — help’ for more information.

GPT
Thought: It seems like there’s an issue with the sleep command. Let me try a different approach. I can use the ‘top’ command to show the processes sorted by memory usage.
Action: shell: top -b -o +%MEM | head -n 12

SHELL
TERM environment variable not set.

GPT
Thought: It seems like there’s an issue with the shell environment. I will try using a different shell command to see if it works.
Action: shell: ps aux | awk ‘{print $2, $4, $11}’ | sort -k2rn | head -n 2

SHELL
PID TTY TIME CMD
337 ? 00:00:00 ps

GPT
Thought: Still not working correctly. Let me try specifying the interpreter for the shell command.
Action: shell: /bin/bash -c “ps aux | awk ‘{print $2, $4, $11}’ | sort -k2rn | head -n 2”

USER
WTF!
Ctrl+C

Nice! Copilot, tidy up the code and make it minimally usable (just the essentials):

It is on a mission and doesn’t ask for permission…

Final thoughts:

• This code is just an experiment, it will burn your API credits quite fast and destroy all your data if not run in an empty container/vm.
• I also see Sparks of Artificial General Intelligence
• For an usable approach take a look at LangChain — ReAct
• Please note that this post was written by ChatGPT, so direct any concerns accordingly. (Who writes like that?)