Writing Silly LLM Agent in Haskell

6 min read Original article ↗

Inspired by Fly.io article I figured that writing an Agent in Haskell (which I recently picked up) would be a nice exercise, so hey, why not do it?

This post is walk through full implementation of a GNU Sed agent (that doesn’t work correctly, but probably because I’m using it wrong).

Design

When thinking about agent design my thought process was as follows:

  • I need to produce a prompt that will consistently return GNU Sed compatible expression…
  • …once I have that, I want to run it through local ollama server
  • …at least few times, because I might get so-so result after first attempt
  • …then pick the most popular option
  • …and feed it to GNU Sed with other parameters

Word on the Prompt

I dislike creating prompts. It feels like a game of whac-a-mole with indeterminate results. Thankfully commercial models got so good, that they can be made to work in that direction.

Prompt took 2 attempts and I didn’t care to tweak it further. In fact, I find it good enough, because seldom results are not adhering to instructions.

Types

I believe that, with Haskell, reasonable start is types definition, so I started with these:

newtype LLMPrompt      = LLMPrompt      T.Text
newtype LLMInstruction = LLMInstruction T.Text
data    LLMCommand     = LLMCommand     T.Text [T.Text]
newtype LLMResult      = LLMResult      T.Text deriving (Show, Eq, Ord)

data    Command        = Command        T.Text [T.Text]
newtype Output         = Output         T.Text deriving (Show, Eq, Ord)

class Agent a where
  run :: a -> LLMResult -> IO Output

newtype SedAgent = SedAgent ()

I think those are rather obvious. 4 types for LLMs where Command is a composition of LLM... types. Finally there’s a Output newtype for the end result.

The T.Text [T.Text] shape is made for Shelly.run (which I knew I wanted to use), that takes command and list of arguments. T is alias to Data.Text.

Agent is represented by Agent typeclass with a single function - “run” that transforms LLMResult into IO Output. When working on this specific part of the code, I thought that maybe better idea would be to make agents monads - so that interactions are made within the context. That design makes sense, but would made initial implementation much more verbose.

makeCommand :: LLMCommand -> LLMInstruction -> LLMPrompt -> Command
runCommand :: Command -> IO LLMResult
runCommandNTimes :: Integer -> Command -> IO [LLMResult]
pickBest :: (Ord a) => [a] -> a
main :: IO ()

I hope those are also straightforward - along with run function for the Agent typeclass. Time for implementation!

Implementation

makeCommand (LLMCommand c args) (LLMInstruction ins) (LLMPrompt prompt) =
  let promptFull = (T.replace "$$$PROMPT$$$" prompt ins) in
  Command c (args ++ [promptFull])

runCommand (Command c args) = shelly $
  Shelly.run (T.unpack c) args
  >>= pure . LLMResult . T.strip

runCommandNTimes n c = sequence [runCommand c | _ <- [1..n]]

pickBest = last . last . (L.sortOn length) . L.group . L.sort

instance Agent SedAgent where
  run _ (LLMResult input) = shelly $
    Shelly.run (T.unpack "gsed") ["--sandbox", "-ne", input, testFile]
    >>= pure . Output . T.strip

makeCommand is using naïve method of replacing keyword within original prompt. Both SedAgent’s run and runCommand are simple wrappers for Shelly that return IO results. Because GNU Sed can run external commands, I added --sandbox argument to prevent any troubles. Also, in SedAgent there is a reference to testFile, which is part of hardcoded section:

llama3 = LLMCommand "ollama" ["run", "llama3:latest"]
sedInstruction = T.pack <$> readFile "assets/sed_instructions.txt"
testFile = "assets/sed_test_file.txt"
testPrompt = "replace zeroes with unicode empty circle"

While llama3 is a reasonably good helper, test* and sedInstruction exist to simplify initial implementation. Files can be found at my haskell-toys repo under “SedAgent” directory.

Main Function

With all components implemented, I wanted to see an output. A sidenote here, I’m using “do” notation, even though I learned it’s not really that great for learning Haskell. But since the idea for this post was in my head the moment I started writing code I wanted it to be readable first.

Main function was implemented like this:

main = do
  instruction <- sedInstruction
  let llmCommand = makeCommand llama3 (LLMInstruction instruction) (LLMPrompt testPrompt)
  llmResults <- runCommandNTimes 50 llmCommand
  let bestResult = pickBest llmResults
  (Output result) <- Main.run (SedAgent ()) bestResult
  let (LLMResult bestResultStr) = bestResult
  putStrLn "Best Result: "
  putStrLn . T.unpack $ bestResultStr
  putStrLn "Output:"
  putStrLn . T.unpack $ result

(Including some debug output)

End result and final word

Let’s see the expression for the testPrompt “replace zeroes with unicode empty circle”. Voilá:

Best Result:
s/0/⚫️/2p
Output:
1 20 3⚫️0 4000 5000

...done

Successful failure!

I knew that would happen, and this - dear audience - is why you never should run any agents1

…

Ok, I’ll admit: there are many fundamental flaws in my approach:

  • Demanding sed expression, that’s a sequence of characters and special chars is most likely hard mode for LLMs
  • I just took off-the-shelf llama3 model and called it a day, I bet there are better (not to mention commercial ones)
  • Prompt wasn’t optimized for the conditions in which it’ll be ran
  • There are much better ways to pick best result..
  • …and validate it, too

One relevation I got from this excercise is that writing a good agent is going to be hellishly difficult and I’ll (personally) treat agents as personal Perl script: only run those I written myself.

Full, runnable, project with the state of this article can be found at my haskell-toys repository.

Main.hs

{-# LANGUAGE OverloadedStrings #-}
module Main (main) where
import qualified Data.Text as T
import qualified Data.List as L
import Shelly

newtype LLMPrompt      = LLMPrompt      T.Text
newtype LLMInstruction = LLMInstruction T.Text
data    LLMCommand     = LLMCommand     T.Text [T.Text]
newtype LLMResult      = LLMResult      T.Text deriving (Show, Eq, Ord)

data    Command        = Command        T.Text [T.Text]
newtype Output         = Output         T.Text deriving (Show, Eq, Ord)

class Agent a where
  run :: a -> LLMResult -> IO Output

newtype SedAgent = SedAgent ()

makeCommand :: LLMCommand -> LLMInstruction -> LLMPrompt -> Command
runCommand :: Command -> IO LLMResult
runCommandNTimes :: Integer -> Command -> IO [LLMResult]
pickBest :: (Ord a) => [a] -> a
main :: IO ()

makeCommand (LLMCommand c args) (LLMInstruction ins) (LLMPrompt prompt) =
  let promptFull = (T.replace "$$$PROMPT$$$" prompt ins) in
  Command c (args ++ [promptFull])

runCommand (Command c args) = shelly $
  Shelly.run (T.unpack c) args
  >>= pure . LLMResult . T.strip

runCommandNTimes n c = sequence [runCommand c | _ <- [1..n]]

pickBest = last . last . (L.sortOn length) . L.group . L.sort

instance Agent SedAgent where
  run _ (LLMResult input) = shelly $
    Shelly.run (T.unpack "gsed") ["--sandbox", "-ne", input, testFile]
    >>= pure . Output . T.strip

llama3 = LLMCommand "ollama" ["run", "llama3:latest"]
sedInstruction = T.pack <$> readFile "assets/sed_instructions.txt"
testFile = "assets/sed_test_file.txt"
testPrompt = "replace zeroes with unicode empty circle"

main = do
  instruction <- sedInstruction
  let llmCommand = makeCommand llama3 (LLMInstruction instruction) (LLMPrompt testPrompt)
  llmResults <- runCommandNTimes 50 llmCommand
  let bestResult = pickBest llmResults
  (Output result) <- Main.run (SedAgent ()) bestResult
  let (LLMResult bestResultStr) = bestResult
  putStrLn "Best Result: "
  putStrLn . T.unpack $ bestResultStr
  putStrLn "Output:"
  putStrLn . T.unpack $ result