Re-imagining Platform Engineering With AI (1): Exploring AI Tools And Platforms

The crossroads of Artificial Intelligence and Platform Engineering represents a frontier brimming with potential for innovation and transformation. LLM’s integration into platform engineering disciplines is not just about automating trivial tasks; it’s about reimagining the ways applications are developed, deployed, and managed.

The core concept of Platform Engineering was formed to reduce the cognitive burden of application developers in order to drive innovations more efficiently. Most common approach today is to slice up the duties based on different roles in an organization, codify them and orchestrate the pieces later into the actual infrastructure-facing specifications. In those cases today, we are seeing the complexities being “shifted down” onto the platform engineers in order to do so. We are not actually eliminating complexities but simply shoving them somewhere else. With the emergence of LLMs, one of the outs is to re-imagine the possibility that these complexities can be stuffed and absorbed into a “black-box”, namely, a large language model.

Finding and embracing the right path is not just a slogan — it’s imperative for those poised to lead the charge in this new frontier. We too are on an eternal journey to find the value propositions and the quintessential arsenal of tools that are instrumental in sculpting the future of AI-driven platform engineering. On this journey, we will keep track of the essential thinking process, tools, ideas, attempts, results and reflections.

Starting with the first blog post in this series, we’ll cover a curated list of tools, frameworks and platforms we saw that are either interesting, or indispensable in the exploration of AI applications.

Concepts — LLMs and RAG

The concept and modus operandi of LLM is likely explained way too many times, so we won’t elaborate them here. In very few words, LLMs represent a technique to generate and predict content based on gigantic datasets fed into a lengthy and expensive training process, which results in billions of parameters that are the basis for prediction. The implications being, LLMs are generally knowledgeable but will likely to perform poorly against specific areas.

RAG stands for Retrieval-Augmented Generation — which represents a relatively new approach to more cost-effectively improve LLM output so it remains relevant, accurate, and in-context. It’s particularly an excellent choice for those with limited resources to train or even fine-tune a model. RAG works by storing and retrieving relevant information from external data sources (usually vector database), then augment the user query by introducing the retrieved information, and eventually send the enhanced prompt to LLMs in order to get more in-context response.

LangChain

LangChain is one of the earliest frameworks designed to simplify the creation of LLM applications, where its major value proposition is to provided standardized interfaces and components for the interactions with LLMs(model I/O, retrieval, agents, and more), and off-the-shelf chains that can be composed into more complex tasks. It started with a set of Python and JavaScript libraries but was later extended with the LangSmith development platform.

The encapsulation of LangChain is remarkable. Using LangChain, we were able to have an application up and running without much prior knowledge of LLMs. It took less than 40 lines of code to build an ChatBot that does research all over the Internet and return the answer with sources, all thanks to the abstract concepts like RetrievalQAWithSourcesChain and WebResearchRetriever.

It’s also straightforward to set up an RAG-based chain, with the source being either static files, web pages, databases, or more. LangChain provides abstractions for every step along the RAG workflow: Load -> Split -> Store -> Retrieve -> Generate.

AutoGen

AutoGen is another popular, open-source framework trying to accelerate the development of LLM applications, with its initial focus on employing multiple agents that converses with one another to solve more complex tasks. Similar to LangChain, it also provides abstractions that wraps around the interactions with LLMs. The multi-agent framework focuses on simplifying the orchestration, automation and optimization of a complex LLM workflow.

Here’s an interesting demostration of AutoGen in the works in a multi-agent workflow to solve a complex task, where a group chat is established and 5 different roles are defined:

• Admin: Corresponds to a human who can continuous send instructions to writer to refine the post.
• Planner: In charge of determining what information is needed in order to complete this task.
• Engineer: In charge of generating Python and Bash code to collect said information.
• Executor: Executes the code written by the Engineer and respond with results.
• Writer: Compose an article based on previous information and user input.

CrewAI

Crew.ai is another relatively new and open-source framework built on top of LangChain, and targeting to enable sophisticated multi-agent interactions. Similar to AutoGen, it employs a role-based approach, where each agent is given a role, a goal and a backstory. On top of that, CrewAI introduces the delegation of tasks among agents. CrewAI also believes in a more controlled and process-driven workflow, rather than an autonomous one initially introduced by AutoGen(it has since supported a more customized approach).

AutoGPT

We are finally out of the framework realm. AutoGPT is a star project in the open source world with a whopping 160k GitHub stars in less than a year, cracking the top 25 of all time. It started as an autonomous agent capable of handling a myriad of tasks with minimal human intervention. The idea is that, given a very specific background including backstory, goal-setting and instructions, the agent is then set to act as autonomously as possible, following a pattern that resembles the OODA Loop (Observe, Orient, Decide, Act) against the GPT-4 APIs. The pattern drives AutoGPT to make plans, make observations, make decisions, act and even reflect on itself. To achieve that, the ability to execute atomic tasks is needed, such as Google search, data persistence, write and execute codes, etc. AutoGPT provides built-in support for said capabilities.

While AutoGPT has an unimaginable ceiling with its vision to automously solve ANY tasks, the project is expanding its horizon with more concrete use cases that are based on its foundation of autonomous agents, with the introduction of Forge and Benchmark, which are used to Build-Your-Own agents and to measure the performance of it.

Lepton

Lepton.ai is yet another platform to solve the problems in the full lifecycle of an AI application, with its primary focus on deploying and managing the Large Language Models behind the user-facing applications. It comes with a set of open-source libraries that abstracts away the complexity of research and modeling code, the dependencies and the infrastructure complexity, etc, enabling a more friendly experience for AI developers. Lepton also provides a CLI and a hosting platform to accommodate the delivery of AI Applications including the deployment and launch of models like those found on Huggingface. Again without must prior knowledge in LLMs, we were able to deploy a model (locally and on the cloud) with running a few commands in less than 10 minutes.

Dify

If we think about LangChain as a set of tools like hammers and nails that lays the foundation for LLM application development, there are also proved value-add to build opinionated platforms on top of the toolkits to service a targeted segment of the users. One such example to optimize for developer experience is the low-code or even no-code platforms to develop LLM apps. We’ve come across Dify.ai which is an open-source LLM application platform with a self-hosted mode that can be used to host an agent platform inside an enterprise. It provides an end-to-end solution for building agents with customized tools and datasets that can be leveraged to solve tasks.

GLMs

GLMs is another AI application platform that impressed me quite a bit, with its emphasis on no-code generation of agents based on chat (That’s right, you don’t need to know how to write code at all). In our experiment, it generated an agent out of a one-sentence description that it’s “a sports reporter specializing in finding out scores and results of tournaments”. The agent successfully reported the UEFA champions league games and results this morning. If the agents needs further customization (such as customized prompt template), that can also be done with no-code mode. Afterwards, we also did the publishing of this agent with a single click. GLMs also supports RAG with a file as its external data source for the agent.

To sum it up…

The development frameworks, utilities and platforms for AI Agents are emerging quickly and gaining quite a lot of momentum, creating a flourishing ecosystem. It not only represents the rapid movement in the AI space but also creating unprecedented possibilities for AI developers. We are extremely excited about this development, and hereby starting this series where we embark on a journey to explore the possibilities of AI and Platform Engineering telling a story together. If you would like to share some interesting ideas in this area, feel free to leave a message for us!

We can be found on Slack here and Github here.