Three layers of context for useful AI

Summary: In this post, I describe how we have integrated the use of Large Language Models (LLMs) into our product, along with some considerations and experiences that guided us. My goal is to organize my thoughts and provide others with ideas on how they can use AI in their products. All reference data is notional.

As a founding CTO in a software company, I’ve been searching for a convincing way to integrate AI into our product, particularly LLMs. My team has explored a wide range of models and use cases, drawing ideas from team members, customers, investors, and partners.

Finding an angle that leverages the strengths of LLMs without becoming a fancy Lorem Ipsum generator or a shiny demo that fails in real-world scenarios has been challenging. It has also been tough to come up with compelling questions to ask the LLM. Often, I found myself testing whether the AI could infer something I already knew rather than being genuinely curious. This is not a good method for identifying real use cases.

However, we now have two use cases that I am enthusiastic about. One of them enables users to ask questions about selected content. The other needs more time.

An example of the first use:

This use case allows users to draw a rectangle within the app, identifying the HTML elements within the selection. This approach has two key advantages: the user sees all the data in scope, and it limits the amount of data sent to the LLM by design. Many other use cases we considered involved sending unknown amounts of data to the LLM, potentially exceeding context limits. Specific and small data is crucial, and this is the first layer of context.

The three layers of context

In addition to the user prompt and the selected content, we add more two layers of context. They serve different purposes.

Layer 2: DOM Context

The second layer is pulled from the current Document Object Model (DOM) of the app. This context is added as key-value pairs within the HTML structure, becoming more specific further down the hierarchy. When the user draws a rectangle, we traverse the DOM upwards to collect available context. The purpose of this layer is to give the LLM an idea of what the user has selected, helping it to choose relevant parts of its existing corpus. This layer is generic, always present, and up-to-date, nudging the LLM in the right direction. An illustration of the DOM structure for this concept:

Layer 3: RAG

The third layer involves indexing vectors from “background documents” like instruction manuals, regulations, and guidelines in a vector database. This data is fairly static, avoiding continuous indexing as a material part of product operation. The purpose of this layer is to provide the LLM with data it does not have in its corpus. For instance, a financial institution might provide a list of “high-risk countries” based on government guidelines.

Before sending a prompt to the LLM, a single vector is generated for the prompt and the two first context layers. This vector is used to identify presumably relevant parts of the background documents whch becomes the third layer. The goal is to identify relevant sections of the document.

An example

I won’t attempt any formal verification of this 3-layer approach as that too quickly becomes anecdotal. But I will try to illustrate the principle with an experiment using the smallest llama3 model and ollama as provider. Let’s first explore what goes into the different layers of context in a specific example:

Layer 1: Selected content

Credit TX Country Credit amount nok
India kr 2 005 192
Italy kr 0
Norway kr 8 868 686
Sweden kr 50 236
United Kingdom kr 42 844
United States kr 200 000
Total kr 11 166 958

Layer 2: DOM Context

Object name: Convier AS
Object type: CUSTOMER
Project name: fincrime

Layer 3: RAG

As background documents I chose a document about examination policies from the FDIC. The 55-page document consists of around 650 sections. A vector was generated for each section using the embedding model of llama3. These vectors are then added to a vector database.

Prompt:

The selection of a representative prompt without too much bias is tricky, but I’ll go with the following:

How should I check transactions to these countries?

Template:

We put the prompt and the generated context into a template which is then passed to the LLM. I use the following template:

The following is text copied from a website. Given the context provided 
below, answer the user's question. Double check your answer.
Context: {context}
Question: {prompt}

Testing

The following are the responses from the LLM given various degrees of context. The first one without any context is included for completeness.

Without context:

Please provide the text and I'll do my best to help you with checking 
transactions to those countries.

With Level 1 context:

Based on the provided text, it appears that you have credit transactions 
in various countries and the total amount of credits is given. To 
"check transactions" in this context would likely mean to review or 
verify the credit amounts for each country.
Accordingly, if you need to check transactions to these countries, 
I assume you would start by reviewing the credit amounts listed for India, 
Italy, Norway, Sweden, United Kingdom, and United States. This might 
involve comparing these amounts with other financial records or 
conducting additional research to ensure accuracy.

With Level 1 and 2 context:

Based on the provided text, it appears that the data represents credit 
transactions by country. To check transactions to these countries, you 
can simply look up each country's total credit amount in Norwegian 
kroner (NOK) from the table:
India: kr 2 005 192
Italy: kr 0
Norway: kr 8 868 686
Sweden: kr 50 236
United Kingdom: kr 42 844
United States: kr 200 000
Total: kr 11 166 958

With only Level 3 (RAG) context:

Based on the provided context, which appears to be related to regulatory 
requirements for Money Services Businesses (MSBs), I understand that 
you're asking about how to check transactions involving certain countries.
According to the text, international transfers that are not consistent 
with the customer's business could indicate money laundering. Therefore, 
it seems that checking transactions to these countries involves 
verifying that they are consistent with the customer's known legitimate 
business activities.
In more specific terms, this might involve:
* Reviewing transaction records to identify any unusual or suspicious 
  activity.
* Verifying that the frequency and amounts of international transfers align 
  with the customer's expected activity level.
* Conducting enhanced scrutiny on accounts and transactions involving senior 
  foreign political figures (PEPs) and other high-risk individuals.
By taking these steps, you can help prevent potential money laundering and 
fraud, as well as comply with regulatory requirements such as those set by 
FinCEN and the FDIC.

With Level 1, 2 and 3 context:

Based on the provided text, it seems that you are a customer of Convier AS 
and have transactions with various countries. To check transactions to 
these countries, you should follow the regulations for private banking 
accounts mentioned earlier.
According to FinCEN, enhanced due diligence procedures and controls should 
be established where appropriate and necessary with respect to applicable 
accounts and relationships. This means you should review your transactions 
to determine if any of them are suspicious or reportable money laundering 
transactions.
For each country, you should verify the identity of the counterparties 
and ensure that all transactions comply with anti-money laundering 
regulations. You may also need to collect additional information from the 
counterparties, such as tax identification numbers or government-issued 
documentation (if not already obtained).
In your case, since you have transactions with India, Italy, Norway, 
Sweden, United Kingdom, and United States, you should follow these steps:
* Verify the identity of your counterparties in each country.
* Ensure that all transactions comply with anti-money laundering regulations.
* Review each transaction to determine if it is suspicious or 
  reportable money laundering transactions.
By following these steps, you can ensure compliance with regulations and 
prevent potential financial risks associated with money laundering

Assessment

With these types of questions, it’s easy to think “no sh*t, Sherlock” about the generated responses, like “Ensure that all transactions comply with anti-money laundering regulations.”. It also does misunderstand some of the context, like being a customer of Convier AS, when Convier AS is the customer. But it seems clear that as I added more context, the LLM understands for and more of the situation in which the question is asked. With only the RAG context, the answer is quite generic, but with the addition of application and selection context, it seems to understand better what I am trying to achieve.

Concluding remarks

This approach has a couple key strengths which is why I believe in it:

• Generality: It can be used to ask questions I cannot imagine.
• Ease of Maintenance: All added context is either static or automatically generated within the app, with no jobs or external dependencies except the LLM provider.
• No Post-Processing Required: Suitable for smaller, cheaper, faster, or weaker models which can run “locally” with reasonable response times.

This has hopefully provided you with some insights into how we use LLM to provide real value to users.

Please do not hesitate to get in touch if you have any questions or comments! My email address is petter.bjelland convier.com.