The Return of Structure: Data Architecture Lessons for the Agentic Workforce

In the age of AI, it is often assumed that agents will be intelligent enough to discover the information they need on their own. However, the reality is the opposite: AI is pushing companies to structure their data in a more machine-readable way than ever before. Whether the goal is to optimise token burn or manage LLM hallucinations — both of which can cost organisations hundreds of thousands of dollars — architecting context retrieval is now one of the primary levers for achieving ROI from AI initiatives.

To manage this, AI Engineers are turning to a spectrum of techniques, from standard RAG and Knowledge Graphs to emerging hybrids such as GraphRAG and KG-RAG. Yet these tools entail significant trade-offs and a pressing need for governance. Most organisations are currently ill-equipped to manage this complexity. Without a structural map, they risk repeating the mistakes of the early data-lake era, when a lack of governance turned promising technology into chaotic ‘data swamps’.

Knowledge graphs (KG)

Knowledge graphs require information to be structured as a set of Nodes and edges. In Resource Description Framework (RDF) knowledge graphs, each relationship is represented as a triplet (Subject, Predicate, Object). A simple example of this is a User (me), LivesIn, City (Amsterdam). It is worth noting that, normally, these edges are unidirectional.

Knowledge graphs have been in place for decades. When I was working at Facebook, the entire Facebook application had been conceived as a form of entities and associations (unidirectional), each entity and association containing generic data objects. Facebook introduced GraphQL to efficiently query interconnected data structures. Graph-like objects are also present in some underlying database or data engineering structure, for instance, when leveraging tools like Airflow or DBT, dependencies and lineage are available as a “Directed Acyclic graph” (DAG).

These knowledge graphs are often coupled with an ontology, which formalises what each node and edge represent and gives them meaning.

For AI, this means agents can more effectively use it as a source of structured information, which can be particularly beneficial for adding context and enabling retrieval. This is, for instance, what I did when building Qarion’s Control plane, leveraging relationships to better understand the table-to-fields relationship and the ontological mapping to extend the available information. This information was then used by LLMs to provide an automatically generated description of a dataset or field, or to, for instance, estimate the impact of a change.

The main challenge in managing knowledge graphs is the effort required to maintain them. It is easy for Nodes to end up duplicated, for instance, due to bad input such as a Typo, different ways to name the entity, or for bi-directional edges to end up uni-directional, whether it is due to human error or a system error, or previous relationships to no longer be relevant due to the test of time. At scale, they require particular care, governance and a robust system for identifying and resolving issues.

Knowledge graphs, however, do allow for providing highly curated and governed information, away from some of the hallucinations of LLMs. As such, they can serve as a gold standard for how some information can be curated for exposure to the agentic workforce for retrieval.

Retrieval-augmented generation (RAG)

Contrary to knowledge graphs, the goal is to enable systems to retrieve information based on meaning, without requiring predefined relationships in the data.

RAG systems operate on unstructured data by identifying related content through proximity in a learned representation space.

To achieve this, neural networks transform content into vector representations, called embeddings, that map semantic similarity in a multi-dimensional space. In this environment, related ideas are clustered together, allowing the system to retrieve information by simply finding the ‘nearest neighbours.’ However, this mathematical elegance comes with a trade-off: the necessary steps of chunking and tokenising text before embedding often lead to a loss of context, as the structural integrity of the original narrative is fragmented.”

The main advantage of this approach is that it does not require data to follow a predefined structure and can operate directly on raw text. However, it relies on similarity rather than explicit relationships, which can limit its ability to capture less obvious or highly structured dependencies.

It can, however, be quite difficult to debug and often turns into a Blackbox, as it is hard to see why a given chunk maps to a given semantic space. As such, RAGs are particularly well-suited for unstructured search but can sometimes be prone to hallucination or to returning unexpected results

A mixture of both worlds

Microsoft Research first introduced GraphRAG in February 2024, as an attempt to marry both approaches. GraphRAG attempts to leverage LLMs to do entity extraction and linking, generating the same triplets as in the RDF representation of a knowledge graph. At query time, entities are matched, and their surrounding edges and entities are retrieved via graph traversal; this data is used to structure the context and reduce hallucination when invoking the LLM.

KG-RAG (Knowledge Graph-Augmented Generation) is an alternative approach that leverages an existing, curated knowledge graph rather than building one from scratch. Instead of relying on the semantic ‘guesswork’ of vector proximity, KG-RAG maps the user query to specific, governed nodes within the graph. By traversing the predefined relationships from those nodes, the system retrieves high-fidelity context — often referred to as the ‘Gold Standard’ — and injects it directly into the prompt to ensure the LLM remains grounded in fact.

There are major distinctions between the two approaches:

• GraphRAG aims to build knowledge, while KG-RAG looks to exploit knowledge.
• GraphRAG requires much less effort to set up than KG-RAG, which requires a highly curated knowledge graph already available.
• GraphRAG aims to decrease hallucination, while KG-RAG aims to be the gold standard for information retrieval (facts).

Strategic Implications

Most organiSations are still approaching AI initiatives with the same mindset they had during the Data Science PoC wave of the 2010s — treating them as experiments rather than as foundational capabilities that require engineering excellence and data maturity.

This approach is risky. The shift toward agentic AI has far-reaching organisational implications that most companies are not yet prepared to address: governance, ownership, budget allocation, risk management, and operating model design.

Building and maintaining high-quality knowledge graphs and ontologies is not a cheap or lightweight endeavour. It demands sustained investment, clear accountability, and robust processes — areas where many AI projects currently fall short. Questions that leaders must answer include:

• Who owns the ontology and the “gold standard” knowledge layer?
• How do we govern the continuous curation and evolution of entities and relationships?
• What level of investment are we willing to make in data fidelity versus accepting higher token costs and hallucination risks?

Just as organisations had to mature their data platforms and practices to move beyond data lakes, they now need to evolve their data and AI operating models for the agentic era. Companies that have already shifted their mindset — treating data not as a byproduct but as a strategic product — and have built strong governance across both structured and unstructured assets will hold a significant competitive advantage.

Those who invest early in a governed knowledge foundation will be able to deploy more reliable, cost-efficient, and trustworthy AI agents, while others risk building on shaky ground.

Conclusion

The AI Age is often portrayed as the triumph of unstructured, fluid intelligence over rigid systems. Yet the rise of the Agentic Workforce reveals the opposite truth: the more autonomous our AI agents become, the more they depend on structured, governed knowledge.

Standard RAG offers intuition at scale. GraphRAG accelerates discovery. But only KG-RAG, grounded in well-maintained ontologies and human-curated relationships, delivers the reliability and governance required for enterprise deployment.

In an era where every token has a cost and every hallucination carries business risk, structural integrity is no longer optional — it is a core competitive advantage.

Organisations need to determine their investment and risk appetite, but recognising that to fully leverage the benefit of AI, only those with a solid knowledge foundation will be positioned to deploy trustworthy, cost-efficient agentic systems. Those who don’t risk building increasingly expensive houses on sand.

The return of structure is not a step backward. It is the necessary foundation for the next phase of AI value creation.