Ontologies are having a moment again. AI engineers are rediscovering RDF and OWL as a way to keep probabilistic agents inside deterministic boundaries, and reaching for off-the-shelf vocabularies like schema.org and Dublin Core because the models have already seen them. Latent Space covered the revival after UC Berkeley’s Frank Coyle argued at the AI Engineer World’s Fair that probabilistic models need logical guardrails, and Neo4j CEO Emil Eifrem pitched a shared semantic layer that thin agents could run on instead of each one being wired to its own data sources by hand.
I’ve seen three large efforts to make knowledge machine-readable at scale: Smart Tags at Microsoft in 2001, which I helped build; the two SharePoints that shipped the same year, which I watched from across the company; and question answering on google.com in 2005, where I was the PM and later managed the engineering team. Each of them delivered technically but ran into a wall anyway. Maintenance is the reason usually given.
It’s the third of three hard problems. Somebody has to design the schema. Somebody has to populate it before it’s worth anything. And somebody has to keep it current forever. Three problems, solved by people, over and over, and for two decades nothing worth doing it for. None of the three projects were Semantic Web projects, and that’s the point: the formalisms differed but the problems did not. What has changed is that an agent can do the upkeep, and the payoff has arrived, or at least enough people believe it has that RDF and OWL are back on the table. I want to say up front that I’m building a company in this space.
I believed in all three projects at the time. What makes this revival different is not that I believe harder, but that keeping a model current can now happen as a byproduct of the work instead of being somebody’s job.
Three attempts
Before joining Microsoft in 1998 to work on semantic annotation, I did a PhD in natural language processing. We productized research from labs and universities and shipped Smart Tags in Office XP in 2001.
As an aside: the better-known Smart Tags story is the browser one. Microsoft pulled that feature from Internet Explorer 6 in June 2001 after publishers objected to Microsoft turning their words into links of its choosing. That fight was about who controls a web page. The Office version shipped, and faded for another reason.
Every smart tag had two components: a recognizer and an action. The recognizer decided that a string was of a certain type. The action offered you verbs for it. Email this person, add this date to your calendar, look up this stock. We shipped those and let you build your own as well.
Microsoft shipped an SDK so that companies could define the types that mattered in their own business and attach the actions worth taking against them. That’s where it broke down, before maintenance was ever the issue. Developers turned out not to be good at identifying those types, or at inventing the verbs to go with them. Neither is the work of a software engineer. And it wasn’t really anyone’s job.
If nobody owns the work, there are two ways out: make it someone’s job, or build something that doesn’t depend on the work. Microsoft tried both at once, and I had a front-row seat. In 2001 it shipped two SharePoints. Portal Server, released to manufacturing that March, was the serious one. It organized documents by category, with category folders, document profiles and property definitions as first-class objects. It was built for the companies best equipped to do that properly, large enough to have a person, or a whole team, whose actual job was owning the taxonomy. Team Services was the other one, broadly available that May with Office XP and FrontPage 2002. It was somewhere to put files and stand up a site, and it required no particular expertise.
Portal Server, and the human-curated taxonomy features in it, was set up for success. Microsoft was behind it, and its customers were large enough to hire taxonomists. The other version caught on. Team Services became Windows SharePoint Services in 2003 and SharePoint Foundation in 2010, and that lineage, rather than Portal Server’s, is the one every later version descends from.
Two bets, same company, same year, and everything since has been built on the one that required no expertise. Price is not the explanation. Customers at that size bought Microsoft software on enterprise agreements, where the incremental cost of Portal Server was small. The difference was demand.
We launched question answering on google.com in 2005. It offered short factual answers above the search results. The system behind it was called DAFFIE, the Database of All Fact, Fiction, Information and Exaggeration, and I wrote about how it organized everything around entities and their attributes. The approach inverted Smart Tags. Rather than ask developers to define types, we crawled the web and got the attributes and values ourselves.
Where the page already had structure, that generalized. A table is a table, and you can read one without knowing the domain in advance. Everywhere else it was bespoke, code written by hand for one source at a time, using the information-extraction tools of the day.
One line in the launch post says exactly where the edge was. DAFFIE knew about “celebrities, countries of the world, the planets, the elements, electronics, movies, and anything else we’ve thought of so far.”
“Anything else we’ve thought of so far.” Nothing was typed in by hand. But the generic extractors only reached as far as structure that already existed, and past that every new domain needed an engineer to write its own.
That’s the cold start problem, and it’s separate from maintenance: before anything can go stale, somebody has to decide which entities exist and get the first ones in. The schema was a trap in both directions. Constrain it and you leave information on the table, because the web is more various than any list you can sit down and write. Leave it open and you get nonsense, because the web is chaotic. Misspelled entities (who can even spell Britney Spears?), nonsensical attributes (click here: 12), values that don’t line up with their attributes (height: blue).
Britney Spears was a pop star when we built this. She had albums, gave concerts, was about five foot four and was born in December. There was also an accountant in Ohio named Britney Spears: different height, different birthday, no albums. And a dog, shorter than either and lighter by a lot, with no known albums. Keeping the three of them apart was a struggle, but losing it does not leave you with a gap in the data. It leaves you with one entity the system is sure about, with the average height of a pop star, an accountant and a dog. And it’s equally wrong about nearly everything else.
Back at Microsoft, the same choice was being made a second time. The SharePoint story did not end in 2001. In 2003 the capitulation became architectural. Portal Server was rebuilt on top of Windows SharePoint Services, and Microsoft said so itself: the new release “extends the capabilities of Windows SharePoint Services”. The product that had led with a taxonomy now sat as a paid layer on the engine that asked for none.
Taxonomy came back yet again. SharePoint Server 2010 added the Managed Metadata Service, a central term store and a term management tool. But look at who Microsoft said it was for: enterprise metadata management is a set of features “that enable taxonomists, librarians, and administrators to create and manage terms and sets of terms across the enterprise.”
Nine years on, the answer to keeping a company’s vocabulary current was still a job title you had to staff. That’s not the upkeep problem being solved. It’s the upkeep problem being priced based on salaries.
The parts of the Semantic Web that survived are the ones where the vocabulary is somebody else’s problem: schema.org markup and JSON-LD, where you tag against a shared vocabulary you did not have to invent. What never arrived is the one many enterprise pitches were about, an ontology of your own business, affordably kept accurate by your own people. The revival is picking up the parts that worked.
Three problems, not one
Maintenance is the reason usually given, including by Richard MacManus, who wrote the Latent Space piece, and it’s right as far as it goes. It’s also the last of three, and the first two had already stopped most projects before upkeep became the question.
Designing the schema. Deciding which types matter in a business is not software engineering, and it was nobody’s job. Smart Tags shipped an SDK for exactly this but the types never came. Portal Server assumed a taxonomist and got adopted by the organizations that could staff one. On DAFFIE we tried both ends of the continuum and neither end worked. Constrain the schema and you lose most of the web, open it up and a pop star, an accountant and a dog collapse into one entity.
Populating it. A schema is worth nothing empty, and the first entries are the expensive ones. That’s the cold start problem, and it is separate from upkeep: before anything can go stale, somebody has to decide which entities exist and get the first ones in. DAFFIE ran without anyone entering a fact, but each new domain depended on an engineer to write its extractor first.
Keeping it current. This is the one people usually name, and it’s the one that never ended. Here’s what maintaining a hand-built ontology actually involved.
An ontology is a description of a world that keeps moving. Someone has to add the new product line, retire the discontinued one, reconcile the two teams that use the same word for different things, and notice when a relationship that used to be one-to-one became one-to-many. That was usually just one person, often with another job. The business changed faster than they could edit.
So the schema drifted from reality. And that’s worse than no schema, because systems keep trusting it.
You don’t have to take my word for what that work looked like. In December 2005, Unisys wrote up its SharePoint deployment for Dublin Core’s corporate group. The metadata belonged to a Corporate Knowledge Architect. The attributes and their details were “maintained on a spreadsheet.” They had “been unable to automatically populate the lists of controlled vocabularies,” and change control was “an issue with this method.” That’s a large, competent company doing everything right, and the state of the art required a dedicated person or team and a spreadsheet.
That’s the maintenance story, and on its own it explains less than it looks like it does, because plenty of difficult, tedious technologies won anyway. The usual postmortem adds that the Semantic Web was too academic, too complicated, too far from what working developers wanted. SQL is the counterexample. It came out of a research paper in 1970, it rests on relational algebra, and it asks people to think in a way nothing in ordinary life prepares them for. It won completely anyway. Difficulty is not what decided this.
What differs is who consumes the structure, and whether that consumer exists yet. Model your own database well and the consumer is the application you’re building. You write the schema for it, ship it, and your project advances.
Put the same care into RDF and the consumer, and benefit, might be hypothetical. You might be structuring data for your own application, but you might equally be paving the way for applications nobody has built. That’s roughly what happened around 2001. The vocabularies were the easy half. What never arrived was the population of agents that were supposed to traverse them, so the annotations sat there waiting for a reader.
That’s what has actually changed, and it has nothing to do with the formalism. An AI agent today can be a general-purpose consumer of structured data. Nobody has to build a tool to read what you wrote down, which means structure created this morning can be used this afternoon. The wait between doing the work and benefiting from it goes from years, or never, to the same day.
schema.org is the case that shows the mechanism. Put the markup in the head of your pages and search engines read it, and can show a richer result against your own listing. The reward comes quickly and it’s yours, and it’s one part of this stack that got adopted at scale.
Deferred reward is why any hand-maintained model of a company decays. It’s exactly what happens to the wiki nobody has updated since the person who wrote it left. Potential future benefit isn’t enough to compel people to do work, unless they’re passionate about it.
What’s actually different now
The maintainer no longer has to be a human. Prasenjit Sarkar, quoted in the Latent Space piece, put it about as well as it can be put: when an agent maintains the ontology as part of its own operation, updating definitions when it hits edge cases, the maintenance problem changes character. It doesn’t disappear. It changes from a job somebody has to remember to do into something that happens as a byproduct of the work.
That matters because maintenance is the only one of the three that never ends. You design the schema once and make the first entries once, but the upkeep continues for as long as anyone relies on it.
The other two are still open. AI may tackle them as well, but it’s too early to say. Populating a model automatically is early work, and deciding which types a business runs on is harder still. I haven’t seen anyone do either.
The obvious objection is the one my own Britney example raises. An agent can be confidently wrong too, and a model that drifts quietly is worse than a wiki everyone knows is stale. The difference is what scale does to ambiguity. On the open web, Britney Spears is a pop star, an accountant in Ohio, a dog and an album, and DAFFIE had to tell them apart across every page that used the name. We found heuristics that helped. It nonetheless remained a research problem. In the messages a business exchanges with its partners and vendors, Perot Systems is a company, and a second Perot Systems is unlikely to turn up in any of its threads. Ambiguity doesn’t disappear at that scale. The same name can be a customer in one thread and a supplier in another, and two people in your contacts can share a name. But a few hundred entities that mostly have one referent each is a different scale of problem than the open web, where a single name can mean a pop star, an accountant and a dog.
None of which means an agent will always get it right. It means the system has to be built for resilience, and there are two principles I’d suggest to whoever designs it. Require a person to approve any significant change, before it lands rather than in a report you read afterward. And make the result editable, so a person who spots a wrong line can fix it in place instead of filing a bug against a pipeline. An agent that writes to a model nobody can correct is the 2005 problem with a faster engine.
That is cheaper than it sounds, because the checking rides along with the use. Have the agent log what it learned before it updates a page and there’s a record of which message produced which change. DAFFIE had logs too; what it lacked was a reviewable number of them. A page about one customer is read by whoever is about to talk to that customer, or by the agent drafting for them, so a wrong line gets caught in the course of using it. The check is a byproduct of the work, the same as the update.
Why keep a model at all? Put the messages in a vector store, let the agent read them, and skip the schema. That works until the agent has to be right about the same thing twice. Retrieval returns what resembles the question, so an agent asked about a customer will happily hand back the thread about a competitor with a similar name, and when you correct it, there’s nowhere to put the correction, so tomorrow it reads the same messages and makes the same mistake.
That argues for a light ontology rather than a heavy one. The types carry the weight. Knowing that this entity is a customer, that one a competitor and that one a product lets an agent fetch the right set, and gives a correction something to attach to. You may not need the part that sank the earlier attempts, a fixed list of attributes every customer has to carry.
What we built in 2005 had a growing list of types and a growing set of attributes, and the shape was fixed: everything had to be an entity with attributes and values. What I’d build now is narrower in one direction and much wider in the other. A list of the types a company runs on, a list of the actual entities under each, and a page of prose about each one instead of a row of attributes. A page can hold anything true about that customer. One is watching costs because a new product line isn’t catching on. Another is distracted by a patent infringement suit. Neither fits in a CRM, and nobody enumerated “distracted by a patent infringement suit” in advance, because the agent reading the messages is not filling in a form.
The other half of the equation
Even if all three problems disappeared tomorrow, that would not be enough to make ontologies worth building. An ontology can be simple to build and still not repay the effort. Last time the work was hard and the reward never arrived at all. What you get for building one has to improve, and it has, for two reasons that did not hold in 2001 or 2005.
The first is that most of what a company knows never reaches a system of record. In a 2018 Panopto survey of 1,001 US employees at organizations of 200 people or more, run with YouGov, 42% of institutional knowledge was unique to the person holding it: not written down, and not known by a colleague either. That’s the launch that slipped, the partner gone quiet, the two people answering the same customer differently. Twenty years ago that knowledge existed too, but it sat in conversations nothing could read. Now it sits in messages, which a language model reads well. I’ve written elsewhere about the missing middle of company knowledge, and Jack Dorsey and Roelof Botha make the case for organizing a company around a model of itself in From Hierarchy to Intelligence.
The second is that agents need it. A model answering from raw message history re-derives the same conclusions on every run and pays for it every time. A maintained layer amortizes that, and it gives a correction somewhere to live that the next run will read. That’s why the ontology revival is happening among people building agents rather than among people building search.
Neither of those makes the schema easier to design. They make having one more valuable, and that value is what was missing last time.
Guardrails are the easy part. Deciding what to model and keeping it current are the hard ones, and all three of the systems I’ve described ran into both. Only upkeep now has a credible answer. On the evidence of the last twenty years, whenever a person is the one keeping a model current, that’s where the decay starts.
Whether it’s possible to learn the schema and populate it are still open questions. That’s why I’m building on this idea again rather than just telling you what failed before.