Skip to content
← all posts
RAGKnowledge ManagementAI EngineeringGoogle OKFProduction Systems

Beyond Retrieval: Google's Open Knowledge Format and What It Means for RAG

Ask any engineering team how they're grounding their LLMs, and the answer is almost always RAG. Retrieval-augmented generation has become the default architecture — the thing you reach for when the model needs to work with your organization's knowledge instead of its own training data.

But RAG makes a quiet assumption: that the knowledge it retrieves from is well-organized, discoverable, and semantically coherent. In most enterprises, that assumption is wrong. The real problem isn't retrieval. The real problem is upstream — the knowledge itself is fragmented, inconsistent, and scattered across a dozen systems that were never designed to talk to each other.

Google's announcement of the Open Knowledge Format on June 12 tackles exactly this upstream problem. And understanding what OKF is — and what it isn't — matters for anyone building production AI systems.


The Problem RAG Doesn't Solve

In my experience, we've built RAG pipelines across multiple product surfaces. Every time we do, the hardest part is never the retrieval layer. It's getting the knowledge into a state where retrieval produces useful results.

The pattern is always the same. Business knowledge lives in Confluence wikis, Google Docs, Slack threads, database schemas, internal runbooks, and the heads of three people who've been at the company for five years. You build a RAG pipeline, point it at your corpus, and the retrieval results are noisy — because the corpus is noisy. Conflicting definitions of the same metric in different documents. Stale runbooks that describe systems that no longer exist. Tribal knowledge that was never written down.

RAG retrieves from a corpus. It does not fix the corpus. If the knowledge upstream is fragmented, inconsistent, or incomplete, retrieval just serves that fragmentation faster — with the veneer of AI confidence.

This is the knowledge quality problem. Most teams solve it by throwing more embedding models and re-ranking at it. That helps with relevance. It doesn't help with correctness, completeness, or consistency.


What OKF Actually Is

OKF — Open Knowledge Format — is Google's open-source, vendor-neutral specification (currently v0.1) for standardizing how organizational knowledge is packaged and shared with AI agents. It is not a product. It is not a database. It is a file format.

An OKF bundle is a directory of Markdown files with YAML frontmatter. Each file represents a single concept — a metric definition, a database table description, an API endpoint, a runbook. The concepts are interconnected via standard Markdown links, forming a navigable knowledge graph.

Anatomy of an OKF Bundle

ElementDescription
File formatStandard Markdown with YAML frontmatter
One file per conceptEach file describes exactly one thing — a metric, a table, an API, a runbook
Required metadatatype field in frontmatter (e.g., metric, table, api, runbook)
Optional metadatatitle, description, resource, tags, timestamp
LinkingStandard Markdown links between files create a navigable concept graph
StorageAny Git repo or filesystem — no proprietary runtime or database required

The key design decisions are deliberate. Human-readable AND machine-parseable. No proprietary SDK. No vendor lock-in. No special runtime. You can open an OKF bundle in any text editor and understand what's in it. An AI agent can parse the same bundle programmatically and navigate between concepts via the link graph.

OKF treats knowledge the way good engineering treats code: each concept has a single source of truth, with explicit metadata and explicit relationships. It's version-controllable, reviewable, and diff-able.

OKF vs RAG: Write Path vs Read Path

This is where most of the confusion lives. OKF and RAG are not competing approaches. They operate on different sides of the knowledge lifecycle.

OKF (Write Path)

Structures knowledge at rest. Defines how organizational knowledge is authored, curated, and maintained. Focuses on consistency, completeness, and navigability of the knowledge corpus itself.

RAG (Read Path)

Retrieves knowledge at runtime. Defines how an agent finds and surfaces relevant knowledge at inference time. Focuses on relevance, latency, and contextual fit of retrieved results.

OKF is about the write path — how knowledge gets organized, governed, and maintained before any AI system touches it. RAG is about the read path — how an agent queries that knowledge at inference time to ground its responses.

Treating them as competing is like arguing whether database schemas or SQL queries are more important. You need both. A perfectly structured knowledge base with no retrieval mechanism is inert. A retrieval pipeline over a messy corpus is noisy. The value comes from the combination.


Where RAG Still Wins

RAG remains the right architecture for a set of scenarios that OKF doesn't — and isn't designed to — address.

Dynamic, high-volume retrieval. When you need to search across millions of documents at inference time, semantic search over vector embeddings is the answer. OKF bundles are curated knowledge — they don't scale to the firehose of unstructured data that enterprises generate daily.

Real-time and external data. Web search augmentation, live API data, streaming content — anything where the knowledge changes faster than a curation process can keep up. RAG handles this naturally. OKF is designed for knowledge that's been reviewed and stabilized.

Unstructured exploration. When the agent doesn't know what it's looking for — research tasks, open-ended analysis, exploratory queries — RAG excels because retrieval can surface unexpected connections. OKF's curated graph is powerful for known concepts but can't surface what hasn't been explicitly authored.

RAG's strength is breadth and speed over large, messy corpora. It's the right tool when you can't — or shouldn't — pre-curate every piece of knowledge an agent might need.

Where OKF Changes the Game

OKF shines in exactly the areas where RAG struggles most: governed, curated, high-stakes knowledge for agentic systems.

Version-controlled knowledge. Because OKF bundles live in Git, every change to organizational knowledge has an author, a timestamp, a commit message, and a diff. When your AI agent tells a customer that your API rate limit is 1,000 requests per minute, you can trace that fact to a specific commit by a specific author. Try doing that with a chunk retrieved from a vector store.

Agent-navigable concept graphs. The Markdown link structure between OKF files creates an explicit knowledge graph. An agent can start at a metric definition, follow links to the underlying table schema, then to the API that serves it, then to the runbook for when it breaks. This is structured navigation, not probabilistic retrieval — and the difference matters for complex agentic workflows.

Audit-friendly knowledge governance. In regulated industries — and we operate in one — you need to demonstrate that the knowledge your AI systems act on is current, reviewed, and authorized. OKF's file-per-concept structure with explicit metadata makes this auditable by design.

OKF represents a shift in how we think about knowledge in AI systems. Knowledge stops being "documents the AI searches through" and becomes infrastructure — versioned, governed, and engineered with the same rigor as the code that consumes it.

Google's release of an Enrichment Agent that auto-drafts OKF documents from BigQuery datasets signals something important: they're betting that the bottleneck in AI grounding is not retrieval — it's knowledge authoring. Automating the write path is where the leverage is.


The Hybrid Architecture

The production architecture that makes sense is not OKF or RAG — it's OKF feeding into RAG.

3Knowledge Stack LayersOKF → Indexing → RAG Retrieval
2Complementary PathsWrite path (OKF) + Read path (RAG)

The stack works in three tiers:

  1. Knowledge tier (OKF). Curated, version-controlled OKF bundles define the authoritative knowledge. Subject matter experts author and review concepts. Changes go through pull requests. This is your source of truth.
  2. Indexing tier. OKF bundles are indexed into your vector store and search infrastructure. The structured metadata — type, tags, resource — enriches the embeddings and enables filtered retrieval.
  3. Retrieval tier (RAG). At inference time, your agent queries the index. Results from OKF-sourced content carry provenance metadata — the agent knows not just what it retrieved but where it came from, who authored it, and when it was last updated.
This is the same pattern we use for databases: schemas define the structure and constraints of the data (write path), queries retrieve it at runtime (read path). Nobody argues about whether schemas or queries are more important. Knowledge systems should work the same way.

The OKF-sourced documents in your RAG pipeline become high-trust, high-provenance results that can be ranked and weighted differently from unstructured documents. Your agent can distinguish between a curated metric definition and a Slack message that mentions the same metric in passing.


What This Means for Your Stack

If you're building production AI systems today, the action item is not to rip out your RAG pipeline and replace it with OKF. It's to recognize that you probably have a knowledge quality problem that no amount of retrieval optimization will fix.

The most common mistake in AI grounding is treating knowledge as "just documents" — dumping everything into a vector store and hoping the embedding model will sort out relevance, correctness, and currency. It won't. The model doesn't know that one document is an authoritative definition and another is a draft someone abandoned in 2023.

OKF offers a concrete, open-standard way to address the write path. It's v0.1 — early, evolving, and not yet battle-tested at enterprise scale. But the core insight is sound: knowledge needs structure, governance, and explicit relationships before retrieval can do its job well.

Whether you adopt OKF specifically or build your own knowledge curation layer, the principle is the same. Invest in the write path. Treat knowledge as infrastructure. Version it, review it, govern it — then let RAG do what it does best over a corpus you actually trust.

// key takeaway

RAG solves the retrieval problem. OKF solves the knowledge problem. The teams that build production AI systems treating knowledge with the same rigor as their codebase — versioned, reviewed, governed, and structured — will outperform teams that keep throwing retrieval optimizations at a fundamentally broken corpus.