Ask an AI assistant how to calculate a business metric, and the answer may depend on a table schema, a policy, a runbook, and an approved query. When those details are spread across different systems, finding the current and trustworthy answer becomes difficult.
The Open Knowledge Format (OKF) specification provides a simple shared format: ordinary Markdown files with YAML metadata. It helps teams organize, review, and share the context that people and AI agents need.
1. Why AI Systems Need a Knowledge Format
Modern language models can reason over a document when that document is placed in their context. The difficult part is usually not generating text. It is finding the right, current, organization-specific context, then knowing whether it is safe to rely on.
A reliable answer to “What does weekly active users mean?” may depend on definitions, schemas, policies, and approved queries scattered across different tools.
This fragmentation creates several practical problems:
- Every new agent integration needs custom connectors and custom interpretations of metadata.
- Important explanations, examples, and operational details are hard to move between tools.
- A generated answer can sound confident even when its supporting knowledge is stale or unreviewed.
- Teams cannot easily review knowledge changes with the pull requests, diffs, and history that they already use for code.
OKF formalizes the “LLM wiki” pattern: maintain a growing collection of files that agents can navigate and update, while people curate it through familiar engineering workflows. It is relevant to agentic systems, but it is not restricted to agents. A search index, documentation site, graph visualizer, or ordinary text editor can consume the same bundle.
1.1 The key design choice: make the knowledge itself portable
An OKF bundle is just a directory tree. Each non-reserved Markdown file is a concept, such as a dataset, API endpoint, runbook, business metric, or playbook. Its path is its identity within the bundle.
This choice gives the knowledge four useful properties:
- Human-readable: a teammate can open and review a concept without special software.
- Agent-readable: an agent can parse frontmatter, retrieve a body, and follow ordinary Markdown links.
- Versionable: Git can show who changed a definition, when it changed, and what changed.
- Portable: the same directory can be copied, archived, mounted, committed, or served by a different system.
OKF is intentionally a format, not a platform. It does not prescribe a database, vector store, model provider, query engine, or agent framework. It can complement a knowledge graph, a retrieval-augmented generation (RAG) system, or a metadata catalog, but it does not replace them.
2. What OKF Specifies, and What It Leaves Open
The format is minimal by design. Every concept document needs only one mandatory frontmatter key: type. The producer may add fields that are useful for its domain, and consumers must tolerate types and fields they do not recognize.
This creates a small interoperability contract without forcing every organization into one global ontology.
| OKF specifies | OKF does not specify |
|---|---|
| Markdown concept files and YAML frontmatter | A mandatory database or hosting service |
A required type field | A centrally registered taxonomy of concept types |
Reserved index.md and log.md filenames | A search, retrieval, or serving implementation |
| Link and path conventions | Domain schemas such as OpenAPI, Avro, or Protobuf |
| Optional provenance, trust, lifecycle, and computation metadata | A universal access-control policy or execution sandbox |
For example, a producer can use type: BigQuery Table, type: Incident Playbook, or type: Metric. A consumer that does not understand Incident Playbook should still preserve and display it as a generic concept rather than failing.
Here is a small bundle for product analytics:
product-analytics/
├── index.md
├── log.md
├── datasets/
│ ├── index.md
│ └── events.md
├── metrics/
│ ├── index.md
│ └── weekly-active-users.md
├── computations/
│ └── weekly-active-users.md
└── playbooks/
└── event-ingestion-delay.mdThe hierarchy helps an agent progressively discover the collection. Rather than loading every document into a limited context window, it can read the root index.md, choose a relevant directory, and then open a particular concept.
Normal Markdown links add relationships that a folder tree cannot express. For example, the weekly-active-users metric can link to its event table, its sanctioned computation, and the playbook for delayed events. If concepts are vertices and links are directed edges, the bundle can be viewed as a graph:
$$
G = (V, E), \qquad E = {(u, v) : \text{concept } u \text{ links to concept } v}.
$$
The directory supplies coarse organization, while $G$ captures cross-cutting relationships such as “depends on,” “documents,” or “uses for joins.” OKF does not impose a formal edge taxonomy. The surrounding prose explains the relationship, which keeps the base format simple.
3. Anatomy of a Concept Document
Every ordinary concept is a UTF-8 Markdown file with two parts:
- YAML frontmatter at the top, delimited by
---, contains concise fields that a tool can filter, route, or display. - A Markdown body contains the explanation, schemas, examples, links, and operational detail that people and models need to read.
The frontmatter pattern is already familiar to many static-site and note-taking tools. It uses YAML for structured metadata and standard Markdown for the body.
3.1 The smallest valid concept
The following document is conformant because type is non-empty:
---
type: Team Glossary Term
---
# Activation
An activation occurs when a newly registered user completes a qualifying action.In practice, use the recommended fields whenever they are available:
title: a reader-friendly display name.description: a one-sentence summary for search results, indexes, and previews.resource: a canonical URI for the underlying asset, if there is one.tags: short labels for cross-cutting categories.
The body has no mandatory headings. However, structural Markdown is strongly preferable to a long wall of prose. Headings, tables, lists, fenced code blocks, and links provide useful retrieval boundaries for both people and agents. # Schema, # Examples, and # Computation have conventional meanings in the specification.
Two filenames have special meaning at every directory level:
index.mdlists contents and supports progressive disclosure. It usually has headings followed by linked entries with short descriptions.log.mdrecords date-grouped updates, newest first, using ISO 8601 dates such as2026-07-29.
Neither of the above filenames may be used for a concept document. index.md and log.md are structural files: they organize and chronicle the bundle. Only concept files carry a type field.
There is one narrow exception: a root-level index.md may include okf_version: "0.2" in its frontmatter to declare which spec version the bundle targets. That single key aside, it remains a navigation listing, not a concept.
4. Trust Is Metadata, Not a Vibe
Plain Markdown alone answers “what does this document say?” Agent-maintained knowledge also needs to answer: “Where did this claim come from? Who checked it? Is it still current?”
OKF makes these questions queryable through optional frontmatter families. Optional does not mean unimportant. It means a consumer can still use a small hand-written bundle, while a high-stakes workflow can demand stronger signals.
4.1 Provenance with sources
The sources field records materials from which a concept was derived. Each source entry requires a resource and may include a stable id, title, author, usage_count, and last_modified.
Use a stable source ID when a specific claim in the body needs attribution. A Markdown footnote label can match that ID, allowing a consumer to connect a claim to structured source metadata without relying on a fragile positional list.
sources:
- id: metric-policy
resource: https://intranet.example.com/product/metrics-policy
title: Product metrics policy
author: human:analytics-governance
last_modified: 2026-07-10
- id: wau-dashboard
resource: https://analytics.example.com/dashboards/42
title: Weekly product-health dashboard
usage_count: 3200
usage_window: { from: 2026-07-01, to: 2026-07-28 }These are signals, not a universal truth score. A dashboard viewed 3,200 times is likely alive and visible, but it is not automatically correct. The specification deliberately avoids storing a single credibility number because organizations weight authority, recency, and adoption differently.
4.2 Generated, verified, and stale are different states
Three related fields carry distinct information:
generatedsays who or what last produced the current content and when.verifiedrecords one or more independent checks against the concept’s sources or resource.stale_aftersets an absolute date on or after which the concept should be treated as stale.
Actors use a simple convention: <producer>/<version> for tools and agents, human:<id> for people, and process:<id> for automated processes. Consumers derive a trust tier for a concept $c$ from its verified field:
$$
\operatorname{trust\,tier}(c) =
\begin{cases}
0 & \text{if no verification is recorded} \\
1 & \text{if only non-human verifiers are recorded} \\
2 & \text{if at least one verifier has the } \mathrm{human:} \text{ prefix.}
\end{cases}
$$
The labels for these values are unverified, machine-confirmed, and human-reviewed. This is a useful routing signal, not authorization. A human-reviewed document can still be wrong, and an unverified document may still contain useful early research.
Freshness is a plain date comparison:
$$
\operatorname{stale}(c, t) = \mathbb{1}[t \geq \operatorname{stale_after}(c)].
$$
The absolute-date design is intentional. A consumer can determine staleness without guessing when a relative time-to-live began. Use status: draft, status: stable, or status: deprecated to communicate lifecycle state. When missing, status defaults to stable.
For time-sensitive systems, this metadata works well with the broader principle of point-in-time correctness: a consumer should be able to identify the policy and evidence that were valid at the time a claim or result was used.
4.3 Attested computations close an important gap
A concept can accurately define a number and still leave an agent room to run the wrong query. For example, an agent may interpret “weekly active users” correctly but select an incorrect event filter or time boundary.
An Attested Computation is a dedicated OKF concept that holds the sanctioned way to calculate a value. It declares:
runtime, such asbigquery,postgres,dbt, orpython.parameters, the typed values an agent is allowed to provide.computation, either inline under# Computationor stored in a referenced file.executor, which identifies run instructions and the receipt fields that must be returned.attester, deterministic code that inspects the receipt and returns a verdict.
The important separation is between verification and attestation. Verification checks whether a document definition still matches policy. Attestation checks whether one particular result was produced by the sanctioned computation with the supplied parameters. The latter is runtime evidence, not a permanent assertion stored in the bundle.
5. Build a Small, Useful OKF Bundle
Start with one high-value workflow instead of attempting to document an entire organization. A metric that repeatedly causes confusion, a frequently used table, or a recurring incident is an excellent first concept.
5.1 Author a metric concept
The following example describes weekly active users and links to a separate computation. It uses all major trust and lifecycle fields, but they remain optional in the format.
---
type: Metric
title: Weekly active users
description: Distinct registered users who completed a qualifying event in a rolling seven-day window.
tags: [product, engagement, wau]
status: stable
generated: { by: catalog-sync/1.4.0, at: 2026-07-28T02:15:00Z }
verified: { by: human:product-analytics, at: 2026-07-28T10:00:00Z }
stale_after: 2027-01-01
sources:
- id: metric-policy
resource: https://intranet.example.com/product/metrics-policy
title: Product metrics policy
author: human:analytics-governance
last_modified: 2026-07-10
---
# Definition
Weekly active users (WAU) are distinct `user_id` values associated with a
qualifying event during the preceding seven complete days. Exclude internal
test accounts and users who have deleted their account.[^metric-policy]
The value is produced by the [approved WAU computation](/computations/weekly-active-users.md).
The event fields are documented in the [events dataset](/datasets/events.md).
[^metric-policy]: Product metrics policyThe metric is readable on its own. A consumer can follow the links only when it needs implementation detail, which is precisely the progressive-disclosure behavior that keeps agent context focused.
5.2 Put the executable definition in its own concept
Here is a matching computation concept. The code shown is deliberately simple. In production, the executor and attester should point to reviewed, access-controlled artifacts, such as a packaged agent skill that wraps the run instructions.
---
type: Attested Computation
title: Weekly active users calculation
description: Approved BigQuery calculation for rolling seven-day weekly active users.
status: stable
runtime: bigquery
parameters:
- { name: window_end, type: date, required: true }
executor:
resource: /references/skills/run-bigquery.md
receipt: [job_id, executed_sql, result]
attester:
resource: /references/attesters/wau_sql_check.py
generated: { by: catalog-sync/1.4.0, at: 2026-07-28T02:15:00Z }
verified: { by: human:product-analytics, at: 2026-07-28T10:00:00Z }
stale_after: 2027-01-01
---
# Computation
```sql
SELECT COUNT(DISTINCT user_id) AS weekly_active_users
FROM `analytics.events`
WHERE event_timestamp >= TIMESTAMP_SUB(TIMESTAMP(@window_end), INTERVAL 7 DAY)
AND event_timestamp < TIMESTAMP(@window_end)
AND is_internal_user = FALSE
AND account_deleted_at IS NULL;
```Notice the boundary around window_end. Documentation must define such details clearly because different interpretations of “the last seven days” can yield different values. This is one reason that agent knowledge must include not only facts but also approved procedures.
5.3 Add a small validator to continuous integration
The format is simple enough to validate without a specialized platform. The following runnable Python script checks the core OKF conformance rules for concept files. It uses PyYAML to parse frontmatter.
from __future__ import annotations
import re
import sys
from pathlib import Path
import yaml
FRONTMATTER = re.compile(r"\A---[ \t]*\r?\n(.*?)\r?\n---[ \t]*(?:\r?\n|\Z)", re.DOTALL)
RESERVED_FILENAMES = {"index.md", "log.md"}
def load_frontmatter(path: Path) -> dict:
"""Return one concept's YAML frontmatter or raise a clear error."""
match = FRONTMATTER.match(path.read_text(encoding="utf-8"))
if not match:
raise ValueError("missing or malformed YAML frontmatter")
data = yaml.safe_load(match.group(1))
if not isinstance(data, dict):
raise ValueError("frontmatter must be a YAML mapping")
return data
def validate_bundle(bundle_root: Path) -> int:
errors: list[str] = []
for path in bundle_root.rglob("*.md"):
if path.name in RESERVED_FILENAMES:
continue
try:
frontmatter = load_frontmatter(path)
except (OSError, ValueError, yaml.YAMLError) as error:
errors.append(f"{path}: {error}")
continue
concept_type = frontmatter.get("type")
if not isinstance(concept_type, str) or not concept_type.strip():
errors.append(f"{path}: required non-empty 'type' is missing")
for source in frontmatter.get("sources", []):
if not isinstance(source, dict) or not source.get("resource"):
errors.append(f"{path}: every 'sources' item needs 'resource'")
if concept_type == "Attested Computation" and not frontmatter.get("runtime"):
errors.append(f"{path}: an Attested Computation needs 'runtime'")
if errors:
print("OKF validation failed:")
print("\n".join(f" - {error}" for error in errors))
return 1
print(f"OKF validation passed for {bundle_root}")
return 0
if __name__ == "__main__":
root = Path(sys.argv[1]) if len(sys.argv) == 2 else Path(".")
raise SystemExit(validate_bundle(root))This is intentionally a narrow validator. A production implementation can additionally check links, enforce an organization-specific type vocabulary, verify timestamps, build indexes, flag stale concepts, and invoke deterministic attesters. Keep those stricter policies separate from baseline OKF conformance so that a useful external bundle does not become unreadable merely because it lacks your local conventions.
6. How OKF Fits Into RAG and Agent Architectures
OKF is not a retrieval algorithm. It is a durable source representation that can improve the inputs to retrieval and agent workflows.
6.1 Is OKF competitive with, or complementary to, RAG?
OKF and retrieval-augmented generation (RAG) operate at different layers, so they are usually complementary rather than competitive. RAG is an application architecture: it retrieves relevant evidence at question time and supplies that evidence to a language model. OKF is a knowledge representation and interchange format: it defines how concepts, their metadata, and their links can be stored and exchanged before any retrieval system is built.
| Question | OKF | RAG |
|---|---|---|
| Primary job | Make organizational knowledge portable, inspectable, and reviewable | Find relevant evidence and ground a model response at query time |
| Main artifact | A directory of Markdown concepts with YAML metadata and links | A retrieval pipeline, index, retriever, and generation step |
| When it acts | During authoring, review, sharing, and indexing | When a user or agent asks a question |
| Can it work alone? | Yes, for documentation, Git-based review, static sites, or direct agent navigation | Yes, over PDFs, web pages, databases, or other document sources |
| What it does not provide | Ranking, embeddings, vector search, or answer generation | A portable, human-friendly canonical format for the source knowledge |
An OKF bundle can therefore be one high-quality source for a RAG system, but it does not require one. A team may use OKF as a reviewed knowledge base that people browse directly or that agents traverse through index.md files. Conversely, a RAG system can retrieve from an existing wiki, ticket system, or database without converting its contents to OKF.
The combination is useful when a team wants both a reliable source layer and fast query-time retrieval. OKF contributes explicit concept boundaries, lifecycle and trust metadata, direct source links, and Git-friendly review. A RAG pipeline contributes chunking, indexing, filtering, ranking, reranking, and context assembly. Keeping these responsibilities separate also makes replacement easier: a team can change vector databases or retrievers without rewriting the knowledge bundle, or publish the same bundle to a documentation site and an agent workflow.
6.2 Document ingestion and retrieval
A document-ingestion pipeline can scan a bundle, parse the frontmatter, apply a chunking strategy to the Markdown body, and store embeddings and metadata in a retrieval index. At query time, a system may filter on type, tags, status, trust tier, or stale state before semantic retrieval.
For example, a production assistant could prefer status: stable concepts, surface a warning for content whose stale_after date has passed, and require human-reviewed concepts before answering a sensitive financial-policy question. These are product decisions, not universal OKF mandates.
The bundle also gives RAG evaluation a clearer target. Instead of judging only whether a final answer resembles a reference answer, evaluate whether the system retrieved the intended concept, followed relevant links, cited sources correctly, and respected trust or freshness requirements.
An agent does not need every file at once. A reasonable navigation loop is:
- Read the bundle-root
index.mdto identify relevant domains. - Read a domain
index.mdto locate promising concepts. - Read the selected concept and inspect its frontmatter before treating the body as authoritative.
- Follow only the links needed to resolve a definition, dependency, or computation.
- State the concept’s source, trust tier, and freshness when these materially affect the answer.
This approach is especially helpful when an agent performs retrieval and action together. A plain answer may cite a metric definition, while a tool-using agent can discover the linked Attested Computation and use the declared, parameter-limited procedure.
6.4 Data catalogs and knowledge graphs
Organizations can export existing catalog metadata into initial concepts, then enrich those concepts with prose, examples, policy links, and review signals. Because links form a graph, the same bundle can feed a visualization or graph-analysis tool. The official reference repository includes a static visualizer and sample bundles, including GA4, Stack Overflow, and Bitcoin datasets, in the OKF reference implementation.
Do not confuse an OKF link graph with a full semantic knowledge graph. A graph database may carry typed edges, logical constraints, inference rules, and scalable graph queries. OKF offers a lightweight, inspectable interchange layer that can coexist with those systems; see when to use a knowledge graph over plain RAG for that tradeoff.
7. Practical Guidance and Failure Modes
The simplicity of Markdown files is a strength only when teams establish responsible operating habits.
7.1 Recommended practices
- Begin with a small, high-value bundle. Document a few concepts that agents and people repeatedly need. Build confidence before attempting broad coverage.
- Make ownership visible. Assign a team or reviewer for high-impact concepts, and define review expectations for
draft,stable, anddeprecatedcontent. - Treat agent-generated content as a change proposal. Store
generatedmetadata, require source links, and route important updates through review. - Set realistic freshness dates. A stale concept is safer when clearly flagged than when silently assumed to be current.
- Use durable, direct sources. Prefer canonical policies, schemas, and source repositories over a link to a chat transcript or a temporary dashboard view.
- Keep concepts focused. One concept should answer one coherent question. Split long documents into linked concepts when their review cycles or trust states differ.
- Validate in pull requests. At minimum, check parseable frontmatter and non-empty
type. Add organization-specific checks gradually. - Preserve unknown fields. If a tool reads and writes OKF, it should not silently remove extensions it does not understand.
7.2 Common mistakes
- Using OKF as an access-control system: trust tiers are advisory metadata. Enforce permissions in the storage, retrieval, and execution layers.
- Treating
usage_countas an accuracy score: popularity can show liveness, not correctness or suitability for every use case. - Putting secrets in a bundle: Markdown is easy to copy and index. Store credentials and sensitive values in an appropriate secret-management system, then reference safe operational procedures. See protecting privacy in the age of AI for the broader data-handling concerns this raises.
- Allowing an agent to rewrite an approved query: an Attested Computation is useful only when the consumer binds declared parameters rather than letting the model alter the computation.
- Hiding stale content: retain deprecated or stale concepts for historical links, but make their lifecycle state prominent to consumers.
- Overdesigning the initial taxonomy: the specification intentionally avoids a central type registry. Start with descriptive types that serve your workflow, then standardize locally when patterns emerge.
For a wider view of agent reliability, pair these practices with guardrails for LLMs. OKF can make evidence and approved procedures visible, but it cannot by itself stop prompt injection, enforce authorization, or guarantee that an agent follows instructions.
7.3 Limitations and When Another Representation Is Better
OKF is a good fit when knowledge must be reviewed, moved, and read by both humans and tools. It is not the best primary representation for every kind of information.
- Use OpenAPI, Protobuf, or Avro when strict machine-to-machine schemas and generated clients are the primary need.
- Use a relational database or data warehouse for large, frequently updated structured data and analytical queries.
- Use a graph database when typed relations, formal constraints, and high-scale graph traversal are central requirements.
- Use a secrets manager for credentials and access tokens.
- Use domain-specific governance and access-control systems when legal, compliance, or row-level security rules must be enforced.
The most effective architecture is often hybrid: source systems hold authoritative operational data, specialized schemas describe strict interfaces, and an OKF bundle captures the contextual knowledge that helps people and agents use those systems correctly.
Summary
Open Knowledge Format is a small open specification for representing knowledge as a portable directory of Markdown concepts with YAML frontmatter. Its minimal core makes it easy to adopt, while OKF’s optional provenance, verification, freshness, lifecycle, and attested-computation fields address the trust questions that become essential in agent-maintained knowledge bases.
Start by choosing one recurring decision or operational workflow. Create a small bundle with a root index, two or three focused concepts, direct source links, and a simple validation check. Then connect it to the retrieval or agent workflow that needs the knowledge. Read the full OKF specification, inspect the reference bundles and tooling, and adapt the format with local conventions that make your team’s knowledge easier to trust and reuse.
Silpa brings 5 years of experience in working on diverse ML projects, specializing in designing end-to-end ML systems tailored for real-time applications. Her background in statistics (Bachelor of Technology) provides a strong foundation for her work in the field. Silpa is also the driving force behind the development of the content you find on this site.
Happy is a seasoned ML professional with over 15 years of experience. His expertise spans various domains, including Computer Vision, Natural Language Processing (NLP), and Time Series analysis. He holds a PhD in Machine Learning from IIT Kharagpur and has furthered his research with postdoctoral experience at INRIA-Sophia Antipolis, France. Happy has a proven track record of delivering impactful ML solutions to clients.
Subscribe to our newsletter!
