Coding agents: Read AGENTS.md before working in this repository.
Cottontail aims to provide a unified storage substrate for agentic search.
Cottontail is built around annotative indexing, a unified approach to indexing
that currently supports inverted indices, links, graph structures, JSON, other
structured text, and dynamic indices. Transactions support immediate
updates with ACID properties for multiple concurrent readers and writers.
Planned support includes indexed regular expression search and dense retrieval;
you can see these and other plans developing in the ai/ directory.
This repository also includes an initial version of a metadata layer for Cottontail, called Meadowlark. Meadowlark provides a simple filesystem-like layer over Cottontail, directly supporting JSONL and TSV files with structural annotations, as well as content-oriented support for flat text files and code. Meadowlark's metadata helps agents explore and understand previously unseen databases, with the long-term goal of establishing a universal standard for agentic stores.
Cottontail is the reference implementation of annotative indexing described in:
Charles L. A. Clarke. 2025. Annotative Indexing. Information Retrieval Research Journal 1, 1 (2025), 109–136. https://doi.org/10.54195/irrj.19910
Current command-line utilities live in apps/. Historical experiments
and benchmarks live in expr/. Meadowlark-specific code lives in
meadowlark/. The core Cottontail library lives in src/.
- MS MARCO demo: build a Meadowlark collection, explore its structure and metadata, add ranking annotations, and run BM25 retrieval.
- SSR agentic search demo: watch the video, build and serve the 1.6 TB ClimbMix collection, and explore Boolean search through the web interface, command-line clients, or an agent.
- Examples from the IIRJ paper: TREC collection building and updating, and JSON tables.
On Ubuntu or Debian, install a C++ compiler, Git, curl, and the zlib headers:
sudo apt update
sudo apt install build-essential git curl zlib1g-devCottontail is built with Bazel. The easiest way to
install Bazel is to install
Bazelisk under the name bazel:
sudo curl -L \
https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-amd64 \
-o /usr/local/bin/bazel
sudo chmod +x /usr/local/bin/bazelThe slightly magical part is the filename: /usr/local/bin/bazel is actually
Bazelisk, a small launcher that downloads and runs Bazel. The first build will
therefore take longer and requires an internet connection. On an ARM64 Linux
system, use bazelisk-linux-arm64 instead of bazelisk-linux-amd64.
Now clone and build Cottontail:
git clone https://github.com/claclark/Cottontail.git
cd Cottontail
make buildingThe compiled command-line tools will be in bazel-bin/apps/. Use make fast
instead when you want an optimized build.