Skip to content
UWaterlooIRPublic

About

ixg: index grep built on Cottontail

Resources

Stars

0 stars

Watchers

0 watching

Forks

 
 

Repository files navigation

Cottontail

Coding agents: Read AGENTS.md before working in this repository.

Cottontail aims to provide a unified storage substrate for agentic search. Cottontail is built around annotative indexing, a unified approach to indexing that currently supports inverted indices, links, graph structures, JSON, other structured text, and dynamic indices. Transactions support immediate updates with ACID properties for multiple concurrent readers and writers. Planned support includes indexed regular expression search and dense retrieval; you can see these and other plans developing in the ai/ directory.

This repository also includes an initial version of a metadata layer for Cottontail, called Meadowlark. Meadowlark provides a simple filesystem-like layer over Cottontail, directly supporting JSONL and TSV files with structural annotations, as well as content-oriented support for flat text files and code. Meadowlark's metadata helps agents explore and understand previously unseen databases, with the long-term goal of establishing a universal standard for agentic stores.

Cottontail is the reference implementation of annotative indexing described in:

Charles L. A. Clarke. 2025. Annotative Indexing. Information Retrieval Research Journal 1, 1 (2025), 109–136. https://doi.org/10.54195/irrj.19910

Current command-line utilities live in apps/. Historical experiments and benchmarks live in expr/. Meadowlark-specific code lives in meadowlark/. The core Cottontail library lives in src/.

Demos

  • MS MARCO demo: build a Meadowlark collection, explore its structure and metadata, add ranking annotations, and run BM25 retrieval.
  • SSR agentic search demo: watch the video, build and serve the 1.6 TB ClimbMix collection, and explore Boolean search through the web interface, command-line clients, or an agent.
  • Examples from the IIRJ paper: TREC collection building and updating, and JSON tables.

Installation

On Ubuntu or Debian, install a C++ compiler, Git, curl, and the zlib headers:

sudo apt update
sudo apt install build-essential git curl zlib1g-dev

Cottontail is built with Bazel. The easiest way to install Bazel is to install Bazelisk under the name bazel:

sudo curl -L \
  https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-amd64 \
  -o /usr/local/bin/bazel
sudo chmod +x /usr/local/bin/bazel

The slightly magical part is the filename: /usr/local/bin/bazel is actually Bazelisk, a small launcher that downloads and runs Bazel. The first build will therefore take longer and requires an internet connection. On an ARM64 Linux system, use bazelisk-linux-arm64 instead of bazelisk-linux-amd64.

Now clone and build Cottontail:

git clone https://github.com/claclark/Cottontail.git
cd Cottontail
make building

The compiled command-line tools will be in bazel-bin/apps/. Use make fast instead when you want an optimized build.

About

ixg: index grep built on Cottontail

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages