diff --git a/.kb/agents.md b/.kb/agents.md index 4181069d..36a94f25 100644 --- a/.kb/agents.md +++ b/.kb/agents.md @@ -2,43 +2,67 @@ This repository follows strict conventions for organizing agent-oriented knowledge documents, and this document is required reading for any agent that wants to read or write these. -Read the top-level `.kb/agents.md` before continuing below. +The design of this structure has the following key goals: +- **Mechanical** - Encourages agents to read/write the knowledge base as main actors. +- **Generic** - Benefits any agentic workflow, no matter the editor or platform. +- **Distilled** - Avoids the use of verbose task or plan logs that pollute the context window. +- **Hierarchical** - Takes information closer to its context, avoids bloat in a single place. +- **Human** - Preserves information readily available and reviewable in a friendly format. -# Overview +IMPORTANT: Read the `.kb/agents.md` file. -Every directory in this repository, including the root, may have its own `AGENTS.md` file and `.kb/` subdirectory. The `AGENTS.md` file provides a general view for the directory, while more specific information is found in `.kb/*.md` files with dashed lowercase names (e.g. `.kb/special-relativity.md`). -The design of this structure has the following key goals: +# Laws -- **Mechanical** - Agents are the main actors reading and writing the knowledge base. -- **Generic** - Benefits any agentic workflow, no matter the editor or platform. -- **Distilled** - Avoids the use of verbose task or plan logs that pollute the context window. -- **Hierarchical** - Avoids excessive information in a single place that also pollutes the context window. -- **Human** - Information is readily available and reviewable in a friendly format. +## §1 - Organization + +Every directory in this repository may have its own `.kb/` subdirectory with `.kb/*.md` files with dashed lowercase names such as `.kb/special-relativity.md`. + +Every filename must be _globally unique_ inside the repository, so section references such as agents.md§1 are possible. +## §2 - Headers -# Important +The following are the ONLY top-level headers allowed in `.kb/*.md` files in this repository. Keep the section order identical to that presented below. -- Read local `AGENTS.md` files upon navigating directories. -- Keep the `AGENTS.md` and `.kb/*.md` files updated whenever there is something relevant to be documented or updated. The top-level `.kb/agents.md` must NOT be edited unless explicitly requested. -- Follow the header conventions outlined below. Only the _Preface_ header is required, and the other headers should be omitted if empty or trivial. +- _Preface_ - A brief introduction outlining the scope and relevance of a specific `*.md` file. This section MUST be at the top of every `.kb/*.md` file so agents can easily grep for it, and the last line of this section MUST be "IMPORTANT: Read the `.kb/agents.md` file." so rules are respected. +- _Laws_ - Essential directives outlining critical constraints, behaviors, and rules. The Laws are defined in _terse_ subsections `## § - ` as in this file, internally referenced as §3 and externally as agents.md§3 (this law). +- _Concepts_ - Structural design, details, references, etc. This is the only section that accepts open-ended subsections. +Keep two empty lines before top headers, and one empty line before subheaders. -# Headers +## §3 - Process -The following are the ONLY top-level headers allowed across the `.kb/*.md` files in this repository, to maintain semantic standardization across projects. Keep the ordering of created sections the exact same you see below. -Sub-headers are okay. +- Read the _Knowledge Base_ section inside the `AGENTS.md` file at the top of the repository. +- Keep the `.kb/*.md` files up-to-date: add or update content whenever something relevant needs documenting. The `.kb/agents.md` must NOT be edited unless explicitly requested. +- Follow the header conventions in §3. Only the _Preface_ header is required, and the other headers should be omitted if empty or trivial. +- If you find incorrect organization that breaks any of the laws in this document, fix it. +- Never change the subject of a Law - if obsolete, replace its header with `## §<N> - Revoked`. -- _Preface_ - A brief introduction outlining the scope and relevance of a specific `*.md` file. This section MUST be at the top of every `.kb/*.md` file so agents can easily grep for it, and the last line of this section MUST be "Read the top-level `.kb/agents.md` file before continuing below." so rules are followed. -- _Overview_ - High-level summary of the directory, subsystem or knowledge base layout at large. Do NOT use this to list files or directories. -- _Important_ - Essential directives for the agent outlining critical constraints, behaviors, or rules. -- _Headers_ - Global registry of header definitions, uniquely hosted at the root `.kb/agents.md` (this file). Do NOT use this header in any other document. -- _Architecture_ - Structural design details or boundary explanations for a given component. Only use this for software architecture concepts, NOT for directory outlining. Also avoid using this as a code reference (keep that in the code itself). -- _Directory_ - Only in `AGENTS.md` files, it briefly outlines the contents and structure of the directory the `AGENTS.md` file is in, and potentially nested small directories that do not justify their own `AGENTS.md` file. For `.kb/*.md` files, use _Documents_ instead. -- _Documents_ - Only in `AGENTS.md` files, it outlines the content of `.kb/*.md` files and also immediately nested `AGENTS.md` child files, to aid agent navigation. It MUST be the last section in the file. It's okay to also mention such filenames inline when they are relevant elsewhere in the text. +## §4 - AGENTS.md -For the _Directory_ and _Documents_ listings, format items as a dashed list starting with the file or directory name surrounded by backticks, a dash, and then a brief description: +The repository's `AGENTS.md` file MUST START with an ordered list of ALL `.kb/*.md` files, in this format: ``` -- `filename` - Terse summary. +# Knowledge Base + +IMPORTANT: Read the `.kb/agents.md` file. + +- [.kb/agents.md](.kb/agents.md) - Structure and rules of the knowledge base. You MUST read this file. +- [.kb/filename.md](.kb/filename.md) - Terse summary. +- [subdir/.kb/filename.md](subdir/.kb/filename.md) - Another terse summary. ``` + +# Concepts + +## Changelog + +- **2026-11-03** - Added revocations. Moved Headers before Process. Changed sentence pointing to .kb/agents.md, enforced it for AGENTS.md. +- **2026-10-30** - Major overhaul. Added: Laws, Concepts; Removed: Overview, Important, Architecture, Directory, Documents, AGENTS.md in subdirectories. +- **2026-09-24** - Enforced ordering of all headers. Minor tweaks for clarity. +- **2026-06-08** - Public release. + +## Version updates + +Only when EXPLICITLY requested to update the version of the knowledge base, replace this file with: + +https://raw.githubusercontent.com/canonical/knowledge-base/refs/heads/main/.kb/agents.md diff --git a/.kb/repo-overview.md b/.kb/repo-overview.md new file mode 100644 index 00000000..3cc02dfa --- /dev/null +++ b/.kb/repo-overview.md @@ -0,0 +1,48 @@ +# Preface + +This document holds the top-level overview of the Chisel repository. + +IMPORTANT: Read the `.kb/agents.md` file. + + +# Laws + +## §1 - Deterministic behavior + +Behavior must be deterministic and predictable, failing on the side of correctness and preserving existing user content by default. + +## §2 - Provider independence + +Chisel's operation and logic, including validation, must not rely on package contents or on information from package providers (currently the archive, in the future the bin store). Do as much as possible without fetching from package providers, so that operations stay fast and can work without network access. + +## §3 - Compatibility + +Do not break compatibility of the chisel-releases format or the public manifest packages, backward or forward, without a compelling reason. + + +# Concepts + +## Purpose + +Chisel cuts fine-grained slices out of Debian packages: it takes a selection of slices defined in chisel-releases, resolves their dependencies, downloads the corresponding packages from Ubuntu archives, extracts the selected files into a target root filesystem, runs Starlark mutation scripts, and generates manifests at paths requested by the selected slice definitions. The codebase is split across command entry points, an internals tree holding the carving pipeline, and public packages defining the manifest format. + +## Compatibility guidance + +When introducing a new feature or fixing a bug, consider both backward and forward compatibility: new behavior must keep working with previous Chisel releases, and parsing of release YAML files must accept or reject values so that future format evolutions remain possible. + +## Contents + +- `cmd/` - Binary entry points for the `chisel` application. +- `internal/` - Core internal packages: slice orchestration, setup, extraction, archive, cache, and supporting utilities. +- `public/` - Public API packages for external consumers. +- `snap/` - Snap packaging configuration for Chisel. +- `tests/` - End-to-end integration test suite. +- `docs/` - Documentation assets. +- `go.mod` - Go module definition. +- `go.sum` - Go dependency checksums. +- `spread.yaml` - Spread test configuration. +- `workshop.yaml` - Workshop configuration. +- `.golangci.yaml` - golangci-lint configuration. +- `README.md` - Project readme. +- `SECURITY.md` - Security policy. +- `LICENSE` - Project license. diff --git a/AGENTS.md b/AGENTS.md index f1ce01c3..d9a3fc8f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,45 +1,11 @@ -# Preface +# Knowledge Base -This repository hosts Chisel, a software tool for carving and cutting Debian packages. This `AGENTS.md` is the root index into the dispersed knowledge base; read it to orient yourself, then follow the links below for details. +IMPORTANT: Read the `.kb/agents.md` file. -Read the top-level `.kb/agents.md` file before continuing below. - - -# Overview - -Chisel cuts fine-grained slices out of Debian packages. It takes a selection of slices defined in chisel-releases, resolves their dependencies, downloads the corresponding packages from Ubuntu archives, extracts the selected files into a target root filesystem, runs Starlark mutation scripts, and generates manifests at paths requested by the selected slice definitions. The codebase is split across command entry points, an internals tree holding the carving pipeline, and public packages defining the manifest format. - - -# Important - -- Prefer deterministic, predictable behavior that fails on the side of correctness and preserves existing user content by default. -- Keep Chisel as independent as practical from incidental package archive contents. -- Treat changes to the chisel-releases format and the public manifest packages as compatibility-sensitive. - - -# Directory - -- `cmd/` - Binary entry points for the `chisel` application. -- `internal/` - Core internal packages: slice orchestration, setup, extraction, archive, cache, and supporting utilities. -- `public/` - Public API packages for external consumers. -- `snap/` - Snap packaging configuration for Chisel. -- `tests/` - End-to-end integration test suite. -- `docs/` - Documentation assets. -- `go.mod` - Go module definition. -- `go.sum` - Go dependency checksums. -- `spread.yaml` - Spread test configuration. -- `workshop.yaml` - Workshop configuration. -- `.golangci.yaml` - golangci-lint configuration. -- `README.md` - Project readme. -- `SECURITY.md` - Security policy. -- `LICENSE` - Project license. - - -# Documents - -- `.kb/agents.md` - General rules for the knowledge base reading and writing. -- `cmd/AGENTS.md` - CLI entry points. -- `internal/AGENTS.md` - Core internal packages. -- `public/AGENTS.md` - Public API packages. -- `snap/AGENTS.md` - Snap packaging. -- `tests/AGENTS.md` - Integration testing. +- [.kb/agents.md](.kb/agents.md) - Structure and rules of the knowledge base. You MUST read this file. +- [.kb/repo-overview.md](.kb/repo-overview.md) - Chisel overview, behavioral laws, and root directory layout. +- [cmd/.kb/chisel-entry-points.md](cmd/.kb/chisel-entry-points.md) - Binary entry points for the `chisel` application. +- [internal/.kb/internal-packages.md](internal/.kb/internal-packages.md) - Core internal packages and the carving pipeline. +- [public/.kb/public-packages.md](public/.kb/public-packages.md) - Public API packages and the license boundary. +- [snap/.kb/snap-packaging.md](snap/.kb/snap-packaging.md) - Snap packaging configuration. +- [tests/.kb/integration-suite.md](tests/.kb/integration-suite.md) - Spread-based end-to-end integration testing. diff --git a/cmd/AGENTS.md b/cmd/.kb/chisel-entry-points.md similarity index 57% rename from cmd/AGENTS.md rename to cmd/.kb/chisel-entry-points.md index 98b65bf4..2a5d219b 100644 --- a/cmd/AGENTS.md +++ b/cmd/.kb/chisel-entry-points.md @@ -1,14 +1,36 @@ # Preface -This document serves as the local knowledge base index for the `cmd/` directory. It outlines the scope and purpose of the entry points for the `chisel` binary. +This document covers the CLI entry points for the `chisel` binary in the `cmd/` directory. -Read the top-level `.kb/agents.md` file before continuing below. +IMPORTANT: Read the `.kb/agents.md` file. -# Overview -The `cmd/` directory contains the main binary entry points for the `chisel` application. Instead of housing core business logic, this directory is responsible for initializing the application, wiring up the CLI subsystem, and handling version information. Core logic is deferred to the packages in `internal/`. +# Laws -# Directory +## §1 - CLI boundary + +Do not add core business logic to `cmd/`; commands only wire up flags and arguments and delegate to the packages in `internal/`. + + +# Concepts + +## Scope + +The `cmd/` directory contains the main binary entry points for the `chisel` application, responsible for initializing the application, wiring up the CLI subsystem, and handling version information. + +## Debug commands + +The `debug` subcommand group has no stability guarantee: its commands may be removed without notice and may not work on non-development systems, so they are registered as hidden. + +## Version injection + +The application version is injected at build time by `mkversion.sh` (via `go:generate`), which writes the generated `version_generated.go`. + +## Testing hooks + +Tests override the `Stdin`/`Stdout`/`Stderr`/`ReadPassword` variables instead of touching the `os` streams directly. + +## Contents - `chisel/` - Contains the primary executable entry point for the `chisel` application. - `chisel/main.go` - Application bootstrap and CLI initialization. diff --git a/internal/AGENTS.md b/internal/.kb/internal-packages.md similarity index 92% rename from internal/AGENTS.md rename to internal/.kb/internal-packages.md index d71d9ee9..189c4147 100644 --- a/internal/AGENTS.md +++ b/internal/.kb/internal-packages.md @@ -1,14 +1,17 @@ # Preface -This is the index file for the `internal/` directory's knowledge base. It provides context about the core internal packages of Chisel. +This document covers the core internal packages and the carving pipeline in the `internal/` directory. -Read the top-level `.kb/agents.md` file before continuing below. +IMPORTANT: Read the `.kb/agents.md` file. -# Overview + +# Concepts + +## Scope The `internal/` directory houses the core business logic, components, and utilities of Chisel. It coordinates slice selection, dependency resolution, package fetching, file extraction, filesystem mutations, and manifest output. These packages are not part of Chisel's public API and must not be imported by external consumers. -# Architecture +## Pipeline Chisel's carving pipeline is coordinated by `slicer/`: @@ -23,7 +26,7 @@ flowchart LR Mutate --> Manifest["Generate requested manifests"] ``` -# Directory +## Contents - `slicer/` - Main orchestrator for a Chisel run. Receives a slice selection, drives all other internal packages (setup, archive, cache, deb, fsutil, scripts, manifestutil) to completion, and writes the final filesystem and manifest. - `setup/` - Parses chisel-releases YAML definitions into the `Release` model and resolves slice dependencies, path conflicts, and package contention. diff --git a/public/AGENTS.md b/public/.kb/public-packages.md similarity index 64% rename from public/AGENTS.md rename to public/.kb/public-packages.md index 3dd180c0..4a322ac4 100644 --- a/public/AGENTS.md +++ b/public/.kb/public-packages.md @@ -1,18 +1,28 @@ # Preface -This document describes the scope of the `public/` directory, which contains Chisel's public API packages intended for consumption by external tools. +This document covers the public API packages in the `public/` directory. -Read the top-level `.kb/agents.md` file before continuing below. +IMPORTANT: Read the `.kb/agents.md` file. -# Overview + +# Laws + +## §1 - License boundary + +Packages in `public/` and their dependencies must remain Apache-2.0 licensed. + + +# Concepts + +## Scope The `public/` directory houses the two packages that form Chisel's stable public contract. They provide a generic sorted JSON-lines database format and the Chisel manifest schema built on it, enabling third-party tools such as SBOM generators and vulnerability scanners to consume Chisel output without depending on internal packages. -# Important +## License check -- When changing these packages or their dependencies, run `go run .github/scripts/external-packages-license-check.go` to verify the Apache-2.0 license boundary. +When changing these packages or their dependencies, run `go run .github/scripts/external-packages-license-check.go` to verify the license boundary. -# Directory +## Contents - `jsonwall/` - Implements a generic database format with one JSON object per line, with fields and entries sorted for efficient search and iteration. - `manifest/` - Defines Chisel manifest schema 1.0 through the `Package`, `Slice`, `Path`, and `Content` record types, and uses `jsonwall` for serialization and deserialization. The carving pipeline applies Zstandard compression when generating manifest files. diff --git a/snap/AGENTS.md b/snap/.kb/snap-packaging.md similarity index 63% rename from snap/AGENTS.md rename to snap/.kb/snap-packaging.md index 365776be..2e042696 100644 --- a/snap/AGENTS.md +++ b/snap/.kb/snap-packaging.md @@ -1,13 +1,23 @@ # Preface -This document describes the scope of the `snap/` directory within the Chisel repository, providing context for automated agents navigating and modifying the snapcraft configuration. +This document covers the snap packaging configuration in the `snap/` directory. -Read the top-level `.kb/agents.md` file before continuing below. +IMPORTANT: Read the `.kb/agents.md` file. -# Overview + +# Laws + +## §1 - Strict confinement + +The snap must use strict confinement. + + +# Concepts + +## Scope The `snap/` directory contains the configuration necessary to build and package Chisel as a snap application. It defines the application metadata, execution confinement, and compilation steps for the `chisel` application using Go. -# Directory +## Contents - `snapcraft.yaml` - The primary manifest file defining the snap package. It configures strict confinement, utilizes the Go plugin, and outlines the build steps which depend on `cmd/mkversion.sh` for version injection. diff --git a/tests/AGENTS.md b/tests/.kb/integration-suite.md similarity index 51% rename from tests/AGENTS.md rename to tests/.kb/integration-suite.md index 266facab..5f967acb 100644 --- a/tests/AGENTS.md +++ b/tests/.kb/integration-suite.md @@ -1,20 +1,32 @@ # Preface -This file serves as the local knowledge base index for the `tests/` directory. It outlines the scope and procedures for end-to-end integration testing within the Chisel repository. +This document covers the end-to-end integration test suite in the `tests/` directory. -Read the top-level `.kb/agents.md` file before continuing below. +IMPORTANT: Read the `.kb/agents.md` file. -# Overview -The `tests/` directory contains the Spread integration test suite. It exercises the `chisel` binary end to end, covering behavior that cannot be validated within individual Go packages. +# Laws -# Important +## §1 - End-to-end scope -- **Execution**: Integration tests are run with `spread`, not with `go test`. Each test scenario is a subdirectory containing a `task.yaml` with shell-based assertions. -- **No build tags**: Unlike Go-based integration test suites, these tests require no `//go:build` directives. They are entirely shell-driven. -- **Pre-built binary**: Spread compiles and provisions the `chisel` binary as part of the test environment setup defined in `spread.yaml` at the repository root. +Integration tests must exercise the compiled `chisel` binary end to end, not package internals in isolation. -# Directory +## §2 - Spread runner + +Integration tests must be run with `spread`. + + +# Concepts + +## Scope + +The `tests/` directory contains the Spread integration test suite, covering behavior that cannot be validated within individual Go packages. + +## Running the suite + +Tests are not run with `go test`. Each test scenario is a subdirectory containing a `task.yaml` with shell-based assertions, and requires no `//go:build` directives. Spread compiles and provisions the `chisel` binary as part of the test environment setup defined in `spread.yaml` at the repository root. + +## Contents - `basic/` - Core slice extraction scenario verifying that files are correctly written to the target root filesystem and that mutation scripts are applied. - `find/` - Tests for the `chisel find` command, covering search by slice name and package.