Repository navigation
ai: update knowledge base to new structure #347
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Preface | ||
|
|
||
| This document holds the top-level overview of the Chisel repository. | ||
|
|
||
| IMPORTANT: Read the `.kb/agents.md` file. | ||
|
|
||
|
|
||
| # Laws | ||
|
|
||
| ## §1 - Deterministic behavior | ||
|
|
||
| Behavior must be deterministic and predictable, failing on the side of correctness and preserving existing user content by default. | ||
|
|
||
| ## §2 - Provider independence | ||
|
|
||
| Chisel's operation and logic, including validation, must not rely on package contents or on information from package providers (currently the archive, in the future the bin store). Do as much as possible without fetching from package providers, so that operations stay fast and can work without network access. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Chisel's operation and logic (...) must not rely on package contents or on information from (...) the archive". are we changing chisel in a way in which i was not aware of? 😄 more seriously, the previous workding was better imo: "as independent as practical from incidental package archive contents". as written it sounds like its going very strongly against the literal thing chisel is supposed to do. also, leave bin store out until it exists? no strong opinions on that one though tbh. |
||
|
|
||
| ## §3 - Compatibility | ||
|
|
||
| Do not break compatibility of the chisel-releases format or the public manifest packages, backward or forward, without a compelling reason. | ||
|
|
||
|
|
||
| # Concepts | ||
|
|
||
| ## Purpose | ||
|
|
||
| Chisel cuts fine-grained slices out of Debian packages: it takes a selection of slices defined in chisel-releases, resolves their dependencies, downloads the corresponding packages from Ubuntu archives, extracts the selected files into a target root filesystem, runs Starlark mutation scripts, and generates manifests at paths requested by the selected slice definitions. The codebase is split across command entry points, an internals tree holding the carving pipeline, and public packages defining the manifest format. | ||
|
|
||
| ## Compatibility guidance | ||
|
|
||
| When introducing a new feature or fixing a bug, consider both backward and forward compatibility: new behavior must keep working with previous Chisel releases, and parsing of release YAML files must accept or reject values so that future format evolutions remain possible. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. reads a bit weird. how about something like "new versions of chisel must be able to keep reading old releases, and, where possible, release format changes stay parseable by older chisel versions." edit: i mean the "new behavior must keep working with previous Chisel releases" bit |
||
|
|
||
| ## Contents | ||
|
|
||
| - `cmd/` - Binary entry points for the `chisel` application. | ||
| - `internal/` - Core internal packages: slice orchestration, setup, extraction, archive, cache, and supporting utilities. | ||
| - `public/` - Public API packages for external consumers. | ||
| - `snap/` - Snap packaging configuration for Chisel. | ||
| - `tests/` - End-to-end integration test suite. | ||
| - `docs/` - Documentation assets. | ||
| - `go.mod` - Go module definition. | ||
| - `go.sum` - Go dependency checksums. | ||
| - `spread.yaml` - Spread test configuration. | ||
| - `workshop.yaml` - Workshop configuration. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As an aside, this should go to a dotted file. |
||
| - `.golangci.yaml` - golangci-lint configuration. | ||
| - `README.md` - Project readme. | ||
| - `SECURITY.md` - Security policy. | ||
| - `LICENSE` - Project license. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's please avoid such listings, unless they are really adding value. Pretty much everything in the list at the moment is obvious from context at the moment. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,45 +1,11 @@ | ||
| # Preface | ||
| # Knowledge Base | ||
|
|
||
| This repository hosts Chisel, a software tool for carving and cutting Debian packages. This `AGENTS.md` is the root index into the dispersed knowledge base; read it to orient yourself, then follow the links below for details. | ||
| IMPORTANT: Read the `.kb/agents.md` file. | ||
|
|
||
| Read the top-level `.kb/agents.md` file before continuing below. | ||
|
|
||
|
|
||
| # Overview | ||
|
|
||
| Chisel cuts fine-grained slices out of Debian packages. It takes a selection of slices defined in chisel-releases, resolves their dependencies, downloads the corresponding packages from Ubuntu archives, extracts the selected files into a target root filesystem, runs Starlark mutation scripts, and generates manifests at paths requested by the selected slice definitions. The codebase is split across command entry points, an internals tree holding the carving pipeline, and public packages defining the manifest format. | ||
|
|
||
|
|
||
| # Important | ||
|
|
||
| - Prefer deterministic, predictable behavior that fails on the side of correctness and preserves existing user content by default. | ||
| - Keep Chisel as independent as practical from incidental package archive contents. | ||
| - Treat changes to the chisel-releases format and the public manifest packages as compatibility-sensitive. | ||
|
|
||
|
|
||
| # Directory | ||
|
|
||
| - `cmd/` - Binary entry points for the `chisel` application. | ||
| - `internal/` - Core internal packages: slice orchestration, setup, extraction, archive, cache, and supporting utilities. | ||
| - `public/` - Public API packages for external consumers. | ||
| - `snap/` - Snap packaging configuration for Chisel. | ||
| - `tests/` - End-to-end integration test suite. | ||
| - `docs/` - Documentation assets. | ||
| - `go.mod` - Go module definition. | ||
| - `go.sum` - Go dependency checksums. | ||
| - `spread.yaml` - Spread test configuration. | ||
| - `workshop.yaml` - Workshop configuration. | ||
| - `.golangci.yaml` - golangci-lint configuration. | ||
| - `README.md` - Project readme. | ||
| - `SECURITY.md` - Security policy. | ||
| - `LICENSE` - Project license. | ||
|
|
||
|
|
||
| # Documents | ||
|
|
||
| - `.kb/agents.md` - General rules for the knowledge base reading and writing. | ||
| - `cmd/AGENTS.md` - CLI entry points. | ||
| - `internal/AGENTS.md` - Core internal packages. | ||
| - `public/AGENTS.md` - Public API packages. | ||
| - `snap/AGENTS.md` - Snap packaging. | ||
| - `tests/AGENTS.md` - Integration testing. | ||
| - [.kb/agents.md](.kb/agents.md) - Structure and rules of the knowledge base. You MUST read this file. | ||
| - [.kb/repo-overview.md](.kb/repo-overview.md) - Chisel overview, behavioral laws, and root directory layout. | ||
| - [cmd/.kb/chisel-entry-points.md](cmd/.kb/chisel-entry-points.md) - Binary entry points for the `chisel` application. | ||
| - [internal/.kb/internal-packages.md](internal/.kb/internal-packages.md) - Core internal packages and the carving pipeline. | ||
| - [public/.kb/public-packages.md](public/.kb/public-packages.md) - Public API packages and the license boundary. | ||
| - [snap/.kb/snap-packaging.md](snap/.kb/snap-packaging.md) - Snap packaging configuration. | ||
| - [tests/.kb/integration-suite.md](tests/.kb/integration-suite.md) - Spread-based end-to-end integration testing. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,36 @@ | ||
| # Preface | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My vote is to drop this one as well. This is an unrelated collection of low-value ideas. |
||
|
|
||
| This document serves as the local knowledge base index for the `cmd/` directory. It outlines the scope and purpose of the entry points for the `chisel` binary. | ||
| This document covers the CLI entry points for the `chisel` binary in the `cmd/` directory. | ||
|
|
||
| Read the top-level `.kb/agents.md` file before continuing below. | ||
| IMPORTANT: Read the `.kb/agents.md` file. | ||
|
|
||
| # Overview | ||
|
|
||
| The `cmd/` directory contains the main binary entry points for the `chisel` application. Instead of housing core business logic, this directory is responsible for initializing the application, wiring up the CLI subsystem, and handling version information. Core logic is deferred to the packages in `internal/`. | ||
| # Laws | ||
|
|
||
| # Directory | ||
| ## §1 - CLI boundary | ||
|
|
||
| Do not add core business logic to `cmd/`; commands only wire up flags and arguments and delegate to the packages in `internal/`. | ||
|
|
||
|
|
||
| # Concepts | ||
|
|
||
| ## Scope | ||
|
|
||
| The `cmd/` directory contains the main binary entry points for the `chisel` application, responsible for initializing the application, wiring up the CLI subsystem, and handling version information. | ||
|
|
||
| ## Debug commands | ||
|
|
||
| The `debug` subcommand group has no stability guarantee: its commands may be removed without notice and may not work on non-development systems, so they are registered as hidden. | ||
|
|
||
| ## Version injection | ||
|
|
||
| The application version is injected at build time by `mkversion.sh` (via `go:generate`), which writes the generated `version_generated.go`. | ||
|
|
||
| ## Testing hooks | ||
|
|
||
| Tests override the `Stdin`/`Stdout`/`Stderr`/`ReadPassword` variables instead of touching the `os` streams directly. | ||
|
|
||
| ## Contents | ||
|
|
||
| - `chisel/` - Contains the primary executable entry point for the `chisel` application. | ||
| - `chisel/main.go` - Application bootstrap and CLI initialization. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,17 @@ | ||
| # Preface | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same. |
||
|
|
||
| This is the index file for the `internal/` directory's knowledge base. It provides context about the core internal packages of Chisel. | ||
| This document covers the core internal packages and the carving pipeline in the `internal/` directory. | ||
|
|
||
| Read the top-level `.kb/agents.md` file before continuing below. | ||
| IMPORTANT: Read the `.kb/agents.md` file. | ||
|
|
||
| # Overview | ||
|
|
||
| # Concepts | ||
|
|
||
| ## Scope | ||
|
|
||
| The `internal/` directory houses the core business logic, components, and utilities of Chisel. It coordinates slice selection, dependency resolution, package fetching, file extraction, filesystem mutations, and manifest output. These packages are not part of Chisel's public API and must not be imported by external consumers. | ||
|
|
||
| # Architecture | ||
| ## Pipeline | ||
|
|
||
| Chisel's carving pipeline is coordinated by `slicer/`: | ||
|
|
||
|
|
@@ -23,7 +26,7 @@ flowchart LR | |
| Mutate --> Manifest["Generate requested manifests"] | ||
| ``` | ||
|
|
||
| # Directory | ||
| ## Contents | ||
|
|
||
| - `slicer/` - Main orchestrator for a Chisel run. Receives a slice selection, drives all other internal packages (setup, archive, cache, deb, fsutil, scripts, manifestutil) to completion, and writes the final filesystem and manifest. | ||
| - `setup/` - Parses chisel-releases YAML definitions into the `Release` model and resolves slice dependencies, path conflicts, and package contention. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,18 +1,28 @@ | ||
| # Preface | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same.. those files look like directory listings. |
||
|
|
||
| This document describes the scope of the `public/` directory, which contains Chisel's public API packages intended for consumption by external tools. | ||
| This document covers the public API packages in the `public/` directory. | ||
|
|
||
| Read the top-level `.kb/agents.md` file before continuing below. | ||
| IMPORTANT: Read the `.kb/agents.md` file. | ||
|
|
||
| # Overview | ||
|
|
||
| # Laws | ||
|
|
||
| ## §1 - License boundary | ||
|
|
||
| Packages in `public/` and their dependencies must remain Apache-2.0 licensed. | ||
|
|
||
|
|
||
| # Concepts | ||
|
|
||
| ## Scope | ||
|
|
||
| The `public/` directory houses the two packages that form Chisel's stable public contract. They provide a generic sorted JSON-lines database format and the Chisel manifest schema built on it, enabling third-party tools such as SBOM generators and vulnerability scanners to consume Chisel output without depending on internal packages. | ||
|
|
||
| # Important | ||
| ## License check | ||
|
|
||
| - When changing these packages or their dependencies, run `go run .github/scripts/external-packages-license-check.go` to verify the Apache-2.0 license boundary. | ||
| When changing these packages or their dependencies, run `go run .github/scripts/external-packages-license-check.go` to verify the license boundary. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems strange to ask the bot to be doing this every time it touches a file. This should probably be going to something like hacking.md next to other conventions that we follow, without asking for the action to be repeated on every change. If we want to have such a per-change action, it should be a single repository-wide static check, and then we don't need to tell what it is doing because if it fails it will say so. |
||
|
|
||
| # Directory | ||
| ## Contents | ||
|
|
||
| - `jsonwall/` - Implements a generic database format with one JSON object per line, with fields and entries sorted for efficient search and iteration. | ||
| - `manifest/` - Defines Chisel manifest schema 1.0 through the `Package`, `Slice`, `Path`, and `Content` record types, and uses `jsonwall` for serialization and deserialization. The carving pipeline applies Zstandard compression when generating manifest files. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,23 @@ | ||
| # Preface | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This one might stay, in the hope it grows to something useful. But let's rename it to snap-package.md, and drop the directory listing at the bottom. |
||
|
|
||
| This document describes the scope of the `snap/` directory within the Chisel repository, providing context for automated agents navigating and modifying the snapcraft configuration. | ||
| This document covers the snap packaging configuration in the `snap/` directory. | ||
|
|
||
| Read the top-level `.kb/agents.md` file before continuing below. | ||
| IMPORTANT: Read the `.kb/agents.md` file. | ||
|
|
||
| # Overview | ||
|
|
||
| # Laws | ||
|
|
||
| ## §1 - Strict confinement | ||
|
|
||
| The snap must use strict confinement. | ||
|
|
||
|
|
||
| # Concepts | ||
|
|
||
| ## Scope | ||
|
|
||
| The `snap/` directory contains the configuration necessary to build and package Chisel as a snap application. It defines the application metadata, execution confinement, and compilation steps for the `chisel` application using Go. | ||
|
|
||
| # Directory | ||
| ## Contents | ||
|
|
||
| - `snapcraft.yaml` - The primary manifest file defining the snap package. It configures strict confinement, utilizes the Go plugin, and outlines the build steps which depend on `cmd/mkversion.sh` for version injection. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My vote is to drop this entire file for now. It's not pulling its weight at the moment.