diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000000..96a30a6568 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,23 @@ +--- +name: Bug report +about: Report a reproducible problem with PALM. +--- + +## Problem + +What did you expect, and what happened instead? Identify the stage: analysis, generation, repair, or coverage. + +## Reproduction + +Include the command and a small target or function that reproduces the problem. + +## Environment + +- PALM commit: +- Operating system: +- `rustc --version`: +- `cargo llvm-cov --version` (for coverage problems): + +## Relevant output + +Include the error output. For model-related problems, optionally include the model identifier and a relevant response or request summary. Remove private information; do not attach API keys or `api.json`. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000..41186adc91 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,11 @@ +## Problem + + + +## Change + + + +## Validation + + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 44732dabe5..ffed0c462e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,13 +18,15 @@ Use the existing check relevant to your change: | Generation, repair, result handling, or test execution | `python3 scripts/check_minimal.py` | | Task scheduling, request budgets, or failure cleanup | `python3 scripts/check_tasks.py` | | Coverage annotations or preprocessing | `python3 scripts/check_coverage.py` | +| Coverage denominators or target-specific baseline maps | `cargo test -p utgen --locked coverage_baselines_match_test_build_modes -- --ignored` | +| Library/binary ownership, target selection, or dependency context | `python3 scripts/check_mixed.py` | | Analysis, focal context, or method identity | `cargo build --workspace --release --locked` then `python3 scripts/check_bytes.py` | The checks use temporary targets and local model fixtures. They do not need API keys or call an external model service; dependency installation may need network access. The full suite runs in CI for code changes. Documentation changes should preserve working links, command examples, and the distinction between offline validation and real-model results. ## Report a problem -Open a [GitHub issue](https://github.com/cppbear/PALM/issues) with: +Use the [bug report template](https://github.com/cppbear/PALM/issues/new/choose) with: - The PALM commit, operating system, `rustc --version`, and `cargo llvm-cov --version`. - The command you ran, expected behavior, actual behavior, and relevant error output. @@ -37,4 +39,6 @@ Run PALM on a working copy of a target crate: preprocessing changes its source a Keep the change focused and explain the problem, resulting behavior, and checks you ran. Add a regression case for a reproduced behavior change when the existing checks do not cover it. Real-model calls are not required for ordinary fixes; use local fixtures or existing saved candidates where practical. +The PR template has three short sections: problem, change, and validation. Use the checks relevant to the change; if none are needed, explain why. + Keep credentials, target build directories, generated answers, coverage reports, and local trial logs out of commits. Preserve the pinned toolchain and dependency lockfile unless changing them is part of the task. diff --git a/README.md b/README.md index 672513f51e..b664c5a505 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,59 @@ [![Build and test](https://github.com/cppbear/PALM/actions/workflows/ci.yml/badge.svg)](https://github.com/cppbear/PALM/actions/workflows/ci.yml) -PALM combines program analysis and large language models to generate Rust unit tests. It extracts path constraints and function context, generates candidates, repairs compilation errors, and reports test results and coverage with test code excluded. +PALM generates Rust tests by combining program analysis with large language models. It extracts path constraints and code context, generates candidates, repairs compilation errors, and reports execution results and coverage with test code excluded. -This is the maintained implementation of ***PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage***, published at ASE 2025. The repository includes changes made after the paper's experiments. +This is the maintained implementation of ***PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage***, published at ASE 2025. The current code includes changes made after the paper's experiments. -[Paper](https://doi.org/10.1109/ASE63991.2025.00223) · [Preprint](https://arxiv.org/abs/2506.09002) · [中文技术指南](docs/palm-rust-unit-test-generation.md) · [CLI reference](utgen/README.md) · [Citation](#citation) +[Quick start](#quick-start) · [Examples](examples/README.md) · [CLI reference](utgen/README.md) · [中文技术指南](docs/palm-rust-unit-test-generation.md) · [Paper](https://doi.org/10.1109/ASE63991.2025.00223) · [Preprint](https://arxiv.org/abs/2506.09002) + +## What PALM does + +- Extracts representative path constraints and the focal function's dependencies for model prompts. +- Generates unit tests or integration tests, with explicit function selection and request budgets. +- Uses compiler diagnostics to repair candidates that do not compile. +- Reports compilation, execution, and coverage results for each focal function, preserving library/binary ownership. + +## How it works + +```mermaid +flowchart TD + A["Rust package"] --> B["Path constraints and code context"] + B --> C["Model-generated tests"] + C --> D["Compilation check"] + D -->|Compiler errors| E["Model repair"] + E --> D + D -->|Compiles| F["Test execution and coverage"] +``` + +Repair addresses compilation errors. Runtime assertion failures and candidate timeouts remain failed test outcomes. + +## A small example + +The [minimal fixture](examples/minimal/README.md) includes this function, whose [support module](examples/minimal/src/support.rs) checks whether the input is positive: + +```rust +pub fn classify(value: i32) -> i32 { + if support::positive(value) { + 1 + } else { + 0 + } +} +``` + +The offline check supplies fixed local model responses with assertions for both paths. These excerpts omit the test wrappers and helper assertions: + +```rust +assert_eq!(classify(2), 1); +assert_eq!(classify(-1), 0); +``` + +The same check exercises compilation repair with an intentionally invalid candidate, then validates execution, coverage, and source restoration. It uses real Rust tools but does not measure model quality. [Run it below](#quick-start), then use the [validation guide](docs/minimal-pipeline.md#outputs-and-coverage-interpretation) to interpret its output. ## Prerequisites -Use a native Rust build environment with Git, [rustup](https://rustup.rs/), the stable Rust toolchain, and a C linker. The validation scripts also require Python 3.9 or later. The offline pipeline is checked on Linux CI and has been exercised on macOS with Apple Silicon. +Use a native build environment with Git, [rustup](https://rustup.rs/), stable Rust, a C linker, and Python 3.9 or later for the validation scripts. Tool and dependency downloads may require network access. 1. Install the pinned analysis toolchain and components: @@ -25,7 +69,7 @@ Use a native Rust build environment with Git, [rustup](https://rustup.rs/), the cargo +stable install cargo-llvm-cov --version 0.6.16 --locked ``` -Use the same nightly toolchain for the target crate. The supported end-to-end workflow operates on one standalone crate at a time; analysis handles `lib` and `bin` targets. Building, installation, and the offline checks do not require model credentials. +Use the same nightly for the target package. `rust-analyzer` provides editor support; compiler analysis uses components such as `rustc-dev`. Building, installation, and the offline checks need no model credentials. ## Quick start @@ -38,85 +82,90 @@ cargo build --workspace --locked python3 scripts/check_minimal.py ``` -The script uses fixed responses from a local HTTP server and real analysis, compilation, repair, and coverage tools. It works in temporary copies, prints their location, and finishes with `Minimal analysis/generation/repair/coverage checks passed.` The check verifies pipeline behavior; it does not measure model quality. See [minimal pipeline validation](docs/minimal-pipeline.md) for the checked outcomes and output files. +The script works in temporary copies, prints their location, and finishes with: + +```text +Minimal analysis/generation/repair/coverage checks passed. +``` -For model-based generation, install the tools below and follow the [two-function trial](examples/README.md#prepare-a-small-model-trial). That walkthrough uses a working copy, an explicit function list, and separate generation and repair request budgets. +See [minimal pipeline validation](docs/minimal-pipeline.md) for the checked scenarios. For a real model, install the tools below, [select a runtime configuration](utgen/README.md#prerequisites), and follow a [bounded example](examples/README.md). ## Installation -No model configuration is needed to build or install the tools. `utgen gen` and `utgen fix` load configuration at runtime; changing the API address, key, or model takes effect on the next command without rebuilding. - -From the repository root, run: +From the repository root: ```sh ./install.sh ``` -The script installs `brinfo`, `focxt/call_chain`, `focxt`, and `utgen`. It locates the repository from its own path, so it can also be invoked from another directory. To select tools, use `./install.sh brinfo utgen`; `./install.sh --help` lists the options. Cargo diagnostics remain visible and installation stops on the first failure. Ensure that the installation directory, typically `$HOME/.cargo/bin`, is in `PATH`. +This installs brinfo, call_chain, focxt, and utgen. Ensure Cargo's binary directory, typically `$HOME/.cargo/bin`, is in `PATH`. The installer also works from another directory; `./install.sh --help` lists tool selection options. See [build and installation checks](docs/build-validation.md) for verification and the component READMEs below for individual installation commands. -Alternatively, install each tool from the repository root: - -```sh -cargo install --path brinfo --locked -cargo install --path focxt/call_chain --locked -cargo install --path focxt --locked -cargo install --path utgen --locked -``` +Model configuration is loaded at runtime by `gen` and `fix`. Changing the API address, key, or model takes effect on the next command without rebuilding. ## Workflow -To measure existing tests before preprocessing, run `utgen coverage -p `. It runs tests once, excludes test code from coverage, and writes `coverage.xml` and `coverage.json` without model configuration. Keep this baseline copy separate from the generation copy. +Use a fresh working copy of the target package: preprocessing changes its source and test layout, and generation may add a test dependency. -On a fresh working copy of a standalone crate, run `utgen pre-process -p ` before `utgen analyze -p `. Analysis clears Cargo's check cache, runs brinfo and focxt, and verifies their outputs. It rejects existing `brinfo/` or `focxt/` directories so previous analysis results cannot be mixed into the run. +1. Run `utgen pre-process`, then `utgen analyze` on that copy. +2. Generate candidates with `utgen gen --requirement --context`, using a function list and request cap for an initial trial. +3. Run `utgen fix` with the same function list and mode, then inspect request reports and per-function statistics. -Then use `utgen gen` and `utgen fix` for generation and compilation repair. Pass `--requirement --context` to `utgen gen` to include path constraints and focal context. For integration tests, pass `--integration` to both `gen` and `fix`. Preprocessing and test checks modify the target tree; use a working copy of the target project. +Use `--integration` on both generation and repair commands for integration tests. The [examples](examples/README.md) provide complete commands for small unit, integration, and mixed-target trials. A completed command does not imply every candidate passed; [result fields](utgen/README.md#output) report the outcomes. -Start with the [minimal pipeline check](docs/minimal-pipeline.md), which uses fixed local model responses and verifies analysis, generation, repair, source restoration, and coverage. The [bytes analysis check](docs/bytes-analysis.md) validates analysis on the larger bundled target without a model service. For model-based generation and repair, follow the [small model trial](examples/README.md#prepare-a-small-model-trial) using an explicit function list and separate request limits, then inspect the request reports and test statistics. See [utgen usage](utgen/README.md) for command details and supported scope. +To measure existing tests, run `utgen coverage -p ` before preprocessing, on a separate copy. It needs no model configuration and retains `coverage.xml` and `coverage.json`. -Generation or repair completing does not mean every candidate passes. Inspect both request reports and test statistics. Repair addresses compilation errors; runtime failures and five-second candidate timeouts remain failed test outcomes. +## Supported scope + +| Area | Current scope | +| --- | --- | +| Target package | One standalone Cargo package with ordinary lib/bin targets, including custom names and entry files. The complete workflow does not support multi-package workspaces. | +| Test modes | Unit tests use the recorded lib/bin target. Function-level integration tests require a library and compile-check external access. | +| Shared definitions | A definition compiled into both lib and bin is represented by lib. This does not test every binary variant. | +| Rust features | Analysis follows the active build configuration. There is no general macro expansion or automatic feature-matrix exploration. | +| Toolchain | `nightly-2025-03-19` and cargo-llvm-cov `0.6.16`. | +| Platforms | Full offline suite on Linux CI; exercised locally on macOS Apple Silicon. Windows has not been validated. | + +See the [CLI reference](utgen/README.md) for source-processing boundaries, cache compatibility, timeouts, and coverage interpretation. ## Project Structure | Directory | Purpose | | --- | --- | -| [brinfo](brinfo/README.md) | Extract function condition chains from HIR and MIR. | -| [focxt](focxt/README.md) | Construct code context for each focal function. | -| [focxt/call_chain](focxt/call_chain/README.md) | Extract calls and type dependencies through `cargo call-chain`. | -| [utgen](utgen/README.md) | Generate tests, check compilation, repair tests, and collect coverage. | -| [build-utils](build-utils/README.md) | Configure the compiler library search path during builds. | -| [examples](examples/README.md) | Minimal and mixed-target fixtures, the bundled bytes target, and scoped trial instructions. | +| [brinfo](brinfo/README.md) | Extract condition chains from HIR and MIR. | +| [focxt](focxt/README.md) | Construct context for each focal function. | +| [focxt/call_chain](focxt/call_chain/README.md) | Extract calls and type dependencies. | +| [utgen](utgen/README.md) | Generate, compile-check, repair, and evaluate tests. | +| [build-utils](build-utils/README.md) | Configure compiler library search paths during builds. | +| [examples](examples/README.md) | Minimal and mixed-target fixtures, plus the bundled bytes target. | +| [docs](docs/README.md) | Technical, usage, validation, and research documentation. | | `docker/` | Container build and run scripts. | -| [docs](docs/README.md) | Technical guide, validation guides, and ASE 2025 materials. | ## Docker -Run `docker/docker-build` to prepare an image with the pinned analysis toolchain, Python, and cargo-llvm-cov. Both Docker scripts locate the repository from their own paths. Mirrors can be configured in `docker/Dockerfile` if needed. +Run `docker/docker-build`, then `docker/docker-run` to open a shell with the repository mounted at `/home/palm/palm`. Follow [Installation](#installation) inside the container. The image includes the pinned compiler and coverage tools; files written in the mounted repository remain on the host after the container exits. -Run `docker/docker-run` to open a shell in the repository mounted at `/home/palm/palm`, then follow [Installation](#installation). The container is removed when the shell exits; files written in the mounted repository remain on the host. Use working copies of target crates as described above. - -The [Docker CI workflow](.github/workflows/docker.yml) checks image construction and tool installation when Docker or build-environment files change. Its manual `full_pipeline` option also runs the minimal offline pipeline inside the container; see [CI scope](docs/build-validation.md#ci-and-further-checks). +Both scripts locate the repository from their own paths. See [Docker validation and CI scope](docs/build-validation.md#ci-and-further-checks) for the checked environment and optional container pipeline check. ## Development checks -From the repository root, using the pinned toolchain: - ```sh cargo build --workspace --locked cargo test --workspace --locked ``` -Default tests use local model-response fixtures and do not contact a model service. The real-service test is opt-in; see [utgen testing](utgen/README.md#testing). Dependency downloads may still require network access. See [build and installation checks](docs/build-validation.md) for commands and CI scope. - -The [GitHub Actions workflow](.github/workflows/ci.yml) runs on pull requests and pushes to main, canceling superseded runs for the same PR or branch. Changes limited to READMEs and the selected documentation paths under `docs/` receive a lightweight patch whitespace check. Other changes and manual workflow runs execute the full offline suite on Linux. +Default checks use local model responses. See [CONTRIBUTING.md](CONTRIBUTING.md) for the regression check relevant to a change, and [build validation](docs/build-validation.md) for CI scope. The real-model request test is [explicitly opt-in](utgen/README.md#testing). ## Contributing -Bug reports and focused pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, relevant checks, and the information to include in a report. +Bug reports and focused pull requests are welcome. Include a reproducer and relevant validation as described in [CONTRIBUTING.md](CONTRIBUTING.md). ## Documentation -- [Technical guide in Chinese](docs/palm-rust-unit-test-generation.md): architecture, data flow, and usage. -- [ASE 2025 materials](docs/ase2025/README.md): the original poster and presentation. +- [Examples](examples/README.md): bounded unit, integration, and mixed-target trials. +- [Technical guide in Chinese](docs/palm-rust-unit-test-generation.md): architecture, data flow, and implementation. +- [CLI reference](utgen/README.md): configuration, commands, and outputs. +- [Research materials](docs/ase2025/README.md): conference poster and presentation, with revision notes. +- [Documentation index](docs/README.md): all guides and validation entry points. ## Citation diff --git a/docs/README.md b/docs/README.md index 7e6b489b6b..3904fb0f8f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,9 +11,9 @@ PALM's implementation, technical guide, and conference materials are available i | [Technical guide in Chinese](palm-rust-unit-test-generation.md) | Condition chains, context construction, generation, repair, and usage. | | [Minimal pipeline](minimal-pipeline.md) | Deterministic program-analysis, generation, repair, and coverage validation. | | [Bytes analysis](bytes-analysis.md) | Analysis artifacts, method identities, constructor context, and timing output. | -| [bytes example](../examples/README.md) | Commands for analyzing a working copy of the bundled target. | +| [Examples](../examples/README.md) | Minimal and mixed-target fixtures, bytes analysis, and bounded unit/integration model trials. | | [utgen reference](../utgen/README.md) | Configuration, CLI behavior, and output files. | -| [ASE 2025 materials](ase2025/README.md) | Original poster and presentation PDFs. | +| [ASE 2025 materials](ase2025/README.md) | Conference poster and presentation PDFs, with revision and statistical notes. | | [Build and installation checks](build-validation.md) | Build, tests, CLI entry points, installation, and CI scope. | The conference PDFs describe the work presented in 2025. The technical guide and command reference describe the implementation maintained here; the current code includes changes made after that presentation. diff --git a/docs/ase2025/README.md b/docs/ase2025/README.md index bed0fe3247..5cfa8f2891 100644 --- a/docs/ase2025/README.md +++ b/docs/ase2025/README.md @@ -1,8 +1,12 @@ # ASE 2025 Materials -These are the original presentation materials for **PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage**. +Conference presentation materials for **PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage**. - [Poster](research-2647-poster.pdf): one A0 page. - [Presentation](research-2647-pre.pdf): 33 pages. +This repository copy updates the poster's materials QR code to this page and adds acceptance-rate denominator notes to the poster and slides 28 and 32. The experimental figures remain unchanged; the materials describe the paper's experiments, not a new evaluation of the current implementation. + +The reported acceptance rate is **80 / (80 + 5) = 94.1%**, among submissions with a decision at the time of the paper. Of 91 submitted tests, 80 were accepted, 5 were rejected, and 6 were pending; the pending submissions are excluded from that rate. See the [paper](https://arxiv.org/abs/2506.09002) for the study and its scope. + For installation and commands for the current code, see the [project README](../../README.md) and [technical guide](../palm-rust-unit-test-generation.md). diff --git a/docs/ase2025/research-2647-poster.pdf b/docs/ase2025/research-2647-poster.pdf index 2247d7283d..f89495f2e7 100644 Binary files a/docs/ase2025/research-2647-poster.pdf and b/docs/ase2025/research-2647-poster.pdf differ diff --git a/docs/ase2025/research-2647-pre.pdf b/docs/ase2025/research-2647-pre.pdf index bdb44e0823..588892a853 100644 Binary files a/docs/ase2025/research-2647-pre.pdf and b/docs/ase2025/research-2647-pre.pdf differ diff --git a/docs/assets/palm-tech-route.png b/docs/assets/palm-tech-route.png deleted file mode 100644 index a5da83fc71..0000000000 Binary files a/docs/assets/palm-tech-route.png and /dev/null differ diff --git a/docs/bytes-analysis.md b/docs/bytes-analysis.md index da78d06a2d..7b710e7f26 100644 --- a/docs/bytes-analysis.md +++ b/docs/bytes-analysis.md @@ -23,7 +23,7 @@ The script prints its temporary working directory and finishes with `All bytes a | Source identity | Exported function code matches its source location, accounting for brinfo's indentation normalization. | | Method identity | Distinct generic trait implementations, reference receivers, default methods, and recoverable macro-generated methods retain the correct bodies and impl headers. Inactive `cfg` alternatives do not replace active methods. | | Constructor context | Constructors returning `Self` or the concrete impl type retain their bodies, including the checked generic and macro-generated cases. Unrelated methods remain subject to context trimming. | -| Function lists | The checked-in two- and eight-function selections match the analysis index and their expected representative condition chains. | +| Function lists | The unit-mode `bytes-smoke-2.txt` and `bytes-smoke-8.txt` selections match the analysis index and their expected representative condition chains. This script does not check the separate integration or mixed-target trial lists. | | Source preservation | Analysis leaves the prepared sources and manifest unchanged. | ## Logs and Timings diff --git a/docs/minimal-pipeline.md b/docs/minimal-pipeline.md index 0422d6181e..48c646f863 100644 --- a/docs/minimal-pipeline.md +++ b/docs/minimal-pipeline.md @@ -13,6 +13,8 @@ python3 scripts/check_minimal.py The script uses the [minimal fixture](../examples/minimal/README.md) in temporary copies. It prints the working directory and finishes with `Minimal analysis/generation/repair/coverage checks passed.` The checked-in fixture remains unchanged. +The [homepage example](../README.md#a-small-example) shows excerpts from its fixed responses for `classify`. Those assertions are illustrative excerpts, not a complete generated file or a model-quality result. The full script also exercises deliberate compilation failures and runtime failures; the final success message means those outcomes and cleanup matched expectations. + Run the related checks when changing coverage/preprocessing or task scheduling: ```sh diff --git a/docs/palm-rust-unit-test-generation.md b/docs/palm-rust-unit-test-generation.md index 39752d7f63..9702c56caf 100644 --- a/docs/palm-rust-unit-test-generation.md +++ b/docs/palm-rust-unit-test-generation.md @@ -17,17 +17,15 @@ PALM 的思路是把程序分析纳入提示词构建过程：先用 Rust 编译 ## 2. 总体技术路线 -![PALM 技术路线图](assets/palm-tech-route.png) - 整体流程可以概括为四步： 1. 条件链提取：`brinfo` 通过 rustc API 同时获取 HIR 和 MIR，构建 CFG，遍历执行路径，恢复源码级条件，并输出每个函数的条件链。 2. 上下文构建：`focxt` 和 `focxt/call_chain` 分析函数调用、使用到的类型、模块树和可见性，生成面向提示词的上下文代码片段。 3. 提示词生成：`utgen` 读取条件链、函数源码、上下文和提示词模板，构造完整测试生成任务。 -4. 测试生成与修复：`utgen` 调用 LLM 生成测试，临时插入项目运行 `cargo build --tests` 和 `cargo llvm-cov`，再用编译错误驱动 LLM 修复失败测试。 +4. 测试生成与修复：`utgen` 调用 LLM 生成候选，按所属 Cargo 目标检查编译；单元模式临时插入源码，集成模式写入临时测试目标。编译错误用于驱动 LLM 修复，测试执行和覆盖率采集在后续阶段完成。 ```mermaid -flowchart LR +flowchart TD A["目标 Rust crate"] --> B["brinfo: HIR/MIR + CFG"] B --> C["brinfo/brdata/*.json 条件链"] A --> D["focxt/call_chain: 调用和类型依赖"] @@ -35,9 +33,9 @@ flowchart LR C --> F["utgen: Prompt 构建"] E --> F F --> G["LLM 生成测试"] - G --> H["cargo build --tests 编译筛选"] - H --> I["cargo llvm-cov 覆盖率统计"] - H --> J["LLM 修复编译错误"] + G --> H["按目标检查编译，不执行测试"] + H -->|编译通过| I["测试执行与覆盖率统计"] + H -->|编译错误| J["LLM 修复编译错误"] J --> H ``` @@ -109,9 +107,9 @@ flowchart LR - `brinfo/src/bin/cargo-brinfo.rs` - 实现 `cargo brinfo` 子命令。 - 读取当前 crate 的 Cargo metadata。 - - 对 lib/bin target 执行 `cargo check`。 + - 对选定的 lib/bin target 执行 `cargo check`；混合目标由 `utgen analyze` 逐个调度。 - 设置 `RUSTC_WRAPPER` 指向自身，使编译过程回到 `cargo-brinfo`。 - - 通过 `BRINFO_TOP_CRATE_NAME` 判断当前 rustc 调用是否属于目标 crate。 + - 同时核对 crate 名、目标类型和入口源码路径，判断当前 rustc 调用是否属于选定目标，避免把同名库依赖当作 bin 分析。 - 对目标 crate 调用 `brinfo`，对依赖 crate 设置 `BRINFO_BE_RUSTC=1` 让它表现为普通 rustc。 - `brinfo/src/bin/brinfo.rs` @@ -163,7 +161,7 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` ### 5.5 `brinfo` 输出产物 -在目标 crate 目录下生成： +单目标分析在目标 crate 目录下生成以下产物： | 输出 | 说明 | | --- | --- | @@ -175,6 +173,8 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` | `brinfo/tmp//cond_map.json` | HIR 条件映射结果。 | | `brinfo/tmp//cfg.dot` | CFG dot 图。 | +混合目标分析的原始产物位于 `brinfo/targets///brinfo/`。根目录的 `brinfo/name_map.json` 和 `brinfo/brdata/` 是筛选归属后的汇总结果；调试文件不会复制到根目录。汇总标识与原始标识不同，查看某个目标的调试文件时，应使用该目标原始 `name_map.json` 中的标识。 + ## 6. `focxt/call_chain` 模块说明 `focxt/call_chain` 负责从编译器层面提取函数调用和类型依赖，安装后提供 `cargo call-chain`。 @@ -191,7 +191,7 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` - 遍历 MIR basic blocks 中的 `TerminatorKind::Call`，提取调用函数。 - 遍历 local declarations 和调用参数，递归收集 ADT、数组、切片、裸指针、tuple 等类型的子类型。 -输出在目标 crate 的 `focxt/` 目录下： +单目标分析输出在目标 crate 的 `focxt/` 目录下： | 输出 | 说明 | | --- | --- | @@ -199,6 +199,8 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` | `focxt/basic_blocks/.txt` | basic blocks 和 locals 调试信息。 | | `focxt/impl_informations.json` | 函数所属模块、函数名、结构体名、trait 名、完整名和编码名。 | +混合目标的这些原始文件保存在 `brinfo/targets///focxt/`。根目录 `focxt/impl_informations.json` 是汇总索引，保留最终被测函数的目标归属；原始调用依赖和调试文件仍按目标保存。 + ## 7. `focxt` 模块说明 `focxt` 将调用/类型依赖转换为可放入提示词的上下文代码。 @@ -207,19 +209,21 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` `focxt/src/main.rs` 的流程是： -1. 解析 `--crate `。 -2. 执行 `run_call_chain`，先 `cargo clean`，再在目标 crate 下运行 `cargo call-chain`。 -3. 读取 `focxt/impl_informations.json`。 -4. 创建 `CrateContext` 并解析 crate。 +1. 解析 `--crate `，通过 Cargo metadata 获取目标名称和入口；混合目标由 `utgen analyze` 传入当前选择。 +2. 执行 `run_call_chain`，先 `cargo clean`，再为选定目标运行 `cargo call-chain`。 +3. 从当前目标的输出目录读取 `focxt/impl_informations.json`。 +4. 用目标 crate 名、入口文件和输出目录创建 `CrateContext`，解析该入口对应的模块树。 5. 建立模块树和完整名称。 6. 生成扩展后的调用/类型依赖。 7. 为每个函数输出上下文 `.rs` 文件。 +分析 bin 时，本地定义与包内库的定义分开处理；引用到的库函数和类型放入标明来源的依赖段。最终被测函数列表采用 lib 优先规则，不会因此丢弃 bin 构建上下文所需的原始定义。 + ### 7.2 模块树与 AST 上下文 - `focxt/src/collect_context/crate_context.rs` - - 读取 `Cargo.toml` 获取 crate 名。 - - 识别 `src/main.rs` 和 `src/lib.rs` 入口。 + - 接收上层从 Cargo metadata 取得的目标 crate 名、入口和输出目录。 + - 每次解析一个选定入口，支持 Cargo 声明的自定义名称与入口路径。 - 组织多个 `ModContext`。 - 输出模块树、函数名清单、调试上下文。 @@ -239,7 +243,7 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` ### 7.3 `focxt` 输出产物 -在目标 crate 目录下生成： +单目标分析在目标 crate 目录下生成： | 输出 | 说明 | | --- | --- | @@ -250,6 +254,8 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` | `focxt/mod_trees/mod_tree*.txt` | 模块树列表。 | | `focxt/functions/function*.txt` | 完整函数名列表。 | +混合目标的原始上下文、依赖和调试文件位于 `brinfo/targets///focxt/`。根目录 `focxt/` 仅保留最终被测函数的 `.rs` 上下文与汇总 `impl_informations.json`。原始文件名使用各目标索引中的编码，不能直接套用汇总编码。 + ## 8. `utgen` 模块说明 `utgen` 是生成和执行测试的主模块。 @@ -261,6 +267,7 @@ Rust 的一个核心挑战是 MIR 的去糖。比如高级语法 `match color` ```text utgen pre-process --project-dir [--work-dir ...] utgen analyze --project-dir +utgen coverage --project-dir utgen gen --project-dir [--work-dir ...] [--tasks ] [--integration] [--requirement] [--context] [--oracle] utgen fix --project-dir [--work-dir ...] [--tasks ] [--integration] ``` @@ -274,9 +281,10 @@ utgen fix --project-dir [--work-dir ...] [--tasks ] [ | `-p, --project-dir` | 项目根目录。路径会相对于当前目录解析并 canonicalize。 | | `-w, --work-dir` | 工作目录，可重复传入或用逗号分隔，相对路径基于当前命令执行目录，默认等于 project dir。 | | `-t, --tasks` | 默认 `4`，必须大于 `0`。生成和修复均限制同时活跃的被测函数任务数；同一函数内的条件链、生成阶段或修复轮次依次执行。 | -| `--functions-file` | 可选 UTF-8 函数清单，每行一个 `brinfo/name_map.json` 中的完整名称。生成、修复及其统计均按清单筛选；空行忽略、重复项去重，不支持通配符。 | +| `--request-timeout` | `gen` 和 `fix` 的单次模型请求期限，默认 `180` 秒，必须大于 `0`；不限制 Cargo 子进程。 | +| `--functions-file` | 可选 UTF-8 函数清单，每行一个 `brinfo/name_map.json` 中的索引键，也可使用能唯一定位的 Rust 路径。生成、修复及其统计均按清单筛选；空行忽略、重复项去重，不支持通配符。 | | `--max-requests` | 可选的整次命令请求尝试次数上限，必须大于 `0`；默认不设总上限。所有并发任务和重试共享该额度。 | -| `-i, --integration` | 生成集成测试，放到 `tests/` 目录，按分析结果的 `visible` 标志筛选函数并进行编译检查。 | +| `-i, --integration` | `gen` 和 `fix` 均支持集成模式，使用 `tests/` 中的测试目标；生成时选择满足分析可见性要求的库函数，实际可访问性仍由编译确认。 | | `-r, --requirement` | 在提示词中加入条件链约束。 | | `-c, --context` | 在提示词中加入 `focxt/.rs` 上下文。 | | `-o, --oracle` | 开启三段式生成：输入范围、测试前缀、oracle 分别生成。该模式是可选实验模式，不是按路径任务分解本身。 | @@ -357,8 +365,8 @@ export PALM_CONFIG="$(pwd)/utgen/res/api.json" - `generation_tests` 根据 `--oracle` 选择三段式生成或完整生成。 - `gen_full_tests` 使用 `test_prompt.json` 一次性生成测试。 - `gen_tests_cot` 使用 `input -> prefix -> oracle` 三步流程。 - - `check_unit` 会临时把测试插入被测源码文件，运行 `cargo build --tests`，记录每个候选是否可编译，然后恢复文件。 - - `check_integration` 会写入临时 `tests/palm_candidate.rs`，只编译该集成测试目标。 + - `check_unit` 会临时把测试插入被测源码文件，运行 `cargo test --lib --no-run` 或 `cargo test --bin --no-run`，记录每个候选是否可编译，然后恢复文件。 + - `check_integration` 会写入临时 `tests/palm_candidate.rs`，以 `cargo build --test palm_candidate` 检查编译，不执行测试。Cargo 仍可能构建包内库和 bin。 - `utgen/src/gene/cot/*.rs` - `input_infer.rs`：生成输入范围。 @@ -389,26 +397,28 @@ export PALM_CONFIG="$(pwd)/utgen/res/api.json" 核心文件： - `utgen/src/run/run.rs` - - `run_test` 在覆盖率模式下执行： + - `run_test` 按保存的目标选择单元测试 harness。以 lib 为例，覆盖率模式执行： ```sh - cargo llvm-cov --tests --ignore-run-fail --branch --cobertura --output-path coverage.xml + cargo llvm-cov --lib --ignore-run-fail --branch --cobertura --output-path coverage.xml cargo llvm-cov report --json --output-path coverage.json ``` - - 错误模式下执行： + - 错误模式只收集编译诊断，不执行测试。以 lib 为例： ```sh - cargo test --tests --message-format json + cargo test --lib --no-run --message-format json ``` - `gen_test_rate` 读取生成结果，运行覆盖率，计算每个被测函数的测试数、编译通过数、运行通过数、行覆盖和分支覆盖。 +bin 的单元模式将 `--lib` 替换为 `--bin `；集成模式使用对应的 `--test ` 目标。独立 `utgen coverage` 和原有测试对比仍使用包级 `--tests`，与选定候选的统计范围不同。 + - `utgen/src/run/coverage.rs` - 覆盖率入口复用预处理的测试识别规则，在测试模块、独立测试函数及 `cfg(test)` 辅助函数/方法前临时添加 `#[coverage(off)]`。整个测试模块（含生成的 `llmtests`）排除后，其中的辅助代码默认不计入覆盖率分子和分母；测试专用 `impl` 按方法处理，带文件级 `#![cfg(test)]` 的模块整文件排除。被调用的生产函数仍参与统计。 - 已有 coverage 属性和 feature 开关予以复用；条件属性只在原条件不成立时补充，条件由编译器判断，显式 `coverage(on)` 保留。在 crate 根文件同一行临时开启 `coverage_attribute`，保持原行号，不覆盖编译 flags。成功或编译/导出报错返回前恢复源文件原始内容；强制终止后的恢复留待后续处理。 - 测试只执行一次，先输出 XML，再由 `report --json` 导出同一份运行数据。第一条命令保留默认清理，避免候选之间累积旧覆盖率。 - - `utgen coverage -p <原始 crate 工作副本>` 可在预处理前测量原有测试，无需模型配置和分析产物；结果保留在该副本的 `coverage.xml`、`coverage.json`。断言失败仍导出覆盖率，编译或报告错误则返回失败。当前支持源码位于 `src/` 的独立 crate，非入口文件若是合法表达式片段（如 `include!` 引入的表达式），预处理与覆盖率均原样保留，不自动改写片段内部的测试代码；入口文件与其他解析错误仍严格检查。不展开宏，也不自动推断未标记的公共辅助函数是否仅用于测试。 + - `utgen coverage -p <原始 crate 工作副本>` 可在预处理前测量原有测试，无需模型配置和分析产物；结果保留在该副本的 `coverage.xml`、`coverage.json`。断言失败仍导出覆盖率，编译或报告错误则返回失败。源码处理范围为独立 package 的 `src/` 及 Cargo 声明的 lib/bin 入口文件，不会遍历任意位于 `src/` 外的模块文件。非入口文件若是合法表达式片段（如 `include!` 引入的表达式），预处理与覆盖率均原样保留，不自动改写片段内部的测试代码；入口文件与其他解析错误仍严格检查。不展开宏，也不自动推断未标记的公共辅助函数是否仅用于测试。 - `utgen/src/run/coverage_json.rs` - 解析 `coverage.json` 中的 branch 信息。 @@ -436,7 +446,8 @@ export PALM_CONFIG="$(pwd)/utgen/res/api.json" - `utgen/src/run/llm_fix.rs` - 读取 `utgen/generation/pre_fix`，复制或合并到 `utgen/generation/llm_fix`。 - - 对不可编译且未修复的测试，临时插入源码并运行 `cargo test --tests --message-format json`。 + - 单元模式把候选临时插入源码，按保存的目标运行编译诊断：lib 使用 `cargo test --lib --no-run --message-format json`，bin 将 `--lib` 替换为 `--bin `。修复读取编译错误而非普通警告。 + - 集成模式将候选导入、辅助代码和测试体写入临时 `tests/palm_candidate.rs`，以 `cargo test --test palm_candidate --no-run --message-format json` 收集诊断，不插入生产源码。Cargo 仍可能编译依赖库及包内 bin。 - 解析编译错误，抽取错误相关代码片段。 - 构造 `rustassistant_prompt.json` 中定义的 ChangeLog 格式要求。 - 调用 LLM 修复，应用 ChangeLog，重新编译。 @@ -481,8 +492,8 @@ utgen/fixed_result/.json 安装 nightly 工具链和组件： ```sh -rustup install nightly-2025-03-19 -rustup component add --toolchain nightly-2025-03-19 rust-src rustc-dev llvm-tools-preview +rustup toolchain install nightly-2025-03-19 --profile minimal \ + --component rust-src --component rustc-dev --component llvm-tools-preview --component rust-analyzer ``` 安装覆盖率工具： @@ -496,10 +507,10 @@ cargo +stable install cargo-llvm-cov --version 0.6.16 --locked ```toml [toolchain] channel = "nightly-2025-03-19" -components = ["rust-src", "rustc-dev", "llvm-tools-preview"] +components = ["rust-src", "rustc-dev", "llvm-tools-preview", "rust-analyzer"] ``` -本仓库根目录已经有 `rust-toolchain.toml`，示例项目 `examples/bytes/` 也包含对应配置。 +将上述内容保存为 `rust-toolchain.toml`。本仓库与示例均固定同一 nightly；`rust-analyzer` 用于编辑器支持，程序分析依赖 `rustc-dev` 等编译器组件。运行验证脚本还需要 Python 3.9 或更高版本，以及 Git、rustup、stable Rust 和本机 C 链接器；完整准备步骤见[项目 README](../README.md#prerequisites)。 构建和安装无需密钥。运行 `gen`、`fix` 前准备 `utgen/res/api.json`，并按 8.2 节设置 `PALM_CONFIG` 或传入 `--config`： @@ -541,6 +552,7 @@ cargo install --path utgen --locked ```sh cargo brinfo --help +cargo call-chain --help focxt --help utgen --help ``` @@ -560,7 +572,7 @@ utgen pre-process -p -w -w 预处理会： - 将所选 crate 根目录的 `tests` 重命名为 `tests.bak`；已有备份不会被覆盖。 -- 将 `src` 中测试专属模块，以及带 `#[test]` 或路径末段为 `test` 的属性的函数替换为空白，保持换行和字节位置。 +- 访问所选 package 的 `src/` 和 Cargo 声明的 lib/bin 入口文件，将可识别的测试专属模块、函数、impl 与方法替换为空白，保持换行和字节位置。 模块识别解析 `cfg` 表达式，仅移除能确定在关闭 `test` 时不可用的模块；保留 `cfg(not(test))`，对未知 feature/target 条件保守处理。预处理修改目标项目源码树，没有完整的反向预处理命令，应在临时副本中运行。先预处理，再对准备好的源码分析；空白替换保持字节位置，UTF-8 字符列号仍应以处理后的源码为准。`-w` 的相对路径基于当前命令执行目录。 @@ -572,7 +584,7 @@ utgen pre-process -p -w -w utgen analyze -p ``` -也可以手动进入准备好的目标 crate 根目录提取条件链。已有 `cargo check` 缓存可能使编译器包装器不执行，因此需要先清理： +只有一个普通 lib/bin 目标时，也可以手动进入准备好的目标 crate 根目录提取条件链。已有 `cargo check` 缓存可能使编译器包装器不执行，因此需要先清理： ```sh cargo clean @@ -585,7 +597,7 @@ cargo brinfo focxt -c ``` -`focxt` 内部会运行 `cargo call-chain`，因此需要先安装 `focxt/call_chain`。 +`focxt` 内部会运行 `cargo call-chain`，因此需要先安装 `focxt/call_chain`。混合目标必须使用 `utgen analyze`；未指定目标的底层命令会拒绝多个目标，不能用上面的手动流程替代。 ### 10.5 生成测试 @@ -624,7 +636,7 @@ utgen fix -p 修复阶段读取 `utgen/generation/pre_fix`，针对不可编译测试调用 LLM 生成 ChangeLog 并重新验证。修复结果写入 `utgen/generation/llm_fix`，覆盖率统计写入 `utgen/fixed_result`。 -`fix` 默认将测试插入源码，以单元测试方式修复和统计。集成测试需要在生成和修复时都传入 `--integration`：修复阶段把单个候选写入临时集成测试目标，只编译该目标；可以修改导入、模块级辅助代码和测试体，并单独保存各候选的导入与辅助代码。整体执行和逐候选覆盖率统计都保持集成测试模式，不再把候选插入源码。覆盖率阶段仍会临时添加排除测试代码的属性并恢复源文件。 +`fix` 默认将测试插入源码，以单元测试方式修复和统计。集成测试需要在生成和修复时都传入 `--integration`：修复阶段把单个候选写入临时集成测试目标，以该目标收集编译诊断，不执行测试；可以修改导入、模块级辅助代码和测试体，并单独保存各候选的导入与辅助代码。整体执行和逐候选覆盖率统计都保持集成测试模式，不再把候选插入源码。Cargo 可能同时构建依赖库和包内 bin；覆盖率阶段仍会临时添加排除测试代码的属性并恢复源文件。 候选缓存记录生成模式，模式不匹配时会报错；没有模式字段的旧缓存按单元测试处理，旧集成测试缓存应在新的工作副本中重新生成。两条命令还应传入相同的函数清单。运行时断言失败不属于编译修复流程的目标。 @@ -642,14 +654,18 @@ utgen pre-process -p "$palm_example_dir" utgen analyze -p "$palm_example_dir" ``` -按单元测试模式生成和修复： +先使用已有的两函数清单进行单元测试试跑，分别限制生成和修复的请求尝试次数： ```sh -utgen gen -p "$palm_example_dir" --requirement --context -utgen fix -p "$palm_example_dir" +utgen gen -p "$palm_example_dir" --requirement --context \ + --functions-file "$palm_repo/examples/bytes-smoke-2.txt" --max-requests 8 +utgen fix -p "$palm_example_dir" \ + --functions-file "$palm_repo/examples/bytes-smoke-2.txt" --max-requests 8 ``` -切换到 integration、其他提示词或其他模型时，使用新的目标副本并重新分析，避免复用旧的生成缓存。更详细的示例见 [examples/README.md](../examples/README.md)。[最小流程验证](minimal-pipeline.md)使用固定本地响应检查生成、修复、执行和覆盖率；[Bytes 分析检查](bytes-analysis.md)核对默认配置下导出的 663 个函数及其上下文。真实模型实验按示例中的函数清单和请求额度单独开展，并检查请求报告及测试结果。 +每次命令结束后先检查状态和请求报告；重试命令会开始新的请求额度。清单限制候选生成及其统计，若工作副本中保留 `tests.bak/`，原有集成测试对比仍按包执行。只验证候选时，可在预处理后把该备份移到工作副本之外保存。 + +切换到 integration、其他提示词或其他模型时，使用新的目标副本并重新分析，避免复用旧的生成缓存。更详细的示例见 [examples/README.md](../examples/README.md)，其中也提供限额的集成测试和混合目标试跑。[最小流程验证](minimal-pipeline.md)使用固定本地响应检查生成、修复、执行和覆盖率；[Bytes 分析检查](bytes-analysis.md)核对默认配置下导出的 663 个函数及其上下文。 完成后重点查看临时目标目录中的以下路径： @@ -675,14 +691,14 @@ utgen --config utgen/res/api.json gen -p --requirement --con ### 11.2 `cargo brinfo` 在 workspace 根目录失败 -`cargo-brinfo` 需要定位当前 package。对于 workspace，建议进入具体 crate 目录运行，或者明确使用目标 crate 的 manifest。 +`cargo-brinfo` 需要定位当前 package。底层单目标分析可进入具体 package 目录，或明确传入其 manifest。该操作不表示 PALM 已支持 workspace 的完整生成流程；`utgen analyze` 当前要求一个独立 Cargo package，混合 lib/bin 目标也应由它统一调度。 ### 11.3 `brinfo` 编译失败或找不到 rustc 私有 crate 确认使用 `nightly-2025-03-19`，并安装： ```sh -rust-src rustc-dev llvm-tools-preview +rustup component add --toolchain nightly-2025-03-19 rust-src rustc-dev llvm-tools-preview ``` 这些 crate 使用了 `#![feature(rustc_private)]`，必须依赖 nightly 和 rustc-dev。 diff --git a/examples/minimal/README.md b/examples/minimal/README.md index 0850d9f5a0..6d14282ac0 100644 --- a/examples/minimal/README.md +++ b/examples/minimal/README.md @@ -3,3 +3,7 @@ This standalone library exercises five focal functions: a branch, a struct method, an inline module, and a two-level cross-module call chain. It also includes existing unit/integration tests and a test sharing a line with production code, and test-only helpers. A helper can record a test execution for the coverage regression, which checks that XML and JSON do not cause duplicate runs. Its test dependencies are locked for the pinned nightly. Run the [minimal pipeline check](../../docs/minimal-pipeline.md) from the PALM repository root. The script copies this directory before preprocessing or analysis; generated data is not written into this fixture. + +For a small demonstration, start with `classify` in [src/lib.rs](src/lib.rs). It calls the [support module](src/support.rs), which checks whether the value is positive. Fixed local model responses produce assertions such as `classify(2) == 1` and `classify(-1) == 0`. The check also deliberately supplies an invalid argument to `nested::double`, then repairs it using real compiler diagnostics. + +The fixture demonstrates pipeline behavior, not model quality. After running it, use the [output guide](../../docs/minimal-pipeline.md#outputs-and-coverage-interpretation) to distinguish request counts, candidate outcomes, and coverage. Its success message means the expected checks passed, including cases that intentionally fail compilation or execution. diff --git a/focxt/README.md b/focxt/README.md index e1f043d122..e9385b6852 100644 --- a/focxt/README.md +++ b/focxt/README.md @@ -30,7 +30,7 @@ Mixed packages use `utgen analyze`, which selects each Cargo entry and preserves ## Output -Paths are relative to the target crate: +For single-target analysis, paths are relative to the target crate: | Path | Contents | | --- | --- | @@ -41,6 +41,8 @@ Paths are relative to the target crate: Additional text files under `focxt/` describe parsed declarations and module trees for debugging. Generation also requires [brinfo](../brinfo/README.md) output from the same prepared source. +For mixed packages, `utgen analyze` writes the selected focal contexts and combined `impl_informations.json` at the paths above. Raw contexts, calls/types, and debug files stay under `brinfo/targets///focxt/`; they are not copied into the top-level `focxt/`. Use each target's raw index to locate its artifacts: raw encoded identifiers differ from the combined identifiers. + ## Scope and checks Function contexts use call-chain's compiler identities and source positions, including distinct generic trait impls and reference receivers. Recoverable macro-generated method bodies are associated with their enclosing source impl; this is not general macro expansion. diff --git a/focxt/call_chain/README.md b/focxt/call_chain/README.md index 21d3a88496..87479613c4 100644 --- a/focxt/call_chain/README.md +++ b/focxt/call_chain/README.md @@ -29,7 +29,7 @@ The wrapper analyzes a single ordinary `lib` or `bin` target through `cargo chec ## Output -Paths are relative to the target crate: +For direct single-target analysis, paths are relative to the target crate: | Path | Contents | | --- | --- | @@ -37,4 +37,6 @@ Paths are relative to the target crate: | `focxt/callsandtypes/.json` | Direct calls and type dependencies for each function. | | `focxt/basic_blocks/.txt` | MIR basic blocks and locals for debugging. | +When `utgen analyze` processes a mixed package, these raw files live under `brinfo/targets///focxt/`. The top-level `focxt/impl_informations.json` is the combined focal index; it does not replace the raw target indices for looking up calls/types or debug files. + call-chain supplies compiler data; focxt produces the per-function `.rs` context files used by generation. Changes to shared data should be checked with both tools. See [minimal pipeline validation](../../docs/minimal-pipeline.md) and [bytes analysis validation](../../docs/bytes-analysis.md).