[diskann-benchmark] Add type reflection to enable documentation of benchmark inputs. by hildebrandmw · Pull Request #1459 · microsoft/DiskANN

Mark Hildebrand (hildebrandmw) · 2026-10-03T01:13:53Z

This has been a bugbear of mine for quite some time. It's been sitting mostly done for long enough that maybe it's just time to either get it in, or abandon it all together.

This implements type inspection for inputs in diskann-benchmark. Currently, we can get example JSON input schemas using (for example)

cargo run --package diskann-benchmark -- inputs graph-index-build

While this serves as a starting point, it does not document what the various fields are supposed to do. This is particularly annoying for enums, where the accepted values or syntaxes have to basically be reverse engineered from the source code.

Running the above command with this PR will also bring up an imperfect but IMO acceptable list of the high-level types. If there were rust-doc strings on these fields, they would also be rendered, providing an easy way to document fields and types for benchmark users.

A full list of registered types (again, far from perfectly rendered, but adequate) can be retrieved with

cargo run --package diskann-benchmark -- type-info

Documentation for specific types can be retrieved by naming it:

cargo run --package diskann-benchmark -- type-info

The heavy lifting here is done with a procedural macro Reflect (implemented here in diskann-macros). Most code can be updated by deriving Reflect and calling it a day.

Please see the docstring on Reflect in diskann-benchmark-runner/src/reflect/mod.rs for the trait definition and supported attributes.

Only serde(rename_all = "..."), serde(rename = "..."), and the enum related serde(tag = "...")/serde(content = "...") are supported. Attributes like serde(default), serde(try_from = "...") are not supported because benchmark inputs should be as simple as possible. Fancy constructs like this are a sign that the input DTO is too complicated and should be simplified.

Why is this PR so damn big?

That is an excellent question. A large amount of the complexity in this PR comes from needing to support serde's annoying corner cases and testing the infrastructure. The file diskann-macros/src/reflect/repr.rs has a good summary of the weirdness that arises from serde support.

Other than that, there is a chicken and egg problem. The proc-macro is useless without diskann-benchamrk-runner/src/reflect/tree.rs. The constructs in said file are really annoying to build properly without the proc-macro. Further, the AST structures in tree.rs need the Reflect trait (via Reflection) to work. Finally, all of this needs to be wired in through the benchmark registry and applied to our existing inputs, otherwise it's a bunch of code doing nothing.

Suggested Reviewing Order

I would recommend starting with diskann_benchmark_runner/src/reflect/mod.rs. This:

Defines the Reflect trait that is used for type inspection.
Defines the Reflection type for working with type-erased Reflect implementations.
Adds visit_with for exploring transitively reachable types from a given Reflection.
Implements Reflect for a number of primitive types.
The implementation of the Reflection type is as a reference to a vtable, initialized by a concrete Reflect implementation.

Next, I would look at diskann-benchmark_runner/src/reflect/tree.rs. This introduces the Type enum that is used to represent rust types we wish to reflect over. You will observe that these types are an absolute pain to implement manually. Must more so to keep in-sync with an actual type. That's where the procedural macro comes in.

Switch to diskann-macros/src/reflect/mod.rs. This is, for all intents and purposes, the entry point for the macro. All it does is try to parse a DeriveInput and emit the corresponding code. diskann-macros/src/reflect/repr.rs introduces an intermediate representation used to parse input syntax, process the supported attributes, legalize the attributes, and finally emit the builders for the Types. The structure of this is very similar to tree.rs. You can explore diskann-macros/src/reflect/attributes.rs at your leisure. This is what handles parsing the supported serde and reflect attributes and does some level of sanity checking on these.

diskann-macros/tests/* includes compile tests for the various error cases related to incorrect or otherwise illegal attributes. These validate the error messages associated with these cases, helping ensure that we provide somewhat actionable errrors.

Back in diskann-benchmark-runner I suggest looking at src/reflect/test/unit.rs for unit tests related to successful applications of the Reflect macro. It's neighbor serde.rs is kind of a mess and mainly tries to ensure that the representation of the Type emitted by Reflect agrees with how serde handles the same data structure.

Next, src/internal/typemap.rs is a registry for associating Reflect::type_name() with the associated Reflection type. The biggest service this provides is to support transactional registrations. The function TypeMap::register takes a list of Reflections and registers all transitively reachable types. So for example, registering Foo where Foo is defined as

struct Foo {
   bar: usize,
   baz: String,
}

will also attempt to register usize and String. If two types have the same typename but different type-ids, this is an error and all the types registered during the call are rolled back and removed. Additionally, the returned Transaction will roll-back a registration if dropped without calling Transaction::complete. This is useful for staging changes in Registry.

Finally, src/registry.rs wires up TypeMap and automatically registers reachable types for inputs. src/app.rs makes the type map reachable from the CLI.

In the rest of diskann-benchmark-runner:

src/utils/required.rs adds RequiredOption. This behaves like Option but with a critical difference for deserialization. Where serde treats a missing fields whose value is Option<T> as None, RequiredOption requires an explicit null in the input. This is meant to handle the lack of macro support for serde(deserialize_with = ...), which we currently use to achieve the same effect.
The rest of the changes are either testing related, or applying Reflect to diskann-benchmark-runner exported types.

The changes in diskann-benchmark are all about applying the Reflect trait. Many cases simply require deriving Reflect and using RequiredOption. There are some additional cases that deserve special attention:

diskann-benchamrk/src/inputs/graph_index.rs: StartPointStrategy. This currently uses serde(remote = "...") to deserialize to the type in diskann. This PR changes this to be a DTO for the diskann type.
diskann-benchamrk/src/inputs/graph_index.rs: DynamicRunbookParams. This type had a resolved_gt_directory field that was lazily initialized and not serialized. This PR reuses the gt_directory fields (behaving like InputFile) and stores the resolve path there directly.
diskann-benchmark/src/inputs/disk.rs: The DiskIndexOperation does a bunch of crazy things involving serde features, conditionally compiled fields, and deserializing remote types. This Reflect implementation is left as a stub.

…ildebr/reflect

Copilot

🟡 Changes recommended

Recursive derive bounds, input compatibility, and output redirection issues remain unresolved.

3 open issues

What changed in this PR

Adds runtime type reflection so benchmark users can inspect input fields, enum choices, and documentation through the CLI.

Changes:

Introduces the Reflect derive macro, type metadata, rendering, and transactional registration.
Adds type-info and extends input examples with documentation.
Updates benchmark inputs and adds compile-time and CLI regression coverage.

File	Description
diskann-macros/tests/ui/pass/supported.rs	Tests supported derive attributes.
diskann-macros/tests/ui/fail/unsupported_union.stderr	Records union rejection diagnostic.
diskann-macros/tests/ui/fail/unsupported_union.rs	Tests unsupported unions.
diskann-macros/tests/ui/fail/tagging.stderr	Records tagging diagnostics.
diskann-macros/tests/ui/fail/tagging.rs	Tests invalid tagging combinations.
diskann-macros/tests/ui/fail/serde_attributes.stderr	Records Serde attribute diagnostics.
diskann-macros/tests/ui/fail/serde_attributes.rs	Tests unsupported Serde attributes.
diskann-macros/tests/ui/fail/reflect_attributes.stderr	Records reflection attribute diagnostics.
diskann-macros/tests/ui/fail/reflect_attributes.rs	Tests invalid reflection attributes.
diskann-macros/tests/ui/fail/name_collisions.stderr	Records name-collision diagnostics.
diskann-macros/tests/ui/fail/name_collisions.rs	Tests renamed field and variant collisions.
diskann-macros/tests/compile.rs	Runs compile-pass and compile-fail fixtures.
diskann-macros/src/reflect/mod.rs	Connects parsing to macro expansion.
diskann-macros/src/reflect/attributes.rs	Parses documentation and supported attributes.
diskann-macros/src/lib.rs	Exposes the derive macro.
diskann-macros/Cargo.toml	Defines the procedural macro crate.
diskann-inmem/integration/support/tolerance.rs	Reflects empty tolerance inputs.
diskann-inmem/integration/support/datatype.rs	Reflects integration data types.
diskann-inmem/integration/store/simple.rs	Reflects simple-store inputs.
diskann-inmem/integration/store/mod.rs	Reflects shared store setup.
diskann-inmem/integration/store/intrusive.rs	Reflects intrusive-store inputs.
diskann-inmem/integration/store/checked.rs	Reflects checked-store inputs.
diskann-inmem/integration/index/runner.rs	Reflects index inputs and adopts required options.
diskann-benchmark/src/utils/mod.rs	Reflects similarity measures.
diskann-benchmark/src/multi_vector/driver.rs	Reflects multi-vector tolerances.
diskann-benchmark/src/main.rs	Adds registration timing output.
diskann-benchmark/src/inputs/multi_vector.rs	Reflects multi-vector configuration.
diskann-benchmark/src/inputs/graph_index.rs	Reflects graph inputs and simplifies serialization.
diskann-benchmark/src/inputs/flat.rs	Reflects flat-search inputs.
diskann-benchmark/src/inputs/filters.rs	Reflects filter inputs and removes a default.
diskann-benchmark/src/inputs/exhaustive.rs	Reflects exhaustive-search configuration.
diskann-benchmark/src/inputs/disk.rs	Adds a disk reflection placeholder.
diskann-benchmark/src/inputs/bftree.rs	Reflects BfTree inputs and adopts required options.
diskann-benchmark/src/index/inmem2.rs	Reflects inputs and changes quantization optionality.
diskann-benchmark/src/index/inmem/scalar.rs	Adapts start-point strategy access.
diskann-benchmark/src/index/inmem/product.rs	Adapts start-point strategy access.
diskann-benchmark/src/index/bftree/streaming_utils.rs	Uses the resolved ground-truth path directly.
diskann-benchmark/src/index/benchmarks.rs	Adapts graph input consumers.
diskann-benchmark/src/disk_index/benchmarks.rs	Reflects disk tolerances.
diskann-benchmark/example/graph-index-bftree-stream.json	Supplies the consolidation threshold.
diskann-benchmark-simd/src/lib.rs	Reflects and documents SIMD inputs.
diskann-benchmark-runner/tests/rendered_reflections.txt	Adds rendering baselines.
diskann-benchmark-runner/tests/benchmark/type-info-missing/stdout.txt	Records unknown-type output.
diskann-benchmark-runner/tests/benchmark/type-info-missing/stdin.txt	Exercises unknown-type lookup.
diskann-benchmark-runner/tests/benchmark/type-info-list/stdout.txt	Records registered type names.
diskann-benchmark-runner/tests/benchmark/type-info-list/stdin.txt	Exercises type listing.
diskann-benchmark-runner/tests/benchmark/type-info-list-all-features/stdout.txt	Records feature-enabled type names.
diskann-benchmark-runner/tests/benchmark/type-info-list-all-features/stdin.txt	Exercises feature-enabled listing.
diskann-benchmark-runner/tests/benchmark/type-info-list-all-features/features.txt	Enables fixture features.
diskann-benchmark-runner/tests/benchmark/type-info-describe/stdout.txt	Records type documentation output.
diskann-benchmark-runner/tests/benchmark/type-info-describe/stdin.txt	Exercises named-type documentation.
diskann-benchmark-runner/tests/benchmark/test-3/stdout.txt	Updates input documentation baseline.
diskann-benchmark-runner/tests/benchmark/test-2/stdout.txt	Updates optional-field documentation baseline.
diskann-benchmark-runner/src/ux.rs	Shares baseline-test helpers.
diskann-benchmark-runner/src/utils/required.rs	Adds required nullable values and tests.
diskann-benchmark-runner/src/utils/num.rs	Reflects constrained numeric values.
diskann-benchmark-runner/src/utils/mod.rs	Exports `RequiredOption`.
diskann-benchmark-runner/src/utils/datatype.rs	Reflects shared data types.
diskann-benchmark-runner/src/test/typed.rs	Reflects and documents typed fixtures.
diskann-benchmark-runner/src/test/gated.rs	Reflects feature-gated fixtures.
diskann-benchmark-runner/src/test/dim.rs	Reflects dimension fixtures.
diskann-benchmark-runner/src/registry.rs	Registers input and tolerance type metadata.
diskann-benchmark-runner/src/reflect/tree.rs	Defines reflection metadata structures.
diskann-benchmark-runner/src/reflect/test/unit.rs	Tests derived metadata.
diskann-benchmark-runner/src/reflect/test/mod.rs	Connects reflection test modules.
diskann-benchmark-runner/src/reflect/render.rs	Renders type documentation with baselines.
diskann-benchmark-runner/src/reflect/mod.rs	Defines reflection traits and traversal.
diskann-benchmark-runner/src/lib.rs	Exports reflection APIs.
diskann-benchmark-runner/src/internal/typemap.rs	Adds transactional type registration.
diskann-benchmark-runner/src/internal/mod.rs	Connects the type map.
diskann-benchmark-runner/src/input.rs	Requires reflection for raw inputs.
diskann-benchmark-runner/src/files.rs	Reflects input file paths.
diskann-benchmark-runner/src/app.rs	Adds type documentation commands.
diskann-benchmark-runner/dev/main.rs	Clarifies feature argument parsing.
diskann-benchmark-runner/Cargo.toml	Adds reflection and testing dependencies.
Cargo.toml	Registers the macro workspace crate.
Cargo.lock	Records dependency changes.

_{Review effort: Balanced}

+        for ty in self.types.iter() {
+            predicates.push(parse_quote!(#ty: #path::Reflect));


        pub(super) build: BuildParams,
        pub(super) search: KnnSearch,
-        pub(super) quantization: Option<Quantization>,
+        pub(super) quantization: RequiredOption<Quantization>,


        filters::register_benchmarks(&mut registry)?;
        multi_vector::register_benchmarks(&mut registry)?;

+        println!("registration took {}us", now.elapsed().as_micros());


Codecov Comments Bot (codecov-commenter) · 2026-10-21T17:40:20Z

Codecov Report

❌ Patch coverage is 88.09339% with 306 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.04%. Comparing base (79c5fcf) to head (12f508c).

Files with missing lines	Patch %	Lines
diskann-benchmark-runner/src/reflect/test/serde.rs	86.77%	48 Missing ⚠️
diskann-benchmark/src/inputs/graph_index.rs	31.57%	39 Missing ⚠️
diskann-benchmark-runner/src/reflect/render.rs	89.55%	35 Missing ⚠️
diskann-benchmark-runner/src/reflect/tree.rs	87.96%	26 Missing ⚠️
diskann-benchmark/src/inputs/bftree.rs	71.76%	24 Missing ⚠️
diskann-benchmark-runner/src/registry.rs	87.42%	21 Missing ⚠️
diskann-benchmark-runner/src/reflect/test/unit.rs	93.05%	15 Missing ⚠️
diskann-benchmark/src/index/inmem2.rs	7.14%	13 Missing ⚠️
diskann-benchmark-runner/src/reflect/mod.rs	90.75%	11 Missing ⚠️
diskann-benchmark/src/inputs/exhaustive.rs	0.00%	11 Missing ⚠️
... and 25 more

❌ Your patch status has failed because the patch coverage (88.09%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1459      +/-   ##
==========================================
- Coverage   92.10%   92.04%   -0.07%     
==========================================
  Files         590      601      +11     
  Lines      121495   123884    +2389     
==========================================
+ Hits       111905   114028    +2123     
- Misses       9590     9856     +266

Flag	Coverage Δ
miri	`92.04% <88.09%> (-0.07%)`	⬇️
unittests	`91.99% <88.09%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark-runner/dev/main.rs	`0.00% <ø> (ø)`
diskann-benchmark-runner/src/files.rs	`100.00% <100.00%> (ø)`
diskann-benchmark-runner/src/internal/mod.rs	`100.00% <ø> (ø)`
diskann-benchmark-runner/src/utils/required.rs	`100.00% <100.00%> (ø)`
diskann-benchmark/src/index/benchmarks.rs	`76.15% <100.00%> (+0.16%)`	⬆️
...kann-benchmark/src/index/bftree/streaming_utils.rs	`85.36% <100.00%> (+2.75%)`	⬆️
diskann-benchmark/src/index/inmem/product.rs	`78.63% <100.00%> (ø)`
diskann-benchmark/src/index/inmem/scalar.rs	`69.86% <100.00%> (ø)`
diskann-benchmark/src/inputs/disk.rs	`54.87% <100.00%> (+1.34%)`	⬆️
diskann-macros/src/lib.rs	`100.00% <100.00%> (ø)`
... and 36 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Aditya Krishnan (arkrishn94) · 2026-10-22T20:51:50Z

+///
+/// * `#[serde(rename = "name")]`: Describe with the given name instead of its Rust name.
+///
+/// * `#[serde(rename_all = "...)]`: Rename all fiels of this struct variant with the given


nit: typo p

Aditya Krishnan (arkrishn94) · 2026-10-22T21:01:20Z

+///
+/// * `#[serde(rename = "name")]`: Describe with the given name instead of its Rust name.
+///
+/// ## Reflect Attributes


Just for understanding - when would you want to use Reflection attributes over Serde attributes?

Aditya Krishnan (arkrishn94)

Thanks Mark, this is quite a feature. I have a super high level question - for folks using and contributing to benchmark for the first time, how can we make this feature discoverable? I'm trying to make sure that folks who aren't deeply familiar with the intricacies of diskann-benchmark can add define them correctly without much external review/feedback.

Mark Hildebrand added 30 commits May 26, 2026 17:00

Checkpoint.

6ded6e4

Checkpoint.

7a16b93

Merge remote-tracking branch 'origin/main' into mhildebr/reflect

784ff09

Checkpoint.

6f735df

Enum?

502e154

More derives!

57c5c33

Progress!

a91e2c9

Let's see if we can wire it up ...

4598bc1

Refactor

0deff77

Clean-up.

84097c3

Slowly expanding.

fae0fe9

Serde tests.

979d84f

Checkpoint.

9e81f18

More progress?

288505f

diskann-benchmark-simd.

8b2d91e

Registry tests.

900cc26

trybuild test and docs.

0530df6

minimal CLI hookup.

89f5ace

CLI baselines.

a550a07

CLoser!

8a29df9

Finish porting.

78a2698

Remove temp files.

3036872

Merge remote-tracking branch 'origin/main' into mhildebr/reflect

dee0443

Prepare to remove internal tagging.

fea7ce4

Prepare for implementation overhaul.

d76c0ef

Move to new impl.

9ae63ee

Checkpoint.

2961e80

Big overhaul.

cd7bdb0

Reorg.

9490d2d

Rename

1bddeb4

Mark Hildebrand added 12 commits October 8, 2026 21:31

AAAAHHHHHH

a9302a8

Partial support for tolerance inputs.

67ed305

Consolidate benchmark input type erasure.

1a22016

Add small docs.

7522c15

Finish the job.

b79e8bf

Merge remote-tracking branch 'origin/main' into mhildebr/reflect

8f04748

Merge remote-tracking branch 'origin/mhildebr/simplify-input' into mh…

1da8230

…ildebr/reflect

Rename again.

69df89f

Merge remote-tracking branch 'origin/main' into mhildebr/reflect

4e5e51e

Fixups.

8203633

Fix doctests.

7005228

Merge remote-tracking branch 'origin/main' into mhildebr/reflect

79c0bc9

Mark Hildebrand (hildebrandmw) marked this pull request as ready for review October 21, 2026 00:20

Mark Hildebrand (hildebrandmw) requested review from a team and a balanced review from Copilot October 21, 2026 00:20

Copilot started reviewing on behalf of Mark Hildebrand (hildebrandmw) October 21, 2026 00:20

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Rename and cleanup.

12f508c

Aditya Krishnan (arkrishn94) reviewed Oct 22, 2026

View reviewed changes

Aditya Krishnan (arkrishn94) approved these changes Oct 22, 2026

View reviewed changes

Navigation Menu

Repository navigation

[diskann-benchmark] Add type reflection to enable documentation of benchmark inputs. - #1459

[diskann-benchmark] Add type reflection to enable documentation of benchmark inputs.#1459
Mark Hildebrand (hildebrandmw) wants to merge 43 commits into
mainfrom
mhildebr/reflect

Mark Hildebrand (hildebrandmw) commented Oct 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Oct 21, 2026

Uh oh!

Aditya Krishnan (arkrishn94) Oct 22, 2026

Uh oh!

Aditya Krishnan (arkrishn94) Oct 22, 2026

Uh oh!

Aditya Krishnan (arkrishn94) left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		for ty in self.types.iter() {
		predicates.push(parse_quote!(#ty: #path::Reflect));

Conversation

Mark Hildebrand (hildebrandmw) commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why is this PR so damn big?

Suggested Reviewing Order

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Oct 21, 2026

Codecov Report

Uh oh!

Aditya Krishnan (arkrishn94) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Aditya Krishnan (arkrishn94) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Aditya Krishnan (arkrishn94) left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Mark Hildebrand (hildebrandmw) commented Oct 3, 2026 •

edited

Loading