Skip to content

feat: generate typed contract reads from schemas (DXP-537) - #124

Open
jmlago wants to merge 3 commits into
v0.19-devfrom
legatus/dxp-537
Open

jmlago wants to merge 3 commits into
v0.19-devfrom
legatus/dxp-537

Conversation

@jmlago

@jmlago jmlago commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Merge hold

Do not merge into frozen v0.19-dev / 0.19.0. Passing tests and advisory E2E do not override this release freeze. Hold until the next development branch exists and retarget there, or until the repository owner explicitly approves inclusion. The currently declared active branch is still v0.19-dev.

Package-wide typing enablement is now separately proposed in #125, with an unreleased compatibility note and a package-wide strict audit/known-issues list. This feature depends on that independently reviewed change. The identical marker/package-data/ContractMethod prerequisite edits remain in this branch so the feature can be tested; merge the prerequisite only when authorized, then rebase this PR so those edits disappear from its diff before merging this feature. The prerequisite reports 373 existing strict errors in 33 SDK files, not a strict-clean SDK. Neither PR is authorized to bypass the freeze.

PR #123 remains open and this PR currently reports MERGEABLE. Both touch genlayer_py/__init__.py; rebase the second to land and preserve both exports when that happens. There is no current conflict to resolve.

Summary

Contract reads currently return generic calldata values even when a schema describes their structure. Add generate_contract_types(schema, class_name="Contract", strict=False), which emits an importable Python module with typed view methods, public result aliases, and nested TypedDict records. A generated Storage(client, address).get_balance(owner) exposes schema argument and result types while forwarding through the existing SDK read path.

Support GenVM primitive types, records, unions, dictionaries, repeated and fixed arrays. Preserve record field names literally. GenVM's _repr_type dataclass branch copies field names directly; _escape_dict_prop is used for keyword parameters, not these records. Removed the inferred record $$ unescaping, so $$name stays $$name.

Unsupported method or parameter names now skip only that method, with a safely escaped comment in the generated source. strict=True preserves fail-fast behavior. Invalid class names and unsupported schema types still raise ValueError. Tests cover method, positional-parameter and keyword-parameter names, including newline-containing hostile names.

Generated imports use defining modules for strict-compatible exports. The README usage section is alongside the other contract examples, before Key Features and License.

Ticket: https://linear.app/genlayer-labs/issue/DXP-537/generate-contract-types-from-schema

Assumptions

  • Generate reads and result types; writes and deployment stay on the existing SDK API. This scope is explicit in the README and was accepted in the review follow-up.
  • Keyword parameters remain required, agreeing with genlayer-js#248 after its update. Schema defaults are not available. Positional parameters are positional-only; keyword parameters remain keyword-only.
  • Return Python source without network calls, runtime execution, or automatic file writes. Callers choose the output path and regeneration time.
  • Preserve decoded values: dictionaries use TypedDict, addresses use CalldataAddress, arrays use lists. Fixed-array length is not enforced; element types form a union.
  • Generated reads use client defaults, including the connected account. Custom options use client.read_contract.
  • Class/method/parameter names must be ASCII Python identifiers without a leading underscore. The default skips only methods with unsupported names and emits diagnostic comments; strict mode raises. Record field names are literal.
  • Export <Class>_<method>_Result aliases; internal aliases prevent generated-name collisions. Output is deterministic.
  • Integration coverage uses actual SDK schema/read actions and calldata codecs against a mocked provider. Live E2E is separately requested as advisory.
  • Interpret the placeholder <100 ms requirement as local generation latency, not network read latency. Human review and release-owner approval remain necessary.

Validation

  • uv run pytest tests/unit --disable-warnings -q -m "not testnet": 282 passed, 17 deselected. No wheel builds, network installs, or mypy invocation occur in this unit suite.
  • uv run pytest tests/packaging -m packaging -q: 1 passed. This explicitly builds the wheel, installs it in isolation, verifies the marker is packaged, runs strict mypy on generated code and valid usage, and verifies rejection of an invalid address argument, missing keyword argument, and unknown result field.
  • Packaging checks were moved out of tests/unit and behind the registered packaging marker. Default pytest runs exclude them; the explicit check fails rather than skips if uv is absent. README documents its network/uv requirements and command.
  • uv run pytest --collect-only -q: 334 selected, 1 packaging check deselected by default.
  • Black and git diff --check pass.
  • Prior performance measurement: a 100-method schema averaged 1.02 ms/generation over 100 runs; not a timing-sensitive test.

Advisory E2E is requested separately. The release freeze and prerequisite review are merge gates regardless of its result.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 19b3cbae-fbc5-4ca0-ae99-171cd9c0b40c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

/run-e2e

@ci-core-e2e-runner

Copy link
Copy Markdown
Contributor

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

@danieljrc888 danieljrc888 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at ea97a81. Two things: a merge gate that no code change resolves, and a short list of design points to settle while it waits.

Merge gate (blocking)
Same situation as #123: the base is v0.19-dev, frozen for the v0.6 train until 0.19.0 stable is tagged as part of the Studio v0.123 release in progress. A new top-level export (generate_contract_types) is public API that would ship in 0.19.0 without an rc cycle. It also changes ContractMethod in genlayer_py/types/contracts.py from a plain class to a TypedDict; the fix is right, but it is a behaviour change to an existing exported type on a frozen line. Hold until the next development branch exists and retarget there, or until the repo owner explicitly decides to include it in 0.19.0. The PR body says E2E is advisory and not a merge gate; the freeze is the gate.

What I verified

  • Generated bindings forward to read_contract with the argument shapes it expects (genlayer_py/contracts/actions.py), and result types match the Python decoder (no JSON-safe conversion exists in this SDK, so CalldataAddress, bytes, dict, list are right).
  • Names reach the generated source only after isidentifier / iskeyword / leading-underscore checks, and keys only through repr, so a hostile schema cannot inject code. This matters more here than in the JS generator because the output is imported and executed.
  • _typing.Never and the rest are fine for requires-python >= 3.12.

Design points to settle

  1. Inconsistent with the JS implementation of the same feature (genlayer-js#248, DXP-538). There keyword parameters are optional and write methods are included; here keyword parameters are all required and only views are generated. A contract method with a defaulted keyword argument is callable without it through client.read_contract but not through the generated binding. Pick one rule for both SDKs. Since the schema does not expose defaults, optional keyword arguments (omitted from the call when not passed) is the choice that never rejects a valid call.
  2. $$ key unescaping. The generator treats $$name as an escaped $name field and rejects other $ keys. I could not find that convention anywhere in this repo. Link the GenVM schema source that defines it, or drop it; inventing an escape rule here means silently mistyping a field if GenVM does something else.
  3. One unsupported name fails the whole contract. A single view method or parameter that fails _identifier raises for the entire schema. Skipping that method with a comment in the output, or raising only in a strict mode, keeps the generator usable on contracts the author does not control.
  4. Conflict with #123. Both PRs edit genlayer_py/__init__.py; whichever lands second needs a rebase.

Minor

  • The README section is appended after ## License; move it next to the other contract usage docs.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Legatus audit: Changes requested (commit ea97a81)

Audit of PR #124 (DXP-537): typed contract reads generated from the schema

Head reviewed: ea97a817a749ad6e37594341e5683acc23f04f7e. Base v0.19-dev is correct: it matches support/ci/ACTIVE_DEV_BRANCH and docs/BRANCHING.md, and the PR has never been retargeted.

The PR adds generate_contract_types(schema, class_name). It produces the source of a Python module with one class per contract, a method for each view, and TypedDict and type-alias result types. The work stays within the ticket. The schema handling matches what GenVM actually produces (genvm/runners/genlayer-py-std/src/genlayer/_internal/get_schema.py): plain values, array, dict and any, $or, $dict, [{"$rep": T}], fixed-length tuples and dataclass records. Name checks block code injection and name clashes. Write methods are left out.

Checks I ran

  • Tests: CI runs uv run pytest tests/unit -m "not testnet". I got 281 passed. The 6 failures are all testnet-marked smoke tests, which CI leaves out of this job. test_codegen.py passes 28/28.
  • Static typing: I generated a module from a realistic schema (record with a nested record, $dict of a union, a tuple, an optional value, a null return, a write method) and ran mypy and pyright on it.
    • The generated module itself type-checks cleanly.
    • Using it wrongly is caught: a wrong value type, an unknown key, a missing keyword argument, and calling a write method.
  • CI: Tests on 3.12 and 3.13, branch policy and CodeRabbit pass. E2E Tests (advisory) was still running when I looked; I did not wait for it.

Findings (most serious first)

  1. Must fix: for anyone installing from PyPI, the generated module fails mypy by default, and address arguments lose their type checking.

    • Where: genlayer_py/contracts/codegen.py:130-133 (the generated imports), plus packaging: the package has no py.typed marker.
    • What happens: I built the wheel (uv build) and installed it, not in editable mode, in a clean venv. Running mypy on the freshly generated module gives:
      storage_types.py:6: error: Skipping analyzing "genlayer_py.types": module is installed, but missing library stubs or py.typed marker [import-untyped]
    • Effect: the SDK's types become Any inside the generated module. storage.get_user("0x00", ...), where a CalldataAddress is required, is no longer flagged. That same call is flagged when the package is visible to the checker.
    • Fix: add genlayer_py/py.typed and make sure it ships in the wheel. Packaging is setuptools with pyproject.toml; the wheel I built has 0 py.typed files.
    • Test to add: run mypy on a generated module, or at least check that py.typed is packaged.
    • The README promises that "editors can check dictionary keys and their types". That promise is only partly kept today.
  2. Should fix: the generated module fails mypy --strict.

    • Where: codegen.py:132, which emits from genlayer_py.types import CalldataAddress, CalldataEncodable.
    • What happens: genlayer_py/types/__init__.py re-imports these names without __all__, so strict mode reports Module "genlayer_py.types" does not explicitly export attribute "CalldataAddress" (the same for CalldataEncodable). The file says "Do not edit", so users can't fix it themselves.
    • Fix: generate the imports from the modules that define these names (genlayer_py.types.calldata, genlayer_py.client.genlayer_client), or add __all__ to genlayer_py/types/__init__.py.
  3. Should fix: the README section is in the wrong place.

    • Where: README.md:428.
    • "### Generate typed contract reads" is appended after ## License, which makes it a subsection of License. Move it next to the other usage sections, for example after "Debugging transaction execution" (around line 390).
  4. Minor, no change needed:

    • A method entry missing readonly, params, kwparams or ret raises KeyError, not the documented ValueError. GenVM always emits these keys, so this is a robustness point only.
    • A public method whose name starts with _ makes the whole generation fail instead of skipping that method. This is documented.
    • The ADR mentions {"$ref": ...} schema entries. Today's get_schema.py never emits them, and they are rejected with ValueError, which is acceptable.
  5. Note on the tests: the "integration" test (test_sdk_schema_to_decoded_read_integration) runs the real client, schema action and calldata codec against a mocked provider. That is reasonable for this repo. There is no live Studio/localnet e2e test, and no test that runs a type checker on generated code. The test from item 1 would close the most valuable of those gaps.

What must change before merge

  • Ship a py.typed marker in genlayer_py, so that generated bindings type-check in user projects that install from PyPI, with a test or packaging check that keeps it shipped.
  • Make the generated imports pass mypy --strict.
  • Move the README section out from under ## License.

@danieljrc888

Copy link
Copy Markdown
Contributor

Update on design point 1: genlayer-js#248 now makes keyword parameters required as well (558c5fa3), so the two SDKs agree on that rule and the kwargs part of the point is settled. The remaining difference is scope (JS generates writes, this generates reads only), which is fine if the README says so, and it does. Points 2 and 3 and the merge gate are unchanged.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

/run-e2e

@ci-core-e2e-runner

Copy link
Copy Markdown
Contributor

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

/run-e2e

@ci-core-e2e-runner

Copy link
Copy Markdown
Contributor

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

@danieljrc888

Copy link
Copy Markdown
Contributor

Re-checked at 192b75d. The requested-changes state stays, and this commit widens the scope in a way that makes the freeze concern stronger.

New in this commit: genlayer_py/py.typed
Adding the PEP 561 marker is not local to the generator. It tells every consumer's type checker to read the inline annotations of the whole package. Until now mypy treated genlayer_py imports as untyped; after this, any project that type-checks against the SDK starts seeing whatever is incomplete or wrong in the existing annotations (the ContractMethod definition this PR fixes is one example of what is in there). That is a package-wide behaviour change for downstream CI. It may well be worth doing, but it needs its own PR, a mypy --strict pass or a known-issues list over the package, and a release note. It must not ride into 0.19.0 inside a typegen feature on a frozen branch.

New test does network installs inside the unit suite
tests/unit/contracts/test_codegen_wheel.py runs uv build, then uv run --isolated --with <wheel> --with mypy==2.4.1, which resolves and downloads the SDK's dependencies plus mypy on every unit run. It is skipped silently when uv is missing, so it can also pass by not running. Move it behind a marker (or a dedicated CI job) so the unit suite stays offline and deterministic, and fail instead of skip in the job that is meant to run it.

Resolved

  • README placement.
  • Keyword-argument rule now agrees with genlayer-js#248.

Still open

  • Merge gate: frozen v0.19-dev.
  • $$ key unescaping: source for the convention, or remove it.
  • One unsupported name fails the whole schema.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Legatus audit: Approved (commit 192b75d)

Re-audit of PR #124 (DXP-537): typed contract reads generated from the schema

I approve commit 192b75d4aa1dc76bcd646daa6c36419d2186023f: all three changes from the previous round are fixed, and I found nothing new that blocks the merge. Base is still v0.19-dev, which is correct.

Previous findings

  1. py.typed marker, so installed users get type checking (fixed).
    • The PR adds genlayer_py/py.typed and lists it in pyproject.toml package-data.
    • I built the wheel and confirmed it contains genlayer_py/py.typed.
    • A new test, tests/unit/contracts/test_codegen_wheel.py, builds the wheel and installs it in a separate environment with mypy==2.4.1. It then runs scripts/check-generated-types.py.
      • The script first checks that it is using the installed wheel, not the source tree, and that the marker is present.
      • It runs mypy --strict on a generated module and on correct usage of it.
      • It checks that wrong usage gives exactly 3 errors: a str where an address is required, a missing keyword argument, and an unknown key.
    • Run by hand against the installed wheel, the script passes and reports those 3 errors.
  2. Imports that fail mypy --strict (fixed). codegen.py:130-131 now imports from genlayer_py.client.genlayer_client and genlayer_py.types.calldata. The generated __init__ gains -> None. Generated code now passes mypy --strict.
  3. README section under ## License (fixed). "Generate typed contract reads" now comes right after the debugging section and before Key Features, Documentation, Contributing and License. That part of the change only moves existing text.

Tests

  • Locally: uv run pytest tests/unit -m "not testnet" (the command CI uses) gives 271 passed, 17 deselected.
  • One local glitch, not caused by the PR: in some isolated runs the wheel test crashed with SIGSEGV. The crash was inside mypy itself, or in one run inside the setuptools build step, on my NixOS Python. It disappeared once the cache was warm, and the next three runs passed. I don't attribute it to the PR.
  • CI: Tests on Python 3.12 and 3.13 pass, as do branch policy and CodeRabbit. E2E Tests (advisory) was still running when I looked; I did not wait for it.

Notes (not blocking)

  • The wheel test needs network access. It runs uv build and downloads mypy==2.4.1, adding about 10 s. It is skipped only when uv is missing, so pytest tests/unit on a machine without PyPI access will fail. A skipif for that case or a marker would be kinder to offline developers.
  • Shipping py.typed changes things for SDK users. Projects that run mypy will now have their own calls into the SDK checked against its annotations, which may show new errors for them. This is worth a CHANGELOG line when the release is cut.
  • Correction to an early finding of mine in this round: I first thought the new test left a build/ directory in the source tree. That was wrong. Plain uv build builds the wheel from an sdist in a temporary directory. The stray directory came from my own uv build --wheel runs. That led me to a small packaging problem that already existed, reported below.

LEGATUS-TICKET
team: DXP
title: genlayer-py: setuptools packages.find picks up build/ so repeated in-tree wheel builds ship duplicate stray packages
repo: genlayer-py
blocks: no
pyproject.toml [tool.setuptools.packages.find] has where = ["."] and exclude = ["tests*", "scripts*"], which is the same on origin/v0.19-dev. It does not exclude build*. In a clean checkout of 192b75d, running uv build --wheel -o <dir> . three times in a row gave wheels with 0, 57 and 114 entries under build/, for example build/lib/build/lib/genlayer_py/__init__.py. Each build packages the previous build's build/lib output again. Expected: every wheel contains only genlayer_py/. publish.yml runs rm -rf -- dist build *.egg-info before uv build, so released wheels are not affected; only local or manual builds are. Suggested fix: add "build*" (and possibly "dist*") to exclude.
LEGATUS-END

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

/run-e2e all

@ci-core-e2e-runner

Copy link
Copy Markdown
Contributor

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

/run-e2e

@ci-core-e2e-runner

Copy link
Copy Markdown
Contributor

E2E status was updated. Follow the current E2E and merge checks on this PR. Detailed diagnostics are available internally.

@danieljrc888

Copy link
Copy Markdown
Contributor

Re-reviewed at 2aab8ff. The three code points from my review are addressed:

  • $$ keys are kept literally, so no unsourced escaping convention remains.
  • An unsupported method or parameter name now skips only that method with a source comment, and strict=True restores the hard failure.
  • The wheel build and mypy install moved to an opt-in packaging marker outside tests/unit.

My review stays as changes requested for two reasons that are not about this code: the base is the frozen v0.19-dev, and this now depends on #125. After #125 lands, rebase so py.typed, pyproject.toml and types/contracts.py drop out of this diff.

One small thing: addopts = -m "not packaging" is replaced, not combined, when a caller passes its own -m (the last one wins). CI is unaffected because it targets tests/unit, but pytest -m "not testnet" from the repo root will run the packaging test with a network install. Either skip inside the test unless an env var is set, or document the exact command.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants