Skip to content

[python] Repro: property access detaches container returned by __getitem__ - #12359

Draft
Josiah Vinson (jovinson-ms) wants to merge 1 commit into
microsoft:mainfrom
jovinson-ms:jovinson-microsoft-python-model-base-container-detach-repro
Draft

Josiah Vinson (jovinson-ms) wants to merge 1 commit into
microsoft:mainfrom
jovinson-ms:jovinson-microsoft-python-model-base-container-detach-repro

Conversation

@jovinson-ms

Copy link
Copy Markdown

Problem

In the generated model runtime (packages/http-client-python/generator/pygen/codegen/templates/model_base.py.jinja2), _RestField.__get__ deserializes list/dict/set values into a new container, writes that new container back into obj._data[rest_name], and sets _deserialized_<rest_name>. Any container a caller already got through obj["rest_name"] is now detached from _data, so mutations made through it are silently dropped when the model is serialized.

Repro

src = SourceStorageLocation(location="https://x.blob.core.windows.net/c", prefix="p", extensions=[".txt"])
filters = src["extensions"]
_ = src.extensions          # replaces _data["extensions"] with a new list
filters.append(".csv")
print(src.as_dict()["extensions"])
  • Expected: ['.txt', '.csv']. Dict access and property access share one backing container.
  • Actual: ['.txt']. The append is lost.

Observed with @azure-tools/typespec-python 0.63.10 / @typespec/http-client-python 0.39.0.

What this PR adds (tests only, no fix)

In tests/unit/test_model_base_serialization.py:

  • test_list_getitem_reference_survives_attribute_access: fails (list[str], the case above)
  • test_dict_getitem_reference_survives_attribute_access: fails (dict[str, str] variant)
  • test_list_attribute_reference_mutation_serializes: passes. Guard: property access first, then mutate, still serializes correctly and must keep working after a fix.

The failing tests are intentionally left plainly failing (not xfail). This package's tests have no xfail convention, and the goal of this draft is to show a red test that the fix turns green. If maintainers prefer, they can be switched to @pytest.mark.xfail(strict=True, reason=...).

A .chronus changeset with changeKind: internal is included because packages/http-client-python/tests/** is not excluded by changedFiles.

Local run

model_base.py.jinja2 was rendered (unbranded, corehttp) into a temporary specialwords._utils.model_base package. unit/test_model_base_serialization.py was then run with --noconftest, because its conftest auto-starts the Spector mock server:

FAILED test_list_getitem_reference_survives_attribute_access - AssertionError: assert ['.txt'] == ['.txt', '.csv']
FAILED test_dict_getitem_reference_survives_attribute_access - AssertionError: assert {'a': '1'} == {'a': '1', 'b': '2'}
2 failed, 123 passed

The guard test and all existing tests in that file pass.

Likely location and suggested fix direction

_RestField.__get__ in model_base.py.jinja2:

if isinstance(deserialized, (dict, list, set)):
    obj._data[self._rest_name] = deserialized   # <-- replaces the container
    object.__setattr__(obj, cache_attr, True)
    return deserialized

Suggested direction:

  • If deserialization didn't convert anything (deserialized == item and same type), keep the existing _data container (return item) instead of replacing it.
  • Otherwise, update the existing container in place so outstanding references stay attached, e.g. item[:] = deserialized for lists, item.clear(); item.update(deserialized) for dicts/sets.

Found during review of Azure/azure-sdk-for-python#49627.

Created by Copilot on behalf of Josiah Vinson (Josiah Vinson (@jovinson-ms))

… container

Add failing unit tests showing that a list/dict obtained via obj["key"]
is detached from the model once the attribute is accessed, because
_RestField.__get__ replaces obj._data[key] with a newly deserialized
container. Also add a passing guard for property-then-mutate.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: beceb370-8e32-49c3-9976-c8b3fb3619cf
@microsoft-github-policy-service microsoft-github-policy-service Bot added the emitter:client:python Issue for the Python client emitter: @typespec/http-client-python label Oct 22, 2026
@pkg-pr-new

pkg-pr-new Bot commented Oct 22, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-python@12359

commit: 08f5b6c

Josiah Vinson (jovinson-ms) pushed a commit to jovinson-ms/azure-sdk-for-python that referenced this pull request Oct 22, 2026
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849
@github-actions

Copy link
Copy Markdown
Contributor

All changed packages have been documented.

  • ✅ @typespec/http-client-python
Show changes

@typespec/http-client-python - internal ✏️

Add failing unit tests reproducing a container returned by __getitem__ becoming detached after property access in generated model_base.

@github-actions

Copy link
Copy Markdown
Contributor

Python emitter diff

Baseline gh:721154b2ab55bfc78ab4850e86c40487f66241f2 vs this PR.

No changes to generated output.

Rendered diff: inline on the run summary, or the emitter-diff-html artifact.

Informational check (eng/emitter-diff); does not block the PR.

@azure-sdk-automation

Copy link
Copy Markdown

You can try these changes here

🛝 Playground 🌐 Website 🛝 VSCode Extension

Josiah Vinson (jovinson-ms) added a commit to Azure/azure-sdk-for-python that referenced this pull request Oct 26, 2026
…ors, add test-resources readiness probe (#49627)

* [Health Deidentification] Remove stale generated files and regenerate

Fixes pyright reportArgumentType (issubclass) and pylint C0123 failures in
the healthdataaiservices CI pipeline.

- Delete stale _serialization.py, _model_base.py, _vendor.py, aio/_vendor.py
  left over from an older emitter (only referenced each other).
- Point tsp-location.yaml at the relocated spec
  (specification/healthdataaiservices/data-plane/HealthDataAIServices @ c7d9e33)
  and regenerate with @azure-tools/typespec-python 0.63.10. No API changes.
- Emitter moves packaging to pyproject.toml (Python >=3.10, azure-core >=1.37.0)
  and adds types.py; bump to 1.1.0b2 and note it in CHANGELOG/README.
- Tests: exclude Accept header from matching (regenerated delete_job no longer
  sends it) and disable the background LRO poller in the sync create/cancel/delete
  test, which otherwise races the test's own GETs in playback.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [Health Deidentification] Note new types module in CHANGELOG

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* Opt out of TypedDict generation for azure-health-deidentification

Regenerated with generate-typeddict: false so the JSON overloads match
1.1.0b1 and the inaccurate request-body TypedDicts (types.py) are removed.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [Health Deid] Replace fixed sleep with retry/backoff in test-resources-post.ps1

Role assignment and network ACL propagation can take minutes after
deployment, causing intermittent 403 AuthorizationFailure on upload.
Retry container access and each blob upload with exponential backoff
(5s initial, 60s cap, 5 min budget) and fail with a clear error.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6

* [Health Deid] Keep a 30s minimum wait after the network rule update

The retry loop proves the test principal's RBAC and the network rule, but
not the DeID managed identity's role assignment. Restore the previous 30s
wait as a minimum measured from the network rule update; time spent on
retries and uploads counts toward it.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6

* [DeID] Surface job errors in live test failures; random live job names

- Add wait_for_job/wait_for_job_async test helpers that report job status,
  error, summary, and unsuccessful documents instead of azure-core's generic
  'Operation returned an invalid status OK', and fail explicitly when the job
  is not terminal after the timeout (now 600s).
- Generate a fresh uniquifier for every live run (including
  AZURE_SKIP_LIVE_RECORDING); only persist it when recording.
- Samples print job.error before re-raising on job failure.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 40cdcc81-accc-4b2e-af24-fd79c6480740

* [DeID] Sync README snippet with deidentify_documents sample

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 40cdcc81-accc-4b2e-af24-fd79c6480740

* [DeID] Make sample job diagnostics best-effort; always close async credentials

The diagnostic get_job lookup in the samples' error handling could raise
(for example 404 when creation was rejected) and replace the original
error. It is now wrapped so the original exception is always re-raised.
The async samples close the credential via async with.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [DeID] Point tsp-location at spec commit with generate-typeddict: false

Spec PR Azure/azure-rest-api-specs#47496 adds generate-typeddict: false to
the Python emitter options. Regenerating from that commit produces no
changes to the generated code.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [DeID] Point tsp-location at merged spec commit

Azure/azure-rest-api-specs#47496 merged as 6de3840. Regenerating from it produces no changes to the generated code.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [DeID] Keep job diagnostics when document listing fails

describe_job(_async) now fetches the job and lists documents in separate try blocks, so a listing failure (even on a later page) no longer discards the job status/error or documents already collected; the listing error is appended instead.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [DeID] Preserve model container identity across dict and property access

The generated _RestField.__get__ replaces the backing list/dict/set in _data
when a mutable field is first read as a property, detaching any reference
previously returned by model["field"]. Wrap __get__ in patch_sdk() to copy
the deserialized contents into the original container instead, until the
emitter template is fixed. Adds unit tests for all dict/property access
patterns plus a guard that fails once the generated code no longer needs
the shim.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [DeID] Link microsoft/typespec#12359 as the shim removal tracker

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

* [DeID] Fix cspell failure in container-identity shim comment

The shim's TODO quoted the emitter template's repository path, which
contains "pygen" and fails the cspell check. Describe the template by
name instead.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849

---------

Co-authored-by: Josiah Vinson <[email protected]>
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: ef2497f1-4ec3-4283-9448-f9679c8b7849
Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6
Copilot-Session: 40cdcc81-accc-4b2e-af24-fd79c6480740

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

emitter:client:python Issue for the Python client emitter: @typespec/http-client-python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants