From e6a40da1c352716835c88f5dc7d9f4764453de4c Mon Sep 17 00:00:00 2001 From: jdsika Date: Thu, 24 Sep 2026 13:07:39 +0200 Subject: [PATCH] feat(generators): translate class-level boolean expressions alike to SHACL and JSON Schema gen-shacl dropped class-level any_of, all_of, exactly_one_of and none_of, so a class stating "a code or a name is required" generated shapes that accepted everything. gen-json-schema translated them, but only for the declaring class, and read an absent slot differently depending on where an expression was nested. Both generators now give class expressions one semantics, shared in linkml.generators.common.class_expression: - A class expression constrains every instance of its class, so a class carries the expressions of its ancestors and mixins, translated in its own context. - The specification does not say what a slot condition means for an absent slot. As an SQL CHECK constraint (ISO/IEC 9075) reads a condition on null, a condition that doesn't decide whether its slot may be absent is unknown for an absent slot. An instance is invalid only when an expression is definitely false, and exactly_one_of holds when exactly one member is definitely true. The operators compose, and gen-json-schema no longer rejects an instance meeting only the first member of an exactly_one_of while accepting one meeting none. gen-shacl emits sh:or, sh:and, sh:xone and sh:not (the operators' exact_mappings, SHACL 4.6) over anonymous member shapes, each in its "not false" form or, under sh:not, its "definitely true" form, where such a condition also has sh:minCount 1. Conditions translate required, value_presence, the cardinalities, minimum/maximum_value, pattern, equals_string(_in), equals_number and range; a parameter SHACL allows once per shape that one condition needs twice moves into an sh:and member. An operator whose members use anything else is skipped whole, with one warning naming the shapes it is missing from. gen-json-schema builds the same forms with anyOf, allOf, oneOf and not, resolves conditions on the induced slot, and also enforces cardinalities (minItems / maxItems, or minProperties / maxProperties for a slot inlined as a dict, whose values are the values of the dict) and range in class-level conditions. value_presence: ABSENT on several slots of one expression now requires each to be absent rather than not all of them present, in rules too. An is_a member names a class that every instance of the class carrying the expression belongs to (the class or an ancestor), that only some belong to (through a descendant), or that none belongs to, since an instance belongs to its class and that class's ancestors. Only the second case checks the instance: sh:class in SHACL, and in JSON Schema, which carries no type, a reference to the class's definition. Before, a class inherited its parent's any_of over its own subclasses as sh:class and $ref to itself, which recursed without end in JSON Schema and, in SHACL, rejected a node typed only as a subclass. The slot loop's range dispatch and sh:path computation move into _add_range and _slot_iri; output for schemas without class expressions is unchanged. The compliance tests test_class_any_of and test_class_any_of_with_required run for SHACL. Signed-off-by: jdsika --- docs/generators/json-schema.rst | 10 + docs/generators/shacl.rst | 92 + docs/schemas/advanced.md | 30 + .../generators/common/class_expression.py | 150 ++ .../src/linkml/generators/jsonschemagen.py | 174 +- .../linkml/src/linkml/generators/shaclgen.py | 430 ++++- .../test_boolean_slot_compliance.py | 22 +- .../test_generators/test_jsonschemagen.py | 243 +++ tests/linkml/test_generators/test_shaclgen.py | 1531 ++++++++++++++++- .../test_validator/test_validation_context.py | 20 +- 10 files changed, 2625 insertions(+), 77 deletions(-) create mode 100644 packages/linkml/src/linkml/generators/common/class_expression.py diff --git a/docs/generators/json-schema.rst b/docs/generators/json-schema.rst index 87c85552cb..e696d66cd7 100644 --- a/docs/generators/json-schema.rst +++ b/docs/generators/json-schema.rst @@ -172,6 +172,16 @@ LinkML supports analogous elements: Use of these elements will be translated into the appropriate JSON-Schema construct. +At the class level, a class carries the expressions of its ancestors and mixins. +An ``is_a`` member names a class every instance already belongs to, or none can, +unless only some do, through a descendant: then the instance must match that +class's definition, since a JSON object carries no type. +A slot condition is unknown for an absent slot unless it decides whether the slot +may be absent, and an instance is invalid only when an expression is definitely +false, as described under "Class-level expressions and absent slots" in +:doc:`Advanced features `. +The SHACL generator reads them the same way. + Inlining ^^^^^^^^ diff --git a/docs/generators/shacl.rst b/docs/generators/shacl.rst index 3e88f0090f..9322186e44 100644 --- a/docs/generators/shacl.rst +++ b/docs/generators/shacl.rst @@ -84,6 +84,98 @@ Example Output: shacl:targetClass . +Class Expressions +^^^^^^^^^^^^^^^^^ + +Class-level boolean expressions become the SHACL logical constraint components +their metamodel definitions map to (`SHACL §4.6 +`__): + +================== ===================================================== +LinkML SHACL, on the class's ``sh:NodeShape`` +================== ===================================================== +``any_of`` ``sh:or`` over the member shapes +``all_of`` ``sh:and`` over the member shapes +``exactly_one_of`` ``sh:xone`` over the member shapes +``none_of`` one ``sh:not`` per member +================== ===================================================== + +Each member becomes an anonymous node shape. ``is_a`` names a class that every +instance of the shape's class belongs to (the class itself or an ancestor), and +then adds nothing; one that only some instances belong to, because a descendant +of the class has it among its ancestors, and then gives ``sh:class``; or one that +none belongs to, and then gives a shape nothing conforms to. An instance belongs +to its class and that class's ancestors, by ``is_a`` and by ``mixins``, so a +class that may also be another is declared as a subclass with that mixin. +Nested expressions recurse. Each entry of ``slot_conditions`` gives an +``sh:property`` whose path is that of the slot as induced for the class, so +``slot_usage`` applies: + +* ``required``, ``value_presence`` and the cardinalities give ``sh:minCount`` / + ``sh:maxCount``; +* ``minimum_value`` / ``maximum_value`` give ``sh:minInclusive`` / + ``sh:maxInclusive``, and ``equals_number`` gives both, so that ``5`` also + matches ``5.0``; +* ``pattern`` gives ``sh:pattern``; +* ``equals_string`` and ``equals_string_in`` give ``sh:in``; on an enum slot the + values are the permissible values as the enum renders them, the IRI of their + ``meaning`` where they have one; +* ``range`` gives the same class, type or enum constraint as a slot's range. + +A shape may have at most one value of ``sh:minInclusive``, ``sh:maxInclusive`` +or ``sh:in``, and of ``sh:pattern``, whose component also takes ``sh:flags`` +(`SHACL §2.1.1 `__). Where one condition +needs one of them twice, for example ``minimum_value`` next to +``equals_number``, the second value goes into an ``sh:and`` member of the +property shape, where it applies to the same values. + +A slot condition is unknown for an absent slot unless it decides whether the slot +may be absent, and an instance violates an expression only when the expression is +definitely false, as described under "Class-level expressions and absent slots" in +:doc:`Advanced features `. +The JSON Schema generator reads them the same way. In SHACL, each expression gets +its "not false" form, and, under ``sh:not``, its "definitely true" form, where +such a condition also requires its slot (``sh:minCount 1``). ``exactly_one_of`` +becomes ``sh:xone`` over the "definitely true" forms of its members. + +.. code-block:: yaml + + GeodeticReferenceSystem: + slots: [code, name] + any_of: + - slot_conditions: + code: + required: true + - slot_conditions: + name: + required: true + +.. code-block:: turtle + + ex:GeodeticReferenceSystem a sh:NodeShape ; + sh:or ( [ sh:property [ sh:path ex:code ; sh:minCount 1 ] ] + [ sh:property [ sh:path ex:name ; sh:minCount 1 ] ] ) ; + ... + +A class expression constrains every instance of its class, so the shape of a +class carries the expressions of its ancestors and mixins as well, as it carries +their slots. Each is translated in the context of that class, where +``slot_usage`` applies. A node typed only with a subclass, as ``rdflib_dumper`` +and the JSON-LD context produce it, is therefore checked without an +``rdfs:subClassOf`` triple in the data. The ``sh:class`` that ``is_a`` gives +recognises instances of subclasses only where the data graph states the +``rdfs:subClassOf`` (`SHACL §4.1.1 +`__), as it does for a +slot's range. + +An operator whose members use anything else is skipped as a whole and logged as +a warning, because leaving out one member would change what the operator +admits. That covers, for example, ``has_member`` or a slot-level ``any_of`` +inside a slot condition, a condition on a name that is not a slot, a condition +on the identifier slot (the node's IRI rather than a property), and +``equals_string`` on a slot whose range does not hold strings. + + Command Line ^^^^^^^^^^^^ diff --git a/docs/schemas/advanced.md b/docs/schemas/advanced.md index c56a808379..2f173f2443 100644 --- a/docs/schemas/advanced.md +++ b/docs/schemas/advanced.md @@ -64,6 +64,36 @@ The following LinkML constructs can be used to express boolean constraints: These can be applied at the class or slot level. The range of each of these is an array of *expressions*. +### Class-level expressions and absent slots + +At the class level, each member of `any_of`, `all_of`, `exactly_one_of` and `none_of` is a class expression whose `slot_conditions` constrain the instance's slots. The expressions of a class also constrain the instances of its subclasses, and of the classes that use it as a mixin. + +A slot condition needs a meaning when its slot is absent. The JSON Schema and SHACL generators read it as an SQL `CHECK` constraint reads a condition on a null value: an instance is invalid only when an expression is definitely false. + +- A condition that decides whether its slot may be absent is true or false as usual. Such a condition sets `value_presence: PRESENT` or `ABSENT`, `required: true`, a minimum or exact cardinality of at least 1, or a maximum or exact cardinality of 0. +- Any other condition is *unknown* when its slot is absent. +- `any_of`, `all_of` and `none_of` combine these as "or", "and" and "not". An unknown member doesn't make `any_of` true, nor `none_of` false. +- `exactly_one_of` holds when exactly one member is definitely true. + +```yaml +classes: + Sample: + none_of: + - slot_conditions: + status: + equals_string: retracted +``` + +A `Sample` without `status` is valid: the condition is unknown, so `none_of` isn't false. To require the slot, state it with `status: {required: true}`. + +| Expression | `{}` | `{label: A}` | `{label: B}` | +|---|---|---|---| +| `any_of: [label = A]` | valid | valid | invalid | +| `none_of: [label = A]` | valid | invalid | valid | +| `exactly_one_of: [label = A, note = B]` | invalid | valid | invalid | + +[Rules](#rules) are read differently: their preconditions require their slots, and so do their postconditions unless the rule is `open_world`. + ### Unions as ranges [any_of](https://w3id.org/linkml/any_of) can be used to express that a range must satisfy any of a set of ranges. diff --git a/packages/linkml/src/linkml/generators/common/class_expression.py b/packages/linkml/src/linkml/generators/common/class_expression.py new file mode 100644 index 0000000000..ba4bd04860 --- /dev/null +++ b/packages/linkml/src/linkml/generators/common/class_expression.py @@ -0,0 +1,150 @@ +"""Shared semantics of slot conditions in class-level boolean expressions. + +The class-level operators ``any_of``, ``all_of``, ``exactly_one_of`` and +``none_of`` combine class expressions whose slot conditions constrain slots of +an instance. The specification does not say what a condition means for an +absent slot. Generators that translate class expressions read it the same +way, as an SQL CHECK constraint (ISO/IEC 9075) reads a condition on a null +value: a condition that doesn't state presence is *unknown* for an absent +slot, and an expression is violated only when it is definitely false. + +Each expression therefore has two forms: + +* its **"not false" form**, which the instance must satisfy, and in which a + condition holds for an absent slot unless it states presence; +* its **"definitely true" form**, used under a negation, in which a condition + that doesn't state presence also requires its slot. + +``none_of`` takes its members in the opposite form, ``exactly_one_of`` counts +the members that are definitely true, and ``any_of`` / ``all_of`` keep the +form. + +>>> from linkml_runtime.linkml_model.meta import SlotDefinition +>>> value_bounds(SlotDefinition("label", equals_string="A"), definite=False) +(0, None) +>>> value_bounds(SlotDefinition("label", equals_string="A"), definite=True) +(1, None) +>>> value_bounds(SlotDefinition("label", required=True, value_presence="ABSENT"), definite=True) +(0, 0) +>>> value_bounds(SlotDefinition("tags", minimum_cardinality=2, maximum_cardinality=3), definite=False) +(2, 3) +""" + +from linkml_runtime.linkml_model.meta import PresenceEnum, SlotDefinition +from linkml_runtime.utils.schemaview import SchemaView + +_PRESENT = PresenceEnum(PresenceEnum.PRESENT) +_ABSENT = PresenceEnum(PresenceEnum.ABSENT) + + +def states_presence(condition: SlotDefinition) -> bool: + """Whether *condition* decides whether its slot may be absent. + + Such a condition is definitely true or false for an absent slot: + ``value_presence: PRESENT`` or ``ABSENT``; ``required: true``, unless + ``value_presence`` overrides it; a minimum or exact cardinality of at least + 1, which an absent slot fails; and a maximum or exact cardinality of 0, + which it satisfies. Other bounds, ``required: false`` and ``UNCOMMITTED`` + leave absence open, so adding one never changes the verdict on an absent + slot. + + >>> states_presence(SlotDefinition("label", equals_string="A")) + False + >>> states_presence(SlotDefinition("label", required=False, equals_string="A")) + False + >>> states_presence(SlotDefinition("tags", maximum_cardinality=5)) + False + >>> states_presence(SlotDefinition("tags", maximum_cardinality=0)) + True + >>> states_presence(SlotDefinition("tags", minimum_cardinality=1)) + True + """ + if condition.value_presence is not None: + if condition.value_presence in (_PRESENT, _ABSENT): + return True + elif condition.required: + return True + lower = (condition.minimum_cardinality, condition.exact_cardinality) + upper = (condition.maximum_cardinality, condition.exact_cardinality) + return any(bound is not None and bound >= 1 for bound in lower) or 0 in upper + + +def value_bounds(condition: SlotDefinition, definite: bool) -> tuple[int, int | None]: + """The least and the greatest number of values *condition* allows its slot. + + ``value_presence`` takes precedence over ``required``. In the "definitely + true" form (*definite*), a condition that doesn't state presence also + requires the slot. The greatest number is ``None`` when unbounded. + """ + lower, upper = [0], [] + if condition.value_presence is not None: + if condition.value_presence == _PRESENT: + lower.append(1) + elif condition.value_presence == _ABSENT: + upper.append(0) + elif condition.required: + lower.append(1) + if definite and not states_presence(condition): + lower.append(1) + for bound, target in ( + (condition.minimum_cardinality, lower), + (condition.exact_cardinality, lower), + (condition.maximum_cardinality, upper), + (condition.exact_cardinality, upper), + ): + if bound is not None: + target.append(int(bound)) + return max(lower), min(upper) if upper else None + + +def class_membership(sv: SchemaView, cls_name: str, target: str) -> bool | None: + """Whether an instance of the class *cls_name* is an instance of the class *target*. + + A LinkML instance belongs to its class and to that class's ancestors, by + ``is_a`` and by ``mixins``. So every instance of *cls_name* is one when + *target* is *cls_name* or an ancestor (``True``); some may be when *target* + is an ancestor of a descendant of *cls_name*, which the instance may belong + to (``None``); and none is otherwise (``False``). + + A class carries the expressions of its ancestors, which may name any of + these: a ``Person`` carries its parent's ``any_of: [{is_a: Person}, + {is_a: Organization}]``, whose first member it satisfies and whose second + it cannot. A value of ``linkml:Any`` may be an instance of any class, and + every instance is one of ``linkml:Any``. + + >>> sv = SchemaView(''' + ... id: https://example.org/membership + ... name: membership + ... classes: + ... Any: {class_uri: linkml:Any} + ... Agent: {} + ... Person: {is_a: Agent} + ... Student: {is_a: Person} + ... Organization: {is_a: Agent} + ... ''') + >>> class_membership(sv, "Student", "Person") + True + >>> print(class_membership(sv, "Agent", "Person")) + None + >>> class_membership(sv, "Person", "Organization") + False + >>> print(class_membership(sv, "Any", "Person")) + None + >>> class_membership(sv, "Person", "Any") + True + """ + if target in sv.class_ancestors(cls_name) or _is_any(sv, target): + return True + if _is_any(sv, cls_name): + return None + if any(target in sv.class_ancestors(descendant) for descendant in sv.class_descendants(cls_name)): + return None + return False + + +def _is_any(sv: SchemaView, cls_name: str) -> bool: + """Whether the class *cls_name* is ``linkml:Any``, the class of every value.""" + cls = sv.get_class(cls_name) + if cls is None or not cls.class_uri: + return False + return cls.class_uri == "linkml:Any" or sv.expand_curie(cls.class_uri) == "https://w3id.org/linkml/Any" diff --git a/packages/linkml/src/linkml/generators/jsonschemagen.py b/packages/linkml/src/linkml/generators/jsonschemagen.py index 9cc776f5e3..8086896e2b 100644 --- a/packages/linkml/src/linkml/generators/jsonschemagen.py +++ b/packages/linkml/src/linkml/generators/jsonschemagen.py @@ -14,6 +14,7 @@ from linkml.generators.common import build from linkml.generators.common.array import ArrayRangeGenerator, ArrayRepresentation from linkml.generators.common.build import RangeResult +from linkml.generators.common.class_expression import class_membership, value_bounds from linkml.generators.common.lifecycle import LifecycleMixin from linkml.generators.common.subproperty import get_subproperty_values from linkml.generators.common.type_designators import ( @@ -275,24 +276,12 @@ def add_property( self["required"].append(canonical_name) # JSON Schema does not have a very natural way to express that a property cannot be present. - # The apparent best way to do it is to use: - # { - # properties: { - # foo: ... - # }, - # not: { - # required: ['foo'] - # } - # } - # The {required: [foo]} subschema evaluates to true if the foo property is present with any - # value. Wrapping that in a `not` keyword inverts that condition. + # The apparent best way to do it is `not: {required: [foo]}`: the {required: [foo]} + # subschema evaluates to true if the foo property is present with any value, and `not` + # inverts that. Each absent property gets its own `not`, in `allOf`, since + # `not: {required: [foo, bar]}` would only forbid foo and bar together. if value_disallowed: - if "not" not in self: - self["not"] = {} - if "required" not in self["not"]: - self["not"]["required"] = [] - - self["not"]["required"].append(canonical_name) + self.setdefault("allOf", []).append({"not": {"required": [canonical_name]}}) def add_keyword(self, keyword: str, value: Any): if value is None: @@ -626,29 +615,20 @@ def handle_class(self, cls: ClassDefinition) -> None: class_subschema["allOf"] = [] class_subschema["allOf"].extend(rule_subschemas) - if cls.any_of is not None and len(cls.any_of) > 0: - class_subschema["anyOf"] = [self.get_subschema_for_anonymous_class(c, False) for c in cls.any_of] - - if cls.all_of is not None and len(cls.all_of) > 0: - if "allOf" not in class_subschema: - class_subschema["allOf"] = [] - class_subschema["allOf"].extend([self.get_subschema_for_anonymous_class(c, False) for c in cls.all_of]) - - if cls.exactly_one_of is not None and len(cls.exactly_one_of) > 0: - class_subschema["oneOf"] = [self.get_subschema_for_anonymous_class(c, False) for c in cls.exactly_one_of] - - if cls.none_of is not None and len(cls.none_of) > 0: - # properties_required=True so absent slots make their branch fail; otherwise - # properties is vacuously true and `not(anyOf)` rejects instances missing the slot. - new_not = {"anyOf": [self.get_subschema_for_anonymous_class(c, True) for c in cls.none_of]} - if "not" in class_subschema: - existing_not = class_subschema.pop("not") - if "allOf" not in class_subschema: - class_subschema["allOf"] = [] - class_subschema["allOf"].append({"not": existing_not}) - class_subschema["allOf"].append({"not": new_not}) - else: - class_subschema["not"] = new_not + # A class expression constrains every instance of its class, so a class carries the + # expressions of its ancestors and mixins, translated in its own context. + for owner in self.schemaview.class_ancestors(cls.name): + for operator in self.CLASS_EXPRESSION_OPERATORS: + members = getattr(self.schemaview.get_class(owner), operator) or [] + if members: + expression = self.get_subschema_for_class_operator(cls, operator, members, definite=False) + for keyword, value in expression.items(): + if keyword == "allOf": + class_subschema.setdefault("allOf", []).extend(value) + elif keyword not in class_subschema: + class_subschema[keyword] = value + else: + class_subschema.setdefault("allOf", []).append({keyword: value}) class_subschema = self.after_generate_class( ClassResult.model_construct(schema_=class_subschema, source=cls), self.schemaview @@ -674,6 +654,120 @@ def handle_class(self, cls: ClassDefinition) -> None: if key not in self.top_level_schema: self.top_level_schema[key] = value + CLASS_EXPRESSION_OPERATORS = ("any_of", "all_of", "exactly_one_of", "none_of") + _CLASS_OPERATOR_KEYWORDS = {"any_of": "anyOf", "all_of": "allOf", "exactly_one_of": "oneOf"} + + def get_subschema_for_class_operator( + self, cls: ClassDefinition, operator: str, members: list[AnonymousClassExpression], definite: bool + ) -> JsonSchema: + """The subschema of the class-level boolean expression *operator* over *members*, for *cls*. + + A condition that doesn't state presence is unknown for an absent slot, + and an expression is violated only when it is definitely false (see + :mod:`linkml.generators.common.class_expression`). With *definite* the + subschema accepts the instances for which the expression is definitely + true, otherwise those for which it is not false. ``none_of`` takes its + members in the opposite form, ``exactly_one_of`` counts the members + that are definitely true, and ``any_of`` / ``all_of`` keep the form. + """ + if operator == "none_of": + return JsonSchema( + {"not": {"anyOf": [self.get_subschema_for_class_expression(cls, m, not definite) for m in members]}} + ) + form = True if operator == "exactly_one_of" else definite + return JsonSchema( + { + self._CLASS_OPERATOR_KEYWORDS[operator]: [ + self.get_subschema_for_class_expression(cls, m, form) for m in members + ] + } + ) + + def get_subschema_for_class_expression( + self, cls: ClassDefinition, expr: AnonymousClassExpression, definite: bool + ) -> JsonSchema: + """The subschema of one member *expr* of a class-level boolean expression of *cls*. + + Each slot condition constrains the slot as induced for *cls*, so + ``slot_usage`` applies. Its values must satisfy its value operators + and its ``range``; the number of values must lie within + :func:`~linkml.generators.common.class_expression.value_bounds`, where + an empty list or dict counts as no value. *definite* selects the form, as in + :meth:`get_subschema_for_class_operator`. + """ + subschema = JsonSchema() + conjuncts: list[JsonSchema] = [] + for slot_name, condition in expr.slot_conditions.items(): + slot = self._class_expression_slot(cls, slot_name) or condition + if self.use_curies: + prop_name = self._curie(slot) + else: + prop_name = self.aliased_slot_name(slot) + values = self.get_subschema_for_slot(condition, omit_type=True, include_null=False) + lower, upper = value_bounds(condition, definite) + if slot.multivalued and self._inlined_as_dict(slot): + # the values of a slot inlined as a dict are the values of the dict + prop = JsonSchema({"type": "object", "additionalProperties": values}) + prop.add_keyword("minProperties", lower or None) + prop.add_keyword("maxProperties", upper) + elif slot.multivalued: + prop = JsonSchema.array_of(values, include_null=False, required=False) + prop.add_keyword("minItems", lower or None) + prop.add_keyword("maxItems", upper) + else: + prop = values + if condition.range is not None: + # the values in the representation of the slot, as a list, a dict or a single value + typed = self.get_subschema_for_slot( + SlotDefinition( + slot.name, + range=condition.range, + multivalued=slot.multivalued, + inlined=slot.inlined, + inlined_as_list=slot.inlined_as_list, + ), + include_null=False, + ) + prop = JsonSchema({"allOf": [prop, typed]}) if prop else typed + if slot.multivalued: + subschema.add_property(prop_name, prop, value_required=lower > 0) + else: + subschema.add_property(prop_name, prop, value_required=lower > 0, value_disallowed=upper == 0) + if lower > 1: + # a single-valued slot holds at most one value + conjuncts.append(JsonSchema({"not": {}})) + for operator in self.CLASS_EXPRESSION_OPERATORS: + members = getattr(expr, operator) or [] + if members: + conjuncts.append(self.get_subschema_for_class_operator(cls, operator, members, definite)) + if expr.is_a is not None: + # `is_a: ` requires an instance of the expression to be an instance of . Every + # instance of *cls* is one, or none is, unless is a descendant of *cls*: then, in + # JSON, only the instance's structure tells, so it must match the definition of . + membership = class_membership(self.schemaview, cls.name, expr.is_a) + if membership is None: + conjuncts.append(self.get_subschema_for_slot(AnonymousSlotExpression(range=expr.is_a, inlined=True))) + elif not membership: + conjuncts.append(JsonSchema({"not": {}})) + if conjuncts: + subschema.setdefault("allOf", []).extend(conjuncts) + return subschema + + def _inlined_as_dict(self, slot: SlotDefinition) -> bool: + """Whether the values of multivalued *slot* are inlined as a dict, keyed by their identifier.""" + return ( + self.schemaview.is_inlined(slot) + and not slot.inlined_as_list + and get_range_associated_slots(self.schemaview, slot.range)[0] is not None + ) + + def _class_expression_slot(self, cls: ClassDefinition, slot_name: str) -> SlotDefinition | None: + """The slot a condition of a class expression of *cls* names, as induced for *cls*, if any.""" + sv = self.schemaview + if slot_name in sv.class_slots(cls.name): + return sv.induced_slot(slot_name, cls.name) + return sv.get_slot(slot_name) + def get_subschema_for_anonymous_class( self, cls: AnonymousClassExpression, properties_required: bool = False ) -> None | JsonSchema: diff --git a/packages/linkml/src/linkml/generators/shaclgen.py b/packages/linkml/src/linkml/generators/shaclgen.py index eb6b48e9d6..f1ee237952 100644 --- a/packages/linkml/src/linkml/generators/shaclgen.py +++ b/packages/linkml/src/linkml/generators/shaclgen.py @@ -2,7 +2,7 @@ import os import string from collections.abc import Callable -from dataclasses import dataclass +from dataclasses import dataclass, fields import click from jsonasobj2 import JsonObj, as_dict @@ -11,12 +11,23 @@ from rdflib.namespace import RDF, RDFS, SH, XSD from linkml._version import __version__ +from linkml.generators.common.class_expression import class_membership, value_bounds from linkml.generators.common.subproperty import get_subproperty_values, is_uri_range from linkml.generators.shacl.shacl_data_type import ShaclDataType from linkml.generators.shacl.shacl_ifabsent_processor import ShaclIfAbsentProcessor from linkml.utils.generator import Generator, shared_arguments from linkml.utils.language_tags import LanguageTagResolver -from linkml_runtime.linkml_model.meta import ClassDefinition, ElementName, PresenceEnum +from linkml_runtime.linkml_model.meta import ( + AnonymousClassExpression, + AnonymousSlotExpression, + ClassDefinition, + ClassExpression, + Element, + ElementName, + PresenceEnum, + SlotDefinition, + SlotExpression, +) from linkml_runtime.utils.formatutils import underscore from linkml_runtime.utils.rdf_canonicalize import canonicalize_rdf_graph from linkml_runtime.utils.yamlutils import TypedNode, extended_float, extended_int, extended_str @@ -211,6 +222,8 @@ def as_graph(self) -> Graph: for pfx in self.schema.prefixes.values(): g.bind(str(pfx.prefix_prefix), pfx.prefix_reference) + self._class_expressions_added: set[tuple[URIRef, str, str]] = set() + self._class_expression_problems: dict[tuple[str, str, str], list[str]] = {} for c in sv.all_classes(imports=not self.exclude_imports).values(): def shape_pv(p, v): @@ -251,12 +264,7 @@ def shape_pv(p, v): self._add_annotations(shape_pv, c) order = 0 for s in sv.class_induced_slots(c.name): - # fixed in linkml-runtime 1.1.3 - if s.name in sv.element_by_schema_map(): - slot_uri = URIRef(sv.get_uri(s, expand=True)) - else: - pfx = sv.schema.default_prefix - slot_uri = URIRef(sv.expand_curie(f"{pfx}:{underscore(s.name)}")) + slot_uri = URIRef(self._slot_iri(s)) pnode = BNode() shape_pv(SH.property, pnode) @@ -404,21 +412,7 @@ def st_node_pv(p, v): f" require range 'string' and not '{r}'" ) - if r in all_classes: - cls_def = sv.get_class(r) - is_any = cls_def and getattr(cls_def, "class_uri", None) == "linkml:Any" - self._add_class(prop_pv, r) - if not is_any: - if sv.get_identifier_slot(r) is not None: - prop_pv(SH.nodeKind, SH.IRI) - else: - prop_pv(SH.nodeKind, SH.BlankNodeOrIRI) - elif r in sv.all_types(): - self._add_type(prop_pv, r) - elif r in sv.all_enums(): - self._add_enum(g, prop_pv, r) - else: - add_simple_data_type(prop_pv, r) + self._add_range(g, prop_pv, r) if s.equals_string: # Map equal_string and equal_string_in to sh:in self._and_equals_string(g, prop_pv, [s.equals_string]) @@ -439,13 +433,372 @@ def st_node_pv(p, v): if default_value: prop_pv(SH.defaultValue, default_value) + self._add_class_expressions(g, class_uri_with_suffix, c) + if self.emit_rules: self._add_rules(g, class_uri_with_suffix, c) + self._report_class_expression_problems() return g LINKML_ANY_URI = "https://w3id.org/linkml/Any" + # ------------------------------------------------------------------- + # Class expressions → sh:or / sh:and / sh:xone / sh:not + # ------------------------------------------------------------------- + + # The SHACL logical constraint component, taking a list, of each list operator. + _LIST_OPERATORS = {"any_of": SH["or"], "all_of": SH["and"], "exactly_one_of": SH.xone} + _CLASS_EXPRESSION_OPERATORS = ("any_of", "all_of", "exactly_one_of", "none_of") + + # The fields of an anonymous class expression that carry meaning, derived from + # the metamodel, so that a new semantic field is reported as untranslatable + # instead of being ignored. Every other field is metadata. + _CLASS_EXPRESSION_FIELDS = frozenset(f.name for f in fields(ClassExpression)) | {"is_a"} + + # Slot-condition fields translated by _slot_condition_shape: those deciding + # whether the slot is present, and those constraining its values. + _SLOT_CONDITION_PRESENCE_FIELDS = frozenset( + {"required", "value_presence", "minimum_cardinality", "maximum_cardinality", "exact_cardinality"} + ) + _SLOT_CONDITION_VALUE_FIELDS = frozenset( + {"minimum_value", "maximum_value", "pattern", "equals_string", "equals_string_in", "equals_number", "range"} + ) + _SLOT_CONDITION_FIELDS = _SLOT_CONDITION_PRESENCE_FIELDS | _SLOT_CONDITION_VALUE_FIELDS + + # Parameters a shape may have at most one value of: sh:minInclusive and + # sh:maxInclusive (SHACL §4.3), sh:in (§4.8.3), sh:datatype and sh:nodeKind + # (§4.1), and sh:pattern, as a parameter of a component with more than one + # parameter (§2.1.1). A condition that needs one of them twice gets the + # second value in an sh:and member. + _SINGLE_VALUE_PARAMETERS = frozenset( + {SH.minInclusive, SH.maxInclusive, SH["in"], SH.pattern, SH.datatype, SH.nodeKind} + ) + + # Fields on a slot condition / class expression that carry no constraint + # semantics: they never change which instances satisfy the condition, so + # they are ignored by the operator accounting below. Anything set on a + # condition that is neither here nor explicitly translated by a converter + # makes the rule untranslatable — the converters must SKIP such a rule + # rather than emit a query that silently drops a conjunct (which would + # widen the trigger or narrow the check: a mis-translation, not a skip). + # Derived from the metamodel: the metadata every ``element`` carries, minus + # anything that is a ``slot_expression`` operator. + _NON_OPERATOR_FIELDS = frozenset(f.name for f in fields(Element)) - frozenset( + f.name for f in fields(SlotExpression) + ) + + @classmethod + def _set_operator_fields( + cls, condition: SlotDefinition | AnonymousSlotExpression | AnonymousClassExpression + ) -> set[str]: + """Return the names of the constraint-bearing fields actually set on a + rule condition or class expression. + + A field counts as *set* when it is not ``None`` and not an empty + collection (SchemaView materialises unset multivalued fields as empty + lists / dicts). Scalars are never judged by truthiness, so legitimate + falsy constraints such as ``minimum_value: 0`` or + ``equals_string: ""`` still count as set. Metadata fields + (:data:`_NON_OPERATOR_FIELDS`) are excluded. + + The converters compare this set against the exact operator set they + translate and skip the rule on any mismatch, so an unrecognised (or + future-metamodel) operator can never be silently dropped. + """ + return { + name + for name, value in vars(condition).items() + if not name.startswith("_") + and name not in cls._NON_OPERATOR_FIELDS + and value is not None + and not (isinstance(value, list | dict) and not value) + } + + def _add_class_expressions(self, g: Graph, shape_uri: URIRef, cls: ClassDefinition) -> None: + """Emit the class-level boolean expressions of *cls* as SHACL logical constraints. + + Each operator is mapped to the SHACL logical constraint component with the + same semantics (`SHACL §4.6 `_): + + * ``any_of`` → ``sh:or``, ``all_of`` → ``sh:and``, ``exactly_one_of`` → + ``sh:xone``, each over a list of the member shapes; + * ``none_of`` → one ``sh:not`` per member. A shape's values of ``sh:not`` + are separate constraints that all apply (SHACL §2.1.1), so the node must + conform to none of the members. + + Every member becomes an anonymous node shape: ``is_a`` gives ``sh:class`` + for a class that some instances of *cls* belong to through a descendant, + nothing for *cls* or an ancestor, which every instance belongs to, and a + shape nothing conforms to for any other class + (:func:`~linkml.generators.common.class_expression.class_membership`); + each slot condition gives an ``sh:property`` on the path of the slot as + induced for *cls* (so ``slot_usage`` applies), and nested expressions + recurse. + + The specification doesn't say what a slot condition means for an absent + slot. As with an SQL CHECK constraint (ISO/IEC 9075), which is + satisfied unless its condition is false, a class expression is violated + only when it is definitely false, and a condition that doesn't state + presence (:func:`~linkml.generators.common.class_expression.states_presence`) is unknown + for an absent slot. + Each operator is therefore translated in its "not false" form, and, + under a negation, in its "definitely true" form, where such a condition + requires its slot. ``exactly_one_of`` holds when exactly one member is + definitely true. + + A class expression constrains every instance of its class, so the shape + of *cls* carries the expressions of its ancestors and mixins as well, as + it carries their slots. Each is translated in the context of *cls*, + where ``slot_usage`` may refine a slot it names. Classes that share a + ``class_uri`` share one shape, which carries each expression once. + + An operator whose members use anything that cannot be translated is + skipped as a whole, with a warning: dropping one member would change what + the operator admits. + """ + sv = self.schemaview + for owner in sv.class_ancestors(cls.name): + for operator in self._CLASS_EXPRESSION_OPERATORS: + members = getattr(sv.get_class(owner), operator, None) or [] + if not members or (shape_uri, owner, operator) in self._class_expressions_added: + continue + self._class_expressions_added.add((shape_uri, owner, operator)) + reason = next(filter(None, (self._untranslatable(cls, m) for m in members)), None) + if reason is not None: + classes = self._class_expression_problems.setdefault((owner, operator, reason), []) + if cls.name not in classes: + classes.append(cls.name) + continue + self._add_logical_constraint(g, shape_uri, cls, operator, members, definite=False) + + def _report_class_expression_problems(self) -> None: + """Warn once about each class expression that is not translated, naming the + class shapes it is missing from unless that is only the declaring class.""" + for (owner, operator, reason), classes in self._class_expression_problems.items(): + shapes = "" if classes == [owner] else f" (in the shapes of {', '.join(map(repr, classes))})" + logger.warning( + "Class %r: %s is not translated to SHACL, because it uses %s%s.", owner, operator, reason, shapes + ) + + def _add_logical_constraint( + self, + g: Graph, + subject: URIRef | BNode, + cls: ClassDefinition, + operator: str, + members: list[AnonymousClassExpression], + definite: bool, + ) -> None: + """Add the logical constraint for *operator* over *members* to *subject*. + + With *definite* the constraint holds when the expression is definitely + true, otherwise when it is not false. ``none_of`` gives one ``sh:not`` + per member, in the opposite form, since an expression is not false + exactly when its negation is not definitely true. ``exactly_one_of`` + counts the members that are definitely true, in either form. The + other list operators keep the form. + """ + if operator == "none_of": + for member in members: + g.add((subject, SH["not"], self._class_expression_shape(g, cls, member, not definite))) + return + predicate = self._LIST_OPERATORS[operator] + member_form = True if operator == "exactly_one_of" else definite + shapes = [self._class_expression_shape(g, cls, m, member_form) for m in members] + list_node = BNode() + Collection(g, list_node, shapes) + g.add((subject, predicate, list_node)) + + def _class_expression_shape( + self, g: Graph, cls: ClassDefinition, expr: AnonymousClassExpression, definite: bool + ) -> BNode: + """Build the anonymous node shape for one class expression *expr*, in its + "definitely true" form with *definite*, otherwise its "not false" form.""" + node = BNode() + + def node_pv(p, v): + if v is not None: + g.add((node, p, v)) + + if expr.title is not None: + node_pv(RDFS.label, Literal(expr.title, lang=self._resolve_language(expr))) + if expr.description is not None: + node_pv(RDFS.comment, Literal(expr.description, lang=self._resolve_language(expr))) + if expr.is_a is not None: + # Every instance of *cls* is an instance of ``is_a``, or none is, unless ``is_a`` is a + # descendant of *cls*: only then does the node's type decide. + membership = class_membership(self.schemaview, cls.name, expr.is_a) + if membership is None: + self._add_class(node_pv, expr.is_a) + elif not membership: + node_pv(SH["not"], BNode()) # a shape nothing conforms to + for slot_name, condition in expr.slot_conditions.items(): + node_pv(SH.property, self._slot_condition_shape(g, cls, slot_name, condition, definite)) + for operator in self._CLASS_EXPRESSION_OPERATORS: + members = getattr(expr, operator) or [] + if members: + self._add_logical_constraint(g, node, cls, operator, members, definite) + return node + + def _slot_condition_shape( + self, g: Graph, cls: ClassDefinition, slot_name: str, condition: SlotDefinition, definite: bool + ) -> BNode: + """Build the property shape for the condition on *slot_name*, in its + "definitely true" form with *definite*, otherwise its "not false" form.""" + slot = self._condition_slot(cls, slot_name) + pnode = BNode() + repeated = [] + + def prop_pv(p, v): + if v is None: + return + if p in self._SINGLE_VALUE_PARAMETERS and (pnode, p, None) in g: + repeated.append((p, v)) + else: + g.add((pnode, p, v)) + + prop_pv(SH.path, URIRef(self._slot_iri(slot))) + if condition.title is not None: + prop_pv(SH.name, Literal(condition.title, lang=self._resolve_language(condition))) + if condition.description is not None: + prop_pv(SH.description, Literal(condition.description, lang=self._resolve_language(condition))) + + lower, upper = value_bounds(condition, definite) + if lower: + prop_pv(SH.minCount, Literal(lower)) + if upper is not None: + prop_pv(SH.maxCount, Literal(upper)) + + if condition.minimum_value is not None: + prop_pv(SH.minInclusive, Literal(condition.minimum_value)) + if condition.maximum_value is not None: + prop_pv(SH.maxInclusive, Literal(condition.maximum_value)) + if condition.pattern is not None: + prop_pv(SH.pattern, Literal(condition.pattern)) + value_range = condition.range or slot.range + for values in ( + [condition.equals_string] if condition.equals_string is not None else [], + condition.equals_string_in, + ): + if values: + in_node = BNode() + Collection(g, in_node, self._string_value_terms(value_range, values)) + prop_pv(SH["in"], in_node) + if condition.equals_number is not None: + # A value comparison, unlike the slot loop's sh:hasValue: 5 matches 5.0, + # and like every other value constraint in a condition it holds when the + # slot is absent. + prop_pv(SH.minInclusive, Literal(condition.equals_number)) + prop_pv(SH.maxInclusive, Literal(condition.equals_number)) + if condition.range is not None: + self._add_range(g, prop_pv, condition.range) + if repeated: + # Each repeated parameter in a member shape of its own: all of them hold + # for every value, as they would on the property shape itself. + members = [] + for p, v in repeated: + member = BNode() + g.add((member, p, v)) + members.append(member) + and_node = BNode() + Collection(g, and_node, members) + g.add((pnode, SH["and"], and_node)) + return pnode + + def _string_value_terms(self, r: ElementName | None, values: list[str]) -> list[URIRef | Literal]: + """The RDF terms of the ``equals_string`` / ``equals_string_in`` *values* of a slot with range *r*. + + Permissible values of an enum are rendered as :meth:`_add_enum` renders them, as + the IRI of their ``meaning`` where they have one; anything else is a plain literal. + """ + sv = self.schemaview + if r in sv.all_enums(): + pvs = sv.get_enum(r).permissible_values + return [ + URIRef(sv.expand_curie(pvs[v].meaning)) if v in pvs and pvs[v].meaning else Literal(v) for v in values + ] + return [Literal(v) for v in values] + + def _condition_slot(self, cls: ClassDefinition, slot_name: str) -> SlotDefinition | None: + """The slot a condition of *cls* names, as induced for *cls*, or ``None`` if there is none.""" + try: + return self.schemaview.induced_slot(slot_name, cls.name) + except ValueError: + return None + + def _type_uri(self, r: ElementName | None) -> str | None: + """The expanded datatype IRI of type range *r*, or ``None`` when *r* is not a type. + + Resolved through the induced type, so a type derived with ``typeof`` + inherits the ``uri`` of its ancestor. A built-in type name in a schema + that does not import ``linkml:types`` resolves as the main slot loop + resolves it (:class:`ShaclDataType`). + """ + sv = self.schemaview + if r in sv.all_types(): + return sv.get_uri(sv.induced_type(r), expand=True) + builtin = next((t for t in ShaclDataType if t.linkml_type == r), None) + return str(builtin.uri_ref) if builtin is not None else None + + def _is_string_range(self, r: ElementName | None) -> bool: + """Whether a slot with range *r* holds strings, which ``equals_string`` compares against. + + True for an enum, whose permissible values are rendered as their + ``meaning`` IRI or as a plain literal (as :meth:`_add_enum` renders + them); for a type whose datatype is ``xsd:string``, whose values are + plain literals; and for no range at all, whose values are untyped and + compared as strings, as the JSON Schema generator compares them. A + type with any other datatype, including one derived from ``string`` + (``xsd:anyURI``, ``xsd:token``, ...), holds typed literals or IRIs + that a string literal does not match. + """ + if r is None or r in self.schemaview.all_enums(): + return True + return self._type_uri(r) == str(XSD.string) + + def _untranslatable(self, cls: ClassDefinition, expr: AnonymousClassExpression) -> str | None: + """Return what in class expression *expr* of *cls* cannot be translated, or ``None``.""" + sv = self.schemaview + unknown = self._set_operator_fields(expr) - self._CLASS_EXPRESSION_FIELDS + if unknown: + return f"'{sorted(unknown)[0]}'" + if expr.is_a is not None and expr.is_a not in sv.all_classes(): + return f"is_a '{expr.is_a}', which is not a class" + for slot_name, condition in expr.slot_conditions.items(): + unknown = self._set_operator_fields(condition) - self._SLOT_CONDITION_FIELDS + if unknown: + return f"'{sorted(unknown)[0]}' in the condition on slot '{slot_name}'" + slot = self._condition_slot(cls, slot_name) + if slot is None: + return f"a condition on '{slot_name}', which is not a slot" + if slot.identifier: + # An identifier is the node's IRI, not a property arc. + return f"a condition on the identifier slot '{slot_name}'" + if condition.range is not None and not self._is_known_range(condition.range): + return f"the unknown range '{condition.range}' in the condition on slot '{slot_name}'" + value_range = condition.range or slot.range + if (condition.equals_string is not None or condition.equals_string_in) and not self._is_string_range( + value_range + ): + return f"equals_string on slot '{slot_name}', whose range '{value_range}' does not hold strings" + for operator in self._CLASS_EXPRESSION_OPERATORS: + for member in getattr(expr, operator) or []: + reason = self._untranslatable(cls, member) + if reason is not None: + return reason + return None + + def _is_known_range(self, r: ElementName) -> bool: + """Whether *r* names a class, type or enum of the schema, or a built-in type.""" + sv = self.schemaview + return ( + r in sv.all_classes() + or r in sv.all_types() + or r in sv.all_enums() + or any(datatype.linkml_type == r for datatype in ShaclDataType) + ) + # ------------------------------------------------------------------- # Rules → sh:sparql # ------------------------------------------------------------------- @@ -705,6 +1058,37 @@ def _add_class(self, func: Callable, r: ElementName) -> None: range_ref += self.suffix func(SH["node"], URIRef(range_ref)) + def _slot_iri(self, slot: SlotDefinition) -> str: + """The full IRI of *slot*, exactly as ``sh:path`` in the main slot loop renders it. + + An induced slot carries its ``slot_usage`` overrides, so an overridden + ``slot_uri`` yields the same IRI as ``sh:path``; otherwise the query + would use a property the data never uses and never fire. + """ + sv = self.schemaview + if slot.name in sv.element_by_schema_map(): + return sv.get_uri(slot, expand=True) + return sv.expand_curie(f"{sv.schema.default_prefix}:{underscore(slot.name)}") + + def _add_range(self, g: Graph, func: Callable, r: ElementName) -> None: + """Add the value-type constraint for range *r*: a class, type, enum or built-in datatype.""" + sv = self.schemaview + if r in sv.all_classes(): + cls_def = sv.get_class(r) + is_any = cls_def and getattr(cls_def, "class_uri", None) == "linkml:Any" + self._add_class(func, r) + if not is_any: + if sv.get_identifier_slot(r) is not None: + func(SH.nodeKind, SH.IRI) + else: + func(SH.nodeKind, SH.BlankNodeOrIRI) + elif r in sv.all_types(): + self._add_type(func, r) + elif r in sv.all_enums(): + self._add_enum(g, func, r) + else: + add_simple_data_type(func, r) + def _add_enum(self, g: Graph, func: Callable, r: ElementName) -> None: sv = self.schemaview enum = sv.get_enum(r) diff --git a/tests/linkml/test_compliance/test_boolean_slot_compliance.py b/tests/linkml/test_compliance/test_boolean_slot_compliance.py index fd900bf938..dc04f6f2f8 100644 --- a/tests/linkml/test_compliance/test_boolean_slot_compliance.py +++ b/tests/linkml/test_compliance/test_boolean_slot_compliance.py @@ -505,7 +505,7 @@ def test_class_any_of(framework, data_name, s1value, s2value, is_valid): core_elements=["any_of", "ClassDefinition"], ) expected_behavior = ValidationBehavior.IMPLEMENTS - if framework not in [OWL]: + if framework not in [OWL, SHACL]: # TODO: rdflib transformer has issues around ranges expected_behavior = ValidationBehavior.INCOMPLETE # TODO: rdflib transformer has issues around ranges @@ -632,8 +632,22 @@ def test_class_any_of_with_required(framework, nest, op, name, family_name, give core_elements=[op, "ClassDefinition"], ) expected_behavior = ValidationBehavior.IMPLEMENTS - if framework not in [JSON_SCHEMA]: + if framework not in [JSON_SCHEMA, SHACL]: expected_behavior = ValidationBehavior.INCOMPLETE + elif framework == SHACL and 5 in (name, family_name, given_name): + # SHACL validation makes its instances through python dataclasses, which coerce + # the integer to a string, so the range violation never reaches the shapes. A + # row can then only be detected through the operator itself. + present = [value is not None for value in (name, family_name, given_name)] + members = [present[0], present[1] and present[2]] + operator_holds = { + "any_of": any(members), + "all_of": all(members), + "exactly_one_of": sum(members) == 1, + "none_of": not any(members), + }[op] + if operator_holds: + expected_behavior = ValidationBehavior.INCOMPLETE data = {SLOT_S1: name, SLOT_S2: family_name, SLOT_S3: given_name} if nest: @@ -1435,8 +1449,8 @@ def test_class_boolean_with_expressions( """ if (s1_range == CLASS_ANY or s2_range == CLASS_ANY) and framework not in [PYDANTIC]: pytest.skip("Class Any not supported in this test") - if framework == SHACL: - pytest.skip("shaclgen does not support boolean expressions yet") + if framework == SHACL and CLASS_D in (s1_range, s2_range): + pytest.skip("the harness writes a reference as an untyped node, which the slot's sh:class rejects") if framework == PANDERA_POLARS_CLASS: pytest.skip("panderagen does not support boolean expressions yet") if s1_range.endswith("*"): diff --git a/tests/linkml/test_generators/test_jsonschemagen.py b/tests/linkml/test_generators/test_jsonschemagen.py index 3e83ef242d..785da1517a 100644 --- a/tests/linkml/test_generators/test_jsonschemagen.py +++ b/tests/linkml/test_generators/test_jsonschemagen.py @@ -2079,3 +2079,246 @@ def test_type_without_base_defaults_to_string(tmp_path): generated = json.loads(JsonSchemaGenerator(str(schema_file)).serialize()) assert generated["$defs"]["MyClass"]["properties"]["value"]["type"] == ["string", "null"] + + +_CLASS_EXPRESSION_INHERITANCE_SCHEMA = """ +id: https://example.org/class-expressions +name: class_expressions +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/class-expressions/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + a: {} + b: {} +classes: + Marker: + mixin: true + none_of: + - slot_conditions: + a: + equals_string: forbidden + Parent: + slots: [a, b] + any_of: + - slot_conditions: + a: + required: true + - slot_conditions: + b: + required: true + Child: + is_a: Parent + mixins: [Marker] + GrandChild: + is_a: Child +""" + + +@pytest.mark.parametrize("target_class", ["Parent", "Child", "GrandChild"]) +@pytest.mark.parametrize( + "instance,valid_for_parent,valid_for_marker_users", + [ + pytest.param({"a": "1"}, True, True, id="a"), + pytest.param({"b": "2"}, True, True, id="b"), + pytest.param({}, False, False, id="neither"), + pytest.param({"a": "forbidden"}, True, False, id="forbidden-by-mixin"), + ], +) +def test_class_expressions_apply_to_subclasses(target_class, instance, valid_for_parent, valid_for_marker_users): + """A class expression constrains every instance of its class, so the + definitions of its subclasses, and of the classes using a mixin, carry it.""" + json_schema = json.loads( + JsonSchemaGenerator(_CLASS_EXPRESSION_INHERITANCE_SCHEMA, top_class=target_class).serialize() + ) + expected = valid_for_parent if target_class == "Parent" else valid_for_marker_users + assert jsonschema.Draft7Validator(json_schema).is_valid(instance) == expected + + +_TWO_ABSENT_RULE_SCHEMA = """ +id: https://example.org/absent +name: absent +prefixes: + linkml: https://w3id.org/linkml/ +imports: + - linkml:types +default_range: string +classes: + Thing: + attributes: + trigger: {} + a: {} + b: {} + rules: + - preconditions: + slot_conditions: + trigger: + value_presence: PRESENT + postconditions: + slot_conditions: + a: + value_presence: ABSENT + b: + value_presence: ABSENT +""" + + +@pytest.mark.parametrize( + "instance,valid", + [ + pytest.param({"trigger": "t"}, True, id="neither"), + pytest.param({"trigger": "t", "a": "x"}, False, id="one"), + pytest.param({"trigger": "t", "a": "x", "b": "y"}, False, id="both"), + pytest.param({"a": "x", "b": "y"}, True, id="not-triggered"), + ], +) +def test_several_absent_slots_each_must_be_absent(instance, valid): + """value_presence: ABSENT on several slots of one expression requires each + to be absent, not merely that they aren't all present.""" + json_schema = json.loads(JsonSchemaGenerator(_TWO_ABSENT_RULE_SCHEMA, top_class="Thing").serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid(instance) == valid + + +@pytest.mark.parametrize("tags,valid", [(["A"], True), (["A", "B"], False), ([], True)]) +def test_class_expression_condition_uses_the_induced_slot(tags, valid): + """A condition constrains the slot as induced for the class, so a + slot_usage that makes it multivalued applies the condition to every value.""" + schema = """ +id: https://example.org/induced +name: induced +prefixes: + linkml: https://w3id.org/linkml/ +imports: + - linkml:types +default_range: string +slots: + tags: {} +classes: + Thing: + slots: [tags] + slot_usage: + tags: + multivalued: true + all_of: + - slot_conditions: + tags: + equals_string: A +""" + json_schema = json.loads(JsonSchemaGenerator(schema, top_class="Thing").serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid({"tags": tags}) == valid + + +_DICT_INLINED_CONDITION_SCHEMA = """ +id: https://example.org/dict-inlined +name: dict_inlined +prefixes: + linkml: https://w3id.org/linkml/ +imports: + - linkml:types +default_range: string +classes: + Resource: + attributes: + id: {identifier: true} + code: {range: integer} + SpecialResource: + is_a: Resource + slot_usage: + code: {required: true} + Document: + attributes: + licenses: + range: Resource + multivalued: true + inlined: true + all_of: + - slot_conditions: + licenses: CONDITION +""" + + +@pytest.mark.parametrize( + "condition,licenses,valid", + [ + pytest.param({"minimum_cardinality": 1}, {"r1": {"code": 1}}, True, id="min-met"), + pytest.param({"minimum_cardinality": 1}, {}, False, id="min-empty-dict"), + pytest.param({"maximum_cardinality": 1}, {"r1": {}, "r2": {}}, False, id="max-exceeded"), + pytest.param({"range": "SpecialResource"}, {"r1": {"code": 1}}, True, id="range-met"), + pytest.param({"range": "SpecialResource"}, {"r1": {}}, False, id="range-violated"), + ], +) +def test_class_expression_condition_on_a_slot_inlined_as_dict(condition, licenses, valid): + """A condition on a slot inlined as a dict counts and constrains the values + of the dict, which is how the values of such a slot appear in JSON.""" + schema = _DICT_INLINED_CONDITION_SCHEMA.replace("CONDITION", json.dumps(condition)) + json_schema = json.loads(JsonSchemaGenerator(schema, top_class="Document").serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid({"licenses": licenses}) == valid + + +_SUBCLASS_UNION_SCHEMA = """ +id: https://example.org/union +name: union +prefixes: + linkml: https://w3id.org/linkml/ +imports: + - linkml:types +default_range: string +classes: + Agent: + attributes: + agent_id: {} + OPERATOR: + - is_a: Person + - is_a: Organization + Person: + is_a: Agent + attributes: + name: {required: true} + Student: + is_a: Person + attributes: + school: {} + Organization: + is_a: Agent + attributes: + id: {identifier: true} + org_name: {} +""" + + +@pytest.mark.parametrize("operator", ["any_of", "exactly_one_of"]) +@pytest.mark.parametrize( + "target_class,instance,valid", + [ + pytest.param("Person", {"name": "n"}, True, id="person"), + pytest.param("Student", {"name": "n", "school": "s"}, True, id="student"), + pytest.param("Organization", {"id": "o", "org_name": "o"}, True, id="organization"), + pytest.param("Agent", {"agent_id": "1"}, False, id="agent-of-neither-structure"), + ], +) +def test_class_expression_is_a_inherited_by_the_classes_it_names(operator, target_class, instance, valid): + """A class carries its parent's expression over its own subclasses, such as + ``any_of: [{is_a: Person}, {is_a: Organization}]``. An instance of Person satisfies + the first member and cannot satisfy the second, so its definition doesn't refer to + itself. An Agent is a Person in JSON when its structure matches Person's definition.""" + schema = _SUBCLASS_UNION_SCHEMA.replace("OPERATOR", operator) + json_schema = json.loads(JsonSchemaGenerator(schema, top_class=target_class).serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid(instance) == valid + + +@pytest.mark.parametrize( + "target_class,instance,valid", + [ + pytest.param("Person", {"name": "n"}, True, id="person-is-no-organization"), + pytest.param("Organization", {"id": "o"}, False, id="organization"), + pytest.param("Agent", {"agent_id": "1"}, True, id="agent-of-another-structure"), + ], +) +def test_class_expression_is_a_of_a_class_no_instance_belongs_to(target_class, instance, valid): + """``none_of: [{is_a: Organization}]`` holds for every Person, since no Person is an Organization.""" + schema = _SUBCLASS_UNION_SCHEMA.replace("OPERATOR", "none_of").replace(" - is_a: Person\n", "") + json_schema = json.loads(JsonSchemaGenerator(schema, top_class=target_class).serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid(instance) == valid diff --git a/tests/linkml/test_generators/test_shaclgen.py b/tests/linkml/test_generators/test_shaclgen.py index cf6764ad6d..020f756d71 100644 --- a/tests/linkml/test_generators/test_shaclgen.py +++ b/tests/linkml/test_generators/test_shaclgen.py @@ -1,9 +1,12 @@ +import json +import logging from collections import Counter from typing import Any import pytest import rdflib -from rdflib import RDF, RDFS, SH, Literal, URIRef +import yaml +from rdflib import RDF, RDFS, SH, XSD, Literal, URIRef from rdflib.collection import Collection from linkml.generators.shacl.shacl_data_type import ShaclDataType @@ -2972,3 +2975,1529 @@ def conforms(data_ttl: str) -> tuple[bool, str]: for value in ('"x1"', '"za"'): ok, _ = conforms(f"{prefixes}\nex:t2 a ex:SlotPatternWithAnyOf ; ex:tag {value} .") assert not ok, f"tag {value} violates the slot pattern or matches no branch and must be rejected" + + +# Class expressions → sh:or / sh:and / sh:xone / sh:not +# --------------------------------------------------------------------------- + +EX_CE = rdflib.Namespace("https://example.org/class-expressions/") + +_CLASS_EXPRESSION_HEADER = """ +id: https://example.org/class-expressions +name: class_expressions +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/class-expressions/ +imports: + - linkml:types +default_prefix: ex +default_range: string +""" + +_REFERENCE_SYSTEM_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + codeEPSG: + range: integer + coordinateSystemName: {{}} +classes: + ReferenceSystem: + class_uri: ex:ReferenceSystem + slots: [codeEPSG, coordinateSystemName] + {operator}: + - slot_conditions: + codeEPSG: + required: true + - slot_conditions: + coordinateSystemName: + required: true +""" +) + +_LOGICAL_PREDICATES = (SH["or"], SH["and"], SH.xone, SH["not"]) + + +def _list_members(g, shape, predicate): + """Members of the single SHACL list that *shape* has for *predicate*.""" + lists = list(g.objects(shape, predicate)) + assert len(lists) == 1, f"expected one {predicate} list on {shape}, got {len(lists)}" + return list(Collection(g, lists[0])) + + +def _condition(g, member, path): + """The property shape for *path* inside the member shape *member*.""" + shapes = [p for p in g.objects(member, SH.property) if (p, SH.path, path) in g] + assert len(shapes) == 1, f"expected one condition on {path}, got {len(shapes)}" + return shapes[0] + + +def _conforms(shacl_ttl: str, data_ttl: str) -> bool: + """Validate *data_ttl*; meta_shacl makes pyshacl fail on an ill-formed shapes graph.""" + import pyshacl + + conforms, _, _ = pyshacl.validate( + data_graph=data_ttl, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + meta_shacl=True, + ) + return conforms + + +@pytest.mark.parametrize( + "operator,predicate", + [("any_of", SH["or"]), ("all_of", SH["and"]), ("exactly_one_of", SH.xone)], +) +def test_class_expression_list_operator_generates_logical_constraint(operator, predicate): + """any_of, all_of and exactly_one_of become sh:or, sh:and and sh:xone over the member shapes.""" + g = _parse_shacl(_REFERENCE_SYSTEM_SCHEMA.format(operator=operator)) + + members = _list_members(g, EX_CE.ReferenceSystem, predicate) + assert len(members) == 2 + for member, path in zip(members, (EX_CE.codeEPSG, EX_CE.coordinateSystemName)): + condition = _condition(g, member, path) + assert (condition, SH.minCount, Literal(1)) in g + assert (member, SH.targetClass, None) not in g + others = set(_LOGICAL_PREDICATES) - {predicate} + assert not any((EX_CE.ReferenceSystem, p, None) in g for p in others) + + +def test_class_expression_none_of_generates_one_sh_not_per_member(): + """none_of becomes one sh:not per member; the negated constraints all apply (SHACL §2.1.1).""" + g = _parse_shacl(_REFERENCE_SYSTEM_SCHEMA.format(operator="none_of")) + + negated = list(g.objects(EX_CE.ReferenceSystem, SH["not"])) + assert len(negated) == 2 + paths = {path for member in negated for p in g.objects(member, SH.property) for path in g.objects(p, SH.path)} + assert paths == {EX_CE.codeEPSG, EX_CE.coordinateSystemName} + + +@pytest.mark.parametrize( + "operator,properties,expected", + [ + ("any_of", 'ex:codeEPSG 4326 ; ex:coordinateSystemName "WGS 84"', True), + ("any_of", "ex:codeEPSG 4326", True), + ("any_of", "", False), + ("exactly_one_of", "ex:codeEPSG 4326", True), + ("exactly_one_of", 'ex:codeEPSG 4326 ; ex:coordinateSystemName "WGS 84"', False), + ("exactly_one_of", "", False), + ("all_of", 'ex:codeEPSG 4326 ; ex:coordinateSystemName "WGS 84"', True), + ("all_of", "ex:codeEPSG 4326", False), + ("none_of", "", True), + ("none_of", "ex:codeEPSG 4326", False), + ], +) +def test_class_expression_pyshacl_end_to_end(operator, properties, expected): + """End-to-end: each operator admits exactly the instances the metamodel says it holds for.""" + shacl_ttl = ShaclGenerator(_REFERENCE_SYSTEM_SCHEMA.format(operator=operator), mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:rs a ex:ReferenceSystem {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +_FORMAT_PROFILES_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + fileFormat: {} + formatType: {} + version: {} + hasChannel: + range: Channel + multivalued: true + inlined: true +classes: + Channel: + class_uri: ex:Channel + Format: + class_uri: ex:Format + slots: [fileFormat, formatType, version, hasChannel] + any_of: + - description: A single-channel trace. + slot_conditions: + fileFormat: + equals_string_in: [OSI, TXTH] + formatType: + required: true + version: + required: true + - description: A multi-channel container. + slot_conditions: + fileFormat: + required: true + equals_string: MCAP + hasChannel: + required: true +""" +) + + +@pytest.mark.parametrize( + "properties,expected", + [ + ('ex:fileFormat "OSI" ; ex:formatType "SensorView" ; ex:version "3.7.0"', True), + ('ex:fileFormat "MCAP" ; ex:hasChannel ex:ch', True), + ('ex:fileFormat "MCAP" ; ex:formatType "SensorView" ; ex:version "3.7.0"', False), + ('ex:fileFormat "OSI" ; ex:formatType "SensorView" ; ex:hasChannel ex:ch', False), + ], +) +def test_class_expression_any_of_profiles_pyshacl_end_to_end(properties, expected): + """End-to-end: a class-level any_of selects between two complete profiles of a class.""" + shacl_ttl = ShaclGenerator(_FORMAT_PROFILES_SCHEMA, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:ch a ex:Channel . + ex:f a ex:Format ; {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +def test_class_expression_member_metadata(): + """A member's title and description annotate its shape, as they do for a class's NodeShape.""" + g = _parse_shacl(_FORMAT_PROFILES_SCHEMA) + + comments = {str(c) for m in _list_members(g, EX_CE.Format, SH["or"]) for c in g.objects(m, RDFS.comment)} + assert comments == {"A single-channel trace.", "A multi-channel container."} + + +_CONDITIONS_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +enums: + ColourEnum: + permissible_values: + red: {} + blue: {} +slots: + label: {} + size: + range: integer + count: + multivalued: true + exact: + multivalued: true + colour: {} + target: {} + note: {} +classes: + Target: + class_uri: ex:Target + Thing: + class_uri: ex:Thing + slots: [label, size, count, exact, colour, target, note] + all_of: + - title: every condition + slot_conditions: + label: + description: Starts upper case. + required: true + pattern: "^[A-Z]" + equals_string_in: [Alpha, Beta] + size: + minimum_value: 1 + maximum_value: 10 + equals_number: 5 + count: + required: true + minimum_cardinality: 2 + maximum_cardinality: 4 + exact: + exact_cardinality: 3 + colour: + range: ColourEnum + target: + value_presence: PRESENT + range: Target + note: + value_presence: ABSENT +""" +) + + +def test_class_expression_slot_condition_fields(): + """Each supported slot-condition field maps to the SHACL constraint the slot loop uses for it.""" + g = _parse_shacl(_CONDITIONS_SCHEMA) + (member,) = _list_members(g, EX_CE.Thing, SH["and"]) + assert (member, RDFS.label, Literal("every condition")) in g + + def values(path, predicate): + return set(g.objects(_condition(g, member, path), predicate)) + + def in_list(path): + (node,) = values(path, SH["in"]) + return list(Collection(g, node)) + + assert values(EX_CE.label, SH.minCount) == {Literal(1)} + assert values(EX_CE.label, SH.pattern) == {Literal("^[A-Z]")} + assert values(EX_CE.label, SH.description) == {Literal("Starts upper case.")} + assert in_list(EX_CE.label) == [Literal("Alpha"), Literal("Beta")] + + # equals_number is a value comparison; SHACL allows one sh:minInclusive and one + # sh:maxInclusive per shape, so next to the bounds it moves into an sh:and member + assert values(EX_CE.size, SH.minInclusive) == {Literal(1)} + assert values(EX_CE.size, SH.maxInclusive) == {Literal(10)} + (and_node,) = values(EX_CE.size, SH["and"]) + repeated = {(p, o) for m in Collection(g, and_node) for p, o in g.predicate_objects(m)} + assert repeated == {(SH.minInclusive, Literal(5)), (SH.maxInclusive, Literal(5))} + assert values(EX_CE.size, SH["in"]) == set() + assert values(EX_CE.size, SH.minCount) == set() + assert values(EX_CE.size, SH.hasValue) == set() + + # required and a cardinality give one sh:minCount, the stricter of the two + assert values(EX_CE["count"], SH.minCount) == {Literal(2)} + assert values(EX_CE["count"], SH.maxCount) == {Literal(4)} + assert values(EX_CE.exact, SH.minCount) == {Literal(3)} + assert values(EX_CE.exact, SH.maxCount) == {Literal(3)} + + assert in_list(EX_CE.colour) == [Literal("red"), Literal("blue")] + + assert values(EX_CE.target, SH.minCount) == {Literal(1)} + assert values(EX_CE.target, SH["class"]) == {EX_CE.Target} + assert values(EX_CE.target, SH.nodeKind) == {SH.BlankNodeOrIRI} + + assert values(EX_CE.note, SH.maxCount) == {Literal(0)} + assert values(EX_CE.note, SH.minCount) == set() + + +@pytest.mark.parametrize( + "condition,properties,expected", + [ + # outside none_of a value constraint says nothing about presence + ("any_of", "", True), + ("any_of", 'ex:label "b"', False), + # inside none_of it requires the slot, so an absent slot is not rejected + ("none_of", "", True), + ("none_of", 'ex:label "A"', False), + ("none_of", 'ex:label "B"', True), + ], +) +def test_class_expression_presence_semantics(condition, properties, expected): + """A condition holds vacuously for an absent slot, except under none_of, as in the JSON Schema generator.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + f""" +slots: + label: {{}} +classes: + Thing: + class_uri: ex:Thing + slots: [label] + {condition}: + - slot_conditions: + label: + {"pattern: '^[A-Z]'" if condition == "any_of" else "equals_string: A"} +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +def test_class_expression_nested_and_is_a(): + """Nested expressions recurse into the member shape; is_a gives sh:class.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + a: {} + b: {} + c: {} +classes: + Marker: + class_uri: ex:Marker + mixin: true + Thing: + class_uri: ex:Thing + slots: [a, b, c] + any_of: + - all_of: + - slot_conditions: + a: + required: true + - slot_conditions: + b: + required: true + - is_a: Marker + slot_conditions: + c: + required: true + MarkedThing: + is_a: Thing + mixins: [Marker] +""" + ) + # a Thing may be a Marker, since MarkedThing is one, so Thing's shape checks the type; + # open shapes: the instance is also a Marker, whose own shape declares no slots + shacl_ttl = ShaclGenerator(schema, mergeimports=False, closed=False).serialize() + g = rdflib.Graph().parse(data=shacl_ttl) + first, second = _list_members(g, EX_CE.Thing, SH["or"]) + assert len(_list_members(g, first, SH["and"])) == 2 + assert (second, SH["class"], EX_CE.Marker) in g + + prefix = "@prefix ex: ." + assert _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing ; ex:a "1" ; ex:b "2" .') + assert not _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing ; ex:a "1" .') + assert _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing, ex:Marker ; ex:c "3" .') + assert not _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing ; ex:c "3" .') + + +_SUBCLASS_UNION_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +classes: + Agent: + attributes: + agent_id: {} + any_of: + - is_a: Person + - is_a: Organization + Person: + is_a: Agent + attributes: + name: {} + Student: + is_a: Person + attributes: + school: {} + Organization: + is_a: Agent + attributes: + org_name: {} +""" +) + + +def test_class_expression_is_a_inherited_by_the_classes_it_names(): + """A class carries its parent's ``any_of: [{is_a: Person}, {is_a: Organization}]``. + Every Person is a Person and none is an Organization, so Person's shape needs no + ``sh:class`` for the first member and nothing conforms to the second; only Agent's + shape, whose instances may be either, checks the node's type.""" + shacl_ttl = ShaclGenerator(_SUBCLASS_UNION_SCHEMA, mergeimports=False, closed=False).serialize() + g = rdflib.Graph().parse(data=shacl_ttl) + person_member, organization_member = _list_members(g, EX_CE.Person, SH["or"]) + assert (person_member, SH["class"], None) not in g + assert (organization_member, SH["not"], None) in g + agent_members = _list_members(g, EX_CE.Agent, SH["or"]) + assert {g.value(m, SH["class"]) for m in agent_members} == {EX_CE.Person, EX_CE.Organization} + + prefix = "@prefix ex: ." + assert _conforms(shacl_ttl, f'{prefix} ex:p a ex:Person ; ex:name "n" .') + # a Student is a Person without carrying the type: the shape doesn't need the subclass triple + assert _conforms(shacl_ttl, f'{prefix} ex:s a ex:Student ; ex:school "x" .') + assert _conforms(shacl_ttl, f'{prefix} ex:o a ex:Organization ; ex:org_name "o" .') + assert not _conforms(shacl_ttl, f'{prefix} ex:a a ex:Agent ; ex:agent_id "1" .') + assert _conforms(shacl_ttl, f'{prefix} ex:a a ex:Agent, ex:Person ; ex:agent_id "1" .') + + +def test_class_expression_is_a_of_a_class_no_instance_belongs_to(): + """``none_of: [{is_a: Organization}]`` holds for every Person, since no Person is an Organization.""" + schema = _SUBCLASS_UNION_SCHEMA.replace("any_of:\n - is_a: Person\n", "none_of:\n") + shacl_ttl = ShaclGenerator(schema, mergeimports=False, closed=False).serialize() + prefix = "@prefix ex: ." + assert _conforms(shacl_ttl, f'{prefix} ex:p a ex:Person ; ex:name "n" .') + assert not _conforms(shacl_ttl, f'{prefix} ex:o a ex:Organization ; ex:org_name "o" .') + assert _conforms(shacl_ttl, f'{prefix} ex:a a ex:Agent ; ex:agent_id "1" .') + + +_INHERITED_CLASS_EXPRESSION_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + a: {} + b: {} +classes: + Marker: + class_uri: ex:Marker + mixin: true + none_of: + - slot_conditions: + a: + equals_string: forbidden + Parent: + class_uri: ex:Parent + slots: [a, b] + any_of: + - slot_conditions: + a: + required: true + - slot_conditions: + b: + required: true + Child: + class_uri: ex:Child + is_a: Parent + mixins: [Marker] + GrandChild: + class_uri: ex:GrandChild + is_a: Child +""" +) +_CE_PREFIXES = ( + "@prefix ex: .\n@prefix rdfs: .\n" +) + + +@pytest.mark.parametrize("subclass_axioms", [False, True], ids=["typed-only", "with-rdfs-subClassOf"]) +@pytest.mark.parametrize("class_name", ["Parent", "Child", "GrandChild"]) +@pytest.mark.parametrize( + "values,conforms", + [ + pytest.param(' ; ex:a "1"', True, id="a"), + pytest.param(' ; ex:b "2"', True, id="b"), + pytest.param("", False, id="neither"), + ], +) +def test_class_expression_applies_to_subclass_instances(subclass_axioms, class_name, values, conforms): + """A class expression constrains every instance of its class, so the shapes + of its subclasses carry it too. A node typed only with a subclass, as + rdflib_dumper and the JSON-LD context produce it, is checked without an + rdfs:subClassOf triple in the data; with one, the verdict is the same.""" + shacl_ttl = ShaclGenerator(_INHERITED_CLASS_EXPRESSION_SCHEMA, mergeimports=False, closed=False).serialize() + axioms = ( + "ex:Child rdfs:subClassOf ex:Parent . ex:GrandChild rdfs:subClassOf ex:Child .\n" if subclass_axioms else "" + ) + assert _conforms(shacl_ttl, f"{_CE_PREFIXES}{axioms}ex:x a ex:{class_name}{values} .") == conforms + + +@pytest.mark.parametrize("class_name,conforms", [("Parent", True), ("Child", False), ("GrandChild", False)]) +def test_class_expression_of_a_mixin_applies_to_the_classes_using_it(class_name, conforms): + """A mixin's class expression constrains the classes that use the mixin and + their subclasses, and no other class.""" + shacl_ttl = ShaclGenerator(_INHERITED_CLASS_EXPRESSION_SCHEMA, mergeimports=False, closed=False).serialize() + assert _conforms(shacl_ttl, f'{_CE_PREFIXES}ex:x a ex:{class_name} ; ex:a "forbidden" .') == conforms + + +def test_inherited_class_expression_follows_the_subclass_slot_usage(): + """An inherited class expression is translated in the subclass's context, so + its conditions use the subclass's slot_usage, as the subclass's own + property shapes do.""" + schema = yaml.safe_load(_INHERITED_CLASS_EXPRESSION_SCHEMA) + schema["classes"]["Child"]["slot_usage"] = {"a": {"slot_uri": "ex:childA"}} + g = rdflib.Graph().parse(data=ShaclGenerator(json.dumps(schema), mergeimports=False).serialize()) + for shape, path in ((EX_CE.Parent, EX_CE.a), (EX_CE.Child, EX_CE.childA), (EX_CE.GrandChild, EX_CE.childA)): + (or_list,) = g.objects(shape, SH["or"]) + paths = { + p + for member in Collection(g, or_list) + for prop in g.objects(member, SH.property) + for p in g.objects(prop, SH.path) + } + assert paths == {path, EX_CE.b}, (shape, paths) + + +def test_classes_sharing_a_class_uri_carry_an_inherited_expression_once(): + """Classes with the same class_uri share one shape, which carries an + expression they inherit once.""" + schema = yaml.safe_load(_INHERITED_CLASS_EXPRESSION_SCHEMA) + schema["classes"]["Child"]["class_uri"] = "ex:Shared" + schema["classes"]["GrandChild"]["class_uri"] = "ex:Shared" + g = rdflib.Graph().parse(data=ShaclGenerator(json.dumps(schema), mergeimports=False).serialize()) + assert len(list(g.objects(EX_CE.Shared, SH["or"]))) == 1 + assert len(list(g.objects(EX_CE.Shared, SH["not"]))) == 1 + + +@pytest.mark.parametrize( + "child_usage,expected_shapes,warning", + [ + pytest.param( + None, + {EX_CE.Parent: False, EX_CE.Child: False, EX_CE.GrandChild: False}, + "Class 'Parent': any_of is not translated to SHACL, because it uses 'has_member' in the condition on " + "slot 'a' (in the shapes of 'Parent', 'Child', 'GrandChild').", + id="untranslatable-everywhere", + ), + pytest.param( + {"a": {"range": "integer"}}, + {EX_CE.Parent: True, EX_CE.Child: False, EX_CE.GrandChild: False}, + "Class 'Parent': any_of is not translated to SHACL, because it uses equals_string on slot 'a', whose " + "range 'integer' does not hold strings (in the shapes of 'Child', 'GrandChild').", + id="untranslatable-in-subclasses", + ), + ], +) +def test_untranslatable_inherited_class_expression_warned_once(caplog, child_usage, expected_shapes, warning): + """An inherited expression that cannot be translated is skipped in the + shapes where it cannot, and reported once, naming them.""" + schema = yaml.safe_load(_INHERITED_CLASS_EXPRESSION_SCHEMA) + schema["classes"]["Parent"]["any_of"] = ( + [{"slot_conditions": {"a": {"has_member": {"equals_string": "x"}}}}] + if child_usage is None + else [{"slot_conditions": {"a": {"equals_string": "x"}}}] + ) + if child_usage is not None: + schema["classes"]["Child"]["slot_usage"] = child_usage + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = rdflib.Graph().parse(data=ShaclGenerator(json.dumps(schema), mergeimports=False).serialize()) + for shape, translated in expected_shapes.items(): + assert ((shape, SH["or"], None) in g) == translated, shape + messages = [rec.message for rec in caplog.records if "any_of is not translated" in rec.message] + assert messages == [warning] + + +def test_class_expression_untranslatable_operator_skipped_with_warning(caplog): + """An operator with an untranslatable member is skipped whole, with a warning; the others are kept.""" + import logging + + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + tags: + multivalued: true + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [tags, a] + any_of: + - slot_conditions: + tags: + has_member: + equals_string: x + - slot_conditions: + a: + required: true + none_of: + - slot_conditions: + a: + equals_string: forbidden +""" + ) + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(schema) + + assert (EX_CE.Thing, SH["or"], None) not in g + assert any( + "any_of" in rec.message and "has_member" in rec.message and "tags" in rec.message for rec in caplog.records + ) + # the translatable none_of is kept, and enforced + shacl_ttl = g.serialize(format="turtle") + assert not _conforms(shacl_ttl, f'{_CE_PREFIXES}ex:t a ex:Thing ; ex:a "forbidden" .') + assert _conforms(shacl_ttl, f'{_CE_PREFIXES}ex:t a ex:Thing ; ex:a "ok" .') + + +def test_class_expression_condition_on_identifier_skipped_with_warning(caplog): + """An identifier is the node's IRI, not a property arc, so a condition on it is not translated.""" + import logging + + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + id: + identifier: true + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [id, a] + exactly_one_of: + - slot_conditions: + id: + pattern: "^ex:" + - slot_conditions: + a: + required: true +""" + ) + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(schema) + + assert (EX_CE.Thing, SH.xone, None) not in g + assert any("exactly_one_of" in rec.message and "identifier" in rec.message for rec in caplog.records) + + +def test_class_expression_absent_leaves_node_shapes_unchanged(): + """Without class-level expressions no node shape gets a logical constraint; slot-level any_of is unaffected.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + value: + any_of: + - range: integer + - range: string +classes: + Thing: + class_uri: ex:Thing + slots: [value] +""" + ) + g = _parse_shacl(schema) + + for shape in g.subjects(SH.targetClass, None): + assert not any((shape, p, None) in g for p in _LOGICAL_PREDICATES) + (value_shape,) = [p for p in g.objects(EX_CE.Thing, SH.property) if (p, SH.path, EX_CE.value) in g] + assert (value_shape, SH["or"], None) in g + + +def test_class_expression_condition_path_is_the_slot_induced_for_the_class(): + """A condition's sh:path is the one the class's own property shape uses, slot_usage and attributes included.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + code: + slot_uri: ex:baseCode + exact mappings: + slot_uri: ex:exactMatch +classes: + Thing: + class_uri: ex:Thing + slots: [code, exact mappings] + slot_usage: + code: + slot_uri: ex:thingCode + attributes: + loc: + slot_uri: ex:thingLoc + any_of: + - slot_conditions: + code: + required: true + - slot_conditions: + exact_mappings: + required: true + - slot_conditions: + loc: + required: true + Other: + class_uri: ex:Other + attributes: + loc: + slot_uri: ex:otherLoc +""" + ) + g = _parse_shacl(schema) + + class_paths = {path for p in g.objects(EX_CE.Thing, SH.property) for path in g.objects(p, SH.path)} + condition_paths = [ + path + for member in _list_members(g, EX_CE.Thing, SH["or"]) + for p in g.objects(member, SH.property) + for path in g.objects(p, SH.path) + ] + assert condition_paths and set(condition_paths) <= class_paths + assert set(condition_paths) == {EX_CE.thingCode, EX_CE.exactMatch, EX_CE.thingLoc} + + +_PRESENCE_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +enums: + Color: + permissible_values: + red: {} + green: {} +slots: + label: {} + note: {} + tags: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + tree_root: true + slots: [label, note, tags] +""" +) + + +def _class_expression_verdicts(expression: dict, obj: dict) -> tuple[bool, bool]: + """Whether the JSON Schema and the SHACL shapes generated for ``Thing`` with + the class-level *expression* accept *obj*. For SHACL, *obj* is loaded + through the generated JSON-LD context.""" + import jsonschema + import pyshacl + + from linkml.generators.jsonldcontextgen import ContextGenerator + from linkml.generators.jsonschemagen import JsonSchemaGenerator + + schema = yaml.safe_load(_PRESENCE_SCHEMA) + schema["classes"]["Thing"].update(expression) + schema = yaml.safe_dump(schema) + json_schema = json.loads(JsonSchemaGenerator(schema, top_class="Thing").serialize()) + context = json.loads(ContextGenerator(schema).serialize())["@context"] + data = rdflib.Graph().parse(data=json.dumps({"@context": context, "@type": "Thing", **obj}), format="json-ld") + shapes = rdflib.Graph().parse(data=ShaclGenerator(schema, mergeimports=False, closed=False).serialize()) + conforms, _, _ = pyshacl.validate(data, shacl_graph=shapes, meta_shacl=True) + return jsonschema.Draft7Validator(json_schema).is_valid(obj), conforms + + +def _is_a(value: str) -> dict: + """A class expression requiring ``label`` to equal *value*.""" + return {"slot_conditions": {"label": {"equals_string": value}}} + + +def _tags(**condition) -> dict: + """A class expression with *condition* on the multivalued ``tags``.""" + return {"slot_conditions": {"tags": condition}} + + +_LABEL_A_OR_NOTE_B = [_is_a("A"), {"slot_conditions": {"note": {"equals_string": "B"}}}] + + +@pytest.mark.parametrize( + "expression,obj,valid", + [ + # a condition is unknown for an absent slot, and only a definitely false expression is a violation + pytest.param({"any_of": [_is_a("A")]}, {}, True, id="any_of-absent"), + pytest.param({"any_of": [_is_a("A")]}, {"label": "Z"}, False, id="any_of-other"), + pytest.param({"any_of": _LABEL_A_OR_NOTE_B}, {"label": "Z"}, True, id="any_of-one-false-one-unknown"), + pytest.param({"none_of": [_is_a("A")]}, {}, True, id="none_of-absent"), + pytest.param({"none_of": [_is_a("A")]}, {"label": "A"}, False, id="none_of-match"), + pytest.param({"none_of": [_is_a("A")]}, {"label": "B"}, True, id="none_of-other"), + pytest.param({"none_of": [{"any_of": [_is_a("A"), _is_a("B")]}]}, {}, True, id="nested-in-none_of-absent"), + pytest.param( + {"none_of": [{"any_of": [_is_a("A"), _is_a("B")]}]}, {"label": "B"}, False, id="nested-in-none_of" + ), + pytest.param({"none_of": [{"any_of": [_is_a("A"), _is_a("B")]}]}, {"label": "C"}, True, id="nested-other"), + # the reading composes: wrapping in a one-member all_of, or negating twice, changes nothing + pytest.param({"all_of": [{"none_of": [_is_a("A")]}]}, {}, True, id="none_of-in-all_of-absent"), + pytest.param({"all_of": [{"none_of": [_is_a("A")]}]}, {"label": "A"}, False, id="none_of-in-all_of-match"), + pytest.param({"none_of": [{"none_of": [_is_a("A")]}]}, {}, True, id="double-negation-absent"), + pytest.param({"none_of": [{"none_of": [_is_a("A")]}]}, {"label": "A"}, True, id="double-negation-match"), + pytest.param({"none_of": [{"none_of": [_is_a("A")]}]}, {"label": "B"}, False, id="double-negation-other"), + # stated presence is definite; required: false only restates the default + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"required": False, "equals_string": "A"}}}]}, + {}, + True, + id="restated-default", + ), + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"value_presence": "ABSENT", "equals_string": "A"}}}]}, + {}, + False, + id="stated-absent", + ), + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"value_presence": "ABSENT", "equals_string": "A"}}}]}, + {"label": "A"}, + True, + id="stated-absent-present", + ), + pytest.param( + {"all_of": [{"slot_conditions": {"note": {"required": True, "value_presence": "ABSENT"}}}]}, + {}, + True, + id="value_presence-over-required", + ), + pytest.param( + {"all_of": [{"slot_conditions": {"note": {"required": True, "value_presence": "ABSENT"}}}]}, + {"note": "x"}, + False, + id="value_presence-over-required-present", + ), + # an empty condition is unknown for an absent slot, and definitely true for a present one + pytest.param({"none_of": [{"slot_conditions": {"label": {}}}]}, {}, True, id="empty-condition-absent"), + pytest.param({"none_of": [{"slot_conditions": {"label": {}}}]}, {"label": "x"}, False, id="empty-condition"), + # exactly_one_of holds when exactly one member is definitely true + pytest.param({"exactly_one_of": _LABEL_A_OR_NOTE_B}, {}, False, id="exactly_one_of-none-known"), + pytest.param({"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "A"}, True, id="exactly_one_of-first"), + pytest.param({"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "Z"}, False, id="exactly_one_of-none-true"), + pytest.param( + {"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "A", "note": "X"}, True, id="exactly_one_of-one-of-two" + ), + pytest.param( + {"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "A", "note": "B"}, False, id="exactly_one_of-both" + ), + # cardinalities count the values of a list, of which an absent slot has none + pytest.param({"all_of": [_tags(minimum_cardinality=2)]}, {"tags": ["a"]}, False, id="minimum-cardinality"), + pytest.param({"all_of": [_tags(minimum_cardinality=2)]}, {"tags": ["a", "b"]}, True, id="minimum-met"), + pytest.param({"all_of": [_tags(minimum_cardinality=2)]}, {}, False, id="minimum-absent"), + pytest.param({"all_of": [_tags(maximum_cardinality=1)]}, {"tags": ["a", "b"]}, False, id="maximum-cardinality"), + pytest.param({"none_of": [_tags(maximum_cardinality=1)]}, {}, True, id="none_of-maximum-absent"), + pytest.param({"none_of": [_tags(maximum_cardinality=1)]}, {"tags": ["a"]}, False, id="none_of-maximum"), + pytest.param( + {"none_of": [_tags(maximum_cardinality=1)]}, {"tags": ["a", "b"]}, True, id="none_of-maximum-over" + ), + pytest.param({"any_of": [_tags(maximum_cardinality=0), _is_a("A")]}, {}, True, id="maximum-zero-absent"), + # a single-valued slot holds at most one value + pytest.param( + {"all_of": [{"slot_conditions": {"label": {"minimum_cardinality": 2}}}]}, + {"label": "x"}, + False, + id="single-valued-minimum-two", + ), + # a range in a condition narrows the slot's values + pytest.param( + {"all_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, {"label": "red"}, True, id="range" + ), + pytest.param( + {"all_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, {"label": "blue"}, False, id="range-other" + ), + pytest.param({"all_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, {}, True, id="range-absent"), + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, + {"label": "red"}, + False, + id="none_of-range", + ), + ], +) +def test_class_expression_absent_slot_semantics(expression, obj, valid): + """The specification doesn't say what a slot condition means for an absent + slot. As with an SQL CHECK constraint, an expression is violated only when + it is definitely false: a condition that doesn't state presence is unknown + for an absent slot. exactly_one_of holds when exactly one member is + definitely true. The JSON Schema and SHACL generators decide alike.""" + assert _class_expression_verdicts(expression, obj) == (valid, valid) + + +@pytest.mark.parametrize( + "condition,tags,conforms", + [ + # a bound an absent slot satisfies and a present one may too leaves absence open + pytest.param({"maximum_cardinality": 1}, [], True, id="at-most-one-absent"), + pytest.param({"maximum_cardinality": 1}, ["a"], False, id="at-most-one"), + pytest.param({"maximum_cardinality": 1}, ["a", "b"], True, id="more-than-one"), + # a maximum of 0 decides that the slot is absent + pytest.param({"maximum_cardinality": 0, "pattern": "^A"}, [], False, id="at-most-zero-absent"), + pytest.param({"maximum_cardinality": 0, "pattern": "^A"}, ["A"], True, id="at-most-zero-present"), + ], +) +def test_class_expression_none_of_cardinality(condition, tags, conforms): + """Within none_of, a condition is definitely true or false for an absent + slot only when it decides presence, such as a maximum cardinality of 0; + otherwise it requires the slot in its "definitely true" form.""" + schema = yaml.safe_load(_PRESENCE_SCHEMA) + schema["classes"]["Thing"]["none_of"] = [{"slot_conditions": {"tags": condition}}] + shacl_ttl = ShaclGenerator(yaml.safe_dump(schema), mergeimports=False, closed=False).serialize() + values = "" if not tags else " ; ex:tags " + ", ".join(f'"{t}"' for t in tags) + assert _conforms(shacl_ttl, f"{_CE_PREFIXES}ex:x a ex:Thing{values} .") == conforms + + +@pytest.mark.parametrize( + "condition,properties,expected", + [ + # a cardinality is taken literally inside none_of: "not at most zero values" means present + ("maximum_cardinality: 0", "", False), + ("maximum_cardinality: 0", 'ex:tag "x"', True), + ("maximum_cardinality: 1", 'ex:tag "x"', False), + ("maximum_cardinality: 1", 'ex:tag "x", "y"', True), + ], +) +def test_class_expression_none_of_takes_cardinality_literally(condition, properties, expected): + """Inside none_of only a condition silent on presence and cardinality is made to require the slot.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + f""" +slots: + tag: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + slots: [tag] + none_of: + - slot_conditions: + tag: + {condition} +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +def test_class_expression_equals_string_on_enum_uses_the_meaning(): + """equals_string on an enum slot compares against the permissible value as _add_enum renders it.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +enums: + FormatEnum: + permissible_values: + OSI: + meaning: ex:OSI + MCAP: {} +slots: + fileFormat: + range: FormatEnum + hasChannel: {} +classes: + Format: + class_uri: ex:Format + slots: [fileFormat, hasChannel] + any_of: + - slot_conditions: + fileFormat: + equals_string: OSI + - slot_conditions: + fileFormat: + equals_string: MCAP + hasChannel: + required: true +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + g = rdflib.Graph().parse(data=shacl_ttl) + first, second = _list_members(g, EX_CE.Format, SH["or"]) + (osi_in,) = g.objects(_condition(g, first, EX_CE.fileFormat), SH["in"]) + assert list(Collection(g, osi_in)) == [EX_CE.OSI] + (mcap_in,) = g.objects(_condition(g, second, EX_CE.fileFormat), SH["in"]) + assert list(Collection(g, mcap_in)) == [Literal("MCAP")] + + prefix = "@prefix ex: ." + assert _conforms(shacl_ttl, f"{prefix} ex:f a ex:Format ; ex:fileFormat ex:OSI .") + assert not _conforms(shacl_ttl, f'{prefix} ex:f a ex:Format ; ex:fileFormat "MCAP" .') + + +@pytest.mark.parametrize("value,expected", [("5.0", True), ("5", True), ("6.0", False)]) +def test_class_expression_equals_number_compares_values(value, expected): + """equals_number matches the value, whatever numeric datatype carries it.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + size: + range: float + label: {} +classes: + Thing: + class_uri: ex:Thing + slots: [size, label] + any_of: + - slot_conditions: + size: + equals_number: 5 + - slot_conditions: + label: + required: true +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False, closed=False).serialize() + # sh:datatype xsd:float on the class's own property shape would reject an xsd:decimal, + # so every value is given as xsd:float + data = f""" + @prefix ex: . + @prefix xsd: . + ex:t a ex:Thing ; ex:size "{value}"^^xsd:float . + """ + assert _conforms(shacl_ttl, data) is expected + + +_UNTRANSLATABLE_CONDITIONS = { + # an identifier is the node's IRI; here it becomes one only through slot_usage + "identifier": """ +slots: + id: {} + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [id, a] + slot_usage: + id: + identifier: true + any_of: + - slot_conditions: + id: + required: true + - slot_conditions: + a: + required: true +""", + "'undefined', which is not a slot": """ +slots: + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [a] + any_of: + - slot_conditions: + undefined: + required: true + - slot_conditions: + a: + required: true +""", + "does not hold strings": """ +slots: + count: + range: integer + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [count, a] + any_of: + - slot_conditions: + count: + equals_string: "5" + - slot_conditions: + a: + required: true +""", +} + + +@pytest.mark.parametrize("reason", list(_UNTRANSLATABLE_CONDITIONS)) +def test_class_expression_untranslatable_condition_skipped_with_warning(caplog, reason): + """A condition on an identifier, on no slot, or equals_string on a non-string range is not translated.""" + import logging + + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(_CLASS_EXPRESSION_HEADER + _UNTRANSLATABLE_CONDITIONS[reason]) + + assert (EX_CE.Thing, SH["or"], None) not in g + assert any("any_of" in rec.message and reason in rec.message for rec in caplog.records) + + +def test_class_expression_is_a_with_native_names_uses_sh_node(): + """With native names, is_a references the member class's shape, suffix included, as a range does.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + a: {} +classes: + Marker: + class_uri: ex:Marker + mixin: true + Thing: + class_uri: ex:Thing + slots: [a] + any_of: + - is_a: Marker + - slot_conditions: + a: + required: true + MarkedThing: + is_a: Thing + mixins: [Marker] +""" + ) + g = _parse_shacl(schema, use_class_uri_names=False, suffix="Shape") + + (first, _) = _list_members(g, EX_CE.ThingShape, SH["or"]) + assert set(g.objects(first, SH.node)) == {EX_CE.MarkerShape} + assert (first, SH["class"], None) not in g + assert (EX_CE.MarkerShape, RDF.type, SH.NodeShape) in g + + +def _condition_property_shapes(g: rdflib.Graph, shape: URIRef, path: URIRef) -> list: + """The property shapes on *path* that the logical constraints of *shape* reach.""" + own = set(g.objects(shape, SH.property)) + return [node for node in g.transitive_objects(shape, None) if (node, SH.path, path) in g and node not in own] + + +def _class_expression_shacl(classes: str, **kwargs) -> rdflib.Graph: + """The SHACL shapes graph of a class-expression schema with the given ``slots:`` / ``classes:`` YAML.""" + return _parse_shacl(_CLASS_EXPRESSION_HEADER + classes, **kwargs) + + +def test_class_expression_condition_title_names_the_property_shape(): + """A condition's title becomes the sh:name of its property shape.""" + g = _class_expression_shacl( + """ +slots: + a: {} + b: {} +classes: + Thing: + class_uri: ex:Thing + slots: [a, b] + any_of: + - slot_conditions: + a: + title: the a condition + required: true + - slot_conditions: + b: + required: true +""" + ) + (condition,) = _condition_property_shapes(g, EX_CE.Thing, EX_CE.a) + assert set(g.objects(condition, SH.name)) == {Literal("the a condition")} + + +@pytest.mark.parametrize("values,conforms", [('"a", "b"', True), ('"a", "b", "c"', False)]) +def test_class_expression_strictest_maximum_applies(values, conforms): + """With an exact and a maximum cardinality, the stricter bound applies.""" + shacl_ttl = _class_expression_shacl( + """ +slots: + tag: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + slots: [tag] + all_of: + - slot_conditions: + tag: + exact_cardinality: 2 + maximum_cardinality: 3 +""" + ).serialize(format="turtle") + assert _conforms(shacl_ttl, f"{_CE_PREFIXES}ex:t a ex:Thing ; ex:tag {values} .") == conforms + + +@pytest.mark.parametrize("ref,conforms", [(None, True), ("ex:s", False), ("ex:o", True)]) +def test_class_expression_none_of_range(ref, conforms): + """A range in a condition is a value constraint: within none_of it requires + the slot in its "definitely true" form, so only a Special value violates.""" + shacl_ttl = _class_expression_shacl( + """ +slots: + ref: + range: Target +classes: + Target: + class_uri: ex:Target + Special: + class_uri: ex:Special + Thing: + class_uri: ex:Thing + slots: [ref] + none_of: + - slot_conditions: + ref: + range: Special +""", + closed=False, + ).serialize(format="turtle") + value = f" ; ex:ref {ref}" if ref else "" + data = f"{_CE_PREFIXES}ex:s a ex:Special, ex:Target . ex:o a ex:Target . ex:t a ex:Thing{value} ." + assert _conforms(shacl_ttl, data) == conforms + + +def test_class_expression_equals_string_uses_the_condition_range(): + """equals_string is rendered for the condition's own range: an enum value + with a meaning becomes that IRI.""" + g = _class_expression_shacl( + """ +enums: + FormatEnum: + permissible_values: + OSI: + meaning: ex:OSI +slots: + fileFormat: {} + other: {} +classes: + Format: + class_uri: ex:Format + slots: [fileFormat, other] + any_of: + - slot_conditions: + fileFormat: + range: FormatEnum + equals_string: OSI + - slot_conditions: + other: + required: true +""" + ) + (condition,) = _condition_property_shapes(g, EX_CE.Format, EX_CE.fileFormat) + in_lists = [list(Collection(g, node)) for node in g.objects(condition, SH["in"])] + in_lists += [ + list(Collection(g, node)) + for and_list in g.objects(condition, SH["and"]) + for member in Collection(g, and_list) + for node in g.objects(member, SH["in"]) + ] + assert in_lists and all(values == [EX_CE.OSI] for values in in_lists) + + +def test_class_expression_equals_string_on_a_custom_string_type(): + """A type derived from string holds strings, so equals_string on it is translated.""" + g = _class_expression_shacl( + """ +types: + Code: + typeof: string +slots: + code: + range: Code + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [code, a] + any_of: + - slot_conditions: + code: + equals_string: AB + - slot_conditions: + a: + required: true +""" + ) + assert (EX_CE.Thing, SH["or"], None) in g + + +def test_class_expression_builtin_range_without_types_import(): + """A built-in range name in a schema that doesn't import linkml:types gives its datatype.""" + schema = """ +id: https://example.org/class-expressions +name: class_expressions +prefixes: + ex: https://example.org/class-expressions/ +default_prefix: ex +slots: + size: {} + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [size, a] + any_of: + - slot_conditions: + size: + range: integer + - slot_conditions: + a: + required: true +""" + g = _parse_shacl(schema) + (condition,) = _condition_property_shapes(g, EX_CE.Thing, EX_CE.size) + assert (condition, SH.datatype, XSD.integer) in g + + +def test_class_expression_range_with_identifier_is_an_iri(): + """A condition ranged on a class with an identifier expects IRIs, as a slot's range does.""" + g = _class_expression_shacl( + """ +slots: + id: + identifier: true + ref: {} + a: {} +classes: + Target: + class_uri: ex:Target + slots: [id] + Thing: + class_uri: ex:Thing + slots: [ref, a] + any_of: + - slot_conditions: + ref: + range: Target + - slot_conditions: + a: + required: true +""" + ) + (condition,) = _condition_property_shapes(g, EX_CE.Thing, EX_CE.ref) + assert set(g.objects(condition, SH.nodeKind)) == {SH.IRI} + + +_UNTRANSLATABLE_MEMBERS = { + "untranslatable-second-member": ( + "has_member", + """ + - slot_conditions: + a: + required: true + - slot_conditions: + tags: + has_member: + equals_string: x""", + ), + "unknown-condition-range": ( + "unknown range", + """ + - slot_conditions: + a: + range: Undefined + - slot_conditions: + b: + required: true""", + ), + "untranslatable-nested-member": ( + "has_member", + """ + - all_of: + - slot_conditions: + tags: + has_member: + equals_string: x + - slot_conditions: + a: + required: true""", + ), + "is_a-not-a-class": ( + "not a class", + """ + - is_a: Undefined + - slot_conditions: + a: + required: true""", + ), + "equals_string-on-class-range": ( + "does not hold strings", + """ + - slot_conditions: + ref: + equals_string: x + - slot_conditions: + a: + required: true""", + ), + "equals_string_in-on-integer": ( + "does not hold strings", + """ + - slot_conditions: + count: + equals_string_in: ["5"] + - slot_conditions: + a: + required: true""", + ), +} + + +@pytest.mark.parametrize("case", list(_UNTRANSLATABLE_MEMBERS)) +def test_class_expression_untranslatable_member_skips_the_operator(caplog, case): + """An operator with an untranslatable member anywhere, at any depth or + position, is skipped as a whole, with a warning naming the reason.""" + reason, members = _UNTRANSLATABLE_MEMBERS[case] + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + tags: + multivalued: true + ref: + range: Target + count: + range: integer + a: {} + b: {} +classes: + Target: + class_uri: ex:Target + Thing: + class_uri: ex:Thing + slots: [tags, ref, count, a, b] + any_of:""" + + members + + "\n" + ) + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(schema) + assert (EX_CE.Thing, SH["or"], None) not in g + assert any("any_of" in rec.message and reason in rec.message for rec in caplog.records), caplog.text + + +_REPEATED_PARAMETERS_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +types: + Code: + typeof: string + pattern: "^[A-Z]+$" +enums: + LetterEnum: + permissible_values: + A: {} + B: {} + C: {} +slots: + size: + range: integer + label: {} + letter: {} + code: {} + other: {} +classes: + Thing: + class_uri: ex:Thing + slots: [size, label, letter, code, other] + any_of: + - slot_conditions: + size: + minimum_value: 3 + equals_number: 5 + label: + equals_string: A + equals_string_in: [A, B] + letter: + range: LetterEnum + equals_string: B + code: + range: Code + pattern: "^AB" + - slot_conditions: + other: + required: true +""" +) + + +@pytest.mark.parametrize( + "properties,expected", + [ + ('ex:size 5 ; ex:label "A" ; ex:letter "B" ; ex:code "ABC"', True), + ("ex:size 4", False), + ('ex:label "B"', False), + ('ex:letter "A"', False), + ('ex:code "ABc"', False), + ('ex:code "XY"', False), + ('ex:size 4 ; ex:other "x"', True), + ], +) +def test_class_expression_repeated_parameters_stay_well_formed(properties, expected): + """A parameter SHACL allows once per shape, needed twice by one condition, goes into sh:and. + + _conforms runs pyshacl with meta_shacl, which fails on an ill-formed shapes graph. + """ + shacl_ttl = ShaclGenerator(_REPEATED_PARAMETERS_SCHEMA, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing ; {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +@pytest.mark.parametrize( + "extra,properties,expected", + [ + ("", "", True), + ("maximum_cardinality: 5", "", True), + ("minimum_cardinality: 0", "", True), + ("maximum_cardinality: 5", 'ex:tag "A"', False), + ("maximum_cardinality: 5", 'ex:tag "B"', True), + ], +) +def test_class_expression_none_of_presence_is_monotonic(extra, properties, expected): + """Inside none_of, a cardinality that an absent slot satisfies does not flip an absent slot to rejected.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + f""" +slots: + tag: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + slots: [tag] + none_of: + - slot_conditions: + tag: + equals_string: A + {extra} +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected diff --git a/tests/linkml/test_validator/test_validation_context.py b/tests/linkml/test_validator/test_validation_context.py index a674048952..e19c1bda48 100644 --- a/tests/linkml/test_validator/test_validation_context.py +++ b/tests/linkml/test_validator/test_validation_context.py @@ -647,9 +647,9 @@ def _schema_with_value_disallowed() -> SchemaDefinition: def test_cache_hit_preserves_value_disallowed_not_keyword(): - """A slot with value_presence=ABSENT produces a root-level `not` keyword. - Cache-hit must carry it through from defs_class so the validator rejects - instances that include the forbidden field.""" + """A slot with value_presence=ABSENT produces a root-level `not: {required}` + in `allOf`. Cache-hit must carry it through from defs_class so the + validator rejects instances that include the forbidden field.""" schema = _schema_with_value_disallowed() cold = ValidationContext(schema, "ForbidsX").json_schema_validator( @@ -661,8 +661,9 @@ def test_cache_hit_preserves_value_disallowed_not_keyword(): closed=False, include_range_class_descendants=False ) - assert "not" in cold.schema, cold.schema - assert "not" in warm.schema, "value_disallowed `not` keyword must survive cache-hit" + forbids = {"not": {"required": ["forbidden_field"]}} + assert forbids in cold.schema.get("allOf", []), cold.schema + assert forbids in warm.schema.get("allOf", []), "value_disallowed `not` keyword must survive cache-hit" # Instance includes the forbidden field -> both validators reject bad = {"forbidden_field": "anything", "other_field": "ok"} @@ -677,18 +678,19 @@ def test_cache_hit_preserves_value_disallowed_not_keyword(): def test_cache_hit_does_not_leak_value_disallowed_not_to_target_without(): - """Warming with ForbidsX places `not` at root in cached. Hitting cache for - Plain (which has no value_disallowed slots) must NOT inherit `not`.""" + """Warming with ForbidsX places `not: {required}` at root in cached. Hitting + cache for Plain (which has no value_disallowed slots) must NOT inherit it.""" schema = _schema_with_value_disallowed() cache_key = _make_cache_key(schema, include_range_class_descendants=False) ValidationContext(schema, "ForbidsX").json_schema_validator(closed=False, include_range_class_descendants=False) - assert "not" in _json_schema_cache[cache_key] + forbids = {"not": {"required": ["forbidden_field"]}} + assert forbids in _json_schema_cache[cache_key].get("allOf", []) validator = ValidationContext(schema, "Plain").json_schema_validator( closed=False, include_range_class_descendants=False ) - assert "not" not in validator.schema, validator.schema.get("not") + assert forbids not in validator.schema.get("allOf", []), validator.schema.get("allOf") # Plain has no constraint against `forbidden_field` — the leak would falsely # reject this. Confirm it doesn't.