diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-actions.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-actions.rst index 3f37d728dbf2..ed4610c6ea7b 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-actions.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-actions.rst @@ -7,7 +7,29 @@ Customizing library models for GitHub Actions GitHub Actions analysis can be customized by adding library models in data extension files. -A data extension for GitHub Actions is a YAML file of the form: +A data extension for GitHub Actions can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/actions-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml @@ -16,8 +38,8 @@ A data extension for GitHub Actions is a YAML file of the form: pack: codeql/actions-all extensible: data: - - - - + - ["tuple", 1] + - ["tuple", 2] - ... The CodeQL library for GitHub Actions exposes the following extensible predicates: @@ -57,16 +79,23 @@ If there is an Action publisher that you trust, you can include the owner name/o To allow any Action from the publisher ``octodemo``, such as ``octodemo/3rd-party-action``, follow these steps: -1. Create a data extension file ``/models/trusted-owner.model.yml`` with the following content: +1. Create a data extension file ``/models/trusted-owner.model.json`` with the following content: - .. code-block:: yaml + .. code-block:: json - extensions: - - addsTo: - pack: codeql/actions-all - extensible: trustedActionsOwnerDataModel - data: - - ["octodemo"] + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/actions-all", + "extensible": "trustedActionsOwnerDataModel" + }, + "data": [ + ["octodemo"] + ] + } + ] + } 2. Create a model pack file ``/codeql-pack.yml`` with the following content: @@ -78,7 +107,7 @@ To allow any Action from the publisher ``octodemo``, such as ``octodemo/3rd-part extensionTargets: codeql/actions-all: '*' dataExtensions: - - models/**/*.yml + - models/**/*.json 3. Ensure that the model pack is included in your CodeQL analysis. @@ -91,14 +120,21 @@ GitHub's own organizations (``actions``, ``github`` and ``advanced-security``) a To distrust the first-party ``github`` owner, add a data extension file with the following content: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/actions-all - extensible: trustedActionsOwnerDataModel - data: - - ["!github"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/actions-all", + "extensible": "trustedActionsOwnerDataModel" + }, + "data": [ + ["!github"] + ] + } + ] + } With this in place, the query will once again report unpinned tags for Actions published by ``github``. diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-cpp.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-cpp.rst index 7ca619632272..0c163777d192 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-cpp.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-cpp.rst @@ -25,7 +25,29 @@ Syntax used to define an element in an extension file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each model of an element is defined using a data extension where each tuple constitutes a model. -A data extension file to extend the standard CPP queries included with CodeQL is a YAML file with the form: +A data extension file to extend the standard CPP queries included with CodeQL can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/cpp-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml @@ -34,11 +56,11 @@ A data extension file to extend the standard CPP queries included with CodeQL is pack: codeql/cpp-all extensible: data: - - - - + - ["tuple", 1] + - ["tuple", 2] - ... -Each YAML file may contain one or more top-level extensions. +Each data extension file may contain one or more top-level extensions. - ``addsTo`` defines the CodeQL pack name and extensible predicate that the extension is injected into. - ``data`` defines one or more rows of tuples that are injected as values into the extensible predicate. The number of columns and their types must match the definition of the extensible predicate. @@ -79,20 +101,27 @@ This example shows how the CPP query pack models the return value from the ``rea We need to add a tuple to the ``sourceModel(namespace, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/cpp-all - extensible: sourceModel - data: - - ["boost::asio", "", False, "read_until", "", "", "Argument[*1]", "remote", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/cpp-all", + "extensible": "sourceModel" + }, + "data": [ + ["boost::asio", "", false, "read_until", "", "", "Argument[*1]", "remote", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a free function) to be modeled as a source. - The first value ``"boost::asio"`` is the namespace name. - The second value ``""`` is the name of the type (class) that contains the method. Because we're modeling a free function, the type is left blank. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``False``. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``false``. - The fourth value ``"read_until"`` is the function name. - The fifth value is the function input type signature, which can be used to narrow down between functions that have the same name. In this case, we want the model to include all functions in ``boost::asio`` called ``read_until``. @@ -114,20 +143,27 @@ This example shows how the CPP query pack models the second argument of the ``bo We need to add a tuple to the ``sinkModel(namespace, type, subtypes, name, signature, ext, input, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/cpp-all - extensible: sinkModel - data: - - ["boost::asio", "", False, "write", "", "", "Argument[*1]", "remote-sink", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/cpp-all", + "extensible": "sinkModel" + }, + "data": [ + ["boost::asio", "", false, "write", "", "", "Argument[*1]", "remote-sink", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a free function) to be modeled as a sink. - The first value ``"boost::asio"`` is the namespace name. - The second value ``""`` is the name of the type (class) that contains the method. Because we're modeling a free function, the type is left blank. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``False``. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``false``. - The fourth value ``"write"`` is the function name. - The fifth value is the function input type signature, which can be used to narrow down between functions that have the same name. In this case, we want the model to include all functions in ``boost::asio`` called ``write``. @@ -149,20 +185,27 @@ This example shows how the CPP query pack models flow through a function for a s We need to add tuples to the ``summaryModel(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/cpp-all - extensible: summaryModel - data: - - ["boost::asio", "", False, "buffer", "", "", "Argument[*0]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/cpp-all", + "extensible": "summaryModel" + }, + "data": [ + ["boost::asio", "", false, "buffer", "", "", "Argument[*0]", "ReturnValue", "taint", "manual"] + ] + } + ] + } The first five values identify the callable (in this case free function) to be modeled as a summary. - The first value ``"boost::asio"`` is the namespace name. - The second value ``""`` is the name of the type (class) that contains the method. Because we're modeling a free function, the type is left blank. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``False``. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``false``. - The fourth value ``"buffer"`` is the function name. - The fifth value is the function input type signature, which can be used to narrow down between functions that have the same name. In this case, we want the model to include all functions in ``boost::asio`` called ``buffer``. @@ -190,20 +233,27 @@ This function escapes special characters in a string for use in an SQL statement We need to add a tuple to the ``barrierModel(namespace, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/cpp-all - extensible: barrierModel - data: - - ["", "", False, "mysql_real_escape_string", "", "", "Argument[*1]", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/cpp-all", + "extensible": "barrierModel" + }, + "data": [ + ["", "", false, "mysql_real_escape_string", "", "", "Argument[*1]", "sql-injection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a free function) to be modeled as a barrier. - The first value ``""`` is the namespace name. - The second value ``""`` is the name of the type (class) that contains the method. Because we're modeling a free function, the type is left blank. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``False``. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. For a free function, this should be ``false``. - The fourth value ``"mysql_real_escape_string"`` is the function name. - The fifth value is the function input type signature, which can be used to narrow down between functions that have the same name. @@ -229,20 +279,27 @@ Consider a function called ``is_safe`` which returns ``true`` when the data is c We need to add a tuple to the ``barrierGuardModel(namespace, type, subtypes, name, signature, ext, input, acceptingValue, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/cpp-all - extensible: barrierGuardModel - data: - - ["", "", False, "is_safe", "", "", "Argument[*0]", "true", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/cpp-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["", "", false, "is_safe", "", "", "Argument[*0]", "true", "sql-injection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a free function) to be modeled as a barrier guard. - The first value ``""`` is the namespace name. - The second value ``""`` is the name of the type (class) that contains the method. Because we're modeling a free function, the type is left blank. -- The third value ``False`` is a flag that indicates whether or not the model guard also applies to all overrides of the method. For a free function, this should be ``False``. +- The third value ``false`` is a flag that indicates whether or not the model guard also applies to all overrides of the method. For a free function, this should be ``false``. - The fourth value ``"is_safe"`` is the function name. - The fifth value is the function input type signature, which can be used to narrow down between functions that have the same name. diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-csharp.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-csharp.rst index a4b0e26d1bc8..a92776971ac2 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-csharp.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-csharp.rst @@ -25,7 +25,29 @@ Syntax used to define an element in an extension file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each model of an element is defined using a data extension where each tuple constitutes a model. -A data extension file to extend the standard C# queries included with CodeQL is a YAML file with the form: +A data extension file to extend the standard C# queries included with CodeQL can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml @@ -34,11 +56,11 @@ A data extension file to extend the standard C# queries included with CodeQL is pack: codeql/csharp-all extensible: data: - - - - + - ["tuple", 1] + - ["tuple", 2] - ... -Each YAML file may contain one or more top-level extensions. +Each data extension file may contain one or more top-level extensions. - ``addsTo`` defines the CodeQL pack name and extensible predicate that the extension is injected into. - ``data`` defines one or more rows of tuples that are injected as values into the extensible predicate. The number of columns and their types must match the definition of the extensible predicate. @@ -84,20 +106,27 @@ This is the constructor of the ``SqlCommand`` class, which is located in the ``S We need to add a tuple to the ``sinkModel``\(namespace, type, subtypes, name, signature, ext, input, kind, provenance) extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: sinkModel - data: - - ["System.Data.SqlClient", "SqlCommand", False, "SqlCommand", "(System.String,System.Data.SqlClient.SqlConnection)", "", "Argument[0]", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "sinkModel" + }, + "data": [ + ["System.Data.SqlClient", "SqlCommand", false, "SqlCommand", "(System.String,System.Data.SqlClient.SqlConnection)", "", "Argument[0]", "sql-injection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a method) to be modeled as a sink. - The first value ``System.Data.SqlClient`` is the namespace name. - The second value ``SqlCommand`` is the name of the class (type) that contains the method. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``SqlCommand`` is the method name. Constructors are named after the class. - The fifth value ``(System.String,System.Data.SqlClient.SqlConnection)`` is the method input type signature. The type names must be fully qualified. @@ -122,20 +151,27 @@ This is the ``GetStream`` method in the ``TcpClient`` class, which is located in We need to add a tuple to the ``sourceModel(namespace, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: sourceModel - data: - - ["System.Net.Sockets", "TcpClient", False, "GetStream", "()", "", "ReturnValue", "remote", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "sourceModel" + }, + "data": [ + ["System.Net.Sockets", "TcpClient", false, "GetStream", "()", "", "ReturnValue", "remote", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a method) to be modeled as a source. - The first value ``System.Net.Sockets`` is the namespace name. - The second value ``TcpClient`` is the name of the class (type) that contains the source. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``GetStream`` is the method name. - The fifth value ``()`` is the method input type signature. @@ -160,15 +196,22 @@ This pattern covers many of the cases where we need to summarize flow through a We need to add tuples to the ``summaryModel(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: summaryModel - data: - - ["System", "String", False, "Concat", "(System.Object,System.Object)", "", "Argument[0]", "ReturnValue", "taint", "manual"] - - ["System", "String", False, "Concat", "(System.Object,System.Object)", "", "Argument[1]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "summaryModel" + }, + "data": [ + ["System", "String", false, "Concat", "(System.Object,System.Object)", "", "Argument[0]", "ReturnValue", "taint", "manual"], + ["System", "String", false, "Concat", "(System.Object,System.Object)", "", "Argument[1]", "ReturnValue", "taint", "manual"] + ] + } + ] + } Each tuple defines flow from one argument to the return value. The first row defines flow from the first argument (``s1`` in the example) to the return value (``t`` in the example) and the second row defines flow from the second argument (``s2`` in the example) to the return value (``t`` in the example). @@ -178,7 +221,7 @@ These are the same for both of the rows above as we are adding two summaries for - The first value ``System`` is the namespace name. - The second value ``String`` is the class (type) name. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``Concat`` is the method name. - The fifth value ``(System.Object,System.Object)`` is the method input type signature. @@ -192,14 +235,21 @@ The remaining values are used to define the ``access-path``, the ``kind``, and t It would also be possible to merge the two rows into one by using a comma-separated list in the seventh value. This would be useful if the method has many arguments and the flow is the same for all of them. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: summaryModel - data: - - ["System", "String", False, "Concat", "(System.Object,System.Object)", "", "Argument[0,1]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "summaryModel" + }, + "data": [ + ["System", "String", false, "Concat", "(System.Object,System.Object)", "", "Argument[0,1]", "ReturnValue", "taint", "manual"] + ] + } + ] + } This row defines flow from both the first and the second argument to the return value. The seventh value ``Argument[0,1]`` is shorthand for specifying an access path to both ``Argument[0]`` and ``Argument[1]``. @@ -216,14 +266,21 @@ This example shows how the C# query pack models flow through a method for a simp We need to add a tuple to the ``summaryModel(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: summaryModel - data: - - ["System", "String", False, "Trim", "()", "", "Argument[this]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "summaryModel" + }, + "data": [ + ["System", "String", false, "Trim", "()", "", "Argument[this]", "ReturnValue", "taint", "manual"] + ] + } + ] + } Each tuple defines flow from one argument to the return value. The first row defines flow from the qualifier of the method call (``s1`` in the example) to the return value (``t`` in the example). @@ -233,7 +290,7 @@ These are the same for both of the rows above as we are adding two summaries for - The first value ``System`` is the namespace name. - The second value ``String`` is the class (type) name. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``Trim`` is the method name. - The fifth value ``()`` is the method input type signature. @@ -259,15 +316,22 @@ Here we model flow through higher order methods and collection types, as well as We need to add tuples to the ``summaryModel(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: summaryModel - data: - - ["System.Linq", "Enumerable", False, "Select", "(System.Collections.Generic.IEnumerable,System.Func)", "", "Argument[0].Element", "Argument[1].Parameter[0]", "value", "manual"] - - ["System.Linq", "Enumerable", False, "Select", "(System.Collections.Generic.IEnumerable,System.Func)", "", "Argument[1].ReturnValue", "ReturnValue.Element", "value", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "summaryModel" + }, + "data": [ + ["System.Linq", "Enumerable", false, "Select", "(System.Collections.Generic.IEnumerable,System.Func)", "", "Argument[0].Element", "Argument[1].Parameter[0]", "value", "manual"], + ["System.Linq", "Enumerable", false, "Select", "(System.Collections.Generic.IEnumerable,System.Func)", "", "Argument[1].ReturnValue", "ReturnValue.Element", "value", "manual"] + ] + } + ] + } Each tuple defines part of the flow that comprises the total flow through the ``Select`` method. The first five values identify the callable (in this case a method) to be modeled as a summary. @@ -275,7 +339,7 @@ These are the same for both of the rows above as we are adding two summaries for - The first value ``System.Linq`` is the namespace name. - The second value ``Enumerable`` is the class (type) name. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``Select`` is the method name, along with the type parameters for the method. The names of the generic type parameters provided in the model must match the names of the generic type parameters in the method signature in the source code. - The fifth value ``(System.Collections.Generic.IEnumerable,System.Func)`` is the method input type signature. The generics in the signature must match the generics in the method signature in the source code. @@ -318,20 +382,27 @@ The ``RawUrl`` property returns the raw URL of the current request, which is con We need to add a tuple to the ``barrierModel(namespace, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: barrierModel - data: - - ["System.Web", "HttpRequest", False, "get_RawUrl", "()", "", "ReturnValue", "url-redirection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "barrierModel" + }, + "data": [ + ["System.Web", "HttpRequest", false, "get_RawUrl", "()", "", "ReturnValue", "url-redirection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case the getter of a property) to be modeled as a barrier. - The first value ``System.Web`` is the namespace name. - The second value ``HttpRequest`` is the class (type) name. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``get_RawUrl`` is the method name. Getter and setter methods are named ``get_`` and ``set_`` respectively. - The fifth value ``()`` is the method input type signature. @@ -359,20 +430,27 @@ When the ``IsAbsoluteUri`` property returns ``false``, the URL is relative and t We need to add a tuple to the ``barrierGuardModel(namespace, type, subtypes, name, signature, ext, input, acceptingValue, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: barrierGuardModel - data: - - ["System", "Uri", False, "get_IsAbsoluteUri", "()", "", "Argument[this]", "false", "url-redirection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["System", "Uri", false, "get_IsAbsoluteUri", "()", "", "Argument[this]", "false", "url-redirection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case the getter of a property) to be modeled as a barrier guard. - The first value ``System`` is the namespace name. - The second value ``Uri`` is the class (type) name. -- The third value ``False`` is a flag that indicates whether or not the model guard also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model guard also applies to all overrides of the method. - The fourth value ``get_IsAbsoluteUri`` is the method name. Getter and setter methods are named ``get_`` and ``set_`` respectively. - The fifth value ``()`` is the method input type signature. @@ -398,14 +476,21 @@ A neutral model is used to define that there is no flow through a method. We need to add a tuple to the ``neutralModel(namespace, type, name, signature, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/csharp-all - extensible: neutralModel - data: - - ["System", "DateTime", "get_Now", "()", "summary", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/csharp-all", + "extensible": "neutralModel" + }, + "data": [ + ["System", "DateTime", "get_Now", "()", "summary", "manual"] + ] + } + ] + } The first four values identify the callable (in this case the getter of the ``Now`` property) to be modeled as a neutral, the fifth value is the kind, and the sixth value is the provenance (origin) of the neutral. diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-go.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-go.rst index f8f576b16b16..6d1e10ea78ed 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-go.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-go.rst @@ -25,7 +25,29 @@ Syntax used to define an element in an extension file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each model of an element is defined using a data extension where each tuple constitutes a model. -A data extension file to extend the standard Go queries included with CodeQL is a YAML file with the form: +A data extension file to extend the standard Go queries included with CodeQL can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml @@ -34,11 +56,11 @@ A data extension file to extend the standard Go queries included with CodeQL is pack: codeql/go-all extensible: data: - - - - + - ["tuple", 1] + - ["tuple", 2] - ... -Each YAML file may contain one or more top-level extensions. +Each data extension file may contain one or more top-level extensions. - ``addsTo`` defines the CodeQL pack name and extensible predicate that the extension is injected into. - ``data`` defines one or more rows of tuples that are injected as values into the extensible predicate. The number of columns and their types must match the definition of the extensible predicate. @@ -84,20 +106,27 @@ This is the ``Prepare`` method of the ``DB`` type in the ``database/sql`` packag We need to add a tuple to the ``sinkModel``\(package, type, subtypes, name, signature, ext, input, kind, provenance) extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: sinkModel - data: - - ["database/sql", "DB", True, "Prepare", "", "", "Argument[0]", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "sinkModel" + }, + "data": [ + ["database/sql", "DB", true, "Prepare", "", "", "Argument[0]", "sql-injection", "manual"] + ] + } + ] + } The first five values identify the function (in this case a method) to be modeled as a sink. - The first value ``database/sql`` is the package name. - The second value ``DB`` is the name of the type that the method is associated with. -- The third value ``True`` is a flag that indicates whether or not the model also applies to subtypes. This includes when the subtype embeds the given type, so that the method or field is promoted to be a method or field of the subtype. For interface methods it also includes types which implement the interface type. +- The third value ``true`` is a flag that indicates whether or not the model also applies to subtypes. This includes when the subtype embeds the given type, so that the method or field is promoted to be a method or field of the subtype. For interface methods it also includes types which implement the interface type. - The fourth value ``Prepare`` is the method name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -123,20 +152,27 @@ This is the ``FormValue`` method of the ``Request`` type which is located in the We need to add a tuple to the ``sourceModel(package, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: sourceModel - data: - - ["net/http", "Request", True, "FormValue", "", "", "ReturnValue", "remote", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "sourceModel" + }, + "data": [ + ["net/http", "Request", true, "FormValue", "", "", "ReturnValue", "remote", "manual"] + ] + } + ] + } The first five values identify the function to be modeled as a source. - The first value ``net/http`` is the package name. - The second value ``Request`` is the type name, since the function is a method of the ``Request`` type. -- The third value ``True`` is a flag that indicates whether or not the model also applies to subtypes. This includes when the subtype embeds the given type, so that the method or field is promoted to be a method or field of the subtype. For interface methods it also includes types which implement the interface type. +- The third value ``true`` is a flag that indicates whether or not the model also applies to subtypes. This includes when the subtype embeds the given type, so that the method or field is promoted to be a method or field of the subtype. For interface methods it also includes types which implement the interface type. - The fourth value ``FormValue`` is the function name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -163,14 +199,21 @@ This pattern covers many of the cases where we need to summarize flow through a We need to add a tuple to the ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: summaryModel - data: - - ["slices", "", False, "Max", "", "", "Argument[0].ArrayElement", "ReturnValue", "value", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "summaryModel" + }, + "data": [ + ["slices", "", false, "Max", "", "", "Argument[0].ArrayElement", "ReturnValue", "value", "manual"] + ] + } + ] + } The first row defines flow from the first argument (``a`` in the example) to the return value (``max`` in the example). @@ -178,7 +221,7 @@ The first five values identify the function to be modeled as a summary. - The first value ``slices`` is the package name. - The second value ``""`` is left blank, since the function is not a method of a type. -- The third value ``False`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. +- The third value ``false`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. - The fourth value ``Max`` is the function name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -207,14 +250,21 @@ This pattern covers many of the cases where we need to summarize flow through a We need to add a tuple to the ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: summaryModel - data: - - ["slices", "", False, "Concat", "", "", "Argument[0].ArrayElement.ArrayElement", "ReturnValue.ArrayElement", "value", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "summaryModel" + }, + "data": [ + ["slices", "", false, "Concat", "", "", "Argument[0].ArrayElement.ArrayElement", "ReturnValue.ArrayElement", "value", "manual"] + ] + } + ] + } The first row defines flow from the arguments (``a`` and ``b`` in the example) to the return value (``c`` in the example). @@ -222,7 +272,7 @@ The first five values identify the function to be modeled as a summary. - The first value ``slices`` is the package name. - The second value ``""`` is left blank, since the function is not a method of a type. -- The third value ``False`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. +- The third value ``false`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. - The fourth value ``Concat`` is the function name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -250,15 +300,22 @@ This pattern covers many of the cases where we need to summarize flow through a We need to add tuples to the ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: summaryModel - data: - - ["strings", "", False, "Join", "", "", "Argument[0]", "ReturnValue", "taint", "manual"] - - ["strings", "", False, "Join", "", "", "Argument[1]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "summaryModel" + }, + "data": [ + ["strings", "", false, "Join", "", "", "Argument[0]", "ReturnValue", "taint", "manual"], + ["strings", "", false, "Join", "", "", "Argument[1]", "ReturnValue", "taint", "manual"] + ] + } + ] + } Each tuple defines flow from one argument to the return value. The first row defines flow from the first argument (``elems`` in the example) to the return value (``t`` in the example) and the second row defines flow from the second argument (``sep`` in the example) to the return value (``t`` in the example). @@ -268,7 +325,7 @@ These are the same for both of the rows above as we are adding two summaries for - The first value ``strings`` is the package name. - The second value ``""`` is left blank, since the function is not a method of a type. -- The third value ``False`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. +- The third value ``false`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. - The fourth value ``Join`` is the function name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -282,14 +339,21 @@ The remaining values are used to define the ``access-path``, the ``kind``, and t It would also be possible to merge the two rows into one by using ".." to indicate a range in the seventh value. This would be useful if the method has many arguments and the flow is the same for all of them. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: summaryModel - data: - - ["strings", "", False, "Join", "", "", "Argument[0..1]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "summaryModel" + }, + "data": [ + ["strings", "", false, "Join", "", "", "Argument[0..1]", "ReturnValue", "taint", "manual"] + ] + } + ] + } This row defines flow from both the first and the second argument to the return value. The seventh value ``Argument[0..1]`` is shorthand for specifying an access path to both ``Argument[0]`` and ``Argument[1]``. @@ -306,14 +370,21 @@ This example shows how the Go query pack models flow through a method for a simp We need to add a tuple to the ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: summaryModel - data: - - ["net/url", "URL", True, "Hostname", "", "", "Argument[receiver]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "summaryModel" + }, + "data": [ + ["net/url", "URL", true, "Hostname", "", "", "Argument[receiver]", "ReturnValue", "taint", "manual"] + ] + } + ] + } Each tuple defines flow from one argument to the return value. The first row defines flow from the qualifier of the method call (``u`` in the example) to the return value (``host`` in the example). @@ -322,7 +393,7 @@ The first five values identify the function (in this case a method) to be modele - The first value ``net/url`` is the package name. - The second value ``URL`` is the receiver type. -- The third value ``True`` is a flag that indicates whether or not the model also applies to subtypes. This includes when the subtype embeds the given type, so that the method or field is promoted to be a method or field of the subtype. For interface methods it also includes types which implement the interface type. +- The third value ``true`` is a flag that indicates whether or not the model also applies to subtypes. This includes when the subtype embeds the given type, so that the method or field is promoted to be a method or field of the subtype. For interface methods it also includes types which implement the interface type. - The fourth value ``Hostname`` is the method name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -349,20 +420,27 @@ The ``Htmlquote`` function from the `beego` framework HTML-escapes a string, whi We need to add a tuple to the ``barrierModel(package, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: barrierModel - data: - - ["group:beego", "", True, "Htmlquote", "", "", "ReturnValue", "html-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "barrierModel" + }, + "data": [ + ["group:beego", "", true, "Htmlquote", "", "", "ReturnValue", "html-injection", "manual"] + ] + } + ] + } The first five values identify the function to be modeled as a barrier. - The first value ``group:beego`` is the package group name. The ``group:`` prefix indicates that this is a package group, which is used to match multiple package paths that refer to the same package. - The second value ``""`` is left blank since the function is not a method of a type. -- The third value ``True`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. +- The third value ``true`` is a flag that indicates whether or not the model also applies to subtypes. This has no effect for non-method functions. - The fourth value ``Htmlquote`` is the function name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. @@ -389,20 +467,27 @@ Consider a function called ``IsSafe`` which returns ``true`` when the data is co We need to add a tuple to the ``barrierGuardModel(package, type, subtypes, name, signature, ext, input, acceptingValue, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: barrierGuardModel - data: - - ["example.com/example", "", False, "IsSafe", "", "", "Argument[0]", "true", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["example.com/example", "", false, "IsSafe", "", "", "Argument[0]", "true", "sql-injection", "manual"] + ] + } + ] + } The first five values identify the function to be modeled as a barrier guard. - The first value ``example.com/example`` is the package name. - The second value ``""`` is left blank since the function is not a method of a type. -- The third value ``False`` is a flag that indicates whether or not the model guard also applies to subtypes. This has no effect for non-method functions. +- The third value ``false`` is a flag that indicates whether or not the model guard also applies to subtypes. This has no effect for non-method functions. - The fourth value ``IsSafe`` is the function name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. @@ -427,20 +512,27 @@ This example shows how we can model a field read as a source of tainted data. We need to add a tuple to the ``sourceModel(package, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: sourceModel - data: - - ["net/http", "Request", True, "Body", "", "", "", "remote", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "sourceModel" + }, + "data": [ + ["net/http", "Request", true, "Body", "", "", "", "remote", "manual"] + ] + } + ] + } The first five values identify the field to be modeled as a source. - The first value ``net/http`` is the package name. - The second value ``Request`` is the name of the type that the field is associated with. -- The third value ``True`` is a flag that indicates whether or not the model also applies to subtypes. For fields this means when the field is accessed as a promoted field in another type. +- The third value ``true`` is a flag that indicates whether or not the model also applies to subtypes. For fields this means when the field is accessed as a promoted field in another type. - The fourth value ``Body`` is the field name. - The fifth value ``""`` is the input type signature. For Go it should always be an empty string. It is needed for other languages where multiple functions may have the same name and they need to be distinguished by the number and types of the arguments. @@ -460,15 +552,22 @@ Note that packages hosted at ``gopkg.in`` use a slightly different syntax: the m To write models that only apply to ``github.com/couchbase/gocb/v2``, it is sufficient to include the major version suffix (``/v2``) in the package column. To write models that only apply to ``github.com/couchbase/gocb``, you may prefix the package column with ``fixed-version:``. For example, here are two models for a method that has changed name from v1 to v2. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go-all - extensible: sinkModel - data: - - ["fixed-version:github.com/couchbase/gocb", "Cluster", True, "ExecuteAnalyticsQuery", "", "", "Argument[0]", "nosql-injection", "manual"] - - ["github.com/couchbase/gocb/v2", "Cluster", True, "AnalyticsQuery", "", "", "Argument[0]", "nosql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "sinkModel" + }, + "data": [ + ["fixed-version:github.com/couchbase/gocb", "Cluster", true, "ExecuteAnalyticsQuery", "", "", "Argument[0]", "nosql-injection", "manual"], + ["github.com/couchbase/gocb/v2", "Cluster", true, "AnalyticsQuery", "", "", "Argument[0]", "nosql-injection", "manual"] + ] + } + ] + } Package grouping ~~~~~~~~~~~~~~~~ @@ -478,20 +577,31 @@ Since Go uses URLs for package identifiers, it is possible for packages to be im To handle this, the CodeQL Go library uses a mapping from the package path to a group name for the package. This mapping can be specified using the ``packageGrouping`` extensible predicate, and then the models for the APIs in the package will use the the prefix ``group:`` followed by the group name in place of the package path. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/go - extensible: packageGrouping - data: - - ["glog", "github.com/golang/glog"] - - ["glog", "gopkg.in/glog"] - - addsTo: - pack: codeql/go - extensible: sinkModel - data: - - ["group:glog", "", False, "Info", "", "", "Argument[0]", "log-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "packageGrouping" + }, + "data": [ + ["glog", "github.com/golang/glog"], + ["glog", "gopkg.in/glog"] + ] + }, + { + "addsTo": { + "pack": "codeql/go-all", + "extensible": "sinkModel" + }, + "data": [ + ["group:glog", "", false, "Info", "", "", "Argument[0]", "log-injection", "manual"] + ] + } + ] + } .. _threat-models-go: diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-java-and-kotlin.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-java-and-kotlin.rst index a3435002c858..9b3dd4d7a35f 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-java-and-kotlin.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-java-and-kotlin.rst @@ -30,7 +30,29 @@ Syntax used to define an element in an extension file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each model of an element is defined using a data extension where each tuple constitutes a model. -A data extension file to extend the standard Java queries included with CodeQL is a YAML file with the form: +A data extension file to extend the standard Java queries included with CodeQL can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml @@ -39,11 +61,11 @@ A data extension file to extend the standard Java queries included with CodeQL i pack: codeql/java-all extensible: data: - - - - + - ["tuple", 1] + - ["tuple", 2] - ... -Each YAML file may contain one or more top-level extensions. +Each data extension file may contain one or more top-level extensions. - ``addsTo`` defines the CodeQL pack name and extensible predicate that the extension is injected into. - ``data`` defines one or more rows of tuples that are injected as values into the extensible predicate. The number of columns and their types must match the definition of the extensible predicate. @@ -74,9 +96,9 @@ Specifying types in Java and Kotlin models **Nested and inner classes** are denoted by joining the enclosing type and the nested type with a dollar sign (``$``), for example ``Outer$Inner``. This applies both to the type column and to nested types in a signature. For example, the ``Level`` enum nested inside the ``Logger`` interface, nested inside the ``System`` class, is written as ``System$Logger$Level``: -.. code-block:: yaml +.. code-block:: json - - ["java.lang", "System$Logger", True, "log", "(System$Logger$Level,String)", "", "Argument[1]", "log-injection", "manual"] + ["java.lang", "System$Logger", true, "log", "(System$Logger$Level,String)", "", "Argument[1]", "log-injection", "manual"] **Generics** are erased, so type parameters are removed: @@ -85,9 +107,9 @@ Specifying types in Java and Kotlin models For example, ``forEach`` on ``Iterable`` takes a ``Consumer`` argument, so the type is ``Iterable`` and the signature is ``(Consumer)``: -.. code-block:: yaml +.. code-block:: json - - ["java.lang", "Iterable", True, "forEach", "(Consumer)", "", "Argument[this].Element", "Argument[0].Parameter[0]", "value", "manual"] + ["java.lang", "Iterable", true, "forEach", "(Consumer)", "", "Argument[this].Element", "Argument[0].Parameter[0]", "value", "manual"] Examples of custom model definitions ------------------------------------ @@ -109,20 +131,27 @@ This is the ``execute`` method in the ``Statement`` class, which is located in t We need to add a tuple to the ``sinkModel(package, type, subtypes, name, signature, ext, input, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: sinkModel - data: - - ["java.sql", "Statement", True, "execute", "(String)", "", "Argument[0]", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "sinkModel" + }, + "data": [ + ["java.sql", "Statement", true, "execute", "(String)", "", "Argument[0]", "sql-injection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a method) to be modeled as a sink. - The first value ``java.sql`` is the package name. - The second value ``Statement`` is the name of the class (type) that contains the method. -- The third value ``True`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``true`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``execute`` is the method name. - The fifth value ``(String)`` is the method input type signature. @@ -147,20 +176,27 @@ This is the ``getInputStream`` method in the ``Socket`` class, which is located We need to add a tuple to the ``sourceModel(package, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: sourceModel - data: - - ["java.net", "Socket", False, "getInputStream", "()", "", "ReturnValue", "remote", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "sourceModel" + }, + "data": [ + ["java.net", "Socket", false, "getInputStream", "()", "", "ReturnValue", "remote", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a method) to be modeled as a source. - The first value ``java.net`` is the package name. - The second value ``Socket`` is the name of the class (type) that contains the source. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``getInputStream`` is the method name. - The fifth value ``()`` is the method input type signature. @@ -185,15 +221,22 @@ This pattern covers many of the cases where we need to summarize flow through a We need to add tuples to the ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: summaryModel - data: - - ["java.lang", "String", False, "concat", "(String)", "", "Argument[this]", "ReturnValue", "taint", "manual"] - - ["java.lang", "String", False, "concat", "(String)", "", "Argument[0]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "summaryModel" + }, + "data": [ + ["java.lang", "String", false, "concat", "(String)", "", "Argument[this]", "ReturnValue", "taint", "manual"], + ["java.lang", "String", false, "concat", "(String)", "", "Argument[0]", "ReturnValue", "taint", "manual"] + ] + } + ] + } Each tuple defines flow from one argument to the return value. The first row defines flow from the qualifier (``s1`` in the example) to the return value (``t`` in the example) and the second row defines flow from the first argument (``s2`` in the example) to the return value (``t`` in the example). @@ -203,7 +246,7 @@ These are the same for both of the rows above as we are adding two summaries for - The first value ``java.lang`` is the package name. - The second value ``String`` is the class (type) name. -- The third value ``False`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``false`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``concat`` is the method name. - The fifth value ``(String)`` is the method input type signature. @@ -229,15 +272,22 @@ Here we model flow through higher order methods and collection types. We need to add tuples to the ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: summaryModel - data: - - ["java.util.stream", "Stream", True, "map", "(Function)", "", "Argument[this].Element", "Argument[0].Parameter[0]", "value", "manual"] - - ["java.util.stream", "Stream", True, "map", "(Function)", "", "Argument[0].ReturnValue", "ReturnValue.Element", "value", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "summaryModel" + }, + "data": [ + ["java.util.stream", "Stream", true, "map", "(Function)", "", "Argument[this].Element", "Argument[0].Parameter[0]", "value", "manual"], + ["java.util.stream", "Stream", true, "map", "(Function)", "", "Argument[0].ReturnValue", "ReturnValue.Element", "value", "manual"] + ] + } + ] + } Each tuple defines part of the flow that comprises the total flow through the ``map`` method. The first five values identify the callable (in this case a method) to be modeled as a summary. @@ -245,7 +295,7 @@ These are the same for both of the rows above as we are adding two summaries for - The first value ``java.util.stream`` is the package name. - The second value ``Stream`` is the class (type) name. -- The third value ``True`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``true`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``map`` is the method name. - The fifth value ``Function`` is the method input type signature. @@ -286,20 +336,27 @@ This is the ``getName`` method in the ``File`` class, which is located in the `` We need to add a tuple to the ``barrierModel(package, type, subtypes, name, signature, ext, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: barrierModel - data: - - ["java.io", "File", True, "getName", "()", "", "ReturnValue", "path-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "barrierModel" + }, + "data": [ + ["java.io", "File", true, "getName", "()", "", "ReturnValue", "path-injection", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a method) to be modeled as a barrier. - The first value ``java.io`` is the package name. - The second value ``File`` is the name of the class (type) that contains the method. -- The third value ``True`` is a flag that indicates whether or not the model also applies to all overrides of the method. +- The third value ``true`` is a flag that indicates whether or not the model also applies to all overrides of the method. - The fourth value ``getName`` is the method name. - The fifth value ``()`` is the method input type signature. @@ -328,20 +385,27 @@ When the ``isAbsolute`` method returns ``false``, the URI is relative and theref We need to add a tuple to the ``barrierGuardModel(package, type, subtypes, name, signature, ext, input, acceptingValue, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: barrierGuardModel - data: - - ["java.net", "URI", True, "isAbsolute", "()", "", "Argument[this]", "false", "request-forgery", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["java.net", "URI", true, "isAbsolute", "()", "", "Argument[this]", "false", "request-forgery", "manual"] + ] + } + ] + } The first five values identify the callable (in this case a method) to be modeled as a barrier guard. - The first value ``java.net`` is the package name. - The second value ``URI`` is the name of the class (type) that contains the method. -- The third value ``True`` is a flag that indicates whether or not the model guard also applies to all overrides of the method. +- The third value ``true`` is a flag that indicates whether or not the model guard also applies to all overrides of the method. - The fourth value ``isAbsolute`` is the method name. - The fifth value ``()`` is the method input type signature. @@ -367,14 +431,21 @@ A neutral model is used to define that there is no flow through a method. We need to add a tuple to the ``neutralModel(package, type, name, signature, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/java-all - extensible: neutralModel - data: - - ["java.time", "Instant", "now", "()", "summary", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/java-all", + "extensible": "neutralModel" + }, + "data": [ + ["java.time", "Instant", "now", "()", "summary", "manual"] + ] + } + ] + } The first four values identify the callable (in this case a method) to be modeled as a neutral, the fifth value is the kind, and the sixth value is the provenance (origin) of the neutral. diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-javascript.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-javascript.rst index 7ede0d0aff3a..aa33a970cbbf 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-javascript.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-javascript.rst @@ -7,18 +7,42 @@ Customizing Library Models for JavaScript JavaScript analysis can be customized by adding library models in data extension files. -A data extension for JavaScript is a YAML file of the form: +A data extension for JavaScript can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: - data: - - - - - - ... + extensions: + - addsTo: + pack: codeql/javascript-all + extensible: + data: + - ["tuple", 1] + - ["tuple", 2] + - ... + +Each data extension file may contain one or more top-level extensions. The CodeQL library for JavaScript exposes the following extensible predicates: @@ -44,14 +68,21 @@ In this example, we'll show how to add the following argument, passed to ``execa Note that this sink is already recognized by the CodeQL JS analysis, but for this example, you could add a tuple to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sinkModel - data: - - ["execa", "Member[shell].Argument[0]", "command-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sinkModel" + }, + "data": [ + ["execa", "Member[shell].Argument[0]", "command-injection"] + ] + } + ] + } - The first column, ``"execa"``, identifies a set of values from which to begin the search for the sink. The string ``"execa"`` means we start at the places where the codebase imports the NPM package ``execa``. @@ -76,18 +107,21 @@ In this example, we'll show how the ``event.data`` expression below could be mar Note that this source is already recognized by the CodeQL JS analysis, but for this example, you could add a tuple to the ``sourceModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sourceModel - data: - - [ - "global", - "Member[addEventListener].Argument[1].Parameter[0].Member[data]", - "remote", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sourceModel" + }, + "data": [ + ["global", "Member[addEventListener].Argument[1].Parameter[0].Member[data]", "remote"] + ] + } + ] + } - The first column, ``"global"``, begins the search at references to the global object (also known as ``window`` in browser contexts). This is a special JavaScript object that contains all global variables and methods. - ``Member[addEventListener]`` selects accesses to the ``addEventListener`` member. @@ -113,18 +147,21 @@ For example, it would also pick up this irrelevant source: We can refine the model by adding the ``WithStringArgument`` component to restrict the set of calls being considered: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sourceModel - data: - - [ - "global", - "Member[addEventListener].WithStringArgument[0=message].Argument[1].Parameter[0].Member[data]", - "remote", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sourceModel" + }, + "data": [ + ["global", "Member[addEventListener].WithStringArgument[0=message].Argument[1].Parameter[0].Member[data]", "remote"] + ] + } + ] + } The ``WithStringArgument[0=message]`` component here selects the subset of calls to ``addEventListener`` where the first argument is a string literal with the value ``"message"``. @@ -143,14 +180,21 @@ In this example, we'll show how to add the following SQL injection sink: We need to add a tuple to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sinkModel - data: - - ["mysql.Connection", "Member[query].Argument[0]", "sql-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sinkModel" + }, + "data": [ + ["mysql.Connection", "Member[query].Argument[0]", "sql-injection"] + ] + } + ] + } - The first column, ``"mysql.Connection"``, begins the search at any expression whose value is known to be an instance of the ``Connection`` type from the ``mysql`` package. This will select the ``connection`` parameter above because of its type annotation. @@ -162,11 +206,12 @@ This works in this example because the ``connection`` parameter has a type annot Note that there is a significant difference between the following two rows: -.. code-block:: yaml +.. code-block:: json - data: - - ["mysql.Connection", "", ...] - - ["mysql", "Member[Connection]", ...] + "data": [ + ["mysql.Connection", "" /* ... */], + ["mysql", "Member[Connection]" /* ... */] + ] The first row matches instances of ``mysql.Connection``, which are objects that encapsulate a MySQL connection. The second row would match something like ``require('mysql').Connection``, which is not itself a connection object. @@ -188,14 +233,21 @@ There is no type annotation on ``connection``, and there is no indication of wha By adding a tuple to the ``typeModel(type1, type2, path)`` extensible predicate we can tell our model that this function returns an instance of ``mysql.Connection``: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: typeModel - data: - - ["mysql.Connection", "@example/db", "Member[getConnection].ReturnValue"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "typeModel" + }, + "data": [ + ["mysql.Connection", "@example/db", "Member[getConnection].ReturnValue"] + ] + } + ] + } - The first column, ``"mysql.Connection"``, names the type that we're adding a new definition for. - The second column, ``"@example/db"``, begins the search at imports of the hypothetical NPM package ``@example/db``. @@ -209,9 +261,9 @@ The mechanism used here is how library models work for both TypeScript and plain A good library model contains ``typeModel`` tuples to ensure it works even in codebases without type annotations. For example, the ``mysql`` model that is included with the CodeQL JS analysis includes this type definition (among many others): -.. code-block:: yaml +.. code-block:: json - - ["mysql.Connection", "mysql", "Member[createConnection].ReturnValue"] + ["mysql.Connection", "mysql", "Member[createConnection].ReturnValue"] Example: Using fuzzy models to simplify modeling ------------------------------------------------ @@ -228,14 +280,21 @@ In this example, we'll show how to add the following SQL injection sink using a We need to add a tuple for a fuzzy model to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sinkModel - data: - - ["mysql", "Fuzzy.Member[query].Argument[0]", "sql-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sinkModel" + }, + "data": [ + ["mysql", "Fuzzy.Member[query].Argument[0]", "sql-injection"] + ] + } + ] + } - The first column, ``"mysql"``, begins the search at places where the `mysql` package is imported. - ``Fuzzy`` selects all objects that appear to originate from the `mysql` package, such as the `pool`, `conn`, `err`, and `rows` objects. @@ -247,21 +306,31 @@ We need to add a tuple for a fuzzy model to the ``sinkModel(type, path, kind)`` For reference, a more detailed model might look like this, as described in the preceding examples: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sinkModel - data: - - ["mysql.Connection", "Member[query].Argument[0]", "sql-injection"] - - - addsTo: - pack: codeql/javascript-all - extensible: typeModel - data: - - ["mysql.Pool", "mysql", "Member[createPool].ReturnValue"] - - ["mysql.Connection", "mysql.Pool", "Member[getConnection].Argument[0].Parameter[1]"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sinkModel" + }, + "data": [ + ["mysql.Connection", "Member[query].Argument[0]", "sql-injection"] + ] + }, + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "typeModel" + }, + "data": [ + ["mysql.Pool", "mysql", "Member[createPool].ReturnValue"], + ["mysql.Connection", "mysql.Pool", "Member[getConnection].Argument[0].Parameter[1]"] + ] + } + ] + } The model using the ``Fuzzy`` component is simpler, at the cost of being approximate. This technique is useful when modeling a large or complex library, where it is difficult to write a detailed model. @@ -278,20 +347,21 @@ In this example, we'll show how to add flow through calls to `decodeURIComponent Note that this flow is already recognized by the CodeQL JS analysis, but for this example, you could add a tuple to the ``summaryModel(type, path, input, output, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: summaryModel - data: - - [ - "global", - "Member[decodeURIComponent]", - "Argument[0]", - "ReturnValue", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "summaryModel" + }, + "data": [ + ["global", "Member[decodeURIComponent]", "Argument[0]", "ReturnValue", "taint"] + ] + } + ] + } - The first column, ``"global"``, begins the search for relevant calls at references to the global object. In JavaScript, global variables are properties of the global object, so this lets us access global variables or functions. @@ -315,20 +385,21 @@ In this example, we'll show how to add flow through calls to ``forEach`` from th Note that this flow is already recognized by the CodeQL JS analysis, but for this example, you could add a tuple to the ``summaryModel(type, path, input, output, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: summaryModel - data: - - [ - "underscore", - "Member[forEach]", - "Argument[0].ArrayElement", - "Argument[1].Parameter[0]", - "value", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "summaryModel" + }, + "data": [ + ["underscore", "Member[forEach]", "Argument[0].ArrayElement", "Argument[1].Parameter[0]", "value"] + ] + } + ] + } - The first column, ``"underscore"``, begins the search for relevant calls at places where the ``underscore`` package is imported. - The second column, ``Member[forEach]``, selects references to the ``forEach`` member from the ``underscore`` package. @@ -365,18 +436,21 @@ on the incoming request objects: We need to add a tuple to the ``sourceModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sourceModel - data: - - [ - "@example/middleware", - "Member[injectData].ReturnValue.GuardedRouteHandler.Parameter[0].Member[data]", - "remote", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sourceModel" + }, + "data": [ + ["@example/middleware", "Member[injectData].ReturnValue.GuardedRouteHandler.Parameter[0].Member[data]", "remote"] + ] + } + ] + } - The first column, ``"@example/middleware"``, begins the search at imports of the hypothetical NPM package ``@example/middleware``. - ``Member[injectData]`` selects accesses to the ``injectData`` member. @@ -398,14 +472,21 @@ In this example, we'll show how to add the return value of ``encodeURIComponent` We need to add a tuple to the ``barrierModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: barrierModel - data: - - ["global", "Member[encodeURIComponent].ReturnValue", "html-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "barrierModel" + }, + "data": [ + ["global", "Member[encodeURIComponent].ReturnValue", "html-injection"] + ] + } + ] + } - The first column, ``"global"``, begins the search for relevant calls at references to the global object. - The second column, ``Member[encodeURIComponent].ReturnValue``, selects the return value of the ``encodeURIComponent`` function. @@ -425,14 +506,21 @@ Consider a function called `isValid` which returns `true` when the data is consi We need to add a tuple to the ``barrierGuardModel(type, path, acceptingValue, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: barrierGuardModel - data: - - ["my-package", "Member[isValid].Argument[0]", "true", "sql-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["my-package", "Member[isValid].Argument[0]", "true", "sql-injection"] + ] + } + ] + } - The first column, ``"my-package"``, begins the search at imports of the hypothetical NPM package ``my-package``. - The second column, ``Member[isValid].Argument[0]``, selects the first argument of the `isValid` function. This is the value being validated. @@ -458,14 +546,21 @@ Adds a new taint source. Most taint-tracking queries will use the new source. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sourceModel - data: - - ["global", "Member[user].Member[name]", "remote"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sourceModel" + }, + "data": [ + ["global", "Member[user].Member[name]", "remote"] + ] + } + ] + } sinkModel(type, path, kind) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -478,14 +573,21 @@ Adds a new taint sink. Sinks are query-specific and will typically affect one or Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: sinkModel - data: - - ["global", "Member[eval].Argument[0]", "code-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "sinkModel" + }, + "data": [ + ["global", "Member[eval].Argument[0]", "code-injection"] + ] + } + ] + } summaryModel(type, path, input, output, kind) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -500,20 +602,21 @@ Adds flow through a function call. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: summaryModel - data: - - [ - "global", - "Member[decodeURIComponent]", - "Argument[0]", - "ReturnValue", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "summaryModel" + }, + "data": [ + ["global", "Member[decodeURIComponent]", "Argument[0]", "ReturnValue", "taint"] + ] + } + ] + } typeModel(type1, type2, path) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -526,18 +629,21 @@ Adds a new definition of a type. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/javascript-all - extensible: typeModel - data: - - [ - "mysql.Connection", - "@example/db", - "Member[getConnection].ReturnValue", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/javascript-all", + "extensible": "typeModel" + }, + "data": [ + ["mysql.Connection", "@example/db", "Member[getConnection].ReturnValue"] + ] + } + ] + } Types ----- diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-python.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-python.rst index ee4565caff3e..d020282af218 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-python.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-python.rst @@ -7,18 +7,42 @@ Customizing Library Models for Python Python analysis can be customized by adding library models in data extension files. -A data extension for Python is a YAML file of the form: +A data extension for Python can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml - extensions: - - addsTo: - pack: codeql/python-all - extensible: - data: - - - - - - ... + extensions: + - addsTo: + pack: codeql/python-all + extensible: + data: + - ["tuple", 1] + - ["tuple", 2] + - ... + +Each data extension file may contain one or more top-level extensions. The CodeQL library for Python exposes the following extensible predicates: @@ -44,14 +68,21 @@ In this example, we'll show how to add the following argument, passed to ``sudo` Note that this sink is already recognized by the CodeQL Python analysis, but for this example, you could add a tuple to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: sinkModel - data: - - ["fabric", "Member[operations].Member[sudo].Argument[0]", "command-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "sinkModel" + }, + "data": [ + ["fabric", "Member[operations].Member[sudo].Argument[0]", "command-injection"] + ] + } + ] + } - The first column, ``"fabric"``, identifies a set of values from which to begin the search for the sink. The string ``"fabric"`` means we start at the places where the codebase imports the package ``fabric``. @@ -77,14 +108,21 @@ Often sinks are found as arguments to methods rather than functions. In this exa Note that this sink is already recognized by the CodeQL Python analysis, but for this example, you could add a tuple to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: sinkModel - data: - - ["invoke", "Member[Context].Instance.Member[run].Argument[0]", "command-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "sinkModel" + }, + "data": [ + ["invoke", "Member[Context].Instance.Member[run].Argument[0]", "command-injection"] + ] + } + ] + } - The first column, ``"invoke"``, begins the search at places where the codebase imports the package ``invoke``. - The second column is an access path that is evaluated from left to right, starting at the values that were identified by the first column. @@ -100,14 +138,21 @@ Note that the ``Instance`` component is used to select instances of a class, inc Since methods on instances are common targets, we have a more compact syntax for selecting them. The first column, the type, is allowed to contain a dotted path ending in a class name. This will begin the search at instances of that class. Using this syntax, the previous example could be written as: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: sinkModel - data: - - ["invoke.Context", "Member[run].Argument[0]", "command-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "sinkModel" + }, + "data": [ + ["invoke.Context", "Member[run].Argument[0]", "command-injection"] + ] + } + ] + } Continued example: Multiple ways to obtain a type ------------------------------------------------- @@ -124,14 +169,21 @@ Comparing to the previous Python snippet, the ``Context`` class is now found as We could add a data extension similar to the previous one, but with the type ``invoke.context.Context``. However, we can also use the ``typeModel(type1, type2, path)`` extensible predicate to describe how to reach ``invoke.Context`` from ``invoke.context.Context``: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: typeModel - data: - - ["invoke.Context", "invoke.context.Context", ""] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "typeModel" + }, + "data": [ + ["invoke.Context", "invoke.context.Context", ""] + ] + } + ] + } - The first column, ``"invoke.Context"``, is the name of the type to reach. - The second column, ``"invoke.context.Context"``, is the name of the type from which to evaluate the path. @@ -162,18 +214,21 @@ This filename is what we want to mark as a taint source. An example use looks as Note that this source is already recognized by the CodeQL Python analysis, but for this example, you could add a tuple to the ``sourceModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: sourceModel - data: - - [ - "django.db.models.FileField!", - "Call.Argument[0,upload_to:].Parameter[1]", - "remote", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "sourceModel" + }, + "data": [ + ["django.db.models.FileField!", "Call.Argument[0,upload_to:].Parameter[1]", "remote"] + ] + } + ] + } - The first column, ``"django.db.models.FileField!"``, is a dotted path to the ``FileField`` class from the ``django.db.models`` package. The ``!`` at the end of the type name indicates that we are looking for the class itself rather than instances of this class. @@ -201,20 +256,21 @@ In this example, we'll show how to add flow through calls to ``re.compile``. Note that this flow is already recognized by the CodeQL Python analysis, but for this example, you could add a tuple to the ``summaryModel(type, path, input, output, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: summaryModel - data: - - [ - "re", - "Member[compile]", - "Argument[0,pattern:]", - "ReturnValue.Attribute[pattern]", - "value", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "summaryModel" + }, + "data": [ + ["re", "Member[compile]", "Argument[0,pattern:]", "ReturnValue.Attribute[pattern]", "value"] + ] + } + ] + } - The first column, ``"re"``, begins the search for relevant calls at places where the ``re`` package is imported. - The second column, ``"Member[compile]"``, is a path leading to the function calls we wish to model. @@ -236,20 +292,21 @@ In this example, we'll show how to add flow through calls to the built-in functi Note that this flow is already recognized by the CodeQL Python analysis, but for this example, you could add a tuple to the ``summaryModel(type, path, input, output, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: summaryModel - data: - - [ - "builtins", - "Member[sorted]", - "Argument[0]", - "ReturnValue", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "summaryModel" + }, + "data": [ + ["builtins", "Member[sorted]", "Argument[0]", "ReturnValue", "taint"] + ] + } + ] + } - The first column, ``"builtins"``, begins the search for relevant calls among references to the built-in names. In Python, many built-in functions are available. Technically, most of these are part of the ``builtins`` package, but they can be accessed without an explicit import. When we write ``builtins`` in the first column, we will find both the implicit and explicit references to the built-in functions. @@ -261,20 +318,21 @@ Note that this flow is already recognized by the CodeQL Python analysis, but for We might also provide a summary stating that the elements of the input list are preserved in the output list: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: summaryModel - data: - - [ - "builtins", - "Member[sorted]", - "Argument[0].ListElement", - "ReturnValue.ListElement", - "value", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "summaryModel" + }, + "data": [ + ["builtins", "Member[sorted]", "Argument[0].ListElement", "ReturnValue.ListElement", "value"] + ] + } + ] + } The tracking of list elements is imprecise in that the analysis does not know where in the list the tracked value is found. So this summary simply states that if the value is found somewhere in the input list, it will also be found somewhere in the output list, unchanged. @@ -291,14 +349,21 @@ In this example, we'll show how to add the return value of ``html.escape`` as a We need to add a tuple to the ``barrierModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: barrierModel - data: - - ["html", "Member[escape].ReturnValue", "html-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "barrierModel" + }, + "data": [ + ["html", "Member[escape].ReturnValue", "html-injection"] + ] + } + ] + } - The first column, ``"html"``, begins the search at places where the ``html`` module is imported. - The second column, ``Member[escape].ReturnValue``, selects the return value of the ``escape`` function from the ``html`` module. @@ -318,19 +383,21 @@ Consider the function ``url_has_allowed_host_and_scheme`` from the ``django.util We need to add a tuple to the ``barrierGuardModel(type, path, acceptingValue, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: barrierGuardModel - data: - - [ - "django", - "Member[utils].Member[http].Member[url_has_allowed_host_and_scheme].Argument[0,url:]", - "true", - "url-redirection", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["django", "Member[utils].Member[http].Member[url_has_allowed_host_and_scheme].Argument[0,url:]", "true", "url-redirection"] + ] + } + ] + } - The first column, ``"django"``, begins the search at places where the ``django`` package is imported. - The second column, ``Member[utils].Member[http].Member[url_has_allowed_host_and_scheme].Argument[0,url:]``, selects the first argument (or the keyword argument ``url``) of the ``url_has_allowed_host_and_scheme`` function in the ``django.utils.http`` module. This is the value being validated. @@ -356,14 +423,21 @@ Adds a new taint source. Most taint-tracking queries will use the new source. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: sourceModel - data: - - ["flask", "Member[request]", "remote"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "sourceModel" + }, + "data": [ + ["flask", "Member[request]", "remote"] + ] + } + ] + } sinkModel(type, path, kind) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -376,14 +450,21 @@ Adds a new taint sink. Sinks are query-specific and will typically affect one or Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: sinkModel - data: - - ["builtins", "Member[exec].Argument[0]", "code-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "sinkModel" + }, + "data": [ + ["builtins", "Member[exec].Argument[0]", "code-injection"] + ] + } + ] + } summaryModel(type, path, input, output, kind) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -398,20 +479,21 @@ Adds flow through a function call. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: summaryModel - data: - - [ - "builtins", - "Member[reversed]", - "Argument[0]", - "ReturnValue", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "summaryModel" + }, + "data": [ + ["builtins", "Member[reversed]", "Argument[0]", "ReturnValue", "taint"] + ] + } + ] + } typeModel(type1, type2, path) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -426,18 +508,21 @@ In the context of instances, this describes how to obtain an instance of ``type1 Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/python-all - extensible: typeModel - data: - - [ - "flask.Response", - "flask", - "Member[jsonify].ReturnValue", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/python-all", + "extensible": "typeModel" + }, + "data": [ + ["flask.Response", "flask", "Member[jsonify].ReturnValue"] + ] + } + ] + } Types ----- diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-ruby.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-ruby.rst index beccf82327f2..07526ffd9a83 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-ruby.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-ruby.rst @@ -8,18 +8,42 @@ Customizing library models for Ruby Ruby analysis can be customized by adding library models in data extension files. -A data extension for Ruby is a YAML file of the form: +A data extension for Ruby can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: - data: - - - - - - ... + extensions: + - addsTo: + pack: codeql/ruby-all + extensible: + data: + - ["tuple", 1] + - ["tuple", 2] + - ... + +Each data extension file may contain one or more top-level extensions. The CodeQL library for Ruby exposes the following extensible predicates: @@ -45,15 +69,21 @@ In this example, we'll show how to add the following argument, passed to ``tty-c We need to add a tuple to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: sinkModel - data: - - ["TTY::Command", "Method[run].Argument[0]", "command-injection"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "sinkModel" + }, + "data": [ + ["TTY::Command", "Method[run].Argument[0]", "command-injection"] + ] + } + ] + } - The first column, ``"TTY::Command"``, identifies a set of values from which to begin the search for the sink. The string ``"TTY::Command"`` means we start at the places where the codebase constructs instances of the class ``TTY::Command``. @@ -79,18 +109,21 @@ In this example, we'll show how the 'x' parameter below could be marked as a rem We need to add a tuple to the ``sourceModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: sourceModel - data: - - [ - "Sinatra::Base!", - "Method[get].Argument[block].Parameter[0]", - "remote", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "sourceModel" + }, + "data": [ + ["Sinatra::Base!", "Method[get].Argument[block].Parameter[0]", "remote"] + ] + } + ] + } - The first column, ``"Sinatra::Base!"``, begins the search at references to the ``Sinatra::Base`` class. The ``!`` suffix indicates that we want to search for references to the class itself, rather than instances of the class. @@ -113,14 +146,21 @@ In this example, we'll show how to add the following SQL injection sink: We need to add a tuple to the ``sinkModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: sinkModel - data: - - ["Mysql2::Client", "Method[query].Argument[0]", "sql-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "sinkModel" + }, + "data": [ + ["Mysql2::Client", "Method[query].Argument[0]", "sql-injection"] + ] + } + ] + } - The first column, ``"Mysql2::Client"``, begins the search at any instance of the ``Mysql2::Client`` class. - ``Method[query]`` selects any call to the ``query`` method on that instance. @@ -145,14 +185,21 @@ may have many models for the various methods available. Because ``Mysql2::EM::Cl Instead of updating all our models to include both classes, we can add a tuple to the ``typeModel(type, subtype, ext)`` extensible predicate to indicate that ``Mysql2::EM::Client`` is a subclass of ``Mysql2::Client``: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: typeModel - data: - - ["Mysql2::Client", "Mysql2::EM::Client", ""] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "typeModel" + }, + "data": [ + ["Mysql2::Client", "Mysql2::EM::Client", ""] + ] + } + ] + } Example: Adding flow through 'URI.decode_uri_component' ------------------------------------------------------- @@ -165,20 +212,21 @@ In this example, we'll show how to add flow through calls to 'URI.decode_uri_com We need to add a tuple to the ``summaryModel(type, path, input, output, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: summaryModel - data: - - [ - "URI!", - "Method[decode_uri_component]", - "Argument[0]", - "ReturnValue", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "summaryModel" + }, + "data": [ + ["URI!", "Method[decode_uri_component]", "Argument[0]", "ReturnValue", "taint"] + ] + } + ] + } - The first column, ``"URI!"``, begins the search for relevant calls at references to the ``URI`` class. The ``!`` suffix indicates that we are looking for the class itself, rather than instances of the class. @@ -201,20 +249,21 @@ In this example, we'll show how to add flow through calls to ``File#each`` from We need to add a tuple to the ``summaryModel(type, path, input, output, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: summaryModel - data: - - [ - "File", - "Method[each]", - "Argument[self]", - "Argument[block].Parameter[0]", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "summaryModel" + }, + "data": [ + ["File", "Method[each]", "Argument[self]", "Argument[block].Parameter[0]", "taint"] + ] + } + ] + } - The first column, ``"File"``, begins the search for relevant calls at places where the ``File`` class is used. - The second column, ``Method[each]``, selects references to the ``each`` method on the ``File`` class. @@ -240,14 +289,21 @@ In this example, we'll show how to add the return value of ``Mysql2::Client#esca We need to add a tuple to the ``barrierModel(type, path, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: barrierModel - data: - - ["Mysql2::Client!", "Method[escape].ReturnValue", "sql-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "barrierModel" + }, + "data": [ + ["Mysql2::Client!", "Method[escape].ReturnValue", "sql-injection"] + ] + } + ] + } - The first column, ``"Mysql2::Client!"``, begins the search for relevant calls at references to the ``Mysql2::Client`` class. The ``!`` suffix indicates that we want to search for references to the class itself, rather than instances of the class. @@ -269,14 +325,21 @@ Consider a validation method ``Validator.is_safe`` which returns ``true`` when t We need to add a tuple to the ``barrierGuardModel(type, path, acceptingValue, kind)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: barrierGuardModel - data: - - ["Validator!", "Method[is_safe].Argument[0]", "true", "sql-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["Validator!", "Method[is_safe].Argument[0]", "true", "sql-injection"] + ] + } + ] + } - The first column, ``"Validator!"``, begins the search at references to the ``Validator`` class. The ``!`` suffix indicates that we want to search for references to the class itself, rather than instances of the class. @@ -303,14 +366,21 @@ Adds a new taint source. Most taint-tracking queries will use the new source. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: sourceModel - data: - - ["User", "Method[name]", "remote"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "sourceModel" + }, + "data": [ + ["User", "Method[name]", "remote"] + ] + } + ] + } sinkModel(type, path, kind) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -323,14 +393,21 @@ Adds a new taint sink. Sinks are query-specific and will typically affect one or Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: sinkModel - data: - - ["ExecuteShell", "Method[run].Argument[0]", "command-injection"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "sinkModel" + }, + "data": [ + ["ExecuteShell", "Method[run].Argument[0]", "command-injection"] + ] + } + ] + } summaryModel(type, path, input, output, kind) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -345,20 +422,21 @@ Adds flow through a method call. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: summaryModel - data: - - [ - "URI", - "Method[decode_uri_component]", - "Argument[0]", - "ReturnValue", - "taint", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "summaryModel" + }, + "data": [ + ["URI", "Method[decode_uri_component]", "Argument[0]", "ReturnValue", "taint"] + ] + } + ] + } typeModel(type1, type2, path) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -371,18 +449,21 @@ Adds a new definition of a type. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/ruby-all - extensible: typeModel - data: - - [ - "Mysql2::Client", - "MyDbWrapper", - "Method[getConnection].ReturnValue", - ] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/ruby-all", + "extensible": "typeModel" + }, + "data": [ + ["Mysql2::Client", "MyDbWrapper", "Method[getConnection].ReturnValue"] + ] + } + ] + } Types ----- diff --git a/docs/codeql/codeql-language-guides/customizing-library-models-for-rust.rst b/docs/codeql/codeql-language-guides/customizing-library-models-for-rust.rst index 45983f4e62c5..08a03849397a 100644 --- a/docs/codeql/codeql-language-guides/customizing-library-models-for-rust.rst +++ b/docs/codeql/codeql-language-guides/customizing-library-models-for-rust.rst @@ -25,7 +25,29 @@ Syntax used to define an element in an extension file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each model of an element is defined using a data extension where each tuple constitutes a model. -A data extension file to extend the standard Rust queries included with CodeQL is a YAML file with the form: +A data extension file to extend the standard Rust queries included with CodeQL can be written using either JSON or YAML. The JSON format takes the following form: + +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "" + }, + "data": [ + ["tuple", 1], + ["tuple", 2] + // ... + ] + } + ] + } + +Files in the JSON format must use the ``.json`` file extension. Single-line (``//``) and multi-line (``/* ... */``) comments are supported as a non-standard JSON extension. + +A YAML file has the following form: .. code-block:: yaml @@ -34,11 +56,11 @@ A data extension file to extend the standard Rust queries included with CodeQL i pack: codeql/rust-all extensible: data: - - - - + - ["tuple", 1] + - ["tuple", 2] - ... -Each YAML file may contain one or more top-level extensions. +Each data extension file may contain one or more top-level extensions. - ``addsTo`` defines the CodeQL pack name and extensible predicate that the extension is injected into. - ``data`` defines one or more rows of tuples that are injected as values into the extensible predicate. The number of columns and their types must match the definition of the extensible predicate. @@ -99,15 +121,21 @@ This example shows how the Rust query pack models the first argument of the ``sq We need to add a tuple to the ``sinkModel(path, input, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: sinkModel - data: - - ["sqlx_core::query::query", "Argument[0]", "sql-injection", "manual"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "sinkModel" + }, + "data": [ + ["sqlx_core::query::query", "Argument[0]", "sql-injection", "manual"] + ] + } + ] + } - The first value ``sqlx_core::query::query`` is the canonical path of the function to model. Note that this is the internal module path (``sqlx_core::query::query``), not the public re-export path (``sqlx::query``). - The second value ``Argument[0]`` is the access path to the first argument of the function call, which is the SQL query string. This is the location of the sink. @@ -128,15 +156,21 @@ This example shows how the Rust query pack models the return value of the ``reqw We need to add a tuple to the ``sourceModel(path, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: sourceModel - data: - - ["reqwest::get", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "remote", "manual"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "sourceModel" + }, + "data": [ + ["reqwest::get", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "remote", "manual"] + ] + } + ] + } - The first value ``reqwest::get`` is the canonical path of the function. - The second value ``ReturnValue.Future.Field[core::result::Result::Ok(0)]`` is the access path to the output. This compound path is read left to right: @@ -162,14 +196,21 @@ This example shows how the Rust query pack models the return value of ``std::env We need to add a tuple to the ``sourceModel(path, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: sourceModel - data: - - ["std::env::var", "ReturnValue.Field[core::result::Result::Ok(0)]", "environment", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "sourceModel" + }, + "data": [ + ["std::env::var", "ReturnValue.Field[core::result::Result::Ok(0)]", "environment", "manual"] + ] + } + ] + } - The first value ``std::env::var`` is the canonical path to the ``var`` function in the ``std::env`` module. - The second value ``ReturnValue.Field[core::result::Result::Ok(0)]`` selects the ``Ok`` variant of the returned ``Result``. @@ -190,15 +231,21 @@ This example shows how the Rust query pack models taint flow through the ``text` We need to add a tuple to the ``summaryModel(path, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: summaryModel - data: - - ["::text", "Argument[self]", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "taint", "manual"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "summaryModel" + }, + "data": [ + ["::text", "Argument[self]", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "taint", "manual"] + ] + } + ] + } - The first value ``::text`` is the canonical path. Note the format ``::method`` used for inherent methods. Also note that the canonical path uses the internal module path ``reqwest::response::Response``, not just ``reqwest::Response``. - The second value ``Argument[self]`` is the access path to the input. ``Argument[self]`` refers to the receiver of the method call (``response`` in the example). @@ -222,15 +269,22 @@ This example shows how the Rust query pack models taint flow through the ``join` We need to add tuples to the ``summaryModel(path, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: summaryModel - data: - - ["::join", "Argument[self].Reference", "ReturnValue", "taint", "manual"] - - ["::join", "Argument[0]", "ReturnValue", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "summaryModel" + }, + "data": [ + ["::join", "Argument[self].Reference", "ReturnValue", "taint", "manual"], + ["::join", "Argument[0]", "ReturnValue", "taint", "manual"] + ] + } + ] + } Since we are adding flow through a method, we need to add tuples to the ``summaryModel`` extensible predicate. Each tuple defines flow from one input to the output. The first row defines flow from the receiver and the second row defines flow from the first argument. @@ -263,15 +317,21 @@ This example shows how the Rust query pack models a more complex flow through a We need to add tuples to the ``summaryModel(path, input, output, kind, provenance)`` extensible predicate by updating a data extension file: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: summaryModel - data: - - ["::map", "Argument[self].Element", "Argument[0].Parameter[0]", "value", "manual"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "summaryModel" + }, + "data": [ + ["::map", "Argument[self].Element", "Argument[0].Parameter[0]", "value", "manual"] + ] + } + ] + } - The first value ``::map`` is the canonical path. The ``::method`` form matches any type that implements the ``Iterator`` trait. - The second value ``Argument[self].Element`` is the access path to the input — the elements of the iterator (the receiver). @@ -290,14 +350,21 @@ This example shows how the Rust query pack models the ``Option::map`` method as A neutral model prevents generated or inherited models of a specific category (``source``, ``sink``, or ``summary``) from being applied to a callable. This is useful when an automatically generated model incorrectly identifies a callable as, for example, a sink. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: neutralModel - data: - - ["::map", "sink", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "neutralModel" + }, + "data": [ + ["::map", "sink", "manual"] + ] + } + ] + } Since we are adding a neutral model, we need to add a tuple to the ``neutralModel`` extensible predicate. The tuple has three values: @@ -322,15 +389,21 @@ Consider a hypothetical function ``my_crate::sanitize::escape_sql`` which escape We need to add a tuple to the ``barrierModel(path, output, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: barrierModel - data: - - ["my_crate::sanitize::escape_sql", "ReturnValue", "sql-injection", "manual"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "barrierModel" + }, + "data": [ + ["my_crate::sanitize::escape_sql", "ReturnValue", "sql-injection", "manual"] + ] + } + ] + } - The first value ``my_crate::sanitize::escape_sql`` is the canonical path of the function. - The second value ``ReturnValue`` is the access path to the output of the barrier, which means that the return value is considered sanitized. @@ -356,15 +429,21 @@ Consider a hypothetical function ``my_crate::validate::is_safe_path`` which retu We need to add a tuple to the ``barrierGuardModel(path, input, acceptingValue, kind, provenance)`` extensible predicate by updating a data extension file. -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: barrierGuardModel - data: - - ["my_crate::validate::is_safe_path", "Argument[0]", "true", "path-injection", "manual"] - +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["my_crate::validate::is_safe_path", "Argument[0]", "true", "path-injection", "manual"] + ] + } + ] + } - The first value ``my_crate::validate::is_safe_path`` is the canonical path of the function. - The second value ``Argument[0]`` is the access path to the input whose flow is blocked. In this case, the first argument to the function (``user_path`` in the example). @@ -399,14 +478,21 @@ Adds a new taint source. Most taint-tracking queries will use the new source. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: sourceModel - data: - - ["std::env::var", "ReturnValue.Field[core::result::Result::Ok(0)]", "environment", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "sourceModel" + }, + "data": [ + ["std::env::var", "ReturnValue.Field[core::result::Result::Ok(0)]", "environment", "manual"] + ] + } + ] + } sinkModel(path, input, kind, provenance) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -420,14 +506,21 @@ Adds a new taint sink. Sinks are query-specific and will typically affect one or Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: sinkModel - data: - - ["sqlx_core::query::query", "Argument[0]", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "sinkModel" + }, + "data": [ + ["sqlx_core::query::query", "Argument[0]", "sql-injection", "manual"] + ] + } + ] + } summaryModel(path, input, output, kind, provenance) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -442,14 +535,21 @@ Adds flow through a function or method call. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: summaryModel - data: - - ["::text", "Argument[self]", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "taint", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "summaryModel" + }, + "data": [ + ["::text", "Argument[self]", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "taint", "manual"] + ] + } + ] + } neutralModel(path, kind, provenance) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -462,14 +562,21 @@ Prevents generated or inherited models of the specified category from being appl Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: neutralModel - data: - - ["::map", "sink", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "neutralModel" + }, + "data": [ + ["::map", "sink", "manual"] + ] + } + ] + } barrierModel(path, output, kind, provenance) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -483,14 +590,21 @@ Adds a new barrier that stops the flow of taint at the specified element. Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: barrierModel - data: - - ["my_crate::sanitize::escape_sql", "ReturnValue", "sql-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "barrierModel" + }, + "data": [ + ["my_crate::sanitize::escape_sql", "ReturnValue", "sql-injection", "manual"] + ] + } + ] + } barrierGuardModel(path, input, acceptingValue, kind, provenance) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -505,14 +619,21 @@ Adds a new barrier guard that stops the flow of taint when a conditional check i Example: -.. code-block:: yaml - - extensions: - - addsTo: - pack: codeql/rust-all - extensible: barrierGuardModel - data: - - ["my_crate::validate::is_safe_path", "Argument[0]", "true", "path-injection", "manual"] +.. code-block:: json + + { + "extensions": [ + { + "addsTo": { + "pack": "codeql/rust-all", + "extensible": "barrierGuardModel" + }, + "data": [ + ["my_crate::validate::is_safe_path", "Argument[0]", "true", "path-injection", "manual"] + ] + } + ] + } Access paths ------------