[SizzleUnrlsd]

Closes #270. Closes #271.

In the SARIF log, each rule's only text was its id, the confidence the model rates every finding with was dropped, and a finding could not point at a second place. GitHub code scanning, an IDE or CoreTrace's merged log could show a finding's message, but not explain its rule, rank it, or link to where the problem happens.

Changes

• Rule descriptions (SARIF: describe each rule instead of repeating its id #270):
  • Plugins: whoever defines a rule describes it. A plugin declares its rules in Plugin.rules, a mapping from rule id to RuleDescription (a title and what the rule detects). It is optional, so existing plugins load unchanged and their rules are listed by id alone.
  • Engine: the engine describes its own findings in engine.ENGINE_RULES: unsupported-syntax, syntax-error, ambiguous-module and exploitable-vulnerability.
  • Bundled plugins: each declares the rules it reports, and SecretDetector declares the three secret rules.
  • Path to the report: a check gathers these descriptions (engine.rule_descriptions). ProjectAnalysis and FileAnalysis carry them as rules, and the command line passes them to Report.rules.
  • Reporter: the SARIF reporter reads report.rules only. A SARIF rule then carries the title as shortDescription and the description as fullDescription.
• Confidence (SARIF: write each finding's confidence, and the other places it involves #271): each SARIF result carries its finding's confidence (low, medium, high) as the confidence property, as CoreTrace's merged log spells it.
• Related locations (SARIF: write each finding's confidence, and the other places it involves #271):
  • RelatedLocation (a span and a message) and Finding.related let a finding name other places. The SARIF reporter writes them as relatedLocations, numbered from 0.
  • The SARIF message links to each one with an embedded link [text](id), with brackets and backslashes escaped around them. The text and JSON reports keep the message as it is.
  • A taint flow that reaches its sink inside a project function points at the sink: sink python.builtins.eval, reached via relay, where relay is the call the flow goes through, at any depth. The through and sink_line metadata are kept.
  • The cache, which the parallel analysis also goes through, keeps the related locations. CACHE_FORMAT goes to 18, so caches miss once.

Compatibility

• Additive for plugins.
• The JSON report is unchanged.
• In SARIF, properties is now present on every result, with the confidence beside injection, context and origin.
• Three tests compared a result's properties exactly. They now expect the confidence beside the properties they check, and still nothing else, so the guarantee that no other metadata is exported holds.

Tests

Each new test was committed before its implementation and failed for the expected reason:

• test_rule_descriptions:
  • a third-party plugin that declares its rule sees it described in the SARIF log, and one that declares nothing still loads and reports by id;
  • every rule the usage guide documents is described by the bundled plugins;
  • every rule id the code gives a finding, read from the sources (the engine's notes included), is described. This caught syntax-error and ambiguous-module.
• test_reporters:
  • the reporter renders the descriptions the report gives;
  • a rule without one keeps its id alone;
  • the confidence is checked for each level;
  • the related locations are rendered, and the message links to them with escaping.
• test_code_injection: the sink inside a project function is a related location, reached via the first call even two calls away; a sink in the function itself gives none.
• Cache and parallel: test_cache and test_parallel caught that the cache dropped the related locations, before the codec kept them.

Results:

• pytest: 2464 passed, 19 regression tests deselected (network).
• mypy and ruff check . are clean.