[SizzleUnrlsd]

Summary

Closes #272.

The escaping inspection of Django templates now reads each template's markup in one pass that also tracks what the template leaves open at its end and how comments end. Constructs left open, comments and that final context all come from this one pass, so no separate rule or rescan can disagree with it.

The rules:

• Constructs left open: a <script> or <style> block (those names exactly, not <style-switcher>) or an end tag that the template leaves open runs to the end of the template, read once. A variable inside is refused, as in a closed one. This also covers C: a variable in a script left open is refused even where the document ends there and the browser would never run it, an accepted prudent false positive.
• Comments: a comment ends at its close (-->, --!>, or at once for <!--> and <!--->), or at the first template tag inside it, whichever comes first, and what follows that tag is read as markup. An include there may close the comment, and a condition around its <!-- may leave it out. A variable cannot close it, since it is escaped. A comment left open, with no tag after it, hides the rest of the text.
• Final context: a template that ends inside a script or style block, a start tag (or a quoted value) or an end tag hands that construct to the markup following its include. The templates that include it are therefore not established as escaping. A template extending another renders it and ends where it ends, so such an end is handed on through extends too. The extending template's own verdict is unaffected, since its blocks stand where the parent places them, before the parent's end.
• Cost: each construct is matched where it starts, instead of on a copy of the rest of the text, and is read once. The inspection is now linear in the size of the template.

Commit	Change
test(templates): cover markup the inspection already reads right	what must be kept: B, D, a plain comment, comments that close early, a template ending in a comment, an extended one
perf(templates): match markup where it starts instead of on a copy of the rest	match in place; verdicts unchanged
fix(templates): read a script or style block left open to the end of the template	A, C
fix(templates): read an end tag left open to the end of the template	an end tag left open
fix(templates): end a comment at a template tag inside it	G, and D for the right reason
fix(templates): close a comment where a browser closes it	--!>, <!-->, <!--->
fix(templates): let a comment left open hide the rest of the template	a comment left open; linear cost
refactor(templates): name what the inspection of a template establishes	a named tuple instead of a pair; behaviour unchanged
fix(templates): refuse including a template that ends inside a construct	E; one reading returns the verdict and the final context
docs: state how templates left open and comments are read	docs/usage.md
fix(templates): read only script and style elements as raw text	review: custom elements
perf(templates): find a comment close once for the comments before it	review: comments before one close
refactor(templates): record the templates a template extends	extends and includes apart; behaviour unchanged
fix(templates): end a template extending another where that one ends	review: an extended template's open end
docs: state that a template extending another ends where that one ends	docs/usage.md

Review

Three findings on the first ten commits, each reproduced and then fixed in its own commit, with tests that failed before the fix:

An extended template's open end. The three templates were:

• base.html: <p>{% block body %}{% endblock %}</p><script>
• child.html: {% extends "base.html" %}{% block body %}hello{% endblock %}
• page.html: {% include "child.html" %}var name = {{ name }};</script>

Django renders <p>hello</p><script>var name = …;</script>, and Chrome runs it, yet nothing was reported. The open end of base.html is now handed on to the templates extending it, along a chain of extends too. page.html is refused, and the view reports xss. Unlike #273, the block stands before the script, not inside it.

Comments before one close. "<!--{% include 'empty.html' %}" * n + "-->" searched the final close again from each comment, which is quadratic. The close found is now kept for every comment that opens before it.

Characters	main	First ten commits	Now
232,263	0.007 s	0.323 s	0.008 s
464,523	0.012 s	1.241 s	0.017 s
929,043	0.025 s	4.794 s	0.034 s
9,000,003			0.322 s

Custom elements. <style-switcher><p>{{ name }}</p></style-switcher> (or <script-editor>) opened a style or script block, since the tag-name test accepted a word boundary. With nothing closing it, the block ran to the end of the template, and the variable was refused. A tag name now runs to whitespace, / or >, as the tokenizer reads it, so only script and style themselves hold raw text. main accepted this template, and so does this branch now.

Not in this change

#273 stays open. A {% block %} (F) or an {% include %} placed inside another construct inserts markup there that no rule checks.

• F: a block inside the parent's <script>. Still no findings, and the script runs.
• The same family, inside an attribute value: <a title="{% include "q.html" %}{{ name }}">x</a>, with q.html holding " onclick=". Rendered with Django, it becomes <a title="" onclick="alert(1)">x</a>, yet --check still reports no findings.

Results

The reproductions of #272

Each is a Django project with one view rendering page.html with request.GET["name"]:

Project	main	This branch
A, a fragment's <script> closed by its includer	no findings	xss
B, closed in the same template	xss	xss
C, left open to the end of the document	no findings	xss, the prudent false positive
D, a comment closed by an include	xss	xss
E, an includer continuing an included <script>	no findings	xss
G, a closed comment holding an include	no findings	xss
<p>hello</p><!-- {{ name }} -->	escaped	escaped
F, a block inside the parent's <script> (#273)	no findings	no findings

Verdicts

The verdicts before and after the change were compared on 1,618,480 texts: every text of up to five tokens of a small alphabet, plus 500,000 random texts of a wider one. The tokens are comments and their closes, <script>, <style>, tags, end tags, quotes, variables and template tags. Every difference has one of these causes, and none is left unexplained.

Change	Texts	Example (\x00 a variable, \x01 a template tag)
added refusal: variable in a script or style block left open	69,176	<script>\x00
added refusal: variable in an end tag left open	9,797	</a\x00
added refusal: comment closed at once by <!--> or <!--->, or by --!>, where the old reading ran to a later -->	414	<!--><a\x00-->
added refusal: comment ending at a template tag inside it	16	<!--\x01<a\x00-->
lost refusal: comment left open, with no template tag after it, hides the rest of the text	2,543	<!--<a\x00
lost refusal: comment closed by --!>, which the old reading did not close	225	<!--<a\x00--!>

The lost refusals are markup a browser shows nothing of.

Corpus

escaped_templates is identical before and after on the 10 projects of the local corpus that have templates. They include healthchecks (136 of 285 templates escaped), pygoat (101 of 128), Vulnerable_fastapi, uploadbox, DjangoGoat, dvpwa, luigi and flask 3.1.0. The remote battery could not be reached, so superset, sentry, saleor and mitmproxy were not compared.

Cost

templates._inspect was timed in one subprocess per size, with a 60-second limit, on the same machine and under the same load, before (ff49a2c) and after:

Shape	100k before	1M before	1M after	3M after	10M after
"<" * n	0.24 s	16.2 s	0.44 s	1.29 s	4.45 s
"</a" * (n // 3)	1.35 s	> 60 s	0.00 s	0.00 s	0.01 s
"<!--" * (n // 4)	9.90 s	> 60 s	0.00 s	0.01 s	0.04 s
"<script>" * (n // 8)	4.51 s	> 60 s	0.02 s	0.05 s	0.19 s
"<style>" * (n // 7)	4.67 s	> 60 s	0.09 s	0.06 s	0.26 s
"<!--{% if x %}" * (n // 14)	1.11 s	> 60 s	0.06 s	0.20 s	0.62 s
"<p>x</p>" * (n // 8)	0.05 s	3.56 s	0.16 s	0.53 s	1.61 s

--check on a project whose only file is a 300,000-character template of each shape takes 0.2 to 0.3 s. On main, five of these shapes were measured earlier, under heavier load (#272): 0.5 s for the well-formed control and 1.6 to 66.7 s for the others. These timings come from Python 3.12.15 on macOS with an Apple M3.

Tests

The new file tests/test_template_open_constructs.py has 37 tests, added with the commit whose behaviour each one checks. Each inspection and analysis runs in a subprocess with a timeout. No existing test changes, and no regression snapshot changes.

Tests	What they check	On main
Reproductions (6)	A, B, C, D, E and G report xss	A, C, E and G failed
Plain comment (1)	<!-- {{ name }} --> stays escaped and reports nothing	passed (kept)
Comment ends (5)	a condition around the opener, an include inside, <!-->, <!---> and --!>	include inside failed
Browser closes (2)	a handler after <!--> or --!>, before a later -->, is refused	failed
Comment left open (1)	with no template tag after it, it hides the rest	failed
Left open (3)	a variable in a <style>, a <script> or an end tag left open is refused	failed
Final context (7)	an includer of a template ending in a script, a style, a start tag, a quoted value or an end tag is refused; one ending in a comment, or an extended one, stays established	the five includers failed
Cost (7)	the seven shapes above at 3,000,000 characters end within the timeout	six failed
Review (5)	custom elements (2) and an extended template's open end, directly and through two extends (2), are read right; comments before one close end within the timeout (1)	all five failed on the first ten commits

Each commit leaves the suite green, and the tests a commit adds fail on the commit before it, those tests and no other. The refactor adds none.

Each of these mutations of the change makes at least one test fail:

• comments ignoring template tags;
• --> as the only close;
• the final context not handed to includers;
• extends treated as an include;
• a comment left open handed on to includers;
• an unclosed comment searched again at each <!--, which brings back the quadratic cost;
• raw text left open no longer read to the end;
• the word boundary back in the tag-name test;
• the close searched again for every comment;
• no open end handed on through extends;
• an open end handed on through one extends only.

Verification

Python 3.12.15 (macOS), this branch on main at ff49a2c:

• mypy and ruff check .: clean;
• unit suite: 2,430 passed;
• regression: 19 of 19;
• Enterprise plugins at b1c4ed3: 978 passed.