Skip to content

Run formatter test cases from files under test-cases/ - #160

Open
abashev wants to merge 6 commits into
21.X.Xfrom
test/formatter-cases
Open

abashev wants to merge 6 commits into
21.X.Xfrom
test/formatter-cases

Conversation

@abashev

@abashev abashev commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Adding a formatter test meant a Java method with two text blocks and a formatter around them, or an .input/.output pair in testdata/. Now it can be one file in open-java-format/src/test/resources/test-cases/. In order: the suite with one test moved into it, every before-and-after test of FormatterTest and JBangDirectivesTest that needs no settings, reading the cases byte for byte, the testdata/ pairs that need no settings, each as an LF, a CRLF and a CR case, a fix that keeps two pairs other modules read, and the pairs whose input has no final line break, which the second dashes can now follow directly.

The case file

A header, six or more dashes that end a line, the code before formatting, a second run of such dashes and the code expected after it. The header may be empty, and then the file starts with the dashes. There are no settings yet, so the header takes blank lines and # comments, which explain the case; any other line fails the case.

# Why this case exists.
----------
class A{}
----------
class A {}

The file is read as it is. The before block runs from the line after the first dashes up to the second ones. On a line of their own they leave it its last line break; at the end of its last line they leave it none, for an input that does not end in one:

----------
class MyClass {
    private void myFile() {}
}----------
class MyClass {
    private void myFile() {}
}

The after block runs from the line after the second dashes to the end of the file, with a final line break or without. Line breaks stay \n, \r\n or \r, whichever the file has, and .gitattributes keeps git from converting them in *.case files. A line of code that ends in six or more dashes would be a separator itself, so it cannot be part of a case; none is, and such a file fails with three separators found.

The suite

FormatterCasesTest finds the .case files anywhere under test-cases/ on the test classpath and runs one test per file, named by its path without the extension. The before block through formatSourceAndFixImports in the OJF style must give the after block, and a second run must change nothing. A failure first names the line of the case file where the output departs from the after block, with both versions of that line, and then shows both texts:

[test-cases/nonNlsMarkersStayOnTheLineOfTheirLiteral.case, first run, line 25
  expected:                 "third one"); //$NON-NLS-6$
  actual:                   "third one"); //$NON-NLS-5$]

The output has to match the after block byte for byte. A file without exactly two lines of dashes fails its own test, and a missing or empty folder fails the suite rather than passing with no tests.

The tests moved into cases

The first commit moves nonNlsMarkersStayOnTheLineOfTheirLiteral. The second moves every test that formatted a string and compared the result with another string in the code, one file per test and one per row of a parameterized test: 23 tests of FormatterTest and all 40 of JBangDirectivesTest, which goes; its cases are test-cases/jbang/<method>-<row>.case. A one-off generator read the strings from the test sources through javac's tree, so none was retyped, and the comments of the tests became the # headers of their cases.

The 19 of them that formatted in the GOOGLE style now run in OJF, the one style the formatter has. Their inputs are unchanged and each expected output is the OJF one; for all but two it equals the old one with every indent doubled, so the test checks the same thing. The two that wrap at the line limit were checked by hand:

  • wrapMarkdownDocstringComment fitted in 120 columns, so its sentence got three more words and wraps again.
  • wrapMarkdownDocstringRunKeepsTheSlashPrefix now wraps right before @Deprecated. On JDK 25 a formatter from before the fix for Code won't compile after Markdown Javadoc is formatted palantir/palantir-java-format#1803 drops the /// there and makes it an annotation, so the case still catches the bug. It was skipped below JDK 23 and now runs on every JDK; every case has the same output on JDK 21, 25 and 27.

wrapLineComment stays in FormatterTest: its output, in GOOGLE as in OJF, changes on a second run, which the cases check. That is a formatter bug, to be fixed separately. It and the seven other tests that stay in FormatterTest but formatted in GOOGLE use an OJF formatter now; only wrapLineComment and blankLinesImportComment change their expected text.

The testdata pairs

FormatterIntegrationTest checked each pair four times: the formatted input against the output, then the output with LF, CR and CRLF line breaks against itself. 249 of the 256 pairs become three cases each, testdata/<name>-lf.case, -crlf.case and -cr.case, the pair byte for byte with those line breaks. A case formats its before block and has to give the after block, then formats that again and has to change nothing, so the CR and CRLF copies check a whole run with their line breaks, not only the output.

The 28 among them whose input does not end in a line break, JavaFileNoNewLine too, put the second dashes right after the input's last character. Seven pairs stay in testdata/:

  • Other modules read two of them: the benchmarks link PalantirDeeplyNestedCalls.java to palantir-deeply-nested-calls.input, and the IntelliJ plugin's PalantirJavaFormatFormattingServiceTest reads A.input and A.output.
  • ModuleImport needs JDK 23.
  • Under formatSourceAndFixImports B21647014 loses an unused import, Unformatted gets its imports sorted, and M and ojf-issue-24-jbang-script change the blank lines around them.

FileBasedTests.VERSIONED_TESTS keeps only ModuleImport.

Test counts on JDK 21

before after
FormatterTest 43 19
JBangDirectivesTest 40 –
FormatterIntegrationTest 1024 28
FormatterCasesTest – 811
CaseFileTest – 16
module 1658 1425

Every other class is unchanged. Each moved test of FormatterTest and JBangDirectivesTest is one case, and each moved pair is its three cases, where the integration test counted four checks; that is why the module has fewer tests. 14 tests are skipped instead of 15, because the slash-prefix case runs below JDK 23 now. ./gradlew test --rerun passes in every module.

Next

The fix for the unstable comment wrap, with wrapLineComment moved into a case; the pairs left in testdata/; and the other classes that still format in GOOGLE, with Formatter.create() defaulting to OJF.

A formatter test was a Java method with two text blocks and a formatter
around them, or an .input/.output pair in testdata/. Now it can be one
file: src/test/resources/test-cases/<name>.case holds the code before
formatting and the code expected after it, separated by a line of six or
more dashes. A file with two such lines starts with a settings block.
There are no settings yet, so that block takes blank lines and # comments,
which explain the case; any other line fails the case.

FormatterCasesTest finds the .case files anywhere under test-cases/ on
the test classpath and runs one test per file, named by its path without
the extension. The before block through formatSourceAndFixImports in the
OJF style must give the after block, and a second run must change
nothing. A failure first names the line of the case file where the output
departs from the after block, with what the case expects there and what
the formatter wrote, and then shows both texts. A file with no separator
or more than two fails its own test, and a missing or empty folder fails
the suite rather than passing with no tests. CaseFile splits the file
with String.lines(), so both blocks end their lines in \n whatever the
file uses.

nonNlsMarkersStayOnTheLineOfTheirLiteral moves from FormatterTest to the
first case. Its blocks were generated from the method's text blocks and
are byte for byte the strings it compared; its comment became the header.
Changing one character of the expected output fails the case at its line.

Tests on JDK 21: FormatterTest 43 -> 42, FormatterCasesTest 0 -> 1 under
the same name, CaseFileTest 0 -> 12, every other class unchanged; the
module goes from 1658 to 1670 tests, 15 skipped before and after.
…est into test-cases/

Every test that formatted a string and compared the result with a second
string in the code becomes a case file, one per test and one per row of
a parameterized test: 23 tests of FormatterTest and all 40 of
JBangDirectivesTest, which goes. A one-off generator read the strings
from the test sources through javac's tree, so none was retyped, and
the comments of the tests became the # headers of their cases.

The 19 of them that formatted in the GOOGLE style now run in OJF, the
one style the formatter has. Their inputs are unchanged and each
expected output is the OJF one; for all but two it equals the old one
with every indent doubled, so the test checks the same thing. The two
that wrap at the line limit were checked by hand.
wrapMarkdownDocstringComment fitted in 120 columns, so its sentence got
three more words and wraps again.
wrapMarkdownDocstringRunKeepsTheSlashPrefix now wraps right before
@deprecated; on JDK 25 a formatter from before the fix for
palantir#1803 drops the /// there, so the case
still catches it. It was skipped below JDK 23 and now runs on every
JDK, with the same output on 21, 25 and 27, as all the cases.

wrapLineComment stays in FormatterTest: its output, in GOOGLE as in OJF,
changes on a second run, which the cases check. It and the seven other
tests that stay in FormatterTest but formatted in GOOGLE use an OJF
formatter now; only wrapLineComment and blankLinesImportComment change
their expected text.

Tests on JDK 21: FormatterTest 42 -> 19, JBangDirectivesTest 40 -> 0,
FormatterCasesTest 1 -> 64, every other class unchanged; 1670 tests
before and after, 14 skipped instead of 15. Each of the 63 old test
names maps to one case.
A case went through String.lines() and came back with \n after every
line, so a file written with \r\n or \r tested the same thing as one
with \n, and a line break was added where a file had none at its end.
CaseFile now cuts the blocks out of the text as it is: the before block
runs from the line after the first line of dashes to the second, its
last line break included, and the after block from the line after the
second to the end of the file, and the formatter's output has to match
it byte for byte. Every case therefore has exactly two lines of dashes,
and one without a header starts with them: the 15 cases that started
with their code got the leading line, the other 49 had a header already.

.gitattributes keeps git from converting the line breaks of *.case
files, which the comparison now sees.

Tests on JDK 21 and 25: CaseFileTest 12 -> 14, the 64 cases unchanged
and passing, 1672 tests in the module. Copies of a case with \r\n and
with \r pass as well.
…LF, CRLF and CR copies

FormatterIntegrationTest checked each pair four times: the formatted
input against the output, then the output with LF, CR and CRLF line
breaks against itself. 223 of the 256 pairs become three cases each,
testdata/<name>-lf, -crlf and -cr, the pair byte for byte with those
line breaks. A case formats its before block and has to give the after
block, then formats that again and has to change nothing, so the CR and
CRLF copies check a whole run with their line breaks, not only the
output.

33 pairs stay. ModuleImport needs JDK 23. Under formatSourceAndFixImports
B21647014 loses an unused import, Unformatted gets its imports sorted,
and M and ojf-issue-24-jbang-script change the blank lines around them.
And 28 inputs, JavaFileNoNewLine among them, do not end in a line break,
which a case cannot hold: its before block ends where the second line of
dashes starts. FileBasedTests.VERSIONED_TESTS keeps the entries of the
pairs that stay.

Tests on JDK 21: FormatterIntegrationTest 1024 -> 132, FormatterCasesTest
64 -> 733, every other class unchanged; the module goes from 1672 to
1449 tests, as four checks per pair become three cases.
The benchmarks link src/test/resources/PalantirDeeplyNestedCalls.java to
palantir-deeply-nested-calls.input, and the IntelliJ plugin's
PalantirJavaFormatFormattingServiceTest reads A.input and A.output by
path. Moving both pairs into cases broke the build: CI failed in
:open-java-format-benchmarks:processTestResources on the dangling link.
The two pairs go back to testdata/ and their six cases go. Level.java
pointed at palantir-expression-lambdas.input, which moved; its comment
now names the cases.

Tests on JDK 21: FormatterIntegrationTest 132 -> 140, FormatterCasesTest
733 -> 727, 1451 tests in the module; ./gradlew test --rerun passes in
every module.
…ashes ending the input

A case could not hold an input that does not end in a line break: its
before block ran up to a line of dashes, so it always ended in one. Now
six or more dashes are a separator wherever they end a line, so the
before block can stop right before them, as in a closing brace followed
by the dashes, and keeps no line break of its own. The price is that a
line of code that ends in six or more dashes cannot be part of a case.
None of the cases or pairs has one, and such a file fails with three
separators found rather than quietly.

The 28 testdata pairs whose input has no final line break, among them
JavaFileNoNewLine, become their LF, CRLF and CR cases like the other
pairs, and FileBasedTests.VERSIONED_TESTS keeps only ModuleImport.
testdata/ now holds the seven pairs that need a setting or that other
modules read.

Tests on JDK 21: CaseFileTest 14 -> 16, FormatterIntegrationTest
140 -> 28, FormatterCasesTest 727 -> 811; 1425 tests in the module.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant