fix: parse continue followed by a comment as a continue statement

[ybai08]

Summary

• With continue enabled (Lua.runtime.nonstandardSymbol = ["continue"] or the LuaJIT extensions), continue is a soft keyword. parseAction decides between a statement and a name by looking only at the token right after it (Tokens[Index + 3]). A trailing comment makes that token --, so continue -- note was parsed as the name continue and reported exp-in-action and undefined-global.
• A new side-effect-free helper, peekTokenSkipComment, returns the first token after any comments without moving Index. A short comment (--, or // when it is enabled) runs to the end of the line, so the helper returns the newline after it. For a long comment (--[[ ]], --[==[ ]==], /* */) it returns the token after the closing mark. The soft-keyword check now uses this helper, so continue --[[ c ]] = 1 is still an assignment.
• The changelog Unreleased section has a FIX entry that links continue -- comment rejected with runtime.nonstandardSymbol=["continue"] #3474.

Test plan

• test/parser_test/luajit_ext.lua: new LuaJIT-extension cases (continue -- comment, continue --[[ comment ]], a multi-line --[==[ ]==] comment). They fail without the parser change (EXP_IN_ACTION) and pass with it.
• New TestIndividual('continue', ...) cases for Lua 5.4 + nonstandardSymbol = { continue = true }, including the reported snippet. They also guard continue = 1 and continue --[[ comment ]] = 1 staying assignments. A TestIndividualDisabled case covers the setting being off.
• bin/lua-language-server test.lua --name=parser_test passes.
• Full bin/lua-language-server test.lua passes (test finish.), on darwin-arm64 only.

Fixes #3474

[github-actions]

中文：peekTokenSkipComment 在处理短注释时，repeat ... until not token or NLMap[token] 循环依赖 Tokens[index + 1] 恰好是换行符 token 来终止。若文件在短注释后直接结束（无换行），token 为 nil 时循环退出并返回 nil，行为正确；但若短注释后紧跟的是非换行 token（例如 continue --x end，注释与 end 之间无换行），循环会继续向后扫描，可能越过 end 找到后续 token，从而把本应是变量名的 continue 误判为语句，或反之。建议在短注释分支中显式返回“行尾”语义（例如返回一个表示换行的哨兵或直接返回 NLMap 判定结果），而不是依赖 token 流中一定存在换行 token。
English: In peekTokenSkipComment, the short-comment branch relies on Tokens[index + 1] being a newline token to terminate the repeat ... until not token or NLMap[token] loop. If the file ends right after a short comment (no newline), token is nil and it returns nil, which is fine; but if a non-newline token follows the short comment (e.g. continue --x end with no newline between comment and end), the loop keeps scanning past end and may find a later token, misclassifying continue as a statement (or vice versa). Recommend explicitly returning "end-of-line" semantics in the short-comment branch (e.g. a newline sentinel or the NLMap result directly) rather than depending on a newline token always existing in the token stream.

中文：peekTokenSkipComment 中长注释/长字符串分支使用 sfind(Lua, finishMark, Tokens[index] + 2, true) 查找结束标记，但 Tokens[index] 是注释起始 token 的位置。对于 --[[，起始 token -- 位于 Tokens[index]，[[ 紧随其后，从 Tokens[index] + 2 开始搜索 ]] 是正确的；但对于 --[==[，finishMark 为 ]==]，从 Tokens[index] + 2 开始搜索也能覆盖。然而 /* ... */ 分支中 Tokens[index] 是 /* 的位置，从 +2 开始搜索 */ 正确。请确认 Tokens 中位置语义（是 token 起始偏移还是结束偏移）与 +2 的假设一致，否则可能漏检或误检结束标记，导致跨行注释解析错误。
English: In peekTokenSkipComment, the long-comment/long-string branch uses sfind(Lua, finishMark, Tokens[index] + 2, true) to find the closing marker, where Tokens[index] is the position of the comment-start token. For --[[, the start token -- is at Tokens[index] and [[ follows, so searching ]] from Tokens[index] + 2 is correct; for --[==[, finishMark is ]==] and searching from Tokens[index] + 2 also covers it. For /* ... */, Tokens[index] is the position of /* and searching */ from +2 is correct. Please confirm the position semantics in Tokens (token start offset vs. end offset) match the +2 assumption; otherwise the closing marker may be missed or falsely matched, causing incorrect parsing of multi-line comments.

中文：peekTokenSkipComment 只跳过了注释，但没有处理 continue 与注释之间可能存在的空白/换行。例如 continue\n-- comment\nend：Index + 2 指向换行 token，peekTokenSkipComment 返回该换行 token，NLMap[nextToken] 为真，判定为 continue 语句，正确。但 continue -- comment\nend 中，Index + 2 指向 --，函数跳过注释后返回换行 token，也正确。请补充测试覆盖 continue 后跟注释再跟 ;、continue 后跟注释再跟 =（应作为变量名）等边界情况，以验证 soft keyword 判定在注释介入后仍然正确。
English: peekTokenSkipComment only skips comments but does not handle whitespace/newlines that may exist between continue and the comment. For continue\n-- comment\nend: Index + 2 points to the newline token, peekTokenSkipComment returns it, NLMap[nextToken] is true, and it is correctly treated as a continue statement. For continue -- comment\nend, Index + 2 points to --, the function skips the comment and returns the newline token, also correct. Please add tests covering edge cases such as continue followed by a comment then ;, and continue followed by a comment then = (which should be treated as a variable name), to verify the soft-keyword decision remains correct when comments intervene.

中文：测试用例 TestIndividual('continue', 'continue --[[ comment ]] = 1') 期望 continue 作为变量名，但 peekTokenSkipComment 跳过 --[[ comment ]] 后返回 =，= 不在 NLMap 中且不是 ;/块结束符，因此判定为变量名，符合预期。但请确认 TestIndividual 的断言机制确实验证了“作为变量名”而非仅“不报错”，否则该测试可能无法捕获回归。
English: The test TestIndividual('continue', 'continue --[[ comment ]] = 1') expects continue to be treated as a variable name; peekTokenSkipComment skips --[[ comment ]] and returns =, which is not in NLMap and is not ;/block terminator, so it is treated as a variable name, as expected. However, please confirm that TestIndividual's assertion mechanism actually verifies "treated as a variable name" rather than merely "no error"; otherwise this test may not catch regressions.

中文：peekTokenSkipComment 的 repeat index = index + 2 until not Tokens[index] or Tokens[index] > finishOffset 循环在 Tokens[index] 为 nil 时退出，但若 finishOffset 落在某个 token 内部（例如长注释结束标记与后续 token 重叠），循环可能跳过过多 token。建议增加对 Tokens[index] 为 nil 的显式处理并返回 nil，避免后续逻辑误用。
English: The loop repeat index = index + 2 until not Tokens[index] or Tokens[index] > finishOffset in peekTokenSkipComment exits when Tokens[index] is nil, but if finishOffset falls inside a token (e.g. the long-comment closing marker overlaps a following token), the loop may skip too many tokens. Recommend explicitly handling Tokens[index] == nil and returning nil to avoid misuse in subsequent logic.

中文：整体上该修复方向正确，新增的 peekTokenSkipComment 复用了 skipComment 的注释识别逻辑（--、//、/*、长括号），但存在代码重复：skipComment 中已有类似的注释跳过逻辑，建议抽取公共辅助函数以减少维护成本，并确保两处对 nonstandardSymbol['//'] 和长括号的处理保持一致。
English: Overall the fix direction is correct; the new peekTokenSkipComment reuses the comment-recognition logic of skipComment (--, //, /*, long brackets), but there is code duplication: skipComment already contains similar comment-skipping logic. Recommend extracting a shared helper to reduce maintenance cost and ensure both places handle nonstandardSymbol['//'] and long brackets consistently.

[GrayWolf64]

I'm overall skeptical about this fix's approach. Cleaner and more robust ways?

[ybai08]

Fair point; the hand-written token walk duplicated what skipComment already does. Reworked in bb4027a:

• peekTokenSkipComment is gone.
• In parseAction, after continue the parser takes a getSavePoint(), steps past the keyword, skips comments with the existing skipComment() (only comments, so a newline still ends the check), reads the next token, then restores the save point and drops the comment entries it just recorded so parseBreak/the expression path record them once as usual.
• The check on the next token is the same list as before (nil, newline, ;, end, else, elseif, until, )).

That also handles a multi-line long comment correctly: continue --[[ multi\nline ]] = 1 is still an assignment to continue, now covered by a test. (On the bot's note: in continue --x end the end is part of the short comment, so there's no case where a non-newline token follows a short comment on the same line.)

test.lua --name=parser_test, --name=diagnostics, --name=hover and --name=basic pass; on the base commit parser_test fails on while true do continue -- comment\nend.

[GrayWolf64]

1. You seem to be an AI agent
2. the branch is now bulky with special continue handling. judging from the style, a separate parseContinue is definitely better