fix: correct two user-manual examples; write invisible characters as escapes

CHANGELOG.md

@@ -134,6 +134,13 @@ All notable changes to ForgeLM are documented here.
   with `--tokenizer` and show the real `summarize_result` output, with `--output` resolved to the absolute path the tool
   prints, instead of a layout the tool never prints (#395).
 
+- Correct the PII-masking and GDPR-erasure pages of the user manual, in English and Turkish. The PII-masking page shows
+  the real `forgelm ingest --pii-mask` and `forgelm audit` output, and its audit example now reads an unmasked corpus:
+  auditing the masked output prints no `PII` line, so the old example's counts could not have come from it. The
+  GDPR-erasure page no longer says the identifier never leaves the operator's terminal: the audit chain records a salted
+  digest instead, but the identifier is still typed on the command line and `forgelm reverse-pii` prints the matching
+  corpus lines unmasked.
+
 - Make the GDPR, erasure, retention, PII-masking, GPU-cost and chat documentation say what ForgeLM does, so a deployer can copy a
   documented command or YAML block, and quote a documented limit in a DPIA or an access reply, without it being untrue. The GDPR
   / KVKK manual page was rewritten earlier in this cycle; this completes it and the pages around it (#101):

docs/usermanuals/en/compliance/gdpr-erasure.md

@@ -5,7 +5,7 @@ description: Support Article 15 (access) and Article 17 (erasure) requests with
 
 # GDPR Erasure & Access
 
-ForgeLM ships sister subcommands that support two common GDPR data-subject requests: `forgelm purge` removes rows (one step of an Article 17 erasure) and `forgelm reverse-pii` finds rows (an input to an Article 15 response). Neither is a complete response on its own. They share a per-output-dir salt so a compliance reviewer can correlate erasures and access requests for the same data subject in a single tamper-evident audit chain — without the cleartext identifier ever leaving the operator's terminal.
+ForgeLM ships sister subcommands that support two common GDPR data-subject requests: `forgelm purge` removes rows (one step of an Article 17 erasure) and `forgelm reverse-pii` finds rows (an input to an Article 15 response). Neither is a complete response on its own. They share a per-output-dir salt so a compliance reviewer can correlate erasures and access requests for the same data subject in a single tamper-evident audit chain. The chain records a salted digest instead of the identifier; the identifier itself is still typed on the command line, and `reverse-pii` prints the matching corpus lines unmasked.
 
 This page is the operator quick-reference. The deeper deployer-flow walkthrough lives in the [GDPR erasure deployer guide on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/gdpr_erasure.md); per-flag references are in the in-manual [CLI Reference](#/reference/cli) and the dedicated subcommand pages [`purge_subcommand.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/purge_subcommand.md) + [`reverse_pii_subcommand.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/reverse_pii_subcommand.md) on GitHub.
 

docs/usermanuals/en/data/pii-masking.md

@@ -31,7 +31,16 @@ $ forgelm ingest ./policies/ \
     --recursive --strategy markdown \
     --pii-mask \
     --output data/policies.jsonl
-✓ masked 18 PII matches across 12,240 chunks
+Ingestion summary
+  Output JSONL  : /srv/project/data/policies.jsonl
+  Files (in/out): processed=47  skipped=0
+  Chunks        : 12240
+  Total chars   : 28410553
+  By format     : docx=8, md=27, pdf=12
+  Note          : PII masking redacted 18 span(s): email=11, phone=7
+
+Next: forgelm audit /srv/project/data/policies.jsonl --output ./audit/
+Or train: forgelm --config <your.yaml>  (set data.dataset_name_or_path: /srv/project/data/policies.jsonl)
 ```
 
 After ingest, every match is replaced with a placeholder:
@@ -63,22 +72,25 @@ If you need stricter detection — for instance, a high-stakes legal corpus —
 
 ## Audit-only mode
 
-To detect without modifying:
+To detect without modifying, audit a corpus that was ingested without `--pii-mask`. Here `data/policies-raw.jsonl` holds the same documents as the example above:
 
 ```shell
-$ forgelm audit data/policies.jsonl
+$ forgelm audit data/policies-raw.jsonl
 Data audit summary
-  Source        : data/policies.jsonl
+  Source        : /srv/project/data/policies-raw.jsonl
   Total samples : 12240
   Splits        : train
   └─ train: n=12240
      length  min=24 max=1180 mean=212.4 p95=640
      languages (top-3): en=12240
-     PII             : email=18, phone=4
+     PII             : email=11, phone=7
   PII severity   : worst tier = medium
-     by tier      : medium=18, low=4
+     by tier      : medium=11, low=7
+Report written to: audit/data_audit_report.json
 ```
 
+Auditing the masked `data/policies.jsonl` instead prints no `PII` line, because the same patterns no longer match it.
+
 The audit report carries counts per category and split, not row indices or offsets. To find the rows that hold a specific identifier, search for it with `forgelm reverse-pii`.
 
 ## Regional formats

docs/usermanuals/tr/compliance/gdpr-erasure.md

@@ -5,7 +5,7 @@ description: `forgelm purge` ve `forgelm reverse-pii` subcommand'ları ile Madde
 
 # GDPR Silme & Erişim
 
-ForgeLM, iki yaygın GDPR veri-sahibi talebini destekleyen kardeş subcommand'lar ship eder: `forgelm purge` satır kaldırır (Madde 17 silmenin bir adımı) ve `forgelm reverse-pii` satır bulur (Madde 15 yanıtının bir girdisi). Hiçbiri tek başına eksiksiz bir yanıt değildir. Per-output-dir salt'ı paylaşırlar; böylece compliance reviewer aynı veri sahibi için silme ve erişim taleplerini tek bir tamper-evident audit zincirinde — cleartext identifier operatör terminalinden hiç çıkmadan — korele edebilir.
+ForgeLM, iki yaygın GDPR veri-sahibi talebini destekleyen kardeş subcommand'lar ship eder: `forgelm purge` satır kaldırır (Madde 17 silmenin bir adımı) ve `forgelm reverse-pii` satır bulur (Madde 15 yanıtının bir girdisi). Hiçbiri tek başına eksiksiz bir yanıt değildir. Per-output-dir salt'ı paylaşırlar; böylece compliance reviewer aynı veri sahibi için silme ve erişim taleplerini tek bir tamper-evident audit zincirinde korele edebilir. Zincir identifier yerine onun salt'lı bir digest'ini kaydeder; identifier'ın kendisi yine komut satırına yazılır ve `reverse-pii` eşleşen corpus satırlarını maskelemeden yazdırır.
 
 Bu sayfa operatör hızlı-referansıdır. Daha derin deployer akışı [GitHub'daki GDPR silme deployer kılavuzunda](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/gdpr_erasure-tr.md) bulunur; flag-başına referanslar manuel içindeki [CLI Referansı](#/reference/cli) ve GitHub'daki özel subcommand sayfaları [`purge_subcommand-tr.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/purge_subcommand-tr.md) + [`reverse_pii_subcommand-tr.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/reverse_pii_subcommand-tr.md) sayfalarındadır.
 

docs/usermanuals/tr/data/pii-masking.md

@@ -31,7 +31,16 @@ $ forgelm ingest ./policies/ \
     --recursive --strategy markdown \
     --pii-mask \
     --output data/policies.jsonl
-✓ 12,240 chunk üzerinde 18 PII eşleşmesi maskelendi
+Ingestion summary
+  Output JSONL  : /srv/project/data/policies.jsonl
+  Files (in/out): processed=47  skipped=0
+  Chunks        : 12240
+  Total chars   : 28410553
+  By format     : docx=8, md=27, pdf=12
+  Note          : PII masking redacted 18 span(s): email=11, phone=7
+
+Next: forgelm audit /srv/project/data/policies.jsonl --output ./audit/
+Or train: forgelm --config <your.yaml>  (set data.dataset_name_or_path: /srv/project/data/policies.jsonl)
 ```
 
 Ingest sonrası her eşleşme bir placeholder'la değiştirilir:
@@ -63,22 +72,25 @@ Daha katı tespit gerekirse — örneğin yüksek-stake bir hukuk corpus'u — m
 
 ## Sadece-audit modu
 
-Modifiye etmeden tespit için:
+Modifiye etmeden tespit için `--pii-mask` olmadan ingest edilmiş bir corpus'u audit edin. Burada `data/policies-raw.jsonl`, yukarıdaki örnekle aynı dokümanları içerir:
 
 ```shell
-$ forgelm audit data/policies.jsonl
+$ forgelm audit data/policies-raw.jsonl
 Data audit summary
-  Source        : data/policies.jsonl
+  Source        : /srv/project/data/policies-raw.jsonl
   Total samples : 12240
   Splits        : train
   └─ train: n=12240
      length  min=24 max=1180 mean=212.4 p95=640
      languages (top-3): en=12240
-     PII             : email=18, phone=4
+     PII             : email=11, phone=7
   PII severity   : worst tier = medium
-     by tier      : medium=18, low=4
+     by tier      : medium=11, low=7
+Report written to: audit/data_audit_report.json
 ```
 
+Bunun yerine maskelenmiş `data/policies.jsonl` audit edilirse hiç `PII` satırı yazdırılmaz, çünkü aynı pattern'ler artık onda eşleşmez.
+
 Audit raporu kategori ve split başına sayımları taşır; satır indislerini ya da offset'leri değil. Belirli bir tanımlayıcıyı içeren satırları bulmak için onu `forgelm reverse-pii` ile arayın.
 
 ## Bölgesel formatlar

forgelm/ingestion.py

@@ -1380,7 +1380,7 @@ def _read_text_with_bom_strip(path: Path, *, encoding: Optional[str] = None) ->
             raw = path.read_text(encoding="utf-8-sig", errors="replace")
     else:
         raw = path.read_text(encoding=encoding, errors="replace")
-    if raw.startswith(""):
+    if raw.startswith("\ufeff"):
         raw = raw[1:]
     return raw
 

tests/test_ingestion_reliability.py

@@ -482,7 +482,7 @@ def test_txt_with_bom_strips_bom(self, tmp_path):
         )
         all_text = "\n".join(_read_jsonl(out))
         # The BOM character U+FEFF must not survive into the JSONL.
-        assert "" not in all_text
+        assert "\ufeff" not in all_text
         assert "Header Line One" in all_text
 
     def test_md_frontmatter_stripped_by_default(self, tmp_path):
@@ -965,7 +965,7 @@ def test_bom_strip_works_on_latin1_fallback(self, tmp_path):
         out = tmp_path / "out.jsonl"
         ingest_path(str(src), output_path=str(out), strategy="paragraph")
         all_text = "\n".join(_read_jsonl(out))
-        assert "" not in all_text
+        assert "\ufeff" not in all_text
         assert "Hello" in all_text
 
     def test_strip_pattern_catches_escape_shape_redos(self, tmp_path):

tests/test_issue_dependencies.py

@@ -1266,9 +1266,9 @@ def test_gh_api_failure_text_is_flattened_and_defanged(monkeypatch: pytest.Monke
 
 
 def test_clean_strips_terminal_escapes_and_invisible_characters() -> None:
-    cleaned = tool._clean("\x1b[31mred\x1b[0m ‮evil\u0007\n::error::x")
+    cleaned = tool._clean("\x1b[31mred\x1b[0m \u202eevil\u0007\n::error::x")
     assert cleaned == "red evil : :error: :x"
-    assert assign_on_comment._clean("\x1b[1mbold\x1b[0m‮") == "bold"
+    assert assign_on_comment._clean("\x1b[1mbold\x1b[0m\u202e") == "bold"
 
 
 def test_gh_api_hands_a_graphql_answer_with_errors_to_the_caller_but_not_a_rest_one(

tests/test_safety_advanced.py

@@ -1502,7 +1502,7 @@ def test_neutralise_control_tokens_handles_nested_and_overlapping_tokens(self):
     def test_neutralise_control_tokens_fails_closed_when_no_separator_is_inert(self):
         from forgelm.safety._score_generation import _neutralise_control_tokens
 
-        tokens = ["<| a|>", "<|​b|>", "<|⁠c|>"]
+        tokens = ["<| a|>", "<|\u200bb|>", "<|\u2060c|>"]
         with pytest.raises(ValueError, match="inert separator"):
             _neutralise_control_tokens("<| a|>", tokens)