diff --git a/CHANGELOG.md b/CHANGELOG.md index 146f48ba..9a7e929e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -134,6 +134,13 @@ All notable changes to ForgeLM are documented here. with `--tokenizer` and show the real `summarize_result` output, with `--output` resolved to the absolute path the tool prints, instead of a layout the tool never prints (#395). +- Correct the PII-masking and GDPR-erasure pages of the user manual, in English and Turkish. The PII-masking page shows + the real `forgelm ingest --pii-mask` and `forgelm audit` output, and its audit example now reads an unmasked corpus: + auditing the masked output prints no `PII` line, so the old example's counts could not have come from it. The + GDPR-erasure page no longer says the identifier never leaves the operator's terminal: the audit chain records a salted + digest instead, but the identifier is still typed on the command line and `forgelm reverse-pii` prints the matching + corpus lines unmasked. + - Make the GDPR, erasure, retention, PII-masking, GPU-cost and chat documentation say what ForgeLM does, so a deployer can copy a documented command or YAML block, and quote a documented limit in a DPIA or an access reply, without it being untrue. The GDPR / KVKK manual page was rewritten earlier in this cycle; this completes it and the pages around it (#101): diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index e928cec0..27fb4c85 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -5,7 +5,7 @@ description: Support Article 15 (access) and Article 17 (erasure) requests with # GDPR Erasure & Access -ForgeLM ships sister subcommands that support two common GDPR data-subject requests: `forgelm purge` removes rows (one step of an Article 17 erasure) and `forgelm reverse-pii` finds rows (an input to an Article 15 response). Neither is a complete response on its own. They share a per-output-dir salt so a compliance reviewer can correlate erasures and access requests for the same data subject in a single tamper-evident audit chain — without the cleartext identifier ever leaving the operator's terminal. +ForgeLM ships sister subcommands that support two common GDPR data-subject requests: `forgelm purge` removes rows (one step of an Article 17 erasure) and `forgelm reverse-pii` finds rows (an input to an Article 15 response). Neither is a complete response on its own. They share a per-output-dir salt so a compliance reviewer can correlate erasures and access requests for the same data subject in a single tamper-evident audit chain. The chain records a salted digest instead of the identifier; the identifier itself is still typed on the command line, and `reverse-pii` prints the matching corpus lines unmasked. This page is the operator quick-reference. The deeper deployer-flow walkthrough lives in the [GDPR erasure deployer guide on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/gdpr_erasure.md); per-flag references are in the in-manual [CLI Reference](#/reference/cli) and the dedicated subcommand pages [`purge_subcommand.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/purge_subcommand.md) + [`reverse_pii_subcommand.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/reverse_pii_subcommand.md) on GitHub. diff --git a/docs/usermanuals/en/data/pii-masking.md b/docs/usermanuals/en/data/pii-masking.md index 776dbc30..0afdd3e8 100644 --- a/docs/usermanuals/en/data/pii-masking.md +++ b/docs/usermanuals/en/data/pii-masking.md @@ -31,7 +31,16 @@ $ forgelm ingest ./policies/ \ --recursive --strategy markdown \ --pii-mask \ --output data/policies.jsonl -✓ masked 18 PII matches across 12,240 chunks +Ingestion summary + Output JSONL : /srv/project/data/policies.jsonl + Files (in/out): processed=47 skipped=0 + Chunks : 12240 + Total chars : 28410553 + By format : docx=8, md=27, pdf=12 + Note : PII masking redacted 18 span(s): email=11, phone=7 + +Next: forgelm audit /srv/project/data/policies.jsonl --output ./audit/ +Or train: forgelm --config (set data.dataset_name_or_path: /srv/project/data/policies.jsonl) ``` After ingest, every match is replaced with a placeholder: @@ -63,22 +72,25 @@ If you need stricter detection — for instance, a high-stakes legal corpus — ## Audit-only mode -To detect without modifying: +To detect without modifying, audit a corpus that was ingested without `--pii-mask`. Here `data/policies-raw.jsonl` holds the same documents as the example above: ```shell -$ forgelm audit data/policies.jsonl +$ forgelm audit data/policies-raw.jsonl Data audit summary - Source : data/policies.jsonl + Source : /srv/project/data/policies-raw.jsonl Total samples : 12240 Splits : train └─ train: n=12240 length min=24 max=1180 mean=212.4 p95=640 languages (top-3): en=12240 - PII : email=18, phone=4 + PII : email=11, phone=7 PII severity : worst tier = medium - by tier : medium=18, low=4 + by tier : medium=11, low=7 +Report written to: audit/data_audit_report.json ``` +Auditing the masked `data/policies.jsonl` instead prints no `PII` line, because the same patterns no longer match it. + The audit report carries counts per category and split, not row indices or offsets. To find the rows that hold a specific identifier, search for it with `forgelm reverse-pii`. ## Regional formats diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 6c3eba3f..2124db61 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -5,7 +5,7 @@ description: `forgelm purge` ve `forgelm reverse-pii` subcommand'ları ile Madde # GDPR Silme & Erişim -ForgeLM, iki yaygın GDPR veri-sahibi talebini destekleyen kardeş subcommand'lar ship eder: `forgelm purge` satır kaldırır (Madde 17 silmenin bir adımı) ve `forgelm reverse-pii` satır bulur (Madde 15 yanıtının bir girdisi). Hiçbiri tek başına eksiksiz bir yanıt değildir. Per-output-dir salt'ı paylaşırlar; böylece compliance reviewer aynı veri sahibi için silme ve erişim taleplerini tek bir tamper-evident audit zincirinde — cleartext identifier operatör terminalinden hiç çıkmadan — korele edebilir. +ForgeLM, iki yaygın GDPR veri-sahibi talebini destekleyen kardeş subcommand'lar ship eder: `forgelm purge` satır kaldırır (Madde 17 silmenin bir adımı) ve `forgelm reverse-pii` satır bulur (Madde 15 yanıtının bir girdisi). Hiçbiri tek başına eksiksiz bir yanıt değildir. Per-output-dir salt'ı paylaşırlar; böylece compliance reviewer aynı veri sahibi için silme ve erişim taleplerini tek bir tamper-evident audit zincirinde korele edebilir. Zincir identifier yerine onun salt'lı bir digest'ini kaydeder; identifier'ın kendisi yine komut satırına yazılır ve `reverse-pii` eşleşen corpus satırlarını maskelemeden yazdırır. Bu sayfa operatör hızlı-referansıdır. Daha derin deployer akışı [GitHub'daki GDPR silme deployer kılavuzunda](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/gdpr_erasure-tr.md) bulunur; flag-başına referanslar manuel içindeki [CLI Referansı](#/reference/cli) ve GitHub'daki özel subcommand sayfaları [`purge_subcommand-tr.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/purge_subcommand-tr.md) + [`reverse_pii_subcommand-tr.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/reverse_pii_subcommand-tr.md) sayfalarındadır. diff --git a/docs/usermanuals/tr/data/pii-masking.md b/docs/usermanuals/tr/data/pii-masking.md index e5f2d4ed..c5656f65 100644 --- a/docs/usermanuals/tr/data/pii-masking.md +++ b/docs/usermanuals/tr/data/pii-masking.md @@ -31,7 +31,16 @@ $ forgelm ingest ./policies/ \ --recursive --strategy markdown \ --pii-mask \ --output data/policies.jsonl -✓ 12,240 chunk üzerinde 18 PII eşleşmesi maskelendi +Ingestion summary + Output JSONL : /srv/project/data/policies.jsonl + Files (in/out): processed=47 skipped=0 + Chunks : 12240 + Total chars : 28410553 + By format : docx=8, md=27, pdf=12 + Note : PII masking redacted 18 span(s): email=11, phone=7 + +Next: forgelm audit /srv/project/data/policies.jsonl --output ./audit/ +Or train: forgelm --config (set data.dataset_name_or_path: /srv/project/data/policies.jsonl) ``` Ingest sonrası her eşleşme bir placeholder'la değiştirilir: @@ -63,22 +72,25 @@ Daha katı tespit gerekirse — örneğin yüksek-stake bir hukuk corpus'u — m ## Sadece-audit modu -Modifiye etmeden tespit için: +Modifiye etmeden tespit için `--pii-mask` olmadan ingest edilmiş bir corpus'u audit edin. Burada `data/policies-raw.jsonl`, yukarıdaki örnekle aynı dokümanları içerir: ```shell -$ forgelm audit data/policies.jsonl +$ forgelm audit data/policies-raw.jsonl Data audit summary - Source : data/policies.jsonl + Source : /srv/project/data/policies-raw.jsonl Total samples : 12240 Splits : train └─ train: n=12240 length min=24 max=1180 mean=212.4 p95=640 languages (top-3): en=12240 - PII : email=18, phone=4 + PII : email=11, phone=7 PII severity : worst tier = medium - by tier : medium=18, low=4 + by tier : medium=11, low=7 +Report written to: audit/data_audit_report.json ``` +Bunun yerine maskelenmiş `data/policies.jsonl` audit edilirse hiç `PII` satırı yazdırılmaz, çünkü aynı pattern'ler artık onda eşleşmez. + Audit raporu kategori ve split başına sayımları taşır; satır indislerini ya da offset'leri değil. Belirli bir tanımlayıcıyı içeren satırları bulmak için onu `forgelm reverse-pii` ile arayın. ## Bölgesel formatlar diff --git a/forgelm/ingestion.py b/forgelm/ingestion.py index 8f252328..8f263081 100644 --- a/forgelm/ingestion.py +++ b/forgelm/ingestion.py @@ -1380,7 +1380,7 @@ def _read_text_with_bom_strip(path: Path, *, encoding: Optional[str] = None) -> raw = path.read_text(encoding="utf-8-sig", errors="replace") else: raw = path.read_text(encoding=encoding, errors="replace") - if raw.startswith(""): + if raw.startswith("\ufeff"): raw = raw[1:] return raw diff --git a/tests/test_ingestion_reliability.py b/tests/test_ingestion_reliability.py index 944b7c9e..b0ceb469 100644 --- a/tests/test_ingestion_reliability.py +++ b/tests/test_ingestion_reliability.py @@ -482,7 +482,7 @@ def test_txt_with_bom_strips_bom(self, tmp_path): ) all_text = "\n".join(_read_jsonl(out)) # The BOM character U+FEFF must not survive into the JSONL. - assert "" not in all_text + assert "\ufeff" not in all_text assert "Header Line One" in all_text def test_md_frontmatter_stripped_by_default(self, tmp_path): @@ -965,7 +965,7 @@ def test_bom_strip_works_on_latin1_fallback(self, tmp_path): out = tmp_path / "out.jsonl" ingest_path(str(src), output_path=str(out), strategy="paragraph") all_text = "\n".join(_read_jsonl(out)) - assert "" not in all_text + assert "\ufeff" not in all_text assert "Hello" in all_text def test_strip_pattern_catches_escape_shape_redos(self, tmp_path): diff --git a/tests/test_issue_dependencies.py b/tests/test_issue_dependencies.py index 6ebff869..8f3b2453 100644 --- a/tests/test_issue_dependencies.py +++ b/tests/test_issue_dependencies.py @@ -1266,9 +1266,9 @@ def test_gh_api_failure_text_is_flattened_and_defanged(monkeypatch: pytest.Monke def test_clean_strips_terminal_escapes_and_invisible_characters() -> None: - cleaned = tool._clean("\x1b[31mred\x1b[0m ‮evil\u0007\n::error::x") + cleaned = tool._clean("\x1b[31mred\x1b[0m \u202eevil\u0007\n::error::x") assert cleaned == "red evil : :error: :x" - assert assign_on_comment._clean("\x1b[1mbold\x1b[0m‮") == "bold" + assert assign_on_comment._clean("\x1b[1mbold\x1b[0m\u202e") == "bold" def test_gh_api_hands_a_graphql_answer_with_errors_to_the_caller_but_not_a_rest_one( diff --git a/tests/test_safety_advanced.py b/tests/test_safety_advanced.py index 510c1e0c..b9797dfb 100644 --- a/tests/test_safety_advanced.py +++ b/tests/test_safety_advanced.py @@ -1502,7 +1502,7 @@ def test_neutralise_control_tokens_handles_nested_and_overlapping_tokens(self): def test_neutralise_control_tokens_fails_closed_when_no_separator_is_inert(self): from forgelm.safety._score_generation import _neutralise_control_tokens - tokens = ["<| a|>", "<|​b|>", "<|⁠c|>"] + tokens = ["<| a|>", "<|\u200bb|>", "<|\u2060c|>"] with pytest.raises(ValueError, match="inert separator"): _neutralise_control_tokens("<| a|>", tokens)