GitHub - flexorch/flexorch-audit: Zero-dependency PII + quality + noise audit for LLM datasets (TR/EU/US)

Zero-dependency PII detection, quality grading, and noise audit for LLM datasets — in a single function call.

Why

Before feeding documents into an LLM pipeline you need to answer three questions:

1. Does this text contain personal data? Sending PII to a language model is a compliance risk.
2. Is the text quality high enough? Short, noisy, or duplicate records hurt fine-tuning and RAG retrieval.
3. How bad is the noise? Garbled encodings and symbol clutter degrade model output silently.

Most tools that answer these questions require heavy NLP frameworks, model weights, or cloud APIs. flexorch-audit answers all three with one call — using only regex and Python's standard library. No model weights, no network calls, no external packages.

Features

• Quality grade — A/B/C/D composite score: is this text LLM-ready at a glance?
• Noise ratio — line-level symbol clutter detection (noise_ratio); values above 0.20 indicate likely extraction artifacts
• PII detection — 45 types across 15 countries (TR, DE, AT, FR, BE, IT, NL, ES, UK, US, PL, PT, SE, DK, FI) + universal types; all regex-based with checksum validation where a checksum algorithm exists
• Batch audit — audit_batch() aggregates duplicate ratio and PII counts across an entire dataset in one call
• Masking — four strategies: redact, replace (synthetic), token, hash
• Zero runtime dependencies — pure Python stdlib, Python 3.10+

Install

pip install flexorch-audit

Quick start

from flexorch_audit import audit, mask

text = open("contract.txt").read()  # extract from PDF/DOCX first

result = audit(text)               # "und" by default — all detectors active
# result = audit(text, locale="tr")  # restrict to TR-only detectors

result.quality_grade      # "B"
result.quality_score      # 0.73  (0.0–1.0 composite)
result.noise_ratio        # 0.04  (fraction of blank/garbage lines; >0.20 = low quality)
result.detected_language  # "und" (locale you passed in; caller controls language)
result.pii_summary        # [{"type": "email", "count": 2}, {"type": "national_id_tr", "count": 1}]

# Full findings and raw metrics — dict access also works:
result["pii"]    # [{"type": "email", "value": "ali@example.com", "start": 8, "end": 23}]
result["quality"]  # {"completeness": 1.0, "avg_length": 342, "duplicate_ratio": None}
result["noise"]    # {"garbage_ratio": 0.0, "encoding_ok": True}

clean = mask(text, result["pii"], strategy="redact")
# "Contact: [MASKED_EMAIL]"

One-shot redaction

from flexorch_audit import redact_for_llm

clean = redact_for_llm("TCKN: 12345678950, email: ali@example.com", locale="tr")
# "TCKN: [MASKED_NATIONAL_ID_TR], email: [MASKED_EMAIL]"

# Different masking strategies
redact_for_llm(text, locale="tr", strategy="token")   # <PII_NATIONAL_ID_TR_1>
redact_for_llm(text, locale="tr", strategy="hash")    # [3d4f9a1b2c8e7f0a]
redact_for_llm(text, locale="tr", strategy="replace") # static synthetic value

No PII found → original text returned unchanged.

Token estimation

from flexorch_audit import estimate_tokens

estimate_tokens("The quick brown fox jumps over the lazy dog.")  # → 16
estimate_tokens("")  # → 0

Heuristic: words × 4/3 — no tiktoken required. Accuracy within ~15% of the real tokenizer for English and most European languages; treat as a planning estimate for context window sizing and cost forecasting.

Batch audit

from flexorch_audit import audit_batch

texts = [record["text"] for record in dataset]
batch = audit_batch(texts)           # locale="und" by default

batch["duplicate_ratio"]    # 0.12 — fraction of exact-duplicate records
batch["avg_quality_score"]  # 0.78
batch["pii_summary"]        # [{"type": "email", "count": 47}, ...]
batch["results"]            # list of AuditResult, one per text

Country coverage

Universal detectors (always active regardless of locale): email · iban · credit_card · ip · ip_v6

Language detection: flexorch-audit is zero-dependency — no language detection library is included. Pass the correct locale yourself, or use "und" (default) to activate all detectors.

PII types

Universal

Turkey (locale="tr")

Germany (locale="de")

France (locale="fr")

Italy (locale="it")

Netherlands (locale="nl")

Spain (locale="es")

United Kingdom (locale="uk")

United States (locale="us")

EU / International (locale="eu")

Austria (locale="at", also active under "de")

Belgium (locale="be", also active under "fr" and "nl")

Poland (locale="pl")

Portugal (locale="pt")

Sweden (locale="sv")

Denmark (locale="da")

Finland (locale="fi")

Structured field masking

Sometimes a value doesn't match any PII pattern (empty cell, non-standard ID format) but the column name itself already tells you it's sensitive. mask_record_fields() masks by key name instead of scanning the value:

from flexorch_audit import mask_record_fields

row = {"tckn": "12345678950", "customer_email": "ali@example.com", "city": "Istanbul"}
mask_record_fields(row)
# {"tckn": "[MASKED_NATIONAL_ID_TR]", "customer_email": "[MASKED_EMAIL]", "city": "Istanbul"}

Matching is case-insensitive and matches on exact key name or substring ("customer_email" matches because it contains "email"). None values and unmatched keys are left untouched. Returns a new dict — the input is not mutated. This does not replace mask() — use it alongside PII span-detection for tabular/JSON data where key names carry signal that regex on the value alone would miss.

Compliance report

from flexorch_audit import compliance_report

result = audit(text, locale="tr")
report = compliance_report(result)
# {
#     "has_pii": True,
#     "pii_types": ["email", "national_id_tr"],
#     "risk_level": "high",           # "none" | "low" | "medium" | "high"
#     "masking_required": True,
#     "recommendations": [
#         "Apply mask(strategy='redact') before storing or sharing this text.",
#         "Review applicable regulations (KVKK Art. 6, GDPR Art. 9) for special category data handling.",
#     ],
# }

A technical summary only — not a legal document or regulatory opinion. risk_level is "high" when any national-ID-class type is found (TCKN, SSN, credit card, etc.), "medium" for contact-info-class types (email, phone, IBAN, name), "low" otherwise, "none" when no PII was found.

RAG helpers

Zero-dependency chunking and indexing-readiness metadata for RAG pipelines:

from flexorch_audit import audit, chunk_text, rag_metadata

result = audit(text, locale="tr")
meta = rag_metadata(result)
# {"should_index": True, "pii_safe": False, "quality_gate": "pass",
#  "recommended_chunk_size": 512, "estimated_chunks": 4}

if meta["should_index"]:
    chunks = chunk_text(text, strategy="paragraph", max_tokens=meta["recommended_chunk_size"])
    # [{"text": "...", "token_count": 118, "chunk_index": 0}, ...]

chunk_text() supports three strategies: "paragraph" (default, splits on blank lines and merges short ones), "sliding_window" (fixed-size overlapping windows via overlap), and "sentence" (splits on sentence boundaries). rag_metadata() reads quality_grade/pii/avg_length off an AuditResult — should_index is True only for grade A/B, and pii_safe is True only when no PII was detected; combine both before indexing into a vector store.

Noise detection

noise_ratio measures the fraction of lines that are blank or contain symbol clutter:

result = audit("clean line\n@@@garbage\n\nclean")
result.noise_ratio   # 0.5  (2 noisy lines out of 4)

A line is "noisy" when it is blank (after strip) or contains 3+ consecutive characters from @ # ! ~ * =.

Masking strategies

clean = mask(text, result["pii"], strategy="redact")   # default
clean = mask(text, result["pii"], strategy="token")
clean = mask(text, result["pii"], strategy="hash")
clean = mask(text, result["pii"], strategy="replace")

Quality grade

quality_grade (A–D) and quality_score (0.0–1.0) are composite signals:

Score formula: completeness × (0.4 × noise_score + 0.4 × length_score + 0.2)
length_score = min(char_count / 500, 1.0) · noise_score = max(0, 1 − garbage_ratio × 10)

Limitations

• No automatic language detection — flexorch-audit has zero dependencies. Pass locale explicitly, or use the default "und" to activate all detectors. See LIMITATIONS.md.
• Free-standing name detection (without a label prefix) requires NLP/NER — not included.
• replace masking uses realistic pooled synthetic values for national_id_tr, iban_tr/iban_intl, and name; every other type gets a fixed static placeholder (e.g. user@example.com, 000-00-0000) — not a locale-aware generator.
• mask_record_fields() matches by key name (exact or substring), not by inspecting the value — it will not catch a sensitive value sitting under an unrelated key name.
• The library audits plain text. PDF/DOCX parsing, e-invoice extraction, and pipeline orchestration are out of scope.

Integrations

flexorch-audit slots into any LangChain or LlamaIndex pipeline as a pre-load filter — audit quality, detect PII, and optionally mask before your documents reach the LLM.

LangChain — examples/langchain_loader.py

from examples.langchain_loader import AuditedLoader  # copy to your project

loader = AuditedLoader(
    texts=my_texts,
    locale="tr",       # or "de", "fr", "us", "und" (all)
    mask_pii=True,     # redact PII before loading
    min_grade="B",     # skip low-quality documents
)
docs = loader.load()
# doc.metadata → {"quality_grade": "A", "quality_score": 0.91, "pii_summary": [...], ...}

LlamaIndex — examples/llamaindex_reader.py

from examples.llamaindex_reader import AuditedReader  # copy to your project

reader = AuditedReader(locale="tr", mask_pii=True)
docs = reader.load_data(my_texts, min_grade="B")
# doc.extra_info → {"quality_grade": "A", "quality_score": 0.91, "pii_summary": [...], ...}

Both loaders are thin wrappers (~60 lines) with no new dependencies beyond langchain-core or llama-index-core. Copy them into your project — no framework lock-in.

Also available for JavaScript / TypeScript

npm install @flexorch/audit

Contributing

See CONTRIBUTING.md.

License

MIT