Ask HN: How important is document category recognition in data classification?
Classifying files based solely on presence of IDs, numbers and other regex patterns, from experience, leads to much false positives and negatives. A technical manual document with a 9 digit number could trigger a SSN pattern check and get wrongly classified as having personal data /Confidential.
Personal data on a HR document would be treated differently from personal data on a customer list.
Would identifying the document category (tech manual, customer list, employee record) be a good additional layer to classifier logic? I'm trying to build a non-AI, light model and embedding based classifier that can semantically understand documents, and like opinions on how useful such a thing would be.
No comments yet.