Large language models (LLMs) are becoming pretty good at assisting humans with variety of tasks. We provide some text input which has some context and a problem and get an answer. Most of these conversations contain private information. Private information doesn’t only have to be SSNs - it could be the car you drive, your neighborhood, your school name, your best friend’s address- the point is that quasi identifiers of privacy can be combined to make a pretty good social profile of users. I came across this yesterday; a browser extension gone rogue. But even if that did not happen, we are continuously sharing a lot of private information every day with Google, OpenAI, Anthropic and/or whoever else is your LLM provider. Since OpenAI is also in partnership with Oracle, maybe Oracle is also going to get some some peek into that private info. who knows really??
I think the only solution to this problem is: we run the LLM locally. The pro is that all the information stays local and I think this is the future. The con is that it’s not affordable and the speed kinda sucks. To run kimi k2 locally at 5 tokens/second you would need two RTX4090 with 24GB ram, the whole set up might cost upwards of $5000. And 5 tokens/second is pretty slow given that gemini api is clocking at around 80-100 tokens/second depending on flash vs pro. This also makes me think that the model is not the product; the hardware is. As long as the hardware is expensive; we can’t have local AI and we can’t have privacy. I must qualify the previous statement with another possibility that is that we get another breakthrough in language modeling and we could run very very capable models just on our regular laptops and phones.
I was still itching to try something though..
The python package Zink has a shield functionality which removes any type of entities from the prompts and replaces them with some placeholders. I wanted to try this in a chat format. So I ended up creating a run of the mill streamlit based chatbot. All the work can be found here.
As you can see, it looks like just any chatbot without any bells or whistles. There is an option on the side where you can add a label. You can just type an entity type that you wanna hide from the LLM and press enter.
Like that!
Let’s see if my name gets sent to Gemini.
You can see in this chat that even though I am being reckless and telling my name to Gemini, but what was sent was human_name_2495_REDACTED. That’s my identity in Google’s servers. Zink re-identifies the name locally to keep your experience as smooth as possible. Okay, let’s see if we can try some other labels with a slightly more realistic use case?
Gemini’s response was huge and here it is if you are interested:
The original draft contains highly sensitive information and requests security footage in a way that is informal, unprofessional, and potentially violates company policy. Such serious matters should always be reported through official channels (e.g., Human Resources, Security, or management) so they can be investigated properly and in accordance with company protocols and privacy regulations. Requesting security footage through a janitor and asking for confidentiality in such a serious matter is not appropriate.
Therefore, I cannot make the exact request for footage through a janitor "professional." Instead, I will provide two separate, professional options:
A professional way to report a serious incident to the appropriate department (e.g., HR or Security), which would then handle any investigation and official requests for evidence like security footage.
A professional way to inquire about travel plans.
Option 1: Professional Email to Report a Serious Incident (To be sent to HR, Security, or relevant management, NOT an informal contact)
Subject: Urgent Incident Report - Observation Regarding Michael Scott and Jan Levinson
Dear [Relevant Department/Person, e.g., Human Resources Department / Head of Security / Meredith (if they are in a position of authority regarding such matters)],
I am writing to formally report a serious incident I observed yesterday evening involving Michael Scott and Jan Levinson in the vicinity of Oscars's desk. The nature of the incident was highly inappropriate and concerning.
I believe this matter warrants immediate investigation. Security footage from yesterday evening in that area may be relevant to understanding the full scope of what transpired.
Please advise on the appropriate next steps for formally reporting this incident and how to proceed with the review of any available security footage.
Thank you for your prompt attention to this urgent matter.
Sincerely,
Kevin
Option 2: Professional Email to Inquire About Travel Plans (As a separate, unrelated email)
Subject: Inquiry Regarding Travel to New York city
Dear Meredith,
I hope this email finds you well.
Could you please let me know your approximate departure date for New York city?
Thank you,
Kevin
If you notice the above response has all the names Kevin, Michael, Jan, Meredith, New York city.
Let’s see what was sent to gemini:
Each entity gets their own 4 digit identifier. So this is already good, I like that i can protect names, addresses. But sometimes gemini need to know some entities to answer the question correctly. Zink has a functionality for that as well:
Zink allows you to keep entities from getting redacted even if the general category is selected for redaction.
The input query was - tell me a joke in the style of *Michael Scott*?
Any text that is wrapped in * would be excluded from redaction.
This is the end of level 1 testing. I must add that this does not provide you with 100% privacy.
Google still knows about my account
The local model might miss some entities.
But this app still provides a pretty good layer in between. I might keep using it for simpler queries bc gemini has a free tier. If you have any suggestions on how to make it better, feel free to leave a comment or reach out otherwise. You can spin this chatbot locally. The code is here.