We discovered a way to measure LLM bias while building a recruitment tool

While developing an AI tool to help hiring managers prepare for interviews, we stumbled upon what seems to be a novel method for detecting bias in Large Language Models.

By comparing how LLMs (Claude, GPT-4, Gemini, Llama) interpret anonymized vs. non-anonymized versions of the same content, we can measure and quantify bias reduction. The interesting part is that this technique could potentially be used to audit bias in any LLM-based application, not just recruitment.

Some key findings:

- Different LLMs show varying levels of bias reduction with anonymization

- Llama 3.1 showed consistently lower bias levels

- GPT-4 performed better in specific tasks like interview question generation

We've published our methodology and findings on arXiv: https://arxiv.org/abs/2410.16927

We're a boutique AI consultancy, and this research emerged from our work on building practical AI tools. Happy to discuss the technical implementation, methodology, or real-world applications.