Show HN: Phospho – Text Analytics for LLM Apps (Posthog for Prompts)
github.comHello HN! Pierre and Paul here. We are building an open source text analytics tool for user inputs and LLM app outputs The repo is https://github.com/phospho-app/phospho and landing is https://phospho.ai
Most people building with LLMs today don’t have quantified evaluation and usage metrics on the interactions between users and their product. The only solution is to read every message (or a sample) to get a sense of what is going on. You can't improve your product without understanding who your users are and how they are using it. Nobody would launch a website without standard analytics today; the same principle should apply to LLM products.
We made phospho to analyze the large amounts of text from user inputs and LLM app outputs, and give you quantified and actionable insights.
You first log messages and set up semantic events. Eg: “user is talking about sports”, “assistant didn’t quote the source”. We then run asynchronous jobs to detect if events are present in the text or not. To do so, we use GPT-4 for the first few events, and then downsize to smaller fine-tuned models (cheaper & faster). It works with any LLM provider (OpenAI, Mistral, Ollama…). No proxy, no monkey patch, and no OpenAI key needed.
You can link phospho to your users’ feedback, and even use the platform to annotate some messages yourself. This helps you design step by step a custom evaluation pipeline that runs automatically, fits your needs, and enables you to iterate.
Results are available in dashboards, as dataframes, or via API. You can also directly leverage the events in your app to trigger actions in real-time with the API or via webhooks.
Deploy everything with Docker, or use the hosted cloud version. We have Python/Javascript SDK and an API. License is Apache 2.0.
Give it a spin and see where we’re at: https://github.com/phospho-app/phospho We’re interested in both feature requests and roasts. Let us know what you think! I feel like as devs there is an omerta on the real performances of AI apps.Cherry picked twitter screens get social media highlights and money flowing into the field, but we all know that in practice it delivers on 1% of the value promised. Unsure how much of it relies on the models themselves, and how much on the fact that optimizing prompts for a propper signal is so f* hard I hope more tools will make this easier. Will take a look at phospho, thanks man You're welcome! nice landing page. and docs too. design could be prettier but it does the job of showing the damn product which is all we want. so this is like amplitude/heap/whatever but instead of manual tagging of events you use gpt4 for tagging? with a ui/pipeline for iteration. candidly i doubt that you can build this faster/better than amplitude can clone you, but thats the joy of founding things, you have to overcome everyone's doubt including your own! Exactly. Being copied would be the best of compliments. That's why we are open source to also leverage community contributions. I think the market right now moves too fast for traditional analytics companies. Let's see how it goes in the long run. Very interesting, seems really helpful to keep track of things What are platforms like new relic doing on this front? Thank you! New Relic is close to Datadog, more focused on observability for engineers. We're focusing on extracting insights from the text. We focus on NLP more than trace analytics Very cool project with so much potential. Will follow along. Thanks for sharing. <3 How does it compare to portkey which is backed by YC too? Portkey is a proxy, so you will have to call their API, and they will route the request to a LLM provider. Then, they provide you with some observability tools to look at these requests -> LLM router + observability
phospho is not a proxy, you log the messages and feedbacks to our API. We then extract insights and detect events inside the text -> text and product analytics