tl; dr - You can screenshot web pages, impose a grid and then ask a LM pretending to be a user w/ some need where they would click. You can use this to get ideas about what people find confusing, all using Expected Parrot / @ExpectedParrot
Jupyter Notebook for this: https://www.expectedparrot.com/content/johnjhorton/user-study
The curse of knowledge makes it easy to miss obvious things
One the challenges of writing good technical documentation, writing up an experiment, designing a web page, etc. is the so-called "curse of knowledge": you are intimately familiar with the domain and your own jargon. You forget what it is like not to know something and so you can't empathize with that new user/reader/student you need to reach/teach. This is partially why companies lean so heavily on user testing with strangers---get someone without any experience and see what they do with your product. But it's expensive. Andy you need to go talk to real people. Ugh. Can we at least take a first stab with simulation?
Creating a grid for the vision model to point to
First, I had Claude Code whip up a little script that could impose a grid on an image and label each box in sequence. Code here. Then, I loaded that as a "FileStore" object in EDSL (our open source python package). This is what it looks like in a Jupyter Notebook, where FileStore has a nice 'view" method:
Simulating a single user
Then, I asked a simple question---"if you wanted to get paid to participate in studies, where would you click?" Note that I can just reference the file in my question text via a "Scenario"---I don't have to futz around with uploading files myself. I then run this "job" with GPT-4o. Note the fluent interface that lets me build up scenarios, models, questions, etc. capping it all off with a "run":
Pretty good! Note that this kind of chatty answer might be annoying if I were trying to, say, do an evaluation where I knew the right box answers for each user intent. I don't want to have to parse strings to construct a score. In this case, I'd be better off using an EDSL "QuestionNumerical" which will constrain the answer (and fix it ex post if necessary):
Hundreds of users with different goals (but let's do 4)
What if I hundreds of different users with different goals? I can create a bunch of agents with different goals and then run them through the same survey (but taking the motivation out).
This agent "goal" is incorporated into the system prompt. I could declare hundreds of these goals and have appropriate boxes as answers. I can then run them all in parallel, using a simple declarative syntax. I don't have write out the procedural code to loop over agents and actions: they are all just "sent" to the survey. But let's just do 4:
When I look at the results, I learn the model can recognize the Discord icon (surprising) and it understands the login location. But note it missing the "Docs" connection. Maybe this is clear, but my super smart agent missed it and maybe we should be more explicit.
Next steps / elaborations
You could get a *lot* more sophisticated with this, obviously, but I think this shows the basic idea. As the package is open source and all Python---and you can run these jobs with your own keys---you could make this part of your CI/CD flow: change and interface and have it automatically checked in this way.
I'm sure there's probably 6 startups doing something like this, but I personally would rather just have the ability to customize something exactly how I would want it for my purposes without worrying the startup was going to fail (or they would later do some monkey business software licenses).
Anyway, we love talking to people with interesting applications, so feel free to get in touch!