Code search is easy

Earlier this week, Tom MacWright posted Code Search is Hard. He describes the research he his doing to improve the code search experience of Val Town. It was a great read, and you might have seen it trending on Hacker News.

As Val Town's most active user (behind Steve Krouse, one of the founders of Val Town), I for sure agree with Tom that the search feature needs improvements. But while reading his post, I immediately thought of a different approach to the problem. And a few hours later, Val Town Search was born.

Do things that don't scale

How does this new shiny search engine work? Well, it's quite simple.

1. I wrote a Deno script that fetches all vals from the Val Town API.

#!/usr/bin/env -S deno run -A
import * as path from "https://deno.land/std/path/mod.ts";
const dir = path.join(import.meta.dirname!, "..", "vals");
const blocklist = Deno.readTextFileSync(
  path.join(import.meta.dirname!, "blocklist.txt")
)
  .split("\n")
  .map((line) => line.trim())
  .filter((line) => line.length > 0);

let url = `https://api.val.town/v1/search/vals?limit=100&query=+`;
const vals = [];
while (true) {
  console.log("fetching", url);
  const resp = await fetch(url);
  if (!resp.ok) {
    console.error(resp.statusText);
    Deno.exit(1);
  }

  const { data, links } = await resp.json();
  vals.push(...data);

  if (!links.next) {
    break;
  }

  url = links.next;
}

Deno.removeSync(dir, { recursive: true });
Deno.mkdirSync(dir, { recursive: true });
for (const val of vals) {
  const slug = `${val.author.username}/${val.name}`;
  if (blocklist.includes(slug)) {
    console.log("skipping", slug);
    continue;
  }
  const userDir = path.join(dir, val.author.username);
  Deno.mkdirSync(userDir, { recursive: true });

  Deno.writeTextFileSync(path.join(userDir, `${val.name}.tsx`), val.code);
}

2. I pushed the data to a Github Repository (now private)
3. I added a Github Action that runs the script every hour to refresh the data.

#!/usr/bin/env -S deno run -A
import * as path from "https://deno.land/std/path/mod.ts";
const dir = path.join(import.meta.dirname!, "..", "vals");
const blocklist = Deno.readTextFileSync(
  path.join(import.meta.dirname!, "blocklist.txt")
)
  .split("\n")
  .map((line) => line.trim())
  .filter((line) => line.length > 0);

let url = `https://api.val.town/v1/search/vals?limit=100&query=+`;
const vals = [];
while (true) {
  console.log("fetching", url);
  const resp = await fetch(url);
  if (!resp.ok) {
    console.error(resp.statusText);
    Deno.exit(1);
  }

  const { data, links } = await resp.json();
  vals.push(...data);

  if (!links.next) {
    break;
  }

  url = links.next;
}

Deno.removeSync(dir, { recursive: true });
Deno.mkdirSync(dir, { recursive: true });
for (const val of vals) {
  const slug = `${val.author.username}/${val.name}`;
  if (blocklist.includes(slug)) {
    console.log("skipping", slug);
    continue;
  }
  const userDir = path.join(dir, val.author.username);
  Deno.mkdirSync(userDir, { recursive: true });

  Deno.writeTextFileSync(path.join(userDir, `${val.name}.tsx`), val.code);
}

4. I created a simple frontend on top of the Github Search API that allows you to search the data. It's hosted on Val Town (obviously).

That was it. I didn't have to build a complex search engine, I just used the tools that were available to me.

Is this a scalable solution for Val Town? Probably not.
Am I abusing the Github API? Maybe.
Does it work better than the current search feature of Val Town? Absolutely!

I hope that the val.town engineers will come up with a search feature that will put my little project to shame. But for now, you won't find a better way to search for vals than Val Town Search.

PS: This post was written / is served from Val Town