The blurred line between overengineering and anticipating change

Up until the last decade or so, hardware was so limited that engineers spent most of their time squeezing every transistor out of chips we’d now call prehistoric. I’m still amazed by the optimizations video games pulled off in the late 90’s and early 2000’s, like Quake’s fast inverse square root or Crash Bandicoot literally hacking the PS1 to swap data faster.

But times changed. Engineers no longer need low-level tricks to achieve the impossible, so the focus shifted towards writing software that lasts and can evolve without breaking.

A shallow dive into internal software quality

Good code is code that’s easy to pick up, understand, and change. It usually comes down to a few principles:

• Abstraction, hiding complexity behind a simple interface.
• Modularity, organizing into independent components.
• Separation of concerns, dividing into areas of functionality.
• Anticipation of change, designing to accommodate future modifications.
  This is my favorite one, and the main focus of this post.

When you’re starting a new project, it’s tempting to just dive in and get things done. These principles can feel like an unnecessary burden, but that’s where it usually backfires: after the initial euphoria of shipping dozens of features, the codebase turns into a tangled mess, and every new feature takes longer and longer, as you get yourself through delicious spaghetti code.

I learned about the following chart in my uni classes and I wish I’d seen it sooner:

xychart-beta
	x-axis "Time (days)"
	y-axis "Number of features shipped"
	line [0.0, 6.931471824645996, 10.986123085021973, 13.862943649291992, 16.094379425048828, 17.91759490966797, 19.4591007232666, 20.794414520263672, 21.972246170043945, 23.02585220336914, 23.978954315185547, 24.84906768798828, 25.649492263793945, 26.390573501586914, 27.080501556396484, 27.725887298583984, 28.332134246826172, 28.903717041015625, 29.44438934326172, 29.95732307434082, 30.44522476196289, 30.910425186157227, 31.354942321777344, 31.78053855895996, 32.188758850097656, 32.580963134765625, 32.958370208740234, 33.322044372558594, 33.67295837402344, 34.0119743347168, 34.33987045288086, 34.6573600769043, 34.9650764465332, 35.26360321044922, 35.55348205566406, 35.83518981933594, 36.10917663574219, 36.37586212158203, 36.635616302490234, 36.8887939453125, 37.135719299316406, 37.3766975402832, 37.612003326416016, 37.841896057128906, 38.066627502441406, 38.286415100097656, 38.5014762878418, 38.71200942993164, 38.9182014465332, 39.12023162841797, 39.31825637817383, 39.51243591308594, 39.702919006347656, 39.88983917236328, 40.073333740234375, 40.253517150878906, 40.430511474609375, 40.60443115234375, 40.77537536621094, 40.94344711303711, 41.10873794555664, 41.27134323120117, 41.43134689331055, 41.588829040527344, 41.743873596191406, 41.896549224853516, 42.04692840576172, 42.19507598876953, 42.341064453125, 42.48495101928711, 42.626800537109375, 42.76666259765625, 42.90459442138672, 43.040653228759766, 43.17488098144531, 43.307334899902344, 43.438053131103516, 43.56708908081055, 43.69447708129883, 43.82026672363281, 43.94449234008789, 44.06719207763672, 44.18840408325195, 44.30816650390625, 44.426513671875, 44.54347229003906, 44.659080505371094, 44.77336883544922, 44.8863639831543, 44.99809646606445, 45.10859680175781, 45.21788787841797, 45.32599639892578, 45.43294906616211, 45.53876876831055, 45.64348220825195, 45.747108459472656, 45.84967803955078, 45.95119857788086, 46.05170440673828]
	line [0.09090909361839294, 0.7272727489471436, 2.454545497894287, 5.818181991577148, 11.363636016845703, 19.636363983154297, 31.18181800842285, 46.54545593261719, 66.2727279663086, 90.90908813476562]

Overengineering is a myth

When I say overengineering here, I’m talking in the context of software architecture, not distributed systems and microservices, as that isn’t my area of expertise.

As an advocate for clean code, I often ask myself whether I’m overengineering things. But every single time I think I’ve crossed that line, I’m proven wrong as soon as I need to build on top of what I wrote earlier, and everything seems to fall into place so naturally.

That’s when I realize those design choices were actually anticipating change: even without foreseeing the exact change, or having any gut feeling about how the software would evolve 6 months down the line, those choices still turn out to be the right ones.

A practical example

The Quarkdown CLI has a quarkdown create command that generates a new Quarkdown project with a predefined structure.

The requirements for this command were:

• --main-file <name> lets you pick the name of the main file. Default is main.
• Several prompts gather name, description, document type, and other properties, which are then injected as metadata: .docname {...}, .docdescription {...}, .doctype {...}, etc.
• Unless --empty is provided, the main file should include sample content to help the user learn the syntax and the CLI. An images directory is also created, with a sample image referenced in the main file.

At this point, the implementation seems pretty straightforward. Create the file, inject properties, optionally add the sample content and the images directory:

fun createProject(
    mainFileName: String,
    empty: Boolean,
    info: DocumentInfo
) {
    val mainFile = File(mainFileName + ".qd")
    mainFile.writeText(
        """
        .docname {${info.name}}
        .docdescription {${info.description}}
        .doctype {${info.documentType}}
        
            ${if (!empty) """
                # ${info.name}

                Welcome to Quarkdown!

                ## Compiling

                ...
            """.trimIndent() else ""}
        """.trimIndent()
    )
    
    if (!empty) {
        val imagesDir = File("images")
        imagesDir.mkdir()
        // Add sample image to the images directory
    }
}

Although it’s concise and meets the requirements, this implementation is not maintainable:

• If we want to add more properties in the future (and we will), we’d need to modify writeText’s raw string.
• If we want to change the sample content, we’d need to modify yet another nested raw string.

That’s what anticipation of change is about: designing code that can accommodate future modifications without having to rewrite half of it.

A better approach would be:

• Load contents from a JTE (Java Template Engine) template, a simple text file with placeholders.
• Instead of creating a file directly, return an OutputResource, Quarkdown’s abstraction of a file to be created.

fun createProject(
    mainFileName: String,
    empty: Boolean,
    info: DocumentInfo
): OutputResource {
    val templateContent = loadTemplate("main.qd.jte")
    val renderedContent = renderTemplate(templateContent, info, empty)
    return OutputResource(mainFileName + ".qd", renderedContent)
}

The final architecture

That’s clean and reasonable. So why would it be overengineered? Because that’s not what I went with. I went for this monster instead:

• ProjectCreator is the orchestrator: it doesn’t know how to create templates or what initial content to include. Instead, it delegates to two injected strategies:
• ProjectCreatorTemplateProcessorFactory creates JTE template processors, injects the document info, and defines which files to generate via createFilenameMappings() (a map of file names to their template processor).
• ProjectCreatorInitialContentSupplier provides sample content and additional resources (e.g. images). By default, it loads a separate .jte template with sample code plus the image.
  If --empty is provided, the Empty supplier kicks in instead, returning empty content and no resources.

When createResources() is called, ProjectCreator goes through the factory’s file mappings, processes each template, and collects the results.

class ProjectCreator(
    private val templateProcessorFactory: ProjectCreatorTemplateProcessorFactory,
    private val initialContentSupplier: ProjectCreatorInitialContentSupplier,
    private val mainFileName: String,
) {
    fun createResources(): Set<OutputResource> {
        val resources =
            this.templateProcessorFactory.createFilenameMappings()
                .map { (fileName, processor) ->
                    TextOutputResource(
                        fileName ?: mainFileName,
                        processor.process(),
                    )
                }

        return buildSet {
            addAll(resources)
            addAll(initialContentSupplier.createResources())
        }
    }
}

classDiagram
    direction LR
    class ProjectCreator {
        -mainFileName String
        +createResources() Set~OutputResource~
    }
    class ProjectCreatorTemplateProcessorFactory {
        <<interface>>
        +createFilenameMappings() Map~String?, TemplateProcessor~
    }
    class DefaultProjectCreatorTemplateProcessorFactory {
        -info DocumentInfo
    }
    class ProjectCreatorInitialContentSupplier {
        <<interface>>
        +templateContent String
        +createResources() Set~OutputResource~
    }
    class EmptyProjectCreatorInitialContentSupplier {
        +templateCodeContent = null
        +createResources() = emptySet
    }
    class DefaultProjectCreatorInitialContentSupplier {
        +templateCodeContent
        +createResources() = images
    }
    ProjectCreator --> ProjectCreatorTemplateProcessorFactory
    ProjectCreator --> ProjectCreatorInitialContentSupplier
    ProjectCreatorTemplateProcessorFactory <|.. DefaultProjectCreatorTemplateProcessorFactory
    ProjectCreatorInitialContentSupplier <|.. DefaultProjectCreatorInitialContentSupplier
    ProjectCreatorInitialContentSupplier <|.. EmptyProjectCreatorInitialContentSupplier

What proved me wrong

I knew this was a bit much, but I went with it anyway. It felt overengineered… until about a year later, when I added a new document type: docs. This type lets Quarkdown generate multi-page technical documentation and wikis like this one, which need a different structure and different sample content. Here’s what a docs project looks like:

Also, page-N.qd files shouldn’t be generated if --empty is provided.

With the naive implementation, this would have been a nightmare. Imagine the conditional logic and nesting needed just to handle the new files.

With my approach, it was just two small classes:

• DocsProjectCreatorTemplateProcessorFactory overrides createFilenameMappings() to generate two files instead of one: _setup.qd and main.qd.
• DocsProjectCreatorInitialContentSupplier reuses the default’s sample code content and provides the additional page resources (_nav.qd, page-1.qd, page-2.qd, page-3.qd).

No changes to ProjectCreator itself: it remained completely unaware of docs projects. The CLI command just picks the right strategy based on the document type, and the rest falls into place:

return ProjectCreator(
            templateProcessorFactory =
                when {
                    isDocs -> DocsProjectCreatorTemplateProcessorFactory(documentInfo)
                    else -> DefaultProjectCreatorTemplateProcessorFactory(documentInfo)
                },
            initialContentSupplier =
                when {
                    noInitialContent -> EmptyProjectCreatorInitialContentSupplier()
                    isDocs -> DocsProjectCreatorInitialContentSupplier()
                    else -> DefaultProjectCreatorInitialContentSupplier()
                },
            mainFileName,
        )

classDiagram
    direction LR
    ProjectCreator --> ProjectCreatorTemplateProcessorFactory
    ProjectCreator --> ProjectCreatorInitialContentSupplier
    ProjectCreatorTemplateProcessorFactory <|.. DefaultProjectCreatorTemplateProcessorFactory
    ProjectCreatorTemplateProcessorFactory <|.. DocsProjectCreatorTemplateProcessorFactory
    ProjectCreatorInitialContentSupplier <|.. DefaultProjectCreatorInitialContentSupplier
    ProjectCreatorInitialContentSupplier <|.. EmptyProjectCreatorInitialContentSupplier
    ProjectCreatorInitialContentSupplier <|.. DocsProjectCreatorInitialContentSupplier

Takeaway

This example shows why I don’t believe overengineering is a thing in software architecture. Strategy pattern, factory, supplier: they all seemed like overkill for a simple project generator. But when the requirements evolved, the architecture held up without changes to the core ProjectCreator.

This was a relatively simple and small example, but the same principle easily scales up to larger and more complex systems.