OK, Gell-Mann Amnesia may not be the best analogy, but it certainly seems the closest I can find. LLMs seem incredibly helpful when you’re prompting about things you don’t understand well, but the more you go into detail about something you do understand, the more…whelming (something I thought was only possible in Europe)…these tools seem. That makes me start to wonder if these tools get their most dangerous at places where we have some knowledge, but not enough to claim actual expertise.
As I mentioned recently, I used a free model as queryable documentation for setting up a local K3S cluster with the goal of deploying it to a remote server to give me a place to run some personal projects. Every time I got asked for clarification or a decision about how I wanted something, or pushed back on how this was being constructed, I was absolutely correct, with unanimously good ideas. Which is…shockingly…good for someone with very little Kubernetes experience.
Look, I’ve been working in software for nearly 20 years at this point in my life. I’ve been architecting services to run on the cloud for over 10 of those years, including in containers. I’m not coming into this completely ignorant of deploying and running applications, just Kubernetes (and Helm) YAML files. So yes, I did have a pretty good idea of what I was doing, but that doesn’t mean I can walk into something I’ve never worked with before and know how to do it exactly right even if I didn’t know the syntax. Kubernetes isn’t just a thing you use to run containers, even if I was taking pains to avoid building a multi-node cluster (for now). Surely something that I was used to doing wouldn’t necessarily be a good idea in this context. And yet, I had the right answer every time.
It felt good – I was making progress getting a full fledged setup with supporting services, I had help double-checking the YAML syntax and researching error messages. I could ask what something meant or why something wasn’t behaving the way I would have expected. That’s progress and velocity…right? The frustrating thing is that I have no idea. Not just because I don’t have experience with Kubernetes to be able to say so definitively, but because it was too easy.
Let’s assume that the model was giving me the most common Kubernetes setup1 for a containerized web application. I don’t have any special setup to this app – it just needs a couple directories on the host system (1 for the Turso database, and 1 for any SQL migration files to modify data2). Everything else is in the container. So the most common, basic setup is most likely what I need. But is it overblown for my use case? (probably – I jumped straight to Kubernetes) Were there considerations that should have caused me to change what I was doing? Were my instincts right if I was using something like ECS but less so in Kubernetes? Are there parts to a “standard” setup that don’t necessarily hurt to have in mine but aren’t going to do any good either? An actual expert in Kubernetes deployments could answer those questions, but I don’t have one. Just something that I got running in a suspiciously smooth manner. And I have just enough familiarity around the edges of this to think I can adequately supervise this thing.
Except…any time I try to have 1 of these models help with coding, I find myself having to go back over it, forcing it to go back and clean up the logic, and questioning apparent nonsense decisions (a non-trivial amount of which get reverted). That, by the way, covers “good” results, not the times where I have to revert everything it did and to start from scratch, or where it starts doing unnecessary refactors I didn’t ask for3.
At least if you’re using LLMs to research something you have little to no experience in, you know to do follow-up research and make the clanker give sources. When you’re using it to speed run something you’re not familiar with the documentation on, you just try to run it and stop as soon as you see the output you’re looking for. Sure it launches, but that doesn’t mean it was built well, and the honest truth is I turned to LLMs because it’s not like I knew what “good” looked like in this case.
As tempting as it is to say LLMs made me “faster,” I’m not sure, because I don’t know if I’m going to need to back and fix something in a few weeks. All I know is I have a Kubernetes YAML file that’s probably generic, and an app that probably just needs a generic set of YAML files.
Using an LLM certainly feels like a productivity boost, until you deeply review what it does to stuff you do understand well, and realize it’s probably doing everything just as well. That begs the question – what happens if I was going to need some weird one-off Kubernetes setup for something in the future? Could an LLM even handle that? How much back-and-forth hassle would it take me to even get to that point? That’s the frustrating part to this whole “let agents do the work” movement – we jump to using these tools precisely at the most dangerous points to use them, and we fall into the trap of feeling so empowered we’re willing to give up the areas where our expertise makes us valuable, even if we’re skeptical (or at least we should be) when using them for areas where we know we don’t have any relevant expertise to draw on.
I’m finding it’s all too easy to get lured into a false sense of security by the speed of output and forget that thinking (the real kind, not that placeholder text that your harness of choice shows while auto-completing an answer) often does make a difference. It really is just more fun to be competent.
- I’m hoping that by this point models give you content based on actual real source code running in production, and not just watered-down content from tutorials. ↩︎
- I don’t want an admin panel or user roles for this, it’s a family gift registry. I also made a deliberate decision to not make it a self-service sign-up because I want to guarantee it’s limited to my family. I’m willing to write several
INSERTstatements once to create users to keep it locked down. ↩︎ - I suppose I should write a user-level prompt to better control coding agent behavior, but I’d rather these things stop operating as if they’re the brains of the operation. ↩︎