The movie Kingdom of Heaven has 1 of the best knighting scenes in cinema – punctuating the knight’s oath with a hard slap, because “…that’s so you’ll remember it.” While software development doesn’t come with quite the same…”haptic feedback,” the fact is a lot of us have leveled up after getting metaphorically slapped around. It sucks in the moment, but that suffering is so we remember the lessons we learned.
Here’s an example. We had an outage at work recently stemming from all the available sockets on our server getting tied up, preventing users from connecting to our server (while I’ve been known to suck up everyone’s resources, I’m pleased to announce I didn’t do this one). So guess what we’ve been paying special attention to lately (that we probably should have been actively thinking about all along)? Yep, how something impacts our total socket connection pool. We’ve been burned, and we’re determined to not let this happen to ourselves again. Put another way, we’ve learned a hard lesson, we’ve grown, and we’re doing more thorough engineering now.
I recently wiped out environment variables in my local environment for the gift-registry project I’m working on to learn Go, and had to re-create them, including the Gmail integration for testing the email-based login I have set up. No big deal, just look up the SMTP information for Gmail, and bam, done right? I thought so, until my login flow stopped working. If you’ve integrated an Gmail into an app, you probably already knew this was coming, but Gmail doesn’t try to connect on the SMTP port, it tries to connect on the TLS port. I know that now, and now actually have the skills needed to better review any code connecting to Gmail in the future.
There’s a growing, and troubling, belief in the software industry that this pain can be avoided entirely or sidestepped by outsourcing the thinking and understanding parts of software engineering. That’s already not working, but because the hype takes months to hit normal businesses, they’re chasing to catch up to some imagined ideal the companies that started the trend of getting rid of developers for autocomplete are already starting to regret.
Here’s the thing, skipping this pain is making us worse. It seems like every time someone who doesn’t stand to make money selling inference studies the use of LLMs in development, the conclusion is that they’re a net negative. I thought relying on LLMs would retard people’s skillsets, but holy crap, the extent to which it happens and the speed at which it happens is mind-blowing. I’ve watched someone go from an engineer that I respected and thought was a good developer to someone I lost all respect for whatsoever in a matter of weeks as he became more and more clanker-brained.
Taking the time to do things ourselves, screw it up, and have to figure out how to fix it is how we learn. That “pain” of figuring out what’s going wrong, digging for answers, and having to fix things is how we learn. Doing it repeatedly starts to change how we think. It’s so we remember what the “wrong” code looks like as well as the “right” code. Changing how we think when building software is how we grow from brute-forcing just the happy path to knowing how to build resilient systems that don’t fall over at the slightest provocation.
Before anyone tries to think they’re clever and say that thinking about code isn’t the human’s job anymore, I’d like to offer this challenge – stop reviewing the code then. Don’t question the LLM’s output. Let the developers you’re still employing off the hook for the AI-generated code. Not interested? Developers are still important? I thought so – although apparently not enough to give them the time to build full-fledged mental models of the software and problem domain though.
To the software developers – don’t give up the work. I don’t care how fast an LLM generates lines of code. Having to struggle with problems and make ourselves work through them step-by-step isn’t just how we build a meaningful understanding of the code, but also of the problem and why we’re doing what we’re doing. It’s not about going faster, in fact, the relative slowness of doing it by hand forces us to face facts when what we’re doing painful or awkward, and that we need to go back to the drawing board.
Reading articles, LLM summaries, or tutorial videos are good introductions to something, but you’ll never understand anything until you’ve struggled with it. You have to have those facepalm moments where you can’t believe you made such a stupid mistake over and over again to develop the muscle memory needed to stop making them instinctively. You have to develop the instinct for what problems you can potentially run into with a given approach, and the things that can help you mitigate them. And there’s nothing out there that’s ever going to be an effective substitute for that. You just have to get out there, have problems, fix them, and repeat.
If you’ve ever wondered how some developers got so good they can spot issues in your code that you couldn’t, it’s because they’ve been burned by those issues, and learned how to not be burned by it in the future. Over time, they’ve been burned by enough stuff that they get real good at dealing with a lot of different problems. Yes, debugging can suck, but it’s how you get good at not having bugs in the first place. Believe it or not, “(pain) is a good thing.“