Maybe it’s just me, but lately it seems like every problem that doesn’t come directly from customers is resolved by integrating with something that just does it for you, no need to worry. Some of this stuff is really useful because it solves a problem that’s tangential to yours but too much work to just do yourself (for example, a library that renders graphs on a web page based on JSON data, or a framework for setting up a REST API web server). But lately it seems like more and more stuff is coming out that’s handling stuff that really should be handled by the application itself (for example, building an OAuth sign-in widget, or implementing feature flags). I get not wanting to spend a lot of time on something that isn’t the key thing that your business does, but I think we’re wildly overestimating the value we’re bringing to our applications if we outsource everything but proprietary company data, and wildly underestimating our exposure if 1 of these dependencies breaks or has a security problem. Just because it’s not “core” to the main objectives of our software doesn’t mean it’s not important, or so complicated that we can’t do it ourselves.
Look, I understand not wanting to be a “not invented here” organization. We’re here to solve problems and going down other rabbit holes wastes time, especially when it’s a minor piece of the big picture. And sometimes integrating with a service makes a ton of sense because doing it yourself correctly is hard (e.g. having users log in using an existing Google/Facebook/Twitter/whatever account rather than trying to implement a secure user authentication system yourself), or introduces a whole new slew of work (e.g. I heartily recommend integrating with Stripe or Braintree for payments rather than processing it yourself and going through the associated PCI compliance headaches). Pulling in other services and libraries saves time and effort, and when you have a limited number of people to be working on something, those savings aren’t just huge, they’re the types of things that make you able to build a complete application in the first place.
But that said, there’s a case for taking matters into your own hands when feasible. In fact, I’m increasingly of the mind that we should be doing more ourselves in our software, which is why I’m working on a project with a focus on operating it myself. The problem I’ve found, and it seems a common refrain in a lot of things in software and life, is that for any multi-word concept or multi-part definition people tend to focus on 1 part to the expense of everything else. In this case, we get so focused on “core” aspects of our business (generally defined as things that only your company does), that we start effectively refusing to do more and more fundamental development work.
For the kids out there, several years ago something happened known as the “left-pad incident.” Basically, an open source developer who wrote a lot of little npm libraries got notified about a potential trademark concern regarding 1 of his library names. Rather than negotiating a solution, he decided to throw a tantrum and demanded that all of his packages be removed from npm because a company was poised to get something it wanted. Now, as stupid as this sounds so far, I’m just telling you this to get to the real stupidity of this story. 1 of those packages is the aforementioned left-pad, which was used by seemingly everybody writing Node at the time, either directly or as a dependency in 1 of their dependencies. When that package they depended on disappeared from the repositories, builds starting breaking everywhere.
So what was this mythical left-pad whose absence crippled front-end development for a few days? It was a simple function to add whitespace to the beginning of a set of string so that the values could be aligned. That’s it. That’s all there was to it. But a truly surprising number of companies, who should have known better, imported this dependency rather than write a function themselves (because it wasn’t their “core competency” you know), and got screwed when it had a problem.
It’s not just “stupid little function” dependencies that pose potential problems. If you were a Java developer a few years ago you probably remember the “Log4Shell” bug that gave a lot of developers heartburn trying to fix across a lot of repositories. That was an actual logging interface that standardized Java logging – lots of applications were hit by something that was imported into code bases.
Here’s an example of how outsourcing fundamental business operations to a third party can create more problems than it solves. If you make an in-app purchase on your phone, the company who owns that app doesn’t actually have control the purchase. The company who made your mobile OS does (technically the company that made the app store does, but right now that’s the same as the operating system). But if you have a problem with any aspect of a business transaction you’re not going to reach out to whoever coded your phone, you’re going to contact the company whose app you were using just to find out they can’t really help you. That feels bad, as a customer, and as a company. If you want to know what your customers are buying, you have to hope the mobile app stores you’re using to distribute your application are willing to share data that’s pretty important to your business.
So what’s a good approach to drawing the line between “we do core business functions in-house” and “we’re fully committed to only using code we wrote ourselves?” Well, let’s start with the limited take on “core competency” I mentioned earlier – things that only your company is doing (their “secret sauce” if you will). Now let’s take a step back from that limited definition, and think about all dependencies that code has. Those things are part of your core competency too, so you need to own that code as well. Next up, consider all the things your advertise about your software. Anything code that furthers those selling points should be your code, not outsourced to another service provider.
Introducing dependencies introduces risk (to be fair, your own code also introduces risk). The difference is you have control over your own code. Bug in a dependency? You’re either sitting on pins and needles waiting for a fix to be released or scrambling to find and test a replacement. A bug in your own code? You have the power to push other stuff to the side to fix it. The same is true for security vulnerabilities, performance issues, user experience concerns, or anything else. Your code is under your control, and I’m really growing to appreciate the fact that more control gives you more options for anything you could want to do. Sure, there’s tons of services that advertise themselves as being customizable, and they may actually be so customizable that you can do whatever you want to do, but the only way I’ve ever seen to guarantee that is to own the code yourself.
Modern software development increasingly seems like piecing together various libraries, services, and frameworks that “do {X} for you,” with increasingly less and less business-specific logic to be added ourselves. In fact, it seems like the work done by these libraries, services, and frameworks resembles the job interview meme, and the actual “core competency” business logic resembles the actual job meme. I think we’re so seduced by convenience of just having some other software do work for us, that we’ve lost sight of the benefits of controlling our software. Sure, using managed services means you don’t have to manage the servers and containers yourself, but it also limits your ability to scale up in a pinch.
I learned that the hard way when a data feed processed by lambda functions backed up. We could change a few configuration options to allow more allowable concurrent lambda instances, but never could just get the number of instances running that we really wanted because ultimately, that was AWS’s decision, not ours. If we were running our own stacks, we could absolutely have dedicated more hardware to clear the backlog of messages faster.
At this point in my career, every time I see any sort of architecture diagram, I start trying to see if I can pull anything out without failing to meet the business needs or making my life needlessly harder for no benefit. I appreciate not re-inventing the wheel and letting people do what they’re best at, but there’s also quite a bit about software that needs to stay “in-house.” We can’t operate what we don’t actually build and own, and that means that we can’t keep passing the bucks. We don’t need to invent it all here, but success in software is going to come to those who aren’t digitally dropshipping.