I transferred teams at work recently, and spent about a week trying to get their code running on my laptop so I can do useful development work. This is in addition to trying to wrap my head around the existing codebase and figuring out how to test my changes. It’s not that the code is bad, it’s just getting all the ****ing components hooked up, communicating with each other, and playing nicely together an exercise in impossibility. Coming from a group that ran everything in AWS, going back to managing all the third-party services in a development environment makes me want to flip my desk over and start screaming about what the **** is wrong with everyone.
The struggle
I tried setting things up 3 different ways before I finally had any success. First, there’s the VM machines that are sort of the “standard” development environment in the company. Allegedly, you can automatically provision 1 of these bad boys, and once it boots it’ll have all the various services pre-installed and running. They’re also near universally-considered in engineering to suck. In my case, the tool we use for setting up and running these services wasn’t even installed on the VM. This utility is a Ruby utility, and Ruby was conveniently multiple versions too low. I spent a lot of time pestering 1 of my co-workers about this, only to learn that he typically avoids doing anything development-related in that environment.
The next thing I tried was trying to set everything up on my laptop. The downside here is there’s no “nuke and start over fresh” option with that, which means if things get into a bad state I’m going to be hosed. But, that’s what people who actually want to get things done do so it’s worth pursuing. Going this route, HBase and Zookeeper absolutely refused to recognize that each other existed, which meant nothing even started to work correctly.
The route with the most promise was running my code locally, and having all the 3rd-party services running in Docker containers. They talked to each other (thanks to things known to work like the --link flag), and I didn’t have to worry about installation and setup, just setting up schemas (by the way, we also use 2 MySQL databases and each database has a separate process creating the tables and schema). This set up got me up and running in less than a day, at least until the point where issues could be exposed by running the code.
By the way, this is just 2 of the datastores the app and its underlying services use – Solr and Redis are also involved, and Solr had to be set up via copying a bunch of configuration XML files to my Solr container, along with some REST calls made via curl. But compare this to the process of getting started on AWS – “Here’s key and secret, and here’s the identifiers for the dev and production versions of things.” AWS ends up being refreshingly easy.
The real issue
These kinds of set ups make getting people started on a project a disaster. Days get wasted just trying to get development environments set up, to say nothing of repeating the process so you have a testing environment. We wouldn’t tolerate that sort of thing in production, and we shouldn’t in our development setup. Making everybody manually complete every step is an invitation for things to break from development to testing and from testing to production. Not to mention with all the little local, “one-off” tweaks people have to go through to keep their house of cards – I mean, environment – up and running, there’s no telling what kind of configuration issues we’re inventing for ourselves.
Stuff like this makes me love all the things my old team had on AWS. Getting set up meant putting the AWS key and secret along with the name for whatever service we were using (S3 bucket, Kinesis stream, etc.) into a configuration file. Once that was done, I was up and running. It doesn’t have to be AWS, any type of managed service will do. That’s because setting up and configuring your dependencies is largely done for you, and it’s done correctly each time. The end result here is that I can have a development environment up in minutes so I can focus on coding.
Here’s another example – after spending days getting my new team’s app set up, I spent the better part of a day setting up 1 of our apps using Docker containers. The reason it took so long – we had a bug in 1 of the libraries we wrote. I had my environment up in minutes, and was debugging other code my team had written shortly thereafter.
Even if you don’t go through the same exact process (e.g. provisioning physical hardware and configuring it with Chef/Puppet/Ansible/etc.), getting any type of environment set up and configured should be largely, if not completely, automated. They don’t have to be the same environments – you can run on your own hardware in production and just use VMs for development and testing. We can automate getting each type of machine from a new state to a ready to run state, and we should be doing that.
These days, we’re building applications that rely on a lot of external services. As great as this is in production, where we use automation to set up new environments, this convenience is sadly lacking when it comes to setting up a development environment. As a result, a lot of time is getting wasted trying to get people set up just to write code. It’s a problem we’ve solved in our production environments, and now we need to spend some time doing the same with our development environments. It’s hard to really focus on this as a priority, since it’s the type of thing that we typically set up once and then never touch again, so the goes out of sight and then out of mind, until you get a new team member. Our applications are getting more and more complicated architecturally, and getting a development environment up and running isn’t as simple as checking out and running some code. We need to make sure the setup process is as easy as possible. If we can’t guarantee that, people are going to go into development with the impression that the software they’re working on is brittle and bound to fall apart at any moment, if they still have the motivation to try to do anything with it at all. It’s sad, especially coming from a team where a lot of this stuff was handled for me. Hopefully we can move in that direction with this team, because I know from experience there’s a better way to do handle these external dependencies.