So I ran into an interesting issue a while back doing something on a project at work. I was doing something that involved interacting with files on S3, and had updated the AWS libraries my application was using, but I was running into errors every time I tried to do anything that touched S3. The issue wound up being a dependency conflict between my code and an S3 wrapper utility we wrote around the AWS SDK. While the main project’s AWS SDK was up-to-date, the utility had older versions that were causing the errors. Although I had used Amazon’s BOM and thought that made sure that my AWS libraries were all set to a particular version, the reality was that assurance didn’t spread to other libraries my project pulled in. The solution was simple enough – update the AWS SDK versions in our other internal library, but that did lead to the question, how does one manage dependencies between projects?
I did some (admittedly brief) searching on the topic, but couldn’t really find any links to indicate this is something’s that’s been conclusively addressed anywhere. I’m sure this is something larger businesses have run into, so I’m interested to learn how they deal with this. Is there a best practice that we’re missing? Should we have some kind of big-picture view of how all these utilities interact with our applications? Surely somebody’s dealt with this and come up with a workable solution. I’m pretty sure multiple somebodies have done so. So what were their ideas? And why aren’t they more easily found by search engines?
Build tools like Maven and SBT are great for managing dependencies within a project, but they’re no good at managing dependencies between projects (assuming project B is a dependency for project A). There’s always hosting your own private repository and not allowing your developers to upgrade versions without going through some sort of process to make sure all projects with that dependency are updated, but that makes updating 3rd-party libraries a bottleneck, which is bad in cases where you need to pull in a bug fix from said library update. But without that bottleneck, you end up with multiple versions of your dependencies and that creates problems too.
All of this leads to 2 questions. The first one being, is there any way to keep dependencies used by multiple projects in sync using build tools like Maven that isn’t clunky or slows down developers ability to keep moving forward? The second question would be is it even worth it to try? I’m beginning to think that the answer to both questions is “no” – at least not through tools. Sure you can set your dependency versions in Maven to “LATEST” and only pull from a private repository so you can ensure everything’s tested when you upgrade a library. That would require you to have great automated testing of all code that calls said library so you could both be sure an upgrade won’t break anything and keep having to bump up a version of a dependency from bringing all development to a grinding halt. It’s not a bad goal to shoot for in principle, but for development teams with actual code to write and things to accomplish, it’s probably also not high on the priority list.
There’s trying to re-write your services so that 3rd-party libraries aren’t shared between projects, which is harder than it sounds. Your external dependencies are always going to be on the same version, but you still need to make sure your projects use consistent versions of your internal libraries (although that’s probably a lot less likely than using 2 different versions of an external library). For a team that focuses on writing several independent apps, this is a lot of work and I’m not sure the payoff is worth it. This also doesn’t solve the problem of your external dependencies having different versions of libraries your code depends on (it’s happened to me a few times with Jackson). It’s still a start though.
Dependency management is a hard problem, and there are parts of it that don’t seem to be completely figured out, in some parts because it’s really hard and in others because the gains aren’t worth the effort. Still, it’s probably worth thinking about the dependencies I’m bringing into my projects and trying to see if there are some best practices, or even just better practices, I could be taking up to try to make sure all my external libraries are on the same page, or in this case, the same version.