So while I read up on Golang, I thought I’d go ahead and set up some sort of observability infrastructure to have in place as soon as I started coding. After all, part of my goal is to instrument an application for OpenTelemetry and take advantage of that. What I found was that it’s not as simple as “pick something and spin it up” as I had hoped. That makes sense, running these sorts of services is part of platform engineering, there are teams whose full-time job includes deploying and managing these types of services, but I’m still spending longer getting that up and ready than I initially wanted to. But that’s OK – another part of this is to get better at operating software, and stuff like this is good for understanding what all is involved in “running and supporting software that actually has users.”
Ironically, observability tools seem oddly unclear in how you’re supposed to set them up. I tried to get Jaeger set up and running first. Jaeger has a container image that’s actually really easy to get up and running (and test with a demo application). However, Jaeger is just log tracing, but does integrate with Prometheus. Since I wanted to go full observability, I decided it was a requirement to get the 2 hooked up, and that’s where I ran into trouble. This is also where I found the Jaeger documentation to be lacking. Namely, there wasn’t much of any explanation of what the configuration are or how they were supposed to help connect to Prometheus. At this point, I could either pull up the Prometheus documentation and try to work backwards on configuring the Jaeger/Prometheus connection, or go for a more all-in-one observability tool
That led me to SigNoz. You can either sign up for their SaaS offering, or run it on your own Docker Swarm stack (I’m went with the latter). There’s also not a lot of documentation around the configuration, but the configuration files in their repo seem to work well enough. The biggest complaint there is that it’s so involved, the only way to get a Signoz instance running on a local Docker Swarm was to clone the repo, and copy files from their Docker Swarm deployment directory into my own project and deploy it from there. That…works, but it’d be nicer if I could just package the container as part of the compose file I’d be using for code I write.
I also came across OpenObserve. On the surface it seems like another good option – covers metrics, logs and traces, all through a single Docker deployment, and apparently a SQL-based query language (as opposed to what appears to be a more custom syntax for SigNoz). I think I’m going to give that a try just to see if it may be easier to set up and integrate, but I’m getting to the point where it’s getting harder to justify putting off actually working on a gift registry, so I can’t afford to spend a lot of time going down this rabbit hole.
It’s been a few years since I’ve just decided to start doing something I’ve had no historical experience with and with no existing infrastructure I can use as a reference, and it’s easy to forget just how overwhelming that initial attempt to learn something can be. The ironic thing about is the information overload – there’s tons of documentation out there (I’ve got a massive tab group that’s almost entirely official documentation or code in Github). But tons of content on a general topic aren’t useful getting help with specific problems, if that section of the docs have even been written yet. That’ll likely get better as I gain familiarity with any particular piece of this, but that part takes time.
As useful as having something already in place to use as a starting point is, this has made it clear that my “learning new things muscles” have clearly started to atrophy. The good news is, I’m at least aware of the problem and am actively working on fixing that now. Oh sure, I’ve read been lots on new developments of “best practices,” about newer languages that are gaining popularity and some of their main features, and just software development in general, but none of that is actually learning. It’s just surface-level information that can be easily recited, but without the slightest bit of real understanding.
I am remembering the importance of both properly scoping what I want to and also having a good plan around achieving it, specifically a good starting point for building off of. It’s sort of like putting together all the edge pieces of a puzzle, doing that upfront gives me a set of bounds that I know has everything has to fit within. That’s a lot harder to conceptualize with software without a very clear set of deliverables (which I didn’t really define for myself).
A big problem I had with scoping is balancing all the “operational” stuff I want to do (not using managed services, incorporating some sort of observability stack so I can practice instrumenting my code for it, adding in all the little pieces that enable me to run a Docker container as a DIY webapp instead of just I’ll put it on service that does all that for me), and learning and building something in Go. It’s easy to get lost in the weeds of trying to get something configured just the way I may want it, but pausing to remind myself that I’m not interested in running an OpenTelemetry-compatible server but in how integrating with that OpenTelemetry-compatible server improves my code. I want to get a reverse proxy running to route traffic to a Docker container because I want to understand just how hard it is to do myself, and how being the person running it affects my code.
My ability to dive into something I don’t do and learn may be rusty, but I’m starting to get back into the groove. It hasn’t been so long since that I don’t remember the feeling of being in over my head trying to figure something out – every time I’ve come out with better skills. This will be no different. Yes, there’s a lot of other stuff distracting from “just write the code,” but those somethings are part of actually running applications for real – not just on my laptop, and not just on managed hosting as a proof-of-concept. It’s a lot of work (and stressful) to start new on something you don’t know how to do. But, like I keep trying to get my son to appreciate, it’s not about doing fun things because they’re easy, it’s about working at hard things because it’s going to be really cool when we pull them off.