Lately I’ve had the good fortune of working on an app migration effort with a heavy focus on containerization, specifically a couple of batch processes which run daily. Formerly, these processes were run by an enterprise scheduling system which handled alerting. After deciding to rewrite the batch processing functions and containerize them, that left me short alerting. I decided to implement a new alerting solution using Datadog, since we’re already using it to gather metrics for the main application.
In my last post, we discovered how to customize some of Datadog’s pre-packaged integrations to build actionable insights for SQL-backed applications quickly. In this post, we are going to dive a bit deeper and look at how we might integrate Datadog with pre-built/custom metrics tooling (such as a shell script, for example). As I wrapped up the last post, I alluded to a metric “gathered via a different route”; this metric keeps a count of the number of individual client processes running on the host.
“If you can’t measure it, you can’t improve it.” This quote, attributed to Peter Drucker, has held a lot of sway over the years in management circles where high-level business processes are created, iterated, tweaked, and refined to ensure that revenues (and hopefully, profits) lead to healthy, long-lasting companies. At Blue Sentry, our Managed Services offering is built upon this same maxim. We rely on several tools to provide insight into customer environments; in this post, I am going to focus on one of my newfound favorites from this toolset – Datadog – and its ability to capture custom metrics.