Lately I’ve had the good fortune of working on an app migration effort with a heavy focus on containerization, specifically a couple of batch processes which run daily. Formerly, these processes were run by an enterprise scheduling system which handled alerting.
After deciding to rewrite the batch processing functions and containerize them, that left me short alerting. I decided to implement a new alerting solution using Datadog, since we’re already using it to gather metrics for the main application. To do so, I decided on the datadogpy library (since my function was written in Python).
Integrating with the API was dead simple, basically as simple as the README in the repo makes it look. Testing out my batch processing containers, though, I would sometimes get a strange error like this:
No agent or invalid configuration file found
Since I was just using the API with an SDK (e.g. no agent, no config file), the error seemed pretty curious to me. After doing some googling and source-code rummaging, I discovered that this error occurs when the datadog SDK can’t detect a hostname. The fix was as easy as adding a host_name attribute to the options dict passed to the initialize(**options) function. After that, the transient error was no longer a problem.
I don’t understand all of Docker’s internals well enough to understand why some containers get a hostname and some don’t, but this resolves the issue quite nicely. In fact, in the instances when the container did get a hostname, it was a random hash-like string. Overriding it with a custom name creates an additional dimension of discoverability for my events.
Happy coding!