Curated Lists - My Favorite "DevOps" Tools (and Then Some)

devops, tools, curated-list

First off, forgive the title. I fall much more on the side of “devops” as a cultural mindset than a thing we do, but when it comes to searching for and exposing things for search on the interwebs, I’m going with the status quo. Next, a few caveats:

  • I haven’t used every single one of these to great extent (or some even at all), but they are tools that are on my radar. For things that fall into the “never used it, but it looks intriguing” category, I’ve so noted it with a double-asterisk (**) by the name of the tool (also, somes things may get a single asterisk if I’ve used it, but not to a significant degree).
  • Most of these tools are quite general or very specific to AWS. As far as cloud goes, my experience lies in AWS. As that begins to change, so will this document (I imagine I will re-work the layout at that time, too, since it will add a new dimension to consider).
  • Some of these are more developer-focused than SRE-focused, and I’m ok with that for the time being.

And now, down to brass tacks…

Infrastructure Automation

  • Terraform - I suppose my bias here is probably pretty apparent, but I think this one is indispensible.
    • Terraforming - Exports existing AWS infra to Terraform code (HCL) and/or to Terraform statefiles. For HCL code exports, solution is somewhat limited (not all resources supported), and be aware that this doesn’t parametrize your code in any sort of way (e.g. you still have a fair amount of work to do to get your templates looking like they should).
  • CloudFormation - Obviously this one’s landlocked to AWS, but it is a good way to get started learning infra automation if you’re working solely in AWS.
    • CloudFormer - Creates CloudFormation templates based off of existing resources. Solution is somewhat limited (not all resources supported), and be aware that this doesn’t parametrize your code in any sort of way (e.g. you still have a fair amount of work to do to get your templates looking like they should).
  • Troposphere - Python language wrapper around CloudFormation templates
  • botoform - Python/YAML-based declarative infra orchestration for AWS

Secondary Tooling

Most of the secondary tools at this point are serverless frameworks that actually leverage an orchestration tool (such as CloudFormation), but I feel like they are worth calling out.

Infrastructure Automation - Add-Ons/Ecosystem Tools

Terraform

CloudFormation

Configuration Management

  • Ansible - Simple YAML-based declarations allow straightforward configuration management. Sort of my favorite of the bunch.
  • Chef - Chef’s great, especially if you’re cooking something with Windows.
  • SaltStack - OK, I lied. SaltStack really is my favorite tool in this space, but I usually opt for Ansible because of its de-facto agentless nature, and more folks seem to understand and have exposure to Ansible.
  • Packer - Create custom VM images (integrates with tools such as Ansible).
  • Vagrant - Use the images you created with Packer :)
  • Converge - Chef and Terraform get together and have a baby. Seriously, this looks way cool and I can’t wait to play around with it.

Configuration Management - Add-Ons/Ecosystem Tools

Ansible

  • Ansible AWX** - The open-source upstream version of Ansible Tower.
  • ara - Allows you to aggregate and query Ansible run logs (which also seems beneficial for auditing and compliance purposes, though that’s not called out in the README) with a really nice UI to boot.
  • ansible-toolkit - Really helpful tooling if you do much with Ansible Vault en masse. The atk-show-vars tool is really helpful for sorting out variable precedence issues, as well.

SaltStack

Logs

  • awslogs - CLI tool to work with CloudWatch logs
  • cwlogs - Similar to awslogs, but with different syntax

Alternative CLIs/CLI Extensions/CLI Utilities

Altenative CLIs

  • awless - “a powerful, innovative and small surface command line interface (CLI) to manage Amazon Web Services”. Also has some interesting inbuilt state management for resources created using awless.
  • aws-shell - An integrated AWS shell

CLI Extensions

  • oh-my-zsh - If you’re a zsh user, the aws plugin is definitely worth enabling. It exposes agp and asp commands, allowing you to easily manage AWS_PROFILE env vars (w/ tab completion, etc.).

CLI Utilities

  • jq - An indispensible tool to grok JSON output. Yeah, you get the JMESPath output parser in the AWS CLI, and that’s worth mastering, but you can’t use that to grok output from curl or any other tool. The @csv function in jq is pretty handy, too, for ad-hoc reporting (like a list of EC2 instances or S3 buckets or something).

Local Package Management

Service Mocks/Local Development

  • localstack - Emulate/mock AWS services locally
  • sam-local - Run Lambdas locally with a local API Gateway
  • Moto - Mocks for boto

IAM/Credentials/Secret Management - Local

  • aws-mfa - Manages AWS passthrough credentials requiring MFA
  • aws-vault - Manages long-lived AWS credentials by storing them securely in OSX keychain and leveraging STS credentials
  • Keybase - A super-awesome wrapper/augmentation of GPG/PGP. Unbeknownst to many, Terraform also directly integrates with it for securing credentials for transmission.
  • LastPass - I’ve been a happy user of LastPass for years. This is another service that I don’t mind ponying up some cash for (Premium account). They also have a business account with ACL-based controls that work pretty well. I primarily use my account from my CLI using the lastpass-cli tool. [Sidenote: At BlueSentry, we even had a CLI tool that worked with SSH keys stored in LastPass (once again, ACLs, and not wanting anyone to have local copies of client’s SSH keys) to ephemerally deliver keys locally when needed to SSH into customer’s environments (there’s also a Docker container, if that’s your thing).]

IAM/Credentials/Secret Management - Application Deployments

  • Vault - The gold standard in this space (IMO)
  • Torus - An interesting offering that abstracts away some of the complexity of a Vault implementation
  • Conjur - Open-source offering from CyberArk
  • BLESS - A very interesting solution to dispatch short-lived, signed SSH keys (yes, you read that right) using Lambda functions. Vault is capable of similar functionality, but the PaaS implications of BLESS vs running infra for Vault make it worth considering.
  • chamber - Yet another outflowing of one of AWS’ greatest and most unsung services (SSM).

IAM/Credentials - Platform

  • AWS IAM Policy Simulator - I’m always amazed that people don’t know that this tool exists. If you’re actually trying to bolt down IAM policies and roles, this is indispensible.
  • How to Write a Cross-Account Lambda - Simple Lambda function that shows how Lambda can assume-role into other accounts and explains some of the setup necessary from the IAM side of things.
  • aws-azure-login - AWS CLI Access using Azure AD Federation

SSL Cert Management

  • letsencrypt - Free, automatable SSL certs - it really doesn’t get much better than this. I do also realize that ACM certs are free, but it’s nice having a somewhat platform-agnostic solution in this area. It’s pretty easy to upload letsencrypt certs to use in AWS via IAM, too (that’s how I do it).

Host Security

  • Tripwire (open-source) - Open-source version of a widely-used enterprise HIDS tool.
  • OSSEC - Amazing suite of tools, but definitely requires some knowledge and overhead to implement correctly.
  • syschangemon - Not as full-bodied as some of the other solutions, but is easy to deploy and has met my needs for deployments in the past.

Monitoring/Alerting/Self-Healing Infra

  • Datadog - I’m cheap, and you’ll notice that most of the tools in this list are free. Datadog’s not, but it’s worth what you pay for it.
  • Monit - Monitors daemons running on hosts and lets you trigger custom actions when things stop working like they should. A great way to keep your ethproxy up and running.
  • Automatron - Another self-healing service tool.
  • Sensu - Wholesale monitoring/alerting solution. Supports custom actions for triggered alerts.
  • netdata - Nice single pane-of-glass of view for pretty much everything you’d possibly want to monitor for a host. Also supports alerting.
  • cabot** - Self-identifies as a hybrid of nagios, PagerDuty, and others
  • stackstorm/st2** - Self-described ‘IFTTT for Ops’

Logging/Log Aggregration

  • Elastic/ELK Stack - So much the “de-facto” solution in this space that I hesitate to mention it. So, if you’re hearing about it here for the first time, go find some blog posts and start reading up.

CI/CD Servers/Tooling

  • CircleCI - Standard disclaimer as with the other SaaS tools here - I love free, OS stuff you can deploy, configure and manage yourself. But Circle is a fantastic tool; they also offer free projects for public GitHub projects.
  • Jenkins - Once again, if you don’t have some Jenkins experience, you’ve got some catching up to do. Need to knows: Job DSL, how to code a pipeline, and what some of the more common/useful plugins are.
  • drone - Another great open-source CI server
  • LambCI** - A serverless CI/CD framework designed to be deployed on AWS

Build Tooling/Dependency Management

You’ll encounter these locally and in CI/CD contexts.

Mobile

  • fastlane - Don’t try to apply CI/CD practices to an iOS app without this (also supports a lot of Android build functions).
  • cocoapods - Swift dependency manager.
  • gradle - De-facto Android build tool.

Java/misc

  • maven - Where Java projects are being built, Maven’s there.
  • ant - Predecessor to maven. I’ve seen it used in lots of “interesting” contexts outside of pure software project builds, too.

JavaScript/misc

  • grunt - The first of the pack (also my favorite). Like maven or ant, doesn’t really have to be just used to build language-specific projects.
  • gulp - I have used this (though to a lesser extent than Grunt) quite a bit. I like the stream-based concept, but it implies a bit more async programming awareness and understanding (IMO).
  • yarn - Node dependency manager
  • npm - Node dependency manager

Ruby/misc

  • Capistrano - If you’ve worked with a Ruby shop, you’ve probably encountered Capistrano. Works with non-Ruby projects as well.

PHP/misc

  • Composer - PHP project builder/dep manager. Inspired by npm and the like.

API Design

  • Swagger - I love Swagger. CloudFormation has supported the ability to import Swagger specs for API Gateway resources, with Terraform lagging behind. Terraform finally caught up sometime last summer.

Job Orchestration/Scheduling

  • rundeck
  • Nomad - Yeah, Nomad is mostly pitched as a container scheduler, but it can actually do a lot more than that. Read up on its documentation and its capabilities. There are a ton of good videos from HashiConf 2017 highlighting some implementation success stories - check out the Elsevier and the PagerDuty videos.
    • replicator** - Looks almost like an app-tier autoscaling framework for Nomad.

Containerization

  • Docker - Pretty much the same disclaimer applies here as for the ELK stack.
    • kubernetes** - Yeah, shameful that this bears the “haven’t used yet” designation for me. Hopefully the opportunity presents itself shortly.
  • rkt - A different take from Docker’s approach
  • lambci/docker-lambda - Take your existing Lambda code, copy it into a Docker image, and it runs like magic. Seriously, this is a fantastic project that you can leverage when you PoC some Lambda solution that won’t work because need more than five minutes for single function execution or a bit more control over the execution environment than Lambda gives you.

Database Management

CI/CD / Versioning

Other Tools

  • dbatools - SQL Server/Powershell
  • pymssql - Python interface to SQL Server (handy for automating retrieval/ CI/CD of stored procs, triggers, UDFs, etc.)
  • sqlcmd/bcp - Native MS SQL tooling available for non-Microsoft OSes (e.g. Linuxes, macOS)

Reverse Proxy/Traffic Routing/Load Balancing

  • haproxy - Most of my experiencing with load balancing and routing is with HAProxy. It is an extremely powerful solution. There are some projects out there that expose a REST API above HAProxy to make it easier to automate.
  • nginx - Also a very capable load balancer/router/reverse proxy. Also a fantastic webserver :D
  • traefik - Easily orchestrated and automated and can handle pretty insane scale.
  • fabio - Load balancing that works with data dynamically retrieved from Consul (so cool).

Service Discovery/Config Services

  • Consul - Key-value store, service discovery tool, cloud Swiss army knife. Batteries included as far as an HTTP API interface goes.
  • etcd* - Very similar to Consul. Uses RBAC control to data instead of ACLs, which is more intuitive to me.
  • Zookeeper* - Config service used by Kafka, which is the only context in which I’ve really dealt with it. Doesn’t support an HTTP interface natively (boo).

Screencasting

  • asciinema - Awesome terminal session capture utility; share as .mng vids.
  • obsproject - Open-source video capture
  • camtasia - Video capture and editor (not free, but powerful and relatively straightforward to use).

Repo Management/VCS

  • git-subrepo - To my consulting crowd… Cost of external seats for GitHub collaborators got you down? I used to house multiple projects per client internally. Then, under a separate project, use this to front all projects under one repo (a “meta-repo,” as I like to call it), then add the clients’ collaborator seats to the meta-repo (thereby saving you adding the same collaborators to all the other individual projects), though there are certainly many other use-cases.
  • hub - A most awesome tool. Manage PRs from the CLI!
  • legit - Mixed feelings about this one, but I could see where someone might want to use it.
  • gogs - As alluded to earlier, I’m cheap. I used to pay for private GitHub repo hosting. Then I found gogs; then I installed Gogs on my Raspberry Pi; then I migrated all my private GH repos into said Gogs install on my RPi and canceled my GitHub subscription. Also, I’ve tried lots of other solutions in this space (primarily when I used to freelance); most of the other self-hosted git solutions are quite painful and come nowhere near offering the ease of administration and GitHub-esque feature set that Gogs does.

Info Management/Sharing

  • Confluence - Fantastic wiki tool available from Atlassian, though I lament its lack of native support for Markdown
  • dokuwiki - I love, love, love DokuWiki. It’s good quality, solid, free OSS software. It’s only dependency is PHP, so it’s not terribly difficult to run on your local if you’re so inclined (e.g. you do not need a database for this one!).

Project Management

  • taiga.io - A full agile PM suite. A very solid (free) offering.
  • Jira - Atlassian’s agile tracking software. Probably needs no introduction or explanation.
    • go-jira - In past consulting roles, I often played a pseudo-scrummaster role in addition to being a technical performer/lead. While not so much the case anymore, I fell in love with this tool during those days when I was closer to the planning side of things. I found this tool much faster and more effective than playing in the Jira web interface once I get my head around it.
  • Redmine - Full, free, opensource project management suite. I ran my own consulting company for years using Redmine. It’s not perfect, but it gets the job done and is extensible enough for any fairly adept programmer to extend it to meet their needs.

Guides/Info Sources

  • og-aws - AWS Canon
  • lastweekinaws - Great curated weekly roundup with some serious snark
  • Gruntwork blog - These guys know something ‘bout some Terraform. Seriously, a good DevOps resource in general.
comments powered by Disqus