Match Containers to Host Processes

During my presentation Securing Container Workloads on AWS Fargate, I built a demo environment where I could build and run various containers and show the effect they had on the host. While my demo went well, a key piece of feedback is that customers liked how I presented the demo environment by having containers and their host processes on one side. To that end, I’ll show you.

Containers Pane

To show the currently running containers on a given host, use docker ps. The normal format (for v18.09.1) looks like:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
525a7b49ef67        nginx               "nginx -g 'daemon of…"   About an hour ago   Up About an hour    80/tcp                   tender_shirley

However, for this demo, I was only concerned with the name, image, command, and current status (which has the time it’s been running), so I formatted the output using the --format flag, and stuck it inside watch to update every second.

Command
watch -n 1 "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Command}}\t{{.Status}}'"
Output
Every 1.0s: docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Command}}\t{{.Status}}' localhost.localdomain: Sat Feb 23 13:44:58 2019 NAMES IMAGE COMMAND STATUS tender_shirley nginx "nginx -g 'daemon of " Up About an hour

Host Processes Pane

Getting the host processes (and a way to map them to containers) was more difficult. The best tool in Linux for looking at processes is ps (which is where Docker gets the name for docker ps), but this doesn’t give us all the information about a container.

When a container starts, it spawns as a process with a specific process identifier (PID) in the host, but the container sees the PID as 1. This process can also spawn other processes, which will reference ther parent process PPID. Subprocesses will show up with a PPID of the main process PID but inside the container as PPID 1. For my demo, I wanted to show both the processes and subprocesses at the host, and include information about the user running each process.

Thus, I built a script called watchpids.sh. This script gathered the host PIDs, found all of the children PIDs and then fed the list of PIDs to ps, also formatting the list to show the running time of the process, the PID, the PPID, the user associated with the process, and the command run. Again, execution of the script was wrapped in watch.

Script
#!/bin/bash
pidlist() {
local thispid=$1
local fulllist=
local childlist=
childlist=$(ps --ppid $thispid -o pid h)
for pid in $childlist
do
fulllist="$(pidlist $pid) $fulllist"
done
echo "$thispid $fulllist"
}
dockerpids() {
local fulllist=
local dockerpids="$(pidof docker-containerd-shim)"
for p in ${dockerpids}
do
for q in $(ps --ppid $p -o pid h)
do
fulllist="$(pidlist $q) $fulllist"
done
done
echo "$fulllist"
}
ary=($(dockerpids))
DOCKER_PIDS=${ary[@]}
ps -Ho etime,pid,ppid,user,cmd --pid ${DOCKER_PIDS// +/,} 2> /dev/null
view raw watchpids.sh hosted with ❤ by GitHub

With both the containers and processes displayed, map the container STATUS to the host process ELAPSED time to see what processes show up on the host whenever a new container is started.

Terminal Window

Tying it all together, I used tmux to build the container and host process panes on the right, and an area to type commands on the left.

tmux uses either keyboard shortcuts or commands inside the session to change the environment–going for a “scripting” approach, I chose the latter.

Commands
tmux new-session -d -s builder_demo
tmux split-window -h
tmux split-window -dv "watch -n 1 \"docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Command}}\t{{.Status}}'\""
tmux select-pane -t 0
tmux send-keys -t 1 'watch -n 1 ./watchpids.sh' C-m
tmux -2 attach-session -d
Screenshot

Securing Container Workloads on AWS Fargate

When containers first became mainstream (think PyCon 2013 with Solomon Hykes on stage), everyone thought it had potential and began to test running containers on their own, but almost no one set out to put containers in production that day. They wanted to see it battle-tested…which has happened over time. Containers have matured from an emerging technology to production-ready where it’s generally considered safe, but there’s a new problem. Now, we need our business processes, tools, and architecture models to mature as well.

The top ask I hear about containers comes down to security. Containers were built as a way to isolate workloads for one another, but many of the security models from virtual machines do not work in containers, and thus we must evolve our thought process.

To that end, I presented an AWS Online Tech Talk about how to secure container workloads using AWS Fargate (though many of the lessons also apply generally across containers). I demonstrated some quick steps to make your containers more secure during the build process as well as how to enhance visibility and security around containers running in AWS Fargate.

Docker Hugo

After restarting my blog, I wanted a way to automate my workflow. I currently work for AWS, and want to use the features of the cloud to manage and deploy my blog, but for as little cost as possible. The lowest cost for a static site like mine is Amazon S3, which offers to host the objects in the bucket as a static website.

This starts by adopting a solid framework for building static sites. After trying a few, I selected Hugo. I had been using mkdocs for training/tutorials but felt it lacked a good native layout engine and wasn’t a good fit for a blog.

I followed the installation instructions, but wanted something I could containerize (since it’s relevant to my current work). Thus, I created docker-hugo as a simple project to containerize hugo.

For now, this includes a README and a Dockerfile (copied as of August 3, 2018):

FROM centos:latest as builder

RUN yum -y update
RUN curl -sL -o hugo.tar.gz https://github.com/gohugoio/hugo/releases/download/v0.46/hugo_0.46_Linux-64bit.tar.gz && tar zxf hugo.tar.gz hugo

FROM scratch

COPY --from=builder /hugo .

VOLUME /host
WORKDIR /host
EXPOSE 1313

ENTRYPOINT ["/hugo"]

While a simple example, it does combine some newer Docker features. I used a multi-stage build to download the actual binary, then a scratch image for the actual deployment. The README highlights the syntax I use for the command, and an alias for being able to run hugo new posts/docker-hugo.md with all of my environment variables already plugged in. This can also be adapted for a future CI/CD process.