Sunday, 24 April 2016

Best Practices for Docker Container deployments

Container technology like Docker leads to faster testing and deployments of software applications. The following is a list of some best practices to consider when using containers. The key objectives are :
Repeatability
Reliability
Resiliency

To achieve this :

1. Have a single code base, tracked in git with many deploys

  • Docker containers should be immutable
  • Use the environment variables to change anything inside the container
  • Do not build separate images for staging and production

2. Explicitly declare and isolate any dependencies

  • Do not use latest in images. Instead define the exact version e.g. ubuntu:latest vs ubuntu:16.04
  • Its useful to build run times (e.g. Java run time) based on specific images
  • Process -> Base OS -> Run Time (e.g. Java run time) -> Add app

3. Store configuration in the environment

  • Do not have a config.yml or properties.xml
  • Always use environmental variables

4. Treat backing services as attached resources

  • Never use local disk
  • Data will always disappear
  • Connect to network services using connection info from the environment e.g. DB_URL

5. Strictly separate build and run stages

  • Build immutable images and then run them
  • Never install anything on deployments
  • Respect the life cycle : build, run, destroy

6. Execute the application as 1 or more stateless processes

  • Schedule long running processes by distributing them across a cluster of physical hardware

7. Export services via port binding

  • Define ports from environment variables
  • A PID cannot guarantee the port in the container

8. Scale out via process model

  • Horizontally scale by adding instances

9. Maximize robustness with fast start-up and graceful shutdown

  • Quickly scale up when there is a load spike
  • For a data intensive app its tempting to load data into memory as a hot cache. Issue is that this stops having a fast start-up (as need to load data into memory). Could use Redis or memcache as an alternative

10. Keep development, staging and production as simple as possible

  • Run containers in development e.g. DB, caching

11. Treat logs as event streams

  • Log to standard output (stdout) and standard error (stderr)
  • Use something like the ELK stack to collect all the logs
  • Should never need to ssh to check logs
  • There should not be random log files in the container

12. Run admin/management tasks as one off processes

  • Do not use customer containers as one off tasks
  • Reuse app images with specific entry points for tasks

Sunday, 17 April 2016

systemd-analyze - Analyze system boot-up performance

The systemd project has an excellent tool systemd-analyze which allows an analysis of system boot performance.

Running without any parameters displays the startup time in total with figures for the kernel + userspace
$ systemd-analyze
Startup finished in 1.190s (kernel) + 8.312s (userspace) = 9.503s
The option blame prints a list of all running units, ordered by the time they took to initialize. This may be used to optimize boot-up times. The output may be misleading as the initialization of one service might be slow simply because it waits for the initialization of another service to complete.
For example the 3 applications that took longest to initialise :
$ systemd-analyze blame | head -n 3
          7.621s man-db.service
          3.174s docker.service
          1.134s mysqld.service
The option critical-chain prints a tree of the time-critical chain of units (for each of the specified UNITs or for the default target otherwise).

The time after the unit is active or started is printed after the “@” character. The time the unit takes to start is printed after the “+” character. Note that the output might be misleading as the initialization of one service might depend on socket activation and because of the parallel execution of units
For example reviewing the docker.service unit

$ systemd-analyze critical-chain docker.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

docker.service +3.174s
`-network.target @1.287s
  `-NetworkManager.service @830ms +172ms
    `-dbus.service @691ms
      `-basic.target @682ms
        `-sockets.target @682ms
          `-docker.socket @681ms +481us
            `-sysinit.target @681ms
              `-systemd-backlight@backlight:acpi_video0.service @1.321s +41ms
                `-system-systemd\x2dbacklight.slice @1.320s
                  `-system.slice @123ms
                    `--.slice @102ms

The plot option prints an SVG graphic detailing which system services have been started at what time, highlighting the time they spent on initialization
$ systemd-analyze plot > plot.svg

How to paste source code in Vim without losing formatting

When pasting code into Vim automatic indenting can occur and your code formatting will be incorrect. To stop this issue there are two options :

Paste option


Within vim
    : set paste

Vimrc function key mapping


It can be useful to have a mapping in your ~/.vimrc file to quickly allow enable/disable of the paste feature :
" Quickly enable/disable 'paste' option
set pastetoggle=<F10>

Saturday, 16 April 2016

Upgrading to GNOME 3.20 from 3.18

Every 6 months the GNOME team releases a new version of the Desktop. Running Arch Linux its great to be able to have quick access to the new features quickly. Running near to the bleeding edge always has a few quirks and here are a few I encountered.

Extensions claiming they are not supported

This is a common issue for me on every GNOME upgrade. I discovered this check can quickly be disabled as follows :
$ gsettings set org.gnome.shell disable-extension-version-validation "true"

No Topleft Hot Corner Extension stopped working

One of my ‘must have’ extensions is no topleft hot corner which stopped working. I found that removing the extension from the extensions site and then installing it from the AUR works as its a newer version :
$ packer -S gnome-shell-extension-nohotcorner-git
This installs the extension as follows :
$ ls /usr/share/gnome-shell/extensions/nohotcorner@azuri.free.fr/
extension.js  metadata.json

Excessive Padding in Applications

The css selectors changed in GNOME 3.20 which means that the padding on titlebars e.g. Firefox and terminal is large. GNOME provides a config file for users to tweak their own environment ~/.config/gtk-3.0/gtk.css. I updated mine to :
/* shrink headebars */
headerbar {
    min-height: 0;
    padding-left: 2px; /* same as childrens vertical margins for nicer proportions */
    padding-right: 2px;
    padding-top: 1px;
    padding-bottom: 1px;
}

headerbar entry,
headerbar spinbutton,
headerbar button,
headerbar separator {
    margin-top: 1px; /* same as headerbar side padding for nicer proportions */
    margin-bottom: 1px;
}

 /* shrink ssd titlebars */
.default-decoration {
    min-height: 0; /* let the entry and button drive the titlebar size */ 
    padding-left: 2px; /* same as childrens vertical margins for nicer proportions */
    padding-right: 2px;
    padding-top: 1px;
    padding-bottom: 1px;
}

.default-decoration .titlebutton {
    min-height: 5px; /* tweak these two props to reduce button size */
    min-width: 5px;
}

/* reduce the padding size of buttons */
button, entry {
  min-height: 0;
  min-width: 0;
  padding: 2px;
}

/* reduce the padding of tabs */
notebook tab {
  min-height: 0;
  padding-top: 3px;
  padding-bottom: 3px;
}

/* reduce the padding of buttons */
notebook tab button {
  min-height: 0;
  min-width: 0;
  padding: 2px;
  margin-top: 2px;
  margin-bottom: 2px;
}

notebook button {
  min-height: 0;
  min-width: 0;
  padding: 2px;
}

/* add padding for Vte-terminal */
widget {
    padding: 4px;
}
Restart GNOME shell with Alt + F2 then type ‘r’ which will restart the desktop to ally the changes.

Monday, 28 March 2016

chromium - FATAL:setuid_sandbox_client.cc(126)] Check failed: IsFileSystemAccessDenied()

Testing running chromium in Docker I had the following stack trace :
[0329/035746:FATAL:setuid_sandbox_client.cc(126)] Check failed:
 IsFileSystemAccessDenied().
#0 0x55e51a82299e <unknown>
#1 0x55e51a8007ee <unknown>
#2 0x55e51a8934fe <unknown>
#3 0x55e51a7f3f3b <unknown>
#4 0x55e51a7f21a6 <unknown>
#5 0x7fe5e8984710 <unknown>
#6 0x55e51a7f34f9 <unknown>

[1:1:0329/035746:ERROR:nacl_fork_delegate_linux.cc(315)] Bad NaCl helper
 startup ack (0 bytes)
[1:1:0329/035746:FATAL:setuid_sandbox_client.cc(126)] Check failed: 
IsFileSystemAccessDenied().
This is because chromium should not be run as root. From the development list this is by design.

Sunday, 27 March 2016

Arch Linux pacstrap options

When creating a base Arch Linux image from scratch for Docker I found the documentation for pacstrap to be limited.
Pacstrap is installed as part of the arch-install-scripts and I eventually found the following options running as the root user
# pacstrap -h
usage: pacstrap [options] root [packages...]

  Options:
    -C config      Use an alternate config file for pacman
    -c             Use the package cache on the host, rather than the target
    -d             Allow installation to a non-mountpoint directory
    -G             Avoid copying the host's pacman keyring to the target
    -i             Avoid auto-confirmation of package selections
    -M             Avoid copying the host's mirrorlist to the target

    -h             Print this help message

pacstrap installs packages to the specified new root directory. If no packages
are given, pacstrap defaults to the "base" group.
The docker script I used was :
spawn pacstrap -C $PACMAN_CONF -c -d -G -i $ROOTFS base haveged 
$PACMAN_EXTRA_PKGS --ignore $PKGIGNORE

Build Arch Linux base image for Docker

As there is no official Docker Hub build for Arch Linux I thought this would be a good oppertunity to find out how images are created from scratch for OS’s. IMHO its good practice to create your own images to ensure you know exactly the providence of what is installed in your container.
As a starting point I used the mkimage-arch.sh script on the Docker github contrib area.
Script must be run as root
if [ “$(id -u)” != “0” ]; then
printf "This script must be run as root\n" 
exit 1
fi
There are two required packages : expect + pacstrap which can be installed with :
$ pacman -S arch-install-scripts expect
The script checks for this as follows :
hash pacstrap &>/dev/null || {
    printf "Could not find pacstrap. Run pacman -S arch-install-scripts"
    exit 1
}

hash expect &>/dev/null || {
    printf "Could not find expect. Run pacman -S expect"
    exit 1
}
The || is similar to && except it tells the shell only to evaluate the expression after it when the first expression fails.

The hash command affects the way the current shell environment remembers the locations of utilities found. If run without any parameters it shows the path of all commands run since the hash was last reset (hash -r) e.g.
$ hash
hits    command
   1    /usr/bin/git
   1    /usr/bin/vim
   3    /usr/bin/cat
   1    /usr/bin/touch
   1    /usr/bin/mv
   1    /usr/bin/mkdir
   3    /usr/bin/man
  13    /usr/bin/ls
The hash table is a feature of bash that prevents it from having to search $PATH every time you type a command by caching the results in memory. For this use case we use it as a way to test if the command is available.
Next we set the language as UTF-8 :
export LANG="C.UTF-8"
Create a temporary root file system with mktemp :
ROOTFS=$(mktemp -d /tmp/rootfs-archlinux-XXXXXXXXXX)
mktemp creates a temporary directory (or file) based on the template provided to randomize the name. Each X value is replaced with a random string.

Set permissions
chmod 755 "$ROOTFS"
Define the packages to not install for minimal image
PKGIGNORE=(
    cryptsetup
    device-mapper
    dhcpcd
    iproute2
    jfsutils
    linux
    lvm2
    man-db
    man-pages
    mdadm
    nano
    netctl
    openresolv
    pciutils
    pcmciautils
    reiserfsprogs
    s-nail
    systemd-sysvcompat
    usbutils
    vi
    xfsprogs
)
Expanding an array without an index only gives the first element. The $IFS is a special shell variable which stands for Internal Field Separator.
IFS=','
PKGIGNORE="${PKGIGNORE[*]}"
unset IFS
printf "%s""\nPackages not to be installed : $PKGIGNORE\n"
Set pacman.conf to provided conf file
PACMAN_CONF='./arch-docker-pacman.conf'
Define the mirror for pacman :
PACMAN_MIRRORLIST='Server = https://mirrors.kernel.org/archlinux/\$repo/os/\$arch'
Set basic variables to create image
PACMAN_EXTRA_PKGS=''
EXPECT_TIMEOUT=60
ARCH_KEYRING=archlinux
DOCKER_IMAGE_NAME=archlinux
Export pacman mirror
export PACMAN_MIRRORLIST
Use expect to auto reply to pacstrap
expect <<EOF
    set send_slow {1 .1}
    proc send {ignore arg} {
        sleep .1
        exp_send -s -- \$arg
    }
    set timeout $EXPECT_TIMEOUT

     spawn pacstrap -C $PACMAN_CONF -c -d -G -i $ROOTFS base haveged $PACMAN_EXTRA_PKGS --ignore $PKGIGNORE
    expect {
        -exact "anyway? \[Y/n\] " { send -- "n\r"; exp_continue }
        -exact "(default=all): " { send -- "\r"; exp_continue }
        -exact "installation? \[Y/n\]" { send -- "y\r"; exp_continue }
    }
EOF
Remove manual files to save space
arch-chroot "$ROOTFS" /bin/sh -c 'rm -r /usr/share/man/*'
Use haveged to generate random numbers and feed linux random device. You must run pacman-key –init before first using pacman; the local keyring can then be populated with the keys of all official Arch Linux packagers with pacman-key –populate archlinux.
arch-chroot "$ROOTFS" /bin/sh -c "haveged -w 1024; pacman-key --init; pkill haveged; pacman -Rs --noconfirm haveged; pacman-key --populate $ARCH_KEYRING; pkill gpg-agent"
Set local timezone to UTC
arch-chroot "$ROOTFS" /bin/sh -c "ln -s /usr/share/zoneinfo/UTC /etc/localtime"
Set locale to ‘en_US.UTF-8 UTF-8’
echo 'en_US.UTF-8 UTF-8' > "$ROOTFS"/etc/locale.gen
arch-chroot "$ROOTFS" locale-gen
Set pacman mirrorlist
arch-chroot "$ROOTFS" /bin/sh -c "echo $PACMAN_MIRRORLIST > /etc/pacman.d/mirrorlist"
udev is a device manager for the Linux kernel. Udev primarily manages device nodes in the /dev directory and also handles all user space events raised while hardware devices are added into the system or removed from it, including firmware loading as required by certain devices.
udev doesn’t work in containers, rebuild /dev
DEV=$ROOTFS/dev
rm -rf "$DEV"
mkdir -p "$DEV"
mknod -m 666 "$DEV"/null c 1 3
mknod -m 666 "$DEV"/zero c 1 5
mknod -m 666 "$DEV"/random c 1 8
mknod -m 666 "$DEV"/urandom c 1 9
mkdir -m 755 "$DEV"/pts
mkdir -m 1777 "$DEV"/shm
mknod -m 666 "$DEV"/tty c 5 0
mknod -m 600 "$DEV"/console c 5 1
mknod -m 666 "$DEV"/tty0 c 4 0
mknod -m 666 "$DEV"/full c 1 7
mknod -m 600 "$DEV"/initctl p
mknod -m 666 "$DEV"/ptmx c 5 2
ln -sf /proc/self/fd "$DEV"/fd
tar root file system and import image to Docker.
tar --numeric-owner --xattrs --acls -C "$ROOTFS" -c . | docker import - "$DOCKER_IMAGE_NAME"
Options for tar used :
--numeric-owner  Always use numbers for user/group names.
--xattrs         Enable extended attributes support
--acls           Enable the POSIX ACLs support
-C               Change to directory
-c               Create new archive 
Test new image
docker run --rm -t $DOCKER_IMAGE_NAME echo Success
Delete temp root file system
rm -rf "$ROOTFS"
Running the script should finish with
tar: ./etc/pacman.d/gnupg/S.gpg-agent: socket ignored
sha256:82b0356924efb95e5483417dd4f0cef85fce7afbacb75c18632de7bc45edd796
Success
Script available on my docker repository on github.