Using /proc on Linux

/proc on Linux is a pseudo file system that provides an interface to the kernel data structures. It is very useful for troubleshooting and the following highlights some that I have found helpful. The man page can be accessed with full details on Arch Linux with :

# man 5 proc

/proc/cpu

• A collection of CPU and system architecture dependent items
• Useful to find the CPU type, number of cores and flags
• lscpu uses this file to obtain its information
  

  $ cat /proc/cpu
  

/proc/cmdline

• Arguments passed to the Linux kernel at boot time
• Useful to see what options were used by e.g. grub on boot
  

  $ cat /proc/cmdline
  

/proc/loadavg

• The first three fields in this file are load average figures giving the number of jobs in the run queue (state R) or waiting for disk I/O (state D) averaged over 1, 5, and 15 minutes.
• Same numbers used by uptime
  

  $ uptime && cat /proc/loadavg
   20:37:08 up  1:36,  8 users,  load average: 0.51, 2.17, 2.56
   0.51 2.17 2.56 6/681 11570
  

• The fourth field consists of two numbers separated by a slash (/)
• The first of these is the number of currently runnable kernel scheduling entities (processes, threads)
• The value after the slash is the number of kernel scheduling entities that currently exist on the system.
• The fifth field is the PID of the process that was most recently created on the system.

/proc/meminfo

• Reports statistics about memory usage on the system
• Used by free to report the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel
  

   $ cat /proc/meminfo
  

/proc/[pid]/cmdline

• Everything in Linux is a file and each process has a directory under /proc for its /pid/
• As pid 1 is the init script we can
  

  $ cat /proc/1/cmdline
  /sbin/init
  

• This read-only file holds the complete command line for the process

/proc/[pid]/fd/

• A subdirectory containing one entry for each file which the process has open, named by its file descriptor
• Its a symbolic link to the actual file

Best Practices for Docker Container deployments

Container technology like Docker leads to faster testing and deployments of software applications. The following is a list of some best practices to consider when using containers. The key objectives are :
Repeatability
Reliability
Resiliency

To achieve this :

1. Have a single code base, tracked in git with many deploys

• Docker containers should be immutable
• Use the environment variables to change anything inside the container
• Do not build separate images for staging and production

2. Explicitly declare and isolate any dependencies

• Do not use latest in images. Instead define the exact version e.g. ubuntu:latest vs ubuntu:16.04
• Its useful to build run times (e.g. Java run time) based on specific images
• Process -> Base OS -> Run Time (e.g. Java run time) -> Add app

3. Store configuration in the environment

• Do not have a config.yml or properties.xml
• Always use environmental variables

4. Treat backing services as attached resources

• Never use local disk
• Data will always disappear
• Connect to network services using connection info from the environment e.g. DB_URL

5. Strictly separate build and run stages

• Build immutable images and then run them
• Never install anything on deployments
• Respect the life cycle : build, run, destroy

6. Execute the application as 1 or more stateless processes

• Schedule long running processes by distributing them across a cluster of physical hardware

7. Export services via port binding

• Define ports from environment variables
• A PID cannot guarantee the port in the container

8. Scale out via process model

• Horizontally scale by adding instances

9. Maximize robustness with fast start-up and graceful shutdown

• Quickly scale up when there is a load spike
• For a data intensive app its tempting to load data into memory as a hot cache. Issue is that this stops having a fast start-up (as need to load data into memory). Could use Redis or memcache as an alternative

10. Keep development, staging and production as simple as possible

• Run containers in development e.g. DB, caching

11. Treat logs as event streams

• Log to standard output (stdout) and standard error (stderr)
• Use something like the ELK stack to collect all the logs
• Should never need to ssh to check logs
• There should not be random log files in the container

12. Run admin/management tasks as one off processes

• Do not use customer containers as one off tasks
• Reuse app images with specific entry points for tasks

systemd-analyze - Analyze system boot-up performance

The systemd project has an excellent tool systemd-analyze which allows an analysis of system boot performance.

Running without any parameters displays the startup time in total with figures for the kernel + userspace

$ systemd-analyze
Startup finished in 1.190s (kernel) + 8.312s (userspace) = 9.503s

The option blame prints a list of all running units, ordered by the time they took to initialize. This may be used to optimize boot-up times. The output may be misleading as the initialization of one service might be slow simply because it waits for the initialization of another service to complete.
For example the 3 applications that took longest to initialise :

$ systemd-analyze blame | head -n 3
          7.621s man-db.service
          3.174s docker.service
          1.134s mysqld.service

The option critical-chain prints a tree of the time-critical chain of units (for each of the specified UNITs or for the default target otherwise).

The time after the unit is active or started is printed after the “@” character. The time the unit takes to start is printed after the “+” character. Note that the output might be misleading as the initialization of one service might depend on socket activation and because of the parallel execution of units
For example reviewing the docker.service unit

$ systemd-analyze critical-chain docker.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

docker.service +3.174s
`-network.target @1.287s
  `-NetworkManager.service @830ms +172ms
    `-dbus.service @691ms
      `-basic.target @682ms
        `-sockets.target @682ms
          `-docker.socket @681ms +481us
            `-sysinit.target @681ms
              `-systemd-backlight@backlight:acpi_video0.service @1.321s +41ms
                `-system-systemd\x2dbacklight.slice @1.320s
                  `-system.slice @123ms
                    `--.slice @102ms

The plot option prints an SVG graphic detailing which system services have been started at what time, highlighting the time they spent on initialization

$ systemd-analyze plot > plot.svg

How to paste source code in Vim without losing formatting

When pasting code into Vim automatic indenting can occur and your code formatting will be incorrect. To stop this issue there are two options :

Paste option

Within vim

    : set paste

Vimrc function key mapping

It can be useful to have a mapping in your ~/.vimrc file to quickly allow enable/disable of the paste feature :

" Quickly enable/disable 'paste' option
set pastetoggle=<F10>

Upgrading to GNOME 3.20 from 3.18

Every 6 months the GNOME team releases a new version of the Desktop. Running Arch Linux its great to be able to have quick access to the new features quickly. Running near to the bleeding edge always has a few quirks and here are a few I encountered.

Extensions claiming they are not supported

This is a common issue for me on every GNOME upgrade. I discovered this check can quickly be disabled as follows :

$ gsettings set org.gnome.shell disable-extension-version-validation "true"

No Topleft Hot Corner Extension stopped working

One of my ‘must have’ extensions is no topleft hot corner which stopped working. I found that removing the extension from the extensions site and then installing it from the AUR works as its a newer version :

$ packer -S gnome-shell-extension-nohotcorner-git

This installs the extension as follows :

$ ls /usr/share/gnome-shell/extensions/nohotcorner@azuri.free.fr/
extension.js  metadata.json

Excessive Padding in Applications

The css selectors changed in GNOME 3.20 which means that the padding on titlebars e.g. Firefox and terminal is large. GNOME provides a config file for users to tweak their own environment ~/.config/gtk-3.0/gtk.css. I updated mine to :

/* shrink headebars */
headerbar {
    min-height: 0;
    padding-left: 2px; /* same as childrens vertical margins for nicer proportions */
    padding-right: 2px;
    padding-top: 1px;
    padding-bottom: 1px;
}

headerbar entry,
headerbar spinbutton,
headerbar button,
headerbar separator {
    margin-top: 1px; /* same as headerbar side padding for nicer proportions */
    margin-bottom: 1px;
}

 /* shrink ssd titlebars */
.default-decoration {
    min-height: 0; /* let the entry and button drive the titlebar size */ 
    padding-left: 2px; /* same as childrens vertical margins for nicer proportions */
    padding-right: 2px;
    padding-top: 1px;
    padding-bottom: 1px;
}

.default-decoration .titlebutton {
    min-height: 5px; /* tweak these two props to reduce button size */
    min-width: 5px;
}

/* reduce the padding size of buttons */
button, entry {
  min-height: 0;
  min-width: 0;
  padding: 2px;
}

/* reduce the padding of tabs */
notebook tab {
  min-height: 0;
  padding-top: 3px;
  padding-bottom: 3px;
}

/* reduce the padding of buttons */
notebook tab button {
  min-height: 0;
  min-width: 0;
  padding: 2px;
  margin-top: 2px;
  margin-bottom: 2px;
}

notebook button {
  min-height: 0;
  min-width: 0;
  padding: 2px;
}

/* add padding for Vte-terminal */
widget {
    padding: 4px;
}

Restart GNOME shell with Alt + F2 then type ‘r’ which will restart the desktop to ally the changes.