path: root/README.md

---
title: Developing with Docker
author: gitlab.com/xlgmokha/developing-with-docker
date: 2020-10-24
---

```text

      -----------------
     (                 )
    ( A whale of a tale )
     (                 )
      -----------------
             \
              \
               \              ##         .
                \       ## ## ##        ==
                 \  ## ## ## ## ##    ===
                /"""""""""""""""""\___/ ===
               {                       /  ===-
                \______ O           __/
                  \    \         __/
                    \____\_______/


Mo Khan | Software Developer
```

# $ history

Mo Khan | Software Developer | Calgary, AB, Canada.

```bash
 7  GitLab       --type=dev-tools
 6  Cisco        --type=security-product
 5  Uppercut     --type=agency
 4  ARC          --type=information-systems
 3  eCompliance  --type=startup
 2  ThoughtWorks --type=consulting
 1  MediaLogic   --type=agency
 0  DataShapers  --type=startup
```

# License scanning at GitLab

```plaintext
       ***********************
       < Show me the licenses >
       ***********************
              \
               \
                \
                              ##        .
                        ## ## ##       ==
                    ## ## ## ##      ===
                /""""""""""""""""___/ ===
           ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                \______ o          __/
                  \    \        __/
                    \____\______/

```

| Name  | Version | Package Manager | License |
| ----- | ----- | --- | --- |
| @rails/actioncable | 6.0.3-3 | Yarn | MIT |
| @rails/ujs | 6.0.3-2 | Yarn | MIT |
| @yarnpkg/lockfile | 1.1.0 | Yarn | BSD-2-Clause |
| addressable | 2.7.0 | Bundler | Apache-2.0 |
| bcrypt | 3.1.12 | Bundler | MIT |
| cbor | 0.5.9.6 | Bundler | Apache-2.0 |
| device_detector | 1.0.0 | Bundler | LGPL-3.0 |
| devise | 4.7.3 | Bundler | MIT |
| diffy | 3.3.0 | Bundler | MIT |
| docutils | 0.13.1 | Pipenv | BSD-2-Clause |
| elasticsearch | 6.8.2 | Bundler | Apache-2.0 |
| eventmachine | 1.2.7 | Bundler | Ruby AND GPL-2.0 |
| ffi | 1.13.1 | Bundler | BSD-3-Clause |
| ffi-compiler | 1.0.1 | Bundler | Apache-2.0 |
| jmespath | 1.4.0 | Bundler | Apache-2.0 |
| kgio | 2.11.3 | Bundler | LGPL-2.1+ |
| launchy | 2.4.3 | Bundler | ISC |
| msgpack | 1.3.3 | Bundler | Apache-2.0 |
| nokogumbo | 2.0.2 | Bundler | Apache-2.0 |
| rails | 6.0.3.3 | Bundler | MIT |
| vue | 2.6.12 | Yarn | MIT |

# Constraints

* supports multiple versions of:
  * Dotnet Core
  * Golang
  * Java
  * NodeJS
  * PHP
  * Python
  * Ruby
* supports multiple package managers:
  * Bundler
  * pip
  * Pipenv
  * Gradle
  * Maven
* includes system packages for common libraries:
  * libpq-dev
  * libsqlite3-dev
* works in limited connectivity environments

```plaintext
       **************************************
       < Must be deployed as a Docker image >
       **************************************
            \
             \
              \
                            ##        .
                      ## ## ##       ==
                  ## ## ## ##      ===
              /""""""""""""""""___/ ===
         ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
              \______ o          __/
                \    \        __/
                  \____\______/

```


# Constraints - 🤔

We need to build a Docker image that can:

* scan any codebase
* install the version of tools needed
* install the project dependencies
* be small enough to fit on a CD-ROM

```plaintext
                      ---------------
                    < Are you joking? >
                      ---------------

          .---.
         /o   o\
      __(=  "  =)__
       //\'-=-'/\\
          )   (_
         /      `"=-._
        /       \     ``"=.
       /  /   \  \         `=..--.
   ___/  /     \  \___      _,  , `\
  `-----' `""""`'-----``"""`  \  \_/
```

# How does this work?

```text
  -------------
  |    git    |
  -------------     ----------------
  | main*     | --> | gitlab-runner |
  | feature-a |     ----------------
  | feature-b |            |
  -------------     launch container
                           |
                           V
                    ---------------
                    |             | <-----------------
                    | Docker Host | ---------        |
                    |             |         |        |
                    ---------------         |        |
                           |             download    |
                           V                |        |
                    ⭐⭐⭐⭐⭐⭐⭐⭐        V        |
                    ⭐  License   ⭐   ------------  |
                    ⭐  Scanner   ⭐   | registry | -|
                    ⭐⭐⭐⭐⭐⭐⭐⭐   ------------
                           |
                     publish report
                           |
                           V
                    ----------------
                    | gitlab-rails |
                    ----------------
```

# v1.0

1. Scan project for lock files (Gemfile.lock, Pipfile.lock etc)
2. Install project tools (Ruby 2.7.2, Python 3.8.4)
3. Install project dependencies (Rails, Django)
4. Scan for licenses
5. Export JSON report

```plaintext
                      ************
                      | APPLAUSE |
                      ************
```

| Name  | Version | Package Manager | License |
| ----- | ----- | --- | --- |
| @rails/actioncable | 6.0.3-3 | Yarn | MIT |
| @rails/ujs | 6.0.3-2 | Yarn | MIT |
| @yarnpkg/lockfile | 1.1.0 | Yarn | BSD-2-Clause |
| addressable | 2.7.0 | Bundler | Apache-2.0 |
| bcrypt | 3.1.12 | Bundler | MIT |
| cbor | 0.5.9.6 | Bundler | Apache-2.0 |
| device_detector | 1.0.0 | Bundler | LGPL-3.0 |

```plaintext
                ---------------------------
              < Yay! We can detect licenses >
                ---------------------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
```

# Why so slow?

```plaintext
 _____________________________________
< That's cool, but why is it so slow? >
 -------------------------------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
```

# Overview

```text
      ---------------
      |             | <-----------------
      | Docker Host | ---------        |
      |             |         |        |
      ---------------         |        |
             |             download    |
             V                |        |
      ⭐⭐⭐⭐⭐⭐⭐⭐        V        |
      ⭐  License   ⭐   ------------  |
      ⭐  Scanner   ⭐   | registry | -|
      ⭐⭐⭐⭐⭐⭐⭐⭐   ------------
             |
        publish report
             |
             V
      ----------------
      | gitlab-rails |
      ----------------
```

# What happens if the Docker image is too large?

* slow downloads
* more disk space is required
* more bandwidth is consumed

```plaintext
                      ---------------
                    < That's not good >
                      ---------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
```


# Overview

```text
      ---------------
      |             | <------ 10GB -----
      | Docker Host | ---------        |
      |             |         |        |
      ---------------         |        |
             |             download    |
             V                |        |
      ⭐⭐⭐⭐⭐⭐⭐⭐        V        |
      ⭐  License   ⭐   ------------  |
      ⭐  Scanner   ⭐   | registry | -|
      ⭐⭐⭐⭐⭐⭐⭐⭐   ------------
```

```plaintext
            --------------------------------------
          < Yikes! It takes 6 minutes to download >
            --------------------------------------

        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
```


# Docker 101

* Definitions
* Build
* Analyze
* Optimize

```text
        < What are we going to talk about? >
        ----------------------------------
            \
             \
              \
                            ##        .
                      ## ## ##       ==
                  ## ## ## ##      ===
              /""""""""""""""""___/ ===
          ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
              \______ o          __/
                \    \        __/
                  \____\______/
```

# Definitions

| Ruby | Docker |
| -- | -- |
| Class | Image |
| Object | Container |

```ruby
class Person
  attr_reader :name

  def initialize(name)
    @name = name
  end

  def hug(other)
    puts "#{name} 🤗 #{other.name}"
  end
end

mo = Person.new("mo")
me = Person.new("me")

mo.hugs(me)
```

# Definitions - Registry

Registry: stores images and makes them available to others

This include metadata about images and blobs for each layer in the image.

For example:

* https://registry-1.docker.io
* https://registry.gitlab.com

# Architecture

```text
                      ----------
                      | Client |
                      ----------
                      | build  |
                      | pull   |
                      | run    |
                      ----------
                          | (tcp/unix socket)
                          V
                    ---------------
                    | Docker Host |
                    ---------------
                    | Daemon     |
                    | Containers |
                    | Images     |
                    --------------
                          |  A
                          V  |
                     ------------
                     | Registry |
                     ------------
                     | Images   |
                     ------------
```

https://docs.docker.com/get-started/overview/#docker-architecture

# $ docker run -it alpine:latest cat /etc/os-release

```terminal8
docker run -it alpine:latest cat /etc/os-release
```

1. check if "alpine:latest" is on docker host
1. download "alpine:latest" from registry to docker host
1. start a container using the "alpine:latest" image

# Dockerfile

Defines how to build a Docker image.

```file
path: examples/001/Dockerfile
relative: true
lang: docker
```

https://docs.docker.com/engine/reference/builder/

# docker build -t developing-with-docker:latest examples/001/

```terminal32
docker build --network=host -t developing-with-docker:latest examples/001/
```

# docker run developing-with-docker:latest

```terminal32
docker run developing-with-docker:latest
```

# Analysis

A docker image is made up of multiple layers.
Each layer is a snapshot of the filesystem
stored as an archive.

```plaintext
 -----------------      |      -----------------
 (    bread      )      |      | FROM          |
 -----------------      |      -----------------
 < brie cheese   >      |      | RUN           |
 { cranberries   }      |      | COPY          |
 [ sliced turkey ]      |      | RUN           |
 -----------------      |      -----------------
 (    bread      )      |      | ENTRYPOINT    |
 -----------------      |      -----------------
```

```plaintext
                  -------------------
                < Yum. Let's dive in! >
                  -------------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
```

# Analysis - dive

```file
path: examples/001/Dockerfile
relative: true
lang: docker
```

```bash
$ dive developing-with-docker:latest

│ Layers ├─────────────────────────────────────────────────────────
Size    Command
5.6 MB   FROM 5a96ef02e9cab83
  16 MB  apk add ruby
  40 B   #(nop) COPY file:253d38e67af26b201caf4e271248576ba6a7da
  40 B   chmod +x /usr/local/bin/hello
```

https://github.com/wagoodman/dive

# dive - layer details

```bash
┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Permission     UID:GID       Size  Filetree
drwxr-xr-x         0:0     841 kB  ├─⊕ bin
drwxr-xr-x         0:0        0 B  ├── dev
drwxr-xr-x         0:0     383 kB  ├─⊕ etc
drwxr-xr-x         0:0        0 B  ├── home
drwxr-xr-x         0:0     3.9 MB  ├─⊕ lib
drwxr-xr-x         0:0        0 B  ├─⊕ media
drwxr-xr-x         0:0        0 B  ├── mnt
drwxr-xr-x         0:0        0 B  ├── opt
dr-xr-xr-x         0:0        0 B  ├── proc
drwx------         0:0        0 B  ├── root
drwxr-xr-x         0:0        0 B  ├── run
drwxr-xr-x         0:0     226 kB  ├─⊕ sbin
drwxr-xr-x         0:0        0 B  ├── srv
drwxr-xr-x         0:0        0 B  ├── sys
drwxrwxrwx         0:0        0 B  ├── tmp
drwxr-xr-x         0:0      14 MB  ├─⊕ usr
drwxr-xr-x         0:0     1.8 MB  └─⊕ var
```

# License scanner - v1.0

```plaintext
 __________________________________
/ How many layers does the license \
\ scanner image have?              /
 ----------------------------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
```

# docker pull - v1.0

```bash
0a01a72a686c: Pull complete
cc899a5544da: Pull complete
19197c550755: Pull complete
716d454e56b6: Pull complete
4a00dd5b28fc: Pull complete
e039d25729bf: Pull complete
930b12354a74: Pull complete
5c0b45be82b1: Pull complete
2174b0b17785: Pull complete
79e1fdae2dfc: Extracting [================>       ]    322MB/396.1MB
2c46852653ff: Download complete
40e535af8764: Download complete
0c1954047133: Download complete
080be37ae17e: Download complete
4179fc4ef96a: Download complete
87d7bc66884f: Download complete
581b5a43e64d: Download complete
a754f2766934: Download complete
c1e41c3670fe: Download complete
e782eb3070d9: Download complete
50d1fb4b55b2: Download complete
276696690bcf: Download complete
5eec42d5363b: Download complete
2296aa2193e9: Download complete
5fe4c102c0bc: Download complete
97390612da81: Downloading [=====>                ]  81.05MB/174MB
311b1e270e29: Downloading [===========>          ]  42.12MB/189.4MB
53dfbd975f60: Downloading [=>                    ]  25.36MB/843.4MB
3dd2acdebe0f: Waiting
d548f098494f: Waiting
da1cc42017ff: Waiting
cfc3cd025ca9: Waiting
69ea647e6c07: Waiting
1e27d5f85aa2: Waiting
94cf5e06627d: Waiting
30e1f788589d: Waiting
d9238ec317d1: Waiting
e17797fa5e82: Waiting
9003b36c1e4e: Waiting
```

# Layer cake

```plaintext
...
50d1fb4b55b2: Download complete
276696690bcf: Download complete
5eec42d5363b: Download complete
2296aa2193e9: Download complete
5fe4c102c0bc: Download complete
97390612da81: Downloading [=====>                ]  81.05MB/174MB
311b1e270e29: Downloading [===========>          ]  42.12MB/189.4MB
53dfbd975f60: Downloading [=>                    ]  25.36MB/843.4MB
3dd2acdebe0f: Waiting
d548f098494f: Waiting
da1cc42017ff: Waiting
cfc3cd025ca9: Waiting
69ea647e6c07: Waiting
1e27d5f85aa2: Waiting
94cf5e06627d: Waiting
30e1f788589d: Waiting
d9238ec317d1: Waiting
e17797fa5e82: Waiting
9003b36c1e4e: Waiting
                _______________________
                < That's a lot of layers! >
                -----------------------
                    \
                    \
                      \
                                    ##        .
                              ## ## ##       ==
                          ## ## ## ##      ===
                      /""""""""""""""""___/ ===
                 ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                      \______ o          __/
                        \    \        __/
                          \____\______/

```

# Start with a minimal base image

```bash
REPOSITORY                   TAG                 SIZE
debian                       stable-slim         69.2MB
licensefinder/license_finder 6.0.0               6.21GB

```

```plaintext
                  ------------------------------
                < Let's try a smaller base image >
                  ------------------------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
                             `-`
```

```Dockerfile
FROM licensefinder/license_finder:6.0.0
```

to

```Dockerfile
FROM debian:stable-slim
```

# Be picky

*Warning: The content below may be considered offensive.*

```Dockerfile
RUN apt-get update && apt-get install -y \
  build-essential \
  curl \
  git-core \
  sudo \
  unzip \
  wget
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
    apt-get -y install nodejs
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - && \
  echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list && \
  apt-get update && \
  apt-get install yarn
RUN apt-get install -y python rebar
RUN apt-get install -y python-pip && \
    pip install --upgrade pip==$PIP_INSTALL_VERSION
RUN apt-get install -y locales
RUN wget https://packages.erlang-solutions.com/erlang-solutions_${MIX_VERSION}_all.deb && \
    sudo dpkg -i erlang-solutions_${MIX_VERSION}_all.deb && \
    sudo rm -f erlang-solutions_${MIX_VERSION}_all.deb && \
    sudo apt-get update && \
    sudo apt-get install -y esl-erlang && \
    sudo apt-get install -y elixir
RUN apt-get install -y python-dev && \
  pip install --ignore-installed six --ignore-installed colorama --ignore-installed requests --ignore-installed chardet --ignore-installed urllib3 --upgrade setuptools && \
  pip install conan
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF &&\
  echo "deb https://download.mono-project.com/repo/ubuntu stable-xenial main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list &&\
  apt-get update &&\
  apt-get install -y mono-complete &&\
  curl -o /usr/local/bin/nuget.exe https://dist.nuget.org/win-x86-commandline/latest/nuget.exe &&\
  echo "alias nuget=\"mono /usr/local/bin/nuget.exe\"" >> ~/.bash_aliases
RUN wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb &&\
  sudo dpkg -i packages-microsoft-prod.deb &&\
  sudo apt-get update &&\
  sudo apt-get install -y dotnet-runtime-2.1
```

# Group 'apt-get install' together

```Dockerfile
COPY config/install.sh /opt/install.sh
RUN bash /opt/install.sh
```

```bash
#!/bin/bash
apt-get install -y --no-install-recommends \
  apt-transport-https \
  autoconf \
  automake \
  bsdmainutils \
  bzip2 \
  ca-certificates \
  cmake \
  curl \
  gnupg2 \
  make \
  pkg-config \
  re2c \
  rebar \
  zstd
```

*note: avoid installing build tools if you can*

# Deflate

Compress files that are large during build time.

```bash
#!/bin/bash

function deflate() {
  local file=$1
  local dir=$2
  local zstd_command="/usr/bin/zstd -19 -T0"
  tar --use-compress-program "$zstd_command" -cf "$file" "$dir"
}

cd /opt
deflate /opt/asdf.tar.zst asdf

cd /usr/lib
deflate /usr/lib/gcc.tar.zst gcc
deflate /usr/lib/mono.tar.zst mono
deflate /usr/lib/rustlib.tar.zst rustlib

cd /usr/share
deflate /usr/share/dotnet.tar.zst dotnet
```

# Inflate

Decompress files when the container is launched
by hooking into the ENTRYPOINT.

```Dockerfile
ENTRYPOINT ["/run.sh"]
```

```bash
#!/bin/bash

function inflate() {
  local file=$1
  local to_dir=$2
  if [ -f "$file" ]; then
    tar --use-compress-program zstd -xf "$file" -C "$to_dir"
    rm "$file"
  fi
}

inflate /opt/asdf.tar.zst /opt
inflate /usr/lib/gcc.tar.zst /usr/lib
inflate /usr/lib/mono.tar.zst /usr/lib
inflate /usr/lib/rustlib.tar.zst /usr/lib
inflate /usr/share/dotnet.tar.zst /usr/share

sh "$@"
```

# Build specialized packages

```ruby
build do
  env = with_standard_compiler_flags(with_embedded_path)
  configure_command = [
    "--disable-debug-env",
    "--disable-dtrace",
    "--disable-install-capi",
    "--disable-install-doc",
    "--disable-install-rdoc",
    "--disable-jit-support",
    "--enable-shared",
    "--prefix=#{install_dir}",
    "--with-out-ext=coverage,dbm,readline,rdoc,win32,win32ole,...",
    "--without-gdbm",
    "--without-gmp",
    "--without-jemalloc",
    "--without-tk",
    "--without-valgrind"
  ]
  configure(*configure_command, env: env)
  make "-j #{workers}", env: env
  make "-j #{workers} install", env: env
end
```

```bash
# ls -lh /opt/toolcache/ruby-* | awk '{ print $5 " " $9 }'
5.3M /opt/toolcache/ruby-2.4.10-1_amd64.deb
5.3M /opt/toolcache/ruby-2.4.5-1_amd64.deb
5.3M /opt/toolcache/ruby-2.4.9-1_amd64.deb
5.4M /opt/toolcache/ruby-2.5.8-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.0-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.1-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.2-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.3-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.4-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.5-1_amd64.deb
5.6M /opt/toolcache/ruby-2.6.6-1_amd64.deb
5.7M /opt/toolcache/ruby-2.7.0-1_amd64.deb
5.7M /opt/toolcache/ruby-2.7.1-1_amd64.deb
5.7M /opt/toolcache/ruby-2.7.2-1_amd64.deb
```

# Results

| Image  | Tag | Size |
| ----- | ----- | --- |
| license-scanner | 3.28.1 | 1.4GB  |
| license-scanner | 2.8.0 | **9.83GB** |
| license-scanner | 1.5.0 | 4.06GB |

```plaintext
                  ------------
                < 9.83GB to 1.4GB. Better! >
                  ------------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
```

# Summary

* Keep each layer small
* Grouping install steps together
* Cleanup transient artifacts in each layer
* Deflate files at building time
* Inflate files at run time

```plaintext
                  ----------
                < Thank you >
                  ----------
        .---.
       /o   o\
    __(=  "  =)__
     //\'-=-'/\\
        )   (_
       /      `"=-._
      /       \     ``"=.
     /  /   \  \         `=..--.
 ___/  /     \  \___      _,  , `\
`-----' `""""`'-----``"""`  \  \_/
```

[gitlab.com/xlgmokha/developing-with-docker](https://gitlab.com/xlgmokha/developing-with-docker)