GitHub - rosineygp/mkdkr: mkdkr = Makefile + Docker

mkdkr

mkdkr = Makefile + Docker

Super small and powerful framework for build CI pipeline, scripted with Makefile and isolated with docker.

• Dependencies: [ make, docker, bash, git ]
• Two files only (Makefile and .mkdkr), less garbage on your repo
• All power of make, docker and bash
• Shipping and switch among CI engines like Circle CI, GitHub Actions, Gitlab-ci, Jenkins, Travis.. and more using exporter
• Clean and elegant code syntax

Table of contents

• Usage
  • Makefile
  • Execute
  • Result
  • Export
  • Demonstration
    • My Workflow - Configuración automática de CI/CD
• Reason
• Functions
  • @$(dkr)
  • instance:
  • service:
  • dind:
  • run:
  • var-run:
  • login:
  • retry:
  • log:
  • push:
  • pull:
  • cd:
• Includes
  • Explicit
  • Implicit
  • mkdkr.csv
  • Collection
• Builtin Targets
  • _list
• Helpers
• Examples
  • Simple
  • Service
  • DIND
  • Escapes
  • Shell
  • Stdout
  • Pipelines
• Environment Variables
• Migration

Usage

Makefile

Create a file with name Makefile and paste the following content.

Download .mkdkr dynamically.

# Required header
include $(shell [ ! -f .mkdkr ] && curl -fsSL https://git.io/JOBYz > .mkdkr; bash .mkdkr init)

# without shorten url
# include $(shell [ ! -f .mkdkr ] && curl -fsSL https://github.com/rosineygp/mkdkr/releases/latest/download/mkdkr > .mkdkr; bash .mkdkr init)

job:
	@$(dkr)
	instance: alpine
	run: echo "hello mkdkr dynamic!"

OR keep .mkdkr locally.

# Download .mkdkr
curl -fsSL https://github.com/rosineygp/mkdkr/releases/latest/download/mkdkr > .mkdkr

# Required header
include $(shell bash .mkdkr init)

job:
	@$(dkr)
	instance: alpine
	run: echo "hello mkdkr local!"

.gitignore (optional)

.tmp
.mkdkr # only in dynamic config

Execute

Result

start: job


instance: alpine
20498831fe05f5d33852313a55be42efd88b1fb38b463c686dbb0f2a735df45c

run: echo hello mkdkr!
hello mkdkr!

cleanup:
20498831fe05

completed:
0m0.007s 0m0.000s
0m0.228s 0m0.179s

Export

Run your current Makefile in another engine, like travis or github actions, use the dynamic include exporter.

Demonstration

My Workflow - Configuración automática de CI/CD

Author: Martin Algañaraz

Reason

Build pipeline for a dedicated platform can take a lot of time to learn and test, with mkdkr you can test all things locally and run it in any pipeline engine like Jenkins, Actions, Gitlab-ci and others.

Functions

@$(dkr)

Load docker layer for mkdkr, use inside a target of Makefile.

shell-only:
	echo "my local shell job"

mkdkr-job:
	@$(dkr)            # load all deps of mkdkr
	intance: alpine
	run: echo "my mkdkr job"

instance:

Create docker containers, without special privileges.

my-instance:
	@$(dkr)
	instance: ubuntu:20.04     # create a instance

Parameters:

• String, DOCKER_IMAGE *: any docker image name
• String|Array, ARGS: additional docker init args like (--cpus 1 --memory 64MB)

Return:

• String, Container Id

Calling instance: twice, it will replace the last container.

service:

Create a docker container in detached mode. Useful to bring up a required service for a job, like a webserver or database.

my-service:
	@$(dkr)
	service: nginx    # up a nginx
	instance: alpine

Is possible start more than one service.

multi-service:
	@$(dkr)
	service: mysql
	service: redis
	instance: node:12
	run: npm install
	run: npm test

* Instance and services are connected in same network
** The name of service is the same of image name with safe values

replace role 's/:|\.|\/|-/_/g'

Parameters:

• String, DOCKER_IMAGE *: any docker image name
• String|Array, ARGS: additional docker init args like (--cpus 1 --memory 64MB)

Return:

• String, Container Id

instance or dind created after a service, will be automatically linked.

dind:

Create a docker instance with daemon access. Useful to build docker images.

my-dind:
	@$(dkr)
	dind: docker:19
	run: docker build -t my/dind .

Parameters:

• String, DOCKER_IMAGE *: any docker image name
• String|Array, ARGS: additional docker init args like (--cpus 1 --memory 64MB)

Return:

• String, Container Id

run:

Execute a command inside docker container [instance: or dind:] (the last one).

Is not possible to execute commands in a service.

Parameters:

• String|Array, command: any sh command eg. 'apk add nodejs'

Return:

• String, Command(s) output

Usage

my-run:
	@$(dkr)
	instance: alpine
	# run a command inside container
	run: apk add curl

	instance: debian
	# avoid escape to host bash, escapes also can be used (eg. \&\&)
	run: 'apt-get update && \
			apt-get install -y curl'

	# run a command inside container and redirect output to host
	run: ls -la > myfile

	# run a command inside container and redirect output to container
	run: 'ls -la > myfile'

var-run:

Execute a command inside docker container [instance: or dind:] and return stdout at named var.

After created var it is passed to next run: or var-run: execution.

Parameters:

• String, var: any bash valid variable name
• String|Array, command: any sh command eg. 'apk add nodejs'

Return:

• String, Command(s) output

Usage

my-var-run:
	@$(dkr)
	instance: alpine
	# run a command inside container
	var-run: myvar hostname
	run: echo '$$myvar'
	var-run: mynewvar echo '$$myvar'
	run: echo "$$myvar $$mynewvar"

login:

Execute docker login in a private registry. If docker instance exist, execute login inside container otherwise at host.

Parameters:

• String, domain: private registry domain.
• String| user: registry username.
• String| password: registry password.

Return:

• None.

Usage

private-registry:
	@$(dkr)
	login: my.private.registry foo $(MKDKR_PASSWORD)
	instance: my.private.registry/alpine

Execute login at host and download image from private registry (login before instance creation)

private-registry:
	@$(dkr)
	dind: docker:19
	login: my.private.registry foo $(MKDKR_PASSWORD)
	instance:  my.private.registry/alpine

Execute login inside instance and download image from private registry (login after instance creation)

retry:

Execute a command inside docker container [instance: or dind:] (the last one), with retry options.

Is not possible to execute commands in a service.

Parameters:

• Number|Int, attempts: number of attempts before crash.
• Number|Int, sleep: time sleeping before retry.
• String|Array, command: any sh command eg. 'curl https://tomcat:8080'

Return:

• String, Command(s) output

Usage

deploy:
	@$(dkr)
	instance: oc
	retry: 3 10 oc apply -f build.yml
	#the job can run 3 times with a delay of ten seconds

npm:
	instance: alpine
	run: apk add curl
	service: my-slow-service
	retry: 60 1 curl http://my-slow-service:8080

log:

All output steps executed in a job (except log:) is stored and can be reused during future steps.

Parameters:

• Number: The number represent the step in a job and start with 0.

Return:

• Text, Multiline text.

Usage

my-log:
	@$(dkr)
	instance: alpine
	run: apk add curl jq
	run: curl http://example.com
	log: 1 \| jq '.'

push:

Push files/folders to a container job from local filesystem.

Parameters:

• String, from: Target files/folders in local filesystem.
• String, to: Destiny of files/folders inside container.

Return

• None.

Usage

push:
	@$(dkr)
	instance: ansible
	push: /etc/ansible/inventory/hosts.yml
	run: ansible-playbook main.yml

Example

pull:

Pull files/folders from a container job to local filesystem.

Parameters:

• String, from: Target files/folders inside container.
• String, to: Destiny of files/folders in local filesystem.

Return

• None.

Usage

pull:
	@$(dkr)
	instance: debian
	run: curl https://example.com -o /tmp/out.html
	pull: /tmp/out.html .

Example

cd:

Move folder context.

Parameters:

• String, workdir: Set workdir.

Return

• None.

change-folder:
	@$(dkr)
	instance: debian
	cd: /tmp
	run: pwd
	# /tmp

Includes

Is possible create jobs or fragments of jobs and reuse it in another projects, like a code package library.

There are two major behavior of includes:

Explicit

A fragment of job (eg. define) and needs to be called explicitly to work.

TAG=latest

define docker_build =
	@$(dkr)
	dind: docker:19
	run: docker build -t $(REGISTRY)/$(PROJECT)/$(REPOS):$(TAG) .
endef

All definitions will be load at start of makefile, after it is possible to call at your custom job.

my-custom-build:
	$(docker-build)

Implicit

Just a full job in another project.

TAG=latest

docker_build:
	@$(dkr)
	dind: docker:19
	run: docker build -t $(REGISTRY)/$(PROJECT)/$(REPOS):$(TAG) .

The jobs will be load at start and can be called directly.

• No needs to implement the job at main Makefile.
• Very useful for similar projects.

mkdkr.csv

A file with name mkdkr.csv, that contains the list of remote includes.

Needs to be at same place o main Makefile.

commitlint,https://github.com/rosineygp/mkdkr_commitlint.git,master,main.mk
docker,https://github.com/rosineygp/mkdkr_docker.git

The file contains four values per line in following order

* required

Collection

Small collection, use it as example

Builtin Targets

_list

List all targets in Makefile, include extensions.

$ make _list

include

alias: exporter, repos: https://github.com/rosineygp/mkdkr_exporter.git, checkout: v1.5.0, file: main.mk

replace: MKDKR_EXPORTER_TAG=latest to v1.5.0
bash.v4-0:
bash.v4-1:
bash.v4-2:
bash.v4-3:
bash.v4-4:
bash.v5-0:
_coverage.report:
examples.dind:
examples.escapes:
examples.pipeline:
examples.retry:
examples.service:
examples.shell:
examples.simple:
examples.stdout:
_exporter_bitbucket-pipelines:
_exporter_circle-ci:
_exporter_github:
_exporter_gitlab-ci:
_exporter_jenkins_pipeline:
_exporter_shell:
_exporter_travis:
lint.commit:
lint.hadolint:
lint.shellcheck:
test.unit:

The result are sorted by name.

First char target name: [a-zA-Z_]

Helpers

A set of small functions to common pipelines process.

autocommit:
	@$(dkr)
	instance: alpine/git
	git-ssh: ~/.ssh/id_rsa github.com auto@mkdkr.com
	run: git clone git@github.com:rosineygp/mkdkr.git /auto
	run: echo "my new file" \> /auto/my-new-file.txt
	run: git -C /auto add my-new-file.txt
	run: git -C /auto commit -m "my automatic change"
	run: git -C /auto push origin master:feat/auto-push

Examples

Simple

simple:
	@$(dkr)
	instance: alpine
	run: echo "hello mkdkr!"

Is possible to mix images during job, see in example

Makefile

Service

service:
	@$(dkr)
	service: nginx
	instance: alpine
	run: apk add curl
	run: curl -s nginx

Makefile

DIND

Privileged job

dind:
	@$(dkr)
	dind: docker:19
	run: docker build -t project/repos .

Makefile

Escapes

pipes:
	@$(dkr)
	instance: ubuntu:18.04
	run: "find . -iname '*.mk' -type f -exec cat {} \; | grep -c escapes"

More examples at file

Makefile

Shell

Switch to another shell

shell:
	@$(dkr)
	instance: ubuntu
	export MKDKR_SHELL=bash
	run: 'echo $$0'

More examples at file

Makefile

Stdout

Get output by id

Use to filter or apply some logic in last command executed

stdout:
	@$(dkr)
	instance: alpine
	run: echo "hello mkdkr!"
	run: ps -ef
	log: 1

log: 1 return stout form second command ps -ef

stdout:
	@$(dkr)
	instance: debian
	run: apt-get update
	run: apt-get install curl -y
	run: dpkg -l
	log: 2 | grep -i curl && echo "INSTALLED"

log: 2 return stdout from third command dpkg -l and apply filter

Makefile

Pipelines

Group of jobs for parallel and organization execution

pipeline:
	make test -j 3	# parallel execution
	make build
	make pack
	make deploy

Makefile

Environment Variables

• to overwrite the values use: export <var>=<value>
• * auto generated

Migration

Migration from release-0.26, just execute the following script on your terminal at root of your project.

curl https://raw.githubusercontent.com/rosineygp/mkdkr/master/.mkdkr > .mkdkr

mkdkr_migration() {
  sed -i 's/\.\.\.\ job\ /instance:\ /g;s/\.\.\.\ service\ /service:\ /g;s/\.\.\.\ privileged\ /dind:\ /g;s/\.\.\.\ /instance:\ /g;s/\.\.\ /run:\ /g;s/@\$(\.)/@\$(dkr)/g' ${1}
}

export -f mkdkr_migration

mkdkr_migration Makefile

find . -iname *.mk -exec bash -c 'mkdkr_migration "$0"' {} \;