Marks Weblog

If any of you have been paying attention I have lately been looking into ansible.

First a disclaimer, I use ‘puppetserver’ and puppet agents as since they moved away from Ruby to their own scripting language which is pretty much english it is incredibly easy to configure. If / ifelse / else syntax means I can have a simple config for say ‘desktop’ that configures a desktop selecting appropiate packages for Ubuntu/Debian/Fedora/CentOS/Rocky and for specific versions (ie: CentOS8 is missing packages that were in centOS7, debian uses completely different package names etc.) And puppet has a fantastic templating feature that maybe one day in the future ansible will be able to match.

Ansible with the playbook parsing json responses from servers can have the playbook configured to run seperate tasks depending on the results returned in previous steps but yaml files are not plain english and it doesn’t really support targets such as ‘configure desktop reguardless of OS’ and at the moment you are better off having a seperate set of playbooks per-OS type… or more simply it is not as readable or manageble yet.

Ansible is also primarily targeted at (has builtin/sipplied modules for) Linux servers although it supports windoze as well; the main issue with ansible is that most of the modules are written in python(3). In the Linux world that is not really an issue as it is almost impossible to have a Linux machine without python. On RHEL based systems it is so tightly integrated it is impossible to remove (it’s needed by systemd, firewall, dnf etc.); fortunately even though Debian(11) installs it by default it is possible to remove it on that OS so I was able to test the examples here on a machine without python installed; although of course the tests were against Linux machines.

The advantage of ansible is that there are many operating systems that support some variant of ssh/scp/sftp even if they do not support python (yes people, there are operating systems that do not and probably never will have python ported to them; some I work on) and ansible allows modules to be written in any language so modules can in theory be written in any scripting language a target machine supports.

The basic function of an ansible playbook ‘task’ seems to be to use sftp to copy the module for a task from it’s local repository or filesystem to the target machine, also sftp a file containing all the data parameters in the playbook to the target machine, and run the script with the datafile of parameters as argument one for the module(script); at the end of the task it deletes the two files it copied to the target.

What I am not sure of is how it triggers the module to run, on *nix it probably uses ‘sh’, windoze powershell; but can it be configured to run other interpreters such as rexx/clist/tacl(or gtacl) etc. If it can then it can poke its gaping wide security hole into and manage every machine that exists in theory.

By security hole I of course just mean that it needs god-like access from ssh keys alone (you do not want 2FA for every step in a playbook) and despite decades of advice on keeping private ssh keys secure inadvertently published ones still keep popping up on forums that intend you no good; and ‘sudo’ does not care about where you connected from so anybody with that private key and access to you network is god on your infrastructure; of course you could be paranoid and have a seperate key pair per server but it you have hundreds of them a maintenance nightmare.

Anyway, my interest in ansible is primarily in if it can easily manage machines that will never have python installed on them. It is also fair to say that if I am to write anything extremely complicated and possibly destructive I would prefer to do it in bash rather than in a language like python I am not really familiar with yet.

As such I have no need of python modules and need to try to avoid letting any python modules be deployed on machines I am testing against. As noted above I managed to remove python from a Debian11 server so am able to test against that.

As all my lab servers are currently *nix servers I don’t really have the oportunity to test against all the different systems I would like to see if ansible works on them (although I might try and get netrexx or oorexx as targets on a few Linux servers).

This post is about how to write ansible modules using the bash shell rather than python.

It is also worth noting where to place your custom modules so they can be located by ansible. If on a Linux machine using ‘ansible-playbook’ from the command line for testing it is easiest to simply create a directory called ‘library’ under the directory your playbook is in, for testing that would simply be your working directory and they will be found in there. You can also use the environmanet variable ANSIBLE_LIBRARY to specify a list of locations if you want a module available to all your playbooks. Note: there are many other ansible environment variables that can be useful, refer to the ansible docs.

You should also note that the ‘ansible-doc’ command only works against python modules, and while it is recomended that you write the documentation in a Python file adjacent to the module file… don’t do that or the ansible-playbook command will try and run the .py file instead of the .sh file. Just ensure it is well documented in your site specific documentation.

While the example module shown in this post may seem a little complicated one thing you must note is that by default an ansible task will run a ‘gather_facts’ step on the target machine to populate variables such as OS family, OS version, hostname, and basically everything about the target machine. That is done by a python module so is not possible on target machines without python, so the example here sets ‘gather_facts: no’ and obtains whet it needs in the module itself as well as returning the information to the playbook for use.

It’s also a little more complicated than it needs to be in that I was curious as to how variables from the playbook would be passed if indented in the playbook under another value; they are passed as jsom if imbedded, for example

Task entries
  dest: /some/dir
  state: present

Results in the file passed as arg1 to the module on the target machine containing
  dest="/some/dir"
  state="present"

Task entries
  parms:
     dest: /some/dir
     state: present

Results in the file passed as arg1 to the module on the target machine containing
  parms="{'dest': '/tmp/hello', 'state': 'present'}"

Knowing in what format the data values are going to be provided to your script is a rather important thing to know :-). In a bash or sh script you can set the variables simply using “source $1” but you do need to know the values will be in different formats depending on indent level. My example script here will handle both of those above examples but not any further levels of indentation. There will be command line json parsers that could help on Linux but remembering I’m curious about non-linux servers I need the scripts to do everything themselves.

For my hosts file I included a ‘rocky8’, ‘debian11’, and ‘fedora32’ server. The bash example returned the expected results from all of them.

It should also be noted that the test against server name in the example playbook step always returns ‘changed’ as the ‘dummy_file.yml’ uses ‘local_action’ which seems to always return changed as true reguardless of what we stuff in stdout so the command function must return its own data fields. Where only the bash module is used we control that value.

But enough rambling. ‘cd’ to a working directory, ‘mkdir library’.

Create a file my_example_module.yml and paste in the below.

# Demo playbook to run my bash ansible module passing
# parms in 'dest' and 'state' in both ways permitted.
# 
# Note: the last example also shows how to take action based on the returned result,
# in this case pull in a yml file containing a list of additional tasks for a specific
# hostname (in my test env the ansible 'hosts' file for the test had ip-addrs only
# so the hostname test is against the returned values).
# While ansible has builtins for hostname and OS info tests such as
#    when: ansible_facts['os_family'] == "RedHat" and ansible_facts['lsb']['major_release'] | int >= 6
# that is only useful for target hosts that have python installed and you use the default 'gather_facts: yes',
# using a bash module implies the target does not have python so we use 'gather_facts: no' so we have
# to do our own tests; and it is a useful example anyway :-).
--- 
- hosts: all
  gather_facts: no
  tasks:
  - name: test direct parms
    become: false
    my_example_module:
      dest: /tmp/hello
      state: present
    register: result
  - debug: var=result
  - name: testing imbedded parms 
    become: false
    my_example_module:
      parms:
        dest: /tmp/hello
        state: present
    register: result 
  - debug: var=result
  - set_fact: target_host="{{ result.hostinfo[0].hostname}}" 
  - include: dummy_file.yml
    when: target_host == 'nagios2'
    ignore_errors: true

Change the hostname in the target_host test condition near the end of the above file to one of your hostnames, it is an example of running a tasks file for a single host and will be skipped if no hostnames match.

Create a file dummy_file.yml (used by the hostname test step) containing the below

  - name: A dummy task to test it is triggered
    local_action: "command echo '{\"changed\": false, \"msg\": \"Dummy task run\"'"
    register: result
  - debug: var=result

Create a file library/my_example_module.sh and paste in the below

#!/bin/bash 
# =====================================================================================================
#
# my_example_module.sh     - example bash ansible module 'my_example_module'
#
# Description:
#   Demonstration of writing a ansible module using the bash shell
#   (1) handles two parameters passed to it (dest, state) passed at either the
#       top level or indented under a parms: field
#   (2) returns useful host OS details (from os-release) as these are useful
#       for logic branching (ie: centos7 and centos8 have different packages
#       available so you need to know what targets modules run on).
#       ---obviously, this method is only useful for linux target hosts
#       Ansible functions such as
#            when: ansible_facts['os_family'] == "RedHat" and ansible_facts['lsb']['major_release'] | int >= 6
#       are obviously not useful for targets that do not run python where we must
#       set gather_facts: no
#
#   Items of note...
#     * all values in the playbook are passed as var=value in a file sent to
#       the remote host and passed as an argument to the script as $1 so script
#       variables can be set withs a simple 'source' command from $1
#     * data values placed immediately after the module name can be used 'as-is'
#       (see example 1) 
#           dest=/tmp/hello
#           state=present
#       however if values are imbedded they will be passed as a
#       JSON string which needs to be parsed
#       (see example 2 where values are placed under a 'parms:' tag)
#           parms='{'"'"'dest'"'"': '"'"'/tmp/hello'"'"', '"'"'state'"'"': '"'"'present'"'"'}'
#       which after the 'source' command sets the parms variable to
#           {'dest': '/tmp/hello', 'state': 'present'}
#       which needs to be parsed to extract the variables
#     * so, don't imbed more than needed for playbook readability or your script will be messy
#
#     * the 'failed' variable indicates to the ansible caller if the script has failed
#       or not so should be set by the script (failed value in ansible completion display)
#     * the 'changed' variable indicates to the ansible caller if the script has changed
#       anything on the host so should be set by the script (the changed value in ansible
#       completion display). You should set that if your module changes anything, but
#       this example has it hard coded as false in the response as the script changes nothing
#     * oh yes, the output/response must be a JSON string, if you have trouble with your
#       outputs try locating the error with https://jsonlint.com/
#
# Usage/Testing
#    Under your currect working directory 'mkdir library' and place your script
#    in there as (for this example) 'my_example_module.sh'.
#    Then as long as the ansible-playbook command is run from your working
#    directory the playbook will magically find and run module my_example_module
#    Obviously for 'production' you would have a dedicated playbook directory
#    in one of the normal locations or use an envronment variable to set the
#    ansible library path, but for testing you do want to keep it isolated to
#    your working directory path :-)
#
# Examples of use in a playbook,
#    1st example is vars at top level, 2nd is imbedded under parms:
#    We use 'gather_facts: no' as using bash modules implies that the
#    targets are servers without python installed so that would fail :-)
#
#   --- 
#   - hosts: all
#     gather_facts: no
#     tasks:
#     - name: example 1 test direct parms
#       my_module_example:
#         dest: /tmp/hello
#         state: present
#       register: result
#     - debug: var=result
#     - name: testing imbedded parms 
#       my_module_example: 
#         parms:
#           dest: /tmp/hello
#           state: present
#       register: result 
#     - debug: var=result
#
#
# Example response produced
#  {
#  	"changed": false,
#  	"failed": false,
#  	"msg": "test run, parms were dest=/somedir state=missing",
#  	"hostinfo": [{
#  		"hostname": "hawk",
#  		"osid": "rocky",
#  		"osname": "Rocky Linux",
#  		"osversion": [{
#  			"major": "8",
#  			"minor": "4"
#  		}]
#  	}]
#  }
#
# =====================================================================================================

source $1         # load all the data values passed in the temporary file
failed="false"    # default is that we have not failed

# If data was passed as a "parms:" subgroup it will be in JSON format such as the below
# {'dest': '/tmp/hello', 'state': 'present'}
# So we need to convert it to dest=xx and state=xx to set the variables
# Parsing for variable name as well as value allows them to be passed in any order
if [ "${parms}." != "." ];
then
   isjson=${parms:0:1}             # field name in playbook is parms:
   if [ "${isjson}." == '{.' ]    # If it is in json format will be {'dest': '/tmp/hello', 'state': 'present'}
   then
     f1=`echo "${parms}" | awk -F\' {'print $2'}`
     d1=`echo "${parms}" | awk -F\' {'print $4'}`
     f2=`echo "${parms}" | awk -F\' {'print $6'}`
     d2=`echo "${parms}" | awk -F\' {'print $8'}`
     export ${f1}="${d1}"     # must use 'export' or the value of f1 is treated as a command
     export ${f2}="${d2}"
   else
      failed="true"
      printf '{ "changed": false, "failed": %s, "msg": "*** Invalid parameters ***" }' "${failed}"
      exit 1
   fi
fi
# Else data was passed as direct values so will have been set by the source command, no parsing needed

# You would of course always check all expected data was provided
if [ "${dest}." == "." -o "${state}." == "." ];
then
   failed="true"
   printf '{ "changed": false, "failed": %s, "msg": "*** Missing parameters ***" }' "${failed}"
   exit 1
fi

OSHOST="$(uname -n)"                                          # Get the node name (host name)
if [ -r /etc/os-release ];
then
   # /etc/os-release is expected to have " around the values, we don't check in this
   # example but assume correct and strip them out.
   # In the real world test for all types of quotes or no quotes :-)
   OSID=`grep '^ID=' /etc/os-release | awk -F\= {'print $2'} | sed -e 's/"//g'`     # Get the OS ID (ie: "rocky")
   OSNAME=`grep '^NAME=' /etc/os-release | awk -F\= {'print $2'} | sed -e 's/"//g'` # Get the OS Name (ie: "Rocky Linux")
   osversion=`grep '^VERSION_ID=' /etc/os-release | awk -F\= {'print $2'} | sed -e 's/"//g'` # Get OS Version (ie: "8.4")
   OSVER_MAJOR=`echo "${osversion}" | awk -F. {'print $1'}`
   OSVER_MINOR=`echo "${osversion}" | awk -F. {'print $2'}`
   if [ "${OSVER_MINOR}." == "." ];   # Debian 11 (at least what I run) does't have a minor version
   then
      OSVER_MINOR="0"
   fi
   hostinfo=`printf '{"hostname": "%s", "osid": "%s", "osname": "%s", "osversion": [{"major": "%s","minor": "%s"}]}' \
            "${OSHOST}" "${OSID}" "${OSNAME}" "${OSVER_MAJOR}" "${OSVER_MINOR}"`
else
   hostinfo=`printf '{"hostname": "%s", "osid": "missing", "osname": "missing", "osversion": [{"major": "0","minor": "0"}]}' "${OSHOST}"`
fi

# Return the JSON response string with a bunch of variables we want to pass back
printf '{ "changed": false, "failed": %s, "msg": "test run, parms were dest=%s state=%s", "hostinfo": [%s] }' \
	 "${failed}" "$dest" "${state}" "${hostinfo}"
exit 0

Create a file named hosts and enter a list of the hostnames or ip-addresses you want to test against as below (using you machines ids of course). Note that the python override is required for Debian11 servers, it does no harm using it on the others.

localhost
192.168.1.177
192.168.1.187
#192.168.1.9
#test_host ansible_port=5555 ansible_host=192.168.1.9
[all:vars]
ansible_python_interpreter=/usr/bin/python3

[test_group]
localhost
192.168.1.177

Create a file names TEST.sh, simply because it is easier to run that multiple times than type in the entire command. Place into that file

ansible-playbook -i hosts my_example_module.yml
#ansible-playbook -i hosts my_example_nopython.yml

Yes the last line is commented, you have not created that file yet.

You are ready to go. Simply ‘bash TEST.sh’ and watch it run, you have your first working bash module.

Now, you are probably wondering about the commented example in the TEST.sh file above.

As mentioned I am curious as to how to use ansible to manage servers that do not have python installed, and have been thinking about how to do it.

This last example avoids the default ansible python modules, manually copies across the script/module to be executed, manually created the data input file to be used, and manually runs the ‘/bin/bash’ command to execute it, cleans up the files it copied.

While manually using the ‘/bin/bash’ command is overkill for Linux servers where you can just place at the start of the file what script execution program should run the script; it shows how you could in theory use the ‘raw’ function to invoke any script processor on the target machine.

I must point out it’s a badly written example, in that ansible is considered a configuration management tool so must have inbuilt functions to copy files from an ansible server to a managed server so having a manual ‘scp’ step is probably not necessary but I am trying to do it with as little inbuilt functions as possible for this example. Also in a managed environment you would probably not scp files from the local server but use curl/wget to pull them from a git repository; but not all operating systems support tools like wget/curl so knowing a manual scp is a way to get files there is useful.

Anyway this example copies exacly the same module as that used in the above example across to the target server, creates a data file of parms, runs the module explicitly specifying /bin/bash as the execution shell, and deletes the two copied files; just as ansible would in the background.

You could take it a lot further, for example not clean up the script file and have a prior step to see if it already existed on the target and skip the copy if it did, useful if you have a huge farm of servers and the files being moved about are large. But all that is beyong the scope of this post.

The playbook to do that is below. You can create the file my_example_nopython.yml, paste the contents below, and uncomment the line in the TEST.sh file to confirm it works. You must of course change the scriptsourcedir value to the working directory you are using, and it must be a full path; and of course change the host ip-addr used to one of your servers.

# Example of using a playbook to simulate what ansible does.
# MUST have 'gather_facts: no' if the target server does not have python3 installed as
# gathering facts is an ansible default module that is of course written in python.
#
# Obviously in the real world you would not copy scripts from a local filesystem but pull them
# from a source repository (but as examples go this you can copy and work on immediately...
# after updating the hosts (and hosts file) and script source location of course
#
# Uses the my demo bash module as the script to run so we must populate a file with the
# two data values it expects, done with a simple echo to a file in this example.
#
# NOTE: if the target server does not have python installed ansible will still happily
#       (if you disabled facts gathering which of course would cause a failure)
#       as part of it's inbuilt processing copy a python module to the target along with
#       a file containing data values and try to run the module (which of course fails);
#       that is the functionality we are duplicating here as an example, as you can
#       easily build on this to make things a lot more complicated :-)
#       (ansible probably uses sftp to copy the files, as the sftp subsystem needs to be enabled)
#       And we of course copy my example module written in bash as we want the demo to work :-)
#
# Why is a playbook like this important ?.
# Many servers that are non-Linux (even non-*nix) support some form of ssh/scp/sftp.
# Using a playbook like this can let you handle the quirks of those systems where supplied
# ansible default modules cannot.
--- 
- hosts: 192.168.1.177
  gather_facts: no
  vars:
    user: ansible
    scriptsourcedir: /home/ansible/testing/library
    scriptname: my_example_module.sh
  tasks:
  - name: copy script to remote host
    local_action: "command scp {{scriptsourcedir}}/{{scriptname}} {{user}}@{{inventory_hostname}}:/var/tmp/{{scriptname}}"
    register: result 
  - debug: var=result
  - name: create remote parm file
    raw: echo 'dest="/some/dir";state="present"' > /var/tmp/{{scriptname}}_data
    become: false
    register: result 
  - name: run remote script
    raw: /bin/bash /var/tmp/{{scriptname}} /var/tmp/{{scriptname}}_data
    become: false
    register: result 
  - debug: var=result
  - name: remove remote script
    raw: /bin/rm /var/tmp/{{scriptname}} /var/tmp/{{scriptname}}_data
    become: false

So, you have seen how to write and test a module in bash, not too complicated after all. There is one important thing you must always remember though. The output of your module must be valid JSON, get a bracket out of place and it will go splat; so two tips

• if you end up with bad JSON output I find https://jsonlint.com/ is a quick way of finding the problem
• if you are unsure of what data is being placed in the data value input file by ansible place a ‘sleep 30’ command in the script which gives you time on the target machine to look at the files under ~ansible/.ansible/tmp (replace ~ansible with the userid used on the target machine) and ‘cat’ the files under there to see what values are actually being set

Enjoy breaking things.

AWX is the upstream open source project for ansible-tower. If you are interested in ansible-tower it is obviously a good starting point, as it is the starting point for ansible-tower. It is also free to use, not just free as in open source but in that you do not need a RedHat account to use it.

AWX now likes to be installed into a Kubernetes environment using awx-operator; that makes it damn hard to be workable in a home lab situation unless you want to spend a lot of time trying to work out what the installation configuration files are doing and changing them to make them usable (ingress/egress/dns etc).

It is far simpler to get a full working environment using Docker, even if the Docker method has been depreciated. This post explains how to get it installed and working using Docker. There are customisations needed to make it useful for a home lab which are covered here; as as you step though them you will understand why I chose Docker for my home lab, it would be a lot more work to make it usable under Kubernetes (even under MiniKube).

These instructions are valid as of 16th October 2021.

This post also assumes, or at least hopes, you have been using ansible from the command line quite happily already so you understand the last step on SSH keys :-).

But first, why would you want to install AWX ?. The only possible reason would be that you want to get an idea of how ansible-tower works, there is certainly no benefit for any small organisation or for your own home lab over the command line; and it is actually a real step backward for a home lab as you really need a spare machine with at least 6Gb ram running either docker or kubernetes (even to get it to run a playbook with a simple shell ‘ls’ command), plus it makes it a lot harder to do things as it expects by default all resources (EE containers, playbooks etc.) to be provided across the internet and never locally (not even local network)… although with a lot of effort you can change that and some of the steps are covered in this post.

Yes it does have user access controls, organisations, groups etc. to limit who can do what isung AWX or ansible-tower. It does not in any way stop anybody with half a clue from simply bypassing all that and using the command line.

However if you want to learn how to use ansible-tower without going anywhere near a RedHat subscription/licence AWX is the place to start.
And if that is what you are interested in then read on.

1. Follow the install instructions from the AWX github site

First get it up and running, thats simply a case of (almost) following the instructions for installing under Docker on the github site. The difference is do NOT use the “-b version” tag to select the latest version, for 19.4.0 at least that pulls down a git clone with no branch head and simply doesn’t work, so omit the “-b” tag and the instructions are simply…

git clone https://github.com/ansible/awx.git
cd awx
vi tools/docker-compose/inventory   # set pg_password,broadcast_websocket_secret,secret_key etc
dnf -y install ansible              # MUST be installed on the build host, used during the build
make docker-compose-build   # creates docker image quay.io/awx/awx_devel
make docker-compose COMPOSE_UP_OPTS=-d     # starts awx, postgresssql and redis containers -detached

Wait until everything is up and running and then do the next steps

docker exec tools_awx_1 make clean-ui ui-devel            # clean and build the UI
docker exec -ti tools_awx_1 awx_manage createsuperuser    # create the initial admin user, default of awx is recomended
                                                          # note: I ran the above twice to create a personal superuser id also
docker exec tools_awx_1 awx-manage create_preload_data    # optional: install the demo data (actually, it may already have been installed)

At the end of the installation you have a running AWX environment, but it has some major limitations we will be fixing below.

At this point you can use a web browser to port 8043 on the Docker host and test the superuser logon you created works, but there is more work to do before going any further.

2. Customisations needed

Do NOT use the “make” command to start it again from this point, or it will destroy any customisations you make to the yaml file by re-creating it.
The install creates persistent docker volumes for the data so after the install we just need a custom script to stop and start the environment rather than using the Makefile.

Simply create a script as below

cat << EOF > control_script.sh
#!/bin/bash
case "$1" in
	"start") cd awx_default
		docker-compose -f tools/docker-compose/_sources/docker-compose.yml up -d --remove-orphans
		;;
	"stop") cd awx_default
		docker-compose -f tools/docker-compose/_sources/docker-compose.yml down
		;;
	*) echo "Use start or stop"
		;;
esac
exit 0
EOF
chmod 755 control_script.sh

Note we are still using the generated yaml (yml) file, but by using the script rather than the “make” command it will not be overwritten.

As installed you cannot use the “manual” playbooks from the filesystem but only from a remote source repository (which has its own problems discussed in the next bit). You may also want to use your own in-house modules.

To use the manual playbooks you need to have a bind mount to the docker host filesystem, so “vi tools/docker-compose/_sources/docker-compose.yml“, in the volumes section add a additional entry to the existing long list as below

     - "/opt/awx_projects:/var/lib/awx/projects"

Then “sudo mkdir /opt/awx_projects;sudo chmod 777 /opt/awx_projects”. Ok, you would probably chown to whatever userid is assigned to UID 1000 rather than set 777 but whatever works.

After restarting the containers you can now place manual playbooks in the /opt/awx_projects directory on the Docker host and AWX will be able to find and use them from the containers /var/lib/awx/projects directory. But don’t restart the containers yet :-).

Also under the projects directory you would normally have directories for individual projects to contain the playbooks for that project. If under each of those project directories you create a new directory named /library you can place any customised or user written modules you want to run in those playbooks (ie: if you have projects/debian and projects/centos you need /projects/debian/library and projects/centos/library; it is within the individual projects levels you create the library directory not the top level projects directory level for AWX). But worth noting as that is the easiest way of implementing your own custom modules.

The second issue is the resolution of host names; it will work fine if you only want playbooks from internet source repositories and only want to manage machines that are resolvable by the world wide web DNS servers but what if you want to use local docker and source management repositories, and more importantly you need to resolve host names of machines on your internal network if you want AWX to manage them.

On the local docker image repository side; at this point AWX will not use insecure private repositories which is a bit of a pain. However to resolve your own internal hostnames you need to reconfigure Docker to use a DNS that can resolve those hostnames. That is simple in Docker, simply “vi /etc/docker/daemon.json” and insert something like the below

{
  "insecure-registries" : [ "docker-local:5000" ],
  "dns" : [ "192.168.1.181" , "192.168.1.1" , "8.8.8.8" ]
}

In the DNS list above is one of my DNS servers (I use dnsmasq) that can resolve all my server hostnames, my router and google DNS to resolve all the external sites. Customise the list for your environment.

Now at this point you will want to restart docker (to pick up the daemon.json changes) and the containers (for the new volume bind mapping), so assuming you created the control_script.sh above.

./control_script.sh stop
systemctl stop docker
systemctl start docker
./control_script.sh start

When the containers are running again “docker exec -ti tools_awx_1 /bin/sh” and try pinging one or more of your local servers to ensure name resolution is working.

3. SSH keys, credentials

If you have been using ansible normally from the command line before you will be aware that part of the implementation is to create SSH keys, the private key on the ansible ‘master’ server and the public key placed on every server to be managed. The good news is that you can just create a “credential” in AWX for the ansible user and paste that same private key into it.

When setting up the command line ansible you had to ssh to every server to be managed from the ‘master’ and reply Y to the ssh prompt; AWX handles that prompt when it tries to run a job on any server that hasn’t been done for yet which is the only time saver over command line. Having said that you still need the public key on every server to be managed so you need to setup a playbook to do that anyway (personally I use puppet for such things so the keys already existed on all my servers).

So for hosts to manage add the new host to an inventory (maybe create some custom inventories first) and in AWX go to inventories->hosts->runcommand [the run command option is only available if you go into an inventory and list the hosts, not from the hosts page] and select the ‘shell’ module and a simple command such as ‘ls’ to make sure it works (use the default awx-EE and the credentials you added the SSH key to). It should just work… although see below.

4. Performance compared to simple command line

The latest versions of AWX use “execution environments” to run commands. This involves spinning up a new container to run each command.

Great for isolation, bad for performance. A simple shell module “ls -la” command from the command line is done in seconds, from AWX it takes few minutes to do the same simple command as it needs to start an execution environment if one is not running (and even download it if there is a later version container image; tip after the first pull of the contianer image change pull to never).

Inventories, ansible hosts files

If you have been using ansible from the command line you probably have quite a few, or even one large ansible hosts file with lots of nice carefully worked out groupings.

This may be a time to split them out into seperate inventories if you have not already done so, for example a hosts file for debian, one for centos, one for rocky, one for fedora etc. depending on how many entries you want to show up in a GUI display of an inventory.

Or you could just create one inventory and store the entire hosts file into that one inventory. By store of course I mean import.

Copy your current ansible hosts file from /etc/ansible (or as many different hosts files as you have to load into seperate inventories) to the awx_tools_1 container. Create an inventory in AWX for each hosts file, and use awx-manage to import the file. The invetory must be pre-created in AWX before trying to import.

For example I created an inventory called internal_machines, did a quick “docker exec -to awx_tools_1 /bin/sh”, in the container a cd to /var/tmp and “vi xxx”; I did a cat of my existing ansible hosts file with all its hosts and groupings and pasted it into the “vi xxx” file on the container session and saved it. Then simply…

awx-manage inventory_import --inventory-name internal_machines --source xxx

My ansible hosts file was in INI format (lots of [] sections rather than json or yaml), the import command will add all the hosts and groups to the inventory (and the hosts to the hosts enries of course).

It is important to note that you should have run a test job (such as an ‘ls -la’ as mentioned above) against a server you have manually added first in order to start at least one execution environment container first; otherwise the import job will time-out waiting for the container to start and fail to import anything.

Issues

The default execution environment awx-ee:latest seems to want to download for every job even though I have the EE settings to only pull if missing. A major issue with this is that often a job will fail with being unable to pull it from quay.io ‘with failed to read blob’, repeatedly retrying will eventually work. The real major major issue is not being able to use a local insecure registry to store a local copy (and even create custom EE images) for local use which would immediately alleviate it as a major issue. And EE’s don’t seem to share images even though they are identical images, for example currently jobs run on the control-plane EE using awx-ee:latest with no problems but the AWX-EE EE using awx-ee:latest is trying to download it again.

Basically unless it becomes really easy to decouple from the internet and use local container image repositories, local git sources, local playbooks etc. its pretty useless unless you are happy with a 2Gb download every time you want to run a simple “ls” command. Expect the same issues with Ansible-Tower which wants to use RedHat repositories, you need to decouple or at least make your local sources of everything the first place searched.

Irritations

It is easy to mess up groupings when trying to manually add them, and if you do a job that runs and ends with “no matching hosts found” that job is marked as a sucessfull job which I personally thing should be a failure.
Under both “schedules” and “management jobs” only the supplied maintenance jobs are available and there does’t seem to be an easy way to add user schedules, therefore it seems pointless to go to all this work when for automation the command line interface run via cron provides better automation.

AWX and it’s downstream ansible-tower provide good audit trails on what defined to the tower application user changed what and ran what within the tower but there is of course no auditing within the tower of who changed what git playbooks (although git will record that) and of course manual playbooks on the filesystem are by their nature unaudited. Sooo, auditing is not really a good reason to use it. Also it has a large overhead, needs a minimum of 2cpus and 4gb of memry (AWX does, ansible-tower needs more) just to run.

And how to import custom modules I’m still looking for as far as AWX goes; easy to use custom modules from the command line. As mentioned earlier in the post you can create a ‘library’ directory for each project, you could also inject into the docker run command the ansible environment variable to specify library search path and create another volume bind to that directory on your host, but really you would want your modules in the same git repository as your playbooks… but as this post has shown how to setup project playbooks on the local filesystem I suppose it’s not really relevant to this post.

Of course AWX/Tower allows creation of individual users/groups/projects/organisations… just like a well laid out filesystem structure and use of the unix users and groups files. Of course reporting on such a layout seems a bit tricky in AWX where on a filesystem you could just use “tree” to display the projects structure and contents.

Far easier to run ansible from the command line using filesystem based playbooks (my filesystem playbooks are git managed on the host so no advantage in using AWX or ansible-tower there) which has absolutely no overhead requirements other than whatever memory a terminal or ssh session takes. Plus no requirement on Docker or Kubernetes (no execution environment containers), just small fast and simple and if you want regular jobs scheduled by cron.

However if I can one day get insecure registries working for EE images, and figure out how to get playbooks and custom modules deployed from my local gitlab server (rather than having to use the filesystem volume mount) it may be useful; and I might even update this post with how. However by useful I mean as a learning experience, my home lab doesn’t have a spare 2Gb memory I can dedicate to actually using it when I find puppetserver so much more powerfull.

This post is not about the differences between Docker and Kubernetes; it is about the differences needed for containers. And why you should not expect any container from dockerhub.io to run under Kubernetes.

The main difference discussed here is that

• Docker and Docker Swarm run containers from a ‘root’ environment by default
• Kubernetes containers run under the user that started Kubernetes or as defined by the RunasUser and RunasGroup parameters in the deployment yaml file

For Docker this allows complex applications with components running under multiple users to be run from a container, as the container sartup script is run as root the startup script can ‘su’ down to each of the neeed users to start those components. As long as any UID needed for bind mounts is documented that works well.

The same startup logic cannot be used for Kubernetes containers. The container startup script must be owned and executable by the RunasUser/RunasGroup pair and as a general rule all application components would be started by that one user. This helps in keeping containers designed for one purpose as simple and as small as possible (microservice) but makes running an entire application with more than one component in a container difficult.

Many container images designed for docker require to be run in a root environment. Should you try to run an image that has the startup script secured to root or user X with a RunasUser of Y under kubernetes the pod will just go into CrashLoopBackOff status because of permission denied on the script.

It is certanly possible to design containers to run under both environments. What makes it difficult is that there is no environment variable set for a container by docker of kubernetes to let the container know where it is running; not a major issue as you can simply provide your own.

From personal experience with my own small containers designed for docker that I wanted to move only used ‘su’ to move from root to the user I wanted the app to run under, the conversion to support both is simply to pass a custom environment variable to the container if running under kubernetes and if the variable exists assume the RunasUser/RunasGroup were set correctly and just start the app; if not present assume docker and ‘su’ down to the expected user to start the app. Actually as I was writing this post I just thought an easier way would be to simply check if the script is running as root and determine the environment that way, which I am going to use instead :-).

It is certainly possible to start apps under multiple users under Kubernetes; you would simply add ‘sudoers’ entries for the RunasUser to allow it to run the startup commands under the correct user. But if the container is designed to run under either engine would have sudoer entries in the container not required by docker and containers are audited these days :-).

For me personally I only want to build one container per app and have it run under either.

If you are relying on external container images just remember images on dockerhub.io are for Docker, don’t expect them to even start on Kubernetes. Images built explicitly for Kubernetes are not intended to be run as root so should not be run under Docker.

If you are developing your own containers and use both engines, design the container to be able to run under both. All that is needed is the awareness of the fact Docker will run startup scripts as root and Kubernetes prefers not to.

This post is primarily because containers I was moving from docker to minikube were going into CrashLoopBackOff and after sorting that out thought it may be of interest to others as it doesn’t seem to be highlighted anywhere.