Marks Weblog

First let me make it clear this post is on dropbox as used in a Linux environment, specifically for this post Fedora (Fedora 32). Dropbax clients may behave differently in different environments.

What is lansync

The dropbox reference pages promote lansync as a way of saving external bandwidth, in that if a file is updated it can be replicated to other machines replicating that folder across the local subnet/network without each local client needing to retrieve the changed file from the dropbox servers.

Ports lansync and dropbox use

A DuckDuckGo search of “dropbox lansync port numbers” returns the following pages.
The description of lansync is that it wants port 17500 on both udp and tcp.
(reference https://help.dropbox.com/installs-integrations/sync-uploads/lan-sync-overview)
Or 17500 only on tcp (reference https://dropbox.tech/infrastructure/inside-lan-sync)

From observation it uses port 17500 on udp, tcp, and tcp6

[mark@hawk bin]$ netstat -an | grep 17500
[mark@hawk bin]$ dropbox lansync y
[mark@hawk bin]$ netstat -an | grep 17500
tcp        0      0 0.0.0.0:17500           0.0.0.0:*               LISTEN     
tcp6       0      0 :::17500                :::*                    LISTEN     
udp        0      0 0.0.0.0:17500           0.0.0.0:*                          

However, due to the extensive logging on run on one of my servers, a server which does not run dropbox, only udp appears to be used. Worse, dropbox never tops trying, two machines run dropbox and each are retying a server that will never respond at thirty second intervals

Sep 17 14:49:24 mdickinson kernel: DROPPED IN=ens3 OUT= MAC=ff:ff:ff:ff:ff:ff:50:b7:c3:20:19:8f:08:00 SRC=192.168.1.9 DST=255.255.255.255 LEN=161 TOS=0x00 PREC=0x00 TTL=64 ID=34187 DF PROTO=UDP SPT=17500 DPT=17500 LEN=141 
Sep 17 14:49:33 mdickinson kernel: DROPPED IN=ens3 OUT= MAC=ff:ff:ff:ff:ff:ff:74:d0:2b:92:f4:f7:08:00 SRC=192.168.1.187 DST=255.255.255.255 LEN=161 TOS=0x00 PREC=0x00 TTL=64 ID=7163 DF PROTO=UDP SPT=17500 DPT=17500 LEN=141 
Sep 17 14:49:54 mdickinson kernel: DROPPED IN=ens3 OUT= MAC=ff:ff:ff:ff:ff:ff:50:b7:c3:20:19:8f:08:00 SRC=192.168.1.9 DST=255.255.255.255 LEN=161 TOS=0x00 PREC=0x00 TTL=64 ID=54938 DF PROTO=UDP SPT=17500 DPT=17500 LEN=141 
Sep 17 14:50:03 mdickinson kernel: DROPPED IN=ens3 OUT= MAC=ff:ff:ff:ff:ff:ff:74:d0:2b:92:f4:f7:08:00 SRC=192.168.1.187 DST=255.255.255.255 LEN=161 TOS=0x00 PREC=0x00 TTL=64 ID=21945 DF PROTO=UDP SPT=17500 DPT=17500 LEN=141 
Sep 17 14:50:24 mdickinson kernel: DROPPED IN=ens3 OUT= MAC=ff:ff:ff:ff:ff:ff:50:b7:c3:20:19:8f:08:00 SRC=192.168.1.9 DST=255.255.255.255 LEN=161 TOS=0x00 PREC=0x00 TTL=64 ID=2628 DF PROTO=UDP SPT=17500 DPT=17500 LEN=141 
Sep 17 14:50:33 mdickinson kernel: DROPPED IN=ens3 OUT= MAC=ff:ff:ff:ff:ff:ff:74:d0:2b:92:f4:f7:08:00 SRC=192.168.1.187 DST=255.255.255.255 LEN=161 TOS=0x00 PREC=0x00 TTL=64 ID=26919 DF PROTO=UDP SPT=17500 DPT=17500 LEN=141 

Note: when dropbox is running it also uses ports 17600 and 17603, although these are bound to localhost so not a risk as long as you have those ports open for access from localhost (yes I do have some firewall rules that prevent access to services from localhost). That is documented at https://help.dropbox.com/installs-integrations/desktop/configuring-firewall

root@hawk ~]# netstat -an | grep 176
tcp        0      0 127.0.0.1:17600         0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:17603         0.0.0.0:*               LISTEN 

Nowhere can I find documentation on when and why dropbox will use port 17601/tcp. However on multiple occasions on multiple machines I have seen port 17601 tcp in use by /home/mark/.dropbox-dist/dropbox-lnx.x86_64-105.4.651/dropbox (identified by ‘netstat -anp’ and querying the pid; note also seen on earlier versions of dropbox), although only on localhost (only listening on 127.0.0.1). And it appears to stop listening immediately when lansync is turned off ?.

The same cannot be said for the udp port 17500, turning off lansync stops the 30 second polling but leaves the local udp port open until dropbox is restarted.

[mark@hawk bin]$ dropbox lansync n
[mark@hawk bin]$ netstat -an | grep 17500
udp        0      0 0.0.0.0:17500           0.0.0.0:*                          
[mark@hawk bin]$ dropbox stop
Dropbox daemon stopped.
[mark@hawk bin]$ dropbox start
[mark@hawk bin]$ netstat -an | grep 17500
[mark@hawk bin]$ dropbox status
Up to date
[mark@hawk bin]$ 

It is probable that it uses udp to find machines, and tcp to transfer data.

Why not to use lansync, and when it may be ok

In a small replication environment where there may be only two or three machines replicating a folder if the file changes are small it is probably going to generate less network traffic by having all machines use the dropbox servers as the source for the changed file; especially as whichever machine changes the file will push it to the dropbox servers anyway.

Also in a small replication environment where there may be only a few users replicating the directory contents having those few users running lansync to poll every machine in the local subnet every thirty seconds is a large overhead (there may be hundreds of machines on the local subnet having to put up with all the polling).

Then of course there may be other small groups all doing the same thing causing a lot of needless broadcast udp traffic.

In environments such as this the preferred solution would be to do something like have an ownCloud or nextCloud server for each small group, all that needs is a single http server and each group could be assigned its own unique url to their own copy of it. Yes both of those two file sync software products also have clients that perform extensive polling, but only to the ownCloud/nextCloud server they are configured to use if they are doing file syncronisation rather than to every server and desktop in the local subnet/network. And if the files should be on dropbox, that one server can push them there; with maybe one ‘manager’ interacting with dropbox to manage conflicts.

In a large environment lansync would make sense if the majority of users on the same subnet were all interesting in replicating the same set of files. However I would think such a situation would be the exception rather than the norm.

It would also make sense in an environment where internet bandwidth is constrained or chargeable as while generating a lot of additional traffic on the local subnet it should cut down on external internet traffic, although one copy would always be uodated on the dropbox servers.

The main issue with lansync

The main issue with lansync is that is enabled by default. For home users with a small local network of a few PCs, maybe a TV, possible a couple of cell phones and maybe a laptop or tablet, connected to your default home router; having all those devices spammed with UDP traffic every thirty seconds simply because one device may have had dropbox installed on it is a pretty bad default.

In a large commercial environment even where lansync may make sense having it spam from install time before admins get around to setting up routing/firewalls/filtering also seems like a bad default; it should default to off in both large and small environments and turned on if needed.

However as toggling it is as sinple as using the commands ‘dropbox lansync n’ and ‘dropbox lansync y’ it is only an issue if users do not know that is is enabled by default. I only discovered it was the default when the firewall logs on one of my servers starting dropping traffic on port 17500 and I investigated where it was coming from.

Disclaimer

I do use dropbox for personal use, there are some files I want to guarantee a copy of should all my servers suddenly die, as it is only on a desktop and laptop (laptop normally powered off) I have disabled lansync.

A second desktop is just manually kept in sync by copying changes to important files in the dropbox folder to an ownCloud folder syncing across all my desktop machines; and of course bacula does daily backups of both.

For important files do not rely on one solution. An if you are syncronising files across multiple desktops use a solution that does not broadcast udp traffic to servers, TVs, cell phones etc as that is a needless overhead that you do not want; if you use dropbox at home disable lansync.

First let me make it clear I prefer webalizer to awstats; however webalizer is not available in the repositories after CentOS7. It is also worth noting that both webalizer.org and mrunix.net (where webalizer sources/downloads could be manually obtained) appear to be no longer online and there does not appear to be a github source for it either.

Another issue is that webalizer requires, and awstats can use the free geo-ip ‘lite’ country databases from maxmind to resolve ip-addresses to countries; however while still free from December 2019 you must create an account to download them (https://dev.maxmind.com/geoip/geoip2/geolite2/) and we all have far too many internet accounts as it is.

So with webalizer being unavailable and awstats fortunately able to use another free (although out of date) ip lookup database awstats is the logical replacement.

Preparation work

For awstats to use the free ip database you need an additional perl module, you should therefore try to install that first.

perl -MCPAN -e shell
install Geo::IPfree

For awstats to produce PDF reports the ‘htmldoc’ package needs to be installed; that unfortunately is also not in the CentOS8 repositories and would need to be installed using snap. Documentation in installing htmldoc using snap is at https://snapcraft.io/install/htmldoc/centos; however I did not install it as snap is a bloody pain to use and chews up a lot of system resources; the only time I installed snap to get a package I immediately uninstalled it all again. Fortunately if you do not need to generate PDF reports that is no issue. Should the ‘htmldoc’ package ever make it to the CentOS8 repositories I will install it from those at that time.

It is also very important to note that awstats expects all httpd log messages to be in a specific format or it will refuse to process the logs. Anywhere you are defining log files you should use

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog "logs/access_log" combined

You would normally need to set these in a default installation in /etc/httpd/conf/httpd.conf and /etc/httpd/conf.d/ssl.conf; but set anywhere you define log files.

It is also important to note that awstats wants to process only one file, and expects it to be in date order, so if you have seperate access_log and ssl_access_log messages (the default in a CentOS install) you will have issues. What I have done, as I wish to keep seperate files to identify the different traffic for other utilities, is leave those alone but add as an additional entry anywhere that needs logging a new logfile all will share, for example the below line I would have in both the main httpd.conf and ssl.conf files so as well as the unique files all messages from both are also logged to a combined file in the correct LogFormat tagged as ‘combined’.

CustomLog "logs/combined_access_log" combined

This permits the single LogFile name needed by awstats to be used against a file that does contain all the log messages needed.

If your logfiles were not already using the correct format all the existing logfiles cannot be processed by awstats. If you have just configured a new ‘combined_access_log’ as discussed above in the required format restart httpd to start aquiring messages in that file that you can use in later steps.

Installing, configuring, populating

To install awstats it is simple a case of ‘dnf -y install awstats’.

After which there is a bit of editing involved, while it does create a /etc/httpd/conf.d/awstats.conf file all the entries in there refer to /usr/local/awstats but the package files are actually installed in /usr/share/awstats. The best option is to remove everything in that file and replace it with the contents below. The file below is based upon the documentation at https://tecadmin.net/steps-to-configure-awstats-on-centos-and-rhel-system/ but modified to include access to the documentation directory (as being able to browse the documentation is useful) as well as using the latest ‘require ip’ syntax (as ‘allow from’ and ‘require host’ are obsolete and not able to be used without loading compatibility modules).

#
# Content of this file, with correct values, can be automatically added to
# your Apache server by using the AWStats configure.pl tool.
#
# If using Windows and Perl ActiveStat, this is to enable Perl script as CGI.
#ScriptInterpreterSource registry
#
# Directives to add to your Apache conf file to allow use of AWStats as a CGI.
# Note that path "/usr/local/awstats/" must reflect your AWStats install path.
Alias /awstatsclasses "/usr/share/awstats/wwwroot/classes/"
Alias /awstatscss "/usr/share/awstats/wwwroot/css/"
Alias /awstatsicons "/usr/share/awstats/wwwroot/icon/"
ScriptAlias /awstats/ "/usr/share/awstats/wwwroot/cgi-bin/"

Alias /awstatsdoc "/usr/share/doc/awstats/"
<Directory /usr/share/doc/>
   DirectoryIndex index.html
   Require all granted
</Directory>

<Directory "/usr/share/awstats/wwwroot">
   Options +ExecCGI
   AllowOverride None
   Require ip 192.168.1.0/24
</Directory>
<IfModule mod_env.c>
   SetEnv PERL5LIB /usr/share/awstats/lib:/usr/share/awstats/plugins
</IfModule>

You then need to ‘cd /etc/awstats’. You will find a model.conf, awstats.localhost.localdomain.conf, and awstats.HOSTNAME.conf (where HOSTNAME) is the name of your host have been created. Copy any one of them to awstats.your.website.name.conf where your.website.name is your website name, for example if your host is webserver.example.org you need a file named awstats.webserver.example.org.conf; and then open the new file in your favourite editor.

• set the ‘LogFile’ value; if you followed my recomendation of using a combined_access_log above set it to that, otherwise set it to the only one of the files you want to process (masks such as “*access.log” will not work; also logresolvmerge as documented in template file did not work for me)
• set the ‘SiteDomain’ value to match your website name
• set the ‘HostAliases’ values
• set the ‘DNSLookup’ value to ‘0’ unless you want to do lots of dns lookups
• review the comments for ‘EnableLockForUpdate’ and decide if you want to set it to ‘0’
• if using dns lookups set the ‘SkipDNSLookupFor’ list
• you may want to set ip-addr ranges for clients that can browse the results in the ‘AllowAccessFromWebToFollowingIPAddresses’ list; although you are also limiting by ip-address in the conf.d/awstats.conf file above
• set the ‘SkipHosts’ value to avoid reporting on internal hosts that would skew the report values (such as nagios/nrpe or other health checking activity)
• Seach on ‘LoadPlugin=”geoipfree”‘ and uncomment it (if you installed the Geo::IPfree mentioned at the start of this post)
• set ‘UseFramesWhenCGI=0’, browsers such as FireFox will refuse to display any pages presented in frames aas frames are inherently insecure

You should also note the entries for URL alias references (/awstatscss,/awstatsicons etc) throughout the file, initially leave all those as the defaults as they match what is set in the /etc/httpd/conf.d/awstats.conf file; and they must match.

Before you are able to do anything you need to create some initial data. If you have had to change your log message formats after restarting httpd just wait for a while, you need some log messages available before you are able to initialise awstats.

When you have a reasonably large collection of logged messages simply run the command below replacing ‘webserver.example.org’ with your hostname, which is the part between awstats. and .conf in the /etc/awstats directory; the example below would use the config file ‘awstats.webserver.example.org.conf’.

perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=webserver.example.org -update

Keeping the statistics up to date

Installing the package creates a file that will perform updates for all the hosts you have configured in seperate conf files for each host in /etc/awstats. That job is /etc/cron.hourly/awstats.

So once you have all the configuration in the steps above done your data is updated automatically.

You may want to create a report job in /etc/cron.daily or /etc/cron.monthly to produce reports that can be archived in case you need to initialise your collected statistics at some point.

Viewing the statistics with a web browser

I suppose after all that work you actually want to look at some of the statistics. If you use the example /etc/httpd/conf.d/awstats.conf file above you would point your web browser at https://yourhostname/awstats/awstats.pl?config=webserver.example.org where the config value matches the configuration file you used.

If you are used to webalizer output you may find it frustrating to hunt down information in. Some information is also incorrect with default settings although I suppose it is possible to correct it with a lot of manual effort in the configuration file and log format statements. But as webalizer is no longer available awstats is a viable replacement; if a lot harder to get working.

The ability to generate static reports may be useful for archiving, but I have not tested that as they would be most useful in PDF form, which as mentioned above awaits the ‘htmldoc’ package reaching the CentOS8 repos one day.

The first thing to note about this post is that it sets up an insecure local registry facility using the standard registry container. This type of registry is ideal for local ‘internal network’ development use. It is also suitable for standalone docker, no need for a docker swarm.

An important thing to note is that the regsitry server itself is not insecure, it wants TLS traffic by default; however it supports insecure traffic if the client requests it. To use insecure access to the local registry server it is the clients of the registry server that are reconfigured to request insecure communication; and the registry server will permit it.

Configuring an insecure Docker registry allows anyone to ‘push’ images to your registry without authentication, so it must only be used for internal use; never internet facing.

You will notice the official documentation for installing a docker registry container as an insecure installation states that not even basic authentication can be used for an insecure configuration. The may actually be incorrect as the configuration document at https://docs.docker.com/registry/configuration/#htpasswd states basic authentication can be configured without TLS… although the user/password information will be passed in clear text as part of the http header so it seems it is only recomended not to use it.

Installing a local Docker registry

A local registry server can be installed on any server already running Docker or docker-ce. How to setup a local registry server is covered at https://docs.docker.com/registry/deploying/ however it is a little fuzzy on configuring insecure traffic.

The actual document on configuring for insecure use is https://docs.docker.com/registry/insecure/ but it omits the rather import detail that the “insecure-registry” settings must be set on all the clients, not on the docker server running the container. There is a lot of confusion about that easily seen from all the questions about it in forums where everyone assumes it is set on the server providing the registry container; it is set on all the clients. Also note in this document it does state that secure traffic will always be tried first in all cases, the “insecure-registries” entry just allows fallback to insecure traffic, so changing your registry to be secure at a later time is trivial.

It is also important you are aware that by default the container running the registry uses volume persistent storage within the container; this means that while it will survive stopping/starting the container should you delete the container your data will be lost. If you intend to be deleting/reconfiguring the container a lot you probably don’t want to use that default.

I implemented my local registry container to use a directory on the docker host filesystem. The run command I used is below, the importance of the REGISTRY_STORAGE_DELETE_ENABLED environment variable I will discuss later in this post in managing the registry; you would probably normally leave it disabled (the default).

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v /home/docker/data:/var/lib/registry \
  -e REGISTRY_STORAGE_DELETE_ENABLED="true" \
  registry:2

Remember to open firewall port 5000 on your registry docker host, and if any client docker hosts have rules blocking outbound traffic ensure the port is opened on those also.

Configuring the Docker client servers for insecure traffic

The “insecure-registries” setting needs to be configured on the servers running Docker that are to be clients of this local registry. On those servers add to (or create if it does not exist) the file /etc/docker/daemon.json and restart the docker service on those clients when you have done so.

{
  "insecure-registries" : [ "hostname-or-ipaddr:5000" ]
}

Of course if you also wish to use the registry on the Docker server running the regisry container set it there also.
If you have a local DNS server you should use the hostname rather than an ip-address.

Pushing and Pulling using your local Docker registry

In the examples here I have the regsitry container running on a host with a DNS entry of docker-local.myexample.org

To push images to your local registry they must be tagged to refer to your local registry. For example if you have an image named busybox:latest you would

docker tag busybox:latest docker-local.myexample.org:5000/busybox:latest
docker push docker-local.myexample.org:5000/busybox:latest

If you get an error along the lines of https expected but got http check your client insecure-registry entries again. On the client the command ‘docker info’ will list the insecure registries configured near the eand of the reponse.

It is also important to note that the tag is a pointer to what is already there, you could in the above example ‘docker image rm busybox:latest’ (which only removed the old pointer) and change your docker run command to run docker-local:5000/busybox:latest instead of busybox:latest which would work perfectly well.

If you have a hostname:port/ in the image name that hostname:port is the registry used, if omitted only the registry at docker.io is used; which you obviously do not have permission to push to.

Once you have images pushed to your local registry you can pull them with the same syntax

docker pull docker-local.myexample.org:5000/busybox:latest

Managing your local registry

Querying your local registry

Useful commands such as ‘docker search’ only apply to the docker.io registry. You obviously need a way of managing your local registry and keeping track of what is in it.

The ‘v2’ interface of the Docker registry container provides a way of looking up what is in your local registry. This can be done with any web browser.

Remembering that we have no certificates and are using an insecure registry the following URLs are useful for determining what is in your registry. The examples use the same example hostname and image.

To see what is in the local registry

    http://docker-local.myexample.org:5000/v2/_catalog
    http://docker-local.myexample.org:5000/v2/_catalog?n=200

Note the second example above, the default response is only the first 100 entries; that can be changed with the n value. If using the default and there are more than 100 entries a link to click on to provide the next 100 entries is provided in the response.

The above URL only displays the image names. You will need to see what tagged versions are stored in the registry. Should yiu have an image names ‘busybox’ and want to see all tagged versions the URL would be

http://docker-local.myexample.org:5000/v2/busybox/tags/list

Of course there are scripts on github that make all that a lot easier and can be run from the command line where most Unix developers work. One is discussed below.

Deleting images from your local registry

The Docker registry ‘v2’ API provides a DELETE facility. As it is possible to corrupt images if you delete incorrect layers it is better to use some of the utilities users have made available on github for that purpose.

I would suggest, and the examples below use, this utility…

cd ~
mkdir git-3rdparty
cd git-3rdparty
git clone https://github.com/byrnedo/docker-reg-tool.git
dnf -y install jp

The below examples of using the script obviously use different examples than busybox above to provide a few more entries to play with; but using the script is fairly simple as seen below. Note that we use “INSECURE_REGISTRY=true” as we have setup an insecure registry; if using TLS there are parameters to provide certs and credentials which are explained on the github page.

[root@gitlab ~]# cd ~git-3rdparty/docker-reg-tool
[root@gitlab docker-reg-tool]# INSECURE_REGISTRY=true ./docker_reg_tool http://docker-local:5000 list
ircserver
mvs38j
[root@gitlab docker-reg-tool]# INSECURE_REGISTRY=true ./docker_reg_tool http://docker-local:5000 list ircserver
f32
f30
[root@gitlab docker-reg-tool]# INSECURE_REGISTRY=true ./docker_reg_tool http://docker-local:5000 delete ircserver f30
DIGEST: sha256:355a3c2bd111b42ea7c1320085c6472ae42bc2de7b13cc1adf54a815ee74fa45
Successfully deleted
[root@gitlab docker-reg-tool]# INSECURE_REGISTRY=true ./docker_reg_tool http://docker-local:5000 list ircserver
f32
[root@gitlab docker-reg-tool]#

However for the delete example above you will most likely get a response as below

[root@gitlab docker-reg-tool]# INSECURE_REGISTRY=true ./docker_reg_tool http://docker-local:5000 delete ircserver f30
DIGEST: sha256:355a3c2bd111b42ea7c1320085c6472ae42bc2de7b13cc1adf54a815ee74fa45
Failed to delete: 405

Refer back to by ‘docker run’ command above, and the parameter I said I would explain later, specifically the environment parameter ‘REGISTRY_STORAGE_DELETE_ENABLED=”true”‘. If that parameter is not set it is not possible to delete entries from the registry… so for a local regsitry you should probably set it unless you intend to keep infinate amounts of images in your local registry.

Reclaiming space in the registry filesystem

Using the registry ‘v2’ API to delete an image/tag from your local registry does not delete anything except the references to the object. This is normally ideal as it preserves layers; for example if you have seven images based on centos:8 they can share the same base layer to minimise space, deleting one of them removes just the pointer. If all are deleted the layers remain as you may push another centos:8 based image in which case a pointer can be re-added rather than the push having to send all layers of the new image.

However there will be occasions when you do want to remove all unused objects from the registry. For example your local development may have been using f30 base images but all development has moved to f32 so you want to reclaim all space used by the old f30 layers.

In order to do so you need to run the registry garbage cleaner to remove the obsolete objects.

The registry garbage cleaner is provided with the docker registry image, and must be run from within the running container.

It should only be run when the registry is not in use; in read-only mode or with user access locked out in some other way. While there is nothing to stop you running it while the registry is in active use any ‘push’ command in progress while the registry garbage cleanup is in progress will result in a corrupt image being stored.

With all that being said, to change to read-only-mode is a case of stopping the regsitry container/deleteing the registry container/redefining-starting the registry in read-only-mode/doing the garbage collect/stopping the registry container/deleting the registry container/redefining starting the registry on write mode again… and if you are going to do all the work you may as well throw in an extra set of stops/deleted/reconfigures to turn on/off the ability to delete entries. Personally I think for a local development registry that is too much complication and I leave registry storage delete enabled and do not switch to read-only-mode (if you need to ensure read-only much simpler to disable the firewall rule allowing access to port 5000 and add it back when done).

To actually perform the garbage reclaimation; on the server hosting Docker container for the registry simply

docker exec -ti regsitry /bin/sh

bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml
exit

Obviously remove the ‘–dry-run’ flag when you are ready to really perform the garbage cleanup.

Some notes on restricting external access, apache proxy

As noted earlier using an insecure registry supposedly prevents any authentication method being used. While easy to switch to TLS that does not magically enable authentication, a lot of extra work is required.

Setting up basic authentication is reasonably easy, and it seems this can be done in clear traffic without TLS if you really want to. However that will the require all your users to authenticate for not just ‘push’ but also for ‘pull’ requests (via ‘docker login’). That limits it’s usability as ideally users should be able to pull anonymously or why bother making it available to them in the first place.

The simpleset way of setting up authentication where users that ‘push’ must authenticate but anyone can ‘pull’ without authentication would seem to be using Apache as a authenticating proxy, by playing with the recipe provided at https://docs.docker.com/registry/recipes/apache/; changing the restriction on GET to allow everyone should do it. And of course create a conf.d file for the virtualhost in your existing web server configuration rather than use the httpd container the example uses. This however still uses basic htaccess authentication although on the web server itself, the registry itself is still insecure using this method but as that would normally run on a seperate machine to the webserver with all public facing traffic having to go via the web server to reach it that is not so much of an issue. Also notice that the example does not really make sense (htpasswd is run in a httpd docker container and the groups created outside the container), but it does at least indicate that all the auth is done by apache in the apache container and not the by the registry container.

One note on the Apache authentication proxy method linked to above is that the document lists as a drawback that the TLS certs must be moved to the webserver as the TLS endpoint instead of on the registry, and the proxy is to the registry at http://servername:5000/v2. Yes, it does mean you must leave your registry container configured as discussed above in that no certificates are configured on the registry itself, but you no longer need to configure the “insecure-hosts” entry on your clients if they pull via the proxy as they will now get https reposnses (provided by the web server).

Also if you already have a public facing Apache webserver with valid certificates an apache proxy setup may be the way to go as yiu do not need to obtain additional certificates. The issues and benefits with the apache proxy approach are

• issue:if you have not used ‘docker login’ to login to your proxy a ‘push’ request results in ‘no basic auth credentials’, however a ‘pull’ request returns the web servers 400 error page (with ‘require valid user’ for GET even if the option to not proxy apache httpd error pages is commented). However if you do not restrict GET access that is not an issue
• benefit: after using ‘docker login’ users can push/pull as expected
• benefit: configuring the GET method rule to “all granted” rather than “valid user” allows any user to ‘pull’ from your registry, which would be the only point of exposing it to the internet. ‘push’ and delete requests are still denied via the proxy if the user has not isued a ‘docker login’ to you local registry
• issue: the mapping must be to the path /v2 as that is what the docker command requires; an open invite to hackers ?. Exactly what can be done with registry GET requests, are any destructive, thats an unknown
• benefit: you are not required to obtain any new ssl certificates, if your website already has working certificates you can configure the same certificates your website already uses in the virtual host entry for the registry

Using the apache proxy with “require all” for GET and leaving the other http options unchanged results in all ‘push’ requests being denied unless a user in the ‘pusher’ group you defined has used ‘docker login’ while all ‘pull’ requests are being permitted; which is probably want you want if you expose it to the internet for end users to pull images from.

[root@gitlab docker]# docker tag busybox mywebproxy.myexample.org:5043/busybox
[root@gitlab docker]# docker push mywebproxy.myexample.org:5043/busybox
The push refers to repository [mywebproxy.myexample.org:5043/busybox]
514c3a3e64d4: Preparing 
unauthorized: authentication required

[root@gitlab docker]# docker pull mywebproxy.myexample.org:5043/mvs38j:f30
f30: Pulling from mvs38j
33fd06a469a7: Pull complete 
Digest: sha256:46eb3fb42e4c6ffbd98215ea5d008afc6f19be80d5c1b331f7ba23e07c9d8e46
Status: Downloaded newer image for mywebproxy.myexample.org:5043/mvs38j:f30
mywebproxy.myexample.org:5043/mvs38j:f30

While configuring users in a htpasswd and group file on an apache server providing a proxy service can seperate users that can only use GET operations and those that can perform all operations by requiring them to ‘docker login’ via the proxy, if you do intend to allow external users to pull images from your repository my recomendation would be to allow all users to ‘pull’ and no users to ‘push’ (simply have no users in the pusher group) via a public facing proxy configuration. Any ‘push’ processing should only be done from the trusted internal network anyway.

This is my replacement for the script provided on the docker site to create a working  Apache proxy configuraion on an existing Apache web server.

DOCKER_REGISTRY_HOST="docker-local" # hostname running the registry container, may be localhost if running it there
DOCKER_REGISTRY_PORT="5000" # port name used by the insecure container
APACHE_DOCKER_AUTH_DIR="/var/www/html/registry-auth" # directory to use for proxy vhost htpasswd and group data files
USERNAME_PUSH_AUTH="mark" # user to demo push access
USERNAME_PUSH_PASSWORD="pusherpwd" # password for above
SSL_CERT_PATH="/etc/letsencrypt/live/mywebserver.myexample.org" # where are the certs used by the existing website

# Did we have a valid cert directory ?, change nothing if not
if [ ! -d ${SSL_CERT_PATH} ];
then
echo "${SSL_CERT_PATH} is not a directory"
exit 1
fi

# Ensure the directories exist, create the needed files
if [ ! -d ${APACHE_DOCKER_AUTH_DIR} ];
then
mkdir -p ${APACHE_DOCKER_AUTH_DIR}
chown apache:apache ${APACHE_DOCKER_AUTH_DIR}
done
htpasswd -Bbn ${USERNAME_PUSH_AUTH} ${USERNAME_PUSH_PASSWORD} >> ${APACHE_DOCKER_AUTH_DIR}/httpd.htpasswd
echo "pusher: ${USERNAME_PUSH_AUTH}" >> ${APACHE_DOCKER_AUTH_DIR}/httpd.groups"
chown apache:apache ${APACHE_DOCKER_AUTH_DIR}/httpd.htpasswd
chown apache:apache ${APACHE_DOCKER_AUTH_DIR}/httpd.groups"

# Create the proxy configuration file
cat << EOF > /etc/httpd/conf.d/docker-registry.conf
LoadModule headers_module modules/mod_headers.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule unixd_module modules/mod_unixd.so

Listen 5043
<VirtualHost *:5043>
ServerName mywebproxy.myexample.org
SSLEngine on
SSLCertificateFile ${SSL_CERT_PATH}/fullchain.pem
SSLCertificateKeyFile ${SSL_CERT_PATH}/privkey.pem
SSLCertificateChainFile ${SSL_CERT_PATH}/fullchain.pem
SSLCompression off
SSLProtocol all -SSLv2 -SSLv3 -TLSv1
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLHonorCipherOrder on
Header always set "Docker-Distribution-Api-Version" "registry/2.0"
Header onsuccess set "Docker-Distribution-Api-Version" "registry/2.0"
RequestHeader set X-Forwarded-Proto "https"
ProxyRequests off
ProxyPreserveHost on
# no proxy for /error/ (Apache HTTPd errors messages)
ProxyPass /error/ !
ProxyPass /v2 http://${DOCKER_REGISTRY_HOST}:${DOCKER_REGISTRY_PORT}/v2
ProxyPassReverse /v2 http://${DOCKER_REGISTRY_HOST}:${DOCKER_REGISTRY_PORT}/v2
<Location /v2>
Order deny,allow
Allow from all
# MUST match realm to the 'basic-realm' used by default in the registry container
# If you change the realm value used by the registry container change this also
AuthName "basic-realm"
AuthType basic
AuthUserFile "${APACHE_DOCKER_AUTH_DIR}/httpd.htpasswd"
AuthGroupFile "${APACHE_DOCKER_AUTH_DIR}/httpd.groups"
# Read access to any users
<Limit GET HEAD>
Require all granted
</Limit>
# Write access to docker-deployer only
<Limit POST PUT DELETE PATCH>
Require group pusher
</Limit>
</Location>
</VirtualHost>
EOF

chown apache:apache /etc/httpd/conf.d/docker-registry.conf

# Implement !
# The below assumes that your apache configuration uses the default of loading all .conf files
# in the conf.d directory; if you selectivy load files from that directory ensure you add
# the new conf file created by this script.
systemctl restart httpd

# What you MUST do
# your webserver host must accept traffic on port 5043
# your webserver mut be able to pass traffic to port 5000 on the remote (or local) registry host
# Then it should all just work with docker images tagged mywebproxy.myexample.org:5043/imagename[:tag]

Have fun.

This post has been created as it may assist others who want to do the same thing. It is not complicated or world shaking.

Often you will run Fedora (or CentOS/RHEL) servers and do not want a GUI desktop installed. However occasionally you may want to run a X-windows GUI application on that server.

This is normally done simply with a normal SSH connection with the remote server allowing X11Forwarding; if the remote server has all the desktop packages installed and boots into GUI mode there is no problem with that approach. However if you SSH into a remote server with “ssh -X user@remotehost” and get a message saying X11 forwarding failed or the command “echo $DISPLAY” returns nothing then congratulations, you have a perfectly normal server install… however you are not able to run GUI applications, which is fortunately easily fixed.

Before you do anything check your /etc/ssh/sshd_config and make sure you have uncommented ‘X11Forwarding yes’ and restarted sshd since doing so. If you had to make that change retry the ssh and echo command above again as while unlikely on a server install you may have had the needed packages all along. Only if you still get the errors do you need to continue reading this post.

It should also be noted that the “X11Forwarding yes” can be set on a per client basis rather than globally if you preffer, an example of that is commented out at the end of the /etc/ssh/sshd_config file on Fedora.

If you have been trying to get this working already you will have already noticed that all fedora server only installs already have a xorg-x11-server-Xorg package installed, for example F32 at the time of writing this post has “xorg-x11-server-Xorg-1.20.8-1.fc32.x86_64” installed by default on a server only install. Obviously this in insuffient to run X-applications across SSH or I would not be writing this post.

To get the required functionality working simply, on the server you wish to remotely connect to using ssh to run GUI apps simply

dnf -y install gdm
systemctl start gdm.service     # note:is enabled by default so will start after reboots

You can then session ssh into the server again, and ‘echo $DISPLAY’ will now have a value.

To test it works, while sshed onto that remote server, simply

sudo dnf -y install xterm
xterm

If you get an xterm window on your client machine with a prompt of the remote server you have it all working.

Important note: on the client machine you must have the X11 forwarding ports open in your firewall, at a minimum have port 6010 open on your client to accept the forwarded X11 session data.

Obviously installing the GDM package installs a few extra required files. Below is a list of all files installed on my server whan I needed to install the GDM package. As the server still boots up in non-GUI mode (unless for some reason you make the changes necessary to change the default runlevel) installed packages used by a desktop session such as pulseaudio and evolution do not start so there is no impact on the server other than the gdm service being started.

[root@xxxx X11]# dnf install gdm
Last metadata expiration check: 1:52:25 ago on Mon 13 Jul 2020 13:44:13.
Dependencies resolved.
=================================================================================================================
 Package                                    Arch        Version                Repository                   Size
=================================================================================================================
Installing:
 gdm                                        x86_64      1:3.36.2-2.fc32        updates                     561 k
Installing dependencies:
 accountsservice                            x86_64      0.6.55-2.fc32          fedora                      118 k
 accountsservice-libs                       x86_64      0.6.55-2.fc32          fedora                       93 k
 bluez-obexd                                x86_64      5.54-1.fc32            fedora                      196 k
 bolt                                       x86_64      0.9-1.fc32             updates                     203 k
 bubblewrap                                 x86_64      0.4.1-1.fc32           fedora                       51 k
 cheese-libs                                x86_64      2:3.34.0-3.fc32        fedora                      814 k
 clutter                                    x86_64      1.26.4-1.fc32          fedora                      1.1 M
 clutter-gst3                               x86_64      3.0.27-3.fc32          fedora                       87 k
 clutter-gtk                                x86_64      1.8.4-7.fc32           fedora                       48 k
 cogl                                       x86_64      1.22.8-1.fc32          updates                     495 k
 colord-gtk                                 x86_64      0.2.0-3.fc32           fedora                       32 k
 cups-pk-helper                             x86_64      0.2.6-9.fc32           fedora                       91 k
 enchant2                                   x86_64      2.2.8-1.fc32           fedora                       63 k
 evolution-data-server                      x86_64      3.36.3-1.fc32          updates                     2.2 M
 evolution-data-server-langpacks            noarch      3.36.3-1.fc32          updates                     1.4 M
 fdk-aac-free                               x86_64      2.0.0-3.fc32           fedora                      401 k
 flatpak-selinux                            noarch      1.6.4-1.fc32           updates                      23 k
 flatpak-session-helper                     x86_64      1.6.4-1.fc32           updates                      77 k
 geoclue2                                   x86_64      2.5.6-1.fc32           fedora                      137 k
 geoclue2-libs                              x86_64      2.5.6-1.fc32           fedora                       51 k
 geocode-glib                               x86_64      3.26.2-1.fc32          fedora                       72 k
 gjs                                        x86_64      1.64.3-3.fc32          updates                     380 k
 gnome-autoar                               x86_64      0.2.4-2.fc32           fedora                       56 k
 gnome-bluetooth                            x86_64      1:3.34.1-1.fc32        fedora                       44 k
 gnome-bluetooth-libs                       x86_64      1:3.34.1-1.fc32        fedora                      318 k
 gnome-control-center                       x86_64      3.36.3-1.fc32          updates                     5.7 M
 gnome-control-center-filesystem            noarch      3.36.3-1.fc32          updates                      12 k
 gnome-desktop3                             x86_64      3.36.3.1-1.fc32        updates                     576 k
 gnome-keyring-pam                          x86_64      3.36.0-1.fc32          fedora                       29 k
 gnome-online-accounts                      x86_64      3.36.0-1.fc32          fedora                      484 k
 gnome-session                              x86_64      3.36.0-2.fc32          fedora                      383 k
 gnome-session-wayland-session              x86_64      3.36.0-2.fc32          fedora                       13 k
 gnome-session-xsession                     x86_64      3.36.0-2.fc32          fedora                       13 k
 gnome-settings-daemon                      x86_64      3.36.1-1.fc32          updates                     1.0 M
 gnome-shell                                x86_64      3.36.4-1.fc32          updates                     1.5 M
 gsound                                     x86_64      1.0.2-11.fc32          fedora                       34 k
 gssdp                                      x86_64      1.0.4-1.fc32           updates                      51 k
 gupnp                                      x86_64      1.0.5-1.fc32           updates                      96 k
 gupnp-av                                   x86_64      0.12.11-3.fc32         fedora                       92 k
 gupnp-dlna                                 x86_64      0.10.5-12.fc32         fedora                       89 k
 harfbuzz-icu                               x86_64      2.6.4-3.fc32           fedora                       16 k
 hyphen                                     x86_64      2.8.8-13.fc32          fedora                       29 k
 ibus                                       x86_64      1.5.22-7.fc32          updates                     7.4 M
 ibus-gtk2                                  x86_64      1.5.22-7.fc32          updates                      28 k
 ibus-gtk3                                  x86_64      1.5.22-7.fc32          updates                      29 k
 ibus-libs                                  x86_64      1.5.22-7.fc32          updates                     262 k
 ibus-setup                                 noarch      1.5.22-7.fc32          updates                      61 k
 iio-sensor-proxy                           x86_64      3.0-1.fc32             fedora                       57 k
 libappindicator-gtk3                       x86_64      12.10.0-28.fc32        updates                      42 k
 libcanberra                                x86_64      0.30-22.fc32           fedora                       87 k
 libcanberra-gtk3                           x86_64      0.30-22.fc32           fedora                       32 k
 libdbusmenu                                x86_64      16.04.0-15.fc32        fedora                      136 k
 libdbusmenu-gtk3                           x86_64      16.04.0-15.fc32        fedora                       41 k
 libgdata                                   x86_64      0.17.12-1.fc32         fedora                      472 k
 libgee                                     x86_64      0.20.3-1.fc32          fedora                      289 k
 libgnomekbd                                x86_64      3.26.1-3.fc32          fedora                      163 k
 libgtop2                                   x86_64      2.40.0-3.fc32          fedora                      150 k
 libgweather                                x86_64      3.36.1-1.fc32          updates                     2.9 M
 libhandy                                   x86_64      0.0.13-4.fc32          updates                     162 k
 libical-glib                               x86_64      3.0.8-1.fc32           fedora                      187 k
 libimobiledevice                           x86_64      1.2.1-0.3.fc32         fedora                       79 k
 libindicator-gtk3                          x86_64      12.10.1-17.fc32        fedora                       67 k
 libmediaart                                x86_64      1.9.4-9.fc32           fedora                       44 k
 libnma                                     x86_64      1.8.28-1.fc32          updates                     302 k
 libnotify                                  x86_64      0.7.9-1.fc32           fedora                       43 k
 libplist                                   x86_64      2.1.0-3.fc32           fedora                       78 k
 libsbc                                     x86_64      1.4-5.fc32             fedora                       44 k
 libusbmuxd                                 x86_64      2.0.0-2.fc32           fedora                       38 k
 libvncserver                               x86_64      0.9.11-11.fc32         fedora                      272 k
 libwpe                                     x86_64      1.6.0-1.fc32           fedora                       26 k
 libxklavier                                x86_64      5.4-15.fc32            fedora                       67 k
 low-memory-monitor                         x86_64      2.0-4.fc32             fedora                       34 k
 mesa-vulkan-drivers                        x86_64      20.1.2-1.fc32          updates                     3.3 M
 mobile-broadband-provider-info             noarch      20190618-3.fc32        fedora                       67 k
 mozjs68                                    x86_64      68.10.0-1.fc32         updates                     6.8 M
 mutter                                     x86_64      3.36.4-1.fc32          updates                     2.4 M
 network-manager-applet                     x86_64      1.16.0-2.fc32          updates                     208 k
 nm-connection-editor                       x86_64      1.16.0-2.fc32          updates                     864 k
 ostree-libs                                x86_64      2020.3-5.fc32          updates                     398 k
 pipewire                                   x86_64      0.3.6-1.fc32           updates                     110 k
 pipewire-libs                              x86_64      0.3.6-1.fc32           updates                     708 k
 pipewire0.2-libs                           x86_64      0.2.7-2.fc32           fedora                      354 k
 pulseaudio                                 x86_64      13.99.1-4.fc32         updates                     1.0 M
 pulseaudio-module-bluetooth-freeworld      x86_64      1.4-1.fc32             rpmfusion-free-updates      100 k
 python3-cairo                              x86_64      1.18.2-4.fc32          fedora                       94 k
 python3-gobject                            x86_64      3.36.1-1.fc32          updates                      17 k
 rtkit                                      x86_64      0.11-23.fc32           fedora                       58 k
 sound-theme-freedesktop                    noarch      0.8-13.fc32            fedora                      378 k
 speexdsp                                   x86_64      1.2.0-1.fc32           updates                     453 k
 startup-notification                       x86_64      0.12-19.fc32           fedora                       42 k
 switcheroo-control                         x86_64      2.2-1.fc32             updates                      38 k
 upower                                     x86_64      0.99.11-3.fc32         fedora                      176 k
 vulkan-loader                              x86_64      1.2.135.0-1.fc32       updates                     126 k
 webkit2gtk3                                x86_64      2.28.3-1.fc32          updates                      15 M
 webkit2gtk3-jsc                            x86_64      2.28.3-1.fc32          updates                     6.0 M
 webrtc-audio-processing                    x86_64      0.3.1-4.fc32           fedora                      313 k
 woff2                                      x86_64      1.0.2-8.fc32           fedora                       61 k
 wpebackend-fdo                             x86_64      1.6.0-1.fc32           fedora                       36 k
 xdg-dbus-proxy                             x86_64      0.1.2-2.fc32           fedora                       43 k
 xdg-desktop-portal                         x86_64      1.7.2-2.fc32           updates                     434 k
 xdg-desktop-portal-gtk                     x86_64      1.7.1-1.fc32           fedora                      239 k
 xorg-x11-server-Xwayland                   x86_64      1.20.8-1.fc32          fedora                      988 k
 xorg-x11-xauth                             x86_64      1:1.1-3.fc32           fedora                       36 k
 xorg-x11-xinit                             x86_64      1.4.0-6.fc32           fedora                       56 k
 zenity                                     x86_64      3.32.0-3.fc32          fedora                      4.3 M
Installing weak dependencies:
 flatpak                                    x86_64      1.6.4-1.fc32           updates                     1.5 M
 gnome-remote-desktop                       x86_64      0.1.8-2.fc32           updates                      69 k
 libldac                                    x86_64      2.0.2.3-5.fc32         fedora                       41 k
 p11-kit-server                             x86_64      0.23.20-1.fc32         fedora                      189 k
 pinentry-gtk                               x86_64      1.1.0-7.fc32           fedora                       48 k
 rygel                                      x86_64      0.36.2-5.fc32          fedora                      1.0 M
 vino                                       x86_64      3.22.0-17.fc32         fedora                      453 k

Transaction Summary
=================================================================================================================
Install  113 Packages

Total download size: 81 M
Installed size: 344 M
Is this ok [y/N]: y

There are lots of complicated ways to install a local GitLab instance, however for small environments it is easiest to simply use the supplied container image. Instructions on installing that are at https://docs.gitlab.com/omnibus/docker/ and are simple to follow.

Install, start, configure

For my home lab environment installing the GitLab container was simply a case of

1. create a new KVM based on CentOS-8 with 4Gb of memory, 2 vcpus, and a 50Gb qcow2 disk
2. install docker

   dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
   dnf list docker-ce
   dnf install docker-ce --nobest
   usermod -aG docker root
   usermod -aG docker mark
   systemctl enable docker
   systemctl start docker
   

3. in /etc/profile add the required ‘export GITLAB_HOME=”/srv”‘ line (it does not have to be in the global profile but I did not want to have to add it in multiple places)
4. docker pull gitlab/gitlab-ce:latest
5. start it

   docker run --detach \
      --hostname gitlab-local.xxx.xxx.org \
      --publish 443:443 --publish 80:80 --publish 5522:22 \
      --name gitlab \
      --restart always \
      --volume $GITLAB_HOME/gitlab/config:/etc/gitlab:Z \
      --volume $GITLAB_HOME/gitlab/logs:/var/log/gitlab:Z \
      --volume $GITLAB_HOME/gitlab/data:/var/opt/gitlab:Z \
      gitlab/gitlab-ce:latest
   

   note that I use port 5522 for ssh as the server running the container is already using port 22 for it’s ssh server, also the :Z appended to the volume lines are because selinux is on my server

Then point a web browser at the ip-address of the server (http://xxx.xxx.xxx.xxx rather than https) and using the userid root create a new password for the GitLab root user. At that point you can configure a few of the options, such as not letting new users create their own accounts.

Create a normal non-admin user, and setup ssh keys

You will then want to create a non-root/non-admin user for day to day use for your projects.

When defining a new user GitLab wants to send an email to the new user to allow them to create their new password, by default the GitLab container does not support outbound email. If that is an issue for you documentation on setting up email is at https://docs.gitlab.com/omnibus/settings/smtp.html#example-configuration, however it is not required (see next paragraph).

After defining a new user you just “edit” the user, which allows you to set an initial password for the user, when they use it at first logon they are required to change it; so having email is not actually required to define a new user. If you are going to run the container in your home lab you probably do not want emails out of GitLab enabled anyway.

Once done log out from the root user.

Setup a ssh key for the new user, fully documented at https://docs.gitlab.com/ee/ssh/, but briefly covered in the following paragraphs.

Logon to the GitLab interface using the new user you created using the password set when you edited the user above. You will be required to change the password, and then logon again with the new password.
Once logged on go to the user settings and select ssh keys.

On a terminal window on your Linux development desktop use ssh-keygen -t ed25519 -C “some meaningful comment” to create a new ssh key, note that I changed the name of the default file when prompted from id_ed25519 to id_ed25519_local_gitlab so I can clearly identify it, this raises issues also covered below.

Once you have generated the key done copy the contents of the .pub file created for the key to your clipboard.

Paste the ssh key in your clipboard into the ‘Key’ field provided, give it a meaningfull title and add the new key.

Back in the terminal window test the key works with “ssh -T git@xxx.xxx.xxx.xxx”, or in my case as I mapped port 5522 “ssh -p 5522 -T git@xxx.xxx.xxx.xxx”; reply yes to accept the fingerprint; you should see a ‘Welcome to GitLab @username!’ message before being returned to your own hosts shell. Repeat to ensure the fingerprint has been accepted and prompts will not interfere with workflow anymore.

For those used to using ssh to jump around machines it is worth pointing out that the command is in fact git@xxx.xxx.xxx.xxx and not the username of the GitLab account; that can be confusing at a first glance but the userid used must be git, and it magicaly determines what username to map to based on the key used.

It should also be noted that by default one of the key files ssh looks for is ~/.ssh/id_ed25519 whch is the default filename used when generating the key. If like me you have many idendity key files discretely seperated the command for me is “ssh -p 5522 -i ~/.ssh/id_ed25519_local_gitlab -T git@xxx.xxx.xxx.xxx” to select the key I want to use.

Create a project as the new user

While logged on as the newly created user you may want to create groups, but if it is for a home lab environment you probably want to jump directly to starting a new project.

The main reason new users should create a new project via the browser interface is that when a new project is created a ‘help page’ for the new project is displayed showing the git commands needed to setup the git environment and create a new local repository on your development desktop by cloning the new empty project or pushing an existing git repository to your new project. It does assume you are using default ssh identity files however.

Workarounds needed for using a container

The documentation recomends mapping port 22:22 which if used means the host running the container would have to provide ssh services on another port; to me that is silly. So as noted in my example of running the container above I mapped 5522:22 so the host running the container can still provide normal ssh services on port 22. It does mean port 5522 (or whatever port you chose) needs to be opened in your firewall).

If you need to use git with multiple ssh identity files as I do there are a few interesting answers at https://superuser.com/questions/232373/how-to-tell-git-which-private-key-to-use.

I prefer the use of ~/.ssh/config myself to provide different keys to different hosts as that also allows the port to be changed. It does mean you need a seperate dns entry for the hostname and gitlab-hostname as you only want to switch ports when accesing the GitLab ssh port and not interfere with the host running the container providing its normal ssh service on port 22.

Assuming a /etc/hosts file entry as below (or dns entry where both hosts map to the same ip-address)

xxx.xxx.xxx.xxx gitlab-local.xxx.xxx.org realservername.xxx.xxx.org

Using the below example of a ~/.ssh/config file any ssh to gitlab-local.xxx.xxx.org will use the specified key in the users .ssh directory instead of the default plus use port 5522 to connect to the remote hosts ssh service. And ssh to any other host (including realservername.xxx.xxx.org) will (as no overrides have been added for the ‘host *’ catch-all entry) use the normal port and all default ssh parameters.

Host gitlab-local.xxx.xxx.org
  IdentityFile ~/.ssh/id_ed25519_local_gitlab
  Port 5522
Host *

Actually using your project from the standard git command line

Then on your development server to push files you are already working on to the project you created earlier is simply a case of following the git instructions that were shown on the web page displayed when your project was created (obviously it would be easier if you added a hostname to your /etc/hosts or dns rather than use the ip-address, but either work).

cd existing_folder
git init
git remote add origin git@xxx.xxxx.xxxx.xxxx:user_name_you_created/project_name_you_created.git
git add .
git commit -m "Initial commit"
git push -u origin master

And your Gitlab browser interface will show the files are in the project now.

To use any of the devops features you will need a bit more work, but for simply installing a local git based software repository users can work on that’s all that is needed.

So excluding the time it takes to download a CentOS-8 install iso, it takes less than 15mins to create a new CentOS-8 VM (remember to asign a static ip-addr), and about another 15mins to get it all working; so give it a try.