This is for the community edition of puppetserver, although will work for all recent puppet releases.
I use puppetserver as it is supposed to be able to manage up to 100 agents without having to hook it up to a web-server such as apache without any issues, and I have nowhere near that many servers or VMs in my home lab. At the time of this post it is running on a CentOS7 VM with agents running on CentOS7, CentOS8, Fedora32 and Fedora31.
It should also be noted this has been posted primarily for my use, as I needed the info again and it took me a while to remeber where I had placed the documentation; so posting it so I can just search my blog as I need it.
I initially investigated this as I had an interesting issue where on a puppet master server a “puppet cert list –all” did not show a cert for one of the agent nodes, but the agent node had no problem updating from the puppet master server. It turns out that ‘cleaning’ a cert does not stop an agent node updating, as the node still has a signed cert. You have to revoke it on the puppet master and leave it there rather than clean (remove) it from the puppet master to stop an agent updating.
So to stop a node updating just revoke.
For my initial problem I just think the agent server in question had another nodes certificate somehow; but thats another issue.
Anyway I was looking for the documentation a second time because after a full OS upgrade of an agent the agent was unable to retrieve files from the puppet master; it seems to connect OK and handshake OK but the errors were simply that it was unable to access file xxx where xxx was all the files that were relevant to that agent server. The solution in most cases where agent/master communication hits bumps is to generate a new certificate for the agent server.
It took me almost 10mins to find the documentation (after searching my blog first and not finding it). Therefore this blog post so I can find it easily next time by simply searching the blog, and there will be a next time.
To get a new certiificate for an agent follow the steps below
To generate a new certificate for an agent node you must delete all existing certificates for the agent on the agent and master.
- On the puppet agent as the root user, see the paragraph below the list of steps
- systemctl stop puppet
- puppet config print | grep ssldir
- cd to the directory the ssl certificates are stored in identified by the ssldir entry
- rm -rf *
- On the puppet master
- puppet cert clean youragenthostname
- On the puppet agent
- systemctl start puppet
- On the puppet master
- use ‘puppet cert list’ occasionally to wait for the signing request, and ‘puppetcert sign youragenthostname’ when it eventually appears
The reason the work on the agent must be done as the root user is simply because that is the correct environment to work in. If you were to run the ‘puppet print config’ command as a non-root user you would see all the directory paths as $HOME/username/.puppetlabs’ instead of the directories used by the puppet agent itself, in most cases paths under the user home directory would not exist.
If you have a complex environment where you do run copies of the puppet agent under multiple userids (pointless as only the copy running under root can do updates to system files), presumably overriding the hostname, as a way of targeted testing; the you have missed the point of having the puppet master able to serve multiple environments (production, test1, test2) etc. and I would not try to do such a thing. So as this post is to remind me how to fix my environment you are on your own sorting that out should you have gone down the path of running agents under non-root userids.
Special notes on cleaning up the master
Puppet does a good job of preventing you from messing up the certificates on the master, but if you try hard enough you can. Resolving that is similar.
- On the puppet master as root
- systemctl stop puppetserver
- puppet config print | grep ssldir
- cd to the directory the ssl certificates are stored in
- rm -rf *
- systemctl start puppetserver
- On every agent server follow the steps to be taken on the agent server mentioned in obtaining a new certificate above
When the puppetserver is started again it will recreate the keys it needs to function. The reason you must generate new certificates for every agent is simply because you will have deleted the signing key from the puppet master and it will have created a new one when it started, so all agent certificates are invalid and will error when trying to use the puppet master, they must be recreated so they can be signed with the new key.
It is certainly possible to selectively delete certificates on the puppet master to ensure the signing key remains, do not do that. If there is an issue on your puppet master severe enough to require manually deleting ssl keys go for the ‘big bang’ and wipe them all, trying to fiddle about selectively could leave you in a worse unknown position, simpler to regain a consistent environment by deleting them all and effectively starting from scratch.
Depending on how many agents you have you can configure the master to ‘autosign’ tne new certificate requests coming in from the agents; however as you will be manually working through deleting old certificated from the agent servers one by one I see no benefit in that.
UPDATE: 8Jan2022 – a bit of a late update as the change was a wile ago, but please note that more recent versions of puppet have retired the ‘puppet cert’ facility on the puppetserver/master and the puppet command is no longer used to manage puppetserver certificates, you must use puppetserver; you now need to use ‘puppetserver ca list [–all]’ and ‘puppetserver ca clean –certname=xxxx’ or ‘puppetserver ca sign –certname=xxxx’ on the puppetserver machine rather than the ‘puppet cert’ commands from the origional post. All else remains the same.
