This post is based on the “queens” release of OpenStack available from the RDO distribution site.
It is based upon the documentation available at https://www.rdoproject.org/install/packstack/, but modified to install the services I wish to play with which are not installed by the default allinone packstack install, plus I need to add a second compute node.
It is also important to note this post is from 2018/07/25 and the environment is constantly changing, for example
- at 2018/05/15 trying to install container support made the environment un-usable, but now it installs
- at 2018/05/15 there was a lot of manual effort needed to get console support for instances on the second compute node, now console support on the second compute node works ‘out of the box’
The differences between the two dates were discovered as I had to re-install, oops :-(. I had to re-install and discovered the two new ‘improvements’. Immediately prior to that need to re-install I also found there was no reliable documentation on how to use mariadb commands to delete a obsolete compute node, the latest seem to be those that worked well for ocata but not for queens… although the actual error was no cell entries found on a compute node after a new compute node did appear to successfully autoconnect/discover. Anyway, I re-installed.
Packstack by default builds and all-in-one environment, if that is what you want just follow the documentation link mentioned above. It will not install container and heat support so if you want those you may want to read on to see how to change the answers file anyway.
This post is because origionally I wanted to investigate how instances launched on different compute hosts could communicate with each other in their private network range using OpenVSwitch… which obviously needs at least two hosts. And as time goes on I find I need the second compute node, but that is way at the end of the post.
Origionally I used OpenVSwitch across a second ethernet device on each server using a non 192.168.1.0/24 network… which worked OK but added another level of complexity and was not really required; so for simplicity this walkthrough uses only one network card on each server and both cards on on my internal network 192.168.1.0/24 network. Mentioned as a lot of documentation recomends you do it using a private network for OpenVSwitch to communicate between launched cloud instances and if you have a large lab by all means do so, but I am simplifying my home network and this post reflects that… it also makes the post a lot smaller :-).
You will also note during this walkthrough that there is a lot of rebooting and editing of the configuration file and rerunning the install, this is required, using packstack to generate an answers file and attempting to customise that answers file to setup everything you want in one install step will consistently fail and require restarting from the ‘create two new VMs’ step.
This post covers a sucessful install perfomed on a standard intel 4xdual core (8 threads) desktop with 32Gb of memory installed; with a 1Tb spinning disk which is more than enough. The two KVM instances are allocated a total of 23Gb between them which is enough for a well responding working installation. A desktop with these specs can easily be obtained from a custom PC builder for under $1200NZ if ordered without an OS (ie: no expensive windoze) and you install linux.
Step 1 – Build two CentOS7 VMs
You need to create two (or more) CentOS7 virtual machines, both with the minimum package install to keep the footprint small.
-
region1server1 : static ip-address, 15Gb ram (minimum), virtual disk a minimum of 50Gb (a lot larger if you will be taking snapshots). This will be used as the control and network node server
as it is the control node it will also host all the disk images to be deployed and the snapshots, plus if you are using swift storage packstack will create a 20Gb (default) filesystem on this disk, so make it larger not smaller :-).
-
region1server2 : static ip-address, 8Gb ram (more if you plan to run a lot of instances), virtual disk a minimum of 30Gb or more. This will be the compute node.
As it is a virtual machine you can easily shut it down and reconfigure it with more memory later if you need… but the disk space allocated will need to be enough to hold the virtual disk images of the machines you launch on that compute node so you may want to bump disk space up.
- On both you should allocate swap space; I use swapfiles so they can easily be adjusted however from experience once swapping starts to occur everything will grind to a halt so use swapspace as an indicator only… if it is used allocate more real memory to the VMs as soon as possible.
Note that I am using a 192.168.1.0/24 network with 192.168.1.172 as region1server1 (all services) and 192.168.1.162 as region1server2 (a dedicated compute node). The external gateway for both must be a valid gateway as you need the two servers to access the inetrnet to download packages.
At this pont just check the configuration files /etc/sysconfig/network-scripts/ifcfg-eth0 (or ifcfg-en0 depending on what sort of network adapter you configured). You should expect to see something like the below.
On server1
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
PEERDNS="yes"
PEERROUTES="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_FAILURE_FATAL="no"
NAME="eth1"
HWADDR=52:54:00:34:3A:E7
DEVICE="eth0"
ONBOOT="yes"
IPADDR=192.168.1.172
NETMASK=255.255.255.0
BROADCAST=192.168.1.255
GATEWAY=192.168.1.1
DNS1=192.168.1.172
DNS2=192.168.1.1
On Server2
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
NAME="eth0"
HWADDR=52:54:00:7F:56:85
DEVICE="eth0"
ONBOOT="yes"
DNS1="192.168.1.172"
DNS2="192.168.1.1"
IPADDR="192.168.1.162"
PREFIX="24"
GATEWAY="192.168.1.1"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_PRIVACY="no"
You need to switch from using NetworkManager to network (as that is documented as needing to be done for the allinone install), and while the servers are still at a minimal software install this is probably a good time to ensure all the packages installed are up to date. Note that this is recomended from the install documentation, the packstack install does seem to specifically check and configure NetworkManager to be enabled so a bit of a conflict there, but do this for the install anyway.
Also note that firewalld must be disabled. OpenStack uses some rather complex iptables rules and we do not want firewalld getting in the way. net-tools and fuser I always install for trouble-shooting.
As the root user
systemctl disable firewalld
systemctl stop firewalld
systemctl disable NetworkManager
systemctl enable network
yum -y update
yum -y install net-tools psmisc # for ifconfig and fuser
yum -y install screen # needed if running packstack on a non-console session
This would be a good time to reboot both servers to both pick up the latest packages just installed and to make sure the interface on each server comes up with the correct address now you are using “network” instead of NetworkManager; especially as a rather major change to the network configuration will be made a little later and you need to be confident both virtual servers are networked correctly before getting to that point.
After rebooting, redo the steps to disable network manager, because the update re-enabled NetworkManager which we do not want.
systemctl disable NetworkManager
systemctl enable network
Step 2 – Add RDO repositories and configure OpenVSwitch
OpenVSwitch is not available from the CentOS7 repositories and needs to be installed fromthe RDO repositories, so the network configuration changes needed for OpenVSwitch are done at this step not in the prior step when the servers were built. We also want to pick up any packages from the RDO repository that may replace the CentOS7 ones.
As root on both virtual servers, and yes you do need the yum update again as the openstack repository may need to replace CentOS packages.
yum install -y centos-release-openstack-queens
yum -y update
yum -y install openvswitch
You now need to only on the region1server1 server cd to /etc/sysconfig/network-scripts. You will replace the ifcfg-eth0 shown above with an updated ifcfg-eth0 and a new ifcfg-br-ex (for the openvswitch bridge). So edit/create the files on the first virtual server similar to the below… using your machines HWADDR of course.
The ifcfg-eth0 will now look like
DEVICE="eth0"
BOOTPROTO="none"
TYPE="OVSPort"
OVS_BRIDGE="br-ex"
ONBOOT="yes"
DEVICETYPE=ovs
HWADDR-52:54:00:34:3A:E7
The ifcfg-br-ex will look like
TYPE="OVSBridge"
DEVICE="br-ex"
BOOTPROTO="static"
DEVICETYPE=ovs
ONBOOT="yes"
IPADDR="192.168.1.172"
PREFIX0="24"
GATEWAY="192.168.1.1"
DNS1="192.168.1.172"
DNS2="192.168.1.1"
DEFROUTE="yes"
PEERDNS="yes"
PEERROUTES="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_FAILURE_FATAL="no"
USERCTL=yes
After the changes reboot the region1server1 first virtual server; the br-ex interface will now have the servers ip-address assigned and bound to eth0.
[root@region1server1 ~]# ovs-vsctl show
dc3364e6-b4bc-4702-9df0-48cb60e8abcd
Bridge br-ex
Port "eth0"
Interface "eth0"
Port br-ex
Interface br-ex
type: internal
ovs_version: "2.9.0"
Also reboot the second virtual server as we need it to be restarted with the openvswitch package available for use.
The br-ex interface only needs to be created on the first server as that is the server that will be providing the networking functions to all the compute nodes.
Step 3 – Add hosts entries
This is a seperate step to highlight that it must be done, unless your two new virtual servers are in a DNS zone. When packstack installs rabbitmq it must be able to resolve an ip-address to a hostname or the install will fail. I have been caught out by that many times.
In the /etc/hosts file on both virtual servers add an entry for the host and the second host; server1 and server2 must be able to resolve their own names and the other servers name. Use fully qualified domain names, not short names.
Thats so obvious to edit I won’t waste space with pasting an example here.
Step 4 – Install packstack, build allinone, then use answers file, repeatedly
The actual packstack commands should be run from a screen bash session (yum -y install screen;screen bash) as if you need to run it more than once packstack activities reset the network connection so if you are running it from a non-screen ssh session you will get the command killed. Using screen you can just ssh back into the server and use “screen -r” to re-attach to the running packstack command session.
Also I have found that the changes to the configuration file do need to be done one at a time with reboots between them. While it seems reasonable that you could just change all the values in the configuration file in one edit and then run packstack from experience that will not work. The steps below do work, for me anyway.
On the first server only we need to install packstack and build the allinone environment using all the defaults.
As root on the first server only
yum install -y openstack-packstack
packstack --allinone
This should complete successfully, when done reboot.
Log back in, in the root home directory there will be an answers file created, names similar to packstack-answers-20180514-162925.txt; copy that aside to something like answers1.txt which will be the configuration file name we will be using from this point.
Edit the file, set CONFIG_HEAT_INSTALL to y
As root
packstack --answer-file=answers1.txt
This should complete successfully, when done reboot.
Log back in and edit your answers file again
Change the entry CONFIG_COMPUTE_HOSTS !. We will be using our second server as an additional compute host so I change CONFIG_COMPUTE_HOSTS=192.168.1.172 to be CONFIG_COMPUTE_HOSTS=192.168.1.172,192.168.1.162 and yes I do want the control host to be included as a compute host at this time
As root
packstack --answer-file=answers1.txt
This should complete successfully, when done reboot both VMs.
Log back in and edit your answers file again
Change the admin password entry to something you can remember, ie: CONFIG_KEYSTONE_ADMIN_PW=password
Change the mariadb password likewise, ie: CONFIG_MARIADB_PW=password
As root
packstack --answer-file=answers1.txt
This should complete successfully, when done reboot both VMs.
Edit /etc/neutron/plugins/*/ml2_conf.ini, change mechanism_drivers=openvswitch to mechanism_drivers=openvswitch,l2population then reboot both servers again.
And it is done, you have a working system with two compute nodes with heat installed… but no container infrastructure.
As of 2018/07/25 change the answers file to set CONFIG_MAGNUM_INSTALL=n to =y and rerun the packstack command to install container support. (Important: at 2018/05/15 installing container support at any point using packstack consistently caused the install to fail and both VMs to be basically un-usable… this now appears to be fixed but try at your own risk).
Step 5 – Customise, using the dashboard to make it easier
Log onto the dashboard as the admin user using the password you set in the configuration file, or if you did not set one it will be in the file keystonerc_admin in the root directory of the server you ran the packstack install from.
- only one external network seems to be supported. Use the dashboard to display (and note down) the details of the external network created by the default “demo” environment (specifically the external device name, normally “extnet”)… which is a 172.something network which we don’t want anyway
- you cannot delete that external network while it is referenced anywhere; delete the demo user, demo project, the network router, then the external network.
- Then create a new external network (external network and shared must be ticked, actually tick all boxes), in my environment that is a flat network using the “extnet” with no dhcp allowing an ip range of 192.168.1.240,192.168.1.250 (for floating ips) which is outside the range used by my physical network currently.
- Add a new user for yourself, as part of adding the new user you are also able to create a new project for the user. The user must be an admin user if it is to be permitted to associate floating ips to instances
Log onto the dashboard as the new user you have created.
- create a new non-shared network for that tenant user (ie: 10.0.1.0/24 with a ip range of 10.0.1.10,10.0.1.250 ensuring you do not use the full range of .1 to .254 or network creation will fail as it needs a few addresses free for gateway etc.
- create a router for that tenant user attached to the “shared” external network you created above (if it is not visible as an option you forgot to make it shared)
- examine the router details and under the interfaces section select add a new interface, attach your new personal network you have just created
- Under the network tab there will be an option for security group rules, create a new rule to allow ssh and all ICMP inbound. When you launch instances if you need to troubleshoot them (or want to ssh into them or ping them) you can add this rule as an additional rule when you launch an instance. Note: rules can be added/removed from running instances if you do not want to do this at instance launch time
At this point everything should just work for you… with one extra step to be aware of that was mentioned above. You may launch an instance, assign a floating ip, and not be able to ping it from your external network. Thats normal, the default network security group rules do not allow inbound traffic. You will need a custom security group like that mentioned above to permit that.
Additional customisations I always do
- Launch an instance as my tenant user to use as a gateway (I use a F25 cloud image as it only needs a 3Gb disk and 512Mb memory). It will launch by default on the main region1server1 (15Gb) machine which it thinks has the most free memory. Then go to the admin/hypervisor option and disable the region1server1 compute node.leaving only the second compute node available for new instances. The reason for that is that region1server1 will now have used almost all of its 15Gb of memory (top shows me 440k free on that machine now) but openstack does not take into account anything other than instances launched by itself so still thinks there is 14Gb+ free on that server to launch instances into which will obviously cause problems if left enabled. I want my gateway on that server as that server provides the networking to other compute nodes, so is the logical place to put it as if that region1server1 server is down no instances could be accessed anyway
- assign a floating ip address to that gateway instance. For linux users the reason is obvious. On all linux desktops that may want to work with instances just add a route to that gateway. For example if the floating ip assigned was “192.168.1.246” then the command “route add -net 10.0.1.0/24 gw 192.168.1.246” would allow those linux desktops to ssh (or anything else a port was opened for) to instances launched on my 10.0.1.0/24 network… or simply put all instances on the 10.0.1.0/24 network can be access via one floating ip address; which is why my floating ip address range assigned in the examples above was so small, for my personal home use I only need one, maybe 2
- “if you use nagios; as the root user on both CentOS7 VMs yum -y install epel-release” then “yum -y install nrpe nagios-plugins-all”, then “systemctl enable nrpe”, then create a script that runs after all the openstack services have started and created their iptables rules “iptables -I INPUT -p tcp –dport 5666 -m conntrack –ctstate NEW,ESTABLISHED -j ACCEPT” to allow your nagios server to run nrpe checks on the two VMs
Known remaining issues after install
- when trying to setup networks via the web interface the availability zone “nova” seems to be hard coded as the only option even if additional availability zones have been created; this is a change as in Ocata (the last release I had stable) manually created availability zones could be used
- when creating users please note that only users created with the “admin” role can associate floating ips with instances, this is a change from earlier releases. There is discussion on whether this is a bug or lack of documentation at https://bugs.dogfood.paddev.net/horizon/+bug/1394034 should you be curious
