# Table of Contents

- [What is OpenStack Cluster Installer (OCI)](#what-is-openstack-cluster-installer-oci)
  * [General description](#general-description)
  * [What OpenStack services can OCI install?](#what-openstack-services-can-oci-install)
  * [Who initiated the project? Who are the main contributors?](#who-initiated-the-project-who-are-the-main-contributors)
- [How to install your puppet-master/PXE server)](#how-to-install-your-puppet-masterpxe-server)
  * [Minimum requirements](#minimum-requirements)
  * [Installing the package](#installing-the-package)
  * [Installing side services](#installing-side-services)
  * [Getting ready to install servers](#getting-ready-to-install-servers)
- [Using OCI](#Using OCI)
  * [Booting-up servers](#booting-up-servers)
  * [Creating Swift regions, locations, networks, roles and clusters](#creating-swift-regions-locations-networks-roles-and-clusters)
  * [Real certificate for the API](#real-certificate-for-the-api)
  * [Enrolling servers in a cluster](#enrolling-servers-in-a-cluster)
  * [Calculating the Swift ring](#calculating-the-swift-ring)
  * [Installing servers](#installing-servers)
  * [Checking your installation](#checking-your-installation)
  * [Enabling Swift object encryption](#enabling-swift-object-encryption)
  * [Initial cluster setup variable](#initial-cluster-setup-variable)
  * [Adding other types of nodes](#adding-other-types-of-nodes)
- [Advanced usage](#advanced-usage)
  * [Using automated IPMI address configuration](#using-automated-ipmi-address-configuration)
  * [Automatic upgrade of BIOS and IPMI firmware](#automatic-upgrade-of-bios-and-ipmi-firmware)
  * [Customizing the /etc/hosts of all your cluster](#customizing-the-etchosts-of-all-your-cluster)
  * [Customizing the ENC](#customizing-the-enc)
  * [Customizing installed server at setup time](#customizing-installed-server-at-setup-time)
  * [Using a BGP VIP](#using-a-bgp-vip)
  * [Doing a test in OCI's manifests for debug purpose](#doing-a-test-in-ocis-manifests-for-debug-purpose)
  * [Customizing files and packages in your servers](#customizing-files-and-packages-in-your-servers)
  * [Once deployment is ready](#once-deployment-is-ready)
  * [Fixing-up the controllers](#fixing-up-the-controllers)
  * [Adding custom firewall rulles](#adding-custom-firewall-rules)
  * [Adding compute nodes](#adding-compute-nodes)
  * [Adding GPU support in a compute node](#adding-gpu-support-in-a-compute-node)
  * [Multiple Cinder LVM backends](#multiple-cinder-lvm-backends)
- [Managing the OpenStack deployment](#managing-the-openstack-deployment)
  * [Enabling cloudkitty rating](#enabling-cloudkitty-rating)
  * [Installing a first OpenStack image](#installing-a-first-OpenStack-image)
  * [Setting-up networking](#setting-up-networking)
  * [Adding an ssh key](#adding-an-ssh-key)
  * [Creating flavor](#creating-flavor)
  * [Boot a VM](#boot-a-vm)
  * [Add billing](#add-billing)
  * [Add Octavia service](#add-octavia-service)
  * [Setting-up no limits for services resources](#setting-up-no-limits-for-services-resources)
  * [Add Magnum service](#add-magnum-service)
- [Upgrading](#upgrading)

# What is OpenStack Cluster Installer (OCI)

### General description

OCI (OpenStack Cluster Installer) is a software to provision an OpenStack
clusters automatically. This package installs a provisioning machine, which
uses the below components:
- a DHCP server (isc-dhcp-server)
- a PXE boot server (tftp-hpa)
- a web server (apache2)
- a puppet-master

Once computers in the cluster boot for the first time, a Debian live system
is served by OCI over PXE, to act as a discovery image. This live system then
reports the hardware features back to OCI. Computers can then be installed with
Debian from that live system, configured with a puppet-agent that will connect
to the puppet-master of OCI. After Debian is installed, the server reboots, and
OpenStack services are provisionned, depending on the server role in the cluster.

OCI is fully packaged in Debian, including all of the Puppet modules. After
installing the OCI package and its dependencies, no other artificat needs to be
installed on your provisioning server, meaning that if a local debian mirror
is available, the OpenStack cluster installation can be done completely
offline.

### What OpenStack services can OCI install?

Currently, OCI can install:
- Swift (with optional dedicated proxy nodes)
- Keystone
- Cinder (LVM or Ceph backend)
- Glance (File, Swift or Ceph backend, Swift can be external)
- Heat
- Horizon
- Nova (with GPU support)
- Neutron
- Barbican
- Octavia
- Telemetry (Ceilometer, Gnocchi, Panko, Aodh)
- Cloudkitty

There's currently ongoing effort to integrate:
- Magnum
- Designate

All of this in a high availability way, using haproxy and corosync for
the controller nodes for all services.

All services are fully using TLS, even within the cluster.

As a general rule, what OCI does, is check what type of nodes are part
of the cluster, and takes decisions depending on it. For example, if there
are some Ceph OSD nodes, OCI will use Ceph as a backend for Glance, Nova
and Cinder backup.
If there are some Cinder Volume nodes, OCI will use them with the LVM
backend. If there is some Swiftstore nodes, Swift will be used for backups
and Glance images. If there are some Ceph OSD nodes, but
no dedicated Ceph MON nodes, the controllers will act as Ceph monitors.
If there are some Compute nodes, then Cinder, Nova and Neutron will be
installed on the controller nodes. Etc.

The minimum number of controller nodes is 3, though it is possible, with
a bit of hacking to install the 3 controllers on VMs on a single server
(of course, loosing the high availability feature if the hardware fails).

### Who initiated the project? Who are the main contributors?

OCI has been written from scratch by Thomas Goirand (zigo). The work is
fully sponsored by Infomaniak Networks, who is using it in production
in reasonably large clusters. There's been some sporadic contributions
within Infomaniak, plus a few patches from external contributors, but
no major features (yet). Hopefully, this project, over time, will gather
more contributors.

# How to install your puppet-master/PXE server

## Minimum requirements

OCI itself will run fine with about 20 GB of HDD, and a few GB of RAM.
However, to install OpenStack, you will need at least 3 controllers with
a minimum of 16 GB of RAM, 32 GB is recommended, and best is with 64 GB of RAM.
If you want Ceph, a minimum of 3 Ceph OSD is needed, however, we're only
talking when your cluster reaches 100 disks. The Ceph recommendation is
that any given server down doesn't remove more than 10% of the total
capacity. So 10 OSD servers as a start is nice. As for swift, the minimum
number of servers would be 3, but then if one fails, you'll get some timeouts.
So probably it's best to start with at least 6 Swift storage nodes, and maybe
at 2 proxies. For a the other resources, it's really up to you: a few
computes, and probably 2 network nodes and some volume nodes.

If you intend to run the openstack-cluster-installer-poc package to do some
OCI development in a vritualized environment, we recommend a single server
with 1 TB of HDD and 256 GB of RAM. This configuration is enough to
provision 19 VMs where OpenStack will be installed. It's possible to run
with less, but then not a lot of nodes will be available, and you'll have
to tweak down the number of servers in /etc/oci-poc/oci-poc.conf.

## Installing the package

### The package repository

The package is either available from plain Debian Sid/Buster, or from the
OpenStack stable backport repositories. If using Buster with OpenStack
Train is desired, then the below repository must be added to the sources.list file:

```
deb http://buster-train.debian.net/debian buster-train-backports main
deb-src http://buster-train.debian.net/debian buster-train-backports main

deb http://buster-train.debian.net/debian buster-train-backports-nochange main
deb-src http://buster-train.debian.net/debian buster-train-backports-nochange main
```

You may replace buster above by whatever Debian stable distro of the day
(as of writting, only Stretch and Buster are supported), and train by
whatever OpenStack release name of the day (from Rocky to Ussuri right now,
but as you know, we release Debian packages for OpenStack every 6
months...).

The repository key is also available this way:

```
apt-get update
apt-get install --allow-unauthenticated -y openstack-backports-archive-keyring
apt-get update
```

There's also a mirror containing ALL of the OpenStack releases in a single
place, located at:

http://osbpo.debian.net/debian/

### Install the package

Simply install the package:

```
apt-get install openstack-cluster-installer
```

### Install a db server

MariaDB will do:

```
apt-get install mariadb-server dbconfig-common
```

It is possible to the db creation and credentials by hand, or to let OCI handle
it automatically with dbconfig-common. If APT is running in
non-interactive mode, or if during the installation, the user doesn't ask
for the automatic db handling by dbconfig-common, here's how to create the
database:

```
apt-get install openstack-pkg-tools
. /usr/share/openstack-pkg-tools/pkgos_func
PASSWORD=$(openssl rand -hex 16)
pkgos_inifile set /etc/openstack-cluster-installer/openstack-cluster-installer.conf database connection mysql+pymysql://oci:${PASSWORD}@localhost:3306/oci"
mysql --execute 'CREATE DATABASE oci;'
mysql --execute "GRANT ALL PRIVILEGES ON oci.* TO 'oci'@'localhost' IDENTIFIED BY '${PASSWORD}';"
```

One must then make sure that the "connection" directive in
/etc/openstack-cluster-installer/openstack-cluster-installer.conf doesn't
contain spaces before and after the equal sign. Then the db is populated
below.

### Configuring OCI

Make sure the db is in sync (if it is, you'll see table exists errors):

```
apt-get install -y php-cli
cd /usr/share/openstack-cluster-installer ; php db_sync.php
```

Then edit /etc/openstack-cluster-installer/openstack-cluster-installer.conf
and make it looks the way it pleases you (ie: change network values, etc.).

### Generate the OCI's root CA

To handle TLS, OCI is using its own root CA. The root CA certificate is
distributed on all nodes of the cluster. To create the initial root CA,
there's a script to do it all:

```
oci-root-ca-gen
```

At this point, you should be able to browse through OCI's web interface:
```
firefox http://your-ip-address/oci/
```

However, you need a login/pass to get in. There's a shell utility to manage
your usernames. To add a new user, do this:

```
oci-userdb -a mylogin mypassword
```

Passwords are hashed using the PHP password_hash() function using the
BCRYPT algo.

Also, OCI is capable of using an external Radius for its authentication.
However, you still need to manually add logins in the db. What's bellow
inserts a new user that has an entry in the radius server:

```
oci-userdb -r newuser@example.com
```

Note that you also need to configure your radius server address and
shared secret in openstack-cluster-installer.conf.

Note that even if there is an authentication system, it is strongly advised
to not expose OCI to the public internet. The best setup is if your
provisionning server isn't reachable at all from the outside.

## Installing side services

### ISC-DHCPD

Configure isc-dhcp to match your network configuration. Note that
"next-server" must be the address of your puppet-master node (ie: the dhcp
server that we're currently configuring).

Edit /etc/default/isc-dhcpd:

```
sed -i 's/INTERFACESv4=.*/INTERFACESv4="eth0"/' /etc/default/isc-dhcp-server
```

Then edit /etc/dhcp/dhcpd.conf:

```
allow booting;
allow bootp;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
ignore-client-uids On;

subnet 192.168.100.0 netmask 255.255.255.0 {
        range 192.168.100.20 192.168.100.80;
        option domain-name example.com;
        option domain-name-servers 9.9.9.9;
        option routers 192.168.100.1;
        option subnet-mask 255.255.255.0;
        option broadcast-address 192.168.100.255;
        next-server 192.168.100.2;
        if exists user-class and option user-class = "iPXE" {
                filename "http://192.168.100.2/oci/ipxe.php";
        } else {
                filename "lpxelinux.0";
        }
}
```

Carefully note that 192.168.100.2 must be the address of your OCI server,
as it will be used for serving PXE, TFTP and web for the slave nodes.
It is of course fine to use another address if your OCI server does,
so feel free to adapt the above to your liking.

Note that as of OCI version 28 and above, loading the initrd and kernel
is done over HTTP, so using lpxelinux.0 is mandatory (pxelinux.0 should
not be used anymore, as it only supports TFTP).

Also, for OCI to allow query from the DHCP range, you must add your
DHCP subnets to TRUSTED_NETWORKS in openstack-cluster-installer.conf.
Otherwise, hardware reporting will never work.

### tftpd

Configure tftp-hpa to serve files from OCI:

```
sed -i 's#TFTP_DIRECTORY=.*#TFTP_DIRECTORY="/var/lib/openstack-cluster-installer/tftp"#' /etc/default/tftpd-hpa
```

Then restart tftpd-hpa.

## Getting ready to install servers

### Configuring ssh keys

When setting-up, OCI will create a public / private ssh keypair in here:

```
/etc/openstack-cluster-installer/id_rsa
```

Once done, it will copy the corresponding id_rsa.pub content into:

```
/etc/openstack-cluster-installer/authorized_keys
```

and will also add all the public keys it finds under
/root/.ssh/authorized_keys in it. Later on, this file will be copied
in the OCI Debian live image, and in all new systems OCI will install.
OCI will later on use the private key it generated to log into the
servers, while your keys will also be present so you can log into each
individual servers using your private key. Therefore, it is strongly
advise to customize /etc/openstack-cluster-installer/authorized_keys
*before* you build the OCI Debian Live image.

### Build OCI's live image ###

```
mkdir -p /root/live-image
cd /root/live-image
openstack-cluster-installer-build-live-image --pxe-server-ip 192.168.100.2 --debian-mirror-addr http://deb.debian.org/debian --debian-security-mirror-addr http://security.debian.org/
cp -auxf /var/lib/openstack-cluster-installer/tftp/* /usr/share/openstack-cluster-installer
cd ..
rm -rf /root/live-image
```

Is is possible to use package proxy servers like approx,
or local mirrors, which gives the possibility to have your cluster
and OCI itself completely disconnected from internet.

### Configure puppet's ENC

Once the puppet-master service is installed, its external node
classifier (ENC) directives must be set, so that OCI acts as ENC
(which means OCI will define roles and puppet classes to call when
installing a new server with puppet):

```
. /usr/share/openstack-pkg-tools/pkgos_func
pkgos_add_directive /etc/puppet/puppet.conf master "external_nodes = /usr/bin/oci-puppet-external-node-classifier" "# Path to enc"
pkgos_inifile set /etc/puppet/puppet.conf master external_nodes /usr/bin/oci-puppet-external-node-classifier
pkgos_add_directive /etc/puppet/puppet.conf master "node_terminus = exec" "# Tell what type of ENC"
pkgos_inifile set /etc/puppet/puppet.conf master node_terminus exec
```

then restart the puppet-master service.

### Optional: approx

To speed-up package download, it is highly recommended to install approx
locally on your OCI provisionning server, and use its address when
setting-up servers (the address is set in
/etc/openstack-cluster-installer/openstack-cluster-installer.conf).

# Using OCI

## Booting-up servers

Start-up a bunch of computers, booting them with PXE. If everything goes well, they will
catch the OCI's DHCP, and boot-up OCI's Debian live image. Once the server
is up, an agent will run to report to OCI's web interface. Just refresh
OCI's web interface, and you will see machines. You can also use the CLI
tool:

```
# apt-get install openstack-cluster-installer-cli
# ocicli machine-list
serial   ipaddr          memory  status     lastseen             cluster  hostname
2S2JGM2  192.168.100.37  4096    live       2018-09-20 09:22:31  null
2S2JGM3  192.168.100.39  4096    live       2018-09-20 09:22:50  null
```

Note that ocicli can either use a login/password which can be set in
the OCI's internal db, or the IP address of the server where ocicli runs can
be white-listed in /etc/openstack-cluster-installer/openstack-cluster-installer.conf.

## Creating Swift regions, locations, networks, roles and clusters

### Before we start

In this documentation, everything is done through the command line using
ocicli. However, absolutely everything can also be done using the web
interface. It is just easier to explain using the CLI, as this avoids
the necessity of showing snapshots of the web interface.

Here, the only network you'll be adding to OCI would be the OpenStack
intnernal networks. Never, you'll be adding the public networks or the
ones in the OpenStack VMs. For example, one network for the management of
nodes, one for vm-net, one for the ceph-cluster network ... All of the
networks you'll be using on OpenStack, are to be provisioned with
OpenStack itself using the OpenStack API.

### Creating Swift regions and locations

Before installing the systems on your servers, clusters must be defined.
This starts by setting-up Swift regions. In a Swift cluster, there are
zones and regions. When uploading a file to Swift, it is replicated on
N zones (usually 3). If 2 regions are defined, then Swift tries to
replicate objects on both regions.

Under OCI, you must first define Swift regions. To do so, click on
"Swift region" on the web interface, or using ocicli, type:

```
# ocicli swift-region-create datacenter-1
# ocicli swift-region-create datacenter-2
```

Then create locations attached to these regions:

```
# ocicli dc1-zone1 datacenter-1
# ocicli dc1-zone2 datacenter-1
# ocicli dc2-zone1 datacenter-2
```

Later on, when adding a swift data node to a cluster (data nodes are
the servers that will actually do the Swift storage), a location must
be selected.

Once the locations have been defined, it is time to define networks.
Networks are attached to locations as well. The Swift zones and regions
will be related to these locations and regions.

### Creating networks

```
# ocicli network-create dc1-net1 192.168.101.0 24 dc1-zone1 no
```

The above command will create a subnet 192.168.101.0/24, located at
dc1-zone1. Let's create 2 more networks:

```
# ocicli network-create dc1-net2 192.168.102.0 24 dc1-zone2 no
# ocicli network-create dc2-net1 192.168.103.0 24 dc2-zone1 no
```

Next, for the cluster to be reachable, let's create a public network
on which customers will connect:

```
# ocicli network-create pubnet1 203.0.113.0 28 public yes
```

Note that if using a /32, it will be setup on the lo interface of
your controller. The expected setup is to use BGP to route that
public IP on the controller. To do that, it is possible to customize
the ENC and add BGP peering to your router. See at the end of this
documentation for that.

### Creating a new cluster

Let's create a new cluster:

```
# ocicli cluster-create swift01 example.com
```

Now that we have a new cluster, the networks we created can be added to it:

```
# ocicli network-add dc1-net1 swift01 all eth0
# ocicli network-add dc1-net2 swift01 all eth0
# ocicli network-add dc2-net1 swift01 all eth0
# ocicli network-add pubnet1 swift01 all eth0
```

When adding the public network, automatically, one IP address will be
reserved for the VIP (Virtual Private IP). This IP address will later
be shared by the controller nodes, to perform HA (High Availability),
controlled by pacemaker / corosync. The principle is: if one of
the controllers nodes is hosting the VIP (and it's assigned to its
eth0), and becomes unavailable (let's say, the server crashes or the
network wire is unplugged), then the VIP is re-assigned to the eth0
of another controller node of the cluster.

If selecting 2 network interfaces (for example, eth0 and eth1), then
bonding will be used. Note that your network equipment (switches, etc.)
must be configured accordingly (LACP, etc.), and that the setup of
these equipment is out of the scope of this documentation. Consult your
network equipment vendor for more information.

## Real certificate for the API

By default, OCI will generate self-signed certificate for everyting.
Though this works well except a few exceptions (it noticeably doesn't work
for Heat, Magnum and if one wants to enable Swift on-disk encryption), it is
preferable, in production, to use a real API certificate, so that clients
can trust your server. In order to do this, one must first choose a hostname
for the API. This is set this way:

```
# ocicli cluster-set z --vip-hostname cloud-api.example.com
```

Once done, in the OCI server, generate a certificate for this hostname:

```
# oci-gen-slave-node-cert cloud-api.example.com
```

The cd to /var/lib/oci/ssl/slave-nodes/cloud-api.example.com. Then can
be find the cloud-api.example.com.csr (.csr stands for Certificate Signing
Certificate) which can be used to optain a real certificate. Get the
certificate signed, and then replace the .crt, and .pem files with the
real signed content. If you are re-using a wildcard certicate, then you
probably also want to replace the .key file. Note that the .pem file
must contain the certificate *and* the private key, concatenated, and
maybe also all the intermediate certificates.

Once this is done, simply inform OCI that we're using a real signed
certificate:

```
# ocicli cluster-set z --self-signed-api-cert no
```

Now, puppet will be started without using the OCI's root ca as environment,
and ca_file will not be used in all OpenStack configuration files (an
empty string will be set instead).

If you have set your cluster in production before signing the certificate,
it is possible to use, on the puppet server, the oci-update-cluster-certs
utility:

```
# oci-update-cluster-certs z
```

This will replace the certificate cloud-api.example.com everywhere in
the cluster, and restart services to use it. This shell utility is also
useful whenever your SSL certificate expires and needs to be updated.

## Enrolling servers in a cluster

Now that we have networks assigned to the cluster, it is time to add
assign servers to the cluster. Let's say we have the below output:

```
# ocicli machine-list
serial  ipaddr          memory  status  lastseen             cluster  hostname
C1      192.168.100.20  8192    live    2018-09-19 20:31:57  null
C2      192.168.100.21  8192    live    2018-09-19 20:31:04  null
C3      192.168.100.22  8192    live    2018-09-19 20:31:14  null
C4      192.168.100.23  5120    live    2018-09-19 20:31:08  null
C5      192.168.100.24  5120    live    2018-09-19 20:31:06  null
C6      192.168.100.25  5120    live    2018-09-19 20:31:14  null
C7      192.168.100.26  4096    live    2018-09-19 20:31:18  null
C8      192.168.100.27  4096    live    2018-09-19 20:31:26  null
C9      192.168.100.28  4096    live    2018-09-19 20:30:50  null
CA      192.168.100.29  4096    live    2018-09-19 20:31:00  null
CB      192.168.100.30  4096    live    2018-09-19 20:31:07  null
CC      192.168.100.31  4096    live    2018-09-19 20:31:20  null
CD      192.168.100.32  4096    live    2018-09-19 20:31:28  null
CE      192.168.100.33  4096    live    2018-09-19 20:31:33  null
CF      192.168.100.34  4096    live    2018-09-19 20:31:40  null
D0      192.168.100.35  4096    live    2018-09-19 20:31:47  null
D1      192.168.100.37  4096    live    2018-09-21 20:31:23  null
D2      192.168.100.39  4096    live    2018-09-21 20:31:31  null
```

Then we can enroll machines in the cluster this way:

```
# ocicli machine-add C1 swift01 controller dc1-zone1
# ocicli machine-add C2 swift01 controller dc1-zone2
# ocicli machine-add C3 swift01 controller dc2-zone1
# ocicli machine-add C4 swift01 swiftproxy dc1-zone1
# ocicli machine-add C5 swift01 swiftproxy dc1-zone2
# ocicli machine-add C6 swift01 swiftproxy dc2-zone1
# ocicli machine-add C7 swift01 swiftstore dc1-zone1
# ocicli machine-add C8 swift01 swiftstore dc1-zone2
# ocicli machine-add C9 swift01 swiftstore dc2-zone1
# ocicli machine-add CA swift01 swiftstore dc1-zone1
# ocicli machine-add CB swift01 swiftstore dc1-zone2
# ocicli machine-add CC swift01 swiftstore dc2-zone1
```

As a result, there's going to be 1 controller, 1 Swift proxy and
2 Swift data node on each zone of our clusters. IP addresses will
automatically be assigned to servers as you add them to the clusters.
They aren't shown in ocicli, but you can check for them through the
web interface. The result should be like this:

```
# ocicli machine-list
serial  ipaddr          memory  status  lastseen             cluster  hostname
C1      192.168.100.20  8192    live    2018-09-19 20:31:57  7        swift01-controller-1.example.com
C2      192.168.100.21  8192    live    2018-09-19 20:31:04  7        swift01-controller-2.example.com
C3      192.168.100.22  8192    live    2018-09-19 20:31:14  7        swift01-controller-3.example.com
C4      192.168.100.23  5120    live    2018-09-19 20:31:08  7        swift01-swiftproxy-1.example.com
C5      192.168.100.24  5120    live    2018-09-19 20:31:06  7        swift01-swiftproxy-2.example.com
C6      192.168.100.25  5120    live    2018-09-19 20:31:14  7        swift01-swiftproxy-3.example.com
C7      192.168.100.26  4096    live    2018-09-19 20:31:18  7        swift01-swiftstore-1.example.com
C8      192.168.100.27  4096    live    2018-09-19 20:31:26  7        swift01-swiftstore-2.example.com
C9      192.168.100.28  4096    live    2018-09-19 20:30:50  7        swift01-swiftstore-3.example.com
CA      192.168.100.29  4096    live    2018-09-19 20:31:00  7        swift01-swiftstore-4.example.com
CB      192.168.100.30  4096    live    2018-09-19 20:31:07  7        swift01-swiftstore-5.example.com
CC      192.168.100.31  4096    live    2018-09-19 20:31:20  7        swift01-swiftstore-6.example.com
CD      192.168.100.32  4096    live    2018-09-19 20:31:28  null
CE      192.168.100.33  4096    live    2018-09-19 20:31:33  null
CF      192.168.100.34  4096    live    2018-09-19 20:31:40  null
D0      192.168.100.35  4096    live    2018-09-19 20:31:47  null
D1      192.168.100.37  4096    live    2018-09-21 20:31:23  null
D2      192.168.100.39  4096    live    2018-09-21 20:31:31  null
```

As you can see, hostnames are calculated automatically as well.

## Calculating the Swift ring

Before starting to install servers, the swift ring must be built.
Simply issue this command:

```
# ocicli swift-calculate-ring swift01
```

Note that it may take a very long time, depending on your cluster size.
This is expected. Just be patient.

## Installing servers

There's no (yet) a big "install the cluster" button on the web interface, or on
the CLI. Instead, servers must be installed one by one:

```
# ocicli machine-install-os C1
# ocicli machine-install-os C2
# ocicli machine-install-os C3
```

It is advised to first install the controller nodes, manually check that
they are installed correctly (for example, check that "openstack user list"
works), then the Swift store nodes, then the Swift proxy nodes. However,
nodes of the same type can be installed at once. Also, du to the use of
a VIP and corosync/pacemaker, controller nodes *must* be installed roughly
at the same time.

It is possible to see a server's installation log last lines using the
CLI as well:

```
# ocicli machine-install-log C1
```

This will show the logs of the system installation from /var/log/oci,
then once the server has rebooted, it will show the puppet logs from
/var/log/puppet-first-run.

## Checking your installation

Login on a controller node. To do that, list its IP:

```
# CONTROLLER_IP=$(ocicli machine-list | grep C1 | awk '{print $2}')
# ssh root@${CONTROLLER_IP}
```

Once logged into the controller, you'll see login credentials under
/root/oci-openrc.sh. Source it and try:

```
# . /root/oci-openrc.sh
# openstack user list
```

You can also try Swift:

```
# . /root/oci-openrc.sh
# openstack container create foo
# echo "test" >bar
# openstack object create foo bar
# rm bar
# openstack object delete foo bar
```

## Enabling Swift object encryption

Locally on the Swift store, Swift stores the object in clear form. This
means that anyone with physical access to the data center can pull a hard
drive and objects can be accessed from the /srv/node folder.
To mitigate this risk, Swift can do encryption of the objects it stores.
The metadata (accounts, containters, etc.) will still be stored in clear
form, but at least, the data that is stored encrypted.

The way this is implemented in OCI is to use Barbican. This is the reason
why Barbican is provisionned by default on the controller nodes. By default,
encryption isn't activated. To activate it, you must first store the key
for object encryption in the Barbican store. It can be done this way:

```
# ENC_KEY=$(openssl rand -hex 32)
# openstack secret store --name swift-encryption-key \
  --payload-content-type=text/plain --algorithm aes \
  --bit-length 256 --mode ctr --secret-type symmetric \
  --payload ${ENC_KEY}
+---------------+--------------------------------------------------------------------------------------------+
| Field         | Value                                                                                      |
+---------------+--------------------------------------------------------------------------------------------+
| Secret href   | https://swift01-api.example.com/keymanager/v1/secrets/6ba8dd62-d752-4144-b803-b32012d707d0 |
| Name          | swift-encryption-key                                                                       |
| Created       | None                                                                                       |
| Status        | None                                                                                       |
| Content types | {'default': 'text/plain'}                                                                  |
| Algorithm     | aes                                                                                        |
| Bit length    | 256                                                                                        |
| Secret type   | symmetric                                                                                  |
| Mode          | ctr                                                                                        |
| Expiration    | None                                                                                       |
+---------------+--------------------------------------------------------------------------------------------+
```

Once that's done, the key ID (here: 6ba8dd62-d752-4144-b803-b32012d707d0)
has to be entered in the OCI's web interface, in the cluster definition,
under "Swift encryption key id (blank: no encryption):". Once that's done,
another puppet run is needed on the swift proxy nodes:

```
root@C1-swift01-swiftproxy-1>_ ~ # OS_CACERT=/etc/ssl/certs/oci-pki-oci-ca-chain.pem puppet agent --test --debug
```

This should enable encryption. Note that the encryption key must be stored
in Barbican under the user swift and project services, so that Swift has
access to it.

Last, sometimes, "node1" appears when doing "crm status". To clean this
up, simply do:

```
crm_node -R node1 --force
```

## Initial cluster setup variable

To avoid doing too many things when the cluster is in production (like, for
example, starting MySQL to do the initial Galera cluster setup), OCI has a
variable called "initial-cluster-setup". It is on by default on the first
runs, and after all controllers report a successful puppet run, this
variable is automatically set to no. Here's a (probably non-exhaustive) list
of things that OCI does only if initial-cluster-setup is set to yes:

- openstack-api-vip resource in corosync
- Galera cluster
- Make controllers join the rabbitmq cluster
- Heat and Magnum domain users
- Nova cells v2 configuration

At any moment, it is possible to switch the value to yes or no:

```
# ocicli cluster-set z --initial-cluster-setup no
```

however, it is strongly advised to set the value to yes once the cluster is
in production.

## Adding other types of nodes

OCI can handle, by default, the below types of nodes:

- cephmon: Ceph monitor
- cephosd: Ceph data machines
- compute: Nova compute and Neutron DVR nodes
- controller: The OpenStack control plane, running all API and daemons
- swiftproxy: Swift proxy servers
- swiftstore: Swift data machines
- volume: Cinder LVM nodes
- network: DHCP, IPv4 SNAT and IPv6 routing

It is only mandatory to install 3 controllers, then everything else is
optional. There's nothing to configure, OCI will understand what the
user wants depending of what type of nodes is provisioned.

If cephosd nodes are deployed, then everything will be using Ceph:
- Nova (ie: /var/lib/nova/instances over Ceph)
- Glance (images stored on Ceph)
- Cinder (cinder-volume deployed on compute ndoes will be using the Ceph backend)

Though even with Ceph, setting-up volume nodes will add the LVM
backend capability. With or without volume nodes, if some OSD nodes
are deployed, cinder-volume and cinder-backup with Ceph backend will
be installed on the compute nodes.

Live migration of VMs between compute nodes is only possible if using
Ceph (ie: if some Ceph OSD nodes are deployed), or if using the
--block-migration option.

Ceph MON nodes are optional. If they aren't deployed, the Ceph MON and
MGR will be installed on the controller nodes.

Network nodes are optional. If they aren't deployed, the controllers
will act as SNAT and IPv6 routing nodes, and the DHCP servers will be
installed on the compute nodes.

# Advanced usage
## Using automated IPMI address configuration

Because it may take too much time to manage this manually, OCI offers the
possibility to automatically configure IPMI addresses of all discovered
servers. And because it is possible that in your network setup, there's
multiple IPMI networks depending on where the server is physically located,
OCI offers the possibility to automatically choose an IPMI network depending
on which DHCP network a server boots on the Debian live image.

The first thing to do is to define an IPMI network, set it with the role
"ipmi", and then make it match the IP address of the DHCP network:

```
# ocicli network-create ipmi 192.168.200.0 24 zone-1 no
# ocicli network-set ipmi --role ipmi --ipmi-match-addr 192.168.100.0 --ipmi-match-cidr 24
```

Once this is done, the automatic_ipmi_numbering=yes option must be set in
/etc/openstack-cluster-installer/openstack-cluster-installer.conf.

When this option is set, each time a server reports its hardware
configuration, OCI will check if it has a correct IPMI IP. If not, OCI will
ssh into the server and perform the necessary "ipmitool" commands to set a
valid network configuration. When doing so, the IP address will be reserved
in the "ips" table of OCI, making sure that never, an IP is used twice.

With the above example, if a server PXE boots on the 192.168.100.0/24
network, then it will automatically be assigned an IPMI ip address on the
192.168.200.0/24 network. Note that the IPMI password is randomly choosen.
As we're using openssl rand -base64, it may be a good idea to make sure that
your OCI server has a good source of entropy.

If previously, some servers had their IPMI address already set to something
that matches the IPMI network, but OCI didn't record it, it is possible to
get this IP address recorded in OCI's database. Just typing this command is
enough to do so:

```
# ocicli ipmi-assign-check
```

This command will ask OCI to go through each and every machine recorded in
the database, and check the detected IPMI address. If this address exists in
the database, nothing is done. If not, a new record will be added to the
database for this machine, to avoid later address conflict.

## Automatic upgrade of BIOS and IPMI firmware

Upgrading the BIOS and IPMI firmware of servers can take a really long time
if managing a large number of servers. So OCI offers the possibility to
perform these upgrades automatically. This is controled using a
configuration file that can be find in here: 
/etc/openstack-cluster-installer/oci-firmware-upgrade-config.json. Here is
an example valid configuration file:

```
{
	"CL2800 Gen10": {
		"BIOS": {
			"version": "2.1.0",
			"script": "/root/hp-bios-upgrade-2.1.0"
			},
		"IPMI": {
			"version": "2.22",
			"script": "/root/hp-ipmi-upgrade-2.22"
			}
	},
}
```

With the above, if OCI finds an HP Cloud Line CL2800 server that has the
BIOS firmware lower than 2.1.0, it will attempt to upgrade it by launching
the script /root/hp-bios-upgrade-2.1.0. To add the said script, the live
image must be customized. To do so, simply add some files under the folder
/etc/openstack-cluster-installer/live-image-additions. Every files that
are there will be added to the live image. Then the live image must be
regenerated:

```
# openstack-cluster-installer-build-live-image
```

Once this is done, reboot servers that must be upgraded. As they boot
on the live image, the upgrade will be performed. For reference, here is an
example hp-bios-upgrade-2.1.0 script, which will be dumped here:
/etc/openstack-cluster-installer/live-image-additions/root/hp-bios-upgrade-2.1.0.

```
#!/bin/sh

set -e
set -x

cd /root
tar -xvzf CL2600_CL2800_Gen10_BIOS_v2.1.0_11052019_Linux.tgz
cd CL2600_CL2800_Gen10_BIOS_v2.1.0_11052019_Linux/FlashTool/
./flash_bios.sh
reboot
sleep 20000
```

The "sleep 20000" is to make sure the OCI agent doesn't restart before the
machine is rebooted. YMMV depending on the upgrade that needs to be
performed.

## Customizing the /etc/hosts of all your cluster

It is possible to add some entries on all of the /etc/hosts of clusters, if
adding some entries to this file on the OCI server:

/etc/openstack-cluster-installer/hosts_append

All what OCI generates is located between these tags:

```
# OCISTA_MAINTAINED: Do not touch between these lines, this is a generated content.
... some generated content ...
# OCIFIN_MAINTAINED: Do not touch between these lines, this is a generated content.
```

Then it's possible to add some entries to each individual /etc/hosts
manually after the above tag, and these entries will be preserved.

## Customizing the ENC

In /etc/openstack-cluster-installer/hiera, you'll find 2 folders and a
all.yaml. These are to allow one to customize the output of OCI's ENC.
For example, if you put:

```
   ntp:
      servers:
         - 0.us.pool.ntp.org iburst
```

in /etc/openstack-cluster-installer/hiera/all.yaml, then all nodes will
be configured with ntp using 0.us.pool.ntp.org to synchronize time.

If we have a swift01 cluster, then the full folder structure is as follow:

```
/etc/openstack-cluster-installer/hiera/roles/controller.yaml
/etc/openstack-cluster-installer/hiera/roles/swiftproxy.yaml
/etc/openstack-cluster-installer/hiera/roles/swiftstore.yaml
/etc/openstack-cluster-installer/hiera/nodes/-hostname-of-your-node-.yaml
/etc/openstack-cluster-installer/hiera/all.yaml
/etc/openstack-cluster-installer/hiera/clusters/swift01/roles/controller.yaml
/etc/openstack-cluster-installer/hiera/clusters/swift01/roles/swiftproxy.yaml
/etc/openstack-cluster-installer/hiera/clusters/swift01/roles/swiftstore.yaml
/etc/openstack-cluster-installer/hiera/clusters/swift01/nodes/-hostname-of-your-node-.yaml
/etc/openstack-cluster-installer/hiera/clusters/swift01/all.yaml

```

## Customizing installed server at setup time

Sometimes, it is desirable to configure a server at setup time. For example,
it could be needed to configure routing (using BGP) for the virtual IP to be
available at setup time. OCI offers all what's needed in order to enrich the
server configuration at install time, before puppet agent even starts.

Say you want to configure swift01-controller-1 in your swift01 cluster, add
quagga to it, and add some configuration files. Simply create the folder,
fill content in it, and add a oci-packages-list file:

```
# mkdir -p /var/lib/oci/clusters/swift01/swift01-controller-1.example.com/oci-in-target
# cd /var/lib/oci/clusters/swift01/swift01-controller-1.example.com
# echo -n "quagga,tmux" >oci-packages-list
# mkdir -p oci-in-target/etc/quagga
# echo "some conf" >oci-in-target/etc/quagga/bgpd.conf
```

When OCI provision the baremetal server, it looks if the oci-packages-list
file exists. If it does, the packages are added when installing. Then the
oci-in-target content is copied into the target system.

## Using a BGP VIP

The same way, you can for example, decide to have the VIP of your
controllers to use BGP routing. To do that, write in
/etc/openstack-cluster-installer/roles/controller.yaml:

```
   quagga::bgpd:
      my_asn: 64496,
      router_id: 192.0.2.1
      networks4:
         - '192.0.2.0/24'
      peers:
         64497:
            addr4:
               - '192.0.2.2'
            desc: TEST Network
```

Though you may want to do this only for a specific node of a single
cluster of servers, rather than all. In such case, simply use this
filepath scheme:
/etc/openstack-cluster-installer/clusters/cloud1/nodes/cloud1-controller-1.example.com.yaml

For all controllers of the cloud1 cluster, use:
/etc/openstack-cluster-installer/clusters/cloud1/roles/controller.yaml

## Doing a test in OCI's manifests for debug purpose

If you would like to test a change in OCI's puppet files, edit them
in /usr/share/puppet/modules/oci, then on the master run, for example:

```
# puppet master --compile swift01-controller-1.example.com
# /etc/init.d/puppet-master stop
# /etc/init.d/puppet-master start
```

then on swift01-controller-1.example.com you can run:

```
# OS_CACERT=/etc/ssl/certs/oci-pki-oci-ca-chain.pem puppet agent --test --debug
```

## Customizing files and packages in your servers

If you wish to customize the file contents of your hosts, simply write
any file in, for example:

```
/var/lib/oci/clusters/swift01/swift01-controller-1.example.com/oci-in-target
```

and it will be copied in the server you'll be installing.

The same way, you can add additional packages to your server by adding their
names in this file:

```
/var/lib/oci/clusters/swift01/swift01-controller-1.example.com/oci-packages-list
```

Packages must be listed on a single line, separated by comas. For example:

```
quagga,bind
```

### Enabling Hiera for environment

If you need to enable Hiera, you can do it this way:
```
# mkdir -p /etc/puppet/code/environments/production/manifests/
# echo "hiera_include('classes')" > /etc/puppet/code/environments/production/manifests/site.pp
# cat /etc/puppet/code/hiera/common.yaml
---
classes:
  - xxx
...
```
#!/bin/sh

set -e
set -x

# Once deployment is ready

There's currently a few issues that need to be addressed by hand. Hopefully,
all of these will be automated in a near future. In the mean while, please
do contribute the fixes if you find out how, or just do as per what's below.

## Fixing-up the controllers

Unfortunately, sometimes, there's some scheduling issues in the puppet
apply. If this happens, one can try to relaunch the puppet thing:

```
# OS_CACERT=/etc/ssl/certs/oci-pki-oci-ca-chain.pem puppet agent --test --debug 2>&1 | tee /var/log/puppet-run-1
```

Do this on the controller-1 node first, wait until it finishes, then restart
it on the other controller nodes.

## Adding custom firewall rules

OCI is using puppet-module-puppetlabs-firewall, and flushes iptables on each
run. Therefore, if you need custom firewall rules, you also have to do it
via puppet. If you want to do apply the same firewall rules on all nodes,
simply edit the site.pp like this in /etc/puppet/code/environments/production/manifests/site.pp:

```
hiera_include('classes')

firewall { '000 allow monitoring network':
  proto       => tcp,
  action      => accept,
  source      => "10.3.50.0/24",
}
```

Note that the firewall rule is prefixed with a number. This is mandatory.
Also, make sure that this number doesn't enter in conflict with an already
existing rule.

What's done by OCI is: protect the controller's VIP (deny access to it from
the outside), and protect the swiftstore ports for account, container and
object servers from any query not from within the cluster. So the above will
allow a monitoring server from 10.3.50.0/24 to monitor your swiftstore
does.

If you wish to have the above applied only to a specific node, it's possible
to do so by only matching some hostnames. Here's a simple example, with a
different IP allowed depending on the machine roles:

```
hiera_include('classes')

node /^z-controller.*/ {
  firewall { '000 allow monitoring network':
    proto       => tcp,
    action      => accept,
    source      => "10.1.2.0/24",
  }
}

node default {
  firewall { '000 allow monitoring network':
    proto       => tcp,
    action      => accept,
    source      => "10.3.4.0/24",
  }
}
```

## Adding compute nodes

With latest version of OCI, this is performed automatically: after a compute
node runs puppet with success, it calls oci-report-puppet-success, which
contacts the provisionning node, which in its turn ssh to one of the
controllers to run "nova-manage cell_v2 discover_hosts". So what's bellow is
only needed if the compute node didn't install correctly directly.

To add the compute node to the cluster and check it's there, on the controller, do:

```
# . oci-openrc
# su nova -s /bin/sh -c "nova-manage cell_v2 discover_hosts"
# openstack hypervisor list
+----+-------------------------------+-----------------+---------------+-------+
| ID | Hypervisor Hostname           | Hypervisor Type | Host IP       | State |
+----+-------------------------------+-----------------+---------------+-------+
|  4 | swift01-compute-1.example.com | QEMU            | 192.168.103.7 | up    |
+----+-------------------------------+-----------------+---------------+-------+
```

There's nothing more to it... :)

## Adding GPU support in a compute node

Currently, only Nvidia boards are supported, however, we welcome
contributions. First, locate your GPU in your compute host. Here's
an example with an Nvidia T4 board:

```
# lspci -nn | grep -i nvidia
5e:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
```

When you have that, simply enter it with ocicli:
```
# ocicli machine-set 1CJ9FV2 --use-gpu yes --gpu-vendor-id 10de --gpu-produc-id 1eb8 --gpu-name nvidia-t4 --vfio-ids 10de:1eb8+10de:0fb9
```

Please note that the IDs in the --vfio-ids must be separated by +, not by
a comma (conversion is done later on by OCI and Puppet).

This will populate /etc/modprobe.d/blacklist-nvidia.conf to blacklist the
Nvidia driver and a few others, /etc/modules-load.d/vfio.conf to load the
vfio-pci module, and /etc/modprobe.d/vfio.conf with this content (to allow
exposing devices to guests):

```
options vfio-pci ids=10de:1eb8,10de:0fb9
```

The /etc/default/grub should then be modified by hand to add this:

```
intel_iommu=on
```

reboot the compute machine, apply puppet on both the compute and the
controllers.

Now, let's create the Glance image and Nova flavor to use this new
GPU and start the instance:

```
# openstack image set bionic-server-cloudimg-amd64_20190726_GPU --property img_hide_hypervisor_id='true'
# openstack flavor create --ram 6144 --disk 20 --vcpus 2 cpu2-ram6-disk20-gpu-nvidia-t4
# openstack flavor set cpu6-ram20-disk20-gpu-nvidia-t4 --property pci_passthrough:alias=nvidia-t4:1
# openstack server create --image bionic-server-cloudimg-amd64_20190726_GPU --nic net-id=demo-net --key-name demo-keypair --flavor cpu6-ram20-disk20-gpu-nvidia-t4 my-instance-with-gpu
```

In the instance, we can use Cuda and check for it:

```
# wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.168-1_amd64.deb
# apt-get update
# apt-get install cuda cuda-toolkit-10-1  nvidia-cuda-toolkit
# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  430.26  Tue Jun  4 17:40:52 CDT 2019
GCC version:  gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
```

## Multiple Cinder LVM backends

If using more than one type of LVM backend (for example, SSD and HDD), it
may be useful to select the name of the backend when setting-up a new Volume
node. This is done this way:

```
# ocicli machine-set 1CJ9FV2 --lvm-backend-name HDD_1
```

# Managing the OpenStack deployment
## Enabling cloudkitty rating

First, enable the hashmap module:

```
cloudkitty module enable hashmap
cloudkitty module set priority hashmap 100
```

Note that the error 503 may be just ignored, it still works, as "module
list" shows. Now, let's add rating for instances:

```
cloudkitty hashmap group create instance_uptime_flavor
cloudkitty hashmap service create compute
cloudkitty hashmap field create 96a34245-83ae-406b-9621-c4dcd627fb8e flavor
```

The above ID is the one of the hashmap service create. Then we reuse the ID
of the field create we just had for the -f parameter, and the group ID for
the -g parameter below:
```
cloudkitty hashmap mapping create --field-id ce85c041-00a9-4a6a-a25d-9ebf028692b6 --value demo-flavor -t flat -g 2a986ce8-60a3-4f09-911e-c9989d875187 0.03
```

## Installing a first OpenStack image

```
wget http://cdimage.debian.org/cdimage/openstack/current-9/debian-9-openstack-amd64.qcow2
openstack image create \
	--container-format bare --disk-format qcow2 \
	--file debian-9-openstack-amd64.qcow2 \
	debian-9-openstack-amd64
```

## Setting-up networking

There's many ways to handle networking in OpenStack. This documentation only
quickly covers one way, and it is out of the scope of this doc to explain
all of OpenStack networking. However, the reader must know that OCI is
setting-up compute nodes using DVR (Distributed Virtual Routers), which
means a Neutron router is installed on every compute nodes. Also,
OpenVSwitch is used, using VXLan between the compute nodes. Anyway, here's
one way to setup networking. Something like this may do it:

```
# Create external network
openstack network create --external --provider-physical-network external --provider-network-type flat ext-net
openstack subnet create --network ext-net --allocation-pool start=192.168.105.100,end=192.168.105.199 --dns-nameserver 84.16.67.69 --gateway 192.168.105.1 --subnet-range 192.168.105.0/24 --no-dhcp ext-subnet

# Create internal network
openstack network create --share demo-net
openstack subnet create --network demo-net --subnet-range 192.168.200.0/24 --dns-nameserver 84.16.67.69 demo-subnet

# Create router, add it to demo-subnet and set it as gateway
openstack router create demo-router
openstack router add subnet demo-router demo-subnet
openstack router set demo-router --external-gateway ext-net

# Create a few floating IPs
openstack floating ip create ext-net
openstack floating ip create ext-net
openstack floating ip create ext-net
openstack floating ip create ext-net
openstack floating ip create ext-net

# Add rules to the admin's security group to allow ping and ssh
SECURITY_GROUP=$(openstack security group list --project admin --format=csv | q -d , -H 'SELECT ID FROM -')
openstack security group rule create --ingress --protocol tcp --dst-port 22 ${SECURITY_GROUP}
openstack security group rule create --protocol icmp --ingress ${SECURITY_GROUP}
```

## Adding an ssh key

```
openstack keypair create --public-key ~/.ssh/id_rsa.pub demo-keypair
```

## Creating flavor

```
openstack flavor create --ram 2048 --disk 5 --vcpus 1 demo-flavor
openstack flavor create --ram 6144 --disk 20 --vcpus 2 cpu2-ram6-disk20
openstack flavor create --ram 12288 --disk 40 --vcpus 4 cpu4-ram12-disk40
```

## Boot a VM

```
#!/bin/sh

set -e
set -x

NETWORK_ID=$(openstack network list --name demo-net -c ID -f value)
IMAGE_ID=$(openstack image list -f csv 2>/dev/null | q -H -d , "SELECT ID FROM - WHERE Name LIKE 'debian-10%.qcow2'")
FLAVOR_ID=$(openstack flavor show demo-flavor -c id -f value)

openstack server create --image ${IMAGE_ID} --flavor ${FLAVOR_ID} \
	--key-name demo-keypair --nic net-id=${NETWORK_ID} --availability-zone nova:z-compute-1.example.com demo-server
```

## Add billing

The below script will rate "demo-flavor" at 0.01:

```
cloudkitty module enable hashmap
cloudkitty module set priority hashmap 100
cloudkitty hashmap group create instance_uptime_flavor_id
GROUP_ID=$(cloudkitty hashmap group list -f value -c "Group ID")

cloudkitty hashmap service create instance
SERVICE_ID=$(cloudkitty hashmap service list -f value -c "Service ID")

cloudkitty hashmap field create ${SERVICE_ID} flavor_id
FIELD_ID=$(cloudkitty hashmap field list ${SERVICE_ID} -f value -c "Field ID")

FLAVOR_ID=$(openstack flavor show demo-flavor -f value -c id)

cloudkitty hashmap mapping create 0.01 --field-id ${FIELD_ID} --value ${FLAVOR_ID} -g ${GROUP_ID} -t flat
```

The rest may be found here: https://docs.openstack.org/cloudkitty/latest/user/rating/hashmap.html

Also, add the role rating to the admin:

```
openstack role add --user admin --project admin rating
```

Note: currently, after installing the cluster, all ceilometer agents must be
restarted in order to obtain metrics, even though they appear to be well
configured.

## Add Octavia service
### Scripted setup
All of what's done below can be done with 2 helper scripts:

```
oci-octavia-amphora-secgroups-sshkey-lbrole-and-network 
oci-octavia-certs
```

First, edit /usr/bin/oci-octavia-amphora-secgroups-sshkey-lbrole-and-network
header. There, you'll find these values:

```
# Set to either flat or vlan
OCTAVIA_NETWORK_TYPE=flat
# Set to the ID of the Octavia VLAN if the above is set to vlan
OCTAVIA_NETWORK_VLAN=876
# Set this to a value that matches something listed in /etc/neutron/plugins/ml2/ml2_conf.ini
# either in [ml2_type_flat]/flat_networks or in [ml2_type_vlan]/network_vlan_ranges
OCTAVIA_PHYSNET_NAME=external1

OCTAVIA_SUBNET_RANGE=192.168.104.0/24
OCTAVIA_SUBNET_START=192.168.104.4
OCTAVIA_SUBNET_END=192.168.104.250
OCTAVIA_SUBNET_GW=192.168.104.1
OCTAVIA_SUBNET_DNS1=84.16.67.69
OCTAVIA_SUBNET_DNS2=84.16.67.70
```

Edit them to your taste. If you're running with vlan, then the value for
OCTAVIA_NETWORK_TYPE must be vlan, and the value for OCTAVIA_PHYSNET_NAME
must be "external". The IPs described above must be routable from the
controller nodes.

Once edit is done, run the first script, then tell OCI what security group
and network boot to use like this:

```
ocicli cluster-set CLUSTER_NAME --amp-secgroup-list SECGROUP_ID_1,SECGROUP_ID_2d5681bb2-044c-4de2-9f81-c3ca7d91abb6
ocicli cluster-set ver1 --amp-boot-network-list LOAD_BALANCER_NETWORK_ID
```

These IDs may be found in the logs when running
oci-octavia-amphora-secgroups-sshkey-lbrole-and-network, or in
/etc/octavia/octavia.conf under amp_secgroup_list and amp_boot_network_list.

Now, run oci-octavia-certs on one of the controllers, then
copy over /etc/octavia/.ssh and /etc/octavia/certs to the
other controllers.

```
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/certs/ root@z-controller-2:/etc/octavia/certs/
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/certs/ root@z-controller-3:/etc/octavia/certs/
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/.ssh/ root@z-controller-2:/etc/octavia/.ssh/
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/.ssh/ root@z-controller-3:/etc/octavia/.ssh/
```

Now, restart octavia-worker, octavia-health-manager
and octavia-housekeeping. The copy can be done this way:

That's it, it should work now!

### Manual setup
If you wish to do things manually, here's how it works.

Create the Amphora image. This can be done with DIB (Disk Image Builder)
like this:

```
sudo apt-get install openstack-debianimages
/usr/share/doc/openstack-debian-images/examples/octavia/amphora-build
openstack image create --container-format bare --disk-format qcow2 --file debian-buster-octavia-amphora-2019.09.11-11.52-amd64.qcow2 --tag amphora debian-buster-octavia-amphora-2019.09.11-11.52-amd64.qcow2
```

Create the Octavia network. If, like in the PoC package, you are
running with a specific br-lb bridge bound to an external network called
external1, something like this will do:

```
openstack network create --external --provider-physical-network external1 --provider-network-type flat lb-mgmt-net
openstack subnet create --network lb-mgmt-net --allocation-pool start=192.168.104.4,end=192.168.104.250 --dns-nameserver 84.16.67.69 --dns-nameserver 84.16.67.70 --gateway 192.168.104.1 --subnet-range 192.168.104.0/24 lb-mgmt-subnet
```

The above example is for when you're not running with vlan, but have
a specific network card for the Octavia network.

Then we need s specific security groups for Octavia (make sure to use
/root/octavia-openrc, not the admin's one):

```
openstack security group create lb-mgmt-sec-grp
openstack security group rule create --protocol icmp lb-mgmt-sec-grp
openstack security group rule create --protocol tcp --dst-port 22 lb-mgmt-sec-grp
openstack security group rule create --protocol tcp --dst-port 9443 lb-mgmt-sec-grp
openstack security group rule create --protocol icmpv6 --ethertype IPv6 --remote-ip ::/0 lb-mgmt-sec-grp
openstack security group rule create --protocol tcp --dst-port 22 --ethertype IPv6 --remote-ip ::/0 lb-mgmt-sec-grp
openstack security group rule create --protocol tcp --dst-port 9443 --ethertype IPv6 --remote-ip ::/0 lb-mgmt-sec-grp

openstack security group create lb-health-mgr-sec-grp
openstack security group rule create --protocol udp --dst-port 5555 lb-health-mgr-sec-grp
openstack security group rule create --protocol udp --dst-port 5555 --ethertype IPv6 --remote-ip ::/0 lb-health-mgr-sec-grp
```

Then we create an ssh keypair:

```
mkdir /etc/octavia/.ssh
ssh-keygen -t rsa -f /etc/octavia/.ssh/octavia_ssh_key
chown -R octavia:octavia /etc/octavia/.ssh
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/.ssh/ root@z-controller-2:/etc/octavia/.ssh/
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/.ssh/ root@z-controller-3:/etc/octavia/.ssh/
. /root/octavia-openrc
openstack keypair create --public-key /etc/octavia/.ssh/octavia_ssh_key.pub octavia-ssh-key
```

Make the certs as per the upstream tutorial at https://docs.openstack.org/octavia/latest/admin/guides/certificates.html

Rsync the certs to the other 2 controllers:

```
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/certs/ root@z-controller-2:/etc/octavia/certs/
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --delete /etc/octavia/certs/ root@z-controller-3:/etc/octavia/certs/
```

Edit octavia.conf and set amp_boot_network_list and amp_secgroup_list IDs.

Then restart all Octavia services on all controllers.

Create the load-balancer_admin role and assign it:

```
openstack role create load-balancer_admin
openstack role add --project admin --user admin load-balancer_admin
```

Now, one must set, with ocicli, the boot network and security group list for
the amphora:

```
ocicli cluster-set swift01 \
	--amp-boot-network-list 0c50875f-368a-4f43-802a-8350b330c127 \
	--amp-secgroup-list b94afddb-4fe1-4450-a1b8-25f36a354b7d,012584cd-ffde-483b-a55a-a1afba52bc20
```

Then we can start using Octavia:

```
openstack loadbalancer create --name lb-test-1 --vip-subnet-id ext-subnet
```
How to use the load balancer is described here:

https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html

### Using Octavia as an HTTPS load balancer for 2 web servers

The OpenStack documentation has all what you need at:
https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html

However, here's an example creating a loadbalancer with an HTTPS
certificate.

Creating the load balancer for the "foo" service:
```
openstack loadbalancer create \
    --name lb-foo \
    --vip-subnet-id pub01-subnet2
```

Create the certificate and store it in Barbican. First, create a normal
x509 certificate, with the key, crt and ca-chain files. Then convert it
to a pkcs12 cert using this command:

```
openssl pkcs12 -export -inkey server.key -in server.crt -certfile ca-chain.crt -passout pass: -out server.p12
```

Then we store it in Barbican, and keep its resulting address:
```
openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)"
```

Creating the listener:
```
openstack loadbalancer listener create \
    --name lb-foo-https \
    --protocol TERMINATED_HTTPS \
    --protocol-port 443 \
    --default-tls-container-ref https://z-api.example.com/keymanager/v1/secrets/e2e590a4-08b7-40e7-ab52-c06fd3a0a2dd \
    lb-foo
```

Creating the pool:
```
openstack loadbalancer pool create \
    --name pool-foo-https \
    --protocol TERMINATED_HTTPS \
    --listener lb-foo-https \
    --lb-algorithm ROUND_ROBIN
```

Creating the pool members:
```
openstack loadbalancer member create \
    --name foo-member-1-https \
    --address 10.4.42.10 \
    --protocol-port 443 \
    --subnet-id e499c943-09bb-46b7-8463-8d83ce51e830 \
    pool-foo-https
openstack loadbalancer member create \
    --name foo-member-2-https \
    --address 10.4.42.4 \
    --protocol-port 443 \
    --subnet-id e499c943-09bb-46b7-8463-8d83ce51e830 \
    pool-foo-https
```

## Setting-up no limits for services resources

As some services may spawn instances, like for example Octavia or Magnum, it
may be desirable to set no limit for some resources of the services project:

```
openstack quota set --secgroup-rules -1 --secgroups -1 --instances -1 --ram -1 --cores -1 --ports -1 services
```

The quota will apply for the virtual resources the services project will
create, for example, use openstack loadbalancer quota show PROJECT_NAME to
set the max number of loadbalancer for a project.

## Add Magnum service

First, upload the coreos image and set the property correctly:

```
openstack image create --file coreos_production_openstack_image.img coreos_production_openstack_image.img
openstack image set --property os_distro=coreos coreos_production_openstack_image.img
```

Then create the COE template:

```
openstack coe cluster template create k8s-cluster-template \
    --image coreos_production_openstack_image.img --keypair demo-keypair \
    --external-network ext-net --dns-nameserver 84.16.67.69 --flavor demo-flavor \
    --docker-volume-size 5 --network-driver flannel --coe kubernetes
```

Then create the Magnum cluster:

```
openstack coe cluster create k8s-cluster \
                      --cluster-template k8s-cluster-template \
                      --master-count 1 \
                      --node-count 2
```

Looks like coreos wouldn't work for k8s. Instead:

```
wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180419.0/CloudImages/x86_64/images/Fedora-Atomic-27-20180419.0.x86_64.qcow2
openstack image create \
                      --disk-format=qcow2 \
                      --container-format=bare \
                      --file=Fedora-Atomic-27-20180419.0.x86_64.qcow2 \
                      --property os_distro='fedora-atomic' \
                      fedora-atomic-latest
openstack coe cluster template create kubernetes-cluster-template \
	--image fedora-atomic-latest --keypair demo-keypair \
	--external-network ext-net --dns-nameserver 84.16.67.69 \
	--master-flavor demo-flavor --flavor demo-flavor \
	--docker-volume-size 5 --network-driver flannel \
	--coe kubernetes
```

# Upgrading
## From stretch-rocky to buster-rocky

### Upgrading compute nodes

First, switch the apt/sources.list to buster, and remove upstream's Ceph
backport repositories. Then remove all traces of Ceph from uptream:

```apt-get purge libcephfs2 librados2 librbd1 python3-rgw python3-rbd python3-rados python3-cephfs librgw2```

This probably will remove some Nova component, do it anyways. Then do the
dist-upgrade. Just hit entry on any prompt, or run in non-interactive mode
for Debconf prompts. Then just run puppet.

### Upgrading volume nodes

### Upgrading your controllers

Upgrading controllers from Stretch to Buster isn't an easy task, so OCI
includes a script to automate the task:

```
oci-cluster-upgrade-stretch-to-buster CLUSTER_NAME
```

It's going to do all for you. It's strongly advise to test this before
doing it on a live cluster. The upgrade takes about 1 hour if running
with 3 controllers.
