The article below is more like a memo to me about how to fast setup DevStack based private cloud on Ubuntu 16.04. It might take several hours or days solving all kind of issue related to different part of OpenStack and system configuration if start process without good stack knowledge. Or detailed step-by-step notes from previous successful installation. I hope it will save some hours for me next time.
Install Ubuntu 16.04
Create bootable USB drive from ubuntu iso
wget http://ubuntu.volia.net/ubuntu-releases/xenial/ubuntu-16.04.3-server-amd64.iso dd if=ubuntu-16.04.3-server-amd64.iso of=/dev/sdb
Install server minimum (+ ssh server)
Remove virtualization tools if accidentally chosen
sudo apt-get remove --purge qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
Update all components
sudo su - apt update apt upgrade
Disable IPv6
cat <<EOF >>/etc/sysctl.conf net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.all.rp_filter=0 net.ipv4.ip_forward=1 net.ipv4.conf.all.proxy_arp = 1 net.ipv4.conf.all.proxy_arp_pvlan = 1 EOF sysctl -p
Switch to static IP
vim /etc/network/interfaces
# The primary network interface auto enp3s0 iface enp3s0 inet static address 10.9.8.254 netmask 255.255.255.0 gateway liutyi.info dns-search local dns-nameservers liutyi.info
visudo
# User privilege specification root ALL=(ALL:ALL) NOPASSWD:ALL # Members of the admin group may gain root privileges %admin ALL=(ALL) NOPASSWD:ALL # Allow members of group sudo to execute any command %sudo ALL=(ALL:ALL) NOPASSWD:ALL
Add ssh key to user(s)
mkdir .ssh chmod 700 .ssh echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv1hNcFg6w3pOLPDdoc+WdBegk1a9VspjV9MaHTeJm6W1InpyBsSEzg/WUeQT9qfIin8dmjX8tRY9clGsp8xzrpiT2I/3mN8dgLvCfpWlNYNUkIPL5LlerbUmm9N0bnTm8OUpykBY8QPU2xJH9SB4fUDCOQ9kQOuoS/0CKNlainXQCrKpt50BHC4QdSsVwemN5Gqj5O2LqzAQGZBfykuY6WJU1ab/0YmgOcwb6QNP/KuI+biceZr3B49rroCN1/YWVCDVd6j7Wu898a5vh3qsQ0q/acBKrI7mdix+Lbc+TJfKfWVl89wAFOCmswMJku+nhj+kt5STDFklvRAFnuPg/ oliutyi@server' >~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys
reboot
reboot
Devstack setup
Clone DevStack
cd /opt git clone https://git.openstack.org/openstack-dev/devstack cd devstack/ git checkout stable/ocata
Create stack user
/opt/devstack/tools/create-stack-user.sh chown stack:stack /opt/devstack/ -R
Create config
vi /opt/devstack/local.conf
Minimal configuration
[[local|localrc]] FLOATING_RANGE=10.9.8.224/28 FIXED_RANGE=10.11.12.0/24 FIXED_NETWORK_SIZE=256 FLAT_INTERFACE=enp3s0 ADMIN_PASSWORD=password_1 DATABASE_PASSWORD=password_2 RABBIT_PASSWORD=password_3 SERVICE_PASSWORD=password_4
More settings
[[local|localrc]] IP_VERSION=4 HOST_IP=10.9.8.254 FLAT_INTERFACE=enp3s0 FLOATING_RANGE=10.9.8.224/28 IPV4_ADDRS_SAFE_TO_USE=10.9.9.0/24 FIXED_RANGE=10.11.12.0/24 FIXED_NETWORK_SIZE=256 ADMIN_PASSWORD=password_1 DATABASE_PASSWORD=password_2 RABBIT_PASSWORD=password_3 SERVICE_PASSWORD=password_4 VOLUME_GROUP_NAME="stack-volumes" VOLUME_NAME_PREFIX="volume-" VOLUME_BACKING_FILE_SIZE=100G
start installation
su - stack cd /opt/devstack/ ./stack.sh
Finish
========================= DevStack Component Timing ========================= Total runtime 749 run_process 55 test_with_retry 3 apt-get-update 2 pip_install 72 restart_apache_server 14 wait_for_service 13 apt-get 13 ========================= This is your host IP address: 10.9.8.254 Horizon is now available at http://10.9.8.254/dashboard Keystone is serving at http://10.9.8.254/identity/ The default users are: admin and demo The password: password_1 2017-10-04 23:56:18.294 | WARNING: 2017-10-04 23:56:18.294 | Using lib/neutron-legacy is deprecated, and it will be removed in the future DevStack Version: ocata Change: 8cb43d098ca88ce57ec216423c009cccd07f1299 Use "pip list" in check_libs_from_git 2017-10-02 11:09:56 -0400 OS Version: Ubuntu 16.04 xenial 2017-10-04 23:56:18.295 | stack.sh completed in 749 seconds.
Connect storage
apt install nfs-common mkdir /storage mount.nfs storage.local:/volume1/backup /storage
Import Images
http://10.9.8.254/dashboard/project/api_access/
export OS_AUTH_URL=http://10.9.8.254/identity/v3 export OS_PROJECT_ID=7da2ddbd9df5404da160a42eb7594ff2 export OS_PROJECT_NAME="demo" export OS_USER_DOMAIN_NAME="Default" unset OS_TENANT_ID unset OS_TENANT_NAME export OS_USERNAME="admin" export OS_PASSWORD=Password1 export OS_REGION_NAME="RegionOne" export OS_INTERFACE=public export OS_IDENTITY_API_VERSION=3
glance image-create --name DB --min-disk 5 --min-ram 1 --disk-format qcow2 --container-format bare --progress --file /storage/openstack/db-2017-10-01.qcow2; glance image-create --name HOME --min-disk 5 --min-ram 1 --disk-format qcow2 --container-format bare --progress --file /storage/openstack/home-2017-10-01.qcow2; glance image-create --name NOC --min-disk 5 --min-ram 1 --disk-format qcow2 --container-format bare --progress --file /storage/openstack/noc-2017-10-01.qcow2; glance image-create --name JIRA --min-disk 5 --min-ram 1 --disk-format qcow2 --container-format bare --progress --file /storage/openstack/jira-2017-10-01.qcow2; glance image-create --name WIKI --min-disk 5 --min-ram 1 --disk-format qcow2 --container-format bare --progress --file /storage/openstack/wiki-2017-10-01.qcow2;
Compress qcow2 images
prepare
before image creation important to fill empty disk spaces with zeroes
# 70GB disk space zeroed by dd if=/dev/zero of=/zerofile bs=1M count=70k rm /zerofile # zero to swapfile (or partition) swopoff -a dd if=/dev/zero of=/swapfile1 bs=1K count=1M mkswap /swapfile1 swapon -a
compress
qemu-img convert -O qcow2 -c /storage/openstack/wiki-2017-10-08.qcow /storage/openstack/wiki-2017-10-08-zip.qcow
result
-rw-r----- user users 8.0G Oct 8 13:22 wiki-2017-10-08.qcow -rw-r--r-- user users 5.5G Oct 8 13:36 wiki-2017-10-08-zip.qcow
Create Floating IP
openstack subnet list
PUBLICSUBNET=a9de3987-7954-4c2a-811d-87d02af4ed91
IPS="10.9.8.225 10.9.8.226 10.9.8.227 10.9.8.228 10.9.8.229 10.9.8.230 10.9.8.231"
for IP in ${IPS}; do
openstack floating ip create --floating-ip-address ${IP} ${PUBLICSUBNET}
done
openstack floating ip list
Open firewall on OpenStack
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0 nova secgroup-add-rule default tcp 80 80 0.0.0.0/0 nova secgroup-add-rule default tcp 443 443 0.0.0.0/0 nova secgroup-add-rule default tcp 8080 8080 0.0.0.0/0 nova secgroup-add-rule default tcp 8090 8090 0.0.0.0/0 nova secgroup-add-rule default tcp 10051 10051 0.0.0.0/0 nova secgroup-add-rule default tcp 10050 10050 0.0.0.0/0
Launching instances
nova image-list nova flavor-list nova boot --flavor 2 --image DB db nova boot --flavor 2 --image HOME home nova boot --flavor 2 --image NOC noc nova boot --flavor 3 --image JIRA jira nova boot --flavor 4 --image WIKI wiki nova list
Assign floating IPs
nova floating-ip-associate db 10.9.8.226 nova floating-ip-associate home 10.9.8.228 nova floating-ip-associate noc 10.9.8.227 nova floating-ip-associate jira 10.9.8.231 nova floating-ip-associate wiki 10.9.8.230
Change Application settings
Change DB settings for Confluence
vi /var/atlassian/application-data/confluence/confluence.cfg.xml /etc/init.d/confluence stop /etc/init.d/confluence start
Change DB settings for JIRA
vi /var/atlassian/application-data/jira/dbconfig.xml /etc/init.d/jira stop /etc/init.d/jira start
Postgres network check
vi /etc/postgresql/9.3/main/pg_hba.conf
Zabbix DB settings
vi /etc/zabbix/zabbix_server.conf vi /etc/zabbix/web/zabbix.conf.php
Nginx
vi /etc/nginx/sites-enabled/zabbix vi /etc/nginx/sites-enabled/wiki vi /etc/nginx/sites-enabled/jira service nginx restart
Restore DevStack instances after unplanned server restart
Backup failed stack data
cp -R /opt/stack/data /storage/openstack/failed-stack
Setup stack (with data lost)
re-install OS might not be needed, sometimes it is enough to run stack again:
su - stack
cd /opt/devstack/ ./stack.sh
- All steps from planed restore. Use available (old, or clear OS) images to re-create virtual machines with the same flavor and disk size.
- Stop virtual machines.
- Exchange disks in /opt/stack/data/nova/instances with disks from backup
rm -f /opt/stack/data/nova/instances/_base/*
cp /storage/openstack/failed-stack/data/nova/instances/_base/* /opt/stack/data/nova/instances/_base/
cp -f cp /storage/openstack/failed-stack/data/nova/instances/${OLD_VM_ID}/* /opt/stack/data/nova/instances/${NEW_VM_ID}/
Automation1
for name in db noc wiki jira home;do ln -s /opt/stack/data/nova/instances/`sudo grep root@$name /opt/stack/data/nova/instances/*/console.log |cut -d/ -f7| sort| uniq` /opt/stack/data/nova/instances/$name; done for name in db noc wiki jira home;do ln -s /storage/openstack/failed-state/nova/instances/`sudo grep root@$name /storage/openstack/failed-state/nova/instances/*/console.log |cut -d/ -f7| sort| uniq` /storage/openstack/failed-state/nova/instances/$name; done for name in db noc wiki jira home;do echo $name; sudo cp -f /storage/openstack/2018-01-31-failed-state/nova/instances/$name/disk /opt/stack/data/nova/instances/$name/disk; sudo chmod 644 /opt/stack/data/nova/instances/$name/disk; sudo chown root:root /opt/stack/data/nova/instances/$name/disk; done
- Start VMs
- See Application re-setup (for planned restore instruction)
Add SSH Key to be used for new instances created from public cloud images
ssh-keygen openstack keypair create --public-key /opt/stack/.ssh/id_rsa.pub keyname nova boot --key-name keyname --flavor 2 --image Ubuntu db2