Upgrading Oracle Linux Automation Manager

9 Upgrading Oracle Linux Automation Manager

The following chapter provides instructions for upgrading Oracle Linux Automation Manager.

Upgrading a Release 1.0.X to a Release 2.0 Single Host Deployment

To upgrade a single host instance of Oracle Linux Automation Manager release 1.0.x to a single host instance of Oracle Linux Automation Manager Release 2.0, do the following:

Log in to a terminal for the Oracle Linux Automation Manager Release 1.0.x version you want to upgrade.
Back up the /etc/tower/SECRET_KEY file to a secure location. For example, you can copy the file to your home directory:
```
sudo cp /etc/tower/SECRET_KEY ~
```

Stop Oracle Linux Automation Manager.

sudo systemctl stop ol-automation-manager

Log in to the user account that controls the database.
```
sudo su - postgres
```
Export the database using the following command that creates a script file containing all the necessary SQL commands and input data to restore the databases. For example, this command creates the olam1.dump file in your database home directory.
```
pg_dumpall > olamv1.dump
```
Exit the user account that controls the database.
```
exit
```
Stop the database server.
```
sudo systemctl stop postgresql
```
Remove ( and optionally backup ) existing database data directory. For example, the following command removes and creates a backup file in the home directory.
```
sudo mv /var/lib/pgsql/data/ ~/data.old
```
Remove the current version of the database.
```
sudo dnf remove postgresql
```
Enable the postgresql 12 or postgresql 13 module stream.
```
sudo dnf module reset postgresql
sudo dnf module enable postgresql:12
```
or
```
sudo dnf module reset postgresql
sudo dnf module enable postgresql:13
```
Note:
For more information about the Postgresql 12 and 13 life cycle, see the appendix discussing the application life cycle for stream modules in Oracle Linux: Managing Software on Oracle Linux.
Enable the Oracle Linux Automation Manager Yum repos for Release 2 as described in Enabling Access to the Oracle Linux Automation Manager Packages.
Update Oracle Linux Automation Manager.
```
sudo dnf update ol-automation-manager
```
Caution:
If you have installed the ansible package from another repository (for example, EPEL) the installation and upgrade process overwrites this package with the ansible-core package.
Note:

The following message that is observed during the upgrade process can safely be ignored as it does not indicate any failure:
```
ValueError: File context for /var/run/tower(/.*)? already defined
```
Install the database.
```
sudo dnf install postgresql-server
```

After the update completes, set up the database.

sudo postgresql-setup --initdb
sudo systemctl start postgresql
sudo su - postgres
psql -d postgres -f olamv1.dump
exit

Run the following command to see if the database is available:
```
sudo su - postgres -c "psql -l |grep awx"
```
The output should resemble something like this:
```
awx       | awx      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
```

Replace /etc/tower/settings.py with /etc/tower/settings.py.rpmnew. For example:

sudo mv /etc/tower/settings.py /etc/tower/settingsold.py
sudo mv /etc/tower/settings.py.rpmnew /etc/tower/settings.py

In the /etc/tower/settings.py file, set the CLUSTER_HOST_ID as follows:
```
CLUSTER_HOST_ID = "hostname or ip address"
```
In the previous example, hostname or ip address is the hostname or IP address of the system running Oracle Linux Automation Manager. If hostname is used, the host must be resolvable.

In the /etc/nginx/nginx.conf, remove all existing configuration and replace it with the following text:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
 
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
 
events {
 worker_connections 1024;
}
 
http {
 log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                   '$status $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for"';
 
 access_log  /var/log/nginx/access.log  main;
 
 sendfile            on;
 tcp_nopush          on;
 tcp_nodelay         on;
 keepalive_timeout   65;
 types_hash_max_size 2048;
 
 include             /etc/nginx/mime.types;
 default_type        application/octet-stream;
 
 # Load modular configuration files from the /etc/nginx/conf.d directory.
 # See http://nginx.org/en/docs/ngx_core_module.html#include
 # for more information.
 include /etc/nginx/conf.d/*.conf;
}

In the /etc/nginx/conf.d/ol-automation-manager-nginx.conf, remove all existing configuration and replace it with the following text:

upstream uwsgi {
    server unix:/var/run/tower/uwsgi.sock;
}

upstream daphne {
    server unix:/var/run/tower/daphne.sock;
}

server {
    listen 443 default_server ssl;
    listen 127.0.0.1:80 default_server;
    listen [::]:443 default_server ssl;
    listen [::1]:80 default_server;

    # If you have a domain name, this is where to add it
    server_name _;
    keepalive_timeout 65;

    ssl_certificate /etc/tower/tower.crt;
    ssl_certificate_key /etc/tower/tower.key;

    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    # intermediate configuration. tweak to your needs.
    ssl_protocols TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;
    # add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";
    # add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";

    location /favicon.ico { alias /var/lib/awx/venv/awx/lib/python3.8/site-packages/awx/ui/build/static/media/favicon.ico; }

    location /static/ {
        alias /var/lib/awx/venv/awx/lib/python3.8/site-packages/awx/ui/build/static/;
    }

    location /websocket {
        # Pass request to the upstream alias
        proxy_pass http://daphne;
        # Require http version 1.1 to allow for upgrade requests
        proxy_http_version 1.1;
        # We want proxy_buffering off for proxying to websockets.
        proxy_buffering off;
        # http://en.wikipedia.org/wiki/X-Forwarded-For
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # enable this if you use HTTPS:
        proxy_set_header X-Forwarded-Proto https;
        # pass the Host: header from the client for the sake of redirects
        proxy_set_header Host $http_host;
        # We've set the Host header, so we don't need Nginx to muddle
        # about with redirects
        proxy_redirect off;
        # Depending on the request value, set the Upgrade and
        # connection headers
        proxy_set_header Upgrade $http_upgrade;
        # proxy_set_header Connection $connection_upgrade;
        proxy_set_header Connection upgrade;
    }

    location / {
        # Add trailing / if missing
        rewrite ^(.*[^/])$ $1/ permanent;
        uwsgi_read_timeout 120s;
        uwsgi_pass uwsgi;
        include /etc/nginx/uwsgi_params;
    }
}

Remove any default configuration for Receptor. Edit /etc/receptor/receptor.conf to contain the following configuration:

---
- node:
    id: <hostname or ip address>
 
- log-level: debug
 
- tcp-listener:
    port: 27199
 
#- work-signing:
#    privatekey: /etc/receptor/work_private_key.pem
#    tokenexpiration: 1m
 
#- work-verification:
#    publickey: /etc/receptor/work_public_key.pem
 
#- tcp-peer:
#    address: 100.100.253.53:27199
#    redial: true
 
#- tls-server:
#    name: mutual-tls
#    cert: /etc/receptor/certs/awx.crt
#    key: /etc/receptor/certs/awx.key
#    requireclientcert: true
#    clientcas: /etc/receptor/certs/ca.crt
 
- control-service:
    service: control
    filename: /var/run/receptor/receptor.sock
 
- work-command:
    worktype: local
    command: /var/lib/ol-automation-manager/venv/awx/bin/ansible-runner
    params: worker
    allowruntimeparams: true
#    verifysignature: true

In the previous example, hostname or ip address is the hostname or IP address of the system running Oracle Linux Automation Manager. If hostname is used, the host must be resolvable.

Prepare an Oracle Linux Automation Manager deployment as the awx user. Do the following:
1. Run the following commands:
```
sudo su -l awx -s /bin/bash
podman system migrate
podman pull container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:latest
awx-manage makemigrations --merge
awx-manage migrate
awx-manage register_default_execution_environments
```
  Note:
  After you finish upgrading Oracle Linux Automation Manager, you can configure whether you want your Execution Environments to always pull the latest olam-ee container image when running playbooks, or use some other option. For more information about these options, see Oracle Linux Automation Manager 2.1: User's Guide.
2. Exit the awx shell environment.
```
exit
```
Restore the /etc/tower/SECRET_KEY file. For example:
```
sudo cp ~/SECRET_KEY /etc/tower/SECRET_KEY
```

In the /etc/tower/settings.py file, add the following lines:

DEFAULT_EXECUTION_QUEUE_NAME = 'tower'
DEFAULT_CONTROL_PLANE_QUEUE_NAME = 'tower'

Restart NGINX.
```
sudo systemctl restart nginx
```

Start Oracle Linux Automation Manager.

sudo systemctl start ol-automation-manager

Upgrading Release 2.0 to Release 2.1

This upgrade is necessary for Oracle Linux Automation Manager to make use of Private Automation Hub execution environment container images and collections.

On all Oracle Linux Automation Manager 2.0 nodes, log in to a terminal.

Run an update:

sudo dnf clean all
sudo dnf update oraclelinux-automation-manager-release-el8
sudo dnf update ol-automation-manager ol-automation-manager-cli uwsgi

Pull the latest olam-ee image using.

sudo su -l awx -s /bin/bash
podman pull container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:latest
exit

Add the following parameter to /etc/tower/settings.py file:
```
# OLAM Reaper Job Status Tracking
REAPER_TIMEOUT_SEC = 60
```
The REAPER_TIMEOUT_SEC parameter specifies the time in seconds befor a job in the Running or Waiting state is considered stuck and transitioned by the reaper into the Failed state. You can modify this parameter in cases where you have playbooks that run longer than 60 seconds. For more information about this parameter, see Oracle Linux Automation Manager 2.1: Installation Guide.

Restart the following services on all nodes:

sudo systemctl restart ol-automation-manager
sudo systemctl restart nginx
sudo systemctl restart receptor-awx

If required, you can change the default execution environment location for new custom execution environment instances to your private automation hub instance. In addition, you might download olam-ee from container-registry.oracle.com to host it on Private Automation Hub to use for control plane instances. To set these default settings, do the following:
1. Edit the /etc/tower/settings.py file as follows:
```
GLOBAL_JOB_EXECUTION_ENVIRONMENTS = [{'name': '<customer_execution_environment>)', 'image': '<private_automation_hub_hostname_or_ip_address>/<customer_execution_environment>:latest'}]
CONTROL_PLANE_EXECUTION_ENVIRONMENT = '<private_automation_hub_hostname_or_ip_address>/olam-ee:latest'
```
2. Register the new default environments using the following commands:
```
sudo su -l awx -s /bin/bash
awx-manage register_default_execution_environments
exit
```
3. Restart Oracle Linux Automation Manager on all nodes.
```
sudo systemctl restart ol-automation-manager
```

Migrating a Single Instance Deployment to a Clustered Deployment

To migrate a single host instance deployment of Oracle Linux Automation Manager to a clustered deployment, do the following:

If you need to upgrade your single instance host to release 2.0, complete the upgrade procedures in Upgrading a Release 1.0.X to a Release 2.0 Single Host Deployment.
Verify that the upgraded instance is working.
In a terminal, stop Oracle Linux Automation Manager.
```
sudo systemctl stop ol-automation-manager
```

Create a database dump file.

sudo su - postgres
pg_dumpall > olamv2upg.dump

Open the firewall port on the remote database as described in Setting Up the Firewall Rules.
Complete the procedures for setting up a remote database in Setting Up a Local or Remote Database with the following exceptions:
1. Before starting the procedure, copy over the dump file to the remote database. For example, using scp.
2. After starting the database in step 7, import the dump file:
```
sudo su - postgres
psql -d postgres -f /dirwithbackup/olamv2upg.dump
exit
```
3. Skip steps 8 through 10 for creating the database user account and creating the database because these are already part of the dump file.
4. Continue the procedure at step 11.
On the remote database, reapply the password to the database user account:
```
sudo -u postgres psql
\password awx
```

Enter and confirm the password for the awx user.

Enter new password for user "awx": 
Enter it again: 
exit

Restart the database.
```
sudo systemctl restart postgresql
```
Return to the upgraded instance, and in the /etc/tower/settings.py file, replace the existing DATABASES fields with the following fields:
```
DATABASES = {
    'default': {
        'ATOMIC_REQUESTS': True,
        'ENGINE': 'awx.main.db.profiled_pg',
        'NAME': 'awx',
        'USER': 'awx',
        'PASSWORD': 'password',
        'HOST': 'database hostname or ip address',
        'PORT': '5432',
    }
}
```
In the previous example, database hostname or ip address is the hostname or IP address of the remote database. If hostname is used, the host must be resolvable. password is the password for your remote database, if you have configured one.
Stop the local database.
```
sudo systemctl stop postgresql
```

Open the ports used for the Service Mesh.

sudo firewall-cmd --add-port=27199/tcp --permanent
sudo firewall-cmd --reload

Start Oracle Linux Automation Manager.

sudo systemctl start ol-automation-manager

Remove the local database, because it is no longer needed.
```
sudo dnf remove postgresql
```
Run the following command:
```
sudo su -l awx -s /bin/bash
```
Remove the tower instance group (queue name) because this is not used in Oracle Linux Automation Manager release 2.
```
awx-manage remove_from_queue --queuename tower --hostname <hostname or IP address>
```
In the previous example, hostname or IP address is the hostname or IP address of the system running Oracle Linux Automation Manager.
Run the following commands.
```
awx-manage provision_instance --hostname=<hostname or IP address> --node_type=control
awx-manage register_queue --queuename=controlplane --hostnames=<hostname or IP address>
exit
```
In the previous example, hostname or IP address is the hostname or IP address of the system running Oracle Linux Automation Manager. Your choice of host name or IP address must match with the host name or IP addressed used when you configured the /etc/receptor/receptor.conf file node ID (see Configuring and Starting the Control Plane Service Mesh). If hostname is used, the host must be resolvable.

In the /etc/tower/settings.py file, replace the following lines.

DEFAULT_EXECUTION_QUEUE_NAME = 'tower'
DEFAULT_CONTROL_PLANE_QUEUE_NAME = 'tower'

with these lines.

DEFAULT_EXECUTION_QUEUE_NAME = 'execution'
DEFAULT_CONTROL_PLANE_QUEUE_NAME = 'controlplane'

Restart Oracle Linux Automation Manager.

sudo systemctl restart  ol-automation-manager.service

The original upgraded node is now converted into a control node. You must now add one more execute node for the upgraded cluster to be fully functional. For all other members of the cluster, follow the procedures described in Preparing the Database and Hosts, with the exception of setting up a remote database, because this is already completed. Then, follow the procedures for installing and configuring all other hosts as part of the cluster, as described in Installing Oracle Linux Automation Manager in a Clustered Deployment.

Migrating Playbooks to Oracle Linux Automation Engine Release 2.0

Test your Oracle Linux Automation Engine release 1.0.x playbooks to verify whether they function properly with Oracle Linux Automation Manager release 2.0. You may need to update your playbooks because the upstream projects have made changes such as, the number of modules, some modules have become collections, and some modules have been consolidated into other modules or collections.