9 Upgrading Oracle Linux Automation Manager
The following chapter provides instructions for upgrading Oracle Linux Automation Manager.
Upgrading a Release 1.0.X to a Release 2.0 Single Host Deployment
To upgrade a single host instance of Oracle Linux Automation Manager release 1.0.x to a single host instance of Oracle Linux Automation Manager Release 2.0, do the following:
- Log in to a terminal for the Oracle Linux Automation Manager Release 1.0.x version you want to upgrade.
- Back up the
/etc/tower/SECRET_KEY
file to a secure location. For example, you can copy the file to your home directory:sudo cp /etc/tower/SECRET_KEY ~
- Stop Oracle Linux Automation Manager.
sudo systemctl stop ol-automation-manager
- Log in to the user account that controls the database.
sudo su - postgres
- Export the database using the following command that creates a script file containing all
the necessary SQL commands and input data to restore the databases. For example,
this command creates the
olam1.dump
file in the database home directory.pg_dumpall > olamv1.dump
- Exit the user account that controls the
database.
exit
- Stop the database server.
sudo systemctl stop postgresql
- Remove ( and optionally backup ) existing database data directory. For example, the
following command removes and creates a backup file in the home directory.
sudo mv /var/lib/pgsql/data/ ~/data.old
- Remove the current version of the database.
sudo dnf remove postgresql
- Enable the
postgresql 12
orpostgresql 13
module stream.sudo dnf module reset postgresql sudo dnf module enable postgresql:12
or
sudo dnf module reset postgresql sudo dnf module enable postgresql:13
Note:
For more information about the Postgresql 12 and 13 life cycle, see the appendix discussing the application life cycle for stream modules in Oracle Linux: Managing Software on Oracle Linux. - Enable the Oracle Linux Automation Manager Yum repository for Release 2.1 as described in Oracle Linux Automation Manager 2.1: Installation Guide.
- Update Oracle Linux Automation
Manager.
dnf install ol-automation-manager-2.0.1-3* receptor-1.2.2-1* uwsgi-2.0.18-4.0.3*
If you're using the CLI to manage Oracle Linux Automation Manager, run the following command:dnf install ol-automation-manager-2.0.1-3* ol-automation-manager-cli-2.0.1-3* receptor-1.2.2-1* uwsgi-2.0.18-4.0.3*
Caution:
If you have installed theansible
package from another repository (for example, EPEL) the installation and upgrade process overwrites this package with theansible-core
package.Note:
The following message that's observed during the upgrade process can be ignored as it doesn't indicate any failure:
ValueError: File context for /var/run/tower(/.*)? already defined
- Install the
database.
sudo dnf install postgresql-server
- After the update completes, set up the database.
sudo postgresql-setup --initdb sudo systemctl start postgresql sudo su - postgres psql -d postgres -f olamv1.dump exit
- Run the following command to see if the database is
available:
sudo su - postgres -c "psql -l |grep awx"
Output similar to the following is displayed:awx | awx | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
- Replace
/etc/tower/settings.py
with/etc/tower/settings.py.rpmnew
. For example:sudo mv /etc/tower/settings.py /etc/tower/settingsold.py sudo mv /etc/tower/settings.py.rpmnew /etc/tower/settings.py
- In the
/etc/tower/settings.py
file, set theCLUSTER_HOST_ID
as follows:CLUSTER_HOST_ID = "hostname or ip address"
In the previous example, hostname or ip address is the hostname or IP address of the system running Oracle Linux Automation Manager. If hostname is used, the host must be resolvable. - In the
/etc/nginx/nginx.conf
, remove all existing configuration and replace it with the following text:user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic. include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; # Load modular configuration files from the /etc/nginx/conf.d directory. # See http://nginx.org/en/docs/ngx_core_module.html#include # for more information. include /etc/nginx/conf.d/*.conf; }
- In the
/etc/nginx/conf.d/ol-automation-manager-nginx.conf
, remove all existing configuration and replace it with the following text:upstream uwsgi { server unix:/var/run/tower/uwsgi.sock; } upstream daphne { server unix:/var/run/tower/daphne.sock; } server { listen 443 default_server ssl; listen 127.0.0.1:80 default_server; listen [::]:443 default_server ssl; listen [::1]:80 default_server; # If you have a domain name, this is where to add it server_name _; keepalive_timeout 65; ssl_certificate /etc/tower/tower.crt; ssl_certificate_key /etc/tower/tower.key; ssl_session_timeout 1d; ssl_session_cache shared:SSL:50m; ssl_session_tickets off; # intermediate configuration. tweak to your needs. ssl_protocols TLSv1.2; ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256'; ssl_prefer_server_ciphers on; # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months) add_header Strict-Transport-Security max-age=15768000; # add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/"; # add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/"; location /favicon.ico { alias /var/lib/awx/venv/awx/lib/python3.8/site-packages/awx/ui/build/static/media/favicon.ico; } location /static/ { alias /var/lib/awx/venv/awx/lib/python3.8/site-packages/awx/ui/build/static/; } location /websocket { # Pass request to the upstream alias proxy_pass http://daphne; # Require http version 1.1 to allow for upgrade requests proxy_http_version 1.1; # We want proxy_buffering off for proxying to websockets. proxy_buffering off; # http://en.wikipedia.org/wiki/X-Forwarded-For proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # enable this if you use HTTPS: proxy_set_header X-Forwarded-Proto https; # pass the Host: header from the client for the sake of redirects proxy_set_header Host $http_host; # We've set the Host header, so we don't need Nginx to muddle # about with redirects proxy_redirect off; # Depending on the request value, set the Upgrade and # connection headers proxy_set_header Upgrade $http_upgrade; # proxy_set_header Connection $connection_upgrade; proxy_set_header Connection upgrade; } location / { # Add trailing / if missing rewrite ^(.*[^/])$ $1/ permanent; uwsgi_read_timeout 120s; uwsgi_pass uwsgi; include /etc/nginx/uwsgi_params; } }
-
Remove any default configuration for Receptor. Edit
/etc/receptor/receptor.conf
to contain the following configuration:--- - node: id: <hostname or ip address> - log-level: debug - tcp-listener: port: 27199 #- work-signing: # privatekey: /etc/receptor/work_private_key.pem # tokenexpiration: 1m #- work-verification: # publickey: /etc/receptor/work_public_key.pem #- tcp-peer: # address: 100.100.253.53:27199 # redial: true #- tls-server: # name: mutual-tls # cert: /etc/receptor/certs/awx.crt # key: /etc/receptor/certs/awx.key # requireclientcert: true # clientcas: /etc/receptor/certs/ca.crt - control-service: service: control filename: /var/run/receptor/receptor.sock - work-command: worktype: local command: /var/lib/ol-automation-manager/venv/awx/bin/ansible-runner params: worker allowruntimeparams: true # verifysignature: true
In the previous example, hostname or ip address is the hostname or IP address of the system running Oracle Linux Automation Manager. If hostname is used, the host must be resolvable.
-
Prepare an Oracle Linux Automation Manager deployment as the awx user. Do the following:
-
Run the following commands:
sudo su -l awx -s /bin/bash podman system migrate podman pull container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:2.0 podman tag container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:2.0 container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:latest awx-manage makemigrations --merge awx-manage migrate awx-manage register_default_execution_environments
Note:
After you finish upgrading Oracle Linux Automation Manager, you can configure whether you want your Execution Environments to always pull the latestolam-ee
container image when running playbooks, or use some other option. For more information about these options, see Oracle Linux Automation Manager 2.2: User's Guide. -
Exit the awx shell environment.
exit
-
- Restore the /etc/tower/SECRET_KEY file. For
example:
sudo cp ~/SECRET_KEY /etc/tower/SECRET_KEY
- In the
/etc/tower/settings.py
file, add the following lines:DEFAULT_EXECUTION_QUEUE_NAME = 'tower' DEFAULT_CONTROL_PLANE_QUEUE_NAME = 'tower'
- Restart NGINX.
sudo systemctl restart nginx
- Start Oracle Linux Automation Manager.
sudo systemctl start ol-automation-manager
Upgrading Release 2.0 to Release 2.1
- On all Oracle Linux Automation Manager 2.0 nodes, log in to a terminal.
- Run an update:
sudo dnf update oraclelinux-automation-manager-release-el8-2.1 sudo dnf update ol-automation-manager-2.1* ol-automation-manager-cli-2.1* uwsgi-2.0.18-4.0.4* receptor-1.3*
- Pull the latest olam-ee image using.
sudo su -l awx -s /bin/bash podman pull container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:2.1.2 podman tag container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:2.1.2 container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:latest exit
- Add the following parameter to
/etc/tower/conf.d/olam.py
file:# OLAM Reaper Job Status Tracking REAPER_TIMEOUT_SEC = 60
The
REAPER_TIMEOUT_SEC
parameter specifies the time in seconds before a job in the Running or Waiting state is considered stuck and transitioned by the reaper into the Failed state. You can change this parameter in cases where you have playbooks that run longer than 60 seconds. For more information about this parameter, see Oracle Linux Automation Manager 2.1: Installation Guide. - Restart the following services on all nodes:
sudo systemctl restart ol-automation-manager sudo systemctl restart nginx sudo systemctl restart receptor-awx
- If required, you can change the default execution environment location for new custom
execution environment instances to the private automation hub instance. In addition, you
might download
olam-ee
fromcontainer-registry.oracle.com
to host it on Private Automation Hub to use for control plane instances. To set these default settings, do the following:- Edit the
/etc/tower/settings.py
file as follows:GLOBAL_JOB_EXECUTION_ENVIRONMENTS = [{'name': '<customer_execution_environment>)', 'image': '<private_automation_hub_hostname_or_ip_address>/<customer_execution_environment>:latest'}] CONTROL_PLANE_EXECUTION_ENVIRONMENT = '<private_automation_hub_hostname_or_ip_address>/olam-ee:latest'
- Register the new default environments using the following commands:
sudo su -l awx -s /bin/bash awx-manage register_default_execution_environments exit
-
Restart Oracle Linux Automation Manager on all nodes.
sudo systemctl restart ol-automation-manager
- Edit the
Upgrading Release 2.1 to Release 2.2
- Log into the local or remote database as root user.
- Calculate and update the memory requirements parameters using the
following:
In the previous example, total_mem_mb is the total memory size in megabytes of the system hosting the database server. For example, if the total available memory on the system were 18 000 MB, then this worksheet would include the following:max_connections = 1024 shared_buffers = total_mem_mb*0.3 work_mem = total_mem_mb*0.03 maintenance_work_mem = total_mem_mb*0.04
max_connections = 1024 shared_buffers = 18000*0.3 work_mem = 18000*0.03 maintenance_work_mem = 18000*0.04
The final numbers to add are as follows:
max_connections = 1024 shared_buffers = 5400MB work_mem = 540MB maintenance_work_mem = 720MB
- Add the calculated values to the
/var/lib/pgsql/data/postgresql.conf
file. - Restart the
database.
sudo systemctl restart postgresql
- For each Oracle Linux Automation Manager 2.1 nodes, sign in to a terminal.
- Stop the system:
sudo systemctl stop receptor-awx sudo systemctl stop ol-automation-manager
- Consider creating a custom settings file in the
/etc/tower/conf.d
folder. The settings you specify in custom settings files supersede the settings in/etc/tower/settings.py
, preventing software updates from changing the custom settings. To create a custom settings file, do the following:-
Create a custom settings file in the
/etc/tower/conf.d
folder. For example:touch
/etc/tower/conf.d/olam.py
- Add any Oracle Linux Automation Manager specific setting to the new file. For example,the CLUSTER_HOST_ID, DATABASES are some parameters you might want to add. For more information, see Setting up Hosts.
- Set the ownership and file permissions for the custom settings file. For example,
sudo chown awx:awx /etc/tower/conf.d/olam.py sudo chmod 0640 /etc/tower/conf.d/olam.py
-
- Update the
system:
sudo dnf update oraclelinux-automation-manager-release-el8-2.2 sudo mv /etc/yum.repos.d/oraclelinux-automation-manager-ol8.repo /etc/yum.repos.d/oraclelinux-automation-manager-ol8.repo.OLD sudo mv /etc/yum.repos.d/oraclelinux-automation-manager-ol8.repo.rpmnew /etc/yum.repos.d/oraclelinux-automation-manager-ol8.repo sudo dnf update ol-automation-manager ol-automation-manager-cli receptor
- If after the update, the
/etc/tower/settings.py.rpmnew
appears, do the following:sudo mv /etc/tower/settings.py /etc/tower/settings.py.OLD sudo mv /etc/tower/settings.py.rpmnew /etc/tower/settings.py
Note:
Consider adding any custom settings made to the/etc/tower/settings.py.OLD
file to the/etc/tower/conf.d
directory in a similar way as described in step 7. - If the
/etc/nginx/conf.d/ol-automation-manager-nginx.conf.rpmnew
file appears, then custom modifications were detected in the/etc/nginx/conf.d/ol-automation-manager-nginx.conf
file. The changes for the upgrade are contained in theol-automation-manager-nginx.conf.rpmnew
file. In the customol-automation-manager-nginx.conf
file find the following:location /favicon.ico { alias /var/lib/awx/venv/awx/lib/python3.9/site-packages/awx/ui/build/static/media/favicon.ico; } location /static/ { alias /var/lib/awx/venv/awx/lib/python3.9/site-packages/awx/ui/build/static/; }
Change them as follows:location /favicon.ico { alias /var/lib/awx/venv/awx/lib/python3.11/site-packages/awx/ui/build/static/media/favicon.ico; } location /static/ { alias /var/lib/awx/venv/awx/lib/python3.11/site-packages/awx/ui/build/static/; }
- Pull the 2.2 olam-ee image using.
sudo su -l awx -s /bin/bash podman system migrate podman pull container-registry.oracle.com/oracle_linux_automation_manager/olam-ee:2.2 awx-manage makemigrations --merge awx-manage migrate awx-manage register_default_execution_environments exit
- Restart the following services on all nodes:
sudo systemctl restart nginx sudo systemctl start receptor-awx sudo systemctl start ol-automation-manager
Migrating a Single Instance Release 2.2 Deployment to a Clustered Deployment
To migrate a single host instance deployment of Oracle Linux Automation Manager to a clustered deployment, do the following:
- If you need to upgrade a single instance host to release 2.2, complete the upgrade procedures in Upgrading Release 2.1 to Release 2.2.
- Verify that the upgraded instance is working.
- In a terminal, stop Oracle Linux Automation Manager.
sudo systemctl stop ol-automation-manager
- Create a database dump file.
sudo su - postgres pg_dumpall > olamv2upg.dump
- Complete the procedures for setting up a remote database in Setting Up a Local or Remote Database with the following
exceptions:
- Before starting the procedure, copy over the dump file to the remote database. For example, using scp.
- After starting and checking the status of the database in step 7 and 8, import the dump
file:
sudo su - postgres psql -d postgres -f /dirwithbackup/olamv2upg.dump exit
- Skip steps 9 through 11 for creating the database user account and creating the database because these are already part of the dump file.
- Continue the procedure at step 12 until you restart the database.
- On the remote database, reapply the password to the database user account:
sudo -u postgres psql \password awx
- Enter and confirm the password for the awx
user.
Enter new password for user "awx": Enter it again: exit
- Return to the Oracle Linux Automation Manager server, and create a custom settings file.
For
example:
sudo touch /etc/tower/conf.d/olam.py
- Add the following
DATABASES
fields to the custom settings file:DATABASES = { 'default': { 'ATOMIC_REQUESTS': True, 'ENGINE': 'awx.main.db.profiled_pg', 'NAME': 'awx', 'USER': 'awx', 'PASSWORD': 'password', 'HOST': 'database hostname or ip address', 'PORT': '5432', } }
In the previous example, database hostname or ip address is the hostname or IP address of the remote database. If hostname is used, the host must be resolvable. password is the password for your remote database, if you have configured one.
- Set the ownership and file permissions for the custom settings
file:
sudo chown awx:awx /etc/tower/conf.d/filename.py sudo chmod 0640 /etc/tower/conf.d/filename.py
In the previous example, filename is the name of the custom settings file. For example,
sudo chown awx:awx /etc/tower/conf.d/olam.py sudo chmod 0640 /etc/tower/conf.d/olam.py
- Stop the local database.
sudo systemctl stop postgresql
- Open the ports used for the Service
Mesh.
sudo firewall-cmd --add-port=27199/tcp --permanent sudo firewall-cmd --reload
- Start Oracle Linux Automation Manager.
sudo systemctl start ol-automation-manager
- Backup and remove the existing database data directory. For example, the following command
creates a backup file in the home directory and then removes the database:
sudo mv /var/lib/pgsql/data/ ~/data.old sudo dnf remove postgresql
- Run the following
command:
sudo su -l awx -s /bin/bash
- Run the following
commands.
awx-manage deprovision_instance --hostname=<hostname or IP address> awx-manage unregister_queue --queuename default awx-manage unregister_queue --queuename tower awx-manage provision_instance --hostname=<hostname or IP address> --node_type=control awx-manage register_queue --queuename=controlplane --hostnames=<hostname or IP address> exit
In the previous example, hostname or IP address is the hostname or IP address of the system running Oracle Linux Automation Manager. The host name or IP address must match the host name or IP addressed used when you configured the
/etc/receptor/receptor.conf
file node ID (see Configuring and Starting the Control Plane Service Mesh). If hostname is used, the host must be resolvable. You only need to unregister tower if it exists and is no longer used. - In the custom settings file (for example,
/etc/tower/conf.d/olam.py
), append theDEFAULT_EXECUTION_QUEUE_NAME
andDEFAULT_CONTROL_PLANE_QUEUE_NAME
fields:DEFAULT_EXECUTION_QUEUE_NAME = 'execution' DEFAULT_CONTROL_PLANE_QUEUE_NAME = 'controlplane'
- If you have playbooks designed to run longer than the default reaper timeout of 60
seconds, add the
REAPER_TIMEOUT_SEC
parameter to the custom settings file to increase the timeout. For example,REAPER_TIMEOUT_SEC=<longest_playbook_time>
In the previous example, <longest_playbook_time> is number of seconds that exceeds the duration of the longest playbook runtime.
- Restart Oracle Linux Automation
Manager.
sudo systemctl restart ol-automation-manager.service
- The original upgraded node is now converted into a control node. You must now add one more execution node for the upgraded cluster to be fully functional. For all other members of the cluster, follow the procedures described in Preparing the Database and Hosts, except for setting up a remote database, because this is already completed. Then, follow the procedures for installing and configuring all other hosts as part of the cluster, as described in Installing Oracle Linux Automation Manager in a Clustered Deployment.
Migrating Playbooks to Oracle Linux Automation Engine Release 2.2
Test any Oracle Linux Automation Engine release 2.1 playbooks to verify whether they function with Oracle Linux Automation Manager release 2.2. You might need to update playbooks because the upstream projects have made changes such as, the number of modules, some modules have become collections, and some modules have been merged into other modules or collections.
Verify that the playbooks are using the correct versions of python as described in Oracle Linux Automation Manager 2.2: Release Notes. For custom execution environments, review the overview section in Oracle Linux Automation Manager 2.2: Private Automation Hub User's Guide.