Upgrading Cloudera Manager Using Tarballs

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This topic describes how to upgrade Cloudera Manager 5.x using tarballs. Tarballs contain both the Cloudera Manager Server and Cloudera Manager Agent in a single file.

In most cases it is possible to upgrade Cloudera Manager without shutting down most CDH services, although you may need to stop some dependent services. CDH daemons can continue running, unaffected, while Cloudera Manager is upgraded. The upgrade process does not affect your CDH installation. After upgrading Cloudera Manager you may also want to upgrade CDH 4 clusters to CDH 5, or upgrade to a more recent minor version of CDH.

Steps to upgrade Cloudera Manager 5 using tarballs:

Step	Description	Link
1	Collect the information you need to upgrade Cloudera Manager. This includes user accounts, passwords, database URLs, and other items. You must gather this information before beginning the upgrade because some information is available only from the Cloudera Manager Admin Console, which is not available during the upgrade process.	Step 1: Collect Upgrade Information
2	Complete the pre-upgrade steps and review special warnings about upgrades.	Step 2: Complete Pre-Upgrade Steps
3	If your Cloudera Manager hosts use an unsupported version of the JDK, you must upgrade the hosts to a supported version of the JDK before upgrading Cloudera Manager. If you plan to upgrade CDH, you must also upgrade the JDK on all cluster hosts.	Step 3: Upgrade the JDK Step 2: Install Java Development Kit Upgrading to Oracle JDK 1.8
4	If the Cloudera Manager host does not have access to the internet, or you install a version lower than the latest version of Cloudera Manager, configure access to the Cloudera Manager software from either the Cloudera public repository or a local package repository that you create.	Step 4: Using an Internal Package Repository
5	If you are upgrading from Cloudera Navigator 2.6 or lower, upgrade the Cloudera Navigator data management component.	Step 5: Upgrading the Cloudera Navigator Data Management Component
6	Upgrade the Cloudera Manager server and agent software.	Step 6: Upgrading the Cloudera Manager Server and Agents
7	Verify and test your upgrade.	Step 7: Verify and Test the Upgrade
8	Upgrade any required Cloudera Navigator components: Cloudera Manager Key Trustee Server Cloudera Navigator Key HSM Cloudera Navigator Key Trustee KMS Cloudera Navigator Encrypt. The Cloudera Navigator Data Management Component is upgraded automatically when you upgrade Cloudera Manager.	Step 8: Upgrading Cloudera Navigator Components
9	(Optional) Upgrade CDH. You are not required to upgrade CDH after upgrading Cloudera Manager. You can upgrade CDH at a later time.	Step 9: (Optional) Upgrade CDH

Step 1: Collect Upgrade Information

Before starting an upgrade, collect the following information:

Host credentials. You must have SSH access and be able to log in using a root account or an account that has password-less sudo permission.
The version of Cloudera Manager used in your cluster. Go to Support > About.
The version of the JDK deployed in the cluster. Go to Support > About.
The version of CDH. The CDH version number displays next to the cluster name on the Home page.
Whether the cluster was installed using parcels or packages. This information displays next to the CDH version on the Home page of Cloudera Manager.
The services enabled in your cluster. Go to Clusters > Cluster name.
Operating system type and version. Go to Hosts and click on a hostname in the list. The operating system type and version displays in the Distribution row in the Details section.

Step 2: Complete Pre-Upgrade Steps

Before beginning a Cloudera Manager upgrade, do the following:

Review the CDH 5 and Cloudera Manager 5 Requirements and Supported Versions for the new versions you are upgrading to.
Read the Cloudera Manager 5 Release Notes.
Read the Cloudera Security Bulletins.
Note the following:
- Cloudera Management Service TLS/SSL configuration
  If you have enabled TLS security for the Cloudera Manager Admin Console, as of Cloudera Manager 5.1, Cloudera Management Service roles communicate with Cloudera Manager using TLS, and fail to start until TLS/SSL properties have been configured.
- Navigator
  If you have enabled auditing with Cloudera Navigator, during the upgrade to Cloudera Manager 5, auditing is suspended and is only restarted when you restart the roles of audited services. You will be instructed to stop some services in a later step.
If you have previously installed Kafka 1.2, and are upgrading from Cloudera Manager 5.4 or lower, remove the Kafka CSD:
1. Determine the location of the CSD directory:
  1. Select Administration > Settings.
  2. Click the Custom Service Descriptors category.
  3. Retrieve the directory from the Local Descriptor Repository Path property.
2. Delete the Kafka CSD from the directory.
Review package (RPM) dependencies. A Cloudera Manager upgrade might introduce new package dependencies. If your organization has restrictions or requires prior approval for installation of packages, see the list of Package Dependencies before upgrading Cloudera Manager.

Step 3: Upgrade the JDK

See:

Step 4: Creating and Using a Package Repository for Cloudera Manager

If the Cloudera Manager host does not have access to the internet, or you install a version lower than the latest version of Cloudera Manager, configure access to the Cloudera Manager software from either the Cloudera public repository or a local package repository that you create.

See: Using an Internal Package Repository

Step 5: Upgrading the Cloudera Navigator Data Management Component

If you are upgrading from Cloudera Navigator 2.6 or lower, upgrade the Cloudera Navigator data management component.

See Upgrading the Cloudera Navigator Data Management Component

Step 6: Upgrading the Cloudera Manager Server and Agents

If your cluster is running the embedded PostgreSQL database, stop all services that are using the embedded database. These can include:
- Hive service and all services such as Impala and Hue that use the Hive metastore
- Oozie
- Sentry
Stop Cloudera Manager Server, Database, and Agent:
1. Use the Cloudera Manager Admin Console to stop any running commands. These include user commands and commands Cloudera Manager automatically triggers in response to a state change or a schedule. You can either wait for commands to complete, or stop any running commands. For more information on viewing and stopping running commands, see Viewing Running and Recent Commands.
  Important: If you do not stop all commands, the Cloudera Manager Server fails to start after upgrade.
2. On the host running the Cloudera Manager Server, stop the Cloudera Manager Server:
```
sudo service cloudera-scm-server stop
```
3. If you are using the embedded PostgreSQL database with Cloudera Manager, stop the database on the host where the database runs, usually the Cloudera Manager Server host:
```
sudo service cloudera-scm-server-db stop
```
  Important:
  If you are not running the embedded database service and you attempt to stop it, you receive a message indicating that the service cannot be found. If instead you get a message that the shutdown failed, the embedded database is still running, probably because services are connected to the Hive metastore. If the database shutdown fails due to connected services, issue the following command:
  - RHEL-compatible 7 and higher:
    sudo service cloudera-scm-server-db next_stop_fast sudo service cloudera-scm-server-db stop
  - All other Linux distributions:
    sudo service cloudera-scm-server-db fast_stop
4. If the Cloudera Manager host is also running the Cloudera Manager Agent, stop the Cloudera Manager Agent:
```
sudo service cloudera-scm-agent stop
```
Download tarballs from the locations listed in Cloudera Manager Version and Download Information.
Copy the tarballs and unpack them on all hosts on which you intend to install Cloudera Manager Server and Cloudera Manager Agents, in a directory of your choosing. If necessary, create a new directory to accommodate the files you extract from the tarball. For instance, if /opt/cloudera-manager does not exist, create it using a command similar to:
```
$ sudo mkdir /opt/cloudera-manager
```
Extract the contents of the tarball to this directory. For example, to copy a tar file to your home directory and extract the contents of all tar files to the /opt/ directory, use a command similar to the following:
```
$ sudo tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager
```
The files are extracted to a subdirectory named according to the Cloudera Manager version being extracted. For example, files could be extracted to /opt/cloudera-manager/cm-5.0/. This full path is needed later and is referred to as tarball_root directory.
On every Cloudera Manager Agent host, configure the Cloudera Manager Agent to point to the Cloudera Manager Server by setting the following properties in the tarball_root/etc/cloudera-scm-agent/config.ini configuration file:

Property Description

server_host Name of the host where Cloudera Manager Server is running.

server_port Port on the host where Cloudera Manager Server is running.
By default, a tarball installation has a var subdirectory where state is stored. In a non-tarball installation, state is stored in /var. Cloudera recommends that you configure the tarball installation to use an external directory as the /var equivalent (/var or any other directory outside the tarball) so that when you upgrade Cloudera Manager, the new tarball installation can access this state. Configure the installation to use an external directory for storing state by editing tarball_root/etc/default/cloudera-scm-agent and setting the CMF_VAR variable to the location of the /var equivalent. If you do not reuse the state directory between different tarball installations, duplicate Cloudera Manager Agent entries can occur in the Cloudera Manager database.
Start Cloudera Manager Server.
The way in which you start the Cloudera Manager Server varies depending on which account you want the Server to run under:
- As root:
```
sudo tarball_root/etc/init.d/cloudera-scm-server start 
```
- As another user. If you run as another user, ensure the user you created for Cloudera Manager owns the location to which you extracted the tarball including the newly created database files. If you followed the earlier examples and created the directory /opt/cloudera-manager and the user cloudera-scm, you could use the following command to change ownership of the directory:
```
sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager
```
  Once you have established ownership of directory locations, you can start Cloudera Manager Server using the user account you chose. For example, you might run the Cloudera Manager Server as cloudera-service. In this case, you have the following options:
  - Run the following command:
```
$ sudo -u cloudera-service tarball_root/etc/init.d/cloudera-scm-server start 
```
  - Edit the configuration files so the script internally changes the user. Then run the script as root:
    1. Remove the following line from tarball_root/etc/default/cloudera-scm-server:
      export CMF_SUDO_CMD=" "
    2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-server to the user you want the server to run as. For example, to run as cloudera-service, change the user and group as follows:
      USER=cloudera-service GROUP=cloudera-service
    3. Run the server script as root:
      $ sudo tarball_root/etc/init.d/cloudera-scm-server start
- To start the Cloudera Manager Server automatically after a reboot:
  1. Run the following commands on the Cloudera Manager Server host:
    - RHEL-compatible and SLES
      $ cp tarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server $ chkconfig cloudera-scm-server on
    - Debian/Ubuntu
      $ cp tarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server $ update-rc.d cloudera-scm-server defaults
  2. On the Cloudera Manager Server host, open the /etc/init.d/cloudera-scm-server file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} to tarball_root/etc/default.
To stop the Cloudera Manager Agent, run this command on each Agent host:
```
$ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed
```
If you are running single user mode, stop Cloudera Manager Agent using the user account you chose. For example, if you are running the Cloudera Manager Agent as cloudera-scm, you have the following options:
- Run the following command:
```
$ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed 
```
- Edit the configuration files so the script internally changes the user, and then run the script as root:
  1. Remove the following line from tarball_root/etc/default/cloudera-scm-agent:
```
export CMF_SUDO_CMD=" "
```
  2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:
```
USER=cloudera-scm
GROUP=cloudera-scm
```
  3. Run the Agent script as root:
```
$ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed 
```
Start the Cloudera Manager Agent according to the account you want the Agent to run under:
- To start the Cloudera Manager Agent, run this command on each Agent host:
```
$ sudo tarball_root/etc/init.d/cloudera-scm-agent start
```
  When the Agent starts, it contacts the Cloudera Manager Server.
- If you are running single user mode, start Cloudera Manager Agent using the user account you chose. For example, to run the Cloudera Manager Agent as cloudera-scm, you have the following options:
  - Run the following command:
```
$ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent start 
```
  - Edit the configuration files so the script internally changes the user, and then run the script as root:
    1. Remove the following line from tarball_root/etc/default/cloudera-scm-agent:
```
export CMF_SUDO_CMD=" "
```
    2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:
```
USER=cloudera-scm
GROUP=cloudera-scm
```
    3. Run the Agent script as root:
```
$ sudo tarball_root/etc/init.d/cloudera-scm-agent start 
```
- To start the Cloudera Manager Agents automatically after a reboot:
  1. Run the following commands on each Agent host:
    - RHEL-compatible and SLES
```
$ cp tarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
$ chkconfig cloudera-scm-agent on
```
    - Debian/Ubuntu
```
$ cp tarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
$ update-rc.d cloudera-scm-agent defaults
```
  2. On each Agent, open the tarball_root/etc/init.d/cloudera-scm-agent file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} to tarball_root/etc/default.
Log in to the Cloudera Manager Admin Console.
Restart all services:
1. On the Home > Status tab, click next to the cluster name and select Restart.
2. In the confirmation dialog box that displays, click Restart.
Upgrade any required Cloudera Navigator components:

Property	Description
`server_host`	Name of the host where Cloudera Manager Server is running.
`server_port`	Port on the host where Cloudera Manager Server is running.

Step 7: Verify and Test the Upgrade

If the commands to update and start the Cloudera Manager Server complete without errors, the upgrade has completed successfully. To verify, check that the server versions have been updated.

Verify that the agents are sending heartbeats to Cloudera Manager:
1. Go to Hosts > All Hosts.
2. Click the column header labeled Last Heartbeat to sort it.
3. Verify that the last heartbeat for each host has occurred within one minute.
In the Cloudera Manager Admin Console, click the Hosts tab.
Click Inspect All Hosts. On large clusters, the host inspector can take some time to finish running. You must wait for the process to complete before proceeding to the next step.
Click Show Inspector Results. All results from the host inspector process are displayed, including the currently installed versions. If this includes listings of current component versions, the installation completed as expected.
Verify that the monitoring features are working as expected; follow the instructions in Testing the Installation.

Step 8: Upgrading Cloudera Navigator Components

Upgrade any required Cloudera Navigator components:

Cloudera Manager Key Trustee Server
Cloudera Navigator Key HSM
Cloudera Navigator Key Trustee KMS
Cloudera Navigator Encrypt.

See Upgrading Cloudera Navigator Components.

The Cloudera Navigator Data Management Component is upgraded automatically when you upgrade Cloudera Manager.

Step 9: (Optional) Upgrade CDH

To upgrade CDH, see Upgrading CDH Using Cloudera Manager.

Upgrade Guide

CLI Installation Guide