Ensure that you have met the following prerequisites:

• You must have these access privileges:

  • Sudo access to the client system

  • Login access to Cloudera Manager

  If you do not have these privileges, then contact your system administrator for help.

• The client system must run an operating system that Cloudera supports for CDH 6.

  https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/installation.html

• The client system must run a compatible version of Oracle JDK 1.8. See the Cloudera documentation Cloudera documentation for a list of versions that are compatible with CDH 6.x releases.

  To verify the version, use this command:

  $ java -version
  

• In the client configuration, ensure that the HDFS property dfs.client.use.datanode.hostname is set to “true”.

  <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>true</value>
    <description>Whether clients should use datanode hostnames when
   connecting to datanodes.
    </description>
  </property>

  This is property is already set to “true” if you download the configuration from Cloudera Manager on Oracle Big Data Appliance. It may not be set to “true” if you acquire the configuration from other sources, including Cloudera.

To install a CDH client on any operating system identified as supported by Cloudera, follow these instructions.

1. Log in to the client system.

2. If an earlier version of Hadoop is already installed, then remove it.

   See the Cloudera documentation for removing an earlier CDH version at Uninstalling Cloudera Software and Managed Software

3. Follow the instructions provided in Document 1943912.1 at My Oracle Support.

After installing CDH, you must configure it for use with Oracle Big Data Appliance.

The commands in this procedure that reference HADOOP_HOME are used to support older Hadoop clients that require this environment variable. The cluster uses YARN (MRv2) and does not use HADOOP_HOME. If no older clients access the cluster, then you can omit these commands.

To configure the Hadoop client:

1. Log in to the client system and download the MapReduce client configuration from Cloudera Manager. In this example, Cloudera Manager listens on port 7180 (the default) of bda01node03.example.com, and the configuration is stored in a file named yarn-conf.zip.

   $ wget -O yarn-conf.zip http://bda01node03.example.com:7180/cmf/services/3/client-config

2. Unzip mapreduce-config.zip into a permanent location on the client system.

   $ unzip yarn-config.zip
   Archive:  yarn-config.zip
     inflating: yarn-conf/hadoop-env.sh
     inflating: yarn-conf/hdfs-site.xml
     inflating: yarn-conf/core-site.xml
     inflating: yarn-conf/mapred-site.xml
     inflating: yarn-conf/log4j.properties
     inflating: yarn-conf/yarn-site.xml

   All files are stored in a subdirectory named yarn-config.

3. Make a backup copy of the Hadoop configuration files:

   # cp /full_path/yarn-conf /full_path/yarn-conf-bak

4. Overwrite the existing configuration files with the downloaded configuration files.

   # cd /full_path/yarn-conf
   # cp * /usr/lib/hadoop/conf

Follow these steps to enable the CDH client to work with a secure CDH cluster.

To configure a CDH client for Kerberos:

1. Log in to the system where you created the CDH client.

2. Install the Java Cryptography Extension Unlimited Strength Jurisdiction Policy Files:

   1. Download the files for your Java version:

      Java 6: http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.html

      Java 7: http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html

      Java 8: http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

   2. Decompress the downloaded file. This example unzips JCE-8:

      $ unzip UnlimitedJCEPolicyJDK8.zip
      Archive:  UnlimitedJCEPolicyJDK8.zip
         creating: UnlimitedJCEPolicy/
        inflating: UnlimitedJCEPolicy/US_export_policy.jar
        inflating: UnlimitedJCEPolicy/local_policy.jar
        inflating: UnlimitedJCEPolicy/README.txt

      Note:

      The JCE-6 files unzip into a directory named jce instead of UnlimitedJCEPolicy.

   3. Copy the unzipped files into the Java security directory. For example:

      $ cp UnlimitedJCEPolicy/* /usr/java/latest/jre/lib/security/

3. Follow the steps for configuring an unsecured client.

   See "Configuring a CDH Client for an Unsecured Cluster."

4. Ensure that you have a user ID on the CDH cluster that had been added to the Kerberos realm.

   See "Creating Hadoop Cluster Users."

5. On the CDH client system, create a file named krb5.conf in the $HADOOP_CONF_DIR directory. Enter configuration settings like the following, using values appropriate for your installation for the server names, domain, and realm:

   [libdefaults]
        default_realm = EXAMPLE.COM
        dns_lookup_realm = false
        dns_lookup_kdc = false
        clockskew = 3600
        ticket_lifetime = 24h
        renew_lifetime = 7d
        forwardable = true
   [realms]
        EXAMPLE.COM = {
             kdc = bda01node01.example:88
             admin_server = bda01node07:749
             default_domain = example.com
    }
   [domain_realm]
         .com = EXAMPLE.COM 

6. Activate the new configuration file:

   export KRB5_CONFIG=$HADOOP_CONF_DIR/krb5.conf
   export HADOOP_OPTS="-Djava.security.krb5.conf=$HADOOP_CONF_DIR/krb5.conf"
   export KRB5CCNAME=$HADOOP_CONF_DIR/krb5cc_$USER

7. Verify that you have access to the Oracle Big Data Appliance cluster.

   See "Verifying Access to a Cluster from the CDH Client."

Follow this procedure to ensure that you have access to the Oracle Big Data Appliance cluster.

To verify cluster access:

1. To access a Kerberos-protected CDH cluster, first obtain a ticket granting ticket (TGT):

   $ kinit userid@realm

2. Verify that you can access HDFS on Oracle Big Data Appliance from the client, by entering a simple Hadoop file system command like the following:

   $ hadoop fs -ls /user
   Found 6 items
   drwxr-xr-x   - jdoe      hadoop              0 2014-04-03 00:08 /user/jdoe
   drwxrwxrwx   - mapred    hadoop              0 2014-04-02 23:25 /user/history
   drwxr-xr-x   - hive      supergroup          0 2014-04-02 23:27 /user/hive
   drwxrwxr-x   - impala    impala              0 2014-04-03 10:45 /user/impala
   drwxr-xr-x   - oozie     hadoop              0 2014-04-02 23:27 /user/oozie
   drwxr-xr-x   - oracle    hadoop              0 2014-04-03 11:49 /user/oracle

   Check the output for HDFS users defined on Oracle Big Data Appliance, and not on the client system. You should see the same results as you would after entering the command directly on Oracle Big Data Appliance.

3. Submit a MapReduce job. You must be logged in to the client system under the same user name as your HDFS user name on Oracle Big Data Appliance.

   The following example calculates the value of pi:

   $ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*jar pi 10 1000000
   Number of Maps  = 10
   Samples per Map = 1000000
   Wrote input for Map #0
   Wrote input for Map #1
        .
        .
        .
   Job Finished in 12.403 seconds
   Estimated value of Pi is 3.14158440000000000000

4. Use Cloudera Manager to verify that the job ran on Oracle Big Data Appliance instead of the local system. Select mapreduce Jobs from the Activities menu for a list of jobs.

The following figure shows the job created by the previous example.

Figure 4-1 Monitoring a YARN Job in Cloudera Manager

Description of "Figure 4-1 Monitoring a YARN Job in Cloudera Manager"

When creating user accounts, define them as follows:

• To run MapReduce jobs, users must either be in the hadoop group or be granted the equivalent permissions.

• To create and modify tables in Hive, users must either be in the hive group or be granted the equivalent permissions.

• To create Hue users, open Hue in a browser and click the User Admin icon. See "Using Cloudera Hue to Interact With Hadoop."

Users do not need login privileges on Oracle Big Data Appliance to run MapReduce jobs from a remote client. However, for those who want to log in to Oracle Big Data Appliance, you must set a password. You can set or reset a password the same way.

To set a user password across all Oracle Big Data Appliance servers:

1. Create a Hadoop cluster user as described in "Creating Hadoop Cluster Users.".

2. Confirm that the user does not have a password:

   # dcli passwd -S user_name
   bda1node01.example.com: jdoe NP 2013-01-22 0 99999 7 -1 (Empty password.)
   bda1node02.example.com: jdoe NP 2013-01-22 0 99999 7 -1 (Empty password.)
   bda1node03.example.com: jdoe NP 2013-01-22 0 99999 7 -1 (Empty password.)

   If the output shows either "Empty password" or "Password locked," then you must set a password.

3. Set the password:

    hash=$(echo 'password' | openssl passwd -1 -stdin); dcli "usermod --pass='$hash' user_name"

4. Confirm that the password is set across all servers:

   # dcli passwd -S user_name
   bda1node01.example.com: jdoe PS 2013-01-24 0 99999 7 -1 (Password set, MD5 crypt.)
   bda1node02.example.com: jdoe PS 2013-01-24 0 99999 7 -1 (Password set, MD5 crypt.)
   bda1node03.example.com: jdoe PS 2013-01-24 0 99999 7 -1 (Password set, MD5 crypt.)

When the trash facility is enabled, you can easily restore files that were previously deleted.

To restore a file from the trash directory:

1. Check that the deleted file is in the trash. The following example checks for files deleted by the oracle user:

   $ hadoop fs -ls .Trash/Current/user/oracle
   Found 1 items
   -rw-r--r--  3 oracle hadoop  242510990 2012-08-31 11:20 /user/oracle/.Trash/Current/user/oracle/ontime_s.dat

2. Move or copy the file to its previous location. The following example moves ontime_s.dat from the trash to the HDFS /user/oracle directory.

   $ hadoop fs -mv .Trash/Current/user/oracle/ontime_s.dat /user/oracle/ontime_s.dat

The trash interval is the minimum number of minutes that a file remains in the trash directory before being deleted permanently from the system. The default value is 1 day (24 hours).

To change the trash interval:

1. Open Cloudera Manager. See "Managing Operations Using Cloudera Manager".

2. On the Home page under Status, click hdfs.

3. On the hdfs page, click the Configuration subtab, and then select View and Edit.

4. Search for or scroll down to the Filesystem Trash Interval property under NameNode Default Group. See Figure 4-2.

5. Click the current value, and enter a new value in the pop-up form.

6. Click Save Changes.

7. Expand the Actions menu at the top of the page and choose Restart.

8. Open a connection as root to a node in the cluster.

9. Deploy the new configuration:

   dcli -C bdagetclientconfig

The following figure shows the Filesystem Trash Interval property in Cloudera Manager.

Figure 4-2 HDFS Property Settings in Cloudera Manager

Description of "Figure 4-2 HDFS Property Settings in Cloudera Manager"