Single node hadoop cluster using Cloudera Manager

  1. Launch an EC2 instance with minimum of 8/16GB RAM. (m4.large/m4.xlarge)
  2. Add the yum repository inside /etc/yum.repos.d/cloudera.repo like below:
    • Choose appropriate CDH version as per your requirement from RHEL6 / RHEL7
    • [cloudera-cdh5]
      # Packages for Cloudera Manager for Hadoop, Version 5.8.2, on RedHat or CentOS 6 x86_64
      name=Cloudera Manager for Hadoop, Version 5.8.2
      baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.8.2/
      gpgkey=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
      gpgcheck=1
  3. Clean YUM repo : sudo yum clean all
  4. Java Installation
    • sudo yum install oracle-j2sdk1.7-1*
  5. Mysql install
    • yum install mysql-server
    • service mysqld start
    • /usr/bin/mysql_secure_installation – Set password
  6. Create cmdb,hivedb,huedb and whatever required as per the services you need
    1. create database cmdb;
      create user 'cmuser' identified by 'myPass@123';
      grant all on cmdb.* to 'cmuser' identified by 'myPass@123';
      grant all on cmdb.* to 'cmuser'@'localhost' identified by 'myPass@123';
      grant all on cmdb.* to 'cmuser'@'%' identified by 'myPass@123';
      grant all on cmdb.* to 'cmuser'@'<<hostname>>' identified by 'myPass@123';
      
      create database hivedb;
      create user 'hive' identified by 'myPass@123';
      grant all on hivedb.* to 'hive' identified by 'myPass@123';
      grant all on hivedb.* to 'hive'@'localhost' identified by 'myPass@123';
      grant all on hivedb.* to 'hive'@'%' identified by 'myPass@123';
      grant all on hivedb.* to 'hive'@'<<hostname>>' identified by 'myPass@123';
  7. Install cloudera-manager-server
    1. yum install cloudera-scm-sever
  8. Install cloudera-manager-agent
    1. yum install cloudera-scm-agent
  9. Edit db.properties like below – sudo vi /etc/cloudera-scm-server/db.properties
    1. # Copyright (c) 2012 Cloudera, Inc. All rights reserved.
      #
      # This file describes the database connection.
      #
      # The database type
      # Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
      com.cloudera.cmf.db.type=mysql
      # The database host
      # If a non standard port is needed, use 'hostname:port'
      com.cloudera.cmf.db.host=localhost
      # The database name
      com.cloudera.cmf.db.name=cmdb
      
      # The database user
      com.cloudera.cmf.db.user=cmuser
      
      # The database user's password
      com.cloudera.cmf.db.password=myPass@123

       

  10. sudo yum install mysql-connector-java
  11. Start the scm-server
    1. sudo service cloudera-scm-server start
      sudo service cloudera-scm-server start
  12. Start the scm-agent
    1. sudo service cloudera-scm-server start
      sudo service cloudera-scm-server start
  13. http://<<hostname/ipaddress>&gt;:7180/ (check inbound rules/disable firewall if not opening)
  14. Continue the steps mentioned and complete the installation.
  15. Below are some commands which might be helpful for setting proper permissions for directories if something gets wrong during installation.
    1. chown -R yarn:yarn hadoop-yarn
      chown -R mapred:mapred hadoop-mapreduce
      chown -R hdfs:hdfs hadoop-hdfs
      chown -R httpfs:httpfs hadoop-httpfs
      chown -R kms:kms hadoop-kms
      chmod 770 /var/lib/hadoop-yarn
      chmod 770 /var/lib/hadoop-hdfs
      chmod 770 spark/
      chmod 770 oozie/
      chmod 770 hive/
      chmod 770 impala
      chmod 770 hbase

       

  16. This is completely verified, please let me know in the comments if you face any issues or need any help.

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s