Category: Big Data

Configure eclipse to run map reduce programs

As hadoop is written in java, we can execute map reduce programs from eclipse itself. All you need to do is to create a plugin that can support map reduce programs. There are lot of ways to  create map reduce plugin, after doing lot of research I came up with simple solution

Download hadoop plugin from here and save it on your desktop. Move plugin to the plugins folder of the eclipse.

Start hadoop and make sure all nodes are up. Start eclipse and create new Project called “Map Reduce”. This project comes into picture when you place plugin at a right place

Now fill up the blanks as shown in the image and that’s it you will get a DFS in your project explorer in which you upload, delete files in your Hadoop Distributed File System.

Configure eclipse for hadoop

create eclipse shortcut in ubuntu

Lot of us download eclipse as zip file extract it to a location and start using it. Every time we have to go to the eclipse path and open the application. Ubuntu doesn’t provide create shortcut as Windows do….

Don’t worry there are few steps which will allow you to create shortcut at a place where you want

Install gnome-tweak-tool using command: sudo apt-get install gnome-tweak-tool

after installing tool use the following command to get popup for creating shortcut:

/usr/bin/gnome-desktop-item-edit /home/user account name(xxxx)/Desktop/ –create-new

shortcut_eclipseFor command press browse button and select path for eclipse and press OK. Done !!!! your shortcut is waiting on your desktop

Install Hadoop in Ubuntu 12.04

Prerequisites to learn hadoop are having good knowledge in Java and basics of Linux commands.

Lot of us face problems while installing Hadoop in our systems even I was one of them. Here I am going to tell you step by step produce to get error free hadoop environment in Ubuntu 12.04.

Install Java in your system:

  • Download Oracle JDK from Oracle download page (I am using  jdk-7u25-linux-i586.tar.gz)
  • Open Terminal (Ctrl+T) and go to the folder where you downloaded Java.
  • Extract zip file, we will get a folder like jdk1.7.0_xx (In my case jdk1.7.0_25). Use the following command for extracting java.

                     tar -xvf jdk-7u25-linux-i586.tar.gz

  • Make a directory /usr/lib/jvm

                     sudo mkdir -p /usr/lib/jvm

  • Move extracted java folder to the directory created above

                     sudo mv jdk1.7.0_25 /usr/lib/jvm

  • Update alternatives of java, javac, javaws and jps to Ubuntu environment

                     sudo update-alternatives –install “/usr/bin/java” “java” “/usr/lib/jvm/jdk1.7.0_25/bin/java” 1

                     sudo update-alternatives –install “/usr/bin/javac” “javac” “/usr/lib/jvm/jdk1.7.0_25/bin/javac” 1

                     sudo update-alternatives –install “/usr/bin/javaws” “javaws” “/usr/lib/jvm/jdk1.7.0_25/bin/javaws” 1

                     sudo update-alternatives –install “/usr/bin/jps” “jps” “/usr/lib/jvm/jdk1.7.0_25/bin/jps” 1

Now we successfully got java configured in our system.

Prerequisites for Hadoop Installation:

  • Hadoop cluster works on SSH Networks so we install SSH and create password less sessions.

                     sudo apt-get install ssh

  • Add dedicated user and group for hadoop related operations lets consider a user called hduser and group as hadoop, we can use any names in place of hduser and hadoop

                     sudo addgroup hadoop  This adds group called hadoop

                     sudo adduser –ingroup hadoop hduser This adds user called hduser and keeps it in hadoop group

  • Login to hduser account through Terminal

                     su hduser Asks for a password, please give password that you have created for hduser in the above step

  • To create password-less SSH connection, create RSA key for hduser. It will ask for “file in which to save the key” just hit enter.

                     ssh-keygen -t rsa -P “”

  • You have to enable SSH access to your local machine with this newly created key

                     cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

  • Lets test above configured SSH setup by connecting to your local machine with hduser user. It will ask for confirmation, type yes to continue.

                     ssh localhost

Now we are set to install hadoop, if we reach this mark it’s very easy to configure hadoop. Continue reading “Install Hadoop in Ubuntu 12.04”