As hadoop is written in java, we can execute map reduce programs from eclipse itself. All you need to do is to create a plugin that can support map reduce programs. There are lot of ways to create map reduce plugin, after doing lot of research I came up with simple solution
Download hadoop plugin from here and save it on your desktop. Move plugin to the plugins folder of the eclipse.
Start hadoop and make sure all nodes are up. Start eclipse and create new Project called “Map Reduce”. This project comes into picture when you place plugin at a right place
Now fill up the blanks as shown in the image and that’s it you will get a DFS in your project explorer in which you upload, delete files in your Hadoop Distributed File System.
Lot of us download eclipse as zip file extract it to a location and start using it. Every time we have to go to the eclipse path and open the application. Ubuntu doesn’t provide create shortcut as Windows do….
Don’t worry there are few steps which will allow you to create shortcut at a place where you want
Install gnome-tweak-tool using command: sudo apt-get install gnome-tweak-tool
after installing tool use the following command to get popup for creating shortcut:
/usr/bin/gnome-desktop-item-edit /home/user account name(xxxx)/Desktop/ –create-new
For command press browse button and select path for eclipse and press OK. Done !!!! your shortcut is waiting on your desktop
Prerequisites to learn hadoop are having good knowledge in Java and basics of Linux commands.
Lot of us face problems while installing Hadoop in our systems even I was one of them. Here I am going to tell you step by step produce to get error free hadoop environment in Ubuntu 12.04.
Install Java in your system:
- Download Oracle JDK from Oracle download page (I am using jdk-7u25-linux-i586.tar.gz)
- Open Terminal (Ctrl+T) and go to the folder where you downloaded Java.
- Extract zip file, we will get a folder like jdk1.7.0_xx (In my case jdk1.7.0_25). Use the following command for extracting java.
tar -xvf jdk-7u25-linux-i586.tar.gz
- Make a directory /usr/lib/jvm
sudo mkdir -p /usr/lib/jvm
- Move extracted java folder to the directory created above
sudo mv jdk1.7.0_25 /usr/lib/jvm
- Update alternatives of java, javac, javaws and jps to Ubuntu environment
sudo update-alternatives –install “/usr/bin/java” “java” “/usr/lib/jvm/jdk1.7.0_25/bin/java” 1
sudo update-alternatives –install “/usr/bin/javac” “javac” “/usr/lib/jvm/jdk1.7.0_25/bin/javac” 1
sudo update-alternatives –install “/usr/bin/javaws” “javaws” “/usr/lib/jvm/jdk1.7.0_25/bin/javaws” 1
sudo update-alternatives –install “/usr/bin/jps” “jps” “/usr/lib/jvm/jdk1.7.0_25/bin/jps” 1
Now we successfully got java configured in our system.
Prerequisites for Hadoop Installation:
- Hadoop cluster works on SSH Networks so we install SSH and create password less sessions.
sudo apt-get install ssh
- Add dedicated user and group for hadoop related operations lets consider a user called hduser and group as hadoop, we can use any names in place of hduser and hadoop
sudo addgroup hadoop This adds group called hadoop
sudo adduser –ingroup hadoop hduser This adds user called hduser and keeps it in hadoop group
- Login to hduser account through Terminal
su hduser Asks for a password, please give password that you have created for hduser in the above step
- To create password-less SSH connection, create RSA key for hduser. It will ask for “file in which to save the key” just hit enter.
ssh-keygen -t rsa -P “”
- You have to enable SSH access to your local machine with this newly created key
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
- Lets test above configured SSH setup by connecting to your local machine with hduser user. It will ask for confirmation, type yes to continue.
Now we are set to install hadoop, if we reach this mark it’s very easy to configure hadoop. Continue reading “Install Hadoop in Ubuntu 12.04”