Well, I configured single-node Hadoop cluster to run on a Macbook once, for a OSCON tutorial on Apache Hadoop. We installed VMWare/Virtualbox though to run a CentOS VM.
Installing Apache Hadoop on OS XThis article explains how to install an HadoopSingle Node Cluster. This article is not specific to Talend and should be helpful, whatever your requirement is for using Hadoop. The topics discussed here are useful if you want to learn Hadoop and set up your own single node cluster, for learning and development.
I'm not quite sure I understand the question. I would use whatever OS you're most comfortable or familiar with for interacting with these technologies. While Hadoop can be installed and run on Windows and Linux, the extreme vast majority o. Feb 11, 2018 Over the weekend, I wanted to learn a little more about distributed computing, and Hadoop seemed like a good starting point. So now let’s see, how can we try to get Hadoop running on a MacOS. Install hadoop with the command, $brew install hadoop. Inside the folder usr/local/Cellar/hadoop/3.1.0/libexec/etc/hadoop and added the commands in the file hadoop-env.sh, export HADOOPOPTS='$HADOOPOPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=' export JAVAHOME='/Library/Java/JavaVirtualMachines/jdk1.8.0151.jdk/Contents/Home'.
OS X is, probably, not the first platform that you will be considering when you're building your large Hadoop cluster; however this is a useful exercise when you're taking your first look at Hadoop.
This tutorial has been written, by installing Hadoop on a MacBook Air running OS X 10.9.1 (Mavericks). You can also install Hadoop on Unix, Linux variants and on a Windows Server.
Prerequisites for Installing Hadoop
There are a few things you need to sort out before installing Hadoop.
Java Version
Check your Java version. You'll need Java 6 (1.6) or higher. At the time of writing, the latest version of Java was Java 7 (1.7).
Note Although this is a universal guide for installing Hadoop, this is primarily a site about Talend. At the time of writing, the latest version of Talend (5.4.1) only supports Java 6 so, if running Talend, you'll need to have two Java versions installed or use Hadoop with Java 6.
Mavericks By default, Mavericks does not include Java and, if you've upgrade to Mavericks, Java will be uninstalled. There are plenty of resources that will explain how to install Java, if you do not already know. Just Google it.
You should receive the following response.
Hadoop User
For security and administration reasons, it is recommended that you create an Hadoop Operating System User. You can create a new user from Launchpad->System Preferences->Users & Groups. If you create the user
hadoop , create the account as a Standard user. If you are running Hadoop on your own personal computer, you may choose to run Hadoop under your own regular account (this is what I've chosen to do). If you choose to run Hadoop under an account name other than hadoop , amend the commands in this tutorial accordingly.
If you are using the hadoop user, you should now log out and log back in using that account.
Open a Terminal Window
The following commands are entered from a command prompt, so you will need to open a Terminal window. You can do this from Launchpad->Other->Terminal.
SSH
To use Hadoop, it will be necessary for Hadoop to have the ability to establish SSH connections to localhost, and to do this without the need to provide a password or passphrase. OS X comes with SSH pre-installed, so there is no need to install any additional software.
Enter the following command.
You will be asked to
Enter file in which to save the key . The default value is /Users/hadoop/.ssh/id_rsa . You have now created an RSA key file that can be used by SSH. A passphrase is not required to use this key file -P ' .
You should receive the following response.
Now that the RSA key pair has been created, we can authorize it's use, using the following command.
You can test the connection and save the RSA key fingerprint by entering the following command. Respond with
yes Free download of notepad++ for mac. , when prompted to save the finger print. Note that if you have followed the preceeding steps correctly, you should not be asked to enter your password or a passphrase.
You should receive the following response.
You can now close this connection by entering the following command. Macos 10.12 sierra download for vmware.
Firefox for os x yosemite. You should receive the following response. Download trial versacheck platinum.
Download the Latest Version of Hadoop
The next step is to download the latest version of Hadoop. There are many ways in which you can install Hadoop and some are more simple than others. For the purposes of this exercise, and to get maximum understanding of Hadoop, I'm going to do a basic install from the Apache download. Ringtones for nokia phones free download.
This documentation has been written for an installation of Hadoop 2.2.0.
Go to the Hadoop Download Page, where you'll find all of the available downloads. I would recommend downloading the latest stable version of Hadoop.
Hadoop is a library framework from the Apache Foundation.
Installing Hadoop
Now that We've downloaded Hadoop, we have the following files. Note that We've downloaded the binary version rather than source code. When you're more familiar with Hadoop, you may want to start exploring the source code. Note that we have also downloaded the MD5 hash file. Remember that you should always validate software that has been downloaded from the Internet.
These instructions assume that you have downloaded Hadoop to your Downloads directory
$HOME/Downloads .
Validating the download
This is the MD5 entry from the file
hadoop-0.23.10.tar.gz.mds . Note that this file contains multiple hashes, depending on the program that you choose to use to perform the validation.
You can use the grep command to view the hash entry.
You should receive the following response.
We can now validate our downloaded file, using the
md5 command.
You should receive the following response. If the download is valid, the two hash values should match.
Uncompress & Extracting the Archive File
Hadoop is downloaded as a gzipcompressedtar file. Now enter the following command.
The downloaded file will now be uncompressed, removing the
.gz extension.
We will now change to the directory
/usr/local , where we will extract the archive file.
To write to the directory
/usr/local , you will need to raise your privileges, as writing to this directory is restricted. Privileges are raised using the sudo command. Enter the following command. Enter your password, when prompted.
You may, periodically, receive the following warning message. Remember that
sudo is a powerful command and should be used with caution, especially when installing software that has been downloaded from the Internet.
Create Symbolic Link
Hadoop will now be located in the directory
/usr/local/hadoop-2.2.0 . You may install as many versions of Hadoop as you wish, with each being installed it a unique directory. It has helpful to be able to refer to the current version of Hadoop simply as /usr/local/hadoop . To do this, we will create a Symbolic link. As you install and use later versions of Hadoop, you can simply re-point the Symbolic link to allow your programs to use the new version; whilst retaining previous versions as needed.
Set File Ownership
We will now set the ownership of the installed files.
![]()
Each file that we create has an owner and a group, which are used in conjunction with the file's permissions. For example, try entering the following command to see the owner and group of the Hadoop home directory.
You should receive the following response.
Mini Hadoop For Mac Os X 10 11 Download Free
We'll now give ownership of the installed files, including the Symbolic link, to Hadoop. This is achieved using the chown command. If user and group are different to those shown in this comment, amend the command accordingly. The correct values should be shown in the output of the previous command, in this example, the values are
hadoop and staff .
Now enter the following command, to see the effect of these recent steps.
You should receive the following response.
Next Steps
These steps have completed the installation of Hadoop. In the next tutorials, we'll look at Configuring Hadoop 2.x, running Hadoop and testing some basic commands.
Microsoft surface ergonomic keyboard mac mapping.
This tutorial contains step by step instructions for installing hadoop 2.x on Mac OS X El Capitan. These instructions should work on other Mac OS X versions such as Yosemite and Sierra. This tutorial uses pseudo-distributed mode for running hadoop which allows us to use a single machine to run different components of the system in different Java processes. We will also configure YARN as the resource manager for running jobs on hadoop.
Hadoop Component Versions
Hadoop Installation on Mac OS X Sierra & El Capitan
Step 1: Install Java
Mini Hadoop For Mac Os X Download
Hadoop 2.7.3 requires Java 7 or higher. Run the following command in a terminal to verify the Java version installed on the system.
Java version '1.8.0_121'
Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
If Java is not installed, you can get it from here.
Step 2: Configure SSH
When hadoop is installed in distributed mode, it uses a password less SSH for master to slave communication. To enable SSH daemon in mac, go to System Preferences => Sharing. Then click on Remote Login to enable SSH. Execute the following commands on the terminal to enable password less login to SSH,
Step 3 : Install Hadoop
Download hadoop 2.7.3 binary zip file from this link (200MB). Extract the contents of the zip to a folder of your choice.
Step 4: Configure Hadoop
First we need to configure the location of our Java installation in etc/hadoop/hadoop-env.sh. To find the location of Java installation, run the following command on the terminal,
Copy the output of the command and use it to configure JAVA_HOME variable in etc/hadoop/hadoop-env.sh.
Modify various hadoop configuration files to properly setup hadoop and yarn. These files are located in etc/hadoop.
etc/hadoop/core-site.xml
etc/hadoop/hdfs-site.xml
etc/hadoop/mapred-site.xml
etc/hadoop/yarn-site.xml
Note the use of disk utilization threshold above. This tells yarn to continue operations when disk utilization is below 98.5. This was required in my system since my disk utilization was 95% and the default value for this is 90%. If disk utilization goes above the configured threshold, yarn will report the node instance as unhealthy nodes with error 'local-dirs are bad'.
Step 5: Initialize Hadoop Cluster
From a terminal window switch to the hadoop home folder (the folder which contains various sub folders such as bin and etc). Run the following command to initialize the metadata for the hadoop cluster. This formats the hdfs file system and configures it on the local system. By default, files are created in /tmp/hadoop-<username> folder.
bin/hdfs namenode -format
It is possible to modify the default location of name node configuration by adding the following property in the hdfs-site.xml file. Similarly the hdfs data block storage location can be changed using dfs.data.dir property.
The following commands should be executed from the hadoop home folder.
Step 6: Start Hadoop Cluster
Run the following command from terminal (after switching to hadoop home folder) to start the hadoop cluster. This starts name node and data node on the local system.
To verify that the namenode and datanode daemons are running, execute the following command on the terminal. This displays running Java processes on the system.
jps
19203 DataNode
29219 Jps 19126 NameNode 19303 SecondaryNameNode
Step 7: Configure HDFS Home Directories
We will now configure the hdfs home directories. The home directory is of the form – /user/<username>. My user id on the mac system is jj. Replace it with your user name. Run the following commands on the terminal,
bin/hdfs dfs -mkdir /user/jj
Step 8: Run YARN Manager
Start YARN resource manager and node manager instances by running the following command on the terminal,
Run jps command again to verify all the running processes,
jps
19203 DataNode
29283 Jps 19413 ResourceManager 19126 NameNode 19303 SecondaryNameNode 19497 NodeManager
Step 9: Verify Hadoop Installation
Access the URL http://localhost:50070/dfshealth.html to view hadoop name node configuration. You can also navigate the hdfs file system using the menu Utilities => Browse the file system.
Access the URL http://localhost:8088/cluster to view the hadoop cluster details through YARN resource manager.
Step 10: Run Sample MapReduce Job
Hadoop installation contains a number of sample mapreduce jobs. We will run one of them to verify that our hadoop installation is working fine.
We will first copy a file from local system to the hdfs home folder. We will use core-site.xml in etc/hadoop as our input,
Verify that the file is in HDFS folder by navigating to the folder from the name node browser console.
Let us run a mapreduce program on this hdfs file to find the number of occurrences of the word 'configuration' in the file. A mapreduce program for word count is available in the hadoop samples.
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep ./core-site.xml output ‘configuration’
This runs the mapreduce on the hdfs file uploaded earlier and then outputs the results to the output folder inside the hdfs home folder. The file will be named as part-r-00000. This can be downloaded from the name node browser console or run the following command to copy it to the local folder.
Print the contents of the file. This contains the number of occurrences of the word 'configuration' in core-site.xml.
cat part*
Finally delete the uploaded file and the output folder from hdfs system,
bin/hdfs dfs -rmr output
Step 11: Stop Hadoop/YARN Cluster
Run the following commands to stop hadoop/YARN daemons. This stops name node, data node, node manager and resource manager.
sbin/stop-yarn.sh
Comments are closed.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |