Posts

Showing posts from October, 2016

Install Apache hadoop Mahout in MacOS without Brew

Image
Here are the steps to install Apache Mahout
1. Download latest package from http://mirror.nexcess.net/apache/mahout/
2. Extract the package
3. Create a directory and put into HDFS
<hadoop_directory> hdfs dfs -put /home/Hadoop/data/mydata.txt /mahout_data/ 4. Run clustering in mahout
<mahout_directory>/bin/mahout seqdirectory -i hdfs://localhost:9000/mahout_data/ -o hdfs://localhost:9000/clustered_data/ 5. The output file will be in clustered_data directory

Read .properties file in Java

Java class Properties file

[Solved] Hive installation error: java.net.URISyntaxException: Relative path in absolute URI

Error:
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
Solution: Edit and update hive-site.xml
<name>hive.exec.scratchdir</name> <value>/tmp/hive-${user.name}</value> <name>hive.exec.local.scratchdir</name> <value>/tmp/${user.name}</value> <name>hive.downloaded.resources.dir</name> <value>/tmp/${user.name}_resources</value>

Read a file and print all lines in Hadoop

Image
#1. Compile this code and run in hadoop like
#2. Put the "transaction.csv" into HDFS


hadoop jar Cat.jar org.myorg.Cat #3. You will get the output in the console itself

Hadoop - Class not found Exception with $ symbol

Error:
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.gamma.WordCount$Map
To solve this
Add:

job.setJarByClass(WordCount.class);

in main()

How to verify Hadoop is running properly?

Image
To verify hadoop's all node are running properly, type

$jps

The output should be like

How to set environment variable permanently in MAC osX

$cd ~
$open -a TextEdit.app .bash_profile

This will open a text editor
there you can edit your environment variable

Ex: export JAVA_HOME=/etc/java/java_path

Save the file

$source .bash_profile

Install Hadoop in Mac

Image
#1. Download the latest hadoop distribution from http://mirrors.ibiblio.org/apache/hadoop/common/ [Ex: hadoop-2.7.3/]

#2. Extract the compressed file anywhere
#3. Then change the following files

hadoop_distro/etc/hadoop/hdfs-site.xml 1 2 3 4 5