Hadoop - Revision history

imported>T1t0: /* References */

2018-08-19T04:27:14Z

References

← Older revision		Revision as of 04:27, 19 August 2018
Line 1,090:		Line 1,090:
	* https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment		* https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
	* http://ksat.me/map-reduce-a-really-simple-introduction-kloudo/ A really nice example to explain MapReduce concept		* http://ksat.me/map-reduce-a-really-simple-introduction-kloudo/ A really nice example to explain MapReduce concept

			* an updated article: https://dzone.com/articles/install-a-hadoop-cluster-on-ubuntu-18041

2018-07-08T02:54:54Z

2015-11-28T03:00:11Z

Define YARN scheduler configuration tags /srv/hadoop/etc/hadoop/yarn-site.xml

← Older revision		Revision as of 03:00, 28 November 2015
Line 551:		Line 551:
	<value>hadoopmaster:8032</value>		<value>hadoopmaster:8032</value>
	</property>		</property>

			<property>
			<name>yarn.scheduler.maximum-allocation-mb</name>
			<value>4096</value>
			</property>

	</pre>		</pre>

2015-11-21T14:58:10Z

Creating a python example

← Older revision		Revision as of 14:58, 21 November 2015
Line 980:		Line 980:
	# writing-an-hadoop-mapreduce-program-in-python/		# writing-an-hadoop-mapreduce-program-in-python/

	~~from operator import itemgetter~~
	import sys		import sys

Line 1,017:		Line 1,016:
	if current_word == word:		if current_word == word:
	print '%s\t%s' % (current_word, current_count)		print '%s\t%s' % (current_word, current_count)

	</pre>		</pre>

2015-11-21T13:37:33Z

Creating a python example

← Older revision		Revision as of 13:37, 21 November 2015
Line 951:		Line 951:
	#!/usr/bin/env python		#!/usr/bin/env python
	#		#
	# from http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/		# from http://www.michael-noll.com/tutorials/\
			# writing-an-hadoop-mapreduce-program-in-python/

	import sys		import sys
Line 976:		Line 977:
	#!/usr/bin/env python		#!/usr/bin/env python
	#		#
	# from http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/		# from http://www.michael-noll.com/tutorials/\
			# writing-an-hadoop-mapreduce-program-in-python/

	from operator import itemgetter		from operator import itemgetter

2015-11-21T13:32:24Z

Creating a python example

← Older revision		Revision as of 13:32, 21 November 2015
Line 940:		Line 940:

	== Creating a python example ==		== Creating a python example ==

	* from http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

	<pre>		<pre>
Line 952:		Line 950:
	<pre>		<pre>
	#!/usr/bin/env python		#!/usr/bin/env python
			#
			# from http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

	import sys		import sys
Line 972:		Line 972:

	* create file reducer.py, mode 755		* create file reducer.py, mode 755

	<pre>		<pre>
	#!/usr/bin/env python		#!/usr/bin/env python
			#
			# from http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

	from operator import itemgetter		from operator import itemgetter

2015-10-31T14:48:49Z

Commands executed at any Slave host

← Older revision		Revision as of 14:48, 31 October 2015
Line 568:		Line 568:
	----		----

	== Commands executed at any ~~Slave~~ host ==		== Commands executed at any Worker host ==

	----		----

	=== Install and apply ~~hadoop~~ configs ===		=== Install Hadoop and apply configs ===

	<pre>		<pre>

2015-10-31T14:44:03Z

Installing hadoop binary package

@@ Line 337: / Line 337: @@
 wget -c http://www.us.apache.org/dist/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz
 exit
 scp hadoop-2.7.1.tar.gz NODE_HOSTNAME_1:                                        # copy to node 1
 scp hadoop-2.7.1.tar.gz NODE_HOSTNAME_2:                                        # copy to node 2
@@ Line 342: / Line 355: @@
 scp hadoop-2.7.1.tar.gz NODE_HOSTNAME_N:                                        # copy to node N
 exit
 </pre>

2015-10-31T14:34:43Z

Hadoop as Cluster - Install and config

← Older revision		Revision as of 14:34, 31 October 2015
Line 282:		Line 282:
	sudo chown -R hadoop:hadoop /tmp/ssh.tgz		sudo chown -R hadoop:hadoop /tmp/ssh.tgz
	sudo mv /tmp/ssh.tgz /srv/hadoop/home		sudo mv /tmp/ssh.tgz /srv/hadoop/home
	su - hadoop		sudo su - hadoop
	tar -xzf ssh.tgz		tar -xzf ssh.tgz
	rm -f ssh.tgz		rm -f ssh.tgz

2015-10-31T14:16:45Z

Creating common ssh authentication for hadoop user

← Older revision		Revision as of 14:16, 31 October 2015
Line 250:		Line 250:
	exit		exit
	</pre>		</pre>


	* copy the file in all nodes		* copy the file in all nodes

	<pre>		<pre>
	~~sudo~~ scp /srv/hadoop/home/ssh.tgz ~~root@~~NODE_HOSTNAME_1:/tmp # copy to node 1		scp /srv/hadoop/home/ssh.tgz NODE_HOSTNAME_1:/tmp # copy to node 1
	~~sudo~~ scp /srv/hadoop/home/ssh.tgz ~~root@~~NODE_HOSTNAME_2:/tmp # copy to node 2		scp /srv/hadoop/home/ssh.tgz NODE_HOSTNAME_2:/tmp # copy to node 2
	... # ...		... # ...
	~~sudo~~ scp /srv/hadoop/home/ssh.tgz ~~root@~~NODE_HOSTNAME_N:/tmp # copy to node N		scp /srv/hadoop/home/ssh.tgz NODE_HOSTNAME_N:/tmp # copy to node N
	sudo rm /srv/hadoop/home/ssh.tgz # remove the file		sudo rm /srv/hadoop/home/ssh.tgz # remove the file
	</pre>		</pre>