Spark - Revision history

imported>T1t0 at 03:26, 8 July 2018

2018-07-08T03:26:33Z

2016-02-27T22:48:18Z

Creating spakr user

← Older revision		Revision as of 22:48, 27 February 2016
Line 58:		Line 58:
	== Commands to run on Master node ==		== Commands to run on Master node ==

	=== Creating ~~spakr~~ user ===		=== Creating Spark user ===

	<pre>		<pre>

2016-02-10T16:12:06Z

@@ Line 139: / Line 139: @@
 # Set the JAVA_HOME
 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre  # or /usr/lib/jvm/java-7-oracle/jre
 # Spark vars
@@ Line 281: / Line 267: @@
 # Set the JAVA_HOME
 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre  # or /usr/lib/jvm/java-7-oracle/jre
 # Spark vars

2016-02-10T13:45:23Z

@@ Line 3: / Line 3: @@
 * java >= 7 (OpenJDK is ok)
 * Spark bin package == 1.5.1 build for Hadoop 2.6 and later
-* Hadoop 2.7.1 already installed
+* Hadoop 2.7.1 already installed in '/srv/hadoop'
 * ssh server (ex openssh-server)
 * Scala? (which version?)
@@ Line 139: / Line 139: @@
 # Set the JAVA_HOME
 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre  # or /usr/lib/jvm/java-7-oracle/jre
 export SPARK_HOME=/srv/spark/spark-current
 export PATH=$SPARK_HOME/bin:$PATH
@@ Line 265: / Line 281: @@
 # Set the JAVA_HOME
 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre  # or /usr/lib/jvm/java-7-oracle/jre
 export SPARK_HOME=/srv/spark/spark-current
 export PATH=$SPARK_HOME/bin:$PATH
@@ Line 526: / Line 558: @@
 # Set the JAVA_HOME
 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre  # or /usr/lib/jvm/java-7-oracle/jre
 # Spark vars

2016-02-08T14:05:36Z

Create the Program

← Older revision		Revision as of 14:05, 8 February 2016
Line 586:		Line 586:
	----		----

	=== Create the Program ===		=== Create the Python Program ===

	* Create a source code file 'wordcount.py' with the follow content:		* Create a source code file 'wordcount.py' with the follow content:

2016-02-08T14:05:20Z

Create the Program

← Older revision		Revision as of 14:05, 8 February 2016
Line 770:		Line 770:
	----		----

	=== Create the Program ===		=== Create the Java Program ===

	* Create a source code file 'src/main/java/WordCount.java' with the follow content:		* Create a source code file 'src/main/java/WordCount.java' with the follow content:
Line 837:		Line 837:
	public static void main(String[] args) {		public static void main(String[] args) {
	if(args.length != 1) {		if(args.length != 1) {
	System.err.println("Usage: ~~SimpleApp~~ <file>" + "\n" +		System.err.println("Usage: WordCount <file>" + "\n" +
	" where file can be a local 'file:///LOCAL_PATH' path" + "\n" +		" where file can be a local 'file:///LOCAL_PATH' path" + "\n" +
	" or a remote 'hdfs://REMOTE_PATH' path");		" or a remote 'hdfs://REMOTE_PATH' path");

2016-02-08T14:04:48Z

Create the Program

← Older revision		Revision as of 14:04, 8 February 2016
Line 654:		Line 654:
	if len(sys.argv) != 2:		if len(sys.argv) != 2:
	sys.stderr.write(		sys.stderr.write(
	"Usage: ~~SimpleApp~~ <file>" + "\n" +		"Usage: WordCount <file>" + "\n" +
	" where file can be a local 'file:///LOCAL_PATH' path" + "\n" +		" where file can be a local 'file:///LOCAL_PATH' path" + "\n" +
	" or a remote 'hdfs://REMOTE_PATH' path")		" or a remote 'hdfs://REMOTE_PATH' path")

2016-02-08T04:11:06Z

← Older revision		Revision as of 04:11, 8 February 2016
Line 575:		Line 575:
	log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n		log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
	</pre>		</pre>

			----

			= Create your first application =

	----		----

2016-02-08T04:10:26Z

2016-02-08T04:08:14Z

python code example added