Spark in python

Just a simple example to get started.

#./pyspark --jars /tmp/mysql-connector-java-5.1.45-bin.jar
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.5.2
      /_/

Using Python version 2.6.6 (r266:84292, Jan 22 2014 09:42:36)
SparkContext available as sc, HiveContext available as sqlContext.
>>> dataframe_mysql = sqlContext.read.format("jdbc").options( url="jdbc:mysql://cmhlcarchapp01:3306/inventory",driver = "com.mysql.jdbc.Driver",dbtable = "inventory",user="root", password="welcome").load()
>>> dataframe_mysql.show()
+---+--------+--------+----------+--------+
|sku|location|in_stock|in_transit|reserved|
+---+--------+--------+----------+--------+
|  1|     955|      10|         1|       1|
|  1|    2009|      10|         1|       1|
|  1|     173|      10|         1|       1|
|  2|     955|       1|         0|       0|
|  2|     955|       1|         0|       0|
|  2|     955|       1|         0|       0|
|  2|     955|       1|         0|       0|
|  3|     173|      10|         5|       3|
|  3|     173|      10|         5|       3|
|  3|     173|      10|         5|       3|
|  3|     173|      10|         5|       3|
|  1|     955|       1|         1|       1|
|  1|     955|     100|         1|       1|
+---+--------+--------+----------+--------+

>>>

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.