首页 > 其他 > 详细

SPARK学习笔记-20160812

时间:2016-08-12 18:22:16      阅读:259      评论:0      收藏:0      [点我收藏+]

What is Spark?

1,Apache Spark is a fast and general engine for large-scale data processing

2,Speed

3,Ease of Use

4,Generality

5,Integrated with Hadoop

 

One Stack to rule them all:        Spark Streaming     ======>     Ad-hoc Queries

                          \\      SPARK      //

                           \\              //

                            =>  Batch Processing <=

 

Hadoop Data Sharing And Spark Data Sharing:

技术分享技术分享

ps:Hadoop频繁与磁盘上的数据进行交互,频繁对数据进行序列化和反序列化

 

Hadoop的Map Reduce类比图技术分享

 

Why Spark fast ?  

1,Memory based computation

2,DAG   有向无环图(Directed Acyclic Graph)

3,Thread model

4,Optimization (eg:delay scheduling)

 

API: Scala  Python  Java  R

 

Cluster Manager: Local  Standalone  Yarn  Mesos

 

技术分享技术分享

 

Dependency:norrow  wide   技术分享

 

技术分享

SPARK学习笔记-20160812

原文:http://www.cnblogs.com/wsongmao/p/5765777.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!