Programming with Apache Spark and Cassandra -draft


Putting the knowledge gained so far in this  and frequent questions that many may ask and what we have asked ourselves.
1.1         What is the need for using Spark ?
Spark gives you horizontal scale ability in a programmer friendly way.
1.2         But what about other options ?
There are other options as well. I have listed them below, which describes and highlights Sparks place in the architecture

Type
Level of Granularity
Descritpion
LoadBalancer
(nginx,haproxy)
Request Level
(usually HTTP requests)
Works well for Request-response type client server protocols. Works also well in context of microservices in application program side
However to scale the processing insdie the application programs this is inadequate
Task Managers
(celery, other MQ based)
Task Level
Helps to scale processing in the application program.Takes care of Task handling. However the onus is on the developer to split application logic to independent tasks. Usually
only the simplest things are really split into tasks. Equally hard problem is combining the outputs
Cluster Computing
(Apache Spark,Hadoop)
Application Level,Function
level
Helps to scale processing in the application layer across. Takes care of all the above. The onus is still on the developer to use this properly. However if the few API*, map, foreach,reduce and groupBy/partionBy are used , the programmer can be written as if it is is running in a single node, in a single thread. The system manages shared RAM across mutiple nodes, shared cores, task scheduling, multithreading etc. *P.S - Spark has an extensive library for machine learning as well,which could be the gateway for future
Multithreading
Function level
Helps to scale the processing inside a single node across nodes. Usually has to be done with care to avoid the complexity of threading related problems which many programmers are unaware
Green Threads
Funciton level/Stack level
Ex Greenlets in Python ; Good for switching stack in IO bound applications ; example socket server etc; Not really parallel, but wait time in one stack frame can be used by other stacks waiting to execute. Rather specific for general purpose usage



How stable is Apache Spark and Apache Cassandra ?
Speaking from our limited experience in running the prototype, all of the Spark and Casandra JVMs survived 20 days of load runs, network problems , application exceptions we threw at them.And that too in a low end cloud lab. Looks to be well written
1.5         What is the most important thing to take care when using Apache Cassandra ?
Data modelling and connected the Primary key and partition key design. It is important to design your primary key and the partition key so that write are distributed as well as read are faster. This is explained well by the Cassandra expert here -> http://www.planetcassandra.org/blog/the-most-important-thing-to-know-in-cassandra-data-modeling-the-primary-key/
The hash of the partition key is used by Cassandra to identify the node in which to store. So choosing a partition key that distributes the load equally among nodes prevent write hotspots. Example can be seem in the performance run page
P,S - There are few trivial but important things , like writing commit log and data(SSTable)  in different partition. This link gives basic info about write path.
1.6         What is the most important thing to take care when using Apache Spark?
Have not come across as single important thing as such, but couple of pointers
1.    Avoid doing any major work in Spark driver , rdd.collect() or the more better rdd.toLocalIteraror() are not good ideas and don't scale; You get OOM error soon
2.    There is no way to share state like counters etc between driver and workers, though in the code it may seem so. Only way is via accumilators ; and there workers cannot read;
3.    The way you partition the RDD may be important for performance; esp for operation like group by etc ; need to test and understand this better


Comments

Post a Comment

Popular posts from this blog

Long running Java process resource consumption monitoring , leak detection and GC tuning

For easily monitoring the JVM metrics there is no better tool than Java VisualVM or its older counterpart JConsole; These two tools comes with the Java JDK. So it is absolutely necessary that you copy a JDK version to some temp directory in which your server runs. {Make sure that you do not install the JDK / put the java executable in the path.} Before going in JVM monitoring, it is essential to understand a little about Java's memory model. Note - This article is written with  HotSpot JVM in mind,which is one of the most commonly used one ; implemented by OpenJDK and Oracle - formerly from Sun. This is not the only one , notable being JRockit and others ; and some details would change with a different JVM. When you are running a Java Process in a Server you allocate the Memory for the Java Virtual Machine. This memory is split to process heap { what you give withe the Xmx Xms command}where it is again subdivided on generational basis (Young, Old)  and the PermGen -...
Read more

OpenLayers Advanced Clustering and Setting dynamic images for OpenLayer Styles via Ajax

Update: - Read the same at  Medium  for better readability; Blogger is not technology blogging friendly JSFiddle Source for this Tutorial Let us start with a simple OpenLayer map In the JSP or the HTML we have <head> <script src="http://openlayers.org/api/OpenLayers.js"></script> <script type="text/javascript" src="<%=request.getContextPath()%>/js/maplister.js"></script> <script type="text/javascript" src="<%=request.getContextPath()%>/js/jquery-1.7.2.min.js"></script> <script src="http://maps.google.com/maps/api/js?v=3&amp;sensor=false"></script> </head> <body> <div id="map-id" style="height:100%;width=100%"></div> </body> And on document load AUI().ready(function(X) { resize("scMapListerportlet", "#map-id"); var iconpath ="<%=request.getContextPath()%>/...
Read more

JavaScript (Prototypal) Inheritance with new and Object.create

//--------------------------- //Inheritance in JavaScript //--------------------------- var Person = function (string){ this.name= string; //if ( !(this instanceof Person) ) //part soln to danger 1 // return new Person(name); }; Person.prototype.get_name = function(){ return this.name; }; //Danger 1 : No compile errors here if new is skipped //Also inside the function would now be the global this object, //not the new instance! The constructor would be overriding all sorts //of global variables inadvertently var manu = Person("Manu"); name; manu.get_name(); //Danger 1 :only run time error here;Crackford's reason var adi = new Person("Adi"); adi.get_name(); //Okay let us try Iheritance with new var Employee = function (name,company){ this.name = name; this.company=company; Person.call(this,name); }; Employee.prototype.get_company = function(){ return this.comany; }; //How to inherit from Person ? //Step 1 - Link the prototypes /...
Read more