Massive Parallel Processing In Cloud - "BIG Data"

Massive Parallel Processing In Cloud - "BIG Data"

How Pyramid of Giza was built? What it takes to run a chip? What's the Universe made up of?

Basic fabric of nature is - 'building block needs to be atomic and should be available in plenty'. Same applies to computing world. There is a limit to scale up, because as we scale up, maintenance overhead increases and soon we reach to a point where scale up advantage is nullified by the corresponding increase in maintenance overhead.

So the way to go forward is nature’s way. Scale out. Concept - when many atomic computing resources are leveraged in parallel, the impact is massive. And that’s what we call MPP - Massive Parallel Processing.

With each passing day internet infrastructure efficiency is getting stronger in terms of payload and speed. Now that forms the foundation for MPP in Cloud. Unlimited atomic computing resource availability (literally) on cloud from many vendors, is another thumbs up for MPP in Cloud to become a reality.

MPP in cloud needs, scale out architecture paradigm to be effective.

Big Data concept is a reality - “Think Scale Out”  

Economics of Compression

Compression is a mechanism to reduce space required to keep a certain thing or entity. In the world of databases this word assumes a greater meaning. In the information age, data is growing at exponential rate. As we have more avenues to capture data.

With this growth in Data, Organization increasingly wants to draw meaning/value out of this data. That entails - Data Engineers now have to maintain huge volume of data and not only that, the need for performance is also growing.

Algorithms for compression have evolved, providing reduction in space required to maintain data. That’s just the beginning.

Trinity of computing resources namely CPU, Memory and IO plays a vital role in the performance. Out of these three, IO is the weakest link in the chain when it comes to data management performance.

So with reduction in size of data using compression, the need for IO decreases resulting into performance increases. But there will be increase in the total CPU cost, as compression and subsequent decompression will need CPU cycles.

A word of caution - High Cardinality data is difficult to compress and frequent updates at granular level will be slower due to compression overhead.

When in need of Performance one needs to take compression factor into consideration.

Intelligence Revisited

In information world, Intelligence is overloaded word.

Today’s dilemma is "Intelligence presupposes problem is known". That’s where the roots of rework and dead end roads start.

Case in point -:  Business Manager asking how business process is doing. Well the answer lies in, what’s the basis. That's were building the target line comes into picture. How do we get to know those target lines? Seems to be million dollar question?

There are two sources to identify the target line.

·         Experience

·         Discovery

Technologist hands are tied up as far as Business Experience is considered. Now we move towards discovery mode. We can get the top performance numbers from past data. And this forms the basis of target lines. Let’s beat our numbers - that’s the way to go. And supplement this with experience of business managers.

Once target lines are set we can measure, and improve. Intelligence enables Progress......

MVC an Insight

A input triggers a process which may produce implicit or explicit output. In the backdrop of this simplicity MVC framework/architecture manifests itself.

MODEL - Core Process Engine
CONTROLLER - Communication Hub (Receives/Sends Instructions/Data)
VIEW - User Interface

In the trio, Controller forms the basic fabric of the architecture. Its the layer which "cements" as well as provides "independence" to Model and View. Infrastructure which provides PERSISTENCE by modulating/handshaking on both sides i.e. VIEW and MODEL.

As the clients now a days have enough memory and CPU power , VIEW (user-interface) can be intelligent. And that's the space which can contribute to scalability , all the UI needs can be taken care of without the server coming into picture.

Business logic and data binding makes the MODEL. Here flexibility is a big challenge , SOA (Service Oriented Architecture) approach makes this area of architecture most refined process of all. Business Processes identification leading to component build up. Data Layer acts as persistent memory while data schema will reflect business cases (Entity-Relationships).

Cloud Gate

A prompt on a black screen to evoke a link , thus the story started of the mother of networks INTERNET. Till today we do the same thing. Light the fire enjoy the meal. Click a link and access the information.

A small lab with few terminals talking to each other , on black screen prompt. Then there was gui's (Graphical User Interface) to communicate between client and servers - I prefer to call them INTRANET ( 'A' is Personal - age old ownership issue).

Dawn of Global Travel Age brought challenges which compelled organizations, to break the narrow confines of ownership, restricted to geographies. Thus arrived the mighty 'E' Era. And Ms (Microsoft) was at the forefront with brilliant leaders at the helm to be the GateWay to this mighty network with 'IE' Internet Explorer.

Germinating the concept of real time cloud - Netscape appeared on the horizon with Thick Interface to INTERNET. We all know how monopoly may suppress ideas for some moments.

Then arrived Professor Google to take on MS but as if it was not enough Mr Apple played the I - card (Iphone, Ipad etc). Connection points or Gateways to Virtual world long dominated by MS - IE slowly gave way to Chrome,Mozilla,Safari ...

As I write this 'E' Era gateways are slowly giving way to 'M' Era -- So M-Apps are the Cloud Gates.

Where the Rubber Kisses the Tarmac

When a business process need to synchronize with other business process for generating synergy (or to weave the process flow) , foremost challenge is -"interfaces need to couple". Often Orchestration faults at this very juncture.

Following Question need to be answered to take the plunge :

- Business Process (BP) Significance (What BP stands for in the Value Chain)
- Interface Exposure ( What the BP expects and delivers)
- Heartbeat ( How BP are aware of each other presence )
- Referee ( In case the touchdown not possible between BP's, managing process needs to be informed for alternatives)

Interface is the key component for a Business Process in order to communicate to the external world. More often technologies involved in creation of Business Process are varied. This leads to Silos and eventually gaps in value chain labeled as "not Fit For Automation".Building an Interface (Adapter) to expose the Business Process, can make it " Fit For Automation".

Standards of engagement need to follow universal principles of communication - Messaging Protocol, Channel Specification, Transmitter and Receiver Identification.

Competition is driving businesses to build new business models in the blink of an eye. Technology need to reuse granular business processes to deliver. Need of the hour is to be a expert weaver of Business Process.

When It Happened One Day ..

Flood came and a business was shut down forever. A human error and we were history. Scary thoughts ,lets think again does these things,happen. Yes Sir/Madam they do. A big question resounds 'So, What we need to DO ?'.

Lets understand what we are up against : There are either of the two : A Natural Disaster or A Human Error (which includes wilful ones also). Depending on the Location and Business Stream a list of the disruptive events can be built. ( to emphasis :: we are only listing events which we don't have control on )

Now the aim is to build a Plan of Recoverability. Answer the following questions :

• Time Required to Recover? - How much time the organisation have or can provide to recover.
• To what Degree we want to Recover? - How much Loss the organisation can bear.
• What's the Cost of Recoverability? - A million dollar question.

Stage is finally set to brainstorm and build the plan. These plans will be function of (business stream,location,threats) so it will be specific and unique.

When It Happened One Day - We Survived.