Feature for Crystal Gazing

At the dawn of the age of artificial intelligence, when predicting future through data sciences has become a second nature for most of the decision makers. Though we may feel elated on the break-through in the data sciences. Still ground needs to be prepared for seeing the results through.

When we want to peep into the future, our launching station is the past observations. Let’s pause - what’s an observation, isn’t it simply a instance of attribute values. Oh, lets simplify - [Sales Executive] how was my car sales yesterday - fine [ 20 (10 sedans, 6 hatchback & 4 suv) , color mix (white 10, gray 7 & black -3]. Is that enough to predict today’s or let’s say tomorrows sales.

Most of us will say. -No.  So, what we need are the influencers of the predicted value. From gut feel sales manager will tell, following are the subset of attributes which may help --Like the day (is it holiday or working day), what time of the year, is there a promotion going on, how is the overall economy doing etc.

These attribute (influencers) are features which a data scientist will search for, to predict values of certain event. Getting these features in right format is the first step in the journey of prediction.

Then comes the support strength of the features in predicting value, here statistics may help in shaping our features (scaling the features, correlation, visual exploration - list goes on...). Experimenting is the core in honing these features for a near perfect prediction.  It’s all about Features...

Let’s bring the black boxes - yes you heard it right “algorithms”. Scoring and evaluating the algorithm result by using training and test set. In simple words we use past data to train the algorithm and build the model. To check the model’s strength in predicting, we hold back a subset of past data, which acts as test data set, thus providing us the feedback to choose the best predicting model. And the Magic ensues. [let the future come to us...]

Nucleus of Information Genesis

   Business Leaders & Managers demand Information of Business Activities in their respective Area of Concern. Technology Group of Organisation are struggling hard to meet these needs - WHY ?

   Information Solution in most of the Organisations today is Demand Driven. So, pull comes from Business for specific information and then the technology group gears up - itself to fulfill those demands. And there is no end to this cycle of demand and fulfillment, this cycle gets more complex and cumbersome as the day passes by. Information fulfillment pipeline keeps getting longer and longer till the Business becomes weary of the response time of technology group to deliver in time.

   How to reign over Information Demand Fulfillment Cycle? Solution lies in approach - Demand Driven to Business Process Driven

   Business Process Driven approach to Information Demand Fulfillment Cycle starts at the root, and builds infrastructure which takes into consideration almost all current and future information demand.

   Information Solution starts by delineating Business Processes which are contributing towards value generation. Each business process is broken down into Business Events. And each business event is now defined using Actor-Action Analogy -

   Business Event becomes basis of developing the Information Solution. Many business event join in a Bus Matrix to provide comprehensive information framework.

   This is a scientific approach as it encompasses almost all possible permutation and combination, business may ask for.

SOID - Service Oriented Information Dissemination

We have best of the tools and technology at our disposal and these are getting enhanced/matured with each passing day. Ms/Mr Manager still running pillars to post to get the information - why so.

Limited awareness of [Monolithic reporting solutions for each silo/department] leading to compartmentalization of information dissemination mechanics. In the world of Ubers and alike Aggregators there are similis to be drawn while looking out for a solution to the problem.

Let’s focus on anatomy of information -> Data along with Metadata (with business significance defined).

Organisation have multiple applications which generate data. Each application can be easily mapped to its respective Data Owners (personnel's who maintain and understands data flow mechanics of the application). These individuals are the information publishers to the Aggregator.

Decision makers are the Information Consumers interacting with the Aggregator as depicted in the exhibit below -

IOT Touch Analytics

B2C businesses, interfaces with consumers on web either through browser or mobile app.  Interaction of consumer to this interface provides the only opportunity for enterprises to make the experience worth indulging / engaging. 

Capturing all the interaction of consumer with this interface is paramount for continuous improvement and personalization-dynamicity of the interface. Most of the solution in market today around web analytics do provide some head way but it falls short in providing the comprehensive picture and that’s where IOT comes in.

A small detour on what IOT  [ Internet of Things ]  Is.  Any object/entity virtual or real having three properties/attribute should qualify for a thing to be termed as IOT 

Unique ID

Sensitivity

Communication Capability [telemetry]

Basic building block of Interfaces are objects. Enabling these objects as IOT is the first step i.e. object need to identify itself uniquely, have action capturing code and ability to log the activity (telemetry).

As customers interact with interface , IOT's are activated and through telemetry- log is built up of all the activities customer is performing.

Baseline for the next step is ready - amalgamation of the log data with master data such as customer demographics etc. Passing the data through machine learning algorithms provides us exploratory, predictive and pattern analytics.  Which in turn acts as input to the interface for personalization.

Instant personalization of interface content is pinnacle of IOT Touch Analytics.

Operational Business Decision - A Process Of Response to Exception

Business Processes have evolved over time. These process are streamlined to almost military precision. Continuous improvement / Kaizen application over the years have made our business efficient. As the businesses exist in dynamic environment , there is a constant flux in the business ecosystem , which poses new challenges here now and then.

Business decision are rudders keeping the ship on-course to value maximization.  Drift in the course is what exception are. So business decision starts from identifying these exception, prime focus is now on reducing the lead time of reporting these to the decision makers (information dissemination as close as possible to the occurrence of exception) , the better it is.

Kind of business information needed depends on Decision Impact Horizon i.e. long term (strategic) --- short term (operational). Transactional information is the one needed for short term decision process. Accumulated transaction forms the basis for long term decision processes.

Pattern deviation detection is the future of exceptional handling in businesses - thus machine learning is the way to go for organization to apply statistical algorithms to the transactional data and detecting pattern deviation as close as possible to the event occurrence aka real time intelligence.

To correct the course, business decisions are taken. Till now we are half way in accomplishing our goal. Impact of business decision needs to be measured, and on this feedback further action may be warranted.

Information Density In Data

Century of Data has just begun, and we all are overwhelmed with the flow and intensity of data coming to us every day. This is incredible feat human race has achieved in a very short span. 

We had to coin new jargon to represent - 'processing of massive amount of data' - Big Data. Soon with the current rate of acceleration of data accumulation we are moving towards handling super big data. 

The challenge now is finding the right, relevant information from data. 'Like a needle in Haystack'. Millions of dollars are poured by businesses in getting information from data. Classifying sources of data on the basis of information density is next logical step to bring sanity to the process of information extraction from data.

Factors of information density - (Inference Quotient * Data Volume * Data Speed * Structural Quotient)

Inference Quotient: Degree to which direct inference can be extracted from data to reach the relevant information

Data Volume: More the volume, lesser the density

Data Speed: Greater the speed, lesser the density

Structural Quotient: Degree of Structuredness (more structured higher density)

Anatomy Of Intelligence

200 million years ago - Dinosaurs ruled this planet, today its human species. What's changed? Is it intelligence which is the differentiating factor? Not exactly, vital component in the mix is the communication and that's implicit in intelligence.

It all starts from observation - recording event and there results - then making conclusions, improvising as more events of similar nature are encountered. In nutshell pattern recognition match and act. The real difference comes when communication kicks in and amplifies the whole process to a level where innumerable permutation combinations are at disposal of the mankind.

Underling current is the streams of data processed to produce applied information and made available to masses. Same analogy holds true in the porous boundaries of business houses. So a great business organization have one great characteristics, it have smooth uninterrupted communication channels for relevant information to flow through the whole organization.

Where does innovation fits into the picture. Lateral thinkers break the patterns to innovate - it’s so unique to human species. Intelligence then becomes the vehicle for innovation to reach out to every individual of our civilization.

Blueprint Data Streams

Organization wish to stay on top of the data generated in-house and by the environment around its ecosystem. But there is a distance to be covered to make the wish come true. 

 So where do we start - question to be asked is “What we have? ". Getting answer to these questions right is half the battle. Data streams are running throughout the organization but most of these streams are in private spaces of the individuals or at the most with close knit teams. As these individuals and teams time and effort is applied in developing or enhancing these data streams.

 Action plan starts with blueprint of these data streams. This is no small undertaking but need to be done to stay on top of the data. Suffice to say data is the oxygen for the being called [organization]. To know data/oxygen [life force] pathways is of paramount importance.

 Two pronged strategy is proposed to develop the blueprint of data streams

 Source Identification ... & Flow Map....

 Blueprint Data Streams is a living document so as the time passes by, delta in the organization data infrastructure and flows needs to be updated in.

Emulating Human Learning

It all starts with the perceptive nerves. Signals are interpreted and conclusions are made, actions are taken. Lets simplify this assume a new situation is perceived - step one compare with past experiences - no match , do a close match , approx match - and that's what experience is all about.

Information age is ripe to emulate human learning. Big data, immense parallel processing grids availability and algorithms. Algorithms that's right - termed machine learning. It’s not an exact science but we humans are all about [not perfection] but intelligent enough to correlate and approximate.

Regression, Classification [Supervised] and Clustering [Unsupervised] are various methods for learning. Supervised Learning draws from historical data while, unsupervised methods can, charter completely unknown territories. So next decade may see ,advent of machines providing second opinions to human judgment.

As I write my thoughts Business houses are in process of deploying technologies to make this a reality. Don't be surprised if next time your teller machine recognize your mood , and provides you options accordingly. As we humans do...

Sample to Population - The Big Data Leap

Capacity of technology to churn huge amount of data has ushered a new era in statistics.

Lets visit Sampling definition " sampling is concerned with the selection of a subset of individuals from within a statistical population to estimate characteristics of the whole population " , 'subset' is the term which expresses statisticians workaround to get as close as possible to the population spread. As we know the act of getting sample right is of paramount importance.

Oh , what if we can get the whole population data - so that's a clear possibility with the advent of Big Data moving in fast. Massive parallel processing of the Map-Reduce Framework on the Distributed File System , its all possible. To top it, likes of Revolution R & Mahout are already there.

Statistics as we know of today is on its journey to accept population as the base data for analysis rather than the sample. Better estimates , predictions are on its way....

Bus Parallel Processing Architecture - Cloud Ready

Business Intelligence conceptual design paves the way for BPPA (Bus Parallel Processing Architecture). Bus Architecture is the epicenter of developing a well knit dimensional model i.e. translate business processes/sub-business processes into star schema's with dimensions drawing on the master data and transactions finding its way to the facts. Knitting is taken care off through conformed/shared dimensions.

Logical layer takes the bus architecture to the implementation turf, developing the database ER model leading to creation of database tables and bringing star schema into existence in relational database. Physical layer provides the infrastructure for DW i.e. servers, networks etc.

BPPA works on the principle of dimensional redundancy across servers (virtual servers). So each fact moves into a dedicated box thus allowing scaling out the enterprise DW - enabling cloud move for our DW. Conformed/shared dimension are redundant on respective star schema. Cost or overhead of this approach is maintaining redundant copies of dimensions. ROI of this is performance gain and ability to port the scaled out solution to multiple small servers and eventually to cloud.

To top up fact partitioning can be done, as most database vendors now provide this - out of the box, thus reducing IO footprints of fact access.

Futuring Now

History has seeds of Future. Identification of these seeds is what a Business Leader needs.

Information age has made data capture of almost all the activities possible. It’s all about unraveling the patterns in these data which is the key to the future or decision making. So were to start - today there is comprehensive disciple of business intelligence. Dimensionalising our data structure and applying permutation and combination to better understand our business processes.

But that's just the beginning. Actuaries’ science is one of the most evolved application of peeping into future on the basis of history. It's time when statistical pattern analysis and prediction takes the central stage for conversion of the business data into artificial intelligence.

With the computing power and technological break-through (i.e. big data, hadoop distributed file system) huge amount of data can be churned in a scale out manner and that too in virtual space of cloud. Artificial intelligence hits the ground reality as the Data quantum used increases and becomes closer to depicting the population (in statistical context).

Business empires will be built on futuring now and technology organisations have a great opportunity in this.

Performance, Elasticity and Scalability

Performance tuning is done for improving system performance. Prerequisite to performance tuning is monitoring. Monitoring forms the bases of building system performance baseline. Baseline provides reference for the performance evaluation of the system.

A system’s ability to accommodate to changing conditions or circumstances is called elasticity. If system is resilient and requires minimal changes in order to accommodate the new conditions, system has elasticity.

In many cases the system performance decreases for example with increasing load volumes or the number of executed applications, resilience of the system to these and maintain the performance is termed as scalability.

All of the three i.e. performance, elasticity and scalability are qualities of the system and are the cumulative result of designing the system with proven methodology. Architectural approach is the deciding factor here. Design of the system from conceptual, logical to physical layer contributes towards these.

Conceptual Layer design needs proper understanding of the Business Process and there relative importance to the business top line. This will ensure elasticity built into the system as, conceptual design will delineate detailed understanding of the business requirement, towards this Bus Architecture is of prime most importance as it provides top-down integrated view of the enterprise.

Logical Layer design requires process and data modeling expertise in order to ensure all permutation and combination in terms of business requirement are developed.

Physical Layer design needs using appropriate capacity planning of hardware and configuration of software/services

UI Crossroads

User Interface, Unification and then Disintegration. So Browser to Apps, WHY -- it’s from need to want journey i.e. from generic to specialized to personalize User Interfaces (UI).

As technology evolves, it allows more customization and sophistication. Today we are standing at another crossroads where ease of use is taking front seat and standards are taking a hit, thus today we create more than one UI for our application - namely Web UI, IPhone UI, Android UI, Windows UI....

Hang on; the verdict is not out still. HTML is making a mega comeback, watch out for HTML5. Platform independent UI is on unveil.

SOA thought process demands platform agnostic UI. But Business world thinks otherwise, till technology proves ROI. Eventually we have seen time and again technology supersedes, profit machines (corporate houses). Case in point: extensive use of SOA framework to integrate diverse systems with ease.

Effort turnaround time points towards Platform Independent UI.

Massive Parallel Processing In Cloud - "BIG Data"

Massive Parallel Processing In Cloud - "BIG Data"

How Pyramid of Giza was built? What it takes to run a chip? What's the Universe made up of?

Basic fabric of nature is - 'building block needs to be atomic and should be available in plenty'. Same applies to computing world. There is a limit to scale up, because as we scale up, maintenance overhead increases and soon we reach to a point where scale up advantage is nullified by the corresponding increase in maintenance overhead.

So the way to go forward is nature’s way. Scale out. Concept - when many atomic computing resources are leveraged in parallel, the impact is massive. And that’s what we call MPP - Massive Parallel Processing.

With each passing day internet infrastructure efficiency is getting stronger in terms of payload and speed. Now that forms the foundation for MPP in Cloud. Unlimited atomic computing resource availability (literally) on cloud from many vendors, is another thumbs up for MPP in Cloud to become a reality.

MPP in cloud needs, scale out architecture paradigm to be effective.

Big Data concept is a reality - “Think Scale Out”  

Economics of Compression

Compression is a mechanism to reduce space required to keep a certain thing or entity. In the world of databases this word assumes a greater meaning. In the information age, data is growing at exponential rate. As we have more avenues to capture data.

With this growth in Data, Organization increasingly wants to draw meaning/value out of this data. That entails - Data Engineers now have to maintain huge volume of data and not only that, the need for performance is also growing.

Algorithms for compression have evolved, providing reduction in space required to maintain data. That’s just the beginning.

Trinity of computing resources namely CPU, Memory and IO plays a vital role in the performance. Out of these three, IO is the weakest link in the chain when it comes to data management performance.

So with reduction in size of data using compression, the need for IO decreases resulting into performance increases. But there will be increase in the total CPU cost, as compression and subsequent decompression will need CPU cycles.

A word of caution - High Cardinality data is difficult to compress and frequent updates at granular level will be slower due to compression overhead.

When in need of Performance one needs to take compression factor into consideration.

Intelligence Revisited

In information world, Intelligence is overloaded word.

Today’s dilemma is "Intelligence presupposes problem is known". That’s where the roots of rework and dead end roads start.

Case in point -:  Business Manager asking how business process is doing. Well the answer lies in, what’s the basis. That's were building the target line comes into picture. How do we get to know those target lines? Seems to be million dollar question?

There are two sources to identify the target line.

·         Experience

·         Discovery

Technologist hands are tied up as far as Business Experience is considered. Now we move towards discovery mode. We can get the top performance numbers from past data. And this forms the basis of target lines. Let’s beat our numbers - that’s the way to go. And supplement this with experience of business managers.

Once target lines are set we can measure, and improve. Intelligence enables Progress......

MVC an Insight

A input triggers a process which may produce implicit or explicit output. In the backdrop of this simplicity MVC framework/architecture manifests itself.

MODEL - Core Process Engine
CONTROLLER - Communication Hub (Receives/Sends Instructions/Data)
VIEW - User Interface

In the trio, Controller forms the basic fabric of the architecture. Its the layer which "cements" as well as provides "independence" to Model and View. Infrastructure which provides PERSISTENCE by modulating/handshaking on both sides i.e. VIEW and MODEL.

As the clients now a days have enough memory and CPU power , VIEW (user-interface) can be intelligent. And that's the space which can contribute to scalability , all the UI needs can be taken care of without the server coming into picture.

Business logic and data binding makes the MODEL. Here flexibility is a big challenge , SOA (Service Oriented Architecture) approach makes this area of architecture most refined process of all. Business Processes identification leading to component build up. Data Layer acts as persistent memory while data schema will reflect business cases (Entity-Relationships).

Cloud Gate

A prompt on a black screen to evoke a link , thus the story started of the mother of networks INTERNET. Till today we do the same thing. Light the fire enjoy the meal. Click a link and access the information.

A small lab with few terminals talking to each other , on black screen prompt. Then there was gui's (Graphical User Interface) to communicate between client and servers - I prefer to call them INTRANET ( 'A' is Personal - age old ownership issue).

Dawn of Global Travel Age brought challenges which compelled organizations, to break the narrow confines of ownership, restricted to geographies. Thus arrived the mighty 'E' Era. And Ms (Microsoft) was at the forefront with brilliant leaders at the helm to be the GateWay to this mighty network with 'IE' Internet Explorer.

Germinating the concept of real time cloud - Netscape appeared on the horizon with Thick Interface to INTERNET. We all know how monopoly may suppress ideas for some moments.

Then arrived Professor Google to take on MS but as if it was not enough Mr Apple played the I - card (Iphone, Ipad etc). Connection points or Gateways to Virtual world long dominated by MS - IE slowly gave way to Chrome,Mozilla,Safari ...

As I write this 'E' Era gateways are slowly giving way to 'M' Era -- So M-Apps are the Cloud Gates.

Where the Rubber Kisses the Tarmac

When a business process need to synchronize with other business process for generating synergy (or to weave the process flow) , foremost challenge is -"interfaces need to couple". Often Orchestration faults at this very juncture.

Following Question need to be answered to take the plunge :

- Business Process (BP) Significance (What BP stands for in the Value Chain)
- Interface Exposure ( What the BP expects and delivers)
- Heartbeat ( How BP are aware of each other presence )
- Referee ( In case the touchdown not possible between BP's, managing process needs to be informed for alternatives)

Interface is the key component for a Business Process in order to communicate to the external world. More often technologies involved in creation of Business Process are varied. This leads to Silos and eventually gaps in value chain labeled as "not Fit For Automation".Building an Interface (Adapter) to expose the Business Process, can make it " Fit For Automation".

Standards of engagement need to follow universal principles of communication - Messaging Protocol, Channel Specification, Transmitter and Receiver Identification.

Competition is driving businesses to build new business models in the blink of an eye. Technology need to reuse granular business processes to deliver. Need of the hour is to be a expert weaver of Business Process.