For the last few years people and industries have been thinking and talking about the Big Data endlessly. The next round is all about applying the machine learning to the Big Data. But, in reality, the era of Big Data is coming to an end and many of us must have seen media reports regarding the pullbacks in the Big Data industry.
This is where the Big Data ends and the actual application of technology begins. This means there will be lesser “let’s roll out and see what happens” projects which will compel decision makers take more rational approach starting with the business problem first.
Analyzing the common occurring problems, finding patterns and creating packaged solutions to those problems will be next big step and this type of implementation can be seen in the finance industry where fraud detection packages wrapped up inside and are good to go.
These fraud detection packages are not new but distributing them with Hadoop or cloud technology is something very new and fresh. Due to frauds, finance industry was losing so much money but now they are fighting back and Hadoop and Spark are the weapons in their armory.
Custom built solutions are not enough and the fraud detection for the credit cards isn’t that different than for the invoicing, insurance and any other common business application. The next step is not to write the very specialize apps for the specific business but rather to identify the big data patterns for solving the common problems which exist across the lines of business.
Custom solution where everyone is solving the similar problem in different ways will stay in the market but the future is in finding the common grounds, developing the patterns and spreading those solutions across various lines of business. In the end, it needs to be customized a little bit to be used in specific business line. This means many of the repetitive tasks need not to be done again and again.
Yes, we have already seen it in the market before. Almost a decade ago, the accounting software was the hot topic and while one can still find the specialized accounting software in the market, most of the companies are using the pre-packaged solution which has been customized to their business specific needs. It is very rare for the skilled software developer to write software for a specific kind of business and let alone for a particular company. Everybody buys them right off the shelf because they are general purpose.
In the world of Big Data, the next big step is going to be “data driven” and using “machine learning” with the help of integrated software. It might be Big Data driven in the end but Big Data will not be the topic of conversation. By then it would become like Ethernet cards: “Taken for granted”.