Executive Viewpoint 2016 Prediction: Altiscale - The Year that Big Data Goes Big Time

Mike Maciag (Profile)
Wednesday, December 30th 2015

In 2015, leading-edge organizations began investing in Apache Hadoop and saw real business use cases deliver positive results. This early traction inspired other organizations to experiment with Hadoop, and we are now seeing Big Data projects graduate from the early adopter phase.

Hadoop Grows Faster by Going to the Cloud

While many initial Hadoop implementations were on-premises, more companies are moving Big Data to the cloud. A recent Gartner survey showed that over fifty percent of current and future Hadoop deployments will include some form of cloud delivery. Cloud-based Hadoop solutions shoulder the most difficult parts of Big Data --the infrastructure, software management, and operations—so that customers can focus on getting greater business value from their data than ever before. I predict that, in 2016, we will see the cloud model for Hadoop gain critical traction and recognition in the market.

Data Governance Rises

In terms of Hadoop adoption growth, it is now recognized that Big Data is increasingly mainstream. As new inflows of data appear,  we will see increasing demands to better understand where the data comes from, what it is and how best to use it.

Jumping the “Skills Gap” with Fully Managed Cloud Services

Growth in Hadoop demand will exceed the growth in the talent pool. According to a recent AtScale survey, 61 percent of respondents view the limited talent pool as the biggest challenge to adoption. In order to bypass the need to hire more data scientists and Hadoop admins from a highly competitive field, organizations will choose fully managed, cloud services that include operational support. This frees up existing data science teams to focus their talents on analysis instead of spending valuable time wrangling complex Hadoop clusters.

Spark Ignites

Hadoop deployments will increasingly leverage Apache Spark as a programming framework, due to its versatility and performance benefits. This year, Spark began to move from the topic of conversation to usage. In 2016 we will see businesses create real Spark and Hadoop use cases. As Spark continues to rapidly evolve and mature, organizations tapping the technology will need to keep pace with updates and upgrades. On-premises Spark deployments are problematic, requiring time, resources, and the ability to adapt to feature leaps. On the other hand, cloud deployments will facilitate these updates and upgrades, so users don’t need to worry about falling behind.