Here Come the Cloud-Based Big Data Management Services

Everyone’s well aware that the explosive growth of data is one of the biggest challenges facing organizations of all sizes across nearly every industry. A growing number of cloud service providers (CSPs) are recognizing that they can provide more than just simple storage and economical compute power to their customers who are buried in data. They also can help them manage their Big Data processing needs on an ongoing basis.

Big Data wouldn’t have gotten so much attention over the past few years if it weren’t for the availability of cloud services that enable organizations to do something with all the data. Organizations have always faced challenges acquiring, analyzing and acting on data. Third-party data processing services have been a lucrative business for many years because most organizations haven’t had and didn’t want to invest in internal systems and staff to collect, collate and interpret the data themselves.

Organizations now can take advantage of unprecedented computing power at unimaginably attractive prices to do amazing things with the escalating amounts data they are handling. However, having access to all these cloud resources doesn’t mean CIOs and other corporate decision makers know how to cost-effectively take advantage of these on-demand services or want to develop the in-house competency to do so.

The Need Is There

Eighty percent of the 400-plus IT professionals in the United States, Europe and Asia-Pacific who responded to a recent survey conducted by Enterprise Management Associates and commissioned by Iland required professional help to get started in the cloud. Nearly half (47 percent) sought help integrating cloud resources with their existing data center operations.

Although the survey didn’t ask the respondents how they would prefer to manage their ongoing data processing needs, my guess is that many would like to enlist the help of a reliable service provider to handle the responsibility.

Given the shortage of data scientists and the reluctance of many organizations to make significant capital investments in their own data centers, the idea of relying on CSPs to provide a new generation of data processing services is becoming more intriguing.

IBM has invested approximately US$24 billion in internal Big Data capabilities and made more than 30 acquisitions of analytics companies. It now boasts more than 15,000 analytics consultants, 6,000 industry solution business partners and 400 IBM mathematicians.

Although many IBM customers have engaged these resources on a project basis to build or improve their in-house operations, a growing number are turning to IBM to manage their ongoing business intelligence and analytics needs. In response to this rising demand, the company announced on July 1 that it was offering its IBM Navigator on its cloud marketplace and delivering the service via Softlayer’s cloud platform.

Will They Make the Leap?

Google recently offered a sneak peek of its new Dataflow cloud-native data processing service. It also is offering its BigQuery analytics capabilities as a fully managed data service. The Google Big Data processing service allows users to import up to 100,000 rows of data per second, and it will process 1 TB of data each month for free.

These services promise to handle an organization’s real-time, high-volume logging or transaction-recording requirements, and make customer data available for analysis more quickly and easily than most organizations could manage to do on their own.

Amazon Web Services, Oracle, Microsoft, and other leading CSPs also are offering varying forms of managed Big Data processing services via their IaaS platforms. The most important variable in each of their offerings is the level of CSP management and individualized configuration, analysis and support offered to customers.

You can bet that the higher-transaction, commodity service-oriented players like AWS and Google will be less inclined to provide dedicated account management and data scientists to individual accounts. IBM and other CSPs that historically have profited from higher-value data processing and outsourcing services will employ similar business models to package, price and deliver their new generation of cloud-based Big Data processing services.

Salesforce.com’s recent acquisition of RelateIQ also could be pointing in this direction.

The question is, how many organizations will subscribe to these services to help them acquire, analyze and act on the data that is gathering all around them, so they can better respond to rising customer demands and competitive pressures?