Feds Make Big Strides in Big Data Management

U.S. government agencies gradually have been getting a handle on utilizing the potential of big data and data analytics.

At the same time, the Obama administration has recognized that significantly more investment will be required for federal big data research in order to guide agencies toward maximizing the value of processing huge chunks of data and improving their data analysis capabilities.

“U.S. federal government agencies are accelerating the pace of change when deploying Big Data and analytics in their organizations,” wrote IDC Government Insights Research Director Adelaide O’Brien in a report published this spring.

During the past 12 to 24 months, federal agencies “have significantly changed the status quo,” she noted.

For example, agencies have expanded the type of analytics techniques used, the number of users with access to big data and analytics solutions, and the data types and sources undergoing analysis. Agencies have started using new metrics or key performance indicators, according to IDC, which reported that nearly 80 percent of the respondents to its survey said they had engaged in all of those expanded big data activities.

Survey Shows Progress

On a functional basis, “operations related processes” rank first with 55.7 percent of investments in big data and analytics. Next are “people related processes,” which include citizen services and case management, at 23.3 percent. The third function for investment is “money related processes” — including budgeting, finance, accounting, fraud prevention and risk management — at 21 percent.

Satisfaction levels are encouraging, the report indicates. For example, 35.7 percent of respondents said they achieved a “much better than expected” level of results using big data and analytics for operations-related processes.

For people-related processes, 41.4 percent of respondents saw “much-better” results, and 35.2 percent placed results for money-related processes in the “much better than expected” category for applications involving the identification of new revenue streams.

Factors that contributed to successful management of big data at federal agencies, according to the report, include the following:

• Leadership: Successful agencies were far more likely to have top managers who emphasized a data driven culture and who “incentivized” the use of technologies. In addition, those agencies with accomplishments in using the technology were more likely to have taken an enterprise-wide approach.
• Pragmatic Approach: Agencies achieving success displayed “enterprise-wide skill sets” and made efforts to develop a broad range of skills from both external and internal resources. Those agencies also used “flexible, centrally governed architecture” that was adaptable to ongoing user needs. Successful agencies were almost four times as likely as lower achievers to have a high level of automation in systems management, resulting in optimized performance and dynamic scalability.
• Integration:Agencies with better results reduced the occurrence of data “silos” and were much more likely to “have all relevant internal and external data at necessary strategic granularity.”

The gradually improving performance of federal agencies in managing big data and analytics could benefit vendors as agencies become more focused in the acquisition of related technologies. However, the learning curve is still pretty steep, according to IDC’s O’Brien.

Vendor Connections Play a Role

“As agencies share best practices and outcomes achieved, the importance of deploying big data solutions will become better known. However, agencies still need to address skills, processes, and knowledge gaps that may exist in acquiring big data,” O’Brien told the E-Commerce Times.

On the other hand, vendors appear to be getting better at making federal agencies comfortable in using Big Data tools.

Providers are offering pilot programs that allow government agencies to deploy a small test using big data and analytics, and determining the feasibility of further rollouts, O’Brien noted. Agencies have been leveraging successful commercial best practices from vendors with financial expertise in using big data to support federal efforts in detecting and preventing money laundering, tax evasion and securities fraud.

In addition, vendors are providing solutions that allow mash-ups of big data with other business transformation components, such as cloud, mobile capabilities and social business technologies, she pointed out.

These elements comprise a “third platform” of IT, according to IDC, going beyond conventional software IT packaging through user implementation. Vendors also are providing Analytics as a Service capabilities.

Feds Bolstering Big Data Actions

Other federal initiatives launched on a government-wide basis have been supporting and encouraging federal agency activities in big data management. A recent example is the Federal Big Data Research and Development Strategic Plan, which the National Science and Technology Council’s Networking and Information Technology Research and Development program released this spring.

The NITRD program sets forth several research objectives designed to promote utilization of big data:

• Create next-generation capabilities by developing big data foundations, techniques and technologies;
• Support research and development to explore the trustworthiness of data, to make better decisions and enable breakthrough discoveries;
• Build and enhance research cyberinfrastructure that enables big data innovation; and
• Increase the value of data through policies that promote sharing and management of data.

The program’s goals include better understanding the privacy, security and ethical dimensions of big data collection, sharing and use; improving the national landscape for big data education and training in order to meet the demand for analytics talent and capacity in the broader workforce; and “supporting a vibrant big data innovation ecosystem with collaboration between government agencies, universities, companies, and nonprofits.”

While the NITRD program is not designed primarily to enhance federal procurement of big data resources, or assist commercial providers in the federal market, the scope of the program involves developing strong relationships with the private sector.

Many agencies already engage in small business innovation research and small business technology transfer programs through which big data projects can be initiated, noted Peter Lyster, deputy director of NITRD.

In addition, major IT players already have been active with federal agencies. For example, the National Aeronautics and Space Administration, the National Science Foundation and the Department of Energy have been working with technology collaborator Topcoder to run a series of ideation challenges around data fusion.

Also, NASA has been working with Amazon to provide access to earth science data, and the National Oceans and Atmospheric Administration has partnered with Amazon, IBM and the Open Cloud Consortium to release all NOAA data for public access.

There is likely “no immediate and simple connection” between the big data strategic research plan and federal contracting, Lyster noted.

“A lot of large government contracts deal with the purchase of large ‘things’ or systems, and one would hope that such systems have professional-grade commercial data management and analytics capabilities as necessary to perform their jobs,” he said.

“Of course, we hope that the research and development that gets funded to develop new technologies and methods for big data will ultimately find their way into professional grade commercial systems,” Lyster told the E-Commerce Times. “It is probably also worth mentioning that the government pours substantial funding into education and training — for example, to graduate students — and in the big data realm, these folks are often snapped up by industry.”