Federal Initiative Aims to Spur Software Development

The allure of Big Data stems from the ability to use advanced technology to manage and manipulate huge amounts of information. Not only that, Big Data offers the opportunity for unlocking information not previously available, and even not previously knowable.

However, the software required to underpin Big Data and related IT advances may not be up to the task. A federal agency within the U.S. Department of Defense now contends that increasingly powerful and complex IT systems will require similar advances in software.

The Defense Advanced Research Projects Agency, or DARPA, has just initiated the Mining and Understanding Software Enclaves program, a research project for generating breakthrough approaches to software development.

Software Vulnerability Challenges

“As computing devices become more pervasive in our daily lives, the software systems that control them have become increasingly more complex and sophisticated. Consequently, despite the tremendous resources devoted to making software more robust and resilient, ensuring that programs are correct, especially at scale, remains a difficult and challenging endeavor,” DARPA said when it launched the project last month.

“Unfortunately, in spite of developers’ best efforts, software errors are at the root of most execution errors and security vulnerabilities,” the agency noted.

The object of the MUSE program is to make significant advances in software development that are more revolutionary than evolutionary. The collective knowledge gleaned from MUSE’s efforts will facilitate new mechanisms for dramatically improving software correctness, and help develop radically different approaches for automatically constructing and repairing complex software, DARPA said.

“The goal is to develop a mechanism that can effectively leverage the collective wisdom of a large corpus of software to eliminate defects in existing software or automatically create — or synthesize — new software based on a desired behavior. It does this by constantly mining a continuously evolving corpus or database of program facts and behaviors,” Suresh Jagannathan, DARPA program manager, told the E-Commerce Times.

Improving Software Integration

Responding to a query involving a simplified analogy, he noted that a student driver may believe he can drive through a traffic light without stopping, and to test that belief he continues driving to see what happens. Alternatively, the operator can observe what other drivers do — or ask other drivers.

“If the overwhelming majority stops at the light, then he can adjust his belief accordingly, presuming that to be the expected protocol. Moreover, a driver who has no idea of whether to cross or not, but only knows the basic operations involved in driving — that is, acceleration, braking — can observe the fact that most drivers stop, and include this behavior as a key feature of a core driving algorithm,” Jagannathan said.

“New beliefs like this can be continuously generated, refined and validated based on new observations and contexts,” he added.

“Today’s software is written like the driver in the first scenario — incorrect assumptions and implementations lead to crashes. The structure and behavior of other programs, like the behavior of other motorists in the example, is not at all integrated within the software development process. MUSE seeks to provide an infrastructure in which this new alternative approach to software construction, development and maintenance can be realized,” he said.

“Software repositories today are estimated to contain more than 100 billion lines of code, and the number continues to grow. Open source software is widely used in mission-critical Defense Department systems, as well as in the commercial world,” Arati Prabhakar, director of DARPA, said in testimony presented at a congressional hearing March 26.

The MUSE program, she said, “aims to harness the scale and complexity of this array of software to instigate a fundamental shift in the way we conceive, design, implement and maintain software.”

Fostering an ‘Always On’ System

“Ideally, we could enable a paradigm shift in the way we think about software construction and maintenance, replacing the existing costly and laborious test, debug, validate cycle with always-on program analysis, mining, inspection and discovery,” Jagannathan said.

“We could see scalable automated mechanisms to identify and repair program errors, as well as tools to efficiently create new, custom programs from existing components based only on a description of desired properties,” he added.

DARPA accepted research proposals on the MUSE effort from organizations outside of the Defense Department through April 15, 2014. DARPA will fund research projects in five technical areas including Evaluator, Artifact Generators, Mining Engine, Analytics, and Infrastructure. Research will be conducted on a multilingual basis and will involve a diversity of programs including C#, Java, Scala, ML, Haskell, Racket, Javascript, Python and others.

While the DARPA effort will focus on open source code, Jagannathan noted that it could apply to proprietary programs as well.

The objective of the MUSE program holds the promise of improvements in software according to Sushil Bajracharya, code search architect at Black Duck Software.

“What’s particularly interesting is the mention of doing deeper program analysis and not treating source code as just text data. In fact, source code is rich in terms of structure and semantics,” he said.

“Doing precise analysis on code collected from the wild is expensive and error-prone, so I am interested in seeing what kind of techniques would emerge for precise program analysis and fact extraction at scale,” Bajracharya told the E-Commerce Times.

“With the right approaches that lie at the intersection of program analysis and probabilistic mining techniques, it might be possible to build novel verification and debugging techniques,” he suggested.

The Federal Buzz: Cyber Forecast

The U.S. government will spend an estimated US$12.3 billion on cybersecurity efforts in 2014, and that level will grow to $14.2 billion annually by 2018, according to a just-released market forecast from Avascent Analytics. The forecast covers all federal agencies over a five-year period, and includes both classified and unclassified components.

“The estimates cover the contract addressable market and mostly relate to enterprise-wide systems. Also, with the Defense Department, there is significant investment in cybersecurity spending for supply chain management related to weapons procurement,” Christopher Meissner, senior associate at Avascent, told the E-Commerce Times.

“The Treasury Department is another leader in cyberspending, especially for the IRS,” he added, “but we were surprised that Health and Human Services spending was not as much as we expected.”