Val IT Case Study: U.S. Dept. of Veteran Affairs 

Transforming Information Technology at the Department of Veterans Affairs
Jonathan Walters
Governing Magazine

Table of Contents


On behalf of the IBM Center for The Business of Government, we are pleased to present this report, “Transforming Information Technology at the Department of Veterans Affairs” by Jonathan Walters of Governing magazine.

Jonathan Walters’ report chronicles the Department of Veterans Affairs’ (VA) efforts to realign and centralize its information technology activities. Describing it as an “ambitious, audacious and arduous crusade,” Walters makes it very clear that this is still very much a work in progress. There are significant hurdles ahead and certain significant adjustments will no doubt need to be made for this ambitious undertaking to be ultimately implemented and sustained by the VA. Yet at the same time, the effort offers the VA’s new leadership a clear and established roadmap for moving the effort forward, because a lot of hard work has been done for them.

In addition to his captivating description of the VA experience, Walters also identifies ten lessons learned—based on the experience of change management at the VA—which are clearly applicable to any organization confronting a change management initiative.

According to February 2008 Government Accountability Office testimony, the department has established and activated three governance boards to facilitate budget oversight and management of its investments. Further, VA has approved an IT strategic plan that aligns with priorities identified in the department’s strategic plan and has provided multi-year budget guidance to achieve a more disciplined approach for future budget formulation and execution. While these steps are critical to establishing control of the department’s ITIT, it remains too early to assess their overall impact. Thus, their effectiveness in ensuring accountability for the resources and budget has not yet been clearly established.

We hope that this particularly timely and informative report will be useful to the new Secretary of Veterans Affairs and his leadership team, as well as executives across government who are also dealing with change management initiatives, including reform of their Information Technology programs.

Albert Morales
Managing Partner
IBM Center for The Business of Government
John Lainhart
IBM Global Business Services

Executive Summary

In October 2005 the U.S. Department of Veterans Affairs launched what many believe to be the most ambitious organizational information technology overhaul and consolidation ever to be attempted in the federal government.

With increasing pressure coming from outside entities, including Congress and the Government Accountability Office, and pushed by a group of influential insiders, the department in 2005 laid out ambitious and controversial plans to consolidate control over what had become a sprawling, aging and unwieldy system of computer and communications
technologies spread across the department’s more than 1,000 medical centers, clinics, nursing homes and veterans’ centers.

According to an October 2005 memo from a former VA chief information officer, the VA CIO had direct control over only 3 percent of the department’s overall ITIT budget and 6 percent of the department’s ITIT personnel. Individual medical directors in the field had virtually complete control over decisions about ITIT investment, which had resulted in a substantially ad hoc and disjointed ITIT system virtually impervious to rapid, nationwide sharing of client information, universal system upgrades or patches, or system-wide distribution of new, proven applications.

As pressing as questions about the functional capacity of the system were, the overall security of the system was equally urgent, with scores of unsecured laptops and thumb drives floating around the system containing confidential information about millions of veterans. Meanwhile, mainframes sat unsecure in frequently precarious circumstances, near old steam and sewage pipes, with no backups available. Indeed, the push to gain control over the sprawling system was reinforced powerfully in 2006, when an IT security breach compromised the confidential information of some 26 million veterans.

The heart of the overhaul was the creation of a governance plan and the adoption of a full set of best IT practices that the VA hopes will ultimately result in a secure, integrated, reliable and responsive IT system aimed at efficiently delivering high quality health care services to veterans, while supporting the thousands of health care professionals who work for the VA.

Under the plan, the department has centralized all IT budgeting, planning and development, including putting full control of the department’s IT budget and staff under the VA’s Office of Information and Technology in Washington, D.C., while placing a premium on encrypting, securing and accounting for every piece of computer hardware in the system.

The ongoing effort has been difficult and controversial. An organization based substantially on an ethic of dispersed authority and control has proved to be an extremely difficult environment in which to try to centralize and consolidate. There has been wholesale resistance to the consolidation effort from a wide variety of powerful players both inside and outside of the VA.

As of this writing, the transformation is a work in progress, but has made substantial progress towards the “One-VA” vision laid out in the consolidation plan. But while the effort is still a work in progress, it has the potential to be a useful model for other large-scale public sector entities wishing to modernize and consolidate similarly unwieldy and dispersed systems.

Some of the VA lessons include the obvious, including the absolutely critical role that leadership plays in any large-scale organizational change effort. But some lessons are less so, including the VA’s novel and innovative approach to the contract that it let for outside help with the transformation effort, an approach that ought to prove useful to any organization
availing itself of outside help with transformation.

Meanwhile, the VA has clearly made substantial progress in consolidating planning, budgeting and personnel and in securing all the information contained in its massive IT system. Specific initiatives include moving data to regional processing centers with improved “fail over” capacity. A new contract is allowing for increasingly rapid and reliable deployment and upgrading of computer hardware and software system-wide.

While questions about the transformation remain—in particular about whether the development arm of the VA is up to quickly designing, testing and deploying new software—the reforms at the VA appear to be taking root. No doubt the growing pains will continue, but with continued work and investment, the VA has the potential to solidly secure its place as a model for how to do large-scale IT transformation in a logistically and politically complicated—some would even say openly hostile—environment.


Drivers for Change at the VA
By 2004 serious pressure was building on the Department of Veterans Affairs (VA) to get a handle on its sprawling information technology empire. Far flung mainframes and a patchwork of computer networks were neither secure nor standardized. The VA’s main operating system was considered out of date and not up to the job of allowing rapid, nationwide sharing of client information or accommodating new uses, applications, or effecting efficient universal patches for software problems or upgrades.

The Government Accountability Office had been calling for a major ITIT realignment and upgrade at the VA for years, and Congress was beginning to pay much closer attention to what was going on at the VA, as well—interest that would take on a new urgency in the wake of a security breach in 2006 that compromised confidential information about 26 million veterans.

At the same time, influential and active insiders at the VA had also been expressing concern about the age, efficiency and security of the whole VA IT system. The concern culminated in early 2005 with then-VA Secretary R. James Nicholson authorizing a system-wide study to come up with options for what a restructured and modernized IT structure at the VA might look like.

The VA had recognized the critical importance of a new IT strategy which was described in the VA Strategic Plan FY 2006-2011:

“Implement a One-VA information technology (IT) framework that enables the consolidation of IT solutions and the creation of cross-cutting common services to support the integration of information across business lines and provide secure, consistent, reliable, and accurate information to all interested parties.”

The VA faced many challenges in achieving its One-VA vision. IT systems and services were completely decentralized. Meanwhile, budget decisions were not only decentralized, but completely detached from any system-wide IT strategy. According to an October 2005 memorandum from a former VA CIO, the CIO had direct control over only 3 percent of the department’s ITIT budget and 6 percent of the department’s ITIT personnel. Since the late 1980’s, the Government Accountability Office (GAO) had pointed out several times that given the department’s large ITIT funding and decentralized management structure, it was crucial for the department CIO to ensure that well-established and integrated processes for leading, managing, and controlling investments were followed throughout the department. Further, a February 2005 contractor assessment of VA’s IT organizational alignment also noted the lack of control for how and where money was spent. The assessment found that project managers within the field facilities had the ability to shift IT money to support varying projects, even projects having nothing to do with ITIT. Also, according to the assessment, the focus of department-level management was only on reporting expenditures to the Office of Management and Budget and Congress, rather than on managing these expenditures within the department.

The resulting plan for ITIT realignment—including single IT authority at the VA central office (VACO)—was revolutionary and unprecedented in the federal government in its range and ambition—and would become even more ambitious as transformation unfolded.

The hallmark of the effort was the scope of change—a complete reorganizing of the IT personnel reporting structure and field operations, procurement policies and development. But the effort was also characterized by the bitter and sustained opposition that it engendered throughout the VA. A health care network that had turned its reputation around in the 1990s by emphasizing field autonomy, flexibility and rigorous performance standards was suddenly being squeezed into a top-down organizational chart that many in the field and at VACO argued would be a recipe for a health care meltdown.

According to informed observers inside and outside the organization, the effort to centralize IT in the sprawling, high-profile and politically charged VA, is far and away the most ambitious undertaking of its kind ever in the federal government.
Because of the breadth and depth of the effort at VA, it clearly has the potential to offer a wide variety of lessons to other large organizations—at all levels of government, federal, state and local—for how and in some cases perhaps how not to go about such an ambitious, audacious and arduous crusade.

Furthermore, given the decentralized and sprawling (some would even argue borderline-anarchic) nature of the organization, the egos involved, the amounts of money involved, the high human and political stakes, along with the high profile of the organization and the constituency it serves, it is easy to argue that if the VA can make significant progress on such an ambitious undertaking, any organization can.

And more organizations no doubt will try, following the lead not only of the VA, but such private sector powerhouses as Hewlitt Packard, Cisco Systems, IBM and Dell, all which realized that the days where decentralized IT nodes in the same organization working in relative isolation—while certainly a positive force for encouraging creativity and innovation— just presents too unwieldy a model in a world of rapidly evolving information technology.

To harness the full power of IT—and, not incidentally, to operate systems in as secure a fashion as possible— leading edge organizations have decided that they need to get a tighter grip on far flung networks of users, and that a premium needs to be placed on IT compatibility, interoperability and standardization.

About VA
To understand how massive the job of centralizing and securing IT systems at the VA has been—and continues to be—it is necessary to understand just how large and complex an organization the VA actually is and to know something of the organization’s tumultuous history.

The Veterans Administration was officially created in 1930 by executive order, signed by Herbert Hoover. At the time the system consisted of 54 hospitals, employing 31,600 people serving 4.7 million veterans. Its focus was on tertiary care for indigent veterans.

Today, the Department of Veterans Affairs is organized into three administrations:

  • Veterans Health Administration (VHA)is responsible for the VA health system
  • Veterans Benefit Administration (VBA) is responsible for ensuring veterans get their pensions and other benefits, including educational benefits
  • National Cemetery Administration (NCA) is responsible for administering burials and operating VA cemeteries

The 500-pound gorilla in the administrative threesome is the VHA, with a budget of about $35 billion, and which oversees a sprawling complex of 155 medical centers, 872 ambulatory clinics, 135 nursing homes, 45 residential rehabilitation treatment programs, 209 veterans’ centers and 108 comprehensive home-care programs. These VA facilities, meanwhile, are affiliated with 107 medical schools, 55 dental schools and more than 1,200 other schools nationally. The VA estimates that its system helps train upwards of 90,000 health professionals a year.

As of 2008, the VA health care system had nearly 7.8 million enrollees, who were being served by more than 200,000 health care and support professionals nationwide.

Also feeding into the complexity of the organization—and how it uses information technology to meet and forward its mission—the VA does considerable amounts of medical research, focusing on areas of particular concern to veterans, from cardiac and diabetes care, to trauma care (physical and mental), to improvements in artificial limbs.

And as if all that wasn’t enough, part of the VA’s mission is also to step up during national disasters as a backup health care system, something it did with distinction in the aftermath of Hurricane Katrina, and a role that absolutely requires a high-functioning and robust IT infrastructure.

Overview of Report
This report chronicles the VA’s efforts to realign and centralize its IT activities. It needs to be stated clearly and up front that it is still a work in progress, that there are significant hurdles ahead and certain significant adjustments that will no doubt need to be made for this ambitious undertaking to ultimately be implemented and sustained by VA. A new Administration in the White House also adds to the weight of the challenge—but at the same time also offers new leadership a clear and established roadmap for moving the effort forward; a lot of hard work has been done for them.

The work that has been done at the VA has been controversial, to say the very least. High level officials have resigned on account of the push, either because they didn’t agree with it or were just burned out by the fight.

Having noted all of that, the VA continues to move forward with the ambitious effort of centralizing all IT, IT staff, administration and development under VACO, creating an “IT Governance Structure,” and building an integrated IT process and organizational model using IBM’s Process Reference Model for IT (PRM-IT) which draws on best practices from both CobiT (Control Objectives for Information and Related Technology) and ITIL (Information Technology Infrastructure Library).

According to February 2008 Government Accountability Office testimony, the department has established and activated three governance boards to facilitate budget oversight and management of its investments. Further, VA has approved an IT strategic plan that aligns with priorities identified in the department’s strategic plan and has provided multi-year budget guidance to achieve a more disciplined approach for future budget formulation and execution. While these steps are critical to establishing control of the department’s IT, it remains too early to assess their overall impact. Thus, their effectiveness in ensuring accountability for the resources and budget has not yet been clearly established.

Proponents of transformation argue that those issues will be worked out as the initial tumult caused by reorganization settles down. Clearly, however, the new VA Secretary Eric K. Shinseki and his top staff are going to necessarily have to become students of the reform effort and also become active advocates in order to keep it moving forward.

IT Governance and New Organizational Structure

To oversee the new approach to IT , the VA defined IT governance, and created an IT governance program, plan and structure consisting of a group of interlocking boards responsible for everything from strategic planning to budgeting and acquisitions.

VA defined IT Governance as: “A structure of relationships and processes to direct and control the enterprise in order to achieve the enterprise’s goals by adding value while balancing risk versus return over ITIT and its processes.” In addition, the ITIT governance plan recognizes that human beings will be involved and so sets out two sets of human resources related guiding principles essentially related to how people will operate within the new ITIT governance program. These include “ITIT Governance Imperatives (see Appendix IIIIIIIII),” which, among other things discuss the need for trust and partnerships in order to make the governance plan successful, and “ITIT Governance Characteristics (see Appendix IV),” a list of the “rules of the game” that ought to infuse the effort, ranging from “builds relationships and processes,” to “ensure that everyone is playing by the same rules and doing the right things right.”

Several critically needed, very specific IT Governance Guiding Principles (see Box 4) were also identified to ensure among other things that ITIT governance was recognized as being critical to VA’s success and that IT resources and IT program execution result in maximum effectiveness and efficiency across VA to meet requirements and deliver benefits set by VA business leaders.

Box 4: VA IT Governance Guiding Principles

  • IT Governance is critical to the success of VA Governance and business needs
  • Business (mission) requirements and benefit realization are the basis for setting IT priorities
  • Business leaders (Administrations and Staff Officers) establish IT requirements, business benefits, and priorities based on VA Strategic Plan
  • Business leaders oversee full life cycle execution of IT program to manage risk
  • The Office of Information and Technology (OI&T) determines technology solutions and IT related life cycle costs
  • VA CIO manages IT resources and IT program execution from maximum effectiveness and efficiency across VA to meet requirements and deliver benefits set by Business Leaders
  • Use existing VA Governance mechanism to maximum extent possible
  • OI&T policies, procedures and processes must be published, communicated, monitored, measured, and reported across VA
  • IT Governance enforcement must be equitable, timely, and consistent
  • Industry/Government best practices and standards are assessed and implemented as appropriate

Under the VA’s IT governance plan, an over-arching VA Executive Board makes policy, formulates budgets and also acts as ultimate arbiter in the case of conflicts among the other boards lower down the organizational chart. A group of sub-boards does everything from capital and workforce planning, to evaluating projects and priorities, to overseeing enterprise-wide IT architecture, including developing technical standards for security, and recommending system improvements.

The VA, with contractor support, did assess 150 VA facilities (hospitals, clinics, etc.) to try and figure out where the VA was with regard to existing IT capacity. But even that comprehensive review didn’t prepare VACO for just how much work would be required to consolidate and update far-flung and often out-dated IT systems, says former VA CIO Bob Howard.

Based on the contractor’s assessment and on VACO judgments about the state of IT at VA, in February 2007 the Secretary approved a new organizational structure for centralized IT management. The structure was based on industry best practices including CobiT (for “Control Objectives for Information and related Technology) and Val IT , both of which provide a framework for IT governance plans, structures and investments, and also ITIL (for “Information Technology Infrastructure Library), a set of state-of-the-art concepts and policies for managing information technology infrastructure, development and operations.

The new structure was developed to create a system to meet the VA’s IT needs that involved building and maintaining key applications, supporting operations, and monitoring IT in five key areas:

  • Enterprise management
  • Business management
  • Business application management
  • Infrastructure
  • Service support

Service support is particularly important, argues Jeff Shyshka. He says that clinicians and administrators in the field continually express the need for on-the-ground ITIT technical staff to help with the day-to-day ITIT issues and problems that pop up at the facility level.

Finally, the VA established a system for evaluating transformation, not only to gauge progress toward its One-VA vision, but to try and ensure that IT is being effectively deployed to support all those working in and being served by the VA.