Comments:"Why a one-room West Virginia library runs a $20,000 Cisco router | Ars Technica"
Marmet, West Virginia is a town of 1,500 people living in a thin ribbon along the banks of the Kanawha River just below Charleston. The town's public library is only open Thursdays, Fridays, and Saturdays. It's housed in a small building the size of a trailer, which the state of West Virginia describes as an "extremely small facility with only one Internet connection." Which is why it's such a surprise to learn the Marmet Public Library runs this connection through a $15,000 to $20,000 Cisco 3945 router intended for "mid-size to large deployments," according to Cisco.
In an absolutely scathing report (PDF) just released by the state's legislative auditor, West Virginia officials are accused of overspending at least $5 million of federal money on such routers, installed indiscriminately in both large institutions and one-room libraries across the state. The routers were purchased without ever asking the state's libraries, cops, and schools what they needed. And when distributed, the expensive routers were passed out without much apparent care. The small town of Clay received seven of them to serve a total population of 491 people... and all seven routers were installed within only .44 miles of each other at a total cost of more than $100,000.
In total, $24 million was spent on the routers through a not-very-open bidding process under which non-Cisco router manufacturers such as Juniper and Alcatel-Lucent were not "given notice or any opportunity to bid." As for Cisco, which helped put the massive package together, the legislative auditor concluded that the company "had a moral responsibility to propose a plan which reasonably complied with Cisco's own engineering standards" but that instead "Cisco representatives showed a wanton indifference to the interests of the public in recommending using $24 million of public funds to purchase 1,164 Cisco model 3945 branch routers."
In other words, the project has been a stellar example of what not to do and how not to do it.
Clay, WV, a tiny outpost among the mountains, has 7 Cisco 3945 routers within .44 miles of each other.A million here, a million there
The routers in question were purchased as part of a much larger grant from the Broadband Technology Opportunities Program (BTOP), which passed out several billion dollars to help upgrade broadband networks across America as part of President Obama's initial stimulus package in 2009. West Virginia's cash was meant to wire up the many "community anchor institutions" such as libraries, schools, police, and hospitals across the state with Internet access delivered over fiber-optic lines. As part of the project, the state also had to purchase some sort of router for each institution. Instead of "right-sizing" the routers for their intended destinations, the state group of officials charged with implementing the grant decided they would make things easy by purchasing the exact same router and installing it everywhere, even in the most rural locations they planned to reach.
This became controversial in 2012 when local newspapers brought the issue to light and questioned whether the state had not just been boondoggled. The Charleston Gazette noted an official in the state's Office of Technology had actually e-mailed his colleagues to say "this equipment may be grossly oversized for several of the facilities in which it is currently slated to be installed" but that the warning was not heeded. The issue quickly escalated to Congress, where officials from the executive branch were grilled about the West Virginia situation and whether the federal government had exercised enough oversight of the project.
The state of West Virginia has now weighed in with its own report on the routers, and it makes for mind-boggling reading. Consider, for instance, how routers were purchased for the state police. When the West Virginia State Police purchased their own routers a few years earlier, they chose Cisco model 2xxx machines at a cost of only $5,000 or so apiece, with only a single Cisco 3xxx model purchased for the largest deployment. In 2010, when the state received its grant money, no one asked the State Police what they wanted or needed; indeed, the police were "never contacted" at all by the Grant Implementation Team. (This was a widespread problem; the report notes no capacity or user needs surveys were ever done before the money was spent). Instead, the team simply ordered 77 Cisco 3945 routers at a cost of $20,661 apiece—that's one $20,000 router for every 13.7 state police employees—and sent them off to the police. (Each router can handle several hundred concurrent users.)
Had the Grant Implementation Team replaced 70 of these routers with the cheaper model, the state could have saved $1.4 million. And that's assuming that the routers were even needed to begin with—in many cases, they were not.
Such cost savings could have been found all over the state. Nearly all of the West Virginia's 172 libraries could have saved $16,000 per router, saving the state $2.8 million more. Many of the state's public schools are likewise small institutions that could have easily used smaller routers and saved another $3.68 million. In total, another $5+ million could have been spent on tech that was actually useful for the state's residents.
What was the grant team thinking?
Cisco's 3900 series of routers. CiscoHow it happened
The state Office of Technology contends the massive routers might save the state money in the long run by supporting cheap VoIP systems instead of standard telephone lines. But the legislative auditor notes that each of the 3945 routers can handle 700 to 1,200 VoIP lines, which means that the 1,164 routers purchased by the state could support up to 1.39 million lines. As the auditor's report dryly notes, only a single library in the entire state has more than eight phone lines; most have one or two. (None use a VoIP system anyway.)
Ironically, the routers can't even be used for VoIP in some key cases. The state police already have a VoIP-based phone system, but the new 3945 series routers did not come with "the appropriate Cisco VoIP modules" to work with the system. The state now has to spend another $84,768 to purchase those modules; without them, the state police can't use the routers, only two of which are actually installed and operating. (For those keeping score at home, this means that 75 $20,000 routers are depreciating in a state police warehouse somewhere in West Virginia.)
The report also lays a good deal of blame on Cisco and on the company's engineer for the project. The engineer told the auditor he was simply following the state's instructions, which required him to spec out a proposal using only routers with "internal dual power supplies"—hence the 3945s. As the auditor dug into the story, demanding to know when this exact request was made, the Cisco engineer said it originated with the state Department of Education. But the engineer was "unable to provide the legislative auditor with any e-mails or other documentation" to this effect.
Enlarge / The legislative auditor's report.The auditor began digging, speaking to many people in West Virginia state government who had been involved with the project. The Department of Education told him that it "did not request or require that the routers for the state's schools have internal dual power supplies. Education would not have made this requirement because unless a school has two power sources the feature of dual power supplies would have no use." A network engineer for the Department of Education confirmed that he had not requested such a feature.
So the auditor went to the state's Office of Technology, which was also involved in the project. An employee there said that dual power supplies had come up, but only for "24/7/365 locations such as regional jails and DHHR state hospitals." VoIP support was discussed "but not required," he added, and he concluded by saying, "It was never implied to put each feature in all routers."
Cisco defended itself by saying it had drawn up a complete spreadsheet of its proposed bid, and the state had reviewed it. If it didn't need or want these features, or if it thought the routers were too large, it should have said so.
The legislative auditor was also apparently quite peeved by this entire investigation. The auditor's office sent off a fairly testy e-mail to Cisco noting that the 3945 routers were not appropriate for most West Virginia deployments—even according to Cisco's own literature. "I would appreciate an explanation as to why you believe the 3945 routers are not oversized and misconfigured for hundreds of locations," the auditor concluded, "and, thus, a significant over expenditure of millions of dollars for Cisco equipment." The Cisco rep responded the state had reviewed his spreadsheets and not objected and that the 3945s were large enough to allow for future expansion.
The auditor then asked the legislature's own tech team what they used. The West Virginia legislature at peak times can have over 600 internal users and numerous guests accessing "multiple Web servers, up to eight simultaneous live audio webcasts, multiple SQL servers, and multiple Google search appliances located in the Legislature's server farms." Despite all this, the legislature doesn't even use a router but instead runs a cheaper Cisco switch... and it has never exceeded capacity.
The auditor asked one of the legislature's network specialists if he would even want a 3945 router; the man said no because "it greatly exceeds the Legislature's needs." And yet somehow more than 1,000 of them had been sent to the very furthest, most rural corners of the state.
Debarment
The report finds plenty of blame to go around. The ultimate cause of the fiasco, it says, was the fact the grant implementers did not conduct a capacity or use study before spending $24 million. They also used a "legally unauthorized purchasing process" to buy the routers, which resulted in only modest competition for the bid. Finally, Cisco is accused of knowingly selling the state larger routers than it needed and of showing a "wanton indifference to the interests of the public."
Getting any of the money back seems unlikely at this point, but the legislative auditor does have one solid recommendation to make. The State Purchasing division should determine whether Cisco's actions in this matter fall afoul of section 5A-3-33d of the West Virginia Code, and whether the company should be barred from bidding on future projects.
Cisco tells Ars "the criticism of the State is misplaced and fails to recognize the forward-looking nature of their vision. The positive impact of broadband infrastructure on education, job creation, and economic development is well established, and we are committed to working with the State to realize these benefits for the people of West Virginia now and into the future."
As for that $5+ million the state could have saved, it would have paid for 104 additional miles of fiber.