The Pennsylvanie State Senate Law and Justice Committee heard testimony regarding the State’s digital public-safety radio system on April 19, 2010. Users of the radio system testified as to horrific performance problems and pleaded to the Committee that the State’s VHF analog conventional radio system be preserved.

The Pennsylvania State radio system is based on the OpenSky product from Harris/Macom, however my research and experience shows that users of large digital public-safety radio systems from other manufacturers report similar problems. The problems aren’t limited to to OpenSky systems. Click here for articles related to this topic.

What happens after the P25 and OpenSky digital radio bubble bursts?  Click here for my opinion.

–

May 29, 2010, 10:00AM

The state of Oregon’s plan to build a sprawling emergency radio network is a year behind schedule, and state officials now say they can’t be sure what the project — estimated at $485 million — will actually cost taxpayers.

The Oregon Wireless Interoperability Network, or OWIN, is designed to let police, firefighters, highway crews, rescue teams and other public safety officials across Oregon talk to each other on their radios, something they can’t do now. The federal government is requiring public safety agencies to switch to new frequencies by the start of 2013.

The Legislature approved OWIN five years ago to meet the deadline, and it’s become one the biggest construction and technology efforts that Oregon has ever tackled.

But the project is in trouble.

The radio network is supposed to be made up of about 300 microwave towers, most stationed atop Oregon ridges and mountains.

But an examination by The Oregonian has found that OWIN officials have only six sites ready for construction. Crews haven’t broken ground on more than a dozen towers that were supposed to be done by now. And an outside consultant found OWIN has been operating without cost controls or a comprehensive schedule that can tell project officials if they can meet their deadlines.

In February, lawmakers got so frustrated with OWIN’s inability to account for its progress that they froze most of the project’s funding, leaving just enough cash to let the agency start some tower sites.

“I’m just waiting for one report that says the OWIN project is stable — just one,” said Senate President Peter Courtney, D-Salem.

Now, state officials and legislators must weigh releasing the project’s funds as they face billions in budget shortfalls and projected deficits that likely mean cuts to police, prisons, education, health care and protections for children, the elderly and the mentally ill.

In two recent interviews, OWIN officials told The Oregonian the project is on schedule. But agency documents, released to the newspaper under the state’s public records law, raise doubts about the claim.

An OWIN official said in a public meeting last week that, under its current pace and schedule, the project won’t be done until late 2013 or 2014. That’s a year behind the deadline set by the federal government. He said OWIN would in some cases have to double its speed to finish on time.

Lindsay Ball, the project’s director, continues to maintain the project is on schedule but said changes are being made to make sure the project moves faster.

“We have X number of sites and we have to get them done by 2013,” Ball said. “Everything that I’ve been told to date is that’s doable.”

In freezing the project’s funding, the Ways and Means Committee also pushed Gov. Ted Kulongoski to move the OWIN project from under the control of the Oregon State Police to the Oregon Department of Transportation, which has a track record of building construction projects.

ODOT Director Matt Garrett said it’s his understanding the project is behind schedule and his agency is still trying to make sense of the project’s deadlines and budget.

“We talk big numbers,” Garrett said. “But we really need to bring scrutiny to the numbers, the budget, the schedule, the timelines and then how this thing rolls itself out.”

When asked what the project will cost, Garrett replied, “We don’t know yet. That’s a reality.”

Ball, who is retiring Aug. 1, is a former director of the state Department of Administrative Services. On the OWIN project, he answers directly to Kulongoski.

The governor declined to be an interviewed for this story. Kulongoski spokeswoman Anna Richter Taylor said the governor thinks Ball has made tremendous progress with the OWIN project.

“Has this project been more challenging than we thought going in? Yes, without question,” Taylor said.

“While there have been bumps along the way, the governor feels the direction it’s going is the right one and he’s confident (ODOT director) Matt Garrett is the right leader to move this forward.”

Lawmakers say they’ve not been told how far behind schedule the OWIN project has fallen.

“Whenever they talked to us, it was always that they were doing well,” said Rep. Tim Freeman, R-Roseburg, who heard briefings on OWIN while serving on a legislative budget panel and as vice-chairman of the House committee overseeing emergency services.

“But it’s been this pie-in-the-sky thing,” Freeman said. “It’s frustrating spending so many millions and not knowing what the end point is or how much it will really cost.”

OWIN to date has spent $22 million, much of it on radio equipment for tower sites belonging to other government agencies that have joined the state radio network.

Getting local agencies to sign on to the OWIN project has been one of Ball’s biggest accomplishments. Those partnerships helped increase political support for the project around Oregon.

“It’s going to mean we can talk to our neighboring counties during times of emergencies,” said Tillamook County Sheriff Todd Anderson. “And it prepares us for any of the natural disasters that can happen here in Oregon.”

OWIN officials say the partnerships allow agencies and the state to share towers, equipment and expenses, saving the state $60 million.

Ball said the project has been held up by delays in getting its construction company, General Dynamics, under contract to build the radio towers sites. The company signed on in April. Documents show that, even if the contractor had been hired earlier, many OWIN sites still would not have been ready.

Even though lawmakers have frozen most of its budget, the OWIN project has enough money to start construction at some sites in June and July.

Courtney, the Senate president, said it’s going to take an independent report that OWIN is on track before lawmakers release the rest of its money.

“This thing has been a monster since I first started to hear the word ‘OWIN,’ and for some reason we can’t get our hands around this thing,” Courtney said.

After months of wrangling with lawmakers, OWIN officials brought in an outside quality assurance consultant to assess the project. The consultant’s conclusion: “The OWIN Project is well managed and under control.”

But the report’s fine print tells a more complex story. The consultant found the OWIN project lacked cost controls and a comprehensive schedule to meet the 2013 deadline. “Internal schedules do not yet support meeting this completion date,” the report says.

ODOT’s Garrett said that coming up with a defensible price tag and proving the project is back on a reasonable schedule is crucial.

“We have to re-instill confidence that this is a good investment for taxpayers of the state of Oregon,” Garrett said. “This is a lot of money at stake.”

– Brent Walth

I see serious ethical issues when a government entity which is directed by a former Motorola employee recruits a staff of former Motorola employees and then enters into an exclusive and unique business arrangement that would allow Motorola to potentially acquire incredibly valuable spectrum and contracts without a competitive selection process.  This is not in the best interest of the cities and counties in the Bay Area.

This is not the first appearance of impropriety with regard to Bay Area SUASI and Motorola.

• The appearance of impropriety (Part 1)
• The appearance of impropriety (Part 2)

Public safety agencies in the Bay Area already benefit from superb wireless data access provided by Verizon,  Sprint and ATT.  The cost is approximately $50 per month per mobile unit, with no capital investment or capital replacement expense for the subscribing agencies. These carriers have all announced plans to migrate from existing 3G technologies to a 4G systems based in LTE or WiMax.  Would  the Federal grant-funded Motorola deal offer any advantage to Bay Area municipalities, or is the benefit focused on Motorola stakeholders?

Does SUASI’s sponsorship of Motorola’s grant proposal uniquely position Motorola to compete with the incumbent wireless carriers?  Will Motorola provide the hardware, software, engineering and system integration services and ongoing operational support for this proposed taxpayer-funded network without competitive bid?   Does SUASI’s action give an unfair competitive advantage to Motorola?  Is this yet another appearance of impropriety at Bay Area SUASI?

Click here to read the full article as it appeared in Mission Critical Communications, Radio Resource International.

Motorola Applies for $50M in Grants for Bay Area Broadband Network – 04/12/2010

–

On February 24, 2010 the Maple Bluff Fire Department (MBFD) in Dane County Wisconsin performed tests of several portable radios operating in simplex analog and simplex P25 digital mode. They compared the sound quality of the Kenwood TK-5210, Motorola XTS-2500, Motorola XTS-5000, Motorola APX-7000, EF Johnson SL 5100, Tait 9135, and ICOM 9011 under normal firefighter working conditions.  The results of the tests are documented in this video.

The Maple Bluff Fire Department is not endorsing any particular manufacturer, vendor, or radio type, and understand there are radios on the market that were not tested. The radios seen in this test were selected by availability at the time of the test through local vendors.

The tests done by the Maple Bluff Fire Department lacked many controls that would typically be found in a scientific test. Nevertheless, I think the MBFD folks did good work and that the evidence recorded in the video is significant. The most important point from my perspective is the vastly different performance of radios in simplex P25 mode. Perhaps every fire department needs to perform similar testing before making the decision to “go digital.” Please remember that there is no FCC requirement to convert to digital operation.

Please download their Radio Test Worksheet to record your opinion of each radio in their test.

Click here to visit the Maple Bluff Fire Department website.

If you have questions about this test, please contact:

Chief Josh Ripp: jripp at villageofmaplebluff.com
Assistant Chief Brad Ingersoll: bingersoll at villageofmaplebluff.com

Interoperability Money Aids Motorola and Other Contractors, but Are First Responders Better Off?

When a cop or a fire fighter pulls out a radio in a television police drama, his message goes through, whether he’s in the basement of a building or deep in a forest. In the real world, clear communication is rarely so easy, particularly among first responders from different disciplines and jurisdictions. This reality was dramatically brought home at the World Trade Center on September 11, 2001, when crucial observations from the police department’s helicopters did not reach fire chiefs, commanders lost radio contact with responders who ascended the towers, and brigades in the north tower did not hear calls to evacuate.

Since then, an unprecedented amount of federal money has been spent on communications gear and technology, expenses traditionally borne by state and local governments. The goal is to fix the communication problems faced on 9/11 — to create “interoperability” that allows first responders from different disciplines and jurisdictions to communicate. From 2004 to 2008, the only years for which detailed figures are available, the Department of Homeland Security (DHS) approved more than $4.3 billion in grant money to improve interoperability among first responders nationwide. DHS officials have said that more grant money has gone to interoperability than to any other initiative, and it continues to be a major focus for DHS grant programs, while also drawing funding from the economic stimulus package.

Yet for years, results have failed to live up to expectations. In 2004, then-DHS Secretary Tom Ridge promised that by year’s end, it would be possible for most first responders to talk to each other in a crisis. But in 2005, Hurricane Katrina proved that the country was nowhere near ready to handle a real disaster. By 2009, DHS officials were still struggling to convince Congress that first responders could reach basic communications goals.

“The fundamental question all of us have is this,” said Rep. David Price, whose subcommittee oversees homeland security funding, at a hearing last March. “Why aren’t we making faster progress toward effective interoperability?”

An Over-Reliance on Hardware

Part of the answer is that early on, Congress directed the vast majority of DHS interoperability dollars towards hardware: portable and car radios; “repeaters” that extended a signal’s range; antennae and tower systems. But an array of communications experts and even DHS officials say that equipment alone cannot create interoperability.

“There was a lack of understanding in the congressional committees about the importance of planning dollars, that you could waste money if you don’t plan,” says Harlin McEwen, a former police chief who chairs the communications committee at the International Association of Chiefs of Police. “They didn’t want to spend money on things you couldn’t see.”

Support for those less tangible needs — sitting down at the table with counterparts, hashing out plans, establishing command-and-control protocols, training users — has come much more slowly. And experts say that without such support, the value of spending on equipment is questionable. A 2007 report from the Government Accountability Office warned that the lack of coordination was leading to investments in “independent interoperability systems that do not always support each others’ needs.”

DHS and Congress have increased support for planning and training. But as recently as 2008, more than 75 percent of interoperability dollars still went towards hardware.

The challenges of interoperability could offer lessons for any of the troubled homeland security initiatives launched after 9/11, from increased airport security to a “virtual fence” laden with sensors at the Mexican border. Billions have been spent in an effort to make America safer, but critics wonder whether the push primarily has benefited a burgeoning homeland security industrial complex. Perhaps the only hands-down winners have been the companies that supply communications equipment — Motorola, in particular. Billions in DHS funding have helped public safety agencies replace aging infrastructure, but the effects on those agencies’ interoperability capabilities vary widely. While some jurisdictions report substantive progress, for many of the responders risking their lives, the communication problems brought into sharp relief on 9/11 linger on.

History’s Lessons

September 11 is only the most recent entry in a litany of disasters that documented the need for interoperability. Among them: the 1982 crash of an Air Florida Flight 90 into a Washington, D.C., bridge; the 1993 bombing of the World Trade Center bombing; the 1995 bombing of an Oklahoma City federal building; and the 1999 shootings at Colorado’s Columbine High School.

In each case, as a grab-bag of agencies converged on the scene, precious time and efficiency were lost. “Many police officers, firefighters, and emergency medical service personnel working in the same city,” wrote a federal advisory commission in 1996, “cannot communicate with each other.”

The challenge is not just incompatible technology, but disparate cultures. “It’s like three high school [sports teams] in a city,” explains Chief Douglas Aiken, communications chair for the International Association of Fire Chiefs. “They each want to win, and they each have their own funding. They might see each other and talk in passing, but they each are doing their own thing.”

When public safety agencies first began using radios, in the 1920s and ’30s, their equipment was built into cars and used primarily for dispatch. As radios became lighter and more portable, responders gradually came to consider them critical tools. The federal government assigned public safety departments slices of the radio spectrum — electromagnetic real estate — and each agency bought a radio system that operated only within specific frequencies. Systems from different manufacturers could not communicate.

Motorola, based in Schaumburg, Illinois, has long been the undisputed leader in public safety communications equipment, controlling about 80 percent of the U.S. market. By 2001 the firm had cemented its good standing with important groups like the Association of Public-Safety Communications Officials (APCO). The company often sponsors APCO’s state and local events at the most generous levels — “gold” for the 2009 national conference, for instance. Motorola has a few competitors — Harris Corporation, based in Melbourne, Florida, first among them — but local officials consider Motorola the safe choice.

When public safety radios began transitioning to digital technology, organizations like the police chiefs’ and fire chiefs’ associations resolved that they would require manufacturers to provide radios that worked together. In 1989, APCO, whose membership includes both government officials and industry players, began working with vendors to design a crucial series of standards for interoperability, known as Project 25 or P25. Radios that met the P25 criteria in theory would work together, regardless of vendor. The standards define eight different interfaces between various parts of a radio system: the first, for instance, allows two radios from different manufacturers to communicate directly to each other. On 9/11, though, the standards were still incomplete.

Few in the federal government had studied interoperability before 9/11, but in the wake of al-Qaeda’s attacks, policymakers were troubled by the chaotic responses, particularly at Ground Zero, where post-action reports identified a host of issues, from technical failures to breakdowns in command procedures. Emergency communications interoperability suddenly became a national crisis, and Congress turned to the industry and to first responders for solutions. Many first responders asked that Congress dedicate more spectrum to public safety. And Greg Brown, now CEO of Motorola, told Congress in 2003 that the “common, and key, requirements to achieving interoperability include spectrum, standards, and money.”

Money was the first to arrive. After 9/11, Congress began funneling emergency funds to all 50 states. Those funding streams turned into more formal grant programs in 2004: the State Homeland Security Grant Program provided a baseline amount of funding to all 50 states, and the Urban Area Security Initiative handed out additional money to cities that faced greater risks. State agencies charged with overseeing homeland security applied to these and other programs for cash to solve problems, including interoperability, then distributed their allocation to state agencies and local governments. These recipients needed new radio systems and Congress was telling them, go out and buy what you need.

The Missing Link: Planning

Department of Homeland Security officials have said that more grant money has gone to interoperability than to any other initiative, and it continues to be a major focus for DHS grant programs, while also drawing funding from the economic stimulus package. Left out of this discussion, however, was one element that everyone now agrees is crucial: planning. While money began flowing towards interoperability, requirements to plan and coordinate have lagged badly behind expenditures on hardware.

“It is as important as the technology,” says Bob LeGrande, who headed the District of Columbia’s interoperability development. “One cannot exist without the other.”

Without strong relationships in place, agencies have worked towards achieving interoperability locally, without thinking on a regional level, leading to what the GAO has called “the narrow and specific use of DHS funding.” Or they have bought equipment that’s incompatible with surrounding jurisdictions. A 2007 audit of interoperability spending in Colorado worried: “Without … the context of a regional or statewide plan, it is unclear how much unmet need will be addressed by these funds.” DHS officials also worry about “islands of interoperability” — places where jurisdictions have upgraded to new technology without considering how they’ll connect with partners using old gear.

With the right equipment, in theory, every responder could talk to every other responder. Simple connectivity, though, does not guarantee coordination. On 9/11, in New York, those police officers and fire fighters whose radios could connect faced a different challenge. So many people were using the tactical channel meant for interoperability that it became overloaded — a cacophony in which important information could not get through.

The DHS vision for interoperability goes beyond networking first responders’ communication equipment. To DHS officials, interoperability means that officers will be able to communicate with anyone they need to, but only when they need to and when they are authorized to do so. To avoid chaos, they say, public safety officials need to establish strong command and control protocols, standard operating procedures, and, most of all, trust.

Robert Desourdis, an engineer and consultant who has studied interoperability, remembers one exercise during which, under pressure, agencies reverted to using their own systems, instead of using the new technology that was meant to connect them. As Col. G. Jerry Russell, director of the Idaho State Police, put it, “Technology without coordination results in inoperability.”

One academic study of DHS grant spending found that strong planning and coordination correlated with success at creating interoperability, while simply increasing funding did not. Chicago, for instance, has received more than $220 million dollars from the Urban Area Security Initiative grant program. Yet, in a 2007 evaluation of cities nationwide, the city earned DHS’ lowest score on governance, which measured the strength of the formal agreements that provide a foundation for communications planning. Erica Chenoweth and Susan Clarke, who conducted the study, concluded that Chicago’s performance “stems from politics rather than resources,” citing problems like disagreements between two neighboring counties, a lack of clear leadership and accountability, and unfamiliarity with standard operating procedures.

Building the sort of relationships that yield real interoperability can take years. LeGrande credits the success of the D.C. area’s system — one of the few to score 100% on the DHS evaluation — to work that began after the Air Florida crash in 1982. Chris Essid, head of the DHS Office of Emergency Communications, remembers how long it took for the planning process to start working well in Virginia, where he became the statewide interoperability coordinator in 2003. “Our first meeting, you had folks from the localities that didn’t trust the state folks,” he says. “It took a while to get these folks to realize — we have the same problem and we can help each other.”

But when DHS started doling out money, a lot of that hard work had simply not been done. In December 2006, the department reported, “Strategic plans for interoperability are the exception rather than the norm.”

And the lack of planning showed. “You had different fiefdoms doing their own thing,” says Veronique Pluviose-Fenton, a former aide to the House Homeland Security Committee. “It was almost like you were throwing money at the problem without measuring whether or not it was a solution.” Another congressional aide said there was “a lot of money floating around, and if a project met the right buzzword, it got funded.”

In Georgia, for instance, $11 million in federal funding helped build the Georgia Interoperability Network. Months after the system was installed, none of the 81 local dispatch centers connected to the network were using it, according to a 2008 audit.

In the first years of the grant programs, not all emergency management departments had the capacity to oversee the money coming in, either. “Many state and local governments were overwhelmed by the volume of money and the grant guidance,” says Bruce Baughman, a former FEMA official who also served as the director of the Alabama Emergency Management Association.

Chasing the Money

As big money began flowing, a variety of companies saw a business opportunity. Raytheon pitched a product called the ACU-1000, a so-called gateway device that could hook multiple networks together to improve communications. Another popular idea: cells on wheels — COWs for short — involved trucks outfitted with radio gear that could be driven to a disaster site.

In this new gold rush, Motorola had a distinct advantage. Most public safety officials wanted to stick with the solution they knew — the land-based radio systems they already owned and operated — and upgrade to interoperable equipment. In this market, that meant P25 equipment, and in 2001, Motorola was the leading supplier, the only supplier of some P25 components for certain frequencies.

In 2001, though, P25 technology was far from perfect. The only standard that was fully developed was the “common air interface” that allowed individual radios to connect with each other. Standards that would link whole systems together were still in flux. Even so, after 9/11, the federal government recommended that grantees buy P25.

That was a boon for Motorola, as well as for competitors like Harris. “[Our] business is well positioned to participate in this emerging opportunity as customers solidify their funding for safety and security,” said a 2002 Motorola disclosure filed with the Securities and Exchange Commission.

Motorola’s government business is not its largest component, but “it’s by far the most profitable thing they do,” says Tavis McCourt, a senior research analyst at investment firm Morgan Keegan, who follows the company. Motorola has reorganized several times since 2001, changing the way it reports profits, so tracking its public safety business is difficult. But in recent years, Motorola has identified homeland security as the driving force behind increased sales in its government business. In 2008, public safety drove the company’s $1.5 billion operating profit in its enterprise mobility sector; its mobile phone business, by contrast, lost $2.2 billion. Motorola recently announced that, starting in 2011, its government segment, which manufactures emergency radio equipment, will be spun off as part of an independent company. Motorola declined to respond to inquiries regarding this story.

Motorola: Lobbying and Campaign Cash

A variety of vendors positioned their products as the solution for interoperability, but none more aggressively than Motorola. Motorola’s lobbying budget nearly doubled, from just under $4 million in 2002 to $7.6 million in 2005, before dropping modestly in more recent years, according to Senate lobbying records. Alongside increasing in-house lobbying activities, outside lobbying assistance mushroomed, as well. In 2002, the company spent just $300,000 on contract lobbyists; in 2005, outside lobbyists reported $2.3 million in business from Motorola. Over that period, the number of lobbyists reporting Motorola as a client increased from 46 to 105. And while, before 9/11, the company’s work in Washington for public safety focused on spectrum regulation, by 2004, interoperability, now a powerful concept, appeared in all of its lobbying reports.

The company’s political action committee also increased its activity after 9/11. Contributions to political candidates jumped from $202,201 in the 2001-02 election cycle to more than $524,000 in 2005-06, before dipping just a bit in 2007-08, according to the Center for Responsive Politics. Top recipients of Motorola contributions included Illinois politicians like Rep. Mark Kirk and Sen. Dick Durbin, as well as members with new security-related duties, like Rep. Christopher Cox (R-Calif.), the first chair of the House Homeland Security Committee, and Rep. Henry Cuellar (D-Texas), now chair of that panel’s emergency communications subcommittee. President Barack Obama did not accept donations from PACs during his presidential campaign, but his Senate campaign received $4,500 from the company between 2004 and 2006.

As money trickled down from the federal level, Motorola ramped up its lobbying in state capitals, as well. In New York, the company’s lobbying budget grew from $48,000 in 2005, when it listed no lobbying issues on disclosure documents, to more than $444,500 in 2008, when its focus was on public safety communications, according to state lobbying records. In California, Motorola’s lobbying increased from more than $74,000 in the 2003-2004 cycle to more than $171,000 in the 2007-2008 cycle.

On a federal level, the company’s effort focused on ensuring its products, including P25 systems, would be eligible for grant dollars. Motorola executives testified at numerous hearings on emergency communications problems, emphasizing the need for more dollars. Lobbyists also held meetings with officials at the Office of Management and Budget and at the Office of Homeland Security, who were hammering out the details on what sort of products grant money could buy.

A Motorola Standard?

That work paid off. “The debate was driven by companies like Motorola and Ericsson [now Harris] to ensure that the solutions that were promoted were these large, statewide systems,” says one homeland security consultant who has worked with both state and federal governments.

Motorola wasn’t the only company peddling its equipment, of course. Homeland security leaders often mention those gateway devices, like Raytheon’s ACU-1000, as a cost-effective option for creating interoperability. (Whereas P25 systems can run into the hundreds of millions, gateways cost much less — nowadays, around $10,000.) But when DHS published its “interoperability continuum,” showing the steps public safety agencies could take to reach interoperability, gateways ranked lower.

Besides the continuum, it was the fine print of those DHS grant programs that provided the strongest push for state and local governments seeking interoperability money, and in 2003 and 2004, that fine print emphasized the systems meeting the P25 standards. “All radios purchased under this grant should be APCO 25 compliant,” stated an official DHS guidance to grant applicants. The language has softened since then, but DHS still says that the highest level of interoperability should include a “standards-based shared system,” in practice, a P25 system.

In the first years after 9/11, procurement requests from state and local officials for P25 systems often garnered only one bid, Motorola’s. An audit of a communications project in Alaska, for instance, reported that, because of the type of equipment the project was looking for — a P25 system — the only vendor able to fulfill their requirements was Motorola.

Although some officials chalk up Motorola’s lead in P25 to its dominance in the industry, others worry that Motorola’s long relationship with and support of APCO gave the company an additional boost. “The Motorola influence gets those standards written in a way that’s beneficial to them,” says one long-time public safety official.

A few different companies were involved in APCO’s standard-setting process from the beginning, but Motorola was by far the biggest and had the most resources to devote to research and development, according to sources familiar with the process.

“There have always been allegations that P25 was a Motorola standard,” says Chief McEwen. That’s because, he says, Motorola was the company that started developing products in cooperation with the public safety community’s push towards standards, while others were more reluctant. “We were encouraging all companies to participate in the standard, and Motorola was the one to do it,” he says.

Failing To Meet Requirements

While some jurisdictions praise their P25 systems, others have run into trouble. Delaware was one of the earlier adopters of P25 equipment, and state officials say the system has served them well. But New York State canceled a contract worth more than $2 billion for a statewide system built by Harris’ predecessor (then called M/A-Com), after the system failed a battery of tests. (The company disputed the test results and argued that the state canceled the contract for its own budgetary reasons.)

It is not clear that P25 equipment sold throughout this decade lives up to its promises, no matter who’s making it. Some essential standards are still in development, notably the one that will connect responders who use different vendors’ equipment and different frequencies. The National Institute for Standards and Technology (NIST) started testing P25 equipment in 2004, and in 2006 Dereck Orr, NIST’s program manager for public safety communications, reported to Congress, “NIST found that none of the available radios met all aspects of the standard.” In 2007, a GAO report recommended that DHS remove the P25 recommendation until the standards were finished.

“While states and localities have purchased Project 25 radios at the direction of DHS, there is little indication that these radios have enhanced interoperability,” the report stated. “Most jurisdictions we visited were not using the Project 25 capabilities.” DHS, however, boasted of “tremendous recent progress with P25” and said that that “muting the P25 language would remove all incentives for manufacturers to participate in completing the P25 standards.”

In 2007, DHS did promise to work with vendors to methodically verify if P25 products can actually interoperate across equipment from different manufacturer; grantees could then buy only those that passed. The department finally certified labs to test products in 2009. The first phase of testing applies only to the first standard — the common air interface, which allows radios to connect with each other. Certificates of compliance will be posted online; the first three were awarded to Harris in December 2009. These tests do not cover newer P25 equipment that links together systems operating on different frequency bands; DHS is working with public safety agencies to pilot a new technology, multi-band radios, which serves the same purpose.

Making Progress

After years of giving out money for equipment, DHS is starting to focus more on planning. Following Hurricane Katrina, Congress created an ongoing interoperability grant program that supports planning, although, at $50 million per year, it represents only a fraction of the total dollars going to interoperability. Congress also pushed DHS to create the Office of Emergency Communications (OEC), which now works toward achieving interoperable emergency communications nationwide. The OEC has focused on tasks like helping states create statewide plans and interoperability governing bodies. For most of 2007, its first year of life, however, the office was short-handed.

Still, by October 2007, all 50 states had submitted plans for how they would achieve interoperability. Many of the original plans needed to be revised, however. As of July 2008, the OEC had created an emergency communications plan that set national interoperability goals for the first time, and reported significant progress. As of July 2009, 46 of 55 milestones set out for the plan’s first year were met.

OEC has also been meeting with statewide interoperability coordinators, many of whom are defining positions that did not exist a few years ago. Essid, the office’s director, says that 70 percent of the requests coming to OEC’s technical assistance lines are for help on governance, standard operating procedures, training, and exercises. Essid recalls one police officer pulling out his radio, and telling him: “I get more training on infectious diseases than on my radio. This thing knows how to do things I have no clue.”

There remains a fundamental dissonance between how legislators and public safety leaders look at interoperability. Congressional leaders often talk as if one more policy decision, like an increase in grant funding, will solve the interoperability problem. Public safety leaders and DHS officials, however, talk about interoperability as a core goal that will need attention for as long as technology advances. In this view, public safety workers regularly need to refresh their communications skills, the way police officers qualify to operate their weapons or fire fighters train with their equipment.

“In Congress, they tend to want to identify a problem, fix it, and go on to something else,” says Chief McEwen. But when it comes to interoperability, he says, “The problem will never be totally solved. It’s a moving target.”

G.W. Schulz of the Center for Investigative Reporting contributed to this article.

(Reprinted with permission)

Hamilton County (Ohio) firefighters hate a communication system that taxpayers spent $35 million on because it doesn’t do what it’s supposed to do when they’re in a burning building: Let them talk to each other.

The radio failures – which some firefighters say happen daily – played a role in the death of two Colerain Township firefighters nearly two years ago and local fire chiefs say if a solution isn’t found other firefighters’ lives are at risk.

The issue surfaced again last week when radio problems resulted in a near miss for Cincinnati Firefighter Kevin Phillips who fell down a set of stairs, which knocked his helmet and air mask loose, causing him to become disoriented.

A transcript of the radio transmissions obtained by The Enquirer shows it took three minutes before fire command at the scene responded to the mayday call by Phillips’ partner, an eternity in a burning building, firefighters say.

Cincinnati Fire Chief Robert Wright did not return two calls for comment. Firefighters at the Glenway blaze say Wright met with them Tuesday, listened to their concerns and asked what would help them.

Hamilton County taxpayers spent $35 million on the digital Motorola system which was activated in 2003, replacing a fire radio system that dated to the 1940s and a police system that was nearly as old. The new system was hailed for allowing agencies to talk to each other, an impossibility on the old system.

But problems crop up when more than one firefighter tries to talk at once, resulting in firefighters getting a busy signal when they try to call and in garbled transmissions due to background noise like engines and burning buildings. In addition, metal construction blocks communication when firefighters are inside large buildings like downtown high rises or hospitals.

“We’re finding out this is happening every day, not just in big fires,” said B.J. Jetter, Sycamore Township Fire Chief and president of the Hamilton County Fire Chiefs Association.

“And not only with fire and EMS, but for police too,” he said. “It’s system wide.”

The issue came to the forefront in April 2008, when Colerain fire Capt. Robin Broxterman and firefighter Brian Schira died in a blaze on Squirrel’s Nest Lane. A review of the radio calls made during the fire showed the firefighters repeatedly made mayday calls, which were never transmitted.

“I don’t think the full impact or level of concern (about the issue) was realized until we were able to see the radio call log from the Squirrel’s Nest fire,” said Colerain Fire Chief Bruce Smith.

Broxterman’s parents are suing over the fatal fire, naming in the lawsuit among others, Motorola.

The problem became evident to all county public safety agencies during the Sept. 14, 2008, windstorm. All agencies were trying to use the radios at once. While that is an extreme situation, it locked up radios and prevented people from communicating.

County communication center officials questioned Motorola about possible fixes, but there isn’t a solution right now, the company told the county, said Mike Bailey, of the Hamilton County Communication Center.

“This is not an extreme emergency at this point, but it is a very big concern,” Bailey said.

A Motorola spokesman in charge of public safety for North America did not return a call for comment.

Cincinnati Fire department spokesman Capt. Michael Washington said the current system is better than the old one, and the department must work with what it has.

As the county grapples with what to do, Jetter said firefighters, police officers and the public are in danger. “It gives me heartburn that we have this situation,” he said.

Fire departments nationwide have reported problems with the digital radios.

The city of Phoenix has the same system, but when fighting a blaze the fire department uses the old analog system that transmits calls radio to radio, instead of through a computer system.

The International Association of Fire Fighters is now recommending fire department not use digital systems in fires.

“Radios are the most important piece of safety equipment a firefighter has,” said Richard Duffy, assistant to the president of the international union. “If you can’t communicate on the fire ground, you put yourself and others at risk.”

Duffy said Motorola should be held accountable.

Jetter said the problem is so bad it’s almost like having no communication at all. “We’re going back to the old days of using runners, where somebody runs in to deliver a message,” he said. “In an age of technology, this shouldn’t be an issue.”

Cincinnati Fire Union President Marc Monahan said last week’s missed mayday call “could have been really bad.” “We’ve had problems, we’ve pointed them out and nothing has been done,” he said. “Hopefully, this is enough evidence that will force some changes.”

The union plans to once again address the issue with fire administrators at the quarterly safety committee meeting Wednesday. Jetter said he’s not sure what the solution is. “We can complain all we want,” he said. “I don’t know how this gets resolved.”

Link to the original article.

Link to a related article with video.

A FICTIONAL TALE OF POLITICAL REALITY

(Ford, Chevy, and Dodge are used herein only to illustrate a point related the procurement of some public-safety radio systems.  All three are fine automakers and the references to these brand names are purely anecdotal.)

County X wanted new squad cars, and it really, really wanted Ford. They looked at Chevy and Dodge, but its employees really, really wanted Ford.

County X found out that Ford was working on a hydrogen-hybrid squad car, and the County thought that this was really cool. Chevy and Dodge have been working on hybrids using various technologies, and Chevy had an electric-gas hybrid but it wasn’t sexy enough for the County. Besides, “it’s a Chevy and we like Fords”. Dodge had not offered any hybrid solution yet in the marketplace. Besides, the DOT has not completed standards yet for hydrogen hybrids.

The State of Y offered a state contract for leasing Fords. County X found out they can side-step all bidding if they “bought” off of the State’s contracts. So County X takes State Y’s specs for Ford (that does not include hydrogen-hybrid as an option), and declares that they do not need open bidding since Fords were on the State’s contract.

No one challenged the “lease versus purchase” nature of the state contract, and no one noticed that hydrogen-hybrid technology was not offered in the State’s contract  County X had no intent to ever lease the cars and intended only to buy cars.

Some folks asked, “Have you seen the hydrogen hybrid before” and “can we test-drive it before we get them”?  “No” was the answer, “…but you know, Ford is a reputable car maker, we believe it will work, most cops drive Fords anyway, so we’re convinced everything will be OK”.

So, County X declares it will be impossible to openly bid this purchase, becomes convinced that while Chevy and Dodge were also working on hybrid squad cars, it could only buy Fords. So, unchallenged, it does not advertise bidding, it does not disclose that they were negotiating with Ford, and processes a Purchase Order  for “10 Ford hydrogen-hybrid squad cars”.

At public meetings, there was enough obfuscation and razzle-dazzle that the purchase order flew right through. Besides “…our cops deserve the best, right?”.

The Ford dealer shows up with a brand-new standard gas-powered Ford squad car with a hydrogen tank under his arm. The Ford guy says, “Do you mind if we consider your county as a BETA test site, since we’ve never done this hydrogen-thing before?”. A loud noise was heard – it was the eyebrows of the Chevy and Dodge dealer snapping to attention.

It turns out that Ford did not finalize the design for the hydrogen-hybrid, it was still designing the parts, and still working on the software that controls the hydrogen-hybrid car’s function. County X says, “sure we’d love to help you, you are a reputable vendor, and we love Fords”. The county then buys hydrogen storage tanks and dispensing equipment with yet another no-bid contract.

Time goes by and the BETA tests are not going well. The county determines that it needs 9 more squad cars and by way of a flawed change-order process, and orders 9 more cars under the same sole-source, no-bid contract. County rules say that if any contract has a change order of more than 50%, it needs to be tossed out and open bidding must take place. But, “our cops deserve the best, right?”.

Time goes on, and the BETA test continues to struggle. In one case, the hydrogen-hybrid car gets so hot, it starts a fire in the County garage.

The County decides to hire a professional engineer, a hydrogen squad car manager who will run the fleet once it hits the street.  It starts holding public meetings with the potential users of the hydrogen-hybrid squad cars. More problems are revealed in those meetings.

The Public Safety Chairman gets out a calculator and determines that they can’t afford to complete the contract for 19 squad cars, but also realizes that hydrogen is expensive, hard to manage (contain, dispense, and use) and declares that “we can’t afford this project”.

So, all pubic meetings stop. The Public Safety Chairman privately meets with the vendor to try to arrive at a “solution”.  Observers sense that there will be no hydrogen-hybrid squad cars forthcoming.

How will this story end? No one knows…yet. The plot to the final ending is being written behind closed doors.

Will the original contract be canceled? Will the change-order be withdrawn? Will the engineer and manager lose their jobs? Will anyone go to jail? “No one got fired for buying Ford” has been a common mantra in County X and State Y, but history might be made in County X.

Here are some links to documents that may be associated with this fictional story.

• Dupage panel ponders rocketing costs – 12/13/2009
• Dupage County IL – RF Committee Packet – 12/17/2009
• DuPage_County Letter – Page 1 of 2
• DuPage County Letter – Page 2 of 2
• Village of Woodridge Interoperability Statement
• Monthly report from John Lozar
• DuPage County procurement policy

How widespread are these problems? I would like to hear your story if you are a first responder and rely on a digital trunked radio system for essential communication. Click here to send me e-mail.

In some cases, public-safety executives issue gag orders that bar employees and contractors from discussing system shortcomings. Employees face disciplinary action for talking about the communications systems that they rely on for personal safety. Sometimes it’s best to call the baby ugly.

Your e-mail and periodic Internet searches have revealed numerous stories about digital trunked radio system problems and related topics. Here are links to a some of the stories.

• Pennsylvania STARNet reported to be unreliable – 04/19/2010
• Philadelphia – Another Digital Radio System Glitch – 04/13/2010
• Washington DC – Digital trunked radio system failure – 03/29/2010
• Milwaukee – Panel recommends borrowing for troubled police radio system – 03/19/2010
• Cincinnati fire radio problems must be resolved quickly – 01/21/2010
• Hamilton County Ohio – Radio tweaks not a solution – 01/20/2010
• York County – Police survey says digital radio is total failure – 01/08/2010
• York County PA Commissioners react to survey – 12/13/2009
• Cumberland County TN – P25 problems – 12/03/2009
• Milwaukee $17.5M digital radio system problems continue – 11/26/2009
• Maine firefighters seek return to analog radios -  11/18/2009
• Scandal in Chicago over $2.25M Motorola contract – 11/10/2009
• Milwaukee – Police suffering from communications breakdown – 10/28/2009
• West Palm Beach Florida digital trunked radio system fails – 10/09/2009
• Digital trunked radio problems in New Zealand – 09/24/2009
• Pennsylvania State radio system not working properly – 09/16/2009
• Indianapolis – Problems persist with fire radios – 09/18/2009
• PA Evening Sun – Radio issues reported nationwide – 09/06/2009
• 4 Fire departments go back to analog radios – 08/20/2009
• York County PA Fire Chiefs may dump digital radios – 08/17/2009
• Lancaster County PA gets stuck with obsolete digital radios – 06/13/2009
• Philadelphia – Digital radio system out for two hours (06/2009)
• Virginia – Auditor critical of work on statewide radio system (06/2009)
• York County PA – Tyco failures (05/24/2009)
• Cleveland bought $2 million radio equipment that can’t be used unless it buys rest of system (04/13/2009)
• State of New York – M/A-Com system worked as contracted (03/30/2009)
• Radio controversy continues in San Mateo County (02/19/2009)
• York County PA – Police union files suit over digital trunked radio system (02/02/2009)
• Milwaukee digital trunked system still offline (02/01/2009)
• Kansas City MO digital radio system needs to be replaced (01/29/2009)
• Pittsburgh PA – Digital trunked radio system overdue, overbudget and obsolete (01/18/2009)
• State of New York cancels $2 Billion digital trunked radio contract with M/A-Com. (01/15/2009)
• State of New York is close to canceling digital trunked radio contract. M/A-Com threatens to sue. (01/10/2009)
• York County PA prepares to turn off new digital trunked radio system (01/10/2009)
• York County’s (YCDES) new digital trunked radio system has problems (12/20/2008)
• Philadelphia gives $36M to Motorola to upgrade obsolete 6 year old digital radio system (12/11/2008)
• Philadelphia – July crash of radio system laid to faulty maintenance (12/11/2008)
• Philadelphia police radio system will cost $40 million to upgrade (12/11/2008)
• KTVU-TV2 report on digital police radio problems in the San Francisco Bay Area (11/20/2008)
• Milwaukee – TMJ4 reports that OpenSky radio system doesn’t work (11/14/2008) (Video)
• Milwaukee – TMJ4 reports that OpenSky radio system doesn’t work (11/14/2008) (PDF)
• WSLS-TV10 story on digital radio problems in Roanoke VA (10/30/2008)
• APB – Can we have our old radios back (09/2008)
• Leesburg Virginia (Loudoun County) Fire Dept Significant Injury Investigation Report (9/2008)
• New York State Wireless Network – Compliance report by Federal Engineering (08/2008)
• N.Y. network doesn’t make grade (Urgent Comm magazine) (Sept 2008)
• State of New York notice of default to M/A-Com on $2B digital trunked radio system (08/29/2008)
• Kanawha County WV – Digital radio outage (08/28/2008)
• West Virginia WSAZ – New digital radio system goes down (08/28/2008)
• Charleston WV – 5 hour outage latest problem with digital radios (08/27/2008)
• P25 Task Group to Investigate Fireground Noise Issues Mission Critical Comm (08/18/2008)
• Jefferson County W. Virginia – Police chiefs report problems with digital radios (07/28/08)
• New York State’s $2 billion emergency comm system in jeopardy (07/28/08)
• Link to TV news video of Philadelphia Police statement of no confidence in Motorola digital trunked radio system (07/23/08)
• Philadephia digital trunked radio system fails again (PDF) (07/23/08)
• West Virginia -State to look at digital radio problems (07/28/08)
• Charleston West Virginia police report problems with digital radios (07/23/2008)
• Link to TV news video about Indianapolis FOP threatening to sue over digital radio issues (07/15/08)
• Indianapolis FOP threatens to sue over digital radio problems (07/15/08)
• Charleston West Virginal concerns about digital trunked radio (07/10/2008)
• Link to TV news video about more radio and telephone problems with the Indianapolis comm center and digital trunked radio system – 07/09/2008
• Philly’s police radio system fails at crucial times (04/09/2008)
• Motorola P25 problems in Indiana (2008)
• Indianapolis firefighters, police union decry move to new radio system (2008)
• MECA – Fire chiefs question new digital radio system (2008)
• Problems with digital radios when background noise is present (2008)
• IAFC and NIST tests confirm problems with digital radios (2008)
• Marion County Indiana fire chiefs won’t use new digital trunked radio system (2008)
• Orlando Florida firefighters refuse to use digital trunked radio system (2008)
• Colerain Ohio – Radio problems during fatal fire (2008)
• Indianapolis firefighters say new digital radio system is flawed (2008)
• Philadelphia needs multi-million dollar radio system upgrades (2008)
• Philadelphia – Police radio system fails (2008)
• Philadelphia Fire – We buried guys because of faulty radios (2008)
• Lancaster County PA stops digital radio project (2008)
• KGO-TV follow-up story on San Mateo County digital trunked radio problems (2008) (video)
• San Mateo County Sheriff’s Deputy comments on digital trunked radio problems (2008) (PDF)
• KGO-TV news story on San Mateo County and Redwood City Police Dept digital trunked radio problems (2008) (video)
• ABC7 news report on Redwood City Police Department problems with San Mateo County digital trunked radio system (2008) (PDF)
• Erie County NY pulls out of statewide digital radio network (2008)
• Reliability of $2.1B statewide communications system in doubt (2008)
  
• Mayday! Mayday! Mayday! I’m trapped inside. (2008)
• Lansing MI problems with EDACS-Provoice (2008)
• New Miami Beach Police Radios May Be The Problem (2008)
  
• Boise Fire Says No to 700 MHz Digital Radios (2007)
  
• Anonymous police department walks away from digital trunked radio (2007)
• Canadian CREST system would cost millions to fix (2007)
• Miami Beach police forgo new radio system for old (2007)
• Battle Creek Michigan police grievance over trunked radio (2007)
• Philadelphia firefighters want faulty radio system fixed (2006)
• Sacramento Fire Chief says digital radios put lives at risk (2006)
• Aurora Colorado trunked radio problems (2005)
• San Mateo County’s digital radio network taken offline (2005)
• San Mateo County’s multi-million dollar failure (2005)
• Philadelphia digital trunked radio system failure (2004)
• Radio Silence – FDNY (2003)
• FDNY Union Chief calls for digital radio replacement (2002)
  

Click here to learn about the real difference between digital and analog.

The Lancaster County government spent more than $13.8 million on the OpenSky digital radio system, even though the system never progressed beyond the testing phase.

The State of New York canceled it’s $2 billion contract for OpenSky products earlier this year. Pennsylvania reports similar problems with its failed attempt to deploy an OpenSky digital trunked radio system.

The State of Pennsylvania still does not have its OpenSky system working as planned, 13 years after commencing work on the project.  The State Office of Public Safety reported in June that it has cost $368 million so far.

In December 2005, the Sacramento Municipal Utility District (SMUD)  contracted for one of the first OpenSky radio systems at an estimated cost of more than $6 million.  The system was intended to provide service over 900 square miles in the California counties of Sacramento and Placer, however it is not sufficiently reliable to handle routine radio traffic even though SMUD has already spent several million dollars.  SMUD has not accepted the OpenSky system and is still using its old analog trunked radio system.

==

Lancaster Online article by Helen Colwell Adams,  07/12/2009

==

Contra Costa County is one of two counties that comprise the controversial East Bay Regional Communications Systems Authority known as EBRCSA.  EBRCSA is leading the effort to construct and operate a regional radio system for first responders. While this is an admirable goal, EBRCSA has focused on an unaffordable product  from a single vendor.  Funding for EBRCSA comes largely from Bay Area SUASI which is heavily influenced by Motorola.

Contra Costa County’s radio bid specification could serve as a model for other public entities that desire to have an incredibly expensive, prematurely obsolete and marginally performing radio system for its first responders.

Here are some key points that you may wish to consider including in your P25 portable radio bid, should you desire to follow the example set by Contra Costa County.

• Make no reference to P25 product certification by an independent testing laboratory.  Managed in partnership with the National Institute of Standards and Technology’s Office of Law Enforcement Standards, the P25 Compliance Assessment Program is designed to provide first-responder agencies with testing information that ensures that P25 equipment operates and interoperates as specified by the standard.
• Include a specification requiring compatibility with an obsolete, proprietary analog trunked radio system known as SmartNet II, even though your city or county does not and never will have a SmartNet II system.  Justify this to potential bidders as a requirement because a neighboring county has a SmartNet II system, even though the neighboring county has announced plans to replace it as soon as possible. Your desired vendor makes SmartNet II equipment and this will help eliminate other bidders from being considered.
• Do not include a specification for P25 Phase II compliance. This will assure that all of the radios will be obsolete in 2017 and the vendor will be able to sell replacement radios. Planned obsolescence is good, right?
• Include superfluous terms and conditions in the bid specification that are not in any way related to mobile and portable radios.   References to asbestos removal, toilet facilities, lead-based paint and land excavation will help your RFQ look more credible.  Long documents may be subjected to less scrutiny. Include as much fluff as possible to get  a high page count.
• Don’t restrict your desired vendor from invalidating all of the terms and conditions in your RFQ by submitting a counter offer, or using a precedence of documents clause in its proposal.
• Specify the IMBE vocoder as a requirement, even though more robust P25 vocoders may be available. Allow your preferred bidder to bid the oldest and most problematic vocoder possible in order to maximize the profit potential for the vendor.
• Do not make reference to proprietary P25 extensions such as OmniLink in your RFQ.  Add these at the time of order, after the bid has been completed and can no longer be easily scrutinized by the public.
• Do not specify any requirement for ambient noise rejection or useful battery life per charge.  A product from a non-favored brand might score much higher than what you want to purchase.
• Specify the battery voltage, the required number of lines on the LCD display, the quantity and layout of the keys on the keypad, and the weight of the radio. This is meaningless to the performance of the radio and may help disqualify products from all but the preferred vendor.
• DO NOT include substantive requirements related to:

1. Extended warranties
2. Factory depot service requirements
3. Perpetual firmware upgrades without additional charge
4. Number of years of guaranteed product support and parts availability

• Include a clause requiring all “work and services” furnished under the contract to be guaranteed for 10 years, but exclude specific warranty requirements for hardware, software and equipment.  Create the appearance of a good specification without actually doing so.
• Use the following language in response to questions asked by potential bidders when confronted with questions regarding vague and ambiguous requirements in the RFQ: “Please take the time and read the entire specifications of this bid. You can answer all of your questions within the specifications of the bid.”

==

By: G.W. Schulz //  Originally published on April 8, 2009

Local officials in California failed to properly account for millions of dollars spent on homeland security efforts in the state, made dubious purchases that may not make communities safer, and could have overpaid millions by not seeking competitive bidding for equipment, according to an audit by the inspector general of the US Department of Homeland Security.

In one example cited, a California county bought a $96,600 generator to provide its public works department with emergency power during a catastrophe but didn’t factor in a $130,000 overhaul of its electrical system needed to accommodate the generator. So nearly two years after the purchase, the new equipment wasn’t ready for a disaster and might never be, county leaders admitted.

Another county entered into a multimillion-dollar contract for communications equipment without competitive bidding, which an independent assessment found could have cost taxpayers as much as 26 percent less if other companies had participated. A third county paid more than a half-million dollars to outfit a witness interview room, which was rejected as an illegitimate use of homeland security funds.

In addition to citing municipalities for irregularities, the audit was critical of the state homeland security department’s oversight of grant spending, saying it’s monitoring “was not sufficient to assure grant funds were spent properly.”

The report noted that only half of the local agencies in California that have received cash since 2001 were visited by state monitors as of December 2007 and the state’s weak controls did not ensure purchases made were “eligible, allowable, and supportable in accordance with federal guidelines.”

According to the audit: “Our reviews of several grant files disclosed that documents such as purchase orders, receipts, or delivery notices, were not present to support millions of dollars in grant expenditures. State officials explained that, in an effort to improve operational efficiency of grant management, subgrantees were not required to provide supporting documentation together with their reimbursement requests.”

In a letter responding to the audit, California Office of Homeland Security Director Matthew Bettenhausen argued that the state has a system in place for approving local expenditures and that grant managers in his office receive extensive training throughout the year. A spokesman for Bettenhausen said “We’re reviewing the report” but declined to address individual findings in the audit.

The March 13 report focused on $265 million given to the state from 2004-2006 through the State Homeland Security Program, only a fraction of the at least $1.7 billion awarded to California between 2003 and 2008 through the most common grant programs made available by the federal government. The state has received additional millions from Washington for projects that exclusively target port security, expanded interoperable radio systems, hiring new firefighters, border protection and more.

In overall dollars received, California is the leading beneficiary of homeland security grants compared to other states, according to Matt Mayer, a former top official in the federal Department of Homeland Security’s Office of Grants and Training.

The audit also found that despite receiving hundreds of millions of dollars in homeland security grants since 2001, California has not adequately evaluated improvements made in emergency response efforts, so it’s difficult to tell which areas of the state are better prepared to respond to a major catastrophe.

Bettenhausen, in his response letter, disputed that California had failed to measure changes in its ability to manage disasters. He wrote that the state has tracked improvements in its emergency response capabilities and can gauge today how it is better equipped for calamity than before California began receiving the grants. “California has gone above and beyond what is required at the federal level in the measurement of preparedness and capabilities,” Bettenhausen wrote.

The audit does not name any of the counties singled out in examples of questionable grant expenditures, and Bettenhausen’s office would not provide them, although it made assurances to the inspector general that the purchases were valid.

Two California counties CIR was able to independently identify from the report, Alameda and Contra Costa, committed $33.4 million to a new emergency communications system using grants funds, but they need at least $34 million more for additional equipment and are unsure where it will come from, according to auditors. So for now, the system is only partially operational. In addition, a contractor may have overcharged them, and auditors are questioning the entire investment.

For the most part, local California recipients of the homeland security grants Congress began handing out after Sept. 11 must purchase what they need first, such as equipment and terrorism response training, before being reimbursed with federal money controlled by Sacramento. The grants have allowed cities of all sizes across the country to buy new mobile command vehicles, gas masks, hazmat suits, decontamination shelters, surveillance equipment for identifying terrorists and more.

A number of states have struggled to comply with rules created to increase accountability over the grant spending such as seeing to it that cities and counties keep their pricey new gear in a “state of readiness” and maintain documents that validate the purchases they’ve made.

Bettenhausen agreed in his response letter that more monitoring visits needed to occur and vowed to ramp up their frequency, but he complained that the amount of grant funds available specifically for such administrative costs are limited and a backlog of older grant years still exists. However, he promised that more on-site inspections will help ensure “funds are being expended as intended.”

Suspect equipment purchases were made in cases identified by the auditors. One of the two aforementioned counties not named hired an engineering firm after determining that the $96,600 generator it had purchased with grant funds would be too difficult for its own employees to install. The consultants discovered that an electrical system would first need to be expensively renovated before the generator could be used. Because the county wouldn’t put up the money needed to cover the previously unrealized costs, the generator has sat inactive for almost two years. A public works director from the county didn’t believe the money would ever be made available, according to the report.

Another recipient spent nearly $600,000 on audio recorders and additional equipment to outfit a witness interview room for law enforcement purposes, but such items do not “enhance preparedness for terrorists’ attacks or natural disasters,” the report found. Auditors recommended that the state recover the costs of both projects.

Bettenhausen countered in his letter that the audio equipment “can be used to monitor and record suspects and witnesses involved in a terrorist event.” Bettenhausen also promised to make sure that the emergency generator was eventually installed.

Many local authorities in California, according to the report’s findings, also are not meeting federal procurement guidelines that require competitive bidding for major homeland security purchases, an obligation that’s designed to make certain taxpayers receive the most value possible. “As a result, the grants’ requirements for fair and open competition in procurement were not always practiced and subgrantees may have paid more than was necessary,” the report states.

Local grant administrators bought large-ticket items but were not familiar with federal procurement rules and did not have records showing what processes they did use to purchase the gear. The report cited examples totaling $11.6 million worth of questionable transactions, including a $294,000 bomb disposal robot, $81,000 worth of night-vision goggles and a $5.1 million communications system.

Multiple purchases were made without competition, or local officials failed to prove that there weren’t enough suppliers of specific goods to compete with one another. Nor did grantees conduct cost analyses to determine whether prices were “fair and reasonable” when only one vendor was used without competition.

The $5.1 million communications system was supposed to be part of a much larger regional network in the San Francisco Bay Area of northern California, designed to serve multiple dispatch centers and emergency radios so that first responders could more readily talk to one another. Two Bay Area counties, Alameda and Contra Costa, which encircle several California cities including Oakland and Berkeley, manage the interoperability project together, one of many similar but expensive efforts that exist today around the country. Because of the system’s original design, according to the report, most of the equipment for an early phase of the undertaking was purchased from the same company that conceived it.

The contractor, which is not named in the report but CIR identified as Motorola by searching documents online, estimated that the larger system ultimately would carry a price tag of about $54.8 million, including the cost of transmission towers, repeaters and other equipment. A later independent assessment, however, found that Motorola had added a substantial premium to the price of as much as 26 percent compared to what the cost may have been if more than one company competed to offer a better deal.

A spokesman for Motorola declined to comment on the audit’s findings. The East Bay Regional Communications System Authority has spent approximately $10 million so far against the contract with Motorola. Bill McCammon, executive director of the authority, told CIR that it has since learned of other less costly alternatives, such as piggy backing on a contract that was competitively bid out to Motorola by Riverside County in southern California. The authority will pursue those options rather than continuing with the existing contract, but McCammon admits some service fees charged by Motorola so far were overpriced.

He said that when Alameda County sought a contractor initially for a new radio system, Motorola was the only bidder. Officials chose to proceed with the company anyway because the Department of Homeland Security requires grants to be obligated quickly for projects each year after the federal awards are announced.

“It was a question of using those dollars or losing those dollars entirely,” McCammon said. “Those funds need to be expended within a certain amount of time.”

Regardless of the system’s estimated cost now, it doesn’t include the additional millions it will take to provide the police, sheriffs, firefighters, paramedics, public works personnel and others who would benefit from interoperability with individual radios, thousands of which may be needed. McCammon isn’t sure how much more that could cost.

As of May 2008, the two counties were looking for a way to meet a funding gap in the project of $34 million, but according to the audit, they don’t know how to fill it. McCammon responds that “we’re going through a very deliberate process to evaluate our options” with the aid of a communications consultant. Nonetheless, if the system does not eventually work, auditors say that all of the money used from homeland security grants for the project – more than $33 million earmarked so far – should be relinquished.

Bettenhausen responded in his letter that the state would not be requiring an “artificial date” for completion of the communications project but would review its implementation plan to see if the system is technically feasible and compatible with other interoperable radio initiatives in the state.

The report, meanwhile, did single out another project in California for commendation. The California National Guard created a unique truck-bound system that can bridge disconnected communications systems in a disaster area. But it would still take a few hours for the system to be deployed during an emergency with the help of a military airlift. An early prototype was used during Hurricane Katrina and also California’s 2007 wildfire season. Six of the units are being manufactured and will be set up in regions across the state. Other states are examining the system’s state-of-the-art design.