WASHINGTON - The Inspector General of the U.S. Department of Transportation has issued a highly critical report on SafeStat, the federal government's system for ranking the safety of motorcoach operators and truckers.
The Inspector General said the problems with SafeStat are so serious and extensive that it may not be smart to continue to make it public.
The Federal Motor Carrier Safety Administration, which manages and operates SafeStat, "will need to demonstrate timely improvement if it is to continue to publicly disclose carrier results across all SafeStat categories," the report said.
The report's findings were anticipated by many in the motorcoach and trucking industries. Operators, safety officials and advocates, insurers and others have pointed to shortcomings in the system for years. Now, their criticisms have been confirmed by the comprehensive analysis of the DOT Inspector General.
For its part, the FMCSA readily acknowledges the system has problems but contends that some of the criticism is overblown, and it points to improvements that have been or are being made to address the problems.
Still, it is clear from the Inspector General's report that the FMCSA must adopt and met a higher standard of quality to assure fairness to motorcoach companies and trucker who may lose business or be placed at a competitive disadvantage by inaccurate SafeStat results.
For the uninitiated, SafeStat is essentially a massive database that helps the FMCSA decide which of the more than 640,000 active interstate motorcoach and trucking companies it will target for its compliance reviews and safety oversight. Because it is impossible to inspect 640,000 companies each year, the FMCSA must have a system for determining who gets the attention. That system is the Motor Carrier Safety Status Measurement System, or SafeStat.
In reading the Inspector General's report on the system, a three-decades-old cliché keeps coming to mind: Garbage in, Garbage Out. Clearly, the No. 1 problem identified by the report is that much of the data foundation of SafeStat is little more than garbage.
The major shortcomings identified in the report are little short of astounding:
1. There is no updated information for 42 percent of the 643,909 active motor carriers in the SafeStat database.
2. SafeStat reported that 11 percent of the active carriers in its data base had no vehicles and 15 percent had no drivers. Yet, many of these same carriers had crashes or inspections reported against their vehicles or drivers. Were that not bad enough, neither the crashes nor the inspections negatively impacted these carriers SafeStat score.
3. There were no reports for the estimated one-third of the large trucks involved in accidents annually, including 37,000 crashes involving interstate carriers.
4. Reporting for about 20 percent of the crashes in SafeStat occurred six months or more after the crashes.
5. Serious moving traffic violations (mainly speeding) are being seriously underreported. In one state, the audit found that roughly 29,000 serious moving traffic violations went uncounted over a three-year period.
6. An estimated 13 percent of the 21,000 crashes, and 7 percent of the more than 1 million inspection transactions occurring in a six-month period, contained carrier identification errors, including failure to the identify a carrier associated with the violation or, in a smaller number of instances, identifying the wrong carrier.
7. Problems with inaccurate data are compounded because there is no effective system is in place to correct errors in data reporting. In fact, when the FMCSA receives a request from a carrier to correct data it has received from a state, there is no system for centrally tracking the resolution of such requests.
The overarching finding and problems identified in the audit were these:
* Data Quality Weaknesses. Significant problems exist with the data motor carriers and the states provide to the FMCSA, and these data problems limit SafeStat's effectiveness and introduce bias into the ranking process. The most serious problems lie with outdated data on the number of vehicles and drivers used by motor carriers, incomplete crash reporting from the states, and late or incomplete inspection reporting, particularly reports on serious moving traffic violations.
* Poor Crash Data. Six states (including 2 of 10 states visited by Inspector General auditors) did not report any crashes in the six-month period reviewed by the IG. The six states were the District of Columbia, Florida, New Hampshire, New Mexico, Pennsylvania and Vermont. One of the 10 states reviewed reported only fatal crashes to FMCSA, but failed to tell the agency about 5,816 non-fatal large truck crashes that occurred in the state.
* Poor Data on Moving Traffic Violations. State reports on roadside inspections are supposed to include data on serious moving traffic violations but often do not. In addition, large state-to-state variations exist in the reporting of traffic violations. For example, California reported 115 serious moving violations to the FMCSA database in fiscal 2001, compared to Indiana, which reported about 25,000.
* Inaccurate Data Records. Even when crash and inspection data reach the FMCSA database, errors or omissions occur during the processing of the data that could influence SafeStat scores. The IG estimates that errors occurred in approximately 13 percent of the crashes and 7 percent during a six-month sample period. A critical mistake that occurred in an estimated 11 percent of the errors was that the wrong carrier was held accountable for a SafeStat-related violation.
* Insufficient Disclosure of Weaknesses in Publicly Disseminated Data. To
mitigate problems posed by public dissemination of inaccurate data, the FMCSA should ensure public users of SafeStat are aware of any known data quality weaknesses. In the past, the SafeStat Web site pointed to problems with data provided by carriers, but did not disclose reporting problems with states. Following the issuance of the IG's draft report in December, the FMCSA placed a more comprehensive disclaimer on its Web site, and promised to provide state specific information in the future.
* Ineffective Systems for Facilitating Data Correction by the States. Motor carriers cannot easily correct inaccurate crash and inspection information in SafeStat because carriers must deal with individual states to obtain corrective actions. In addition, the FMCSA maintains no data on the adjudication or timeliness of correction challenges and does not enforce existing requirements. FMCSA standards call for the correction of data inaccuracies within 7 days, but the Inspector General could find no evidence of this being enforced.
Since the IG audit was initiated, the FMCSA and the Volpe National Transportation Systems Center have developed a prototype for a data quality tracking system to centrally accept and track challenges motor carriers make to the states. The system currently is being put into operation.
* Insufficient Reviews of Data Quality. The audit found that states have limited controls to ensure the quality of data submitted to the FMCSA. Compounding the lack of state controls, the FMCSA does not routinely conduct its own oversight reviews targeted fixing the system.
* Ineffective Use of Sanctions and Incentives. Overall, the FMCSA has not effectively used existing sanctions and incentives to promote better data reporting by states and motor carriers. The FMCSA also has not imposed sanctions, such as the withholding of Motor Carrier Safety Assistance Program grant funds from states for failing to correct data quality problems.
* Slow Implementation of Joint Programs. Although the Motor Carrier Safety Improvement Act of 1999 directed the FMCSA and the National Highway Traffic Safety Administration to take action to improve the collection and analysis of state crash reporting, results have been slow to materialize.
However, the agencies did not complete a memorandum of understanding on the basic approach until February of last year. Since then, the agencies have initiated a plan to bring about immediate improvement in the completeness of crash reporting for targeted states. The IG says rapid implementation of the planned actions is needed and the plans should be extended to cover all states.
The Inspector General noted that many of SafeStat's shortcomings are caused by the states and police agencies that generate and process much of the safety data going into the system, and that this presents challenges for the FMCSA.
At the same time, however, the IG contends the FMCSA could do a much better job of hammering the states to do better.
One consequence of the lousy reporting is that SafeStat rankings are geographically biased against carriers operating in states that provide more complete data. In other words, such data variations across geographic locations mean the FMCSA may fail to target certain high-risk carriers simply because of where they operate.
So, carriers operating in states that do a terrible job of reporting traffic and safety information to the FMCSA stand a better chance of getting away with more safety violations.
At the same time, inaccurate or misleading data provided by carriers can produce errors in SafeStat calculations and cause some high-risk carriers to be missed.
"Correcting data quality problems are critical to ensure SafeStat more effectively targets high-risk carriers for compliance reviews," says the report. "However, quick action is even more important given the continued public dissemination of SafeStat results."
The FMCSA was given a chance to respond to a draft of the Inspector General's report and the agency said it agreed with the concerns about improving data quality and it cited a number of improvements being implemented to address recommendations in the report. The improvements include:
* Hiring a contractor to conduct a new study to revalidate the SafeStat model.
* Implementing an improved system for tracking public challenges to the accuracy of SafeStat data.
* Providing SafeStat users with comprehensive information on data limitations.
* Assigning staff to review monthly state reports that address state data quality issues and to work with the states to resolve them.
* Establishing goals for completeness, accuracy, and timeliness of data.
* Making state grant funding contingent on participation in certain data quality programs.
The IG report said the current data quality problems should not prevent the FMCSA from using SafeStat as an internal decision-making tool. "However, while the data used for SafeStat calculations are sufficient for internal purposes, if public dissemination of SafeStat results is to continue, the data must meet higher standards for completeness, accuracy, and timeliness," the report said.
To check out the SafeStat Web site, go to www.safersys.org.