Biz Views

Op-Ed: Is FEMA using flawed indicator in decision-making process in P.R.?

For decades, government officers and academia have indicated that the problem for doing research in Puerto Rico has been the lack of access to data. Paradoxically, regarding Puerto Rico, today we find ourselves with statistical independence threatened by political forces and with questionable data analysis produced by some organizations. Is inaccurate data analysis common in Puerto Rico?

The undercount of excess deaths following Hurricane María is one example of how inaccurate data analysis has been present under the current regime. Most recently, a group of political spinners shared inaccurate data analysis through social media to say that crime was “under control” in Puerto Rico.

I went on the record via the New York Times regarding how the steady decline in crime had ended in 2015 when the Murder Rate started increasing again, to this day the data has not returned to pre-2015 levels.

My claim has always been that we need appropriate data analysis to aid in the decision-making processes at the government level. Even when the data is available, stakeholders must remember that data is just one part of the process and not the decision itself. The decision must be the result of a comprehensive analysis of the issue. The perils of technocracy can have devastating consequences.

During a discussion regarding research and data analysis in and for Puerto Rico, Arnaldo Cruz Sanabria indicated that the Federal Emergency Management Agency (FEMA) is using the Municipal Financial Health Index produced by Open Puerto Rico. Cruz-Sanabria is co-founder of Open Puerto Rico, research director for Foundation for Puerto Rico and chairman of the Board of Directors of the Puerto Rico Institute of Statistics.

This statement made me wonder about the reliability of this index; after all, I have devoted a good part of my academic career to highlight the shortcoming of a widely used index of health through the lens of language and using physiological activities as validation criteria.

Is FEMA using a reliable indicator in their decision-making process? I accessed the Municipal Financial Health Index produced by Open Puerto Rico for 2015 and 2016. Any index that is guiding decision-making processes in Puerto Rico should be reliable and consistent. It is not clear from the reports what the range of this metric is. It remains unclear how high or low a municipality can go.

For example, in 2015 the highest index was 11.25857 while the highest score for 2016 was 9.1839, both being indexes for Fajardo. Is this a meaningful difference? Does it mean that Fajardo was doing worse in 2016 than in 2015? Was the first value far from the upper limit of the index?

The same happens for the minimum, for example in 2015; the lowest index was -7.44193 for Loíza while the lowest index for 2016 was -7.09409 for Salinas. Does this mean Salinas, in 2016, was doing better than Loíza in 2015? Again, not having a range for this index makes it difficult to compare the scores across years.

In the 2015 report, Open Puerto Rico made the decision of assigning grades to the indexes based on feedback received from focus groups. The issue of comparability emerges when one compares the grade assigned to municipalities based on their index from one year to another. For example, in 2015 Luquillo had an index of 1.13277, which was equivalent to a C. However, in 2016, Bayamón had a 1.10277 and the grade assigned was a B.

In simple terms a score of 1.10 would have resulted in a C in 2015, but not in 2016. The same happens with values below zero. For example, in 2015, Guánica had an index of -3.19237 that resulted in a D. In 2016, Ponce had an index of -3.12272 and this resulted in an F. It is clear that the determination of grades varies by year, and as such, the grades may not be comparable either. More troubling is the fact that in some cases a better score resulted in a lower grade in one year in comparison to another.

An approach to the cutoff points for each grade provides insights about the ranges of each score by year. The data presented in the table below indicates that the ranges for A, B and C were wider for 2015 than for 2016. Conversely, the ranges for D and F covered a wider range for 2016 than for 2015. This brings into question how comparable the index is over time on matters of scoring and grading.

Another concern regarding the Municipal Financial Health Index is the cutoff points within the same year. In 2015, Culebra had an index of 2.82928 resulted in an A. In the same year, an index of 2.82662 resulted in a B for Quebradillas.

This means that an arithmetic difference of 0.00266 resulted in a worse grade for Quebradillas than for Culebra. Is this a meaningful difference? The documentation available does not allow anybody to answer this question.

FEMA and any other agency or organization using this index in their decision-making process should be aware of the limitations it has and lack of comparability from year to year. Puerto Rico needs conscious data analysis to aid in the decision-making process, faulty data analysis can have similar or worse consequences than no analysis at all. 

Comment here