Op-Ed: Is FEMA using flawed indicator in decision-making process in P.R.?
For decades, government officers and academia have indicated that the problem for doing research in Puerto Rico has been the lack of access to data. Paradoxically, regarding Puerto Rico, today we find ourselves with statistical independence threatened by political forces and with questionable data analysis produced by some organizations. Is inaccurate data analysis common in Puerto Rico?
The undercount of excess
deaths following Hurricane María is one example of how inaccurate data
analysis has been present under the current regime. Most recently, a group of
political spinners shared inaccurate data analysis through social media to say
that crime was “under control” in Puerto Rico.
I went on the record via the New
York Times regarding how the steady decline in crime had ended in 2015 when
the Murder Rate started increasing again, to this day the data has not returned
to pre-2015 levels.
My claim has always been that we need appropriate data
analysis to aid in the decision-making processes at the government level. Even
when the data is available, stakeholders must remember that data is just one
part of the process and not the decision itself. The decision must be the
result of a comprehensive analysis of the issue. The perils of technocracy can
have devastating consequences.
During a discussion regarding research and data analysis in
and for Puerto Rico, Arnaldo Cruz Sanabria indicated
that the Federal Emergency Management Agency
(FEMA) is using the Municipal Financial Health Index produced by Open
Puerto Rico. Cruz-Sanabria is co-founder of Open Puerto Rico, research director
for Foundation for Puerto Rico and chairman of the Board of Directors of the
Puerto Rico Institute of Statistics.
This statement made me wonder about the reliability of
this index; after all, I have devoted a good part of my academic career to
highlight the shortcoming of a widely used index of health through the lens of language
and using physiological
activities as validation criteria.
Is FEMA using a reliable indicator in their
decision-making process? I accessed the Municipal
Financial Health Index produced by Open Puerto Rico for 2015
Any index that is guiding decision-making processes in Puerto Rico should be
reliable and consistent. It is not clear from the reports what the range of
this metric is. It remains unclear how high or low a municipality can go.
For example, in 2015 the highest index was 11.25857
while the highest score for 2016 was 9.1839, both being indexes for Fajardo. Is
this a meaningful difference? Does it mean that Fajardo was doing worse in 2016
than in 2015? Was the first value far from the upper limit of the index?
The same happens for the minimum, for example in 2015;
the lowest index was -7.44193 for Loíza while the lowest index for 2016 was
-7.09409 for Salinas. Does this mean Salinas, in 2016, was doing better than
Loíza in 2015? Again, not having a range for this index makes it difficult to
compare the scores across years.
In the 2015 report, Open Puerto Rico made the decision
of assigning grades to the indexes based on feedback received from focus
groups. The issue of comparability emerges when one compares the grade assigned
to municipalities based on their index from one year to another. For example,
in 2015 Luquillo had an index of 1.13277, which was equivalent to a C. However,
in 2016, Bayamón had a 1.10277 and the grade assigned was a B.
In simple terms a score of 1.10 would have resulted in
a C in 2015, but not in 2016. The same happens with values below zero. For
example, in 2015, Guánica had an index of -3.19237 that resulted in a D. In
2016, Ponce had an index of -3.12272 and this resulted in an F. It is clear
that the determination of grades varies by year, and as such, the grades may
not be comparable either. More troubling is the fact that in some cases a
better score resulted in a lower grade in one year in comparison to another.
An approach to the cutoff points for each grade
provides insights about the ranges of each score by year. The data presented in
the table below indicates that the ranges for A, B and C were wider for 2015
than for 2016. Conversely, the ranges for D and F covered a wider range for
2016 than for 2015. This brings into question how comparable the index is over
time on matters of scoring and grading.
Another concern regarding the Municipal Financial
Health Index is the cutoff points within the same year. In 2015, Culebra had an
index of 2.82928 resulted in an A. In the same year, an index of 2.82662 resulted
in a B for Quebradillas.
This means that an arithmetic difference of 0.00266
resulted in a worse grade for Quebradillas than for Culebra. Is this a
meaningful difference? The documentation available does not allow anybody to
answer this question.
FEMA and any other agency or organization using this
index in their decision-making process should be aware of the limitations it
has and lack of comparability from year to year. Puerto Rico needs conscious
data analysis to aid in the decision-making process, faulty data analysis can
have similar or worse consequences than no analysis at all.