Data Issues and Retraction in Our Farm Program
Summary
We recently detected an instance of data fabrication in the smaller of our two farm program’s operating regions (Nellore). We suspect, but do not have proof, that there were more such instances.
We have thus decided to retract all data and impact from that region over the last two years, and have reflected this in our data release and impact numbers.
We take this as a lesson and a nudge for us to improve our data verification systems. We are still considering whether to continue or cease operations in that region.
The Issue
Our Alliance for Responsible Aquaculture (ARA) program works with fish farmers in Andhra Pradesh, India to improve welfare through regular water quality monitoring. Staff visit enrolled ponds, measure parameters like dissolved oxygen, and recommend corrective actions when readings fall outside acceptable ranges. Nellore is our smaller site—46 farms, compared to 138 in Eluru—and from 2022 onwards all data collection there was done by one person, working most of the time without a manager on site.
In September, following concerns raised by other staff, we investigated and established that at least one day’s water quality data had been fabricated—entered into our system for a measurement that had not been taken. Additionally, testimony from other staff who had worked at the office over 2026 suggested that there had been further such instances, though we consider this evidence suggestive rather than conclusive. Unfortunately, we have been unable to verify how many of our water quality readings were affected, or how far back this problem goes. The individual staffer concerned here will shortly no longer be connected with FWI.
The rest of this post discusses the implications of this discovery.
Data and Impact Implications
We only have proof that one data point (in September of this year) was falsified. However, given that we suspect there were others, we have decided to retract all data in the affected region since November 1, 2024.
Our reasoning for this cutoff date is that this is when our measurement method in Nellore changed. Until November 2024, we measured dissolved oxygen with a handheld meter, read at the pond itself. From November 2024, we switched to the Winkler method, which requires fixing a sample at the pond and then titrating it back at our office. The failure we identified is specifically a failure to perform that laboratory step—and before the switch, there was no laboratory step to skip.
We should note that we are uncertain about this as the cutoff point: Team opinion is split about the scope of possible fabrication, and there were signs that good work was being done in Nellore (namely, very positive farmer relationships and phone timeline data). Other options we considered were retracting all data from the region when the relevant staffer was employed (going back to 2022) or just retracting data from this year. Ultimately, I (Haven) made this decision in part because I view it as a compromise given the lack of definite evidence.
The implications of this for our data and impact are the following:
Data: We have updated our latest data release to only include the affected data in a separate file, with the relevant disclaimer. This removed about 1,100 water quality records across 45 ponds—around a quarter of our Nellore data, and about 8% of all ARA measurements we have ever published.
Impact: As our farm program helps fishes via collecting and responding to this water quality data, we have removed any count of impact that occurred after the cutoff date from our Impact Page and 2025 and 2026 OKRs. This lowered our lifetime impact estimate from about 5.7M to 5.3M.
Programmatic Implications
Moving forward, we are planning stronger data verification protocols to prevent these kinds of issues from occurring. While we did have some such protocols (e.g., audits, phone timeline review), we now believe that they were inadequate (e.g., the audits were announced in advance). Our incoming Programs Lead, Sakthi, will be tasked with improving our protocols here.
Regarding our future work in Nellore, we are still considering whether to continue or cease operations. By default, we might think to hire a new staffer (or more) and continue our work, but given the large distance from Nellore to the rest of our operations (a 6-hour drive), it is not clear that continuing is the right option. We will update our blog later with our decision.
