Outcome Evaluation Initial Results: Why We Are Repeating the Study

In late 2025, we completed the first phase of an outcome evaluation study to understand whether our farm program—the Alliance for Responsible Aquaculture (ARA)—is achieving its intended effect: improving water quality by giving farmers measurements and corrective actions. Unfortunately, due to both logistical challenges and a protocol error, the results of the study were inconclusive. We are now repeating the study in early 2026 and expect to publish the results by the end of June.

This post provides an update on the results of our first outcome evaluation, what went wrong, and what we’re changing going forward.

What We Did

Between September and December 2025, we conducted a randomized controlled trial with 50 ARA farms, divided into a treatment and a control group. The goal was to detect whether our water quality interventions led to measurable improvements on treatment farms compared to controls.

Improved water quality is the primary intended outcome of the ARA. Water quality fluctuates on these farms, and we visit farmers monthly to assess whether they need to improve water quality.

The process of helping fishes as part of the ARA program. The outcome evaluation aims to determine whether our primary outcome of improving water quality works as intended.

The outcome evaluation looks at Step 3: Improvement of Critical Parameters. We aimed to reach at least 16 poor water quality events per group to achieve statistical significance. However, while control farms reached this target, we were only able to obtain six usable events from treatment farms (see the table below). Most treatment farm follow-ups became impossible due to prolonged monsoon-related road flooding—a logistical challenge we couldn't overcome.

Table 1: Breakdown of control and treatment farms’ poor water quality events and follow-ups. Most treatment farm events could not be followed-up due to monsoon-related road flooding:

	Control	Treatment
Total measurements taken	400	315
Poor water quality events	29	17
Follow-ups completed	28	6

One of the farm roads after heavy monsoon rains. These dirt roads become muddy and impassable. The rains continued for longer than expected, affecting our ability to collect follow-up measurements for our outcome evaluation study.

What We Found, and Why We’re Cautious

While we conducted a light analysis of the available data, the results were not statistically strong enough to draw conclusions. Nonetheless, some initial observations are worth sharing:

• On control farms, water quality improved (within two days) in 25% of poor-quality events. This is lower than our anticipated 37% self-resolve rate.

• On treatment farms, water quality improved (within two days) in 3 of 6 events (50%).

A 50% success rate in the treatment group is concerning. Based on our internal benchmarks, we would expect a success rate of around 80-90% if the ARA interventions worked as intended. Given the limited sample size, it's difficult to say whether this result reflects natural variability or a more fundamental issue with our program. To gain more certainty, we need to collect more robust data.

Table 2: Water quality improvement rates for control and treatment farms, respectively.

	Control	Treatment
Water quality improved	7 (25%)	3 (50%)
Water quality did not improve	27 (75%)	3 (50%)

One particularly interesting finding: many control farmers took action independently. These included measures like aeration and stopping to feed—interventions we also recommend to treatment farmers. We see multiple reasons why farmers may do this:

• Routine water treatment practices (e.g., water sanitizers, aeration) 

• Reaction to external changes (e.g., weather)

• Reaction to water quality measurements received from a lab or technician (not FWI)

An aerator on an ARA farm: This is one corrective action we suggest to farmers with poor water quality. Our study confirmed that farmers in the control group used their aerators regularly, thus improving water quality without receiving measurements or corrective actions.

A Mistake in Our Protocol

In reviewing our study process, we identified another issue: we had deviated from the standard ARA protocol.

Specifically, while the ARA normally conducts follow-ups three days after detecting a poor-quality event, the study demanded a two-day follow-up. We had hoped this would reduce weather-related variability, but in retrospect, this change compromised the study’s relevance. Because the evaluation should replicate existing ARA protocols, we are now aligning the study with the program's follow-up schedule.

Outcome Evaluation Part 2

We’re rerunning the outcome evaluation with the corrected protocol and better timing. With the monsoon season now over, we expect to be able to reach all farms throughout the study.

The new study begins this week and will run through May 25. We aim to complete the analysis in June and publish our findings shortly thereafter.

Repeating this study is not the outcome we had hoped for. But we believe it is the right step to ensure that our program is impactful and evidence-based, reliably improving water quality on fish farms.