New Study: Outcome Evaluation Of Our Water Quality Improvements

This post discusses our upcoming outcome evaluation to assess the effectiveness of our corrective actions in improving water quality. This evaluation is part of our effort to strengthen the evidence base for the Alliance for Responsible Aquaculture and evaluates an important part of its theory of change.

Why This Study

The Alliance for Responsible Aquaculture (ARA) is FWI’s core program, designed to help farmed fishes by improving water quality and maintaining appropriate stocking densities. While the ARA remains our most promising initiative, there are still key uncertainties in its theory of change. 

We recently wrote about the monitoring plan for the ARA. Now, in line with our value of being evidence-based, we are planning the ARA’s first field study. The objective of this study is to evaluate whether our program (and, in particular, its corrective actions) is/are improving water quality parameters as expected. 

This will be the most in-depth and rigorous evaluation of our program to date, and we’ll commence it on August 1, 2025. We call it an “outcome evaluation” because it evaluates one of the key outcomes of the ARA’s theory of change (see below, circled in red).

The ARA’s theory of change. The red circle highlights the primary outcome we expect from our intervention. We will investigate the assumptions of this outcome with our outcome evaluation.

Study Plans and Details

We are planning to enroll at least 32 farms in Eluru district, Andhra Pradesh, India. Half will be part of the treatment group, receiving standard ARA services: water quality monitoring, corrective actions when parameters fall out of range, and follow-up measurements to verify improvements. The other half will be a control group, receiving monitoring and follow-up visits, but without telling them the results of these water quality readings and without giving corrective actions.

The study design involves at least n = 32 farms.

This design allows us to test whether corrective actions are genuinely responsible for observed water quality improvements. If water quality at control farms improves just as frequently as on treatment farms (which receive their readings and corrective actions), this may indicate that we are not the reason for these water quality improvements.

Two common corrective actions to improve water quality are aeration (blue-yellow paddle wheel device) and water inflow (white pipe), as seen on this ARA farm.

Study Limitations

Just like much of our work, this study will stretch our team’s capabilities—running a clean, scientific study in a field setting in rural India is difficult, to say the least. These are some of the challenges we anticipate and plan for:

• Confounding factors and noise: Past field studies have proven to be extremely difficult because fish farms are dynamic environments, and it is difficult to keep farmers’ actions and external factors like weather stable. While we try to account for all these, there is a chance that the data collected through this study is too noisy to draw firm conclusions. We will thus seriously investigate the reliability of the data before drawing conclusions about our hypothesis. 

  
  

• Low incidence of out-of-range parameters: Only about 10% of our measurements currently fall outside acceptable ranges, with most deviations occurring in dissolved oxygen (DO) rather than pH or ammonia. This limits the parameters for which we can expect statistically meaningful results. If we cannot collect sufficient data on out-of-range measurements, we may have to continue the study beyond the four-month timeframe. This may complicate the point below on getting farmer buy-in.

  
  

• Farmer participation: It can be hard to convince farmers to participate, especially when the control group doesn’t receive water quality readings and improvement recommendations. While our relationships with farmers are strong, onboarding remains a potential hurdle.

  
  

• Limited geographic diversity: All farms are located in Eluru district, which may limit the generalizability of the results. While we expect some findings to be broadly applicable, farm conditions in other regions could differ.

ARA Senior Program Manager, Chaitanya, in conversation with an ARA farmer.

Looking Ahead

Despite the uncertainties, we believe this evaluation is a necessary next step towards the ARA achieving our minimum scaling thresholds. The study will either strengthen our confidence in the program or highlight areas where we need to adapt. 

To minimize additional resource needs, we are designing it to integrate smoothly into our ongoing operations. We feel confident in our team’s ability to complete such a comprehensive project and will share updates as the study progresses.

If you have thoughts or suggestions, please feel free to contact us.