This blog post outlines what we believe is one of the biggest mistakes we’ve made as an organization: launching a pilot study that proved not valuable enough to complete. We publish this for the sake of transparency and in the hope that other organizations will learn from our mistakes.
Main Lessons Learned
Leaders of ground projects need to be extremely focused on digging into the details of the project and making sure that everyone is operating with the required knowledge and a shared vision.
For large projects, make sure the project leads give appropriate time to project evaluation and strategy, rather than getting caught up in low-level tasks.
Put in place regular re-evaluation points, especially for dynamic projects
Even in the face of closing deadlines, it is worth investing time into ensuring strong team communication, especially where there are existing barriers (such as asymmetrical expertise).
For projects aimed at informing stakeholders, make sure that the project is necessary for those stakeholders to want to collaborate with your organization.
In India, building trust with farmers is a necessary part of collecting accurate data and making them engage in welfare improvements for fish.
Our Past Approach: Pilot Study
Between January and May 2021, Fish Welfare Initiative’s main project was a study into improving fish welfare in Andhra Pradesh, India. The study was a scientifically rigorous test into the effect that training farmers in higher welfare practices had on the welfare of their fish, their profits, and the environmental sustainability of their farm. We planned to work with 9 farms to take in-depth measurements across a production cycle. Our aims were to:
Build relationships & showcase the value of fish welfare to our stakeholders:
Farmers
Governments
Markets
Gain specific in-depth knowledge on the effects and cost-effectiveness of some of the components of our expansion (e.g. aerators)
Gain scientific data on how normal farm operations affect fish
In May 2021 we decided to terminate the project early, instead shifting our focus to implementing the Alliance for Responsible Aquaculture (as well as conducting a process evaluation of the Alliance). We decided to cancel the project primarily because:
The impracticalities of running a very rigorous scientific test on the ground were jeopardizing our ability to capture the values we were aiming for
We updated our beliefs on how valuable the aims of the pilot study were and found that the study was not needed for helping fish
Below, we describe some of the main mistakes that we made during the project and that we expect others could make in similar projects.
Main Mistakes
Trying to uphold unrealistically high standards
Fish farms in Andhra Pradesh are complex. Each region has its own set of challenges and each community has their own way of dealing with those challenges. Despite this, early on we decided to push for the study to be as scientific as possible. We understood that there would be many areas where we would need to concede standardization. However, we felt that striving for as much rigour as possible would lead to the strongest results. What we didn’t properly anticipate was how this would affect our workload. Trying to conform to a high scientific standard drastically increased the logistical effort required for an already operationally challenging project.
Our high standards meant that our plans were consistently being proven impractical and requiring being rewritten. For example, we found that one group of farmers did not purchase fry (young fish of a standard size) because they had issues with predatory birds who would eat the smaller fish. As such, they stocked their ponds with yearlings (older fish not of a standard size). This change meant that all the growth rate indicators we planned to measure were going to be considerably less reliable. Unwilling to concede this, we tried to find ways around the problem, such as weighing the fish before being stocked. These changes presented their own logistical challenges, and so on.
Too little time spent critically evaluating the project
Running an intricate project in the field necessarily comes with a heavy workload, and it is inevitable that at least some elements of the plan will require adjusting over time. However, our emphasis on scientific rigour and standardized conditions, on top of our inexperience with field projects, exacerbated these issues, such that the project team spent a lot of time ‘fire fighting’ to try and retain our ambitions for the project. We were also ambitious in the number of supplementary materials we wanted to produce (such as the 6 farmer guides we created).
These time costs led to the project leaders needing to spend all their time helping to execute the project, and thus very little time was left for evaluating the project. This lack of reflection exacerbated the other issues with the project, allowing them to go unchecked, and also meant that we took longer to terminate the project than we perhaps could have otherwise. Regular pre-set re-evaluation points perhaps could have resolved this, but we neglected to set these up until later in the project.
Miscommunication with our ground staff
Communicating across cultural boundaries can be difficult, even in person. However, the coronavirus pandemic meant that multiple members of our team were not able to be in-country. There were many instances where miscommunication wasted time and caused confusion during the project.
For example, aerators require triphase electricity in order to operate. As such, the project leaders set triphase electricity as a requirement for the farms within the project. What the project leaders neglected to mention was that triphase electricity was required to be available in both the mornings and afternoons (as this is when ponds need to be aerated). The farms that the ground staff managed to find only had access to triphase electricity early in the morning. Because the project leaders did not mention the time stipulation, these farms seemed to fit all the requirements, and so were accepted into the study. This, in turn, meant that aerators, one of the main elements we wanted to test, could not be properly assessed on these farms without expensive workarounds such as a generator. It took us weeks to realize this error. This stands both as an example of the level of detail and complexity that is present in working with farmers in India, and how even small details missed can have an outsized impact on the project.
Assuming what our Indian stakeholders wanted
We decided to run a rigorous scientific study primarily to showcase the value of higher welfare farming practices to NGOs, farmers, and government officials in what we felt was the most honest and believable manner (peer-reviewed journals). We believed that a peer-reviewed study was valuable because we had seen positive signs from NGOs, farmers, and government officials that this was indeed a project they were interested in. What we did not realize, however, was that this was not something they saw as a requirement for working with us. It took us a long time to realize that these stakeholders were already happy to act on implementing higher welfare without the study.
Undervaluing building relationships with farmers
In India, building trust with farmers is a necessary part of making improvements for their fish. Without trust, farmers will not be comfortable sharing details about their farm operation, won’t believe in the importance of fish welfare, and are liable to view an RCT study design with scepticism and as unfair to the control farms. Because we wanted to collect data quickly, our project leaders couldn’t give our ground staff enough time to build these relationships of trust before trying to build our plans for the study. This led to us having inaccurate data from farmers who were hesitant to disclose information, and to farmers not engaging in the study as we had hoped.
Our Current Approach: Expansion of the Alliance for Responsible Aquaculture and Process Evaluation
Our main focus in India is now the Alliance for Responsible Aquaculture, where we and our local partner NGOs work with farmers to set stocking densities and water quality levels that increase welfare for fish. Unlike the pilot study, our alliance work is primarily focused on helping fish, rather than on generating scientific results and showcasing a model. As a consequence, although we do still take measurements, they are much less intensive and we are less concerned about standardising conditions. Also, the scope of this project is much broader than the pilot study. Currently, 31 farmers have joined the alliance, affecting roughly 380,000 fish per production cycle. We expect to have more farmers join soon.
We still believe that there is a need for Fish Welfare Initiative to deeply evaluate our impact on fish welfare, and some information will be taken from our partnered farms to help us review this. However, the focus of our evaluation is now on a process evaluation, which evaluates whether a program is being successfully implemented at the operational level (a change that was recommended by our advisors) [1]. We believe this is the right approach because our uncertainties lie more in the implementation of our program, rather than in the impact of our welfare improvements.
Overall Thoughts
Although the pilot study had many issues, we are generally happy with the position it has left us in. Much of the time spent (such as on finding farmers and building relationships with them) has been valuable in establishing the Alliance for Responsible Aquaculture. Most of the farmers who were in the study have now joined the alliance.
Running a local study was also extremely useful in engaging our key stakeholders early on (such as NGOs and government officials). The primary reason we terminated the study was that we found that those stakeholders were more eager to start making fish welfare improvements than we had expected, and so we felt it was time to focus on direct work for fish [2].
We hope that some of these lessons can be valuable for groups considering intricate projects in the field. If you would like to talk more about our experience with the study and lessons learned, please feel free to contact tom@fwi.fish
Endnotes
To read more about process evaluations, see IDinsight’s summary.
Stakeholders also became more eager to start implementation because we successfully simplified our intervention and messaging such that they could better see the value of our project without the need for a study.