Assessing the Societal Benefits of Applied Research and Expert Consulting in Water Science and Technology

Keywords: assessment ; evaluation of applied research ; impact ; outcome ; output ; productivity ; Solar Water ; Drinking-Water ; Disinfection ; Sodis ; Field Reference EPFL-ARTICLE-180144doi:10.14512/gaia.21.2.6View record in Web of Science Record created on 2012-07-20, modified on 2017-07-16

ost research, both fundamental and applied, is supported by public funding. Society thus requires assurances that this investment is used responsibly and that benefits are generated. For fundamental research, where short-term, direct benefits to society are not expected, the focus is on demonstrating the quality of the research, including its novelty and the extent to which it stimulates further research. A variety of bibliometric indicators (impact factor, h-index, etc.) have been developed to assess fundamental research (Butler 2008), though caveats have been raised regarding the application of these indicators beyond their originally-intended uses (EASE 2007).
These indicators are less useful for applied research since they do not address broader impacts on society. Applied research that provides the basis for the production of economic goods can be assessed by the generation of patents or licenses, but these indicators are not applicable to research that supports the creation, protection or sustainable use of public goods (e. g., environmental research). Although it is unlikely that quantitative indicators can be developed that would comprehensively address applied research relating to public goods, it would be useful to have some indicators -both quantitative and qualitative -that demonstrate the benefit of public expenditures on this research.
Transdisciplinary research (i. e., research that extends beyond the boundary of academic disciplines) incorporates stakeholder engagement in the definition of objectives and approaches and in the research itself to foster the co-production of knowledge (Wiesmann et al. 2008, Heim et al. 2011). This has led to a new understanding of collaboration between science and practice in which scientists act not as isolated experts but as partners with stakeholders. This promotes the integration of practical experience with technical expertise and a broader view of possible solutions to applied problems.
A related activity is expert consulting, in which advice is sought or projects are initiated by different clients. The expertise at applied research institutions is valuable to pro spec tive clients from industry, government, universities or other sectors whose needs cannot be met by commercial consulting services. Avoiding unfair competition with private enterprise can be diffi cult as can deter mining the appropriate level of engagement in expert consulting.
Here, we examine these issues using examples from the Swiss Federal Institute of Aquatic Science and Technology (Eawag). In the framework of Eawag's mandate and research portfolio, we discuss the needs and possibilities for the assessment of applied research and expert consulting. We illustrate possible approach-limitations of such indicators. Although we focus on indicators that can be clearly defined (if not always quantified), we emphasize that such indicators must always be complemented by other tools for the assessment of applied research, such as expert peer review.

Applied Research and Expert Consulting at Eawag
Eawag's mandate encompasses research, education, and expert consulting in the field of aquatic science and technology. Eawag fulfills its mandate in collaboration with the two technical universities and three other research institutes that, together, constitute the Domain of the Swiss Federal Institute of Technology (ETH Domain).
Eawag's research portfolio has evolved over time. Over the last two decades, academic programs in environmental sciences and engineering have been established at the Swiss Federal Institutes of Technology in Zürich (ETHZ) and Lausanne (EPFL). Eawag has participated extensively in this development, particularly at ETHZ. Eawag has also intensified its own focus on applied interand transdisciplinary research (see figure) with the goal of bridging between theory and practice.
It is important to bear in mind that "Eawag's research" corresponds to the aggregate research activities performed by its staff. This research is mainly investigator-driven in that researchers obtain external funding to support doctoral students and postdoctoral researchers. This externally-funded research would not, however, be possible without the infrastructure, facilities, administrative and technical support as well as the salaries of permanent staff that are provided by Eawag.
Given the high level of direct federal support that Eawag receives (nearly 80 percent of its total budget) and its emphasis on applied inter-and transdisciplinary research and expert consulting, it is obvious that Eawag has a strong interest in being able to assess the quality, effectiveness, and societal benefits of these activities.

Assessment of Individual Projects or Programs
Individual projects or programs can be evaluated at varying levels of detail and over varying timeframes (Walker et al. 2008). Particularly for transdisciplinary projects, there has been considerable emphasis on assessing the processes of project design, development and implementation (Bergmann et al. 2005, Schiller et al. 2006. Process is clearly important but cannot guarantee that the results will be useful or applied in practice. There has also been an emphasis on social learning as a tangible benefit of transdisciplinary research projects (Walter et al. 2007). These benefits must also be distinguished from the development of solutions to actual problems facing society.
The most immediate products of a research project are typical ly reports in the scientific literature (including conference www.oekom.de/gaia | GAIA 21/2 (2012): 95 -101 96 Janet G. Hering, Sabine Hoffmann, Regula Meierhofer, Martin Schmid, Armin J. Peter FORUM proceed ings, technical reports, etc.). This output is directly controlled by the researchers and is generated during the project or soon after its completion.
The uptake of results by others (i. e., outcomes) generally requires more time and may also depend on the extent to which the output is made accessible to potential stakeholders and/or promoted by the original researchers. Ulti mately, it is hoped that re search projects or programs will have a (beneficial) impact on society, though this may be evident only many years after the research has been concluded. It may also be diffi cult to trace the in fluence of an individual project on societal activities that are subject to many factors (Douthwaite et al. 2007).
Ex post impact assessments (epIAs) can provide useful insights to funding agencies and/or research institutions that support or conduct transdisciplinary research. But they are also challenging, time-consuming, and expensive to complete (Walker et al. 2008).

Use of Indicators
A complementary approach to conducting a detailed epIA is to monitor selected indicators. This has the advantage of being feasible to apply to more projects than could be investigated in detail, but is subject to concerns regarding the validity, reliability, fairness, transparency, independence, cost effectiveness, and behavioral impact of the selected indicators (ARC 2008). If the goal is to compare a (limited) range of projects within a specified domain, it is not useful to create a new set of indicators for each project. Rather some general set of indicators needs to be identified and applied in a standard and transparent manner that is not unduly burdensome.
Distribution of Eawag's research activities according to the type and disciplinary nature of the research. Activities included in expert consulting are indicated by the shaded area. The size of the red circles indicates the inten sity of effort in the various domains; solid circles indicate established activities, and dashed circles indicate current and anticipated intensification of effort, mainly in association with the Eawag-EPFL Center for Applied Ecotoxicology (Ecotox Center).

FIGURE:
FORUM In this context, it is worth noting that client satisfaction is both an important and self-reinforcing indicator for expert consulting. It also poses problems in that the interests of an individual client may not reflect the full dimensions of the issues relevant to the project.
A key concern throughout all assessment processes must be the inherent tendency of individuals and institutions to respond to indicators as incentives. That is, the indicator itself becomes the goal rather than only corresponding to it. This has the potential that effort will be directed toward "gaming the system" rather than solving actual problems.

Defining Relevant Indicators
An appropriate set of indicators can only be defined in the context of the (general) type of research products and the stakeholders who might adopt and apply them (see table). Within Eawag's thematic focus, researchers produce novel conceptual frameworks, models, methods, and processes, and also develop new applications for existing models, methods, and processes.
One general indicator of the value of these outputs to stakeholders is their willingness to support projects or activities through direct or in-kind funding. For industry, this could also include > a various: conceptual frameworks, analysis, processes, models, applications | b i. e., participation of researchers on advisory boards or expert commissions  joint participation in projects funded by the Swiss Commission for Technology and Innovation (CTI). In-kind contributions could include partial funding of analytical instrumentation by the companies that produce such instruments and/or opportunities to beta test new instrumentation, software or other products.
A key to tracking the uptake and implementation of research products is continuous engagement with stakeholders and practitioners. This is not limited to projects funded by stakeholders (and/or conducted with their participation) but also includes the service of Eawag researchers on advisory boards and expert commissions. Such appointments demonstrate the value placed on the expertise of Eawag researchers.
An integral component of Eawag's research is the training of research personnel at the master's, doctoral, and post-doctoral levels. These individuals gain expertise through applied research that they bring with them as they pursue their careers in various sectors. The attractiveness of these individuals to potential employers is an indicator of the value of the training that Eawag provides. Transfer of knowledge gained through applied research is also accomplished through a program of continuing education for practitioners. The level of interest and participation in these courses indicates their value to the target audience.
In the context of applied research, standard bibliometric indica tors provide useful insight into scientific quality but not into so cietal benefits. In addition, bibliometric indicators have been criticized for their inhibitory effect on inter-and transdisciplinary research (Zilahy et al. 2009). This does not, however, negate the importance of publishing the results of applied inter-and transdisciplinary research in the peer-reviewed literature.
In contrast to the emphasis often put on ISI 2 publications (or their equivalent), it is important for the assessment of applied research to focus on non-ISI publications as well, specifically key trade publications with a target audience of practitioners. Recasting research results for such an audience is not a trivial undertaking, and the effort involved must be recognized if this audience is to be reached effectively.

Three Illustrations
Ideally, it would be of interest to conduct a detailed epIA of one or more Eawag projects and to compare the results with an assessment based on the indicators listed in the table. Although such an effort is beyond the scope of this paper, we find it useful to consider selected examples of Eawag's applied research and expert consulting projects. The purpose of this exercise is not an assessment per se, but rather an identification of aspects or characteristics of the projects that could serve as useful indicators for the societal benefit of similar projects in the future. 3

Solar Disinfection (SODIS)
Motivation and project initiation. One of the major public health issues in developing countries is the lack of access to drinking water free from disease-causing microorganisms (pathogens).
In 1991, the SODIS project was initiated at Eawag with the goal of developing a low-cost method for drinking water disinfection based on existing concepts for solar disinfection. 4

Project development.
In an initial five-year phase, the project focused on evaluating the method under varying conditions in the laboratory and in field settings (Wegelin et al. 1994, Sommer et al. 1997. Subsequent pilot projects were conducted from 1996 to 1999 with local partners in Africa, South America, and Asia to eval uate socio-cultural acceptance, applicability and financial viability. Based on promising results of these studies, the project leader decided to combine further research activities with efforts to disseminate information about the SODIS method. This led to an emphasis on engagement with the World Health Organization (WHO) and broad dissemination of project results in stakeholder-oriented publications. Nonetheless, key stakeholders in developing countries, such as local governments and development agencies, were skeptical about the efficiency of SODIS. Thus, a promotion program was started in 1999 focusing on the education of people without access to safe drinking water, and evaluating and documenting project impact to support the advocacy dialogue with local governments. In addition, further research projects were pursued to fill critical knowledge gaps.

Project findings and benefits. The effectiveness of SODIS under field conditions and the benefits of its use were demonstrated.
Targeted research projects have determined the basis for the inactivation process at the cellular level during solar irradiation (Bosshard et al. 2010) and the stability of PET-material used for SODIS (Schmid et al. 2008), and documented the health impact of promoting household water treatment (Conroy et al. 2001, Graf et al. 2010. Research also addressed the critical issue of the factors affecting the adoption and sustained usage of SODIS (Tamas et al. 2009). An independent study has shown compliance above 90 percent for use of SODIS in rural communities in Cambodia and reduced incidence of dysentery among children in the SODIS group (McGuigan et al. 2011). Through co-funding and collaboration with stakeholders, application manuals and supporting materials have been made available in multiple languages.
Through SODIS, Eawag became involved in WHO discussions on household water treatment systems (HWTS), which led to the establishment of the WHO network for HWTS promotion. Collaboration with this network has, in turn, been an important factor in the recognition of SODIS as a valid method for drinking water treatment and has facilitated the uptake of other household-level technologies developed at Eawag.

FORUM
Outlook. Today, SODIS is used by about five million people in more than 30 countries to disinfect drinking water. The method is globally recognized as valid option for drinking water treatment and is promoted by various development organizations. Several governments in Latin America, Asia and Africa have included SODIS in the extension and training curricula of their Ministries of Health and of Education. Despite this success, Eawag, as a research institute, cannot be expected to take on the role of a develop ment agency. Negotiations have recently been completed to transfer the SODIS activities to the aid agency Helvetas.

Rhone-Thur Project Motivation and project initiation. River restoration activities in
Switzerland have received popular support and public funding. However, such projects rarely incorporated scientific analysis and success evaluation as a potential basis for improving future restoration activities.The Rhone-Thur project, initiated in 2002, sought to bring together expertise in ecology, hydromorphology, decision science, and hydraulic engineering to address these gaps. A key aspect of the project was the early identification of stakeholders, who were involved not only in the discussion and participa tory process but also in the elaboration of consensus-based solutions. In addition to scientific publications, a strong emphasis was put on the development of handbooks for decision making (Hostmann et al. 2005) and success evaluation (Woolsey et al. 2005) that could be used in future restoration projects.
Project findings and benefits. The Rhone-Thur project addressed four main topics: local river widening, mitigation of hydropeaking, decision making, and project assessment. A comparative study of five widened river stretches demonstrated that the success of local river widening is highly site-specific, and the extent of improvement of riparian habitat is inherently limited in small-scale (about one kilometer) efforts (Rohde et al. 2005). The mitigation of sub-daily flow fluctuations associated with hydropeaking was identified as a crucial aspect of restoring natural flow regimes (Meile et al. 2011). Clear definition of objectives, identification of appropriate reference systems, and systematic assessment of performance relative to pre-restoration benchmark conditions were found to be critical components of the evaluation of restoration activities (Woolsey et al. 2007). Products of the project were incorporated in the federal regulations and recommendations to the cantons for the planning of restoration projects.
Outlook.Handbooks developed from the Rhone-Thur project are being used by practitioners in river restoration projects. Increasing interest in this topic is motivated by the political decision in late 2010 to allocate funding for the restoration of 4000 kilometers of Swiss rivers and also for mitigation of the ecological impacts of hydropower generation. Follow-on projects from the Rhone-Thur project provide scientific support for future restoration and mitigation efforts. 5

Lake Kivu Project
Motivation and project initiation. The deep waters of the East African Rift Lake Kivu, located between Rwanda and the Democratic Republic of the Congo, contain approximately 60 cubic kilometers of dissolved methane, a valuable natural resource. In addition to the economic benefit, extraction of dissolved gases from the lake would enhance public safety by reducing the likeli hood of a catastrophic gas release. Eawag first became involved in the study of Lake Kivu through participation in an emergency expedition in 2002 when it was feared that lava inflow to the lake from the Nyiragongo eruption might trigger a sudden gas release.

Project development.
Although several pilot projects for methane extraction are in operation or in planning, technical support is needed not only for the extraction technology itself, but also to prevent potential harm to the lake ecosystem and its fisheries. As the deep water of the lake contains high concentrations of nutrients and other dissolved substances, it cannot be released to the surface after methane extraction without significant impacts.
Expert consulting for the government of Rwanda included sim ulations of the impacts of different methane extraction scenarios (Wüest et al. 2009) as well as providing instruction in running simulations for government representatives and training for a local monitoring team. Specialists from Eawag participated in the expert group that formulated guidelines for a safe and environmentally sound extraction of methane and reviewed the environmental impact assessment of a methane extraction project for the Multilateral Investment Guarantee Agency. Research projects were conducted in cooperation with local universities and incorporated local capacity building.

Project findings and benefits.
Research demonstrated that the primary mechanism for gas transport from the depth of the lake is slow upwelling rather than vertical turbulent exchange, as was previously thought. Simulations indicated an increase in methane production from the sediments, consistent with the observed increase in methane concentrations in the deep water and suggesting that they could reach saturation by 2100 (Schmid et al. 2005). Aerobic methane oxidation was identified as an important sink for methane and the primary process preventing methane from reaching the atmosphere (Pasche et al. 2011). Eawag's expert consulting identified extraction scenarios that would have an immediate negative impact on the lake; on the basis of this work, such scenarios have been avoided and new projects are required to follow the guidelines formulated by the expert group. Eawag's re-> FORUM search and expert consulting have stimulated a growing awareness of the complexities of the system on the part of both government agencies and concessionaires.
Outlook. The economic opportunities associated with methane extraction ensure that Lake Kivu will continue to be a focus of active investigation. Currently, a local monitoring group is investigating the effects of the re-injection of degassed water to the lake. In addition to two continuing research projects conducted by Eawag, there is a growing number of local and international research teams working on Lake Kivu.

Generalizations and Recommendations
Despite the differences among these three examples, there are several common points that emerge. The first is the critical importance of key external partners, mainly local stakeholders. The involvement of such partners at an early phase in the project development (through participation and/or co-funding) provides access and opportunities for researchers and is an indicator of the importance of the project to its target audience. Capacity building for external partners can be both an important outcome of the project and also a motivation for continued cooperation (Maselli et al. 2006). A second point is the need to generate products (e.g., handbooks, training manuals, etc.) that are valued and used by stakeholders and environmental consultants. The presentation and level of these materials must reflect the needs of the users. A third point is the need to recognize the limits of research institutions with regard to broad implementation or dissemination. Partnership with stakeholders also allows the participating research institution to stay within its own mandate and not to overextend itself in other domains. Fourth, the drive and dedication of individuals cannot be overestimated. Projects that involve substantial stakeholder engagement can be quite time intensive and generally do not result in the same level of research productivty as more conventional scientific pursuits. Both the individuals involved and the institutions that employ them must have clear expectations regarding these demands and the value of this work. Finally, it should be noted that the most mature of these projects (i. e., SODIS) has been sustained by Eawag for over two decades and has only recently reached the point where it can be feasibly handed off to an external development agency. Assessment of such projects must balance the need for accountability with the time needed for the uptake and diffusion of applied research results.
Some of these points can be related directly to indicators for the evaluation of completed projects. In particular, stakeholder involvement should be sustained for the duration of the project, and a recursive renegotiation of goals and approaches may be required as the project progresses (Wiesmann et al. 2008). Development and use of appropriate products can be an explicit, monitored goal. In addition, desirable follow-up activities or outcomes can be identified and steps taken to facilitate such outcomes during the project period.
Assessment of the societal benefits of applied research and expert consulting is complicated by two key factors: the inherent time lag between completion of the project and its outcomes and eventual impacts and the influence of multiple factors that may obscure the benefits from a single project or activity (Walker et al. 2008). Nonetheless, the aggregate societal benefits that derive from such activities should not be sacrificed because of the difficulty of assigning credit.
An additional concern -particularly in transdisciplinary research projects -is that they often demand a level of integration and synthesis that goes beyond the usual experience of participating researchers. Despite its importance to the overall success of the project, integration may be perceived by individual researchers as an "unnecessary luxury" (Pohl 2005).
An institution like Eawag, which has a strong interest in fostering applied research and expert consulting, must continue to address the issues raised here on an on-going basis. Nonetheless, it may be useful to make some provisional recommendations on the basis of this analysis.
Desired project outcomes should be explicitly included in the design and planning of applied research and expert consulting projects (Walker et al. 2008). Target outputs that would foster such desired outcomes should be identified and incorporated as goals for the project. Integration and synthesis should be explicitly identified as necessary activities throughout the project and as a goal for the project. Responsibilities for integration and synthesis should be assigned and appropriate resources allocated to support these activities.
A reasonable level of effort should be made to track outcomes of completed projects in a manner that allows comparison among projects; appropriate resources should be allocated for this purpose. This information should provide a basis for the research institution to make decisions regarding the focus and design of future projects.
These recommendations are offered in the hope of improving the assessment of societal benefits of future projects in applied research and expert consulting.