Evaluation at all levels is a fundamental function at World Health Organization (WHO). Evaluation of public health surveillance systems is essential to ensure that problems of public health importance are monitored efficiently and effectively. The WHO's Vitamin and Mineral Nutrition Information System (VMNIS) was evaluated from the perspective of public health surveillance and informatics. Steps included: engaging the stakeholders of the evaluation; describing the surveillance system; focusing the evaluation design; gathering credible evidence regarding system performance; justifying and stating conclusions and recommendations, and sharing lessons learned from the evaluation. Following this assessment, WHO has begun major efforts to upgrade and expand the VMNIS and now the database is more flexible and efficient. The database evaluation summarized in this paper provides a good example of how public health evaluation frameworks can lead to improved surveillance and enhanced information systems, thus making progress toward the ultimate goal of improving public health.
Key words: Vitamins and minerals, nutritional status, surveillance system, public health.
La evaluación en todos los niveles es una función fundamental en la Organización Mundial de la Salud (OMS). La evaluación de los sistemas de vigilancia epidemiológica es esencial para asegurar que los problemas de importancia para la salud pública sean monitorizados de manera eficiente y efectiva. El Sistema de Información Nutricional sobre Vitaminas y Minerales (VMNIS, por sus siglas en inglés) se evaluó desde la perspectiva de sistema de vigilancia e informática en salud pública. Los pasos incluyeron: el involucramiento de las partes interesadas en la evaluación, la descripción del sistema de vigilancia, el enfoque del diseño para la evaluación, la recopilación de pruebas fiables sobre el rendimiento del sistema, la justificación y desarrollo de conclusiones y recomendaciones, y la diseminación de las lecciones aprendidas de la evaluación. Después de este proceso, la OMS inició importantes esfuerzos para mejorar y ampliar el Sistema de Información Nutricional sobre Vitaminas y Minerales para lograr una base de datos más flexible y eficiente. Este artículo resume la evaluación de esta base de datos para documentar un buen ejemplo de cómo los marcos de evaluación en salud pública pueden llevar a mejorar la información en los sistemas de vigilancia contribuyendo de esta manera al objetivo final de mejorar la salud pública.
Palabras clave: Vitaminas y minerales, estado nutricional, sistema de vigilancia, salud pública
World Health Organization, Switzerland. College of Public Health Sciences, Chulalongkorn University, Thailand
Evaluation is an essential function at WHO, carried out at all levels of the Organization. It ensures accountability and oversight for performance and results, and reinforces organizational learning in order to inform policy for decisionmakers and support individual learning (1). Public health surveillance is the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding a health-related event for use in public health action to reduce morbidity and mortality and to improve health (2). Such data are used to measure the burden and distribution of disease or risk factors, for identification of populations at risk, or for immediate public health action. Public health information systems are integrated systems for collecting, compiling and disseminating information with the goal of providing a seamless sharing of knowledge to improve public health. Evaluation of public health surveillance systems is essential to ensure that problems of public health importance are being monitored efficiently and effectively.
The WHO Vitamin and Mineral Nutrition Information System (VMNIS) is a surveillance system established in response to the World Health Assembly's call to strengthen the global surveillance of vitamin and mineral deficiencies (3), which affect almost a third of the population worldwide. The VMNIS is managed by Department of Nutrition for Health and Development in WHO. It has served to guide the planning, implementation, and evaluation of public health programmes to prevent and control vitamin and mineral deficiencies worldwide. However, its performance and usefulness have declined recently as stakeholders’ needs and expectations have increased in parallel to their access to information technology, and better knowledge of epidemiology. Therefore, an assessment of the epidemiological and informatics aspects of the VMNIS was conducted to identify opportunities to make the VMNIS a more efficient, comprehensive, and user-friendly surveillance system.
This evaluation was conducted by two authors (NJA and VK) and included the public health surveillance perspective (NJA) and the public health informatics perspective (VK). The assessment of the VMNIS from a public health surveillance perspective used guidelines developed by the U.S. Centers for Disease Control and Prevention (CDC) (4). This six-step evaluation process included: a) engaging stakeholders of the surveillance system in the evaluation, b) describing the surveillance system, c) focusing the evaluation design, d) gathering credible evidence regarding the performance of the surveillance system, e) justifying and stating conclusions and making recommendations, and f) sharing lessons learned from the evaluation.
We identified stakeholders through interviews with VMNIS staff and then invited them to participate in the evaluation of VMNIS through in-depth interviews and through an electronic self-administered questionnaire.
The VMNIS was described by reviewing publicly available online materials, operational manuals, documents and publications developed using data from the surveillance system, interviewing staff and directly observing staff managing the system from data collection to online data dissemination. The focus of the evaluation was decided through structured qualitative interviews and direct observation of the management of the VMNIS and through consensus building with the VMNIS staff and key stakeholders.
To gather credible evidence regarding the performance of VMNIS from a public health surveillance perspective, the system attributes of simplicity, flexibility, data quality, acceptability, sensitivity, positive predictive value, representativeness, timeliness, and stability were described (Table 1).
Its usefulness was evaluated through the administration of a web-based questionnaire to stakeholders identified as potential users of VMNIS in step 1 of the evaluation process. The questionnaire was also sent to those who had previously enquired about the VMNIS databases and to representatives of select departments within WHO and external international partners. The WHO Regional Offices were also involved in this exercise and forwarded to Member States a link to the questionnaire posted on the WHO website.
The questionnaire focused on the use of the information provided through the VMNIS and probed the aspects of the data that are most valuable. It also aimed to gather information on beneficial changes that could be recommended for the VMNIS. Questions gauged the extent to which VMNIS was contributing to the prevention and control of vitamin and mineral deficiency, including increasing the understanding of these deficiencies.
To gather credible evidence regarding the VMNIS from an informatics perspective, we observed staff management and use of the VMNIS databases. We assessed the tools and the database structure and platform used in data management, and reviewed the standard operating procedures for managing the VMNIS database. Additionally, we interviewed VMNIS staff and other key stakeholders to better understand the entire data management process including data collection, analysis, interpretation, and dissemination as well as to identifying opportunities to display the data in a friendly manner.
Numerous individuals, units, departments and clusters within WHO and more than ten external organizations were identified as stakeholders. Internal stakeholders were engaged through face-to-face interviews with thirteen individuals and phone interviews with two individuals. A total of 99 individuals invited to participate completed at least some part of the questionnaire previously described.
The aim of the VMNIS is to monitor the global prevalence of vitamin and mineral deficiencies (2). Components of the WHO's mandate addressed through the VMNIS are: 1) assess the vitamin and mineral status of populations worldwide, 2) monitor and evaluate the impact of strategies for the prevention and control of vitamin and mineral malnutrition, and 3) track related trends over time. The assessment of vitamin and mineral status is conducted by national health authorities, WHO country and regional offices, and international partners. Data from these assessments, disseminated through reports or peer-reviewed manuscripts, are compiled and stored in the VMNIS. The goal of the data compilation is to identify public health problems of nutritional deficiency, track changes over time, and appropriately direct the public health responses.
At the time of the evaluation, the VMNIS was composed of multiple databases. Each database contained information on numerous indicators of one nutrient deficiency: anaemia (iron), vitamin A, or iodine (Table 2).
The management and uses of the VMNIS databases are illustrated in Figure 1. Data extracted from reports and publications onto either a written data extraction sheet or into an electronic spreadsheet were entered through an electronic database application disaggregated by age group, administrative level, sex, location (urban versus rural), and physiological status (pregnant, non-pregnant, lactating, non-pregnant non-lactating). Once the databases were updated with new data, reports were generated in portable document format (PDF) and uploaded to the WHO website (http://www.who.int/vmnis/en/) for public access.
Some data from the VMNIS database (e.g., prevalence of vitamin A deficiency in pre-school aged children, night blindness in women, and anaemia in both women and children) are shared with the Nutrition Landscape Information System (NLiS), an interactive platform through which a user can query by country and obtain information on a number of diverse indicators of nutritional status (5).
The VMNIS staff and other stakeholders periodically exported and analyzed data from the VMNIS databases using separate data management software. They then generated reports on the data available by country or used the data for other analyses to be presented in the peer-reviewed literature. WHO headquarters would generate draft prevalence reports and verify the data with Member States prior to disseminating final global prevalence estimates. WHO would then make final reports available to the public on the WHO website.
With consensus of the VMNIS staff and other key stakeholders, we focused the evaluation on the performance of the VMNIS relative to its ability to complete its stated objectives. This focus was designed to provide the most useful information for improving the VMNIS for more efficiently fulfilling its mandate.
At the time of the evaluation, data from each of the 193 WHO Member States were organized alphabetically. Though Member States often had numerous nutritional surveys, the relatively small number of entries made the VMNIS simpler to manage than if there were many more Member States and surveys. VMNIS staff summarized the data in electronic spreadsheets, which were easily disseminated.
Determining if data should be included in the databases and data extraction were simple tasks; however they required epidemiological expertise. The multi-step process of extracting data to paper or electronic spreadsheets and then inputting those data into the electronic databases make the process of data entry more complex than if data were entered directly into the electronic databases. Additionally, the data entry process was further encumbered by the data entry screens of the database application not matching the data extraction sheets. For each vitamin and mineral there was a data extraction and entry outline comprised of some components of a data dictionary and some components of a procedure manual. Experienced staff had to provide extensive training and mentoring to any new staff supporting the databases.
The databases were moderately flexible; however, more flexibility in terms of indicators to include could be beneficial. The database application was flexible enough to add new variables, but the task required hiring a computer programmer to modify the application. If a variable had not been initially extracted, VMNIS staff had to revisit original reports to re-extract and then enter data, a time consuming process.
VMNIS staff made a classification for data quality according to whether the data was extracted from published, peer-reviewed manuscripts or final reports from Member State or organizations or whether the data were incomplete or obtained from draft reports. These latter remained pending until the original reports were finalized or published as peer-reviewed manuscripts. The data quality classification did not consider collection or survey sampling methods. Nonetheless, VMNIS staff followed well established procedures to optimize data quality. They included in the databases only results from indicators measured with specific, predefined, valid methods; results from surveys with representative sampling methodology that was adequately explained in the report or publication; results from biological indicators measured with specific, predefined high quality laboratory methods; results from vitamin and mineral deficiencies prevalence estimates confirmed and cleared by Member States.
Some weaknesses related to data quality were:
The acceptability of data inclusion into the database was high. The response rate to staff requests for information was approximately 50-60%. After health and nutrition surveys were completed in Member States, VMNIS staff actively collected the data for the VMNIS database. Because of the active surveillance, compiling data into the VMNIS required little additional effort on the part of Member States, regional offices, or partner agencies and no additional effort on the participants. However, it is worth noting that the execution of a survey with nutritional indicators is a difficult, time consuming, and costly endeavor. The collection of biological data often requires a cold-chain and highly skilled field workers, both of which are often lacking in resource-poor settings and may have low response rate from survey participants.
The sensitivity of the VMNIS relied on the precision of the methods used to measure the biological indicators in the field and the laboratory during the executive of the nutrition survey. For example, the haemoglobin cut-off for defining anaemia in pregnant women is 110 g/L and 120 g/L in non-pregnant women. Therefore, aggregation of results from pregnant and non-pregnant women, 1) increases the number of false positives (number of women detected as anaemic who are not anaemic); 2) decreases the positive predictive value; and, 3) causes an overestimation of the prevalence of anaemia in women.
The inconsistency in the data received from Member States results in an important problem for sensitivity, positive predictive value and the quality of the global estimates of vitamin and mineral deficiency. However, if only those surveys with no limitations were used to prepare global estimates, there would be very few data from which draw estimates. Though, there is heterogeneity in research and survey methods to collect data on vitamin and mineral deficiencies, VMNIS accepts some degree of decrease in data quality caused by this heterogeneity in order to compile data at the global level.
The representativeness of the VMNIS varied by nutrient, nutritional indicator, and by each target group. Indicators of the representativeness of the VMNIS were the percentage of each target group covered by the data in the database, and of the population in a WHO geographical region covered by the data (Table 3).
Many months or even years sometimes passed from data collection in the field to WHO receiving a report with data to enter into the database. If staff must extract data from published peer-reviewed manuscripts, the time lag from data collection to VMNIS obtaining the data could be even longer. Data extraction and data entry take from a few minutes in the case of very small reports or reports with very little vitamin and mineral information to many hours for large reports to many days or weeks if follow-up with the Member States is required. The time needed for data extraction and entry is dependent on available human resources.
There are a number of characteristics of the VMNIS that strengthened its stability; however, there is some room for improvement (Table 4). The most important factor for the stability of the VMNIS is the dedication of personnel time and resources for its existence. The stability of the VMNIS is linked to its usefulness. As long as stakeholders consider the VMNIS useful, it is possible to find the resources necessary for its maintenance.
Stakeholders pointed out several characteristics that could improve the usefulness of the VMNIS. The feature most frequently cited on the questionnaire as desirable was the availability of country nutritional profiles (59%), ability to query the database by population group, so that precise data could be extracted from the VMNIS (47%), the provision of data in a downloadable electronic spreadsheet format (47%), the ability to select survey information by year (46%), and the ability to download prevalence estimates based on empirical data (45%). Additional vitamins and minerals of interest cited by stakeholders included folate (60%), iron (55%), zinc (49%), and vitamin B12 (39%). Nineteen percent of questionnaire respondents reported an interest in “other” vitamins or minerals with vitamin D being the most commonly reported “other” nutrient. Stakeholders were divided into those who have used the data from VMNIS and those who have not and both groups showed a high interest (66% and 74% respectively) in national level data and much fewer showed an interest in local-level data (less than 25% in both groups).
The electronic database application was originally developed in 1998, and before that time, data were collected and stored in a simple electronic spreadsheet table. The development of the electronic database application allowed for local access within the WHO computer network and was an appropriate technological advance at that time for the purposes of the VMNIS. Nonetheless, given more recent advances in technology and the needs of stakeholders, the electronic database application could benefit from an upgrade. Such an upgrade would allow the VMNIS database application to be redesigned to meet stakeholders’ needs.
Stakeholders acknowledged that WHO plays an important, and unique, role in maintaining global surveillance of vitamin and mineral deficiency through the VMNIS. Stakeholders noted that the data in VMNIS could be used for different purposes including advocacy, targeting of resources at a global level, monitoring progress over time, understanding the global burden of disease, and making country and regional comparisons.
The VMNIS addresses the important public health problem of vitamin and mineral deficiencies. It is the only surveillance system in the world monitoring the global prevalence of vitamin and mineral status in populations.
The following recommendations summarize the main findings of the evaluation:
We conducted a round table discussion with members of interested parties to share lessons learned from the evaluation. We also shared the findings in the form of a draft report that was circulated among the Evidence and Programme Guidance Unit and the leadership of the Department of Nutrition for Health and Development. These results are being used to update and improve the vitamin and mineral database of the VMNIS and strengthen the global surveillance of vitamin and mineral status in populations.
Following the evaluation, WHO invested in upgrading and expanding the databases into a more comprehensive information system.
The upgrade of the VMNIS Micronutrients Database occurred in three phases. Phase I was database restructuring and was completed in 2011. With the goal of a more efficient database that exploited new innovations in information technology, and included more nutritional indicators, the database was re-structured with three levels of users: editor, approver and administrator. Only surveys representative at least at the 1st administrative level (i.e. state, canton, province) where the sample is population-based, household-based, or facility based, the sample is cross-sectional and uses standard, validated data collection techniques and laboratory methodology are eligible for inclusion. The focus for data entering switched from a nutrient to survey- approach, allowing the extraction of all information relevant to micronutrients from each survey included and extracted into the database. To minimize error, UN or WHO regions, name of first administrative unit in a country, measurement unit conversions, and laboratory methods available are now automatically displayed. New indicators and cutoff values can be added or changed only by the administrator. A quality control system allows validation of the data entered before it is displayed in the database. Data can be entered from outside the WHO internal environment when permission is granted.
Phase II was data migration and population of the new database structure. This phase aimed to transfer the data available in the VMNIS databases on anaemia, vitamin A deficiency and iodine deficiency to the upgraded VMNIS Database. As the new platform included more stringent inclusion criteria and also the extraction of additional data, the migration was not possible and the screening and extraction had to be repeated. Out of the existing 3500 entries, only approximately 950 complied with the new inclusion criteria. Additional data not previously noted were extracted and entered into the VMNIS database. A digital repository of all the surveys and communications previously identified was completed in 2012. The current search strategy includes a retrieval of all published literature through a structured search of PubMed and also includes contacting WHO Regional Offices, other UN agencies, and other partner organizations to solicit information on nutrition, micronutrient and health surveys. The systematic search will be repeated yearly to maintain the database up-to-date.
Phase III, the redesign of the dissemination platform, has been completed. The redesign of the dissemination platform allows the end-user to query and export data by country, micronutrient, indicator, age group, physiological status, and year. The end-user can also generate tables and graphs in addition to seeing the prevalence and distribution of information. Once a survey has been validated by the approver the data are available to the end-user via internet.
The VMNIS is now more flexible and efficient and has taken greater advantage of updates in information technology since its inception in 1991. Other improvements taking place will make the VMNIS more successful in reaching its objectives of monitoring and evaluating the vitamin and mineral status of populations and will provide tools and resources to support the assessment of the global vitamin and mineral nutritional status. The VMNIS provides a good example of how evaluation of surveillance systems can lead to improved surveillance and enhanced information systems, thus making progress toward the ultimate goal of improving public health.
This evaluation was conducted in 2009 by technical staff (NJA and VK) from the US Centers for Disease Control and Prevention (CDC) upon request of the Department of Nutrition for Health and Development. Nancy Jennings Aburto was an Epidemiology Intelligence Service officer with the International Micronutrient Malnutrition Prevention and Control Program (IMMPaCt) team and Vipat Kuruchittham was a fellow in the CDC Public Health Informatics Fellowship Program. This paper summarizes the findings of the evaluation, the recommendations and actions that led to the upgrade and expansion of the vitamin and mineral nutrition surveillance system. We would like to thank the following individuals for their support in this evaluation: Francesco Branca, Shelley McKenna, Jonathan Siekmann, Monika Bloessner, Ann-Beth Moller, John Shannon, Colin Mathers, John Rawlinson, and Carla Abou-Zahr. The authors would like to acknowledge Paule Pillard, Akaki Lekiachvili, Mike Coss, and Rafael Flores-Ayala. Technical support was received from the U.S. Centers for Disease Control and Prevention, especially Mary Serdula from the Division of Nutrition, Physical Activity, and Obesity and the Office of Informatics and Information Resources Management at the National Center for Chronic Disease Prevention and Health Promotion.
Disclaimer: Nancy Jennings Aburto, Luz Maria De-Regil, Juan Pablo Peña-Rosas, Grace Rob and Lisa Rogers are currently staff members of the World Health Organization (WHO). The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions, policy or views of the World Health Organization.
Recibido: 10-07-2013
Aceptado: 10-08-2013