SMART Expert Panel Meeting

Standardized Monitoring and Assessment of Relief and Transitions (SMART)

An Inter-Agency Initiative

_______________________________________________________________

Expert Panel Meeting on the SMART Methodology

July 13 – 15, 2004: Brussels, Belgium

Centre for Research on the Epidemiology of Disasters (CRED)

Summary

The Centre for Research on the Epidemiology of Disasters (CRED) organized the Expert Panel meeting to discuss the development of the SMART Methodology, July 13-15, 2004, Brussels, Belgium. This was part of the consultative process to review the components and options for triangulating nutritional status of children under five, mortality rate, and food security in an integrated survey protocol. The Expert Panel agreed on the technical parameters for combining the assessment of Nutritional Status of Children Under-Five, Crude Mortality Rate (CMR) and food security in an integrated survey protocol. The critical set of core information for each component was outlined which will be used as the base for developing the methodology. The Expert Panel reviewed and endorsed the windows-based analytical software which simplifies data entry and analysis, with quality-assurance checks built into the software program. It reviewed several pilot countries for testing the methodology under various field conditions. It established an inter-agency work-plan which is to have a draft manual ready by end August to be reviewed by the Expert Panel and others, finalized by end September, and then pilot-tested. A meeting of all G-8 donors and other stakeholders will be convened before end 2004 to roll-out the SMART Methodology.

Background

The development of the SMART Methodology is a follow-up to the July 2002 SMART workshop that established a broad-base consensus that Crude Mortality Rate (CMR) and Nutritional Status of Children Under-Five are the most vital, basic public health indicators of the severity of a humanitarian crisis. They are useful for assessing need and prioritizing resources. They monitor the extent to which the relief system is meeting the needs of the population and thus the overall impact of the relief response. The 2002 workshop recommended the development of a generic, standardized methodology and the “SMART Manual for Dummies” and other tools for all humanitarian organizations.[1] The Canadian International Development Agency (CIDA) provided funds for developing the SMART Methodology and pilot-testing in one country.

With the adoption of the SMART Initiative by the G-8 Action Plan on Ending Famine,[2] the development of the SMART Methodology and related activities is a priority of donors and participating organizations. With multi-partner buy-in, SMART is an important focus of G-8 member support for improved worldwide emergency assessment and response systems. G-8 donors are advancing SMART by providing funds for priority actions, participating in technical reviews and meetings, advocating the use of the SMART Methodology in other fora, including the Good Humanitarian Donorship (GHD) Initiative and working with international, non-governmental organizations and humanitarian coordination groups to encourage the adoption of the SMART Methodology.

The following is a summary of the Expert Panel meeting presentations. PowerPoint presentations and discussion notes are posted at smartindicators.org.

SMART Methodology (A. Ralte)

The SMART Methodology will help to determine needs, monitor and report on trends and the impact of humanitarian aid. This will be undertaken using (for the first time) standardized data collection, analysis, interpretation and reporting. It is envisioned as a complete package of a survey protocol, guidelines, and windows-based analytical software.

The audience for the SMART Methodology is all humanitarian organizations, and SMART partners, including host governments. The level of technical difficulty for the methodology is targeted to PVOs/NGOs with technical background and/or training. As noted at the SMART workshop, July 2002: “The methodology should be technically sound and simple, that is do-able by our partners, in particular PVOs/NGOs. This should be the balance.”

The SMART Methodology will have the following elements:

Iterative – starting with SMART Version 1, the Methodology will be regularly upgraded based on operational research (a related SMART activity).
Integrated – nutritional status of children under five, mortality rate, and food security.
Primer – Version 1 will be a primer, or the most basic essential methodology.
Time element – Version 1 will be do-able in critical, acute emergencies.
Linkages – it will be an electronic manual with hyperlinks to references so that there is a layering of guidelines to meet varied needs of organizations.

SMART Version 1 will be based on anthropometry (nutritional status assessment), with the most critical components of mortality (CMR) and food security (HEA “lite” version) integrated. As recommended at the July 2002 workshop, it will be based on viable, current best-practice methodologies until further research determines change.

Integrated Concept (J. Seaman)

The July 2002 workshop indicated that nutritional survey data cannot be interpreted in isolation, and that the food security context needs to be understood. The interpretation of nutrition data is difficult and food security helps put this in context.

The Household Economy Approach (HEA) (Livelihood method) is the only approach that can meet SMART Version 1 criteria. Although HEA is data-heavy and needs intensive training and know-how, HEA “lite” version that uses information from key informants (farmer, tax official, etc.) is an option for SMART Version 1. The critical set of information need is the main features of the economy (usually limited to two to three activities)with assumptions or uncertainties clearly presented in the methodology. HEA started with the “lite” version and then got “heavy” – the essence is in the data structure, and what questions to ask – and, therefore, key informants can be used to construct the basic data set.

The aim is a “joined-up story” that triangulates nutritional status and food security, and with mortality rate. This data will help to tell the “story” as the SMART indicators are considered basic well-being indicators. Food quantity and quality are the basic drivers of mortality and triangulated data should correspond -- if they do not, this should be an alert for the community to investigate what is happening to the population.

Nutrition Assessment (M. Golden)

Mortality rate is the most basic public health vital statistic. It gives an overall measure of the severity of a situation. Wasting and edema rates are the basic measures of nutritional health.

There is a need for both quantitative and qualitative assessments in emergencies, and a need to triangulate data. Surveys must be as simple as possible while providing reliable information. The survey instrument should collect essential but not excess data, and it must not have a long questionnaire - every question added means a compromise on data quality.

We need to systematize how to resolve practical problems and constraints, such as security and arrivals from inaccessible areas, population movement, scattered population, and topography. There is also difficulty in determining age – should we use 110 cm cutoff instead? Measurement errors are rampant due to inadequately trained staff and inaccurate population figures – and most survey data are likely to be overestimates. This is why transparency is needed, including a standard reporting format for all humanitarian organizations.

The critical set of quantitative information needed includes: wasting rates, edema rates, Crude Mortality Rate (CMR), population size, and demography. It also needs context and food security data for interpretation of the quantitative data. A survey can tell us the current level of severity and help to calibrate surveillance data but contextual data is needed to interpret trends or changes. It is also important to note that the diet can be poor and there are many other nutritional deficiencies (vitamin A, iron, iodine, vitamin C, riboflavine, B12, folate, niacin, etc.) that do not cause wasting or edema but can lead to immuno-incompetence and death. See the illustration of the pellagra problem in Angola (PowerPoint presentation).

A review of 400 surveys (increased from 223 surveys reviewed at the time of July 2002 workshop) show that the standard deviation does not change as the population becomes more malnourished. As populations become malnourished, all individuals are affected to a similar extent and the whole distribution moves downwards – and weight loss is the same among malnourished and well-nourished children. Movement in weight-for-height is not necessarily a “trailing indicator” in population terms, and it is a sensitive indicator of stress within the population. Who becomes malnourished may relate more closely to location than other vulnerability indicators. Since vulnerability seems to be geographically determined, geographical targeting is recommended as it is cheaper and more efficient than identifying and targeting individual households.

Basic elements to be considered in anthropometry:

Use systematic random sample where feasible (otherwise, use cluster sampling). Start by choosing person from crowd and go to his/her house, then use EPI-bottle spin method after each house.
Use the 30 x 30 cluster unless data is collected to inform on assumption and there is available expertise in sampling.
Confidence levels should not be fixed – rather it should depend on prevalence and set for practical purpose.
Measure ALL children in household.
Use houses with no children for CMR/demographic and other data collection.
Choose sampling areas based upon characteristic likely to be important determinants – agro-economic zones, tribal/ethnic areas, etc, and not necessarily administrative districts. This will also facilitate interfacing with HEA.
Data triangulation is needed: anthropometry and edema prevalence, with surveillance (incidence/coverage), program coverage, mortality date, population size and structure. Other data is needed to interpret discrepancies and understand causes. Surveys are only one part of a system, and cannot be used in general to decide where to set up a program.
Data interpretation is important – for example, we should not use the “rule of thumb” threshold such as the 10% prevalence cut-off to make decisions without reviewing trend data.

Issues for further discussion and/or research:

Age is conventional and is necessary for surveys that examines weight-for-age and height-for-age, but is not essential for weight-for-height. Should SMART use length/height only? Should this be 65 to 110 cm (current practice), or 65 to 130 cm, or 60 to 100 cm. Or use only age, i.e., 6-59 months?
Prediction of the prevalence of malnutrition from the mean value of the population. Much smaller numbers are needed for a survey. This would allow spatial distribution of wasting to be assessed, and allow more frequent surveys to be undertaken with the same amount of resources.
HIV/AIDS children respond to therapeutic feeding diet but not to supplementary feeding (does this mean that there is a problem with supplementary feeding ration?). There is a rise in malnutrition in HIV-negative children (as well as among HIV- positive children) in high prevalence areas. We should prepare for continued increase in malnutrition rates.
Several other problem areas to be further addressed: unreliable population estimates, sampling in areas of scattered population, and balancing surveillance and survey data needs. For future SMART action – what should we do about micronutrient data and poor diet quality that causes kwashiorkor?

Food Security (J. Seaman)

A livelihood approach has been proposed as this meets the SMART conceptual framework and practical criteria – this will allow a PVO/NGO to explain anthropometric survey and mortality findings, where the PVO is operating at a reasonably local scale. However, existing livelihood approaches, such as the SCF/UK Household Economy Approach (HEA) have potentially large data and skill requirements which are impractical for most PVOs. Therefore, some simplification is required for the SMART Methodology.

The basic logic of livelihood methods (based on A. Sen’s entitlement concept) is to understand the non-crisis economy of a defined population in sufficient quantitative and qualitative detail to be able to estimate the impact of a defined shock on that economy. Specifically, it is to estimate the ability of people in different wealth groups (poor, better-off) to acquire sufficient food to meet their needs, where non-food expenditure (e.g., fuel, health costs) is specified.

The Household Economy Approach (HEA) was originally developed, and has chiefly been used for predicting the impact of a shock, e.g., crop failure from drought, a price rise, on food security – at middle to large scale (e.g., district – national). This requires substantial inputs for data collection and a fully quantified analysis. Household data (a household budget and assets) is obtained for a “typical” household for each (or at least three) wealth groups plus contextual information (such as availability of wild foods, etc.). Data collection is standardized.

Analysis is undertaken by a simple arithmetic simulation, which allows the user to work systematically through a case and develop a hypothesis or hypotheses about the likely impact of a “shock” (to production, prices, etc.) on the households concerned. This works in two steps: 1) an estimate of the direct impact of a shock on food access by household type, and 2) an estimate of the ability of the household to compensate for any deficit in step 1. The model can be expressed on a spreadsheet, or using software (RiskMap). HEA is widely used and has proved to be technically effective. (For illustration, see PowerPoint presentation). The output of an HEA analysis is a logical argument about the most likely connection between a shock and people’s future ability to get sufficient food, where the methods, assumptions and uncertainties are clearly stated.

HEA “lite” (proposed for SMART) uses the basic structure and logic of HEA, i.e., to understand the impact of a shock on an economy. However, information is obtained from key informants rather than by primary data collection, and there is less emphasis on detailed quantification. The aim is to have sufficient information to set nutrition findings in a context, to allow a prediction about likely future changes in nutrition, food needs and relief action. These data techniques have been used by SCF/UK for rapid updating of existing data, e.g., Malawi 2002. (See PowerPoint presentation).

The aim is to put together a “joined up story” about:

What has happened, e.g., flood, earthquake.
How this has affected people’s livelihoods and food access.
Their ability to cope with the shock, and the costs of coping.
The types of external assistance required, bearing in mind the practical constraints.

This should be consistent with and help “explain” nutrition survey findings and mortality. All assumptions and uncertainties should be clear to the end-user.

Issues to be resolved for SMART Methodology Version 1:

Sampling methodology that can be integrated or used in conjunction with the nutrition/mortality survey protocol.
Data analysis program that easily interfaces with the software program for nutritional status and mortality rate.

Mortality (C. Robinson, D. Guha-Sapir)

Many different methods are available to measure mortality such as observations (body counts, burial site activity), participatory appraisal (use of key informants, community mapping or locating health clinics and places of workshop), surveillance (passive – clinic data; active – burial sites, body counts, neighborhood interviews), and surveys. There are three approaches currently used in surveys. These are:

Past household census – determining the number of household members at the beginning of the recall period, the number of births during that period and the status of these individuals.
Current household census – determining the number of household members alive at the moment, the number of deaths and births within the household during the recall period. Migration is critical to include in this formula – and should be included with deaths and births. Note: Meeting participants recommended using this method for SMART since this is the most widely used.
Previous birth history (U5MR) – determining the number of children born in the prior 5 years.

Issues to consider when designing surveys:

What is the context/setting in which we are gathering information? Is this for response or for the record?

What level of precision do we need?

What period of reporting or recall is used? This should be stable and consistent from one survey to the next.

What events contribute to household change, exposure time? How to define household, family?

What kinds of death may be more critical to the life of the population?

How frequently should the surveys be carried out?

Issues to be resolved for SMART Methodology Version 1:

Whether or not to include migration.
Cause of death – what should be the minimum list? We need to review and follow July 2002 workshop recommendations. In addition, since CE-DAT will be used as a data source for the Centre for Human Security on monitoring the human impacts of conflict, conflict or non-conflict related deaths will need to be determined by the SMART Methodology.

Integrated Software Package (J. Erhardt)

A prototype windows-based software has been developed that is simple and flexible for data entry and analysis, with quality assurance checks built into the program. This can be easily translated into other languages. The prototype software is available at

www.nutrisurvey.de/ena/ena.htm.

Technical support will be built in as part of the SMART training and capacity-building. While the software should be free to all organizations, some participants suggested incorporating a maintenance program and user’s fee for de-bugging and upgrading with contributions from users.

Issues to be resolved for SMART Methodology Version 1:

Interfacing HEA data with nutritional status and mortality rate data analysis.
How to build in maintenance of the software program as part of an ongoing SMART activity.

Complex Emergencies Database (CE-DAT) (D. Hargitt)

The objective of CE-DAT is to support decision making and promote effectiveness of prevention and response. There are 8 pilot countries (Afghanistan, Angola, Democratic Republic of Congo, Iraq, Ivory Coast, Rwanda, Sierra Leone, Sudan), and 3 main categories of data (mortality, malnutrition, morbidity). The objective for the next few months is to integrate CE-DAT format with the SMART software, mapping the information, increasing the number of countries and categories, obtain data directly from data source, and use of optical readers for data entry. A reliability scale should be set up for all data on CE-DAT.

Vulnerability: Update on G-8 Discussion (J. Becker)

SMART contributes to larger undertakings that include the G-8 process and initiatives, USAID data policy issues, and other assessment and response initiatives.

A. G-8 Process and Initiatives:

The three U.S. Initiatives under the G-8 Africa and Famine Action Plans are:

Breaking the Cycle of Famine and Increasing Agricultural Productivity in the Horn of Africa
Improving Worldwide Emergency Assessment and Response Systems
Boosting Agricultural Productivity and Rural Development in Food Insecure Countries, especially in Africa

The G-8 Statement on SMART is reflected under Worldwide Emergency Assessments and Response Systems.

B. Assessment Initiatives (outside the G-8 Process) Relevant to SMART:

Humanitarian Initiatives
Initiatives on Agriculture Trade and Development Modeling PARIS 21 (advocacy and SSD plans, statistical capacity indicators, census, rural and agriculture statistics)
OECD WFP-STAT-ODA Statistics (Principles and Good Practice of Humanitarian Donorship)
UNICEF Initiative on Vital Statistics
ECA Ad-Hoc Expert Group on Geo-information in Africa (GIS, GPS, ICT)

(See PowerPoint presentation for details)

Follow-up: A draft scope of work for the Expert Panel on Vulnerability Assessment will be developed and shared with participants and others.

Operational Research (N. Mock)

The session focused on two objectives with emphasis on the first:

Identifying critical gaps in knowledge and research themes related to SMART
Initiating a Research Task Force as part of SMART

Related to the first, the group agreed that there are a number of knowledge gaps that constrain our ability to identify and standardized methods of assessment in complex emergency and post conflict settings. While these gaps do not prevent the launching of SMART Version 1, they are important for further refinement of methods and information use. Priority research themes were reviewed – see PowerPoint presentation for details.

The Expert Panel also concluded that a Task Force for Research (to be established) will convene, probably, virtually to further elaborate, prioritize research and hopefully solicit funding for specific proposals. Nancy Mock agreed to initially coordinate the Task Force. She will solicit membership by email as a follow-up to this meeting.

Pilot-Test of SMART Methodology (A. Ralte)

SMART will be applied in a pilot-country to see if the methodology and the guidelines work under field conditions. The pilot-test will determine ease of use of the methodology and guidelines, and whether any further modifications or fine-tuning will be needed before roll-out of SMART Methodology Version 1.

Participants at the July 2002 workshop had discussed Afghanistan and DRC, and selected Angola as the initial pilot country. Since then because of the G-8 focus, Ethiopia is being considered. Countries such as Ethiopia and Angola where national guidelines are in place make for good candidates. Burundi was recommended due to host government commitment and being a small country might facilitate coordination. Other suggested countries include:

Rwanda – represents a government in transition

Chad - country with high mortality rates

Liberia - where CDC is undertaking population-based maternal mortality study

Malawi - where large-scale household food security assessment has just been completed which might make it useful for comparing the HEA “lite” methodology

Ivory Coast - country currently in emergency phase which contradicts another suggestion to start in a country that is not going through an acute emergency

Follow-up: Recommendations will be reviewed and final selection will be based on further consultations with organizations/individuals unable to participate at the meeting.

Next Steps

Donors (and others) should continue to advocate that funding priorities should be evidence-based and encourage the use of good quality data for making decisions. This will also help tell the story of how humanitarian assistance is impacting lives. Donors should harmonize on assessments, survey and reporting, and develop joint policy/operational guidelines. The SMART Methodology should be a useful tool for practitioners so that all organizations will want to use it voluntarily. Outreach to other G-8 donors (not yet fully on board) and other groups is also an important follow-up.

A. Issues for Further Deliberations:

Access to data – how do we determine who will have access? How do we ensure that data is only being used for the good of the beneficiary?
Protocol for use of data – do we draw limits on what data can be used for? Who owns data that is collected collaboratively?
Future SMART actions – HIV/AIDS, acute versus chronic assessments (also see issues outlined under Nutrition Assessment)

B. SMART Methodology Work Plan:

Expert Panel Meeting, July 13-15, 2004
Draft manual by end August, circulated for review by Expert Panel and others
Finalize manual by end September
Pilot-test: November? Where?
Refine Methodology as needed
Technical Advisory Group (larger meeting) to roll-out SMART Version 1 – before end 2004
Upgrade and apply SMART Methodology in @6 selected countries over 3 years (adding 2 countries per year) with surveys undertaken every 6 months (and/or combine with surveillance where feasible)

[1] See smartindicators.org.

[2] “Building on the consensus reached in the Standardized Monitoring and Assessment of Relief and Transitions (SMART) Initiative, the G-8 will support further activities to improve needs assessment and monitoring of famine and food security. This will include the establishment of a multi-partner experts’ panel to review standards of practice for vulnerability assessments and food security and the development of on-line information systems to disseminate information on vulnerable areas, needs assessments, and the impact of assistance operations.”