There are two discrete skill sets needed on the ground when implementing a survey. The first skill set is focused on administration/logistics, and the second skill set is focused on research design. The first set of skills is needed for administration: budgeting, creating route plans, recruitment, and management of staff. Administration requires a level of familiarity with local conditions; for example, the ability to quickly estimate costs and troubleshoot logistical issues are important here. The second skill set requires knowledge of research methods to ensure that survey implementation is consistent with the research design. Researchers must be able to recognize any deviations from the protocol and address them in a way that leads to as little bias as possible. Important here is a deep understanding of the survey protocols and possible alternatives, in the case that changes need to be made on the ground.
If you will not be present during survey administration, you will need to either hire a firm or individuals who can work together to cover both sets of needs. There are clear advantages to hiring a firm if you have the budget, the biggest being that firms coordinate internally and balance both sets of needs, ensuring that logistics accommodate the design and vice versa. A possible drawback is that firms frequently have their own protocols, and these default procedures are usually at a lower standard than the latest protocol being used in academia. Upgrading protocols is a costly process and firms may push back against the use of stricter, or simply different, practices.
If you will be present during enumeration but you have a small budget, or do not feel you could manage the entire implementation (administration and design) yourself, a good alternative is to hire a field coordinator from a survey or research firm on a consultant basis. This person can help with administration while you take on the design-related work. Additionally, hiring someone for administration locally can do a lot to help with cross-cultural management. The types of management procedures that might work to motivate or sanction employees in the US may not work in another context, so someone who knows what is acceptable and effective can add a lot of value.
When setting up contracts with local firms it is important to get the incentives right—thorough and good work should also be the most profitable for the firm. You can do a lot to set expectations and incentives in the contract. For example, pay on delivery where possible (although it is customary to pay some costs upfront to cover fixed expenses like transport and early salaries). You can also choose to impose financial penalties for late or low-quality data, but be sure to make these requirements clear up front and provide specific rules for what constitutes low-quality work and how late penalties will be assigned.
In addition to direct costs, it is reasonable for a local firm to charge overhead. This can vary from context to context, and it is best to check against the budgets of other similar projects to make sure the rate is reasonable.
As you prepare to begin survey implementation, the first and most important part of the survey is budgeting expected costs. It’s important to be thorough and detailed in putting together your budget; it will be critical throughout implementation and will be closely considered by potential funders.
The total cost of a survey is the sum of fixed costs, like transportation and equipment costs, and variable expenses like salaries, per diems, and administration costs. See the attached budget template for an overview of typical costs and notes on how to estimate them.
Estimating total salary costs before drawing the sample (needed in order to determine the teams and route plans) requires a bit of guesswork. One approach is to estimate the work-hours needed to conduct the survey (survey length x sample size) and divide by some estimated number of enumerators to come up with the number of enumerator days you will need to pay. The per diem may need to cover food and lodging, and make this clear to enumerators so they can plan accordingly. For surveys that will require long fieldwork, it is good practice to pay salary on a rest day each week although some enumerators prefer to work continuously in order to finish sooner and return home. This choice is context-specific.
Per diems cover enumerator’s expenses associated with doing fieldwork. This means lodging for overnight stays, all meals, and sometimes also transportation. Per diems should also be paid on rest days that fall in between work days. In the case that the variation in lodging and food costs is low, it is not important to change the per diem rate according to location. Teams will know when to save and when to spend.
((survey time to complete * sample size)/workable hours in a day)/# of enumerators = number of days
number of days * (daily rate + per diem) + supervisors = approx. total salary cost
It’s important to, ex ante, be as accurate as possible in estimating the full cost of transportation as this is frequently both least flexible and most variable cost. Typically, it is good practice to build in contingency on the cost of fuel, as the price can change over the several months it takes to go from the grant application stage to the implementation stage. If you are budgeting before drawing your sample, pay particular attention to hard-to-reach areas in your population (islands, places without road access) and pad your transport line for the possibility you randomly sample enumeration areas that carry these higher costs.
Later on in this guide we present the benefits of using personal digital assistants (PDAs) or tablets for data collection (see section 3). PDAs/tablets can be either purchased using survey funds or leased from a research firm, university, or other researchers.
Use of a PDA/tablet allows the collection of more accurate and detailed data (Goldstein, 2012) because of:
PDAs/tablets have lower error rates than paper-based surveys (Caeyers, 2010) and have superior quality control options including:
Parallel reporting chains can greatly improve the reliability of data by providing incentives to declare errors and mistakes. Under a parallel structure, oversight staff reports directly to survey manager, while surveying staff reports to field coordinator. The aim is to use the auditing and back checking reports in order to cleanly identify problems, and field supervision to then correct any issues.
Field teams are made up of enumerators and a team leader. Team leaders report to a field manager, or in a case of a large survey, a regional supervisor.
Enumeration teams and field management can quite easily deviate from important protocols—these deviations can range from replacing sampled households based on the ease of getting respondents to creating fake data. In many cases, cutting corners is not easy to detect and can save money and time for the enumerators, field managers, and even the survey research firm. PDAs/tablets can reduce the number of total possible types of fraud, but some level of field supervision is always necessary. A parallel reporting structure, with independent oversight, can help guard against these deviations.
On the oversight side, there are two types of checks that should be conducted– audits and backchecks:
Conducting both audits and backchecks means that for each individual survey there is some non-zero probability that the work will be checked in some way. In the case that there are only backchecks, teams will never be monitored in terms of their adherence to protocols as they sample and conduct interviews. In the case that there are only audits, if a team is not visited on a particular day of work there is no chance to check that they actually interviewed subjects and recorded their responses accurately.
Auditors and backcheckers must report directly to survey management. Imagine an example: Say a village is difficult to find and the team of enumerators chooses a replacement (rather than resampling by the PIs), and the auditors visit the sampled village and uncover it was not surveyed. If this error is communicated to someone also managing the enumerators, their best response is to cover this up or try to fix it without the PI knowing. This prevents having to admit a management mistake, and having to add a day of work to revisit the original or resampled village. If the survey team is notified directly, there is an opportunity to fix the mistake and make personnel changes as needed. Unmonitored communication between auditing teams and enumeration teams can result in a lot of unauthorized fixes and unexplainable data patterns.
If you are working alone, recruit experienced enumerators through contacts at survey firms, NGOs, or universities. It is important that enumerators are experienced, literate, educated, and able to build rapport with subjects. Hiring enumerators who are connected, in some way, with the survey leaders or local coordinator, e.g. through a youth organization or other social tie, can help immensely with oversight as the enumerators have bigger reputational costs if they shirk their duties.
The foremost requirement is that the enumerators speak the required local languages. We know that coethnicity between enumerators and subjects can reduce bias, so recruitment of coethnic interviewers, and balancing across the sample if using treatment and control groups, is important.
Having a team of mostly male enumerators interview a sample with equal numbers of men and women there can introduce response bias. For sensitive questions, such as questions on sexual behavior or violence, it is strongly recommended that women interview other women. If it is difficult to recruit experienced women enumerators, it usually makes sense to hold a special training for women candidates with less experience in order to ensure teams are balanced in the end.
There are several different and necessary phases of pre-testing:
Trainings establish consistent standards for data collection. If you are contracting a survey firm and are not on the ground yourself, training is the most important part of the process to personally attend. It’s a key moment to communicate quality standards, expectations, the intended meaning of each question, and teach important procedures that may be more technical than what the firm is used to, such as a list experiment. It is also a key moment to motivate the team, by communicating the project’s goals and importance. In order to ensure that the each member of the team is prepared to a certain standard, it is a good idea to test each team member at the end of the training period. There should be an expectation that some team members will be asked not to proceed any further with the project as a result of the test, which will emphasize the importance of taking training to heart and taking the test seriously.
Trainings are a key moment for assessment as well. If you train teams together it is easy to spot management issues and leadership capabilities. A good practice is to train teams together, and select team leaders at the end of training—this gives you a few days to gauge skills and also incentivizes trainees to perform during the training.
Trainings set the tone for the rest of fieldwork. Beyond communicating standards and expectations, this is also a key moment to create a culture of participation. Encouraging trainees to speak out about issues with the survey can show that you are open to feedback and increase the chances that they will report adverse events or challenges during the actual data collection.
Training usually takes several more days than you expect. See below for a rough guide to realistic training schedules.
List of documents needed for training:
Before being deployed to the field, each enumerator must:
Data from mock surveys must be individually assessed and feedback given to each enumerator. You can check whether certain enumerators are entering data differently than their peers, for example by entering lots of “Don’t know” or “Refuse” answers, finding low prevalence of sensitive behaviors, or entering data that is logically inconsistent. However, there is a lot that you can’t tell from the data alone. Spending a lot of time observing enumerators while they run surveys can greatly increase the quality of the data by improving their training, allowing you to select the best enumerators more accurately, and allowing you to understand how the questions are being implemented in the field.
To a large extent, the quality of the data collected is determined by the behavior the enumerators. Throughout the hiring, training, and field stages of the survey it is important to get the incentives right so that enumerators are motivated to do high quality work.
Expectations should be laid out clearly during training, in a manual, and reiterated in clearly worded contracts (signed after training).
Basic expectations of enumerators:
As much as possible, make payment dependent on delivery. Enumerators have less and less incentive to stick with the project towards the end of fieldwork. The marginal returns are lower and they may be concerned about finding new work. In order to offset this, it is good practice to withhold a portion of their total salary (+/- 30%) until the end of fieldwork, and sometimes until data has been thoroughly reviewed if using paper instruments that need to be entered manually. At the same time, enumerators are often living paycheck to paycheck and may have expenses to cover during their long absence in the field. It is important to pay an advance up front to allow enumerators to take care of personal expenses that may otherwise make them anxious and unhappy during fieldwork. Having a strong local manager who understands the enumerators financial situations can help you create incentives while still making sure that they perceive the compensation structure as fair and adequate.
Soft incentives help to keep teams happy and motivated throughout work. Some examples are:
Make route plans that set the order in which enumeration areas [EAs] will be covered, and then assign teams to routes. The basic idea is to minimize transportation costs while taking into account regional differences in languages or dialects that could mean that only some teams can work in certain areas.
A good way to visualize route plans is to mark sampled EAs on a map, using different colors to indicate different languages spoken in each EA. [SAMPLE PHOTO HERE]. Rely on team leaders to gauge travel time between sampled points and to suggest best routes.
Write code to clean data and check for errors, and run it on available data as soon as possible. This code should check for nonsensical responses or patterns, both within and across instruments (enumerator error).
If you are using tablets or PDAs, you should begin to run the cleaning and error script on the incoming data as soon as interviewing begins. If you are not using PDAs/tablets, you’ll need to develop a procedure for checking the paper instruments in the field (the job of the team leader), and also begin checking and cleaning of entered data as soon as possible.
After interviewing the team leader needs to review all instruments for completeness and accuracy. If there are missing data or other inconsistencies, the team leader should send the enumerator back to revisit the respondent to correct all problems before leaving the area.
Once instruments are collected, data entry should commence as soon as possible. All data should be entered twice, and any discrepancies should be checked by a supervisor against the paper instrument.
When using tablets or PDAs, checking the data is the responsibility of the RA and PIs. In addition to using a script that checks for patterns and outliers, it is also best practice to record selected portions of the interview and listen to a subsample of responses, both for errors and quality.