Limitations of online survey tools for developing countries

Before putting forward my arguments that should cause researchers from the developing world to tread carefully in choosing their data collection methods, I would like to remind the reader of what happened in 1936 in the United States, when Literacy Digest failed to accurately forecast the Presidential election result. It had been forecast that Alfred Landon would win by 57%, while Franklin D. Roosevelt was predicted to win by 43%. This meant that Alfred Landon was expected to become President of the United States. Literacy Digest had been correct in their previous predictions, but the 1936 forecast went wrong: Roosevelt won by 62%.

This is a very good example of a serious statistical problem known as using a biased sample, a problem that can happen if one's sample does not represent the population probed. This problem can occur either deliberately or accidentally. Literacy Digest included approximately 2.4 million people as their sample size. Meanwhile, George Gallup used a sample size of 50,000 people and actually predicted the results accurately. Predicting the outcomes is not just an issue of sample size only but also of minimising sample biases.

Recent technology

Recent advancements in information technology have provided access to good data collection tools such as Google forms, Survey Monkey, SurveyCTO, etc., which enable researchers to collect their data in a short time and cost-effectively. The data volume collected can be as large as one wishes, but one's sample may be biased and affected by significant sampling errors. Let us consider an example from the developing world.

Twenty-nine percent of Africans have Internet access (World Bank). This means that if one conducts a study using only online survey tools, one may get responses from a good number of people but the 29% of African people who are online may lack some characteristics pertaining to those people who do not have Internet access. This means that the study may be biased.

For some studies — such as those covering macro issues like electoral thinking — the method of data collection that ought to be applied should not only consist of online surveys. This type of survey may not cover an adequate, representative sample and thus have large sampling errors.

Because of low research budgets, some students have been using technology to collect their data, ignoring the fact that their data are not necessarily representative of the developing world's population.

Although 29% of Africans have Internet access, that Internet access is mostly available to people living in urban areas. That is, online surveys will reach fewer people from rural areas.

Effects of biased sampling

Biased sampling leads to obtaining wrong estimators, which may not predict the behaviour of the population. This reduces a researcher's credibility. Researchers are meant to be objective in conducting their studies, seeking to prove their hypotheses but not be biased to favour his or her own viewpoints. Such biases may lead to wrong policies, government failure and/or misallocation of resources.

Conducting unbiased studies means that the estimated coefficients are, on average, true. If an estimator is unbiased, then its probability distribution has an expected value equal to the parameter it is supposed to estimate.

Other issues with online surveys

While online surveys may lead to sampling biases because of the nature of the online population and the nature of the studies themselves, another important problem is known as careless responding. Ward and collaborators observed that respondents to online surveys do not always properly answer the questions when completing surveys, which leads to measurement errors. A further problem associated with online surveys is attrition. Attrition affects statistical estimates of the relationships between personality traits and performance.

All of these issues result in biased data sampling, which can lead to incorrect predictions of behaviourial traits.

How to avoid sampling biases

In some instances, random sampling and huge samples are used to avoid sampling biases. However, huge sample sizes do not remedy biased design. This is the reason behind the Literacy Digest's wrong prediction of the U.S. election results in 1936, despite their sample size of approximately 2.4 million people. After all, George Gallup used a sample size of just 50,000 people and predicted the results correctly.

However, you can still avoid sampling biases by employing the stratified random sampling technique. Surveys may be conducted by defining strata, such as online respondents and respondents in the general population. That is, if a researcher is planning to conduct a study to predict election results, online surveys can be used to target people who have Internet access, but there should be other means to survey people who cannot be reached by online surveys. The number of respondents in one's online survey has to equal the number of respondents in the general population survey.

This is not to say that studies cannot employ online surveys to collect their data, but this is a call for researchers in the developing world to know that some of their national studies may not be effective if only online surveys are used. This may affect studies covering elections, the influence of political parties, etc. Businesswise, marketers must know that their products are not only consumed by people with Internet access, which thus requires additional, ordinary ways of collecting data before introducing a new product.

It is well accepted that the advancement of technology has come with many good things, including simplicity in statistical data collection. However, that has to be taken with a degree of caution as the randomness of online data collection may give rise to huge sampling errors, thus resulting in outcomes that do not reflect the true population.

An estimator should, on average, be equal to the value of the parameter being estimated. And it should approach the value of the population parameter as the sample size becomes increasingly larger. However, in minimising the unbiased nature of one's survey, increasing the sample size will not do any good if the sample size is taken from a sample that does not represent all of the characteristics of the population of interest.

Conclusion

Any study is conducted with the aim to generate minimal criticism, and thus research must always be systematic in explaining a given phenomenon. Irrespective of any motivation that a researcher may have, research must be scientific such that the aim is not just to complete one's project but to have a good basis for one's conclusions on the behaviour of the aspects studied. This is therefore a call to developing nations' researchers who routinely use online surveys only in collecting their data to be watchful of errors which can arise from their data collection methods and employ the best use of surveys to avoid biases leading to incorrect predictions of the behaviours of interest.

Francis Nyonzo, is an economist and researcher. He currently works at JamiiForums. He consults post-graduate students on conducting research. He can be reached at francisnyonzo@gmail.com, and he is open to collaboration in the field of economic reseach.

Further reading

Caughey, D., Berinsky, A. J., Chatfield, S., Hartman, E., Schickler, E., & Sekhon, J. S. (2020). Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias. Cambridge UK: Cambridge University Press. link

Gujarati, D. N. (2008). Basic Econometrics. McGraw-Hill Education. pdf

Ward, M. K., Meade, A. W., Allred, C. M., Pappalardo, G., & Stoughton, J. W. (2017). Careless response and attrition as sources of bias in online survey assessments of personality traits and performance. Computers in Human Behavior, 76, 417–430. link

Wooldridge, J. M. (2015). Introductory econometrics: A modern approach. Cengage learning. pdf