Methodology
Strengths and Limitations
Online surveys have both advantages and limitations when compared to more traditional approaches to asking people about their use of drugs, such as face-to-face researcher led interviews. As with any piece of epidemiological research when critically assessing the validity of the approach adopted, consideration needs to be given to both the questions asked (the research instrument), the method (on-line anonymous cross sectional surveys), the participants (who the respondents are), the recruitment strategy (how the participants were selected and how representative they are of the target population) and who is doing the asking (the research organisation). Such information can guide the reader in determining how useful the findings are.
Because so many media stories in recent years have been based on on-line polls of uncertain validity, Global Drug Survey wishes to be transparent about its methods, its strengths and its weaknesses. We hope that this will help media commentators such as Nigel Hawkes from Straight Statistics and Nate Silver of the New York Times, to make better sense of our findings. Global Drug Survey agrees with Nate Silver that “Providing more information to the reader is usually the right default”. We also wish to comply with a recent request in Straight Talking Statistics made by Nigel Hawkes ‘I suggest they should publish the methodology used in much greater detail, name the survey company used, acknowledge the sample wasn’t random, and provide the full questions and answers’. You can find a copy of the survey here.
Let us consider each of these issues in turn:
The questions asked
The research instrument. The research instrument used by Global Drug Survey (GDS) is a structured self-complete questionnaire, that following an initial drug use screen used in previous studies conducted by the group, tailors questions based on participants’ recent self-reported drug use. The questions, designed by a group of academically trained researchers and clinicians, offer a selection of fixed responses to each stem question. About 65% of the questions have been used in previous surveys conducted by the group to permit the monitoring of trends over time. Sections addressing specialist areas of interest such as prescription drug use, GP consultations and those attempting to define a particular clinical entity, such as urinary problems or abdominal pain associated with ketamine use, have been developed in consultations with experts in that field. The questions and the responses to our surveys have been deemed robust enough to support the publication of research papers in high quality academic journals (see below). The current study was approved by the joint South London and Maudsley NHS and Institute of Psychiatry Ethics Committee.
The method
Online anonymous cross sectional surveys. GDS conducts online anonymous self-complete surveys that tailor themselves automatically based on an individual’s responses to an initial drug use screen. Although limited by the self-nominating nature of the sample (see below), and fixed responses, the methodology adopted by GDS has a proven track record in being able to track drug trends over time and identify the use and harms associated with new drug classes such as the synthetic cathinones (e.g. mephedrone). Such methods, although considerably cheaper and quicker than face to face research interviews, do have their limitations, some specific to the method, others common to any approach enquiring about sensitive issues such as drug use. Common problems include recall bias (simply forgetting what you had done or taken) and response bias (providing the answers you think the research team/survey want to hear).
The participants
Some of these limitations will be due to the on-line method of data collection. As stated recently by Nigel Hawkes in reference to another on line poll, by definition those who respond are likely to be ‘more web-savvy and active online’, and so may not be representative of the wider population (although they may be typical of younger people in general, who have higher rates of drug use, the area of focus of the study). Like other approaches assessing drug use in the general population, such as randomly selected household surveys, online approaches may exclude some populations with high than average levels of drug use, such as the homeless or those in prison. It is also true that those with less access to the web because of geography or economic status may also be excluded. However, with increasing access to the web, this limitation is becoming less significant. Issues of health literacy will also need to be taken into account. We estimate that a reading age of 15 years would be needed to participate fully in the most recent Global Drug Survey. We are working to improve our surveys, to increase access to lower literacy populations.
Conversely, the use of anonymous online approaches to assessing drug use may be considered to have some significant advantages over more traditional approaches. People may be reluctant to tell a complete stranger at their doorstep or over the phone what drugs they have used recently, so that face-to-face studies may often result in under-reporting. By the time the results of most large national household surveys are released they are at least 12 months out of date. Our findings are 2 months old. Finally, because we attract groups with an interest in the use of drugs and alcohol we are able to access large numbers of people who are relevant to our area of study. In our most recent study, we collected 7700 UK responses in 4 weeks, making it almost 3 times larger than the sample of current drug users captured in the most recent British Crime Survey (where 10% of a representative sample of 27,000 people reported drug use in the last year).
Our recruitment (or sampling strategy)
We work with credible media partners to attract a population who have an interest in the use of drugs and alcohol. Participants are not paid to participate. They are not approached or vetted in any way. Our participants nominate themselves. Access comes through our media partners and increasingly through social media networks. But we are clear, our sample is not a random sample and cannot be said to be representative of the general population. It is hard, if not impossible to determine how similar our participants are to other people who came across the on-line survey and who did not choose to participate. Our participants almost certainly represent a group of people who have greater interest in, and use of, drugs and alcohol.
However, as previously stated, given that use of these substances is the focus of the study, such a bias can, for some questions, (such as comparing the harms associated between heavy and light users of a drug) be of benefit. The fact that the majority of our respondents were aged between 18–30 years demonstrates that the demographic of our participants matches the age profile of the largest group of drug users in the UK.
Ultimately of course, the only people that this study, (like so many others) can definitively tell you about are those who have participated. As previously stated, we accept that because of the non-random nature of the sample, our findings cannot be considered to be reflective of patterns in the wider population. However, unlike many selected non-random sampling methods, we do have very good data on our participants’ demographics, other lifestyle activities, their personality profile and overall wellbeing. This is important because the better you can describe your sample, the more of an idea you have of how they compare to the wider population.
From a statistical perspective, errors consequent upon non-random sampling strategies are not easily corrected by post-stratification (weighting) of responses, although by using complex statistical methods and by being able to define more accurately who your sample is, some of these limitations can be minimised.
However, even accepting that the findings cannot be said to be representative of the wider population, they do still provide a useful snapshot of what drugs are being used and how they are impacting upon peoples’ lives. The findings can and do inform policy, health service development and most importantly those who drink, smoke and/or take drugs. As cited earlier our methods have their limitations but have been considered robust enough to lead to scientific publications* in high profile academic journals. Some of these publications are listed at the end of this page.
The research organisation
Global Drug Survey is an independent self-funded data mapping exchange hub founded by Dr Adam R Winstock, a Consultant Addiction Psychiatrist and researcher based in London. His role in GDS is totally independent of his employment within the NHS and his views expressed as part of GDS have nothing to do with his employers or his academic affiliate organisation (beyond the fact that ethics approval was obtained through his local research ethics committee). Adam has an extensive track record as a researcher and educator. His full CV can be found at the bottom of this page. Adam collaborates widely in the development of the survey questions with a range of national and international experts. Many of these are also part of the GDS Expert Advisory Committee, which is made up of well-respected figures with experience in the fields of medicine, psychology, toxicology, epidemiology, statistics, and policy and health service development. GDS is totally independent and is not funded by any government to carry out its research. GDS determines the nature and scope of the questions. GDS provides only composite data reports to its media partners.
Recent publication based on/informed by output from Global Drug Surveys
*Winstock AR, Cottrell. Urinary problems in ketamine users. In press British Journal of Urology International.
Winstock AR, Mitcheson L Managing newer drugs of abuse –clinical review. In press British Medical Journal.
Winstock AR, Mitcheson L, Ramsey J, Marsden J. Mephedrone: use, subjective effects and health risks. Addiction Vol. 106, Issue 11, pages 1991–1996, November 2011.
*Winstock AR, Mitcheson L, De Luca P, Davey Z, Schiffano F ‘mephedrone – new kid on the block’ Addiction 2011 Jan Vol. 105 (10) pp 1685-7
Winstock AR, Marsden J, Mitcheson L What should be done about mephedrone British Medical Journal 2010;340:c1605
*Winstock AR, Mitcheson L Marsden J. Mephedrone, still available but twice the price. The Lancet, Vol. 376, Issue 9752, Page 1537, 6 November 2010
* Directly based on data from GDS on-line methods