Tips for collecting data through an external organisation

By Jessica Leather

Data collection through web-based organisations has become increasingly common in academic research, due to the promise of large sample sizes and lightning-fast data delivery. Companies such as Qualtrics, Amazon Mechanical Turk and YouGov provide researchers with the opportunity to purchase or ‘crowdsource’ participants for online surveys and cognitive tasks. Some offer in-depth client services, including complex sample quotas, data analysis and transcription. However, there is little guidance for researchers about what to expect when working with a company, or how to ensure high quality data is collected.

Early in my PhD, I was told that there was funding available for me to conduct a large-scale online survey using an external organisation. It seemed like a perfect idea, because I was already struggling as a lone researcher to source a specific sample (UK healthcare professionals) in which to explore a sensitive topic (encountering patients who have self-harmed). Since I had previous experience with a similar company, I took up the offer without a second thought. In hindsight, I wish I had put more thought into my decision, and spent more time navigating drawbacks instead of settling for compromises. In light of this, I have written some tips for ECRs who are considering data collection through an external company.

#1 Research the company before contacting them

Corporate websites tend to be quite glossy and persuasive, but they are a worthwhile starting point for basic information. However, I recommend reaching out to your research network to ask for first-hand accounts about working with a company. This will give you a better idea from colleagues about what they are like to work with, and may help you to anticipate common problems. If possible, search for any raw data uploaded as an appendix to published articles, so you know what to expect in terms of data delivery.

I started by approaching a colleague who had worked with the same organisation, and he was kind enough to share the materials he used and a manuscript of his findings. This provided me with a template for how to structure my own survey, and helped me to create a timetable for the negotiation process leading up to data collection. He also gave me fair warning about some of the pitfalls I could watch out for in the drafting phase, which I will elaborate on later.

#2 Decide if this approach is right for your methodology

One regret I have is failing to consider any downsides until I was well into the process. I decided to use a mixed methods approach; this meant I had to make several compromises to my survey questions in order to stay under their time limit, and I was unable to conduct any follow-up interviews myself because of GDPR constraints. While there were challenges to both the qualitative and quantitative parts, I think external data collection is more suited to the detached nature of quantitative data.

I do feel in some ways that I missed some of the richness and flexibility that qualitative data can provide by outsourcing, so make sure you take the time to decide if this might compromise your data. However, a practical advantage is that transcription and copies of the audio files tend to be included in the costs of the project. For quantitative projects, consider what you want your data to look like. Some companies will present it in bespoke Excel templates, which can be much more challenging to analyse than a bog-standard spreadsheet: I strongly advise asking for a raw dataset to facilitate your own analysis, but their premade cross-tabulations may be useful for initial data eyeballing.

#3 Communicate clearly

Once you are ready to approach a company, consider the finer details: how many participants would you like, and of what characteristics, how long will your task take, and when do you need your data returned? Having these answers ready when discussing a potential project with a company makes it much easier for the staff to know if they can fulfil your requests, and provide you with an accurate quote for their services. Always make sure to check with your supervisors/line manager that the cost is feasible to avoid disappointments.

It can be all too easy to miss information in long email chains, which makes talking on the phone a valuable way of consolidating plans and clearing up any misunderstandings. For me, phone calls became milestones during negotiations, and were a great opportunity to discuss the project in more depth than could be afforded in emails. I found that the ‘discussion and planning’ stage took two to three months around all my other work obligations, but this was mainly due to relaying information between the organisation and my supervisory team. You might be surprised how rapid their turnaround estimates are for data collection (e.g. 1000 survey respondents in two weeks), which makes it even more important to stay on top of drafting deadlines to keep progress running smoothly.

#4 Draft your materials, but be prepared for them to change

You will likely need to submit any questionnaires, task materials or interview guides to your ethics board before sharing them with a third party, so spend time getting them ready in advance with your supervisors or colleagues. Make sure you check with your university or funding body whether ethical approval needs to be granted before you can raise a purchase order for the service. Once this is complete, the organisation will make their own edits and suggestions to your materials before starting a pilot.

In my experience, they emphasised streamlining and speed. They deleted similar questions and used processes called ‘logic’ or ‘routes’ to reveal certain items to respondents based on their answers to previous questions (for example, a section of my survey was only answerable by participants who responded ‘yes’ to having a previous encounter with a patient at risk of self-harm). This caused problems because I had included items from existing instruments, so I had to negotiate with them to retain the varying scales and measurements. Organisations often have an agreement to keep tasks under a strict time limit for their participants, so expect to compromise on the content of your materials to a degree.

#5 You are the client, so you are in control

Lastly, it is important to remember that you are a client in the relationship you build with the organisation. It is tempting to take all their advice on board in the name of swift progress and avoiding conflict. Nevertheless, it is worth the trouble of standing your ground about issues that might affect the quality of your data. For example, I was able to insist on some precise wording in my questions, and retained specific scales in spite of their efforts to make the survey more ‘user friendly’. Ultimately, differences of opinion are inevitable, but a little effort will help you to retain autonomy over your project. It is always good to remember that their role is to provide a service for you, not to take over your research! Always have the confidence to speak up if something is not going well, because it is much easier to fix problems before data collection starts than afterwards.

Overall, I would recommend external data collection to anyone who has a clear research design in mind for their project, provided they have weighed up the benefits and drawbacks first. As long as you are willing to be flexible with your materials and adhere to deadlines, your organisation of choice will generally cooperate with you. The process certainly tests your communication and project management skills, so it is a great experience to mention on a CV. On reflection, I am glad I collected my data this way because it has provided the foundation for the rest of my PhD, but I wish I had been more assertive about my requirements.

---

Jessica Leather (@JessZLeather) is a PhD student funded by the Greater Manchester Patient Safety Translational Research Centre (GM PSTRC), and the University of Manchester (jessica.leather@manchester.ac.uk).

 

*Featuring photo by Adeolu Eletu on Unsplash.