By A. Jess Williams.
So, you’re doing a systematic review? Dear God. Panic stations.
At some point, either you or your PI will say “hey, how about we start this off with a systematic review?” Sounds good right? Yep, very logically. But then you feel overwhelmed; how did you get to this point?! Fear not. The joy of a systematic review is that you can tackle it… systematically. Break it up and sort one bit at a time.
1) Define your research question (and build your search strategy around it)
Personally, I like to keep research questions are simple as possible. Just in case old anxiety brain likes to jump in and say, “wait didn’t you mean this by that term?” No. No I did not, so with a clear research question that doubt will be squashed.
Examining pet ownership as a self-harm protective factor in children.
Each of these highlighted words or phrases should be defined and then should act as the core search term which you build your variations around.
2) Look at previous systematic reviews for search terms, ask at library services, check how they run on all databases
It’s not uncommon to have overlapping search terms with at least one other publication. With a PRISMA checklist, you need at least one search strategy with the submission – so in theory you should be able to find previous searches for your terms. These can be a useful starting point to build up the derivatives for each search term. Don’t rely on them too much though, you might need to tweak or add something that a previous researcher overlooked.
Also, if you’re hosted by a university see whether their services include anything on systematic review searches (maybe public libraries do this too). This can be a simple way to learn how to keep your search the same despite using databases which format this search in different ways. You want to keep all those database searches the same.
3) Know your checklists and your quality assessment tools
So first off, both of these will be dependent on what type of review you’re doing. Find the tools which suit your needs. That checklist can be your bullet points for what needs to go in your protocol. Use it as a baseline for what you need to think about during your systematic review. If you use this beforehand, then you don’t need to panic when it comes to journal reviewing and being told you missed something, which can be horrifying to any researcher.
When it comes to quality assessment tools, can you find one which has been tried and tested before instead of making a new one? Keep in mind, that you are meant to edit the tool to what you’re assessing. For example, what do you need controlling for in your review? Are children who don’t self-harm your comparison group? Is it the children who do self-harm but don’t have pets? Is it the children where having a pet isn’t protective? Think about it before you get to extracting stages and write it down somewhere! Make that a rule so halfway through you don’t forget and start having the comparison group as something else.
4) Have a flexible, clear extraction form and don’t panic when you need to leave gaps
The dread of an extraction form is getting through X amount of papers and then realising you need to be extracting Y variable. Then you get the joy of going back through all those papers and checking for Y. Once you get through all that and then you realise you also need Z variable, face meets the table, and you go through the whole process again.
If you can ask someone for an extraction form which they’re happy for you to modify – do it. This will be a good starting point, there will be things you don’t need. But there will also be things you might not have thought of. At the extraction point, you want to make sure you’ve got as many details, you can always reduce these down later. So make sure you have room for all that information. Then trial it! Take 5 papers, see if it works. Do you need more variables? Do you need to split something up and how?
Variable: Pets
Cats: 1
Dogs: 0
Snake: 0
Lizards: 1
Is there something that consistently blank? Please don’t delete it. In my brain, I like things to be neat, tidy, organised. It’s unlikely the extraction sheet is going to have everything filled for each study, because surprise, surprise, studies are different. So don’t panic about gaps.
5) Find a buddy, and double down
If you can, double review and extract. Find someone who’s happy to do this leg work with you, it’ll take some time and effort. Once you have your final search, have someone else review a percentage or the full amount with you. To help the other person and again to solidify in your own mind, make a decision tree or diagram which clearly shows what papers need to have to get through to full-text screening and extraction.
I’d personally also recommend Rayyan, as a software to do title and abstract viewing with your second reviewer. You can input your search, put blind reviewing on, add the other reviewer’s email and both review at the same time. Having done this process without and with Rayyan, I’m very much on the side of do it with. Just makes things a lot easier. It will store your decisions and when you take blind off, you will be able to see included, excluded and conflicts between reviewers. (Only the primary reviewer will have this power!)
Review software Rayyan: https://rayyan.qcri.org/welcome
6) PREREGISTER, PREREGISTER, PREREGISTER
This might be already in your checklist, it is in PRISMA but preregister your study. Journals are going to ask for it. It’s good science. But it also gives you the chance to check that no one else is currently working on a similar review. If you’re anything like me, this will let you breathe easier.
On the topic of Open Science, I’ve seen more and more journals asking for the systematic reviews’ extracted data to be made public. For me (and other people in my office) this caused panic, not because you’ve done something wrong but because it means people can judge your work, which is scary. Particularly because you’re basing your study off other published work, the amount of conversations Office 304 have had about “what if I interpreted this wrong?” isn’t even funny. But preregistering is a nice way to dip your toes into the Open Science world.
Preregistering at Prospero: https://www.crd.york.ac.uk/prospero/
7) Have a systematic review mentor – someone who’s done it before and is willing to be the “can I ask you this dumb question?” person.
If you can find someone who’s done this all before and is happy to be you go-to, ask them to be. Systematic reviews can be a lengthy processes with surprising twists and turns, and it helps to have someone who’s done it before to give the reassuring “ah you’ve reached that point” chat.
A. Jess Williams (@ajesswilliams) is a PhD student within the Institute for Mental Health, University of Birmingham and the Self-Harm Research Group (SHRG), University of Nottingham. (a.williams.10@pgr.bham.ac.uk)
netecr.org 