On Monday, 4th February at 12pm (GMT) we had our first monthly online journal club session of 2019 and we discussed the paper “Machine learning in suicide science: Applications and ethics” by Ryn Linthicum and colleagues.
Here are some notes from thoughts shared in our discussion, kindly summarised by Ian Hussey.
Overview of paper
Machine Learning (ML) has potentential to increase predictive ability relative to traditional statistics, but like any type of statistics its usefulness is dependant on its careful and thoughtful use.
When should we use ML vs traditional statistics?
- For prediction based questions, when large datasets are available.
- What sample sizes are needed? This is a commonly asked question, but is hard to put a single number on, and many are reluctant to do so. As always, it depends on your research question and specific ML strategy. But, in general, think bigger than usual in terms of both participants and variables.
Can ML be used to isolate clustering of individuals?
- Yes, there’s a broad disitnction between clustering/dimension reduction and classification/prediction, i.e., between unsupervised and supervised machine learning.
What are the links between ML and traditional statistics?
- Technically speaking, several forms of unsupervised prediction ML techniques are based in traditional (generalised) linear model stats (e.g., multiple logistic regression). However, these strategies tend to augment them with additional variable selection methods (e.g., LASSO/ridge regression). Some other unsupervised/prediction ML approaches use substantially different underlying maths (e.g., random forest).
- ML, as a label for a more general set of analytic strategies, also generally takes greater care to employ safeguards against overfitting models relative to traditional statistics. For example, separation of datasets into model training and model testing is possible in both traditional and ML approaches, but is far more common in a ML framework.
- The Open Science and ML communities/movements therefore have an overlap in their efforts. ML and Open Science interrelate, and there is a broader narrative within the meta science ecosystem to be had about behaviour of scientists. This applies to ethics decisions too.
Can ML tell us which variables are predictive and which are not?
- In some casese, but many unsupervised ML approaches are geared towards whole model predictive value rather than telling us what the unique contribution of individual variables was. In the case of many of these ML models, the answer to the latter question is not as human-intelligible as is it for traditional (e.g., multiple regression) models.
Lessons to be learning from ML beyond the statistics
- The concepts of and appeal to ML is proving to be a powerful intervention to persuade researchers to improve their analytic practices. That is, psychology researchers have been resistant to improving our analytic practices despite the discussion around poor replicability, p hacking, etc. However, some researchers seem willing to engage with ML as a hot topic, despite some similar underlying messages being advanced by both efforts regarding not overfitting models.
- On the other hand, the popularity of ML risks inviting their unthinking use, as has happened with traditional statistics. It seems important that we don’t put unrealistic expectations on ML and then straw man the analytic approach when it doesn’t live up to unrealistic expectations.
- On both fronts, discussion of ML may also be of use in raising our collective statistical expectations and ambitions about what our fields standards are and should be.
- Less frequently cited benefits of engaging with ML approaches, beyond the maths involved, is that ML requires us to engage with data sources beyond what we are accostumed to. For example, to use and model data that is non psycohlogical, and non mentalistic, such as patient records, social media activity, and physcial movement data. This also serves to legitimise these types of data where they have historically been dismissed by psychological researchers as beyond the interests of our field.
Potential impact of ML methods on theory
- Internalistic, mentalisitic theorising is pervasive in suicide theorising, and interrelates with the types of data we tend to work with. Separately, there is a strong link between our methods of inference and our theories. Given that adoption of ML methods often broadens the type of data that is typically considered, this analytic innovation might induce or require theoretical innovation too.
- The surprising agreement between data science/complexity science researchers and critical suicidologists using exclusively qualitative data is encouraging.
Ethical issues with ML
- There are many emergent ethical and legal questions, which are likely best answered by interdiscplinary efforts.
- Notable that the voices of those with lived experience haven’t really been heard in this debate about algorithmic prediction and intervention decision making yet.
- Risk of pandora’s box research. History shows us that even failed lines of work show great momentum. For example, the literature showing that risk assessment scales are of very limited utility is often cited as a reason to engage in renewed efforts to create a good risk assessment scale, rather than a reason to disengage from the concept. Comparable risks abound ML.
- Efforts to precict behaviour have been around in suicidology for decades and went relatively unequestioned, due to its inefficacy. Perhaps new interest in the ethics of prediction is related to the perceived utility of ML (and in contrast to previous approaches), and if so maybe such fears are at least diagnositic of progress being made in predictive ability.
- Several express concern over the misuse of such algorithmic predictive methods, over one size fits all approaches, and particularly over their use by private, profit motivated corperation rather than public health bodies.