Creating a body of physical activity evidence to test the generalisation of annotation methods for automated evidence synthesis

TL;DR

Researchers tested if methods for analyzing smoking cessation reports could be used for physical activity reports, finding that most methods worked but some adjustments were needed.

Contribution

The study demonstrates the generalizability of annotation methods from smoking cessation to physical activity, adapting codes for behavior-specific outcomes.

Findings

Smoking cessation annotation methods were mostly transferable to physical activity reports.

Behavioral outcome codes required adaptation for physical activity interventions.

111 physical activity intervention reports were successfully annotated using the updated code set.

Abstract

The Human Behaviour-Change Project (HBCP) aims to improve evidence synthesis in behavioural science by compiling intervention reports, annotating them according to an ontology, and using the resulting data to train information extraction and prediction algorithms. The HBCP used smoking cessation as the first ‘proof of concept’ domain but intends to extend its methodology to other behaviours. The aims of this paper are to (i) assess the extent to which methods developed for annotating smoking cessation intervention reports were generalisable to a corpus of evidence relating to a different behaviour, namely physical activity, and (ii) describe the steps involved in developing this second HBCP corpus.

The development of the physical activity corpus took place in four stages: (i) reviewing the suitability of smoking cessation codes already used in the HBCP, (ii) defining the selection criteria and scope of the corpus, (iii) identifying and screening records for inclusion, and (iv) annotating intervention reports using a code set of 200+ entities from the Behaviour Change Intervention Ontology.

Stage 1 highlighted the need to modify the smoking cessation behavioural outcome codes for application to physical activity. One hundred physical activity intervention reports were reviewed, and 11 physical activity experts were consulted to inform the adapted code set. Stage 2 involved narrowing down the scope of the corpus to interventions targeting moderate-to-vigorous physical activity. In stage 3, 111 physical activity intervention reports were identified, which were then annotated in stage 4.

Smoking cessation annotation methods developed as part of the HBCP were mostly transferable to the physical activity domain. However, the codes applied to behavioural outcome variables required adaptations. This paper can help anyone interested in building a body of research to develop automated evidence synthesis methods in physical activity or for other behaviours.

The Human Behaviour-Change Project (HBCP) wants to make it easier to gather and analyse information about how to change people's behaviour. To achieve this, the project collects reports about behaviour change interventions, code them based on an ontology (that is, a classification scheme to organise and represent information within a specific area), and use that data to train computer programs to automatically extract information from reports and make predictions. The project started with smoking cessation as a proof of concept but plans to expand to other behaviours. This study investigates how well the methods used for coding smoking cessation intervention reports could be applied to a new behaviour, physical activity, and describes how this new set of data on physical activity intervention reports was created. Building the HBCP physical activity data set involved: (1) checking if the ontology codes used for smoking cessation would work for physical activity, (2) deciding what kind of physical activity reports to include, (3) finding the reports, and (4) coding the reports using the ontology. During step 1 researchers found that some changes were needed to the codes used for smoking cessation, so they analysed 100 physical activity intervention reports and got feedback from experts to update the codes. After stages 2-4, they ended up with 111 physical activity reports coded using the ontology of behaviour change interventions. In conclusion, the methods used for coding smoking cessation intervention reports could mostly be applied to reports about physical activity interventions, but changes were needed in relation to the target behaviour (for example, how behaviour is measured or whether researchers want people to start vs stop doing something). The aim of this report is to help others looking to build a data set to improve ways in which information on behaviour change interventions is gathered and analysed.

Figures