Clinical validation of the Unesp-Botucatu acute pain scale in sheep undergoing orthopedic surgery
Nuno Emanuel de Oliveira Figueiredo da Silva, Pedro Henrique Esteves Trindade, Gustavo Venâncio da Silva, Flávia Augusta de Oliveira, Marilda Onghero Taffarel, Mayara Travalini de Lima, Rubia Mitalli Tomacheuski, Gustavo dos Santos Rosa, Ana Liz Garcia Alves
TL;DR
This study validated a pain assessment scale for sheep after orthopedic surgery, showing it is reliable and effective for clinical use.
Contribution
The study confirms the clinical validity and psychometric properties of the USAPS for orthopedic postoperative pain in sheep.
Findings
USAPS demonstrated high intra- and inter-observer reliability with ICC values ranging from 0.77 to 0.93.
The scale showed strong concurrent validity with a Spearman correlation of 0.80 compared to the Visual Analogue Scale.
USAPS had high specificity (100%) and acceptable sensitivity (71%), with a cut-off point of ≥4 out of 10.
Abstract
Unesp-Botucatu sheep acute pain scale (USAPS) was validated for assessing postoperative abdominal pain. We aimed to investigate the clinical applicability and test the psychometric properties of USAPS to assess postoperative pain in sheep submitted to orthopedic surgery. Twenty-three healthy sheep undergoing patellofemoral joint arthrotomy were video-recorded for three minutes before and after surgery, after postoperative analgesic rescue, and 24 hours post-surgery. Four evaluators, unaware of the recording time points, randomly assessed all videos twice at one-month intervals. Intra-observer reliability based on the Intraclass Correlation Coefficient (ICC) was very good for all evaluators (ICC: 0.82–0.93). Inter-observer reliability was very good for four of six pairs of evaluators (ICC: 0.84–0.9) and good for two (ICC: 0.77 and 0.80). Principal component analyses and confirmatory…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
Fig 1
Fig 2
Fig 3
Fig 4
Fig 5
Fig 6- —http://dx.doi.org/10.13039/501100001807Fundação de Amparo à Pesquisa do Estado de São Paulo
- —http://dx.doi.org/10.13039/501100003593Conselho Nacional de Desenvolvimento Científico e Tecnológico
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusculoskeletal pain and rehabilitation · Veterinary Practice and Education Studies · Veterinary Pharmacology and Anesthesia
Introduction
Recognizing and quantifying acute pain in sheep is challenging for veterinarians, farm workers, and researchers. Chilean sheep farmers recognize that castration and tail docking cause intense pain, however, 86–93% of them still do not use analgesics [1]. In Northern Ireland, 27% of sheep farmers did not use NSAIDs for pain relief [2].
Beyond use for food production, sheep are a relevant experimental model to investigate and treat human diseases [3]. In 2022, 15,909 sheep were used in European biomedical research [4] and their use has increased in Australia, New Zealand and the United States [5]. From 2014 to 2020 in Great Britain, sheep were used more frequently (1.8% of the total species) than pigs (0.4%) and cattle (0.2%) for research [6]. Mice, fish, rats and birds account for 92% of animals used in the European Union [4]. Sheep is a biomedical model for cardiovascular, orthopedic, reproduction, gene therapy, and neurodegenerative research to test new pharmacological agents and medical devices for clinical applications [5].
Sheep are a good model for translational orthopedic in vivo research in biomechanical (biomaterials and orthopedic implants), biochemical and histological processes of bone physiology because their weight, size, bone, joint structures and healing are similar to humans [7]. These analogies enable us to study numerous musculoskeletal pathologic conditions, such as the repair of fractures (23% of the studies were in sheep) and articular ligaments (19%), limb bone lengthening (9%), treatment of osteoarthrosis (8%) and osteoporosis (2%) and other injuries (39%) [8].
Creating valid species-specific scales to assess acute pain is essential for recognizing pain and determining the need for and effectiveness of analgesic treatment. Before the validation of the Unesp-Botucatu sheep acute pain scale (USAPS) [9] there were three main instruments available to assess acute pain in sheep. They were developed for 1) lambs submitted to orchiectomy and tail docking [10], 2) sheep implanted with ventricular assist devices [11] and 3) behavior evaluation protocol [12]. Recently, a systematic review [13], reported that only the USAPS [9] scored high for the strength of evidence of psychometric validation [13], according to the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) [14] in farm animals.
There are three scales to assess pain by using facial expression: 1) sheep with pododermatitis and mastitis (SPFES) [15], 2) Grimace scale in sheep after osteotomy [16] and the 3) Grimace scale in lambs after tail docking [17]. None of them met the COSMIN criteria to be classified as a high level of evidence and only SPFES showed a moderate level of evidence [18].
Although the psychometric properties of USAPS have been robustly validated to assess postoperative pain in sheep undergoing soft tissue surgery [9], USAPS still needs to be validated in other circumstances. Therefore, based on the use of sheep as a translational model, validation of USAPS for orthopedic surgery is demanded to ensure its versatility. In general, pre-existing tissue injuries leading to pain and greater surgical invasiveness degree are more frequent in animals submitted to orthopedic procedures compared to soft tissue surgery (e.g., laparotomy), requiring a more comprehensive analgesic strategy [19]. These differences may modify the expression of pain behaviors and the cut-off point.
This study aimed to investigate the clinical applicability and test the psychometric properties of USAPS to assess postoperative pain in sheep submitted to orthopedic surgery in addition to soft tissue (abdominal) pain. We hypothesized that USAPS shows a high strength of evidence of validation to assess acute pain after arthrotomy as reported for laparoscopy in sheep.
Materials and methods
This opportunistic study was incorporated into the main project, “Use of heterologous fibrin sealant and mesenchymal stem cells in knee disorders”, approved by the Ethical Committee for the Use of Animals in Research of the School of Veterinary Medicine and Animal Science - São Paulo State University, Botucatu, São Paulo, Brazil - under the protocol 0111/2019. This blinded, randomized, prospective, and horizontal study followed the COSMIN guidelines [14], the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) [20], that grades the quality of evidence and the strength of its recommendations, checklist and terminology guidelines (S1 Table) to ensure methodological quality, and the Animal Research Reporting In Vivo Experiments (ARRIVE 2.0) [21] (S2 Table), adapted to the experimental design.
Animals
We used 23 female Santa Ines sheep (Ovis aries) aged between 6 and 12 months (7.6 ± 2.2) and weighing between 29 and 37 kg (31.1 ± 6.90 kg). After deworming and vaccination on the University farm, the animals were allowed to adapt to the experimental environment for 14 days, where they were allocated in stalls (3.00 x 4.00 x 1.50 m - length x width x height) and kept in groups of four until the end of the experiment. The sheep had no previous knee injuries and underwent clinical examinations (mucosal colour, lymph nodes palpation, rectal temperature, rumen movements, heart, and respiratory rate), laboratory tests (complete blood count, biochemical and copro-parasitological exams), imaging tests (x-ray and ultrasound) and muscle biopsies to confirm their health. They were fasted for 24 hours on food and 12 hours on water.
Anesthetic and analgesic protocol
One hour before surgery, one of the jugular veins was catheterized, and the sheep received 2.2 mg/kg of ceftiofur sodium (CEF 50^®^, Agener, São Paulo, Brazil) intramuscularly (IM) maintained daily for 10 days. Preanesthetic medication was carried out with 0.3 mg/kg of morphine (Dimorf^®^ 1%, Cristália, Itapira, SP, Brazil) and 0.2 mg/kg of midazolam (Dormonid^®^, Farmoquímica, Rio de Janeiro, RJ, Brazil) IM. Induction of anesthesia was performed with 3–5 mg/kg of propofol (Diprivan^®^, Aspen Pharma, Serra, ES, Brazil) until endotracheal intubation was possible. Inhalation anesthesia was delivered with isoflurane (Isoforine^®^, Cristália). Volume-controlled ventilation with a tidal volume of 10 ml/kg, inspired oxygen fraction of 40%, and inspiration/expiration ratio of 1:2 was used (Model 2800C Large Animal Anesthesia Ventilation System®). Peripheral hemoglobin oxygen saturation was monitored using a sensor positioned on the tongue, and 5 ml/kg/hour of lactate Ringer (RL physiological solution ^®^, Cristália) was infused intravenous (IV).
Sheep were subdivided into three echo-guided locoregional anesthesia techniques performed by the same anesthetist: 1 - lumbosacral epidural administration of 0.1mL/kg of bupivacaine (Neocaína^®^ 0.5%, Cristália); 2 – Infiltration between the popliteal artery and capsule of the knee (IPACK) combined with the adductor channel block of the saphenous nerve branch (0.3mL/kg of 0.5% bupivacaine) or 3 – adductor channel block of the sciatic and saphenous nerve branch (0.3mL/kg of 0.5% bupivacaine). Heart rate and rhythm were assessed by electrocardiography, invasive blood pressure was measured into the femoral artery, and expired end-tidal isoflurane concentration was adjusted to maintain mean arterial blood pressure between 60–100 mmHg (Datex Ohmeda Cardiopac 5®). Depth of anesthesia was assured by immobility, absence of palpebral reflexes, mandibular tone, and spontaneous respiratory efforts during mechanical ventilation. When the heart rate or mean arterial pressure rose above 20% of the values registered before surgery, transoperative analgesia was performed with 2.5 µg/kg of fentanyl (Fentanest^®^, Cristália) IV.
Knee arthrotomy with meniscus injury lasted around 1.5–2 hours. An arthrotomy of the left patellofemoral joint was performed. The patella was removed laterally, exposing the chondral surface of the medial femoral condyle, the trochlea, and the medial meniscus. Two lesions were performed: one in the medial meniscus and another at the distal end of the quadriceps muscle. After surgery, no cast was used, and sheep movement was not restricted.
Data collection
Four animals were operated on each day. The procedures began at 09:00 am, and the last evaluation finished by 06:00 pm. The study was conducted from June to December 2019, and the average temperature and humidity of the environment varied between 12–28 ºC and 1–54%, respectively (https://pt.weatherspark.com). Geographical coordinates were: latitude 22º 54’ South; longitude 48º 29’ West and altitude 866 m (https://earth.google.com). An in-person researcher (NEOFS) filmed the animals for 3 minutes at each time point, with a digital camera (Gopro Hero5 Black®) positioned on a tripod. During this time, he scored USAPS. Recording time points were 24h before surgery without fasting (M1 – no pain), 2 h after surgery (M2 - pain), 30 minutes after analgesia (M3 - rescue), and 24h after surgery (M4 – 24h) (Fig 1). The postoperative analgesic intervention was conducted by the on-site evaluator, with 0.5 mg/kg of meloxicam (Maxicam^®^ 2%, Ourofino, Cravinhos, SP, Brazil) and 0.2 mg/kg morphine IV (Dimorf® 1%, Cristália), in sheep scoring ≥ 3 of 10 in USAPS [9]) 2h after surgery, immediately after recording M2 time point in the pen [this cut-off point was considered instead of ≥ 4 calculated for the soft tissue surgery validation study [9], because locomotion was not assessed]. The pain was reassessed after 30 minutes, and sheep that scored ≥ 3 of 10 in USAPS were treated with 25 mg/kg of metamizole (D500 Dipirona®, Zoetis, São Paulo, Brazil) IV. Then, after 30 minutes, if the pain score was still ≥ 3, sheep received a third analgesic rescue with 0.1 mg/kg of morphine IV and 1 mg/kg of ketamine (Cetamin®, Syntec; Santana de Parnaíba, SP, Brazil) IV and the pain was reassessed 15 minutes later. After the end of the experimental period, sheep were treated with meloxicam (0.5mg/kg) and metamizole (25 mg/kg) both IV for five to seven days. In case the USAPS pain score was ≥ 3, 0.1 mg/kg of morphine IV and 20 mg/kg of oral gabapentin were administered and repeated whenever necessary every 8 hours.
Timeline flowchart used for USAPS in the perioperative period of orthopedic surgery.USAPS: Unesp-Botucatu sheep acute pain scale.
Selection and training of evaluators
All evaluators were veterinary anesthesiologists with at least five years of experience evaluating pain scales in animals (FAO, MOF, MTL, and RMT). They were trained with videos demonstrating each behavior (www.animalpain.org) and were instructed on how to fill in the scale (Table 1). Next, they evaluated 10 videos of different pain intensities and after a week, the observers scored the same randomized videos. Because the intra-observer reliability was ≥ 80%, the evaluators were enrolled in the main study evaluation.
Table 1: Unesp-Botucatu sheep acute pain scale (USAPS) updated for orthopedic surgery excluding the locomotion item.
Pain assessment and analgesic rescue
Four evaluators, unaware of the recording time points, watched and scored 86 [23 sheep in four (or 3) time points] randomized (for time points and animals) unedited videos of around three minutes. The videos’ order was randomized using the Microsoft Office Excel function = RANDOMBETWEEN (1;208). The evaluators scored about 21–22 videos weekly for one month (1^st^ phase). After a one-month interval, the evaluators rewatched the same videos in a new random order (2^nd^ phase). After observing each video, they answered if they would administer rescue analgesia (0 = no and 1 = yes), based on their clinical experience. Then, they assessed in half of the videos first the USAPS followed by the Visual Analogue Scale (VAS) and in the other half first VAS and then USAPS in a randomized order (VAS; a straight line 100 mm long, where “0” represents the animal without pain and “100” is the worst possible pain) [22].
Statistical analysis
To estimate sampling size with the power analyses, it was considered that 80% of the animals undergoing surgery would present a score ≥ 4 after surgery (at M2) based on the USAPS cut-off point and none of them would present pain scores ≥ 4 before surgery (at M1). According to the chi-square test, 11 animals were needed for an alpha of 0.01 and a power of 90%.
Two data scientists (PHET e GVS) coordinated the statistical analysis using R software in the RStudio integrated development environment (version 4.1.0; 29 June 2021) [23].
Table 2 contains a detailed statistical description of each statistical method used to test the psychometric property of the USAPS.
Table 2: Statistical methods used for psychometrical testing of Unesp-Botucatu sheep acute pain scale (USAPS).
Results
There was neither difference in the number of transoperative fentanyl doses nor post-operative analgesic rescues between the three groups of locoregional anesthesia.
Rescue analgesics were administered to 17 of 23 sheep after M2 video recording because their USAPS total score was ≥ 3. Therefore, only the data of these 17 sheep were used for statistical analysis at M3. A second analgesic rescue was administered to seven sheep 30 minutes later, and a third was performed on one sheep 30 minutes later.
Distribution of scores
The score ‘0’ was mainly present (80–99%) in all USAPS items before surgery (M1). In all items, scores ‘1’ and ‘2’ increased after surgery (M2), decreased after analgesic rescue (M3), and increased again 24h after surgery (M4; except for activity). The most frequent postures after surgery were ‘arched back’ and ‘moves tail’ (Fig 2).
Percentage of the USAPS scores for each item and frequency of each posture subitem. USAPS: Unesp-Botucatu sheep acute pain scale. Time points: M1 – 24 h preoperative; M2 – 2 h postoperative, before analgesic rescue; M3 – postoperative, 30 minutes after first analgesic rescue; M4 – 24 h postoperative; GM – grouped time points data (M1 + M2 + M3 + M4).
Multiple association and dimensional structure
Horn’s parallel analysis indicated the retention of the first from the five principal components generated by the principal component analysis. Interaction had a satisfactory loading value in the first principal component, and head position and activity were close (Table 3A; Fig 3). According to the confirmatory factor analysis, the comparative fit values index (0.96) and the Tucker-Lewis index (0.92) values, demonstrate the adequacy of the proposed unidimensional structure; however, the root mean square errors of approximation values were above 0.05 (Table 3B).
Table 3: (A) Loading values, eigenvalues, and variance of the USAPS items based on principal components analysis; (B) Confirmatory factor analysis estimates using all USAPS items in a unidimensional structure.
Biplot of the USAPS principal component analysis.Individuals have been placed along the first two principal components (PC) axes and colored according to the time point. Smaller circles indicate each assessment, while larger circles indicate the centroid of each time point and represent the gravitational center of all the vectors when a line is drawn from points of the same color, leading to its geometric center. Therefore, the centroids related to the time points of greatest pain (M2 – after surgery and M4 - 24 h) are positioned to the right, as well as the positioning of the vectors of each item, while the centroid of no pain is positioned to the left (M1- baseline), and the centroid of moderate pain (M3 - rescue) is close to the center of the figure.
Intra-observer reliability
The repeatability of the USAPS total score was very good (> 0.81; 0.73–0.95) for all evaluators and ranged from moderate to very good (0.42–0.93) for all items on the scale (Table 4).
Table 4: Intra-observer reliability of the USAPS (items, sub-items and total score), visual analogue scale and analgesic rescue indication in the perioperative period of sheep submitted to arthrotomy.
Inter-observer reliability
The reproducibility of the total USAPS score according to the matrix correlation was very good (ICC: 0.84–0.9) in all inter-observer comparisons, except for two, which were good (ICC of 0.77 and 0.80). Reproducibility ranged from reasonable to very good for all the USAPS items, except for posture, where comparisons between two observers were poor (Table 5).
Table 5: Inter-observer reliability of the USAPS (items, sub-items and total USAPS), visual analogue scale and analgesic rescue indication in the perioperative period of sheep submitted to arthrotomy.
Criterion validity
Spearman’s coefficient between the USAPS and VAS scores was 0.80 (ICC 0.77–0.83), confirming concurrent criterion validity (rs ≥ 0.80: very strong) [28].
Construct validity and responsiveness
Responsiveness was confirmed because the USAPS total score was significantly higher at M2 when the most intense pain was expected after surgery, and M4 at 24h when analgesia was losing its efficacy, followed by M3 when analgesia was provided and then at M1 when sheep were pain-free (Table 6; Fig 4A). Individually, all the USAPS items responded to pain because their scores increased at all times after surgery compared to the baseline value (M1), but only appetite improved after analgesic rescue (M3).
Table 6: Responsiveness of the USAPS, VAS and rescue analgesia indication. Time point, observer and phase effects.
Box plot of the scores (median/amplitude) of the USAPS according to the: (A) time point; (B) observers and (C) phases of evaluation.The top and bottom box lines represent the interquartile range (25 to 75%), the bold line within the box represents the median, the extremes of the whiskers represent the minimum and maximum values, the black diamonds (♦) represent the mean, the black circles (•) represent outliers. Different letters express statistical differences in the Bonferroni test between time points, where a > b > c > d. USAPS - Unesp-Botucatu sheep acute pain scale. Time points: M1 – 24 h preoperative; M2 – 2 h postoperative, before analgesic rescue; M3 – postoperative, 30 minutes after first analgesic rescue; M4 – 24 h postoperative.
The time point as expected influenced all items. There was an observer effect for posture and interaction items. The USAPS total score of observer 3 was lower than the other observers (Fig 4B). Phase did not affect the USAPS total score (Fig 4C) but did affect VAS scores. Considering the total USAPS score over time, there were no variations among observers (Fig 5A) and phases (Fig 5B), but there was variation for each animal (Fig 5C).
Smooth tendency lines, according to the LOESS method, indicate the scores of USAPS over time points for each evaluator (A), phase (B) and sheep (C).The shaded area corresponds to the standard error of the smooth lines. USAPS: Unesp-Botucatu sheep acute pain scale. Time points: M1 – 24 h preoperative; M2 – 2 h postoperative, before analgesic rescue; M3 – postoperative, 30 minutes after first analgesic rescue and M4 – 24 h postoperative.
Item-total correlation
The correlation coefficient of each item score with the total score excluding the target item ranged from 0.38 to 0.64, so except for ‘appetite’ (0.25), all items showed acceptable item-total correlation and indicated that they did not inflate the result and were not redundant for USAPS (Table 7).
Table 7: Item-total correlation and internal consistency of the USAPS.
Internal consistency
McDonald’s ω (0.75) and Cronbach’s α coefficients (0.84) were acceptable, indicating that the scale has excellent internal consistency (Table 7). Except for ‘interaction,’ the internal consistency was minimally acceptable (Cronbach’s α) or acceptable (McDonald’s ω) when each item was excluded. Considering that the internal consistency value decreased by excluding all items for α and in eight of 10 items for ω, they contributed significantly to the scale’s total score.
Specificity and sensitivity
Most of the USAPS items showed good to excellent specificity (83–99%). The interaction and activity items showed moderate or good sensitivity (82–87%), but the other three items were not sensitive (50–67%). The total USAPS score showed maximum specificity and moderate sensitivity (Table 8).
Table 8: Specificity and sensitivity and its 95% confidence interval of the USAPS.
Optimal cut-off point
The cut-off point for diagnosing pain and recommending analgesia determined by the ROC curve was 3.5 (3.5–3.5) for USAPS with or without appetite and 26 (21.5–27.5) for VAS. Therefore, the optimal cut-off point is ≥ 4 (out of 10). Based on the diagnostic uncertainty zone between 3.1 and 4.9, a USAPS total score of ≤ 3 indicates sheep without pain (true negatives) and ≥ 5 indicates sheep expressing pain (true positives). The cut-off point had 83% (80–86) of specificity or 86% (82–89), and 85% (81–90) of sensitivity or 81% (77–87) excluding appetite respectively. The area under the curve of 91% (89–93) or 90% (88–93) excluding appetite, indicates that the USAPS has a high discriminatory capacity (Fig 6) [26].
ROC curve and area under the curve (AUC). (A) and two-graph ROC curve with the diagnostic uncertainty zone of the USAPS (B). USAPS - Unesp-Botucatu sheep acute pain scale; ROC (receiver operating characteristic) curve with a 95% confidence interval (CI) calculated from 1,001 replications applied to estimate the diagnostic uncertainty zone of the cut-off point of all evaluators, according to the Youden index for the USAPS. Interpretation of AUC ≥ 0.90 - high discriminatory capacity [32,33]. The diagnostic uncertainty zone was 3.1 - 4.9; ≤ 3 indicates pain-free animals (true negative) and ≥ 5 indicates animals suffering pain (true positive). The Youden index was ≥ 4, representative of the cut-off point for the indication of analgesic rescue, even excluding appetite.
The percentage of animals within the diagnostic uncertainty zone ranged from 1% (in M1) to a maximum of 18% (in M3) (Table 9).
Table 9: Percentage of sheep present in the diagnostic uncertainty zone of each observer at each time point for USAPS.
Discussion
The Unesp-Botucatu sheep acute pain scale (USAPS) applied to an orthopedic pain model is a reliable and valid pain assessment instrument according to COSMIN guidelines, as demonstrated for abdominal surgery, confirming our hypothesis. The cut-off point for analgesic intervention is the same as the original study performed on abdominal pain [9], even excluding the locomotion and appetite items.
The USAPS items’ pain-altered scores (1 and 2) were proportionally distributed following the expected pain level for each perioperative time point (M2 > M4 > M3 > M1), indicating the importance of three levels of scores in each item. The exception was the restless behavior for activity item, which occurrence was low like in the original scale [9]. Restless behavior is also low in scales developed for other species, such as cats [34], cattle [35,36], and horses [37,38]. Although it is rare for a sheep to show more movement than usual or to lie down and get up frequently when experiencing pain, this behavior was classified among the five most important ones according to a previous study [39].
Opposite to expected, sheep submitted to soft tissue surgery had higher pain scores (8; 0–12) than after orthopedic surgeries (4; 0–10). This might be explained by the fact that the abdominal study [9] there simulated a field procedure using minimally invasive video laparoscopy in sheep undergoing dissociative anesthesia, opioid and a short-duration local block. In contrast, in the present study, surgery was more invasive and prolonged; arthrotomy was performed with a multimodal anesthetic and analgesic protocol, including either epidural or perineural block with bupivacaine. The anesthetic effect of the perineural administration of 0.5% bupivacaine lasts 110.0 ± 47.3 minutes [40] and the analgesic duration of epidural administration of 0.25% bupivacaine lasts 240 minutes [41]. In addition, the pain was still present 24h after surgery in the present study, when the effect of bupivacaine had been diminished, otherwise, the median USAPS total score was 0 at 24 h after abdominal surgery [9]. Interestingly, the posture item, moving tail and arched back were the most predominant behaviors after orthopedic surgery against extended head and neck and lying down with the head resting on the ground or close to the ground after abdominal surgery, highlighting the importance of testing the pain assessment instrument with different pain models. Another reason for the lower postoperative scores in the present study than in the previous one [9] was that locomotion was not measured in sheep submitted to orthopedic surgery, therefore the maximal score was 10 instead of 12 [9].
Multiple association tests evaluated the main components of the scale that encompass a group of items within the same principal component (dimension) [24,25]. Although the appetite item was loaded into the second principal component, it was not retained. Therefore, the USAPS for orthopedic surgery showed a unidimensional structure, like previously seen for the first validation of USAPS and the scales developed in cattle [35], goats [42] pigs [43], piglets [44], rabbits [45] and donkeys [46]. From a mathematical point of view, a unidimensional structure allows the sum of all USAPS items into a singular USAPS total score with a singular cut-off point, which makes the pain scale easier to apply compared to multidimensional pain scales. Nevertheless, from a biological point of view, USAPS for orthopedic surgery is multifaceted because it includes sensory, motor, emotional, temporal, and physiological attributes.
Repeatability (intra-observer reliability) and reproducibility (inter-observer reliability) of USAPS for orthopedic surgery were very good and greater than the original scale [9], and a scale developed for lambs undergoing acute pain [10], and similar to a sheep locomotion scoring scale [47]. Concurrent criterion validity assesses the measurement efficiency of a scale by comparing it with a validated instrument. Because there is no other gold standard validated sheep pain scale other than USAPS [13], we used the same approach performed in the previous study by comparing USAPS with VAS. The USAPS correlated strongly with VAS [28] like for soft tissue surgery [9]. Another way of confirming concurrent criterion validity was the very good agreement between most evaluators. Predictive criterion validity was confirmed by sensitivity, as 71% of sheep would receive analgesia based on the cut-off point at the time point of most intense pain (M2).
Responsiveness confirmed the ability of USAPS to distinguish differences in pain scores over time points by detecting the effect of pain and analgesic intervention [26]. All USAPS items increased in response to pain, but their scores were little affected by analgesia because the pre and transoperative analgesic protocol, including bupivacaine, produced preventive analgesia after orthopedic surgery compared to abdominal one [9], where local anesthesia was performed with 1% lidocaine, a short-lasting local anesthetic. In fact, six sheep did not require analgesia after arthrotomy. Another explanation for the lower pain score after arthrotomy is the different anesthetic protocol (diazepam/ketamine and epidural anesthesia with 1% lidocaine for laparoscopy against a more complex multi-balanced anesthesia and analgesia protocol for arthrotomy, including one transoperative injection of fentanyl in seven sheep. Indeed, in the current study, pain scores were still high at 24h, like those after surgery, even considering that six sheep received two and one received three additional analgesic rescues. Although laparoscopy does cause acute inflammation like arthrotomy, when the effect of analgesia wore off, we hypothesized that the dysfunctional component triggered by meniscectomy added a more intense nociceptive stimulus at 24h.
Construct validity was confirmed by testing three hypotheses: 1. the postoperative pain scores were higher than the preoperative score (M1 > M2), 2. the scores decreased after analgesia (M2 > M3) and 3. the scores increased again by 24h when the effect of analgesia faded (M2 ≥ M4 > M3 > M1)], known-groups validity, internal relationships among items (internal consistency, item-total correlation and principal component analysis) and relationships to scores of other instruments (strong Spearman correlation between USAPS and VAS).
Most USAPS behaviors may increase when sheep experience pain and after sedation. However, both USAPS and VAS pain scores reduced 30 minutes after analgesia (M3) compared to M2 (after surgery), therefore sheep were apparently experiencing pain in M2. Otherwise, if USAPS measured sedation, it would be more likely that pain scores would remain high in M3 due to sedation (higher scores of interaction, locomotion, head position and activity). Seven of 17 sheep received metamizole, 30 minutes after the first analgesic rescue with meloxicam and morphine. According to the Wilcoxon test the median score of these seven sheep reduced significantly from 3 (0–7) after meloxicam and morphine to 2.5 (0–7) after metamizole (p = 0.002), showing that scores were responsive to a drug which does not affect sedation.
The USAPS posture and interaction items and total score were susceptible to an observer effect, as reported in caprine [42] and leporine [45], however, in the present study, the differences among observers were caused by only one observer. Phases did not affect USAPS, but affected VAS. The individual variability in pain scores was expected because the expression of pain is individual, as reported in bovine [36].
The item-total correlation analyses the homogeneity of the scale items. Since the item-total correlation ranged between 0.30 and 0.70, the items were not redundant when analyzed singularly, contributing to and correlating well with the USAPS total score [26]. The only exception was the appetite.
The scale had excellent internal consistency like for abdominal pain [9]. When each of the five items was excluded, the internal consistency decreased, indicating that all items were relevant [30]. The only exception was the McDonald’s ω coefficient for appetite. Cronbach’s α coefficient is the most used method for assessing internal consistency, however, it assumes that items have equal covariances. To overcome this limitation, McDonald’s ω coefficient [31], was applied, and in this case, internal consistency was acceptable (ω = 0.75).
All USAPS items, except appetite, were specific, and the USAPS total score showed maximum specificity (100%). Three of five USAPS items did not show sensitivity (< 70%): appetite, posture, and head position. Appetite and posture did not show sensitivity in the laparoscopy study either [9]. The scale identifies 100% of animals with no pain (before surgery; scores < 4), preventing unnecessary treatment [26]. However, the scale´s ability to identify animals experiencing pain was moderate (71%), which increases the possibility of an observer not providing analgesic intervention when necessary. One possible explanation for this is that the sensitivity calculation was performed two hours after surgery when the effect of multimodal analgesia was still present, thus preventing the animals from fully expressing pain by this time.
The ROC curve estimated the optimal cut-off point for diagnosing pain and recommended analgesia with good sensitivity and specificity. The AUC of 91% indicated the high diagnostic accuracy of USAPS in distinguishing sheep with or without pain [33].
The cut-off points for indication of analgesia for VAS (≥ 26 of 100) and USAPS (≥ 4 of 10), even without locomotion, were the same as reported for abdominal surgery (≥ 4 of 12) [9]. Appetite item exclusion did not affect the cut-off point. Therefore, this is extra evidence that appetite could be removed without further impairment in the pain diagnosis.
The appetite item was not within the acceptable values for item-total correlation, internal consistency (McDonald’s ω), sensitivity, and specificity, as in the previous study [9] and considering that the cut-off point was not affected by the exclusion of appetite in the previous study [9] and in the current one, it may be excluded from USAPS.
The selection and training of observers are important to ensure reliability in assessing pain [27,48]. Reliability improved when experienced observers assessed pain in horses and piglets compared to veterinary students [49] or less experienced observers respectively [50]. Students tend to underestimate pain in bulls, cats, pigs and sheep compared to an expert [51] and training improved pain recognition in laboratory animals [48,52]. The observers of the current study had previous experience in assessing pain scales in other species. Two of them (FAO and MOF) participated in the original study [9] and all became familiar with the USAPS behaviors by training on the https://animalpain.org/en website (Table 1). They all had intra and inter-rater reliability ≥ 80% after a pretraining phase before scoring USAPS in the main study, which may have favoured the reliability results. Therefore, future research should compare trained versus untrained observers in scoring USAPS.
In the current study, the videos were sent to the observers weekly for a gradual evaluation, to minimize fatigue [45]. USAPS and VAS were assessed in a random order. There are pros and cons of this approach. In our previous studies, the VAS was assessed first to prevent the expectation bias of the behavioral scales from influencing the VAS [9]. Even with this approach, for all observers VAS intra, and inter-rater reliabilities were worse than USAPS, and there was a phase-effect in responsiveness, reinforcing that the behavioral scale is more reliable for assessing pain than VAS, like in cats [53], horses [38], donkeys [46] and rabbits [45].
We demonstrated that implementing statistical weighting based on machine learning algorithms can rank the importance of each USAPS pain-related behavior to identify sheep requiring or not analgesia and improve the diagnostic capacity based on the weighted USAPS [39] compared to the original one [9]. Pain-specific behaviors occurred almost exclusively at the time of greatest pain (M2) and less throughout the other post-operative period. Otherwise, as expected, maintenance behaviors, like activity, locomotion, and interaction, reduced after surgery (M2) [9]. When the USAPS pain-related behaviors were weighted [39], maintenance behaviors were more important for pain diagnosis than the increase in the frequency of specific pain-related behaviors.
The current study has some limitations. There was not another gold standard pain scale to compare against USAPS and better assess concurrent criterion validity. As in the previous study [9], only adult female sheep were assessed. Specifically in lambs, female lambs displayed more pain avoidance behaviors (sum of all) after tail docking than male lambs. In alpha rams, USAPS showed satisfactory sensitivity and specificity for diagnosing pain after orchiectomy [54].
Observers’ presence can inhibit pain expression, increasing the possibility of false negative results [55,56]. The real-time assessment had comparable results to video observation of the Unesp-Botucatu piglet [57] and cattle [58] acute pain scales. However, because remote camera visualization is not commonly available in routine clinical and field situations, we opted to maintain the researcher present to ensure that results are reproducible in a real-world scenario.
Although we did our best to blind the observers, the animals were lame in the left limb after the orthopedic surgery. Therefore, even considering the randomized order of all videos and animals, the observers might recognize the preoperative period videos, but not the other three postoperative time points.
In the future, it is important to evaluate the scale’s psychometric properties in other languages and cultures, like performed in cats [59] and cattle [35,36,60]. Another limitation of the study is the possible influence of the circadian cycle on sheep pain behavioral expression [61], as reported in horses [62], as they reduce activity at night. The current study occurred in the summer and daylight was present at all periods sheep were assessed. Therefore, other studies could test USAPS during nighttime.
Behavior changes caused by sedation and pain may be similar, therefore sedation may be a confounding factor when assessing pain in all species and may increase the chance of a false positive pain diagnosis. Although the residual effect of drugs used in the current study was minimal, considering their half-life and the late first postoperative measurement time, a future study should be performed, with a negative analgesic control group treated only with sedation, to test only the effect of sedation in USAPS.
Conclusion
The USAPS scale has a high strength of evidence following the COSMIN guidelines to diagnose pain in sheep undergoing arthrotomy. It has satisfactory psychometric properties and therefore can be applied for assessing pain in sheep models of orthopedic surgery. The cut-off point is just a guide to improve decision-making on whether to provide analgesia, however, the final judgment should be based on clinical experience and context, to ensure that sheep suffering pain would be treated accordingly.
Supporting information
Table S1Grading of Recommendations, Assessment, Development, and Evaluations (GRADE).(DOCX)
Table S2Animal Research Reporting In Vivo Experiments (ARRIVE 2.0).(PDF)
S1 AppendixDataset used for this study.(XLSX)
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Larrondo C, Bustamante H, Gallo C. Sheep farmers’ perception of welfare and pain associated with routine husbandry practices in Chile. Animals (Basel). 2018;8(12):225. doi: 10.3390/ani 8120225 30487400 PMC 6315487 · doi ↗ · pubmed ↗
- 2Crawford PE, Hamer K, Lovatt F, Behnke MC, Robinson PA. Improving analgesia provision for sheep: an analysis of farm medicine records and attitudes towards pain relief on sheep farms in Northern Ireland. Vet Rec Open. 2023;10(2):e 75. doi: 10.1002/vro 2.75 37876850 PMC 10591906 · doi ↗ · pubmed ↗
- 3Berset CM, Lanker U, Zeiter S. Survey on sheep usage in biomedical research. Animals (Basel). 2020;10(9):1528. doi: 10.3390/ani 10091528 32872575 PMC 7552153 · doi ↗ · pubmed ↗
- 4European Comission. Summary report on the statistics on the use of animals for scientific purposes in the Member States of the European Union and Norway in 2022. Brussels; 2022.
- 5Banstola A, Reynolds JNJ. The sheep as a large animal model for the investigation and treatment of human disorders. Biology (Basel). 2022;11(9):1251. doi: 10.3390/biology 11091251 36138730 PMC 9495394 · doi ↗ · pubmed ↗
- 6UK Government. Annual statistics of scientific procedures on living animals. Great Britain 2020; 2021. 30 p.
- 7Martini L, Fini M, Giavaresi G, Giardino R. Comparative medicine sheep model in orthopedic research: a literature review. Comp Med. 2001;51:292–9. 11924786 · pubmed ↗
- 8Potes JC, Da Costa RJ, Capela ESF, Relvas C, Cabrita AS, Simões JA. The sheep as an animal model in orthopaedic research. Exp Patho Health Sci. 2008;2:29–32.