Does experience matter? Understanding the mechanism of the volume-outcome relationship: Learning-by-doing or economies of scale

TL;DR

This study finds that experience, not just size, improves outcomes for sepsis patients in intensive care units.

Contribution

The study identifies learning-by-doing as the main driver of the volume-outcome relationship in sepsis care.

Findings

Lagged sepsis volume had a stronger association with reduced hospital mortality than contemporaneous volume.

Learning-by-doing effects were more significant than economies of scale in improving patient outcomes.

Results were consistent across different models of learning-by-doing.

Abstract

To evaluate the underlying mechanism of the volume-outcome relationship, namely learning-by-doing and scale economies in patients with sepsis.

Retrospective cohort study of adult patients with sepsis between 1 January 2010 and 31 December 2016 in 231 intensive care units (ICUs) in the UK.

The patient was the primary unit of analysis. Patient and ICU characteristics were included for risk adjustment. Demographic and clinical data were extracted from the Intensive Care National Audit and Research Centre (ICNARC) Case Mix Programme database.

We used the lags of quarterly sepsis volume in the ICU as a measure of the learning-by-doing effect.

The outcome of hospital mortality after ICU admission for sepsis was assessed using a multilevel probit regression model of patients nested in ICUs over quarters.

Critically ill patients with sepsis were identified by the Sepsis-3 consensus criteria.

Our study identified a cohort of 273001 patients with sepsis admitted to 231 ICUs in the UK. Our study finds that in comparison with contemporaneous volume, lagged volume had a stronger association with acute hospital mortality. This implies that the dynamic learning-by-doing effect is more important than the static economies of scale effect. This finding was consistent across alternate specifications of learning-by-doing.

The study provides evidence that the underlying mechanism for the volume-outcome relationship is learning-by-doing and not the static economies of scale. ICUs caring for patients with sepsis tend to improve by experience.

Figures

Economies of scale versus learning-by-doing.The blue arrow refers to movement along the Average Quality (AQ 1) curve reflecting the effect of economies of scale. The red arrow refers to changes in productivity by experience or learning by doing, resulting in a new Average Quality curve AQ-2. The distinction between static economies of scale and dynamic learning-by-doing is important because if the volume-outcome operated entirely through movement along the economies of scale curve (AQ-1), such as by investments in infrastructure and research and development, then equating static marginal quality to marginal volume would be socially optimal. Consider the example of a transitory shock that raises short-term demand such as a pandemic, assuming demand does not exceed a supply threshold. In such a scenario the economies of scale mechanism predicts no long-run gains to quality when volume returns to baseline. In contrast, the learning-by-doing mechanism would predict a permanent.

Equations

Full text

Introduction

The volume-outcome relationship has been a commonly invoked policy initiative aimed at improving the quality of healthcare. This inverse relationship between the caseload volume of patients treated and patient mortality has been described across many health settings and in many countries [1–3]. Despite the large body of literature demonstrating this favourable relationship, most studies have focused on differentiating the effects of selective referral and the true effects of volume. In comparison, there are few studies evaluating the underlying mechanism of the volume-outcome relationship, namely dynamic learning-by-doing or the static effect of economics of scale [3,4]. Resolving this tension between policies that allow providers to accrue experience over time and policies that promote centralisation of services would make a more compelling argument for policies such as minimum volume standards to be firmly established [5–10].

Conceptual framing of dynamic learning-by-doing versus the static scale effect

The term learning-by-doing was introduced by Arrow in 1962 to refer to institutional learning and describes the improvements in outcomes by experience [11]. The basic idea is that the team acquires experience by performing tasks repeatedly and that care improves because of the cumulative effect of skills gained by patients treated in the past. The knowledge is gained through the production of experience as activity increases and has been termed a learning-curve [11]. Learning-by-doing would result in increases in quality attributed to increases in knowledge. The acquisition of knowledge is a product of experience [11]. Even when volume remains unchanged between time periods, improvements in quality attributable to experience still occur. Learning takes place through activity and can be observed as improving performance so that today’s volume improves tomorrow’s outcomes.

The alternate mechanism of the volume-outcome relationship is the static scale effect. Here economies of scale are driven by indivisibilities of critical investments which impact patients’ outcomes and reflect contemporaneous gains provided through consolidation or centralisation. In the economies of scale mechanism, today’s caseload volume affects contemporaneous outcomes. The distinction between these mechanisms is described in Fig 1.

Economies of scale versus learning-by-doing.The blue arrow refers to movement along the Average Quality (AQ 1) curve reflecting the effect of economies of scale. The red arrow refers to changes in productivity by experience or learning by doing, resulting in a new Average Quality curve AQ-2. The distinction between static economies of scale and dynamic learning-by-doing is important because if the volume-outcome operated entirely through movement along the economies of scale curve (AQ-1), such as by investments in infrastructure and research and development, then equating static marginal quality to marginal volume would be socially optimal. Consider the example of a transitory shock that raises short-term demand such as a pandemic, assuming demand does not exceed a supply threshold. In such a scenario the economies of scale mechanism predicts no long-run gains to quality when volume returns to baseline. In contrast, the learning-by-doing mechanism would predict a permanent.

From a policy perspective, the underlying mechanism by which volume leads to positive patient outcomes matters. If the static volume is the underlying mechanism, then any ICU in which volume is concentrated will lead to better outcomes and centralizing care in a small number of large ICUs will lead to better patient outcomes. Alternatively, if experience is the underlying mechanism, then shifting volume from one ICU to another would reduce opportunities for learning in the ICU transferring patients out. This would make ICU consolidation a less beneficial policy.

The evidence supports two key mechanisms in the volume-outcome relationship: static economies of scale and learning-by-doing. Their prominence appears to vary depending on the type of patient cohort under study [9,11]. Existing literature has focused largely on coronary revascularization and elective surgeries, while non-surgical contexts remain underexplored. Most studies use mortality as an outcome to investigate the volume-outcome relationship and its underlying mechanisms. Studies of elective surgeries with low mortality rates tend to support static economies of scale, whereas research on more complex surgeries with higher mortality often attributes improvements to learning-by-doing. For example, in heart transplantation, successive cost reductions suggest that performance improvements are driven by accumulated experience rather than economies of scale [12]. Gaynor et al. examined the effects of scale economies and learning-by-doing in cardiac surgery, concluding that the benefits of volume were primarily due to static scale economies [9]. A major limitation of this work was the year lag structure used to detect a learning-by-doing effect meant that learning over shorter periods was not observable. Similarly, Ho et al. examined coronary angioplasty and found better outcomes with higher annual hospital volumes but no evidence of learning-by-doing [13]. Another U.S.-based study on cardiac procedures also found no learning effects, attributing all volume-related benefits to static scale economies [14]. A common critique of the literature lies in its reliance on cumulative volume for procedures like coronary angioplasty, a treatment already well-established in clinical practice. This approach makes it unclear when practitioners first gained exposure to the procedure, complicating efforts to isolate the effects of experience on outcomes [15]. Another critique is the use of mortality as an outcome for elective cardiac procedures when it is an uncommon event. In cohorts where mortality is more common, such as advanced cancer surgery patients, a substantial learning-by-doing effect has been identified, with lagged volume showing a stronger impact than contemporaneous volume [16]. This effect was particularly pronounced in patients with colon cancer compared to those with breast cancer, suggesting that experience provides greater benefits when managing diseases of higher complexity.

The two major limitations of the current literature on the mechanism of the volume-outcome relationship that we aim to address in this study can be summarised as follows [4]. First, almost all the evidence of the mechanism of the volume-outcome relationship is derived from elective surgical populations and predominantly cardiac and cancer-related surgery. In these surgical populations, there may be selective referral to higher-performing centres. In contrast to elective surgery, patients with sepsis are usually taken to their nearest ICU, removing selection related to ICU quality or severity of illness. Second, all of the previous studies use a fixed-effects approach to account for time-invariant institution-level unobserved heterogeneity [17]. The inclusion of an institutional fixed-effect means that the regression estimates the effects of changes to volume within the hospital rather than the effects of changes to cumulative volume across hospitals on mortality [17]. The fixed-effects specification requires an adequate number of institutions with significant variation in volume to detect a learning-by-doing effect. Previous studies that have failed to show a learning-by-doing effect may not have been suitably powered [17].

Sepsis patients are the ideal cohort to study the underlying mechanism of the volume-outcome relationship. Sepsis is a major public health concern and the leading cause of morbidity and mortality globally with 1 in 5 deaths worldwide attributed to sepsis [18]. The Seventeenth World Health Assembly recognised the importance of strong, functional health systems including access to intensive care services and health system organizational strategies to improve outcomes from sepsis [19]. One strategy to improve outcomes has been centralisation of care [7,20]. Centralisation is predicated on the assumption that the volume-outcome relationship operates through the static scale effect. If the volume-outcome relationship operates through the learning-by-doing mechanism, then patient outcomes would improve by experience, making system-wide centralisation unnecessary. The unsettled question of the underlying learning mechanism in the volume-outcome relationship therefore has clear implications to all stakeholders in the health system.

We make several contributions to the literature on the volume-outcome relationship. First, this study measures volume at a higher frequency than previously done (quarter instead of year). Given the temporal instability of ICU teams and of hospital teams generally, it is likely that learning might occur over shorter time periods and then undergo some decay as teams change. The quarterly time lags are more likely to detect learning over shorter periods than year lags. Second, our study includes many ICUs over several years and employs a mixed-effects probit regression model. This approach is more sensitive to detecting a learning-by-doing effect than previous fixed-effects approaches. Precise estimation using the fixed-effect approach requires data from enough institutions over a significant number of time periods to observe sufficient variation in volume. Previous studies have contained small sample sizes in terms of number of institutions which may therefore be underpowered to detect a learning-by-doing effect [17]. Third, we control for a rich set of patient and ICU characteristics to minimise the risk of omitted variable bias. Lastly, this paper ties the less commonly reported literature on the underlying mechanism of the volume-outcome relationship, namely economies of scale and learning-by-doing, with mortality and provides useful information on how reconfiguring service lines may improve underperforming lower volume ICUs.

Methods

Data

Data was extracted from the Intensive Care National Audit and Research Centre Case Mix Program database which is a clinical database that covers all adult ICUs in England, Wales, Northern Ireland, United Kingdom [21]. Trained data collectors extracted detailed physiological, diagnostic, and sociodemographic data from consecutive adults admitted to ICUs in the United Kingdom participating in the Case-Mix Program database between 1 January 2010 and 31 December 2016 [22]. Approval for the collection and use of patient identifiable data in the CMP was obtained under Section 251 of the National Health Service Act of 2006. The London School of Economics waived the requirement for approval and informed consent because this research involved secondary analysis of an established dataset of anonymised data. We report an observational cohort study, as per Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines [23].

Exposure

The exposure was defined as by the quarterly sepsis volume with contemporaneous quarterly volume being the measure of the static scale effects and the lagged quarterly volume identifying the learning-by-doing effect.

Study outcome

The primary outcome was death before discharge from an acute hospital. Patients who were transferred between ICUs were excluded from the analysis of mortality but included in the estimation of ICU volumes. This was done to avoid confounding results with outcomes from different ICUs. For patients who were readmitted to the ICU, only the first admission was included in the mortality analysis. We chose the unit of analysis to patient mortality. An ICU-level analysis of sepsis volume and mortality would smooth out variability in outcomes across patients [13]. In the case of no observed learning effect, it would be unclear as to whether this is due to data aggregation or a true absence of a learning effect.

Empirical strategy

Baseline characteristics and unadjusted outcomes for the cohort were tabulated using standard summary statistics. We used a multivariate hierarchical probit regression model to assess the association between volume and acute hospital mortality. Our model recognises that individual patients clustered in quarters and nested in ICUs and provides a consistent estimate of the standard errors for clustered data.

Patient-level covariates include age, gender, ethnicity, functional status, co-morbidities, and sociodemographic status as measured by the Index of Multiple Deprivation. We measure patient level severity of illness using the ICNARC_2018_ score. We include dummy variables for the presence of severe co-morbidities involving 7 organs systems. Functional status was categorised by the degree of assistance needed with activities of daily living. ICU characteristics are quarterly caseload volume, academic affiliation (non-university, university, university-affiliated) and quarterly throughput. Quarterly throughput is defined as the number of ICU admissions per ICU bed.

In the first step we will focus on overall learning curves in hospital mortality. We describe the basic model of Benkard with the important difference that the model presented here will involve multiple ICUs where the initial experience is unknown [24]. The simplest specification of the volume-outcome relationship is:

$$[eqn]$$

The dependant variable $[eqn]$ is patient-level mortality of patient i in ICU $[eqn]$ in quarter $[eqn]$ $[eqn]$ is a vector of patient characteristics and $[eqn]$ is a vector for ICU level characteristics. $[eqn]$ refers to quarterly ICU sepsis volume and captures the effects of static scale. The coefficient of $[eqn]$ is the sepsis volume in e preceding quarter and describes the ICU-level learning-by-doing effect. The $[eqn]$ captures to the ICU effect and $[eqn]$ is a classical error term.

We can expand the learning-by-doing component by including four lags of sepsis volume.

$$[eqn]$$

The individual weights of $[eqn]$ are called lag weights and they collectively constitute the lag distribution from $[eqn]$ , with the full set of quarters being $[eqn]$ The lags in volume estimate the effects of learning over time. If there was learning-by-doing and knowledge was passed on from one period to the next, we would observe a larger coefficient with each succeeding time period, i.e., $[eqn]$ . This is because learning-by-doing allows patients treated in the current time period to benefit from experience gained in the preceding time periods.

The ICU effect in the quarter $[eqn]$ makes it unnecessary to know the ICU’s entire production history. We separate out the volume-outcome effects into its static and dynamic components. Instead of using cumulative learning treating all past periods as the same, we use lags of the previous quarters’ volumes of sepsis. We compare the relative size of the coefficients. If the static scale economies is the main mechanism for the volume outcome relationship, then the coefficients of the lagged volumes would be small, i.e., ( $[eqn]$ If the learning-by-going is important then the coefficients on the lagged volume would be a larger proportion of the total effect. This would imply that experience gained in the past impacts the outcomes of the present. If the contemporaneous volume accounts for a larger proportion of the effect, then it would mean there are benefits to static scale. This would imply that any ICU high volume ICU would improve outcomes and that there would be benefits to indivisibilities of investments in infrastructure favouring consolidation of critical care services.

Quarterly volumes are correlated over time, meaning that $[eqn]$ is correlated with $[eqn]$ , as are $[eqn]$ and $[eqn]$ correlated as well as $[eqn]$ and $[eqn]$ . High level correlation between regressors, referred to as multicollinearity, leads to unreliable coefficient estimates with large variances and standard errors. This leads to lag distributions in which the sequence of lag coefficients bounces between large and small and even sometimes positive and negative. We describe the distribution of the correlation coefficients between the volume lags. A weaker correlation between volume lags would support a low risk of multicollinearity.

Sensitivity analysis

The main variable of interest is in capturing the learning mechanism. In the primary analysis we used quarterly lags to identify a linear learning-by-doing-effect. We undertook two sensitivity analyses to identify other specifications of the learning-by-doing mechanism. First, we used monthly sepsis volume to detect any learning that may occur over shorter time periods. Second, we specify quarterly volume a simple square root form to identify a non-linear learning-by-doing relationship.

Results

The study population of adult sepsis patients admitted to the ICU was 273,001. The median quarterly sepsis volume was 63 IQR [46–86]. Table 1 summarises the patient characterises across quartiles of quarterly caseload volumes from 2010 to 2016. The mean age of patients was 63 (95% CI 63-63) years. A minority of patients had a severe comorbidity (20.3%) and most patients were functionally independent (68.0%). The mean ICNARC_2018_ predicted risk of acute hospital mortality was 21.0% (95%CI 21.0b -21.0). Across quartiles of sepsis caseload volume, patients treated in the lowest quartile had higher acute severity of illness scores ICNARC score 21.3 [21.2-21.4] compared with 20.5 [20.5-20.6] in the highest quartile, p < 0.001. Lower sepsis volume ICUs treated more patients with no chronic comorbidities compared with higher volume ICUs where 81.3% in the lowest quartile had no comorbidities compared with 76.3% in the highest quartile. On average, higher volume ICUs operate at higher occupancy and have higher throughput of patients. The trend in monthly, quarterly and six-monthly volumes from 2010 to 2016 and the mean lagged volumes are described in S1 Table in the supporting information.

Table 1: Patient characteristics across quartiles of quarterly sepsis volume.

The distribution of the correlation coefficients between successive lags is described by the violin plots in S1 Fig in the Supporting information. The median correlation between successive lagged quarterly sepsis volumes ranged from 0.255 [IQR 0.056-0.467], to 0.267 [IQR 0.091-0.482]. These values suggest a weak to moderate correlation between lagged volumes and low risk of multicollinearity.

In Tables 2 and 3, we present the marginal effects and coefficients of the contemporaneous and lagged effects of volume on hospital mortality. We show that the lagged effect is significant in each of the models. The sum of the total effect represented by the F-test is more reliably estimated across models. The F-test for the combined effect of the lagged volume remains significant. The test of joint significance returned a p-value of 0.005, < 0.001, < 0.001 and 0.007 for the 1,2,3 and 4^th^ quarterly lag, respectively. In Table 3, we further explore the relative relationship between the contemporaneous and lagged volume. The ratio of the coefficient for the contemporaneous volume to the sum of all the volume coefficients was used to describe the relative importance of static scale economies. The contemporaneous volume accounts for 21%, 34%, 53% and 28% of the total effect of volume, for the 1,2,3, and 4^th^ lag respectively. The results suggest that the contemporaneous volume did not constitute the dominant mechanism for the volume outcome relationship. The ratio of successive lagged coefficients was used to identify the strength of the learning curve. A ratio of greater than 1 would suggest complete knowledge retention. The ratio between the first and second quarter was low, ranging from 3% to 14%. The ratio between the second and third quarter was 166% to 163% and the third and fourth quarters was 193%. This would suggest that learning takes two to three quarters with some degradation of learning.

Table 2: Probit regression results.

Table 3: Probit regression model showing coefficients, the proportion of the contemporaneous volume to the total volume efcts and the ratio of lagged coefficients.

Sensitivity analysis

The first sensitivity analysis using the months as the time epoch for the learning-by-doing effect does not suggest a shorter time window for learning. The tests of joint significance for 1,2,3 and 4-month lags were p = 0.119, p = 0.019, p = 0.022 and p = 0.003 respectively. The second sensitivity analysis explored the model fit of the non-linear specification of volume. This analysis found the primary analysis to have a better model fit by information criteria (Table 4).

Table 4: Sensitivity analysis of the Probit regression model showing coefficients with standard errors, the proportion of the contemporaneous volume to the total volume effects and the ratio of lagged coefficients with volume specified as monthly volume and by the square root of quarterly volume.

Discussion

This study found a significant learning-by-doing effect for patients with sepsis treated in the ICU between 2010 and 2016. This study contrasts with previous studies by Gaynor et al and Ho et al which focused on cardiac procedures [9,13]. Learning-by-doing differs across conditions and experience may be more significant for sepsis than for elective cardiac procedures. Sepsis afflicts a wide spectrum of patients with varied comorbidities and requires a variety of interventions. This is unlike cardiac surgical patients who have similar risk factor profiles and require one of two procedure valve and coronary artery surgery. Outcomes for routine procedures would therefore likely depend on scale effects more than experience

In this study, accumulated experience is de not be completely retained between time periods, reflecting the cyclical nature of emergency medical teams. As doctors and nurses leave the service, the ICU loses the benefits of their accumulated experience [25]. Data from other healthcare contexts are consistent with the idea that the depreciation of experience is related to the staff turnover [26]. The inconsistent coefficients for the learning-by-doing effect with regards to sepsis might relate to the depreciation of organisational learning and the low temporal stability of ICU teams. The depreciation of knowledge for complex medical treatments has important implications for patient care. It is important that ICUs maintain caseload volumes over time to preserve institutional knowledge.

This study has limitations. Firstly, one of the challenges of including both contemporaneous and lagged volumes is the likely multicollinearity. ICUs that treat many sepsis patients in one quarter are likely to treat a large number in the following quarter. We use data from 231 ICUS and assume that the large sample size will contain sufficient variation between ICUs, which would weaken the collinearity between the contemporaneous and lagged volume. We undertook a robustness check that found a weak to moderate correlation between lagged volumes and low risk of multicollinearity. Secondly, this study does not contain details about compliance with evidence-based processes of care. Previous literature suggests higher volume hospitals have higher adherence to processes of care such as antibiotic administration and venous thrombo-embolism prophylaxis than lower-volume hospitals [27]. The adherence to processes of care also does not fully explain the volume-outcome relationship and we therefore contend that conclusions of this study would not be substantially altered by controlling for it [4,28].

Conclusion and recommendations

This study finds that the underlying mechanism by which volume leads to improved outcomes is through learning-by-doing as opposed to the static scale effects of quality. ICUs improve by caring for a large volume of patients distributed over time, enabling continuous refinement of processes and skills. As populations age and medical care become increasingly complex, teams that can refine their skills over time will produce better outcomes. Centralization, while often proposed to enhance ICU quality, may fail to fully leverage the dynamic benefits of learning-by-doing and risks widening socioeconomic disparities in access to care [7,29]. The experience of the COVID-19 pandemic has highlighted how centralised models of care can struggle under the strain of unprecedented demand [30]. Considering the lessons from COVID-19, organizing ICUs to meet minimum volume standards without full centralization offers an optimal solution that supports equitable access while fostering continuous improvement. Policymakers must prioritize flexible and inclusive frameworks that prepare healthcare systems for future crises while maintaining high-quality critical care for all patients.

Strengths and limitations of this study

In terms of completeness, coverage and representativeness, this study includes 100% of all patients admitted to the ICU with sepsis in England Wales and Northern Ireland during the study period.A potential limitation is the collinearity of the contemporaneous and lagged volumes seen in other studies. This study addresses this by having a large sample size with sufficient variation between ICUs which weakens the collinearity between the contemporaneous and lagged volume.This study did not measure compliance with evidence-based processes of care. This unmeasured confounding could have a potential impact on the findings.Whilst the volume-outcome relationship has been described for a range of surgical conditions, less is known about its underlying mechanism. potential explanations are clinical experience and static economies scale.In this multicentre cohort study of 273001 patients across 231 ICUs with sepsis in the UK, the dynamic learning-by-doing effect was found to be more important than the static economies of scale effect.This finding suggests that ICUs caring for patients with sepsis, experience improved outcomes when caring for patients with sepsis. This argues against centralisation beyond meeting minimum volume standards.

Supporting information

S1 TableDescriptive statistics.Trends in monthly, quarterly and six-monthly volumes from 2010 to 2016. Mean lagged volumes, cumulative volumes and difference in volumes.(DOCX)

S1 FigCorrelation between successive lags described by violin plots (A) Volume qVolume q-1, (B) Volume q-1Volume q-2, (C) Volume q-2Volume q-3, and (D) Volume q-3Volume q-4.(DOCX)