This report marks the third iteration of our Medicare Shared Savings Program (MSSP) performance driver study. Our prior two studies focused on using machine learning to assess accountable care organization (ACO) characteristics that were associated with total savings¹ performance in the MSSP. In the initial study, we analyzed performance year (PY) 2015 results, which had a financial benchmark methodology that was similar to MSSP rules of today, but also distinctly different. At the time of the first analysis, Medicare ACO financial benchmarks were based 100% on their own experience and the benchmarks were trended solely based on the national reference population. We updated this study to analyze the PY 2019 results. This was the first performance year of the “Pathways to Success” MSSP rule. At that time, ACOs were measured against their regional benchmark and regional trends. In this third edition of our study, we analyze the PY 2022 total savings performance across the 482 participating MSSP ACOs.

While the MSSP rules in 2022 were consistent with those in 2019, our updated study aims to build upon our prior study by answering the following questions:

• Do the results of our current study reinforce the findings of the PY 2019 study now that there is a larger pool of ACOs under the Pathways to Success financial benchmark methodology?
• Do the key performance drivers change for ACOs further along in their agreement periods?

To better understand the second question, we expanded our study to not look just at ACO characteristics driving total savings in 2022, but to look also at ACO characteristics driving performance between the benchmark period and PY 2022. Consistent with our prior study, we used a machine learning algorithm called a “random forest” to identify which of the roughly 250 ACO characteristics are most strongly associated with financial outcomes, then performed more targeted analysis on the key characteristics noted by the algorithm. For purposes of this analysis, the biggest advantage of the random forest algorithm is that it can handle a large number of features, including features that are highly collinear, for instance, benchmark year (BY) 1 costs and BY2 costs. Based on the results of the random forest, we then dove deeper into specific aspects of the data to better understand these relationships.

Key findings

• Initial Starting Position, a metric that compares an ACO’s Benchmark Year 3 expenditures to its Historical Benchmark, position is the leading indicator of total savings in PY 2022. We explain the importance of this metric in our study but ultimately exclude it as a random forest feature to focus on the supporting components of this metric, including regional efficiency and risk score trends during the benchmark period.
• As in prior studies, ACO regional efficiency was a key predictor of total savings. However, this metric has a slightly negative correlation with ACO trend during the agreement period, thus suggesting it may be challenging for efficient providers to improve their total savings performance from their current position.
• The ACO’s risk score trends were highly predictive of total savings. However, increasing risk scores during the ACO’s benchmark period could be as predictive of total savings as risk score improvement during the agreement period.
• ACOs with higher proportions of specialists relative to primary care physicians (PCPs) tended to perform worse on total savings than ACOs with relatively fewer specialists and more PCPs.
• Among the utilization metrics evaluated, reductions in inpatient hospital and skilled nursing facility (SNF) utilization were identified as the most strongly associated with total savings.

The remainder of this paper explores these findings in more detail. In Appendix 1², we rank the top 20 ACO characteristics identified by each of the random forest models. The Data Sources and Methodology sections include detail on our approach to the analysis. Unless otherwise stated, values in graphs throughout the paper are observed values and not the values predicted by the random forest model.

Note: We did not find any published studies measuring the relationship of as broad a range of ACO characteristics to ACO total savings in a performance year after the implementation of Pathways to Success (2022 or beyond). We found a number of studies that focused on the correlation of select ACO characteristics or performance measures to total savings. Additionally, we found one study that focused on a broader range of ACO characteristics and their association with total savings; however, this study was conducted prior to Pathways to Success, and we would not expect its conclusions to align with our analysis, because of the different program designs.

Additionally, we recommend referencing this Medicare Payment Advisory Commission (MedPAC) article as a conceptual overview of the MSSP for a refresher on any program references made throughout this paper.

Primer on MSSP financial performance components

Before we get deeper into our results, we provide a primer on the MSSP financial performance components we often review to explain the total savings of ACOs. These components are directly derived from an ACO’s MSSP settlement. Total savings is simply the difference between the total benchmark expenditures and total expenditures for a given performance year (PY). In this study we divide total savings by the total benchmark expenditures to normalize savings performance across ACOs with different spending levels. We then typically deconstruct total savings into two subcomponents: agreement period relative improvement and initial starting position.

• Agreement period relative improvement (or simply “relative improvement”) explains how much of an ACO’s total savings is attributed to ACO performance during its current agreement period. Specifically, this is the difference between the ACO’s expenditure trend and its benchmark trend (i.e., the national/regional blended trend and the ACO risk ratio) between benchmark year (BY) 3 and the PY.
• Initial starting position quantifies the ACO’s total savings position as of the ACO’s BY3 expenditures, prior to the first PY. Specifically, this is the difference between an ACO’s regionally adjusted historical benchmark and its BY3 per capita expenditures, expressed as a percentage of the ACO’s per capita benchmark expenditures. It is based on the notion that the BY3 expenditures are the best representation of an ACO’s expenditure levels at the start of an agreement period. The initial starting position is a function of two additional subcomponents: benchmark tailwind and regional efficiency.
• Benchmark Tailwind measures the performance of the ACO during the three-year benchmark period relative to the benchmark trend. If an ACO’s BY3 expenditures are lower than its BY1 and BY2 expenditures after trending and risk-adjusting to BY3, then the ACO will have a “tailwind” going into the agreement period because its BY3 expenditures are lower than the three-year weighted average used to establish the historical benchmark. In the opposite direction, we would refer to it as a benchmark headwind.
• Regional efficiency is the ratio of the ACO’s per capita expenditures for a given year over the risk-adjusted per capita regional expenditures on the same risk basis as the ACO (i.e., multiplied by the ACO’s risk score). In BY3, this metric indicates the favorability of the regional adjustment applied to the ACO’s historical benchmark, where a lower regional efficiency indicates a more favorable regional adjustment.

As an example, an ACO with a total savings of 4% could be explained by a 3% initial starting position and a 1% relative improvement. This would indicate that 3% of the ACO’s performance is driven by the expenditure levels relative to the per capita benchmark expenditures at the beginning of the agreement period and the remaining 1% is driven by ACO’s trend performance relative to the benchmark trend during the agreement period. If the ACO has a regional benchmark adjustment of 1% of total per capita benchmark, then the remaining 2% of the initial starting position would be driven by benchmark tailwind.

What was the breakdown of MSSP PY 2022 total savings across the financial components defined in the previous section?

Figure 2 highlights the averages of each financial performance component across the MSSP ACOs for PY 2022. We see that an ACO’s initial starting position and the relative improvement both account for material portions of the ACO total savings.

During the initial stages of our analysis, we found that the Random Forest analysis identified initial starting position as the most predictive of an ACO’s total savings among all variables we included. Initial starting position had a correlation³ of approximately 52% with total savings. Ultimately, we decided to exclude initial starting position from the final version of our random forest model and instead chose to focus on other features that influence initial starting position. Additionally, we excluded relative improvement from the total savings random forest model, as it is too directly correlated with performance year costs.

Although an ACO’s initial starting position had a slight negative correlation with relative improvement, the data suggests that relative improvement during the agreement period is not strongly influenced by initial starting position. This can be seen in Figure 3, where 69% of ACOs in a positive initial starting position also have a positive relative improvement, compared with 75% of ACOs in a negative initial starting position. Overall, 70% of ACOs have been able to achieve lower risk-adjusted trends than their benchmark trend (a blend of national and regional reference populations).

The 70% of ACOs achieving a positive relative improvement along with averages noted in Figure 2 above, column D, could indicate that ACOs have been effective in lowering Medicare fee-for-service (FFS) population expenditure trend relative to prevailing regional and national trends. However, there are many confounding factors, which we describe below.

ACO regional efficiency was again identified as a key predictor of total savings, consistent with our prior study. However, this metric was slightly negatively correlated with ACOs’ trend performance during the agreement period.

Consistent with our prior study, the random forest found that an ACO’s performance relative to its region on a risk-adjusted per capita expenditure basis was strongly indicative of total savings performance. This may not come as a surprise, given that the Centers for Medicare and Medicaid Services (CMS) makes a favorable benchmark adjustment to ACOs with risk-adjusted per capita costs that are lower than their regional benchmarks.

To better understand this relationship between an ACO’s regional efficiency and total savings percentage, we ran a linear regression with BY3 regional efficiency as the independent variable and total savings percentage as the dependent variable. We ran separate regressions on two ACO cohorts: those ACOs with greater than 100% regional efficiency and those with less than 100% regional efficiency.

Figure 4 demonstrates the strong negative correlation between BY3 regional efficiency and total savings for ACOs with a regional efficiency less than 100%, which is intuitive because a better regional efficiency results in a higher benchmark adjustment. Figure 4 shows that, for ACOs with regional efficiency above 100%, this metric becomes far less useful in predicting total savings. This also makes sense as CMS dampens the regional benchmark adjustment significantly for ACOs with regional efficiency above 100%.

CMS announced there will be no negative regional adjustment for ACOs starting an agreement period in PY 2025. The intention of this change is to make the program more feasible for higher-cost populations (i.e., populations with regional efficiency over 100%). For ACOs serving these populations, their initial starting positions will become solely reliant on benchmark tailwind. Given the inverse relationship between regional efficiency and agreement period relative improvement, this may open more opportunities for ACOs with populations having a total cost of care above their regional benchmarks.

For ACOs with regional efficiency less than 100%, an ACO’s BY3 regional efficiency has a slightly negative correlation with agreement period relative improvement. Although, our analysis does not explain this outcome; however, ACOs with historically lower costs may have more difficulty keeping their expenditure trends lower than their regions, given the already attained efficiency of the ACO. This raises interesting policy questions such as: whether ACOs should be rewarded for attained efficiency in addition to improved efficiency? With the CMS goal of having 100% of Medicare beneficiaries under value-based care by 2031, one argument for continuing to reward attainted efficiency is to keep the incentive of having already efficient ACOs in the program to ensure efficiency is maintained over the long term.

Risk score improvement was identified as highly predictive for both total savings and relative improvement.

In some sense, the idea that increases in risk scores would be associated with greater total savings is obvious. If an ACO’s risk score increases between two time periods, then its benchmark is adjusted upward as well. However, in theory, the increase in benchmark should be offset by the healthcare expenditures burden of the higher-risk population. The findings of our analysis suggest that risk scores are an imperfect measure of expected expenditures, which could be due in part to changes in documentation patterns over time.

On the surface, risk scores do appear to be good predictors of expenditures for MSSP ACOs, with an average correlation of about 70% between ACO risk score and ACO per capita expenditures across the available years in the public use files (PUFs). However, there is a more notable disconnect when comparing trends for risk scores and expenditures. We found that there was a correlation of only about 12% between risk score trends and expenditure trends across a given time period (BY1 to BY3, BY2 to BY3, or BY3 to PY). Additionally, there was virtually no correlation between regional risk score trends and expenditure trends in the same periods (about 2%).

With risk scores and expenditures trending somewhat independently, it is not surprising to see risk scores creating (or negating) total savings for ACOs. We identified a consistent positive correlation (about 26% to 29%) between risk score trend and total savings, across each benchmark trend period (e.g., BY1 to BY3, BY3 to PY). It is interesting that risk score trends during the benchmark period would be as correlated to total savings as risk score trends during the performance period. Although the MSSP applies a cap of +3.0% on risk score growth between BY3 and the PY, there is no comparable risk score growth cap during the benchmark period, resulting in an outsized initial starting position for some ACOs but limited upside during the performance period. This dynamic could be beneficial for ACOs entering into new agreement periods after hitting their risk score caps in the previous agreement period.

Figure 6 shows the correlation between BY1 to BY3 risk score trends and total savings. The figure demonstrates that many of the extreme total savings performances can be linked to this benefit of having no risk score cap during the benchmark period.

There are a number of potential reasons for the disconnect between risk score and expenditure trends, but one likely driver is that the measurement period for risk scores is different from the expenditure measurement period (i.e., the Performance year). Because the CMS-Hierarchical Condition Category (HCC) model is a prospective risk score algorithm, the measurement period for risk scores is the calendar year (CY) prior to the given benchmark or performance year. ACOs starting a new agreement period in 2022 would have a BY3 to PY risk score trend reflecting CY2020 to CY2021 and an expenditure trend reflecting CY2021 to CY2022. During the pandemic, these trends could be very different, with utilization of services rebounding sharply between CY2020 and CY2021 and then stabilizing between CY2021 and CY2022. The MSSP risk score normalization process helped mitigate risk score fluctuation for ACOs during this time; however, the COVID-19 pandemic did create additional variation across ACOs.

ACOs with a higher proportion of specialists relative to primary care physicians (PCPs) tended to perform worse than other ACOs.

In our prior analysis, we considered several variables that measured per capita counts for different provider types participating in each ACO. That analysis found the number of specialists per 1,000 beneficiaries and the total number of specialists associated with an ACO were key predictors of total savings. In this year’s analysis, we slightly altered the provider type variables to be on a percentage of total providers basis (e.g., the percentage of the ACO’s total providers that are PCPs). We evaluated all available provider types in the PUFs: PCPs, specialists, nurse practitioners, physician assistants, and certified nurse specialists.

This change in methodology resulted in a similar, yet stronger, relationship between total savings and the provider type variables. The random forest algorithm identified two of these variables, PCP percentage and specialist percentage, within the top five most important variables for predicting total savings. In Figures 7 and 8, we summarized the PY 2022 total savings for ACOs by different percentage cohorts of PCPs and specialists composition.

PCP percentage had a moderate positive correlation (39%) with total savings, and specialist percentage had a moderate negative correlation (44%) with total savings. Figures 7 and 8 illustrate the increase in average and median total savings for ACOs with progressively higher PCP percentages and the inverse relationship with specialists. This finding indicates that ACOs with PCPs making up higher percentages of their participating providers tended to have higher overall total savings than those ACOs with lower percentages of PCPs. Conversely, ACOs with specialists making up a higher percentage of their participating providers tended to have lower overall total savings than those with lower percentages of specialists.

The random forest algorithm did not identify these metrics as important in predicting relative improvement (i.e., they were both outside of the top 20 most important features identified). Looking directly at the correlation, a PCP percentage is positively correlated (24%) with relative improvement and specialist percentage is negatively correlated (26%) with relative improvement. Because the random forest algorithm did not identify these metrics as important, and the correlations are lower relative to those seen between these metrics and total savings, the conclusions that can be drawn are less definitive. The fact that the relationship between these provider type variables is stronger for total savings than for relative improvement suggests that the ACOs with higher PCP percentages are generating their savings largely through a strong initial starting position.

It is difficult to determine with certainty why ACOs with higher proportions of specialists tend to have less favorable savings. Risk scores may not fully account for the costs of beneficiaries who are assigned to specialists through the MSSP algorithm, particularly if they are assigned due to an acute episode. Alternatively, the incentives may not be as strong for specialty groups to manage total cost of care, as shared savings is likely to comprise a lower portion of their total revenue.

Reductions in inpatient hospital and skilled nursing facility utilization were strongly associated with total savings.

Among the other features we added to the analysis this year were utilization trends from BY3 to the PY. Although utilization metrics are available within the PUF, they are only provided for the PY. Not surprisingly, for most service categories lower utilization trends were correlated with higher total savings and relative improvement. However, we found that BY3 to PY trends for inpatient hospitalizations and skilled nursing facility admissions were most predictive of ACO performance. In fact, they were the two highest-ranked variables in the random forest algorithm for relative improvement, while all other utilization trends were ranked outside the top 20.

In order to most clearly observe this relationship, it is helpful to bifurcate the data between ACOs starting before the COVID-19 pandemic (i.e., 2019 or 2020 agreement period start dates) and ACOs starting after the COVID-19 pandemic (i.e., 2022 agreement period start dates). Although the impact of COVID-19 on access to services had waned by 2022, utilization was still materially below pre-pandemic levels for many service categories. In Figures 9 and 10, we have plotted BY3 to PY inpatient hospitalization trends against relative improvement. Figure 9 is limited to ACOs starting before the COVID-19 pandemic, while Figure 10 is limited to ACOs starting after the COVID-19 pandemic. Although trends are materially lower in Figure 9, the relationship with relative improvement is strikingly similar in both charts.

We did observe that ACOs with more annual wellness visits (AWVs) tended to have greater improvement than other ACOs. The Random Forest did not identify AWVs as particularly strong predictors of either total savings or relative improvement, but we did find that trends for AWVs were negatively correlated with trends for inpatient hospitalizations (24%) and skilled nursing facility admissions (22%). Therefore, it is possible that AWVs had an indirect effect on an ACO’s relative improvement by driving a reduction in more expensive services, such as inpatient hospitalizations and skilled nursing facility admissions. It is also possible that AWVs are useful in improving risk scores by ensuring complete documentation of beneficiary conditions and maintaining assignment of low-cost beneficiaries, as discussed in the June 2019 MedPAC report to Congress.⁴

Other notable ACO characteristics

Many other ACO characteristics were included in our modeling but were not found to be strongly associated with total savings or relative improvement. A key point is that some of these characteristics were correlated with total savings and/or relative improvement but, when considered alongside all other features in our analysis, they did not significantly improve the predictive accuracy of the model. This indicates that other variables do a better job of predicting total savings and/or relative improvement.

Two features that may be considerations for ACOs are track selection and revenue status (high revenue vs. low revenue).

Track selection

ACOs in two-sided tracks tended to have higher total savings. At the far ends of the spectrum, ACOs in the Enhanced Track had an average total savings of approximately 5.1% compared to approximately 2.4% for ACOs in Tracks A and B. Upon further inspection, however, it appears that ACOs may be selecting higher-risk tracks because they expect to be in a favorable starting position (similarly, ACOs with poor starting positions may drop out when they are forced into a two-sided track). Only 21% of ACOs in a one-sided risk agreement had a starting position of +3.0% or greater, compared with 42% of ACOs in a two-sided risk agreement. For each track, the average relative improvement ranged from 0.8% to 2.1%, indicating that the tracks had similar cost trends within an agreement period.

Revenue status

On average, low-revenue ACOs had materially higher total savings (5.0%) than high-revenue ACOs (2.3%). However, this difference is largely explained by the PCP and specialist mixes, which were identified as a key driver of total savings. In low-revenue ACOs, specialists accounted for approximately 25% of participating providers, compared to approximately 46% of participating providers for high-revenue ACOs. When comparing low-revenue and high-revenue ACOs with similar proportions of specialists, the average total savings was only slightly better for low-revenue ACOs.

Many other ACO characteristics were included in our modeling but were not found to be strongly associated with total savings or relative improvement.⁵ They can be grouped into the following categories:

• ACO organization structure, ownership, or size (14 variables)
• MSSP assignment methodology, parameter selections, or tenure (8 variables)
• Assigned beneficiary profile⁶ (67 variables)
• Most quality measures (23 variables)⁷
• Regional information (8 variables)
• Most utilization metrics not related to inpatient hospital or skilled nursing facility (56 variables)
• All others, including various information on risk scores, cost, provider types, and participant mix (47 variables)

Closing thoughts

As in prior studies, our analysis identified regional efficiency as the most important ACO characteristic associated with ACO total savings. This finding has some interesting implications:

1. ACOs that have already attained some efficiencies are positioned well to earn shared savings under the MSSP.
2. The mechanics of the MSSP financial benchmark methodology strongly influence ACO total savings.
3. Regional efficiency is something providers could estimate prior to a performance year to give early indication of likelihood of success.

In addition to regional efficiency, increasing risk scores, and decreasing inpatient (IP) or skilled nursing facility (SNF) admissions were also associated with ACO total savings and relative improvement. ACOs could use this as support to improve coding accuracy and target managing utilization in these service categories. We also found that ACOs with more PCPs and fewer specialists tended to have higher total savings.

In summary, this analysis strengthens our findings from our prior analysis that total savings performance drivers include factors leading up to and during the performance year. These learnings can be used by ACOs to inform their future strategies.

Data sources

This analysis was based primarily on data from the 2022 MSSP public use file (PUF). We incorporated quality metric information from the 2022 MSSP ACO performance results, also made publicly available by CMS. We excluded any variables that were directly related to the performance year total savings calculation, such as shared savings amounts and performance year costs. Based on these two sources alone, we engineered more than 200 features.

We also added additional features using analysis of outside data sources:

• Utilization trends by service category and location (place of service) of beneficiary assignment were both derived from Milliman’s ACO Builder product. Using the Medicare 100% Research Identifiable Files, the Milliman ACO Builder product develops eligibility, expenditures, risk scores, and program-specific regional benchmarks for Medicare FFS beneficiaries nationwide. The product also runs the MSSP assignment methodology on every provider nationwide and calculates performance and supplemental metrics for each participant’s assigned population.
• Regional efficiency factors were based on comparisons of ACO costs to risk-adjusted regional cost data in the MSSP rebasing public use files. Risk-adjusted regional costs for each ACO were weighted based on the ACO’s mix of beneficiaries by county.

Methodology

We used these features to predict total savings percentage with a machine learning algorithm known as a random forest. A random forest averages the predictions from a large number of decision tree models developed from bootstrapped samples of the data. In our case, we used 10,000 decision trees. The R-squared of the models used for this analysis were 44.3% (predicting total savings) and 34.5% (predicting relative improvement).

Although the random forest does not produce coefficients or p-values, as we typically see in linear models, the random forest algorithm provides a useful measure of feature importance, which we utilized in this paper. The feature importance value represents the increase in error if the ACO characteristic was randomized (therefore rendering it useless). The relative magnitude of each number is more important than the actual number itself.

Limitations and qualifications

This information is intended to identify and rank ACO characteristics most associated with total savings percentage for ACOs participating in the MSSP for 2022. Note our conclusions are only appropriate within the scope of variables we chose to include in our study. This information may not be appropriate, and should not be used, for other purposes.

Milliman has developed certain models to estimate the values included in this report. The purpose of these models is to identify ACO characteristics associated with total savings under the MSSP. We have reviewed the models, including their inputs, calculations, and outputs, for consistency, reasonableness, and appropriateness to the intended purpose and in compliance with generally accepted actuarial practice and relevant actuarial standards of practice (ASOPs). The models, including all input, calculations, and output, may not be appropriate for any other purpose. Where we relied on models developed by others, we have made a reasonable effort to understand the intended purpose, general operation, dependencies, and sensitivities of those models.

The models rely on data and information as input to the models. We have relied upon certain data and information provided by CMS for this purpose and accepted it without audit. To the extent that the data and information provided is not accurate, or is not complete, the values provided in this report may likewise be inaccurate or incomplete.

Anders Larson, Cory Gusland, and Hayden Chromy are members of the American Academy of Actuaries, and they meet the qualification standards of the American Academy of Actuaries to render the actuarial opinions contained herein.

---

¹ Total savings as it is used in this study is defined in the next section.

² Appendix 1 is available for download here.

³ Throughout this paper, we use the term “correlation” to refer to the Pearson correlation coefficient, which measures the linear correlation between two sets of data. If a linear regression is applied to the data sets, the R-squared of that model is equal to the square of the Pearson correlation coefficient. As an example, a linear regression using initial starting position to predict total savings has an R-squared of approximately (52.4% x 52.4%) = 27.5%.

⁴ MedPAC (June 2019). Chapter 6: Assessing the Medicare Shared Savings Program’s Effect on Medicare Spending. Report to the Congress. Retrieved March 26, 2024, from https://www.medpac.gov/wp-content/uploads/import_data/scrape_files/docs/default-source/reports/jun19_ch6_medpac_reporttocongress_sec.pdf.

⁵ We defined “not strongly associated with total savings or relative improvement” as any variable that did have a relative importance value greater than 5%. For comparison, the highest relative importance values in our analysis was approximately 70%, and two variables had a relative importance value of at least 15%.

⁶ One exception is that ACOs with a high portion of their members assigned through an assisted living facility tended to have greater relative improvement.

⁷ Certain Consumer Assessment of Healthcare Providers and Systems (CAHPS) scores were modestly associated with relative improvement based on the random forest algorithm. However, the relationship appeared to be complex and not particularly intuitive; therefore, we did not discuss them in detail in this report. Those variables are not included in the count of variables listed here.