ATLAS® displays signs of high-quality care and patient experience feedback for addiction treatment facilities. The quality measures used to inform ATLAS were selected by national experts and tested for reliability and validity. The data that inform these measures undergo multiple forms of validation.
Learn more about the data in ATLAS by clicking each section (+) below.
In healthcare, quality measures are used to evaluate how well treatment processes, structures, and outcomes align with the goal of delivering high-quality care. The information gathered through quality measures may be used to assess, compare, and improve treatment. Quality measures increase transparency and can be used to ensure there is accountability for delivering high-quality care. Additionally, the quality measures can be used for data-driven decisions on how to improve practices and ultimately support better patient health.
ATLAS™ provides information about the services that are available at addiction treatment facilities and whether the facility is using processes that align with what is recommended by experts. This information is displayed publicly on the ATLAS website to:
- Empower individuals to make informed decisions about care.
- Encourage and support data-driven quality improvement by addiction treatment providers.
- Increase factors and contexts that support the use and availability of research-backed practices for addiction treatment.
The approach used by Shatterproof to gather quality information, validate and analyze the data, and display it to the public builds on lessons learned from decades of health quality reporting. ATLAS uses best practices from existing quality comparison websites in healthcare and other industries and provides reliable and valid information in a user-friendly format.
How to Use Quality Data
The Signs of High-Quality Care section of facility profiles displays data reported by the treatment facility via the ATLAS Treatment Facility Survey and validated by Shatterproof. This is where you can find information on whether research-backed best practices for addiction treatment, described in Shatterproof’s National Principles of Care©, are used at the facility. The use and availability of best practices is indicated through specific icons, described below:
The service described in the Principle of Care is available at this facility.
The service is not available at this facility’s physical location but is available through another affiliated site or through a referral.
The service is not available at this facility.
The data are not available for this facility
The Patient Feedback section of facility profiles displays feedback from patients of the facility and their family members. Once 20 responses have been submitted the data is made publicly available. This minimum threshold aims to protect patient anonymity and prevent biased results. Open-text responses are also posted individually as narrative feedback and include the rating (0 – 10) the respondent selected in response to the question “Using any number from 0 to 10, where 0 is the worst treatment facility possible and 10 is the best treatment facility possible, what number would you use to rate this treatment facility?” as context. For multiple-choice questions, the frequency of each response is posted for the facility side-by-side with state averages. Patient feedback is retired after a period of 18 months to allow for the display of more recent evaluations.
In addition to the publicly available data, ATLAS professional users (providers, health insurers, and state agencies) will have access to insurance claims data in their password-protected professional portals when it is available in their state.
Shatterproof convened experts in the healthcare quality measurement and addiction treatment fields to identify the ATLAS quality measures. The measures were tested and refined through a collaborative, multi-step process that involved addiction treatment providers, health insurers, policymakers, and patient advocates. This process is described below:
- First, a Quality Measures Committee, composed of experts in the field, was assembled to identify existing quality measures and reporting mechanisms aligned with the Shatterproof Principles of Care©. Measures were prioritized based on potential impact and practicality of widespread use.
- Focus groups with addiction treatment providers across the country representing all levels of care and key informant interviews with health insurers were conducted to review and comment on the quality measures. Feedback was used to refine the measures to ensure feasibility and impact.
- The National Quality Forum (NQF) next facilitated an unbiased expert review of the measures. The NQF conducted an independent landscape review and held extensive interviews with 15 experts in addiction and convened an Expert Panel to make recommendations on the measures.
- To further inform the Expert Panel recommendations, NQF hosted a public comment period, during which over 250 comments were received from individuals across the country. Following review of these comments, the Expert Panel made final recommendations for measure concepts to be used in the pilot.
- The measure concepts were then turned into claims specifications and survey questions to be gathered through the ATLAS data sources. These survey questions and specifications underwent feasibility and pilot testing prior to ATLAS data collection, with feedback from more than 500 providers across the pilot states. This process included continual iterations with input requested routinely from various stakeholder groups.
For further information on how the ATLAS measure set was developed, and the current measures, please view this webinar.
To ensure the system continues to be meaningful and keeps up with the latest evidence and best practices in addiction treatment, measures will evolve as additional services are supported by a rigorous level of research and as existing measures are further refined or retired.
The Treatment Facility Survey (TFS) is an online survey administered at the facility level that gathers information on the processes, structures, and services. Facilities are asked to complete 12 sections of questions, which align with Shatterproof’s National Principles of Care.
The TFS examines quality concepts (e.g., access to treatment, use of evidence-based behavioral therapies, availability of medication for addiction treatment) that were identified through the quality measure development process. The questions were also informed by various sources, including Medicaid 1115 Substance Use Disorder Demonstration Waiver requirements, Joint Commission requirements, Centers for Medicare and Medicaid Services (CMS) reporting, National Survey of Substance Abuse Treatment Services (N-SSATS) items, and Treatment Episode Data Set (TEDS) items. A small number of survey items have not been designated as quality measures but were added due to relevance to the end user (e.g., question about national accreditation).
The Patient Experience Survey (PES) is a series of multiple-choice questions about two things: the use of treatment practices supported by research, and patient perceptions of care. Also included is one open-ended, or free-text, question. The PES is anonymously completed by patients of the addiction treatment facility or a loved one acting as a patient proxy.
Questions were derived from NQF-endorsed items in the Consumer Assessment of Healthcare Providers and Systems (CAHPS) program and Experience of Care and Health Outcomes (ECHO) Survey. These are developed by the government-run Agency for Healthcare Research and Quality (AHRQ) to assess patient experiences of care at hospitals and in other healthcare settings and are widely used to track the field, inform policy, and, in some cases, impact reimbursement. Prior to data collection, the PES approach was piloted at 50 facilities in New York state, with findings used to refine PES methodology before data collection (e.g., using letters to solicit patient feedback, adding two demographics questions).
PES responses are collected via “Leave a Review” buttons on facility profiles on the ATLAS website. PES responses are analyzed in near real-time and updated every 24 hours. Responses are anonymous but are subject to natural language processing and algorithmic reviews to screen for potential fraudulent responses.
PES data are posted in facility profiles once a 20-response threshold has been reached, since this is the number of responses needed for reliable data.
As part of the ATLAS pilot, Shatterproof tested an innovative approach to gather information on addiction treatment services from Medicaid and commercial health insurance claims data. Partner Medicaid agencies and commercial health plans submitted measures at the addiction treatment parent organization level. The measures were then analyzed by Shatterproof using a first of its kind approach. Given this novel methodology, these measures will not be publicly reported. The utility of these measures is being rigorously assessed, and findings will be used to refine and advance the use of claims-based measures in addiction treatment quality measurement.
The four claims-based measures being tested are:
- Opioid Use Disorder (OUD) Patients Receiving Medications for OUD
- Continuity of Medication Use for OUD among Patients with OUD
- Follow-up After Substance Use Disorder (SUD) Inpatient/Residential Treatment
- SUD-related Hospitalizations or Emergency Department Visits
Claims-Based Measures 1–3 are adapted from NQF-endorsed insurance plan level measures with a new attribution model for the provider parent organization. Measure 4 was recommended by the Expert Panel convened by NQF and was developed for testing in the ATLAS pilot.
The data collection period for the claims-based measures was from October 2019 to February 2020. Five states and eight commercial health insurers agreed to compute the claims-based measures using their data. Data contributors were asked to code the analytic specifications at the substance use disorder (SUD) provider parent organization level. This decision was made because some states do not have facility-specific identifiers (e.g., NPIs) to use when coding claims and, therefore, the claims cannot be attributed to specific physical locations. Following submission, six validation checks (e.g., relative value checks, equivalency checks, outlier checks) were conducted on the measures and additional summary data provided by the data contributors. Contributors were sent questions or requests for resubmission as needed. Claims-based measures are not available for all parent organizations.
Multiple forms of validation are used to ensure provider responses to the Treatment Facility Survey are accurate.
During Survey Completion
- Supporting Documentation: Facilities are required to upload documentation supporting their responses to certain questions. Examples include a request to upload the facility’s license or including a screenshot of an electronic medical record if it’s indicated that the facility utilizes this tool.
- Survey Logic: If conflicting responses are selected during survey completion, the survey logic will indicate an error. The survey completer will need to correct the conflicting responses before they can proceed with the survey.
After Survey Completion, Before Survey Submission
- Facility Attestation: Formal certification of the accuracy of submitted data is completed by designated practice leader (e.g. Facility Director, Chief Executive Officer) or equivalent senior-level personnel prior to Shatterproof submission. If data integrity issues are identified, the designated practice leader will be contacted to reverify the data and submit a correction.
After Survey Submission
- Flagging Outliers/Facilities Subject to Additional Verification: Results that look unusual trigger additional verification processes, including direct outreach to a treatment facility to confirm data accuracy (e.g. OTP facilities that indicated they provide no MAT when in fact they are required by law to do so). Facilities may be required to answer additional questions, provide information on internal processes and/or submit screenshots. Where treatment facilities decline to respond and/or provide data requested, an indicator on public facing site will show “data not available.”
- Two-week data review period: Prior to public reporting of survey information on ATLAS, treatment facilities will be provided with a two-week review window to ensure survey information was incorporated into the public-facing dashboard correctly prior to publication. If there is an error in the facility’s responses during this time, they can correct the information or choose to suppress their data in favor of the phrasing “data not available.”
- Random Audit: A random audit is conducted on 5–15% of facilities who are required to submit additional documentation to verify the accuracy of various responses from each section of the survey.
- Data Analysis: Responses are assessed by using normative data quality thresholds and data that appear contradictory are flagged for in-depth review.
General Guidelines for Treatment Facilities Related to Data Validation
- Submitting De-Identified Data: For all screenshots and documentation submitted via ATLAS, Shatterproof requires practices to de-identify patient data.
- Disclaimer on Data Limitations: Shatterproof acknowledges the limitations of ATLAS data based on the fact that erroneous data might have been submitted by treatment facilities. This recognizes limits due to data integrity.
Shatterproof is also implementing several strategies to prevent instances of gaming or fraudulent responses to the Patient Experience Survey. Before patient feedback can be submitted, all respondents are required to agree to the ATLAS Terms of Service, which specifically prohibits defamation, fraudulent responses, and instances of gaming. Content moderation processes, including Natural Language Processing and human review, are being used to detect and prevent fraudulent or malicious narrative reviews. Providers also may flag patient narrative feedback that they believe is fraudulent or malicious for Shatterproof to review and remove as needed.
Are you interested in downloading the ATLAS data?
If you are interested in obtaining a license to download the ATLAS data set, please contact ATLAS@shatterproof.org.