Codefinder/CRS and Coding in the cloud (CITC): - 100% of production Coding in the cloud (CRS, CITC) environments have been operational since 15:38 EDT on Friday, 2024-07-19. - 100% of production Legacy Codefinder/CRS environments have been operational since 17:00 EDT on Tuesday, 2024-07-23.
360 Encompass in the cloud (360Eitc): - 100% of production 360 Encompass in the cloud environments have been operational since 01:04 EDT on Saturday, 2024-07-20. - 98.18% of non-production 360 Encompass in the cloud environments are operational as of 16:00 EDT on Wednesday, 2024-07-24. - Build and "in progress" environments are still expected to be 100% recovered by the end of the week. This includes deployments in Australia and Canada.
Important! This incident will be marked "resolved" around COB EDT on Thursday, 2024-07-24. In the meantime: - Connexion Integrator Remote Agent installations on the local network may be affected by CrowdStrike while local IT teams resolve issues affecting your local network: this may cause delays in transmission of patient/visit data to 360 Encompass in the cloud. - For down systems, please contact https://support.3mhis.com. - For all other questions, please contact your assigned application administrator.
Posted Jul 24, 2024 - 17:59 CDT
Update
100% of production Coding in the cloud (CRS, CITC) environments have been operational since 15:38 EDT on Friday, 2024-07-19.
100% of production 360 Encompass in the cloud environments have been operational since 01:04 EDT on Saturday, 2024-07-20.
Legacy, non-production and build environments are being recovered presently.
Production deployments of legacy Codefinder/CRS environments are expected to be 100% recovered later today.
The recovery of 100% of non-production 360 Encompass in the cloud environments is expected to be completed no later than 16:00 EDT on Wednesday, 2024-07-24.
Build and "in progress" environments are expected to be 100% recovered by the end of the week.
Posted Jul 22, 2024 - 11:33 CDT
Monitoring
All production 360 Encompass in the cloud systems have been recovered and validated: end-users may access and use managed production environments as usual.
All production Coding in the cloud (CRS in the cloud, CITC) have been recovered and validated.
Non-production and legacy environments will be recovered during the coming days during North America US business hours.
Important! Connexion Integrator Remote Agent installations on the local network may be affected by CrowdStrike while local IT teams resolve issues affecting your local network: this may cause delays in transmission of patient/visit data to 360 Encompass in the cloud.
For down systems, please contact https://support.3mhis.com; for all other questions, please contact your assigned application administrator.
Posted Jul 20, 2024 - 00:04 CDT
Update
Most production 360 Encompass in the cloud systems have been recovered and are available for normal use by end-users.
A small number of systems that encountered more serious issues are still being repaired, validated, etc.
Posted Jul 19, 2024 - 21:58 CDT
Update
Approximately 80% of production 360 Encompass in the cloud systems have been remediated/validated.
ETA on full resolution for all systems is 1-2 hours from the time of this update.
Posted Jul 19, 2024 - 20:48 CDT
Update
Approximately 60% of production 360 Encompass in the cloud systems have been remediated/validated.
ETA on full resolution for all systems is 2-3 hours from the time of this update.
Posted Jul 19, 2024 - 19:40 CDT
Update
Approximately 50% of production 360 Encompass in the cloud systems have been remediated/validated.
ETA on full resolution for all systems is 2-4 hours from the time of this update.
Posted Jul 19, 2024 - 18:39 CDT
Update
Approximately 40% of production 360 Encompass in the cloud systems have been remediated/validated.
ETA on full resolution for all systems is 3-5 hours from the time of this update.
Posted Jul 19, 2024 - 17:31 CDT
Update
Solventum cloud engineers have developed a proprietary rescue utility that will enable engineers to recover systems at a much higher rate of speed that previously. ETA on full recovery four to six hours from the time of this update.
Posted Jul 19, 2024 - 16:59 CDT
Update
Approximately 30% of production 360 Encompass in the cloud systems have been remediated/validated.
Posted Jul 19, 2024 - 15:45 CDT
Update
Engineers continue to recover production 360 Encompass in the cloud systems. At present approximately 15% of production 360 Encompass in the cloud systems have been recovered, validated and released to end-users.
All production coding in the cloud (CRS in the cloud, CITC) systems have been released (see previous update).
ETA on full recovery for production systems is now close of business US Mountain time.
Posted Jul 19, 2024 - 14:38 CDT
Update
Production coding in the cloud (CRS in the cloud, CITC) systems are fully recovered and released to end-users.
360 Encompass systems are being returned to end-users one at a time.
Important! HL7 interfaces to managed 360 Encompass in the cloud systems will remain paused until systems are validated and released.
Target time for full recovery of production systems is close of business, EDT.
Posted Jul 19, 2024 - 12:54 CDT
Update
Production Coding in the cloud (CRS in the cloud, CITC) systems are expected to be restored within the hour.
Posted Jul 19, 2024 - 11:20 CDT
Update
Application administrators have begun the recovery of production systems and will be reaching out to emergency contacts to validate recovered environments.
ETA target for full recovery of all affected environments is close of business today, Friday, 2024-07-19.
Posted Jul 19, 2024 - 11:08 CDT
Update
Engineers have developed a general solution to resolve the issue across all affected systems. Teams are preparing to remediate individual systems presently and will begin the rollout at approximately 09:00 Eastern. No ETA on full recovery at this time.
Posted Jul 19, 2024 - 07:31 CDT
Update
Engineering teams are still working on a general solution to recover platforms affected by the CrowdStrike issue. No ETA on full recovery.
Posted Jul 19, 2024 - 05:40 CDT
Update
Engineers have remediated the CrowdStrike issue and restored normal functionality on several environments, but due to the scope and scale of the issue, there is still no ETA on full recovery of all affected systems.
Posted Jul 19, 2024 - 04:31 CDT
Identified
The root cause of the outage was related to a Crowdstrike update that caused outages worldwide. A fix has been applied to several affected systems and resolved their issues. Teams are working now to apply the fix generally.
No ETA on general resolution.
Posted Jul 19, 2024 - 01:57 CDT
Update
An enterprise-wide issue affecting the Windows operating system has resulted in multiple key services failing.
End-users are unable to access any managed applications at this time.
Posted Jul 19, 2024 - 00:57 CDT
Update
Windows BSOD issues are occurring generally and affecting multiple services.
Posted Jul 19, 2024 - 00:37 CDT
Update
The AWS EC2 service appears to be impaired in the US-EAST-1 region resulting in general failures affecting all applications. We are continuing to investigate.
Posted Jul 19, 2024 - 00:14 CDT
Investigating
Beginning at 23:33 Central, outge alerts began to be received. We are investigating the issue.
Posted Jul 19, 2024 - 00:00 CDT
This incident affected: US Central (PRODUCTION - 360 Encompass, PRODUCTION - Coding and Reimbursement System (CRS) legacy, Non-production - 360 Encompass, Non-production - Coding and Reimbursement System (CRS) legacy, PRODUCTION - Coding in the cloud (CITC), Non-production - Coding in the cloud (CITC)), US Pacific (PRODUCTION - 360 Encompass, PRODUCTION - Coding and Reimbursement System (CRS) legacy, Non-production - 360 Encompass, Non-production - Coding and Reimbursement System (CRS) legacy, PRODUCTION - Coding in the cloud (CITC), Non-production - Coding in the cloud (CITC)), US East (Non-production - Coding and Reimbursement System (CRS) legacy, PRODUCTION - Coding and Reimbursement System (CRS) legacy, Non-production - 360 Encompass, PRODUCTION - 360 Encompass, PRODUCTION - Coding in the cloud (CITC), Non-production - Coding in the cloud (CITC)), and US Mountain (PRODUCTION - 360 Encompass, PRODUCTION - Coding and Reimbursement System (CRS) legacy, Non-production - 360 Encompass, Non-production - Coding and Reimbursement System (CRS) legacy, PRODUCTION - Coding in the cloud (CITC), Non-production - Coding in the cloud (CITC)).