tag:cmswebsolutions.statuspage.io,2005:/historyCMS Web Solutions Status - Incident History2024-03-14T13:11:34-04:00CMS Web Solutionstag:cmswebsolutions.statuspage.io,2005:Incident/202449932024-03-14T13:11:34-04:002024-03-14T13:11:34-04:00Email and Hosting outage<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>13:11</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>12:25</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>12:05</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/200491762024-02-22T11:56:56-05:002024-02-22T11:56:56-05:00Webmail login issues<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>11:56</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>10:42</var> EST</small><br><strong>Investigating</strong> - New incident: Investigating<br />We are investigating an issue preventing some users on Cluster A from logging into Webmail. Our engineering team has been engaged.<br /><br />We will provide an update once we have additional information.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/171191422023-05-03T09:30:33-04:002023-05-03T09:30:33-04:00Partial outage email<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>09:30</var> EDT</small><br><strong>Resolved</strong> - All email systems are operational. This incident has been resolved.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>15:56</var> EDT</small><br><strong>Update</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>15:08</var> EDT</small><br><strong>Monitoring</strong> - Our system is now actively processing backlogged email requests. Users may continue to experience intermittent login issues and delays in sending and receiving email over the next few hours. By 7 PM ET, we expect the majority of the backlog to be cleared and for our email service to be functioning normally.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>14:22</var> EDT</small><br><strong>Update</strong> - Update - Users are still experiencing login issues as well as delays and failures to send and receive email. We're continuing the recovery process to restore service.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>13:03</var> EDT</small><br><strong>Update</strong> - Update - Users are still experiencing login issues as well as delays and failures to send and receive email. We’re continuing the recovery process to restore service.<br /><br />We will provide another update by 2 PM ET.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>12:03</var> EDT</small><br><strong>Update</strong> - Users are still experiencing login issues as well as delays and failures to send and receive email. We’re continuing the recovery process to restore service.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>10:51</var> EDT</small><br><strong>Update</strong> - Update - On May 1, 2023, at approximately 5 PM ET, the system alerted us to an attempted large-scale DDoS attack to the Hosted Email Service. The attack was safely mitigated.<br /><br />However, the large volume of requests caused an interruption to the authentication service. We are now gradually ramping up the email service in accordance with recovery procedure. No data has been or will be lost. <br /><br />We will provide another update by 12 PM ET.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>09:46</var> EDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>09:22</var> EDT</small><br><strong>Update</strong> - Restoration efforts are underway. There will be no data or email loss. Both inbound and outbound emails will be delivered gradually upon full restoration of the service.<br />We will provide another update by 11 AM ET.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>09:08</var> EDT</small><br><strong>Update</strong> - We apologize again for the inconvenience while we try to get this issue resolved.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>20:00</var> EDT</small><br><strong>Identified</strong> - Partial outage. The email team is working as quickly as possible to resolve email issues. We are currently not able to offer an estimated time of resolution at this time, but will continue to keep you updated. We appreciate your patience.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/158135412023-01-10T16:05:20-05:002023-01-10T16:06:02-05:00Login and Email Retreival<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>16:05</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>15:26</var> EST</small><br><strong>Investigating</strong> - We are investigating an issue which is causing some users users to encounter authentication failures and email retrieval errors. Further updates will be provided.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/155142452022-12-20T11:53:35-05:002022-12-20T11:53:36-05:00Email Degradation<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>11:53</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>15:37</var> EST</small><br><strong>Monitoring</strong> - Monitoring</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>15:35</var> EST</small><br><strong>Investigating</strong> - Monitoring - We are currently experiencing an incident affecting our Hosted Email service.<br /><br />A range of our IPs has been blocked by AT&T mail servers. The security operations team has investigated and is reaching out to AT&T to have the block lifted.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/113009572022-09-28T14:49:56-04:002022-09-28T14:49:56-04:00Inbound Mail Delays<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>14:49</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>12:09</var> EDT</small><br><strong>Update</strong> - Users may still experience delays however we are seeing improvements, we are continuing to monitor.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>11:25</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>11:24</var> EDT</small><br><strong>Investigating</strong> - Investigating - We are currently experiencing inbound mail delays. Our engineering team is investigating the issue. <br /><br />We will provide an update once we have additional information.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/111964622022-09-20T14:59:01-04:002022-09-20T14:59:01-04:00Email Login Failures<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>14:59</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>14:38</var> EDT</small><br><strong>Investigating</strong> - The engineering team has completed the capacity increase and continue to monitor the cluster for slowness and other issues. Further updates will be provided as they become available.<br />Sep 20, 2022 - 2:38 EST<br /><br />Update - Our engineers are working on adding more capacity to the cluster to handle more requests. Further updates will be provided upon completion.<br />Sep 20, 2022 - 11:19 EST<br /><br />Update - Our engineers identified an increase in resource usage and are restarting the affected Email Servers to alleviate the load. They continue to investigate the issue.<br />Sep 20, 2022 - 10:19 EST<br /><br />Investigating - Our engineering team is aware of a subset of users that are experiencing login failures/slowness on cluster A webmail and are investigating the issue.</p>tag:cmswebsolutions.statuspage.io,2005:Incident/108844902022-08-23T08:53:47-04:002022-08-23T08:53:47-04:00Email Issue<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>08:53</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>11:50</var> EDT</small><br><strong>Monitoring</strong> - Monitoring - All services are functional and we are currently in a monitoring state. We will provide updates as we know more.<br />Aug 22, 14:26 UTC<br /><br />Update - MAC and outbound are back up. We are experiencing a delay in inbound emails which should clear up within 10 minutes. A small subset of users are currently down and the engineering team is investigating this also. We will provide further updates as soon as we know more.<br />Aug 22, 14:05 UTC<br /><br />Update - We are continuing to investigate this issue.<br />Aug 22, 13:14 UTC<br /><br />Investigating - We are experiencing an issue affecting Prod A Webmail. The Engineering team has been engaged and is currently investigating the issue.<br />Aug 22, 13:04 UTC</p>