tag:tettra.statuspage.io,2005:/historyTettra Status - Incident History2024-03-28T14:18:30-04:00Tettratag:tettra.statuspage.io,2005:Incident/199793542024-02-14T00:19:01-05:002024-02-14T00:19:01-05:00Search not working for a small % of teams<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>00:19</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>18:04</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:tettra.statuspage.io,2005:Incident/198098392024-02-03T22:10:00-05:002024-02-03T22:13:40-05:00Database upgrades<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>22:10</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>21:00</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:tettra.statuspage.io,2005:Incident/194021652023-12-11T11:17:55-05:002023-12-11T11:17:55-05:00Pages and sidebar not loading for read-only users<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:17</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:17</var> EST</small><br><strong>Update</strong> - A fix for this issue has been deployed.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>10:37</var> EST</small><br><strong>Identified</strong> - Read-only users are not able to view pages or the sidebar right now. We're working on rolling back the deploy the caused this issue.</p>tag:tettra.statuspage.io,2005:Incident/178501332023-07-13T22:58:20-04:002023-07-13T22:58:20-04:00Some users unable to create new team<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>22:58</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p>tag:tettra.statuspage.io,2005:Incident/174283012023-05-25T18:00:00-04:002023-05-31T01:08:01-04:00Intermittent issues with SSO Login<p><small>May <var data-var='date'>25</var>, <var data-var='time'>18:00</var> EDT</small><br><strong>Resolved</strong> - We were having issues with our SSO provider integration. This has now been resolved.</p>tag:tettra.statuspage.io,2005:Incident/108956672022-08-23T20:38:26-04:002022-08-23T20:38:26-04:00Tettra web app is down<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>20:38</var> EDT</small><br><strong>Resolved</strong> - A workaround for the DNS issue has been implemented and the app is accessible as per normal</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>20:03</var> EDT</small><br><strong>Identified</strong> - Our hosting provider Heroku is currently experiencing DNS issues, which is affecting the Tettra app’s ability to properly load. We are currently working to identify a potential workaround.<br /><br />https://status.heroku.com/incidents/2453</p>tag:tettra.statuspage.io,2005:Incident/94001432022-02-24T13:03:15-05:002022-02-24T13:03:15-05:00Webapp is down<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>13:03</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:19</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:18</var> EST</small><br><strong>Monitoring</strong> - Connectivity to the webapp is recovering.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>11:45</var> EST</small><br><strong>Identified</strong> - Our hosting provider, Heroku, is experiencing an outage.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>11:41</var> EST</small><br><strong>Investigating</strong> - The Tettra webapp is not responding to requests. We are investigating the issue.</p>tag:tettra.statuspage.io,2005:Incident/93745712022-02-21T15:46:46-05:002022-02-21T15:46:46-05:00Email delivery delayed due to service provider errors<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>15:46</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>14:05</var> EST</small><br><strong>Monitoring</strong> - Our transactional email service provider, Mailgun, is experiencing API issues causing our email delivery to fail. We are monitoring the situation.<br /><br />https://status.mailgun.com/incidents/0hgvxpzzhvcq</p>tag:tettra.statuspage.io,2005:Incident/59450512021-01-04T14:49:38-05:002021-01-04T14:49:39-05:00Slack Instability<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>14:49</var> EST</small><br><strong>Resolved</strong> - We believe slack's outage is now contained to parts of Slack that do not impact Tettra. Please let us know if you see any further issues with the Slack integration.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>13:31</var> EST</small><br><strong>Monitoring</strong> - Slack is starting to come back online, so we hope all issues with our integration will resolve soon.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>11:24</var> EST</small><br><strong>Update</strong> - We are continuing to monitor Slack. You can see their own status page, here: https://status.slack.com/</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>10:52</var> EST</small><br><strong>Investigating</strong> - Slack is currently experiencing system-wide issues. Tettra users may be unable to sign in using Slack or use the Slack integration within Tettra. We are monitoring the situation in case there is anything we can do to mitigate these issues with Slack.</p>tag:tettra.statuspage.io,2005:Incident/58888702020-12-26T15:54:13-05:002020-12-26T15:54:13-05:00Email delivery disabled due to spam attack<p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>15:54</var> EST</small><br><strong>Resolved</strong> - Email delivery and links are operational again. The 'share' page via email functionality has been temporarily disabled while we mitigate further spam attempts.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>12:34</var> EST</small><br><strong>Update</strong> - Links in existing/delivered emails are also disabled while we continue to work on a fix.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>11:10</var> EST</small><br><strong>Identified</strong> - All emails from our application have been suspended since Dec 24 approximately 22:24 UTC due to a spam attack. This includes all automated notification emails, invitation emails, and page share emails.<br /><br /><br />We have identified and suspended the malicious account(s) and are in the process of re-enabling emails.</p>tag:tettra.statuspage.io,2005:Incident/41206392020-05-17T13:36:11-04:002020-05-17T13:36:11-04:00Content embeds are failing<p><small>May <var data-var='date'>17</var>, <var data-var='time'>13:36</var> EDT</small><br><strong>Resolved</strong> - Content embedding functionality has been restored. Thanks for your patience.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>08:22</var> EDT</small><br><strong>Monitoring</strong> - Embedly is still working on a fix. At this time there is no ETA as to when content embedding will be available again. We will update here as we know more. Original incident: http://status.embed.ly/incidents/tdkr78g2qq5x</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>16:03</var> EDT</small><br><strong>Identified</strong> - We've identified an issue with our content embedding provider that is causing content embedding in pages to fail. We'll update this issue once the problem is resolved.</p>tag:tettra.statuspage.io,2005:Incident/39211942020-04-17T17:00:00-04:002020-04-20T13:14:52-04:00Readonly users unable to access Tettra pages<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>17:00</var> EDT</small><br><strong>Resolved</strong> - From approximately Friday 4/17 5pm EST to Saturday 11am EST, readonly users were not able to access Tettra pages on the webapp.</p>tag:tettra.statuspage.io,2005:Incident/36951022020-03-16T17:37:21-04:002020-03-16T17:37:21-04:00Webapp unavailable<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>17:37</var> EDT</small><br><strong>Resolved</strong> - Access to the webapp has returned to normal. We will continue to keep an eye on the underlying Heroku incident that caused this and the outage earlier today: https://status.heroku.com/incidents/1974</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>16:01</var> EDT</small><br><strong>Monitoring</strong> - We are continuing to monitor the app as it recovers.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>15:51</var> EDT</small><br><strong>Investigating</strong> - app.tettra.co is unavailable due to problems with our service provider. We are investigating the issue.</p>tag:tettra.statuspage.io,2005:Incident/36942292020-03-16T13:36:40-04:002020-03-16T13:36:40-04:00Web app unavailable<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>13:36</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:31</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:30</var> EDT</small><br><strong>Monitoring</strong> - The app is loading normally again. We'll keep monitoring until we hear from our hosting provider.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:25</var> EDT</small><br><strong>Update</strong> - Our service provider Heroku is experiencing widespread platform issues. We will update as we have more information.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:21</var> EDT</small><br><strong>Investigating</strong> - Visitors to app.tettra.co are seeing a Cloudflare error. We are investigating the cause.</p>tag:tettra.statuspage.io,2005:Incident/31536662019-11-02T20:26:33-04:002019-11-02T20:26:33-04:00Database SSL Certificate Update<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>20:26</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>20:25</var> EDT</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>20:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>15:59</var> EDT</small><br><strong>Scheduled</strong> - We will be updating our SSL certificates on our primary database during this time, which requires a reboot. We expect minimal service disruption but you may notice a few minutes of slower response times.</p>tag:tettra.statuspage.io,2005:Incident/31157302019-10-26T20:45:30-04:002019-10-26T20:45:30-04:00Database SSL Certificate Update<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>20:45</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>20:15</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>14:19</var> EDT</small><br><strong>Scheduled</strong> - We will be updating our SSL certificates on our primary database during this time, which requires a reboot. Expected downtime is less than 5 minutes.</p>tag:tettra.statuspage.io,2005:Incident/29633932019-09-05T11:47:09-04:002019-09-05T11:47:09-04:00Slowness and Instability<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>11:47</var> EDT</small><br><strong>Resolved</strong> - We have successfully mitigated the connection issues, and will work to assess what we can do to prevent such issues in the future.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>11:31</var> EDT</small><br><strong>Update</strong> - We have narrowed down the issue to problems connecting to where we store data. We have contacted our service provider's support team but are also further investigating ourselves.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>11:10</var> EDT</small><br><strong>Investigating</strong> - We are getting alerts of degraded performance to the point where some users are unable to view their content or log in. We are investigating.</p>tag:tettra.statuspage.io,2005:Incident/26870742019-07-10T20:28:51-04:002019-07-10T20:28:51-04:00Intermittent 500 errors<p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>20:28</var> EDT</small><br><strong>Resolved</strong> - Our billing provider has recovered. Please let us know if you see any more 500 errors.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>17:40</var> EDT</small><br><strong>Identified</strong> - We're aware of intermittent 500 errors being caused by issues with our billing provider. We will update you again when the problems have been resolved.</p>tag:tettra.statuspage.io,2005:Incident/26424192019-07-02T10:19:06-04:002019-07-02T10:19:06-04:00Web application unavailable<p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:19</var> EDT</small><br><strong>Resolved</strong> - Cloudflare has implemented a fix for the network issue and is currently monitoring the results. The application is available again. We are continuing to monitor the situation.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:17</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:11</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with our DNS provider Cloudflare.</p>tag:tettra.statuspage.io,2005:Incident/21968252019-02-09T19:01:00-05:002019-02-09T19:01:01-05:00System-wide update<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>19:01</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>18:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>13:54</var> EST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>13:51</var> EST</small><br><strong>Scheduled</strong> - Between 6-7pm EST on 02/09/2019, we will be performing a scheduled maintenance task. Expected downtime is <5 minutes. All users will be prompted to log back in once the task is complete.</p>tag:tettra.statuspage.io,2005:Incident/20641292018-11-25T05:30:26-05:002018-11-26T11:08:05-05:00Intermittent Webapp Connectivity Issue<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>05:30</var> EST</small><br><strong>Resolved</strong> - Early Sunday morning 5:30AM - 8AM EST the Tettra webapp was intermittently unavailable due to network issues on one of our application servers.</p>tag:tettra.statuspage.io,2005:Incident/19861222018-10-22T11:03:06-04:002018-10-22T11:03:06-04:00Broken links in email notifications<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>11:03</var> EDT</small><br><strong>Resolved</strong> - From about 2018-10-22 3am EDT to 10:30am EDT links in Tettra email notifications were broken due to a DNS error. Our DNS settings have been updated and this issue is now resolved.</p>tag:tettra.statuspage.io,2005:Incident/18736742018-08-22T15:25:18-04:002018-08-22T15:25:18-04:00Web App Down<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>15:25</var> EDT</small><br><strong>Resolved</strong> - This issue has been resolved. We saw a spike in Slack events due to changing our App's name in the Slack directory. We're taking steps to make sure this doesn't happen again.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>14:33</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>14:33</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>14:25</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:tettra.statuspage.io,2005:Incident/18501542018-08-07T16:07:02-04:002018-08-07T17:03:44-04:00Web App Down<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>16:07</var> EDT</small><br><strong>Resolved</strong> - Our web app was down for 8 minutes between 4:07-4:25pm EST. The incident is resolved and we are investigating the cause. Sorry for the issues.</p>tag:tettra.statuspage.io,2005:Incident/18229202018-07-20T10:55:24-04:002018-07-20T10:55:41-04:00SSL errors on app.tettra.co<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>10:55</var> EDT</small><br><strong>Resolved</strong> - From 10:36 - 10:42 am EDT this morning we experienced unexpected downtime due to an SSL certificate migration issue.</p>