All Systems Operational
SaaS Platform ? Operational
90 days ago
99.99 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Jul 5, 2022

No incidents reported today.

Jul 4, 2022

No incidents reported.

Jul 3, 2022

No incidents reported.

Jul 2, 2022

No incidents reported.

Jul 1, 2022

No incidents reported.

Jun 30, 2022

No incidents reported.

Jun 29, 2022
Resolved - After a week of monitoring, we have had no new reports nor indications that the issue has reoccured.

As a result, we will be closing our investigations and marking this issue as resolved, even though the reason for its occurence remains a mystery.

One possible suspicion that has been floated the experts we have consulted is an unexpected behaviour by the HTTPCache layer not allowing cookies to pass through as expected. However, so long as the issue can not be reproduced, we are not able to confirm nor deny this suspicion.

We have documented this issue as a part of our Product team documentation, including how to confirm the issue and which steps to take should the issue occur again.

Please do get in touch should you experience this (or any other) issue.

Jun 29, 09:05 UTC
Monitoring - On Tuesday the 21st of June, a technical expert from SensioLabs (the creators of Symfony, the framework upon which the platform is built) conducted a preliminary analysis of the issue.

The consultant identified three primary suspects:

* Attack
* An error in the platform authentication implementation
* Infrastructure

Summarized, these were his conclusions after reflecting on and investigating his suspicions:


## Attack
It is unlikely for an attack to produce this outcome, nor is there any clear motivation/intention for what an attacker would gain from such an outcome.

Conclusion: Very unlikely


## An error in the platform authentication implementation
The security implementation of the platform has been reviewed. Most of the authentication process is covered by the Symfony framework itself, which is very unlikely to be the culprit as Symfony is battle-tested amongst millions of users.

The only platform-specific code interacting the with the authentication mechanism has been audited, and the conclusion is that it can not be the source of this issue.

Conclusion: Very unlikely


## Infrastructure
This has not been investigated deeply, but this would be very uncommon behavior for a web server.

The issue could, of course, have been caused by a service temporarily hanging (not working as expected). If that is the case, the problem might be impossible to reproduce once the service is back to normal.

Conclusion: Unlikely


## Conclusion & measures
Since there have been identified no likely suspects, and we are currently not aware of any users experiencing the issue, we are temporarily settling for increasing the detail level of logging in the production environment in the hopes of getting a better understanding of the problem should it arise again.


We urge our users to contact us directly and immediately for debugging and analysis should they experience the issue again. Please use the phone number +47 99 01 13 14.

Jun 22, 07:04 UTC
Update - We have established contact with Sensio Labs, the creators of Symfony, which is the framework upon which the platform is built. They have started the process of identifying a suitable consultant profile to look into the issue. We are awaiting their reply.
Jun 20, 08:55 UTC
Update - We have indeed confirmed that users experiencing this issue have the details of user different to the one they logged in as in their cookie and session.

This has been reported as a security issue to the framework upon with the platform is built. We await their feedback.

It seems the issue can be worked around short-term by logging out and logging back in, although it does not appear to guarantee the problem not reoccuring.

If you do experience the problem, please visit https://[yourorganization].wecomplish.no/debug/impersonation and send us the output of the page before logging out, logging back in again and sending us the output of the same page again.

Jun 19, 16:01 UTC
Update - After a day of troubleshooting, we are still unable to pin-point the problem.

Our current hypothesis is that the problem is related to the platform (seemingly only in some cases) storing incorrect session and/or cookie content.

The investigation has proven difficult as, due to our inability to reproduce the issue on our end since 11:10 am, we have had to rely on users still experiencing the problem to help us troubleshoot it.

Now that people are leaving work for the weekend, we are not getting the feedback we require to continue our investigative efforts. We will therefore halt our investigations, and pick them up again as soon as we are either able to reproduce the issue, or have access to the people who do so (presumably no later than Monday morning).

If you experience the issue yourself, please go to https://[yourorganization].wecomplish.no/debug/impersonation and forward the debug output the page produces to your insight consultant.

Jun 17, 15:19 UTC
Investigating - Some users are still experiencing this issue. Resuming investigation.
Jun 17, 12:47 UTC
Monitoring - We have implemented a fix which may have an impact on the issue. We are monitoring in order to see if the issue persists.

Please give notice if you are still experiencing the issue.

Jun 17, 11:26 UTC
Update - We have no viable theories as to what could be causing the issue. We are starting to add more logging and debugging output to the code base in an attempt to trace the issue back to its source. This process is expected to take a couple of hours before yielding any meaningful insight.
Jun 17, 09:48 UTC
Update - We are in communication with our production environment vendor discussing what we think the root cause might be.
Jun 17, 08:52 UTC
Update - This morning (08:10 AM GMT+2), two users of our application reported that they were all of a sudden logged in as someone else, meaning the UI indicator which displays the profile picture of the logged in user changed to someone else in their organization, and they were seeing the data belonging to that user.

Both users where "switched" to the same person (Person X).

The switch happened when they were just navigating around the application.

When we tested, we could reproduce the issue. Just by navigating around on a few pages, I was all of a sudden logged in as Person X.

After logging out and logging in again, we were no longer able to reproduce the issue, meaning that when we continued to navigate around the application, we remained as our own user.

We have inspected the production data around this particular user without anything seeming out of the ordinary, and have copied the production data to our local environment to try and reproduce the issue there, without any luck. Since the issue was not reproducable after logging out and logging in, this seems less likely to be a data issue.

There have been no recent changes to the part of the code base handling authentication to account for such an issue. Also, the fact that the issue can not be reproduced after logging out and logging back in suggests that the issue is not specifically code-related.

Our production environment vendor has had some maintenance work done during the last 24 hours, but seemingly for regions unrelated to ours. Still, a high priority ticket has been reported with them, and they confirmed that they were looking into this 09:49.

We are awaiting their examination and reply.

Jun 17, 08:26 UTC
Investigating - Since aprox. 08:00 this morning, some users in one specific organization is experiencing being switched to another user in their organization.
Jun 17, 08:19 UTC
Jun 28, 2022

No incidents reported.

Jun 27, 2022

No incidents reported.

Jun 26, 2022

No incidents reported.

Jun 25, 2022

No incidents reported.

Jun 24, 2022

No incidents reported.

Jun 23, 2022

No incidents reported.

Jun 22, 2022
Jun 21, 2022

No incidents reported.