After a week of monitoring, we have had no new reports nor indications that the issue has reoccured.
As a result, we will be closing our investigations and marking this issue as resolved, even though the reason for its occurence remains a mystery.
One possible suspicion that has been floated the experts we have consulted is an unexpected behaviour by the HTTPCache layer not allowing cookies to pass through as expected. However, so long as the issue can not be reproduced, we are not able to confirm nor deny this suspicion.
We have documented this issue as a part of our Product team documentation, including how to confirm the issue and which steps to take should the issue occur again.
Please do get in touch should you experience this (or any other) issue.
Posted Jun 29, 2022 - 09:05 UTC
Monitoring
On Tuesday the 21st of June, a technical expert from SensioLabs (the creators of Symfony, the framework upon which the platform is built) conducted a preliminary analysis of the issue.
The consultant identified three primary suspects:
* Attack * An error in the platform authentication implementation * Infrastructure
Summarized, these were his conclusions after reflecting on and investigating his suspicions:
## Attack It is unlikely for an attack to produce this outcome, nor is there any clear motivation/intention for what an attacker would gain from such an outcome.
Conclusion: Very unlikely
## An error in the platform authentication implementation The security implementation of the platform has been reviewed. Most of the authentication process is covered by the Symfony framework itself, which is very unlikely to be the culprit as Symfony is battle-tested amongst millions of users.
The only platform-specific code interacting the with the authentication mechanism has been audited, and the conclusion is that it can not be the source of this issue.
Conclusion: Very unlikely
## Infrastructure This has not been investigated deeply, but this would be very uncommon behavior for a web server.
The issue could, of course, have been caused by a service temporarily hanging (not working as expected). If that is the case, the problem might be impossible to reproduce once the service is back to normal.
Conclusion: Unlikely
## Conclusion & measures Since there have been identified no likely suspects, and we are currently not aware of any users experiencing the issue, we are temporarily settling for increasing the detail level of logging in the production environment in the hopes of getting a better understanding of the problem should it arise again.
We urge our users to contact us directly and immediately for debugging and analysis should they experience the issue again. Please use the phone number +47 99 01 13 14.
Posted Jun 22, 2022 - 07:04 UTC
Update
We have established contact with Sensio Labs, the creators of Symfony, which is the framework upon which the platform is built. They have started the process of identifying a suitable consultant profile to look into the issue. We are awaiting their reply.
Posted Jun 20, 2022 - 08:55 UTC
Update
We have indeed confirmed that users experiencing this issue have the details of user different to the one they logged in as in their cookie and session.
This has been reported as a security issue to the framework upon with the platform is built. We await their feedback.
It seems the issue can be worked around short-term by logging out and logging back in, although it does not appear to guarantee the problem not reoccuring.
After a day of troubleshooting, we are still unable to pin-point the problem.
Our current hypothesis is that the problem is related to the platform (seemingly only in some cases) storing incorrect session and/or cookie content.
The investigation has proven difficult as, due to our inability to reproduce the issue on our end since 11:10 am, we have had to rely on users still experiencing the problem to help us troubleshoot it.
Now that people are leaving work for the weekend, we are not getting the feedback we require to continue our investigative efforts. We will therefore halt our investigations, and pick them up again as soon as we are either able to reproduce the issue, or have access to the people who do so (presumably no later than Monday morning).
Some users are still experiencing this issue. Resuming investigation.
Posted Jun 17, 2022 - 12:47 UTC
Monitoring
We have implemented a fix which may have an impact on the issue. We are monitoring in order to see if the issue persists.
Please give notice if you are still experiencing the issue.
Posted Jun 17, 2022 - 11:26 UTC
Update
We have no viable theories as to what could be causing the issue. We are starting to add more logging and debugging output to the code base in an attempt to trace the issue back to its source. This process is expected to take a couple of hours before yielding any meaningful insight.
Posted Jun 17, 2022 - 09:48 UTC
Update
We are in communication with our production environment vendor discussing what we think the root cause might be.
Posted Jun 17, 2022 - 08:52 UTC
Update
This morning (08:10 AM GMT+2), two users of our application reported that they were all of a sudden logged in as someone else, meaning the UI indicator which displays the profile picture of the logged in user changed to someone else in their organization, and they were seeing the data belonging to that user.
Both users where "switched" to the same person (Person X).
The switch happened when they were just navigating around the application.
When we tested, we could reproduce the issue. Just by navigating around on a few pages, I was all of a sudden logged in as Person X.
After logging out and logging in again, we were no longer able to reproduce the issue, meaning that when we continued to navigate around the application, we remained as our own user.
We have inspected the production data around this particular user without anything seeming out of the ordinary, and have copied the production data to our local environment to try and reproduce the issue there, without any luck. Since the issue was not reproducable after logging out and logging in, this seems less likely to be a data issue.
There have been no recent changes to the part of the code base handling authentication to account for such an issue. Also, the fact that the issue can not be reproduced after logging out and logging back in suggests that the issue is not specifically code-related.
Our production environment vendor has had some maintenance work done during the last 24 hours, but seemingly for regions unrelated to ours. Still, a high priority ticket has been reported with them, and they confirmed that they were looking into this 09:49.
We are awaiting their examination and reply.
Posted Jun 17, 2022 - 08:26 UTC
Investigating
Since aprox. 08:00 this morning, some users in one specific organization is experiencing being switched to another user in their organization.