All services are online

Resolved: built-in extensions have been safely re-enabled, and with that, full service has been restored and this incident is resolved. So sorry for the disruption!

Partially resolved: root cause found an addressed, all spaces seem to have been working smoothly for the last 30 mins. Extensions still disabled. Will re-enable after adding additional safeguards around the root issue. Postmortem to follow

Update: some small number of spaces are still experiencing issues, but most should be back to normal. Still seeing if there's anything we can do to speed up recovery

Resolving: stability still gradually returning to normal, still not intervening unless we're confident it'll speed up recovery. At this rate, should be back to normal in 15 more minutes. Extensions still disabled, will fix soon

Update: disabled built-in extensions (doors, password doors, plants, etc) for now, stability slowly improving. Trying to speed it up. Will fix extensions ASAP after main service is restored. Not the clearly the main cause, but exacerbated the issue

Back to investigating: the fix didn't fix it, may have improved slightly but incident is ongoing. Still working on it as fast as possible

Update: deploy mostly complete, expect ~10 mins of minor instability and then everything should be fixed

Update: rollout not going smoothly, still working on it. ETA now 30 mins

Update: fix is being deployed now, issue should gradually resolve over the next 15-20 mins. Keeping a close eye on it to be sure

Update: Identified likely cause, working on a fix now