What happened?
On January 26th at 7:04pm (PST), Moxion became aware of a DRM playback issue from several DRM key request error reports. Playback for all DRM video was broken: when trying to play a video, a DRM key error message displayed. We launched an urgent investigation.
At 7:46pm (PST) our engineering team identified an issue where our DRM provider had a region specific endpoint outage. In response, our team switched to an alternative DRM provider which we had been testing and were planning to migrate to. This allowed us to accelerate the incident resolution for any newly uploaded assets. At 8:20pm (PST), our old DRM provider resolved the endpoint outage and playback was restored for historical assets.
On January 27th at 11:00pm (PST), we identified an issue with Apple TV DRM playback for newly uploaded assets. The recent switch to a new DRM provider had introduced this issue. In response, Moxion switched back to our old DRM provider and reprocessed the affected assets. At 4:30pm (PST), we deployed a hotfix for the root cause fixing Apple TV DRM playback on our new DRM provider allowing us to proceed with the migration.
Impact
DRM playback of all assets was broken from 7:00pm till 8:00pm Pacific on January 26th. Then, the DRM playback was restored for all newly uploaded assets but was still failing for all historic assets for another 30min.
DRM playback on Apple TV for newly uploaded assets was further broken for 2.5 days and was resolved within hours after the issue had been reported.
Resolution
We enabled a new DRM provider during the outage of the old DRM provider, to resolve the first outage. We deployed a fix for the root cause of the new DRM provider related issue on Apple TV.
Corrective actions
We've improved our test coverage for the Apple TV platform. This will help us to fix potential issues before anything has been rolled out to production.
We've planned updates to improve our DRM endpoint monitoring. This will help us to detect any errors with DRM playback as they come up and to act on them before they become a problem.
References
Further information about this incident is available on the status page (https://status.moxion.io/)