DRM Error: Playback on AppleTV

Incident Report for Moxion

Postmortem

What happened?

On January 26th at 7:04pm (PST), Moxion became aware of a DRM playback issue from several DRM key request error reports. Playback for all DRM video was broken: when trying to play a video, a DRM key error message displayed. We launched an urgent investigation.

At 7:46pm (PST) our engineering team identified an issue where our DRM provider had a region specific endpoint outage. In response, our team switched to an alternative DRM provider which we had been testing and were planning to migrate to. This allowed us to accelerate the incident resolution for any newly uploaded assets. At 8:20pm (PST), our old DRM provider resolved the endpoint outage and playback was restored for historical assets.

On January 27th at 11:00pm (PST), we identified an issue with Apple TV DRM playback for newly uploaded assets. The recent switch to a new DRM provider had introduced this issue. In response, Moxion switched back to our old DRM provider and reprocessed the affected assets. At 4:30pm (PST), we deployed a hotfix for the root cause fixing Apple TV DRM playback on our new DRM provider allowing us to proceed with the migration.

Impact

DRM playback of all assets was broken from 7:00pm till 8:00pm Pacific on January 26th. Then, the DRM playback was restored for all newly uploaded assets but was still failing for all historic assets for another 30min.

DRM playback on Apple TV for newly uploaded assets was further broken for 2.5 days and was resolved within hours after the issue had been reported.

Resolution

We enabled a new DRM provider during the outage of the old DRM provider, to resolve the first outage. We deployed a fix for the root cause of the new DRM provider related issue on Apple TV.

Corrective actions

We've improved our test coverage for the Apple TV platform. This will help us to fix potential issues before anything has been rolled out to production.

We've planned updates to improve our DRM endpoint monitoring. This will help us to detect any errors with DRM playback as they come up and to act on them before they become a problem.

References

Further information about this incident is available on the status page (https://status.moxion.io/)

Posted Feb 04, 2021 - 19:29 UTC

Resolved

This incident has been resolved.

Posted Jan 29, 2021 - 01:37 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jan 29, 2021 - 00:29 UTC

Update

We have identified there is a window of assets affected from approximately 8pm January 27th to 10am January 28th. We are working to fix the issue for these assets. Any newly uploaded assets are unaffected by this issue and will playback normally on AppleTV. Assets uploaded before this time will also playback normally.

Posted Jan 28, 2021 - 19:44 UTC

Identified

We have discovered an issue with DRM and playback on the AppleTV. We have identified the cause, and are working on a fix.

Posted Jan 28, 2021 - 17:42 UTC

This incident affected: Flow Capture (Viewing/Playback).