What happened?

On January 16th, Moxion engineering team was preparing a new set of features to be released. During final testing in our test environment we discovered an issue with Production Watermarking, a new feature that was a part of the upcoming release. The new Production Watermarking feature adds the ability to create a watermark for individual productions on Moxion (in addition to Company, Folder and no Watermark settings). The issue discovered before the release required to update our back-end with some missing default elements. 

We hotfixed that issue, included the fix into the release and finished the testing of the release. On February 16th, at 4:46pm (PST), the release went live on. 

On February 18th, at 10:45pm (PST) Moxion became aware of an issue with Watermark settings being switched off and we launched an urgent investigation. We discovered two problems:

1. Default watermarking settings were changed to a folder watermark instead of a company watermark. As a result, since most folders don't have a watermark configured, the watermarking settings had been turned off for folders with default watermarking. 
2. Folders that had watermarking customized at the folder level before the release were displaying a broken user interface: the watermarking settings window was showing that no watermark had been selected. That was due to the change in the data format we added to the release and it only affected the look of the interface, not the watermarking settings themselves.

On February 19, at 00:12am (PST) we identified the root cause of the issue: migration scripts that were added as a hotfix to the bug discovered during pre-release testing had the side-effect of changing the default watermarking settings for folders.

At 02:30am (PST) we deployed a fix to the issue.

Impact

Company level visual watermarking had been turned off for any folder that had the default watermarking settings, the issue was present for 2 days and 9 hours.

Folders (as well as their subfolder) that had watermarking settings edited in any way had not been affected. 

The issue was resolved within 3 hours 30 minutes after the first report from a customer. 

Resolution

We deployed a fix to the default watermarking settings by making updates to the database. 

Corrective actions 

We've planned to add a new automated integration test to check the default watermarking settings on a new folder. 

We have also planned to put in place enhanced sanity checks for security hotfixes as well additional review on changes that could impact security.