On January 16th, Moxion engineering team was preparing a new set of features to be released. During final testing in our test environment we discovered an issue with Production Watermarking, a new feature that was a part of the upcoming release. The new Production Watermarking feature adds the ability to create a watermark for individual productions on Moxion (in addition to Company, Folder and no Watermark settings). The issue discovered before the release required to update our back-end with some missing default elements.
We hotfixed that issue, included the fix into the release and finished the testing of the release. On February 16th, at 4:46pm (PST), the release went live on.
On February 18th, at 10:45pm (PST) Moxion became aware of an issue with Watermark settings being switched off and we launched an urgent investigation. We discovered two problems:
On February 19, at 00:12am (PST) we identified the root cause of the issue: migration scripts that were added as a hotfix to the bug discovered during pre-release testing had the side-effect of changing the default watermarking settings for folders.
At 02:30am (PST) we deployed a fix to the issue.
Company level visual watermarking had been turned off for any folder that had the default watermarking settings, the issue was present for 2 days and 9 hours.
Folders (as well as their subfolder) that had watermarking settings edited in any way had not been affected.
The issue was resolved within 3 hours 30 minutes after the first report from a customer.
We deployed a fix to the default watermarking settings by making updates to the database.
We've planned to add a new automated integration test to check the default watermarking settings on a new folder.
We have also planned to put in place enhanced sanity checks for security hotfixes as well additional review on changes that could impact security.