As part of improving our security policies in line with our annual Penetration Testing, and as we look to achieve our ISO27001 and SOC2 status we incorrectly identified a storage location as a temporary holding place for files. This resulted in the deletion of all files associated with the Drag & Drop email builder, and those files are permanently and irrevocably deleted which requires affected clients to re-upload them manually.
Here are some quick links to sections of this article:
"Sorry isn't quite enough in this situation. However, I am truly sorry that you're having to spend your time fixing our screwup. Gecko is a customer-first company, and this situation is absolutely unacceptable. We are undertaking a full review of how this happened, and why the risk associated with the deletion of files wasn't mitigated. Our whole purpose is to make your lives easier, and we have failed to do that for those affected by this."
Matt Lanham - CEO & Founder
Our team identified a file storage location that was assumed to be temporary but did not have the correct policy in place to remove files automatically. To resolve this a new deletion policy was added at 4 pm BST on Thursday 29th October 2022 which would remove files that were more than 3 days old.
As part of how Amazon processes these policies it then deleted all matching files at approx 1 am on Friday 30th October 2022.
On Saturday 1st October 2022 our team were alerted to an issue affecting emails where images were not displaying and clients reported them to be deleted, therefore our team began investigating the issue further.
The root cause of this quickly surfaced and related to the change in policy, the storage location was in fact not temporary and was being used to store images that had been uploaded to the Drag & Drop Email Builder functionality that clients with our Promote product have access to.
Unfortunately due to the storage location being assumed to be temporary, it did not have versioning or backups in place, and therefore these files have been permanently and irrevocably deleted.
We have looked at all possible solutions, including seeking the support of Amazon, but unfortunately, we have exhausted all efforts to recover the affected images, and these will need to be re-uploaded by affected customers manually.
Given the very unique nature of images used in emails, the work required to fix this is going to have to be completed by the affected customers. The team at Gecko want to help, but there's just no way for us to know what the images should be, or access them. We're on hand to help in any other way you can think of.
Whilst it's less than we'd like to help with, our team have focussed on providing affected customers with the information in the best, and simplest way possible to help identify the affected templates & files, and help focus the efforts required on the most recent of them.
This issue only affects Promote customers using the Drag & Drop email builder.
We will undertake a series of steps to ensure that we cannot get into the same situation in the future, including fully reviewing all of our data backup processes, here are some of those:
Regardless of our expectation of temporary or not, we have turned on versioning for all storage locations to prevent this from happening in the future.
We will be conducting a full review of the work the team undertook and why the risk of such work was not assessed fully, and clearly not put through QA
Audit all backup policies and ensure that we have the ability to recover data promptly in all areas of our system
Fixing this issue
As mentioned above, fixing the affected images will require manual work from the affected customers, you will need the 2 reports to do this efficiently if you have not received those already, please contact Customer Success.
This file shows all of the affected email templates, alongside the files used in each of them, this is the best file to use if you want to fully fix a specific email template - as you can focus the effort on those specific files. Also, this will allow you to focus on templates that are being used now, rather than perhaps older templates no longer being used.
This file shows all of the files affected, and the templates that the file is used in - this is helpful to see which files are used most often such as branding items, or common images, and so solving the top used files will likely have the most impact, and we have included the last time the file was used in a template to help, and the number of templates using the file.
Steps to fix
Identify the most important email templates (i.e. those just sent out, or being sent out soon) we have included dates in the report
Locate the image you are going to update, and ensure the naming is the same as the report, ensuring it is case sensitive
In Gecko, locate an affected template with a broken image, then click on Browse (rather than dragging in the image) this will open the File Manager
The File Manager can be used to upload all affected files (not just the template you're looking at)
You can then drag images into the file manager, make sure to create any folders that may have been there previously, and ensure the correct naming
Once you think you have uploaded the images, you can refresh the page and see if they are now displaying on the template correctly
If the file is used across multiple templates, then it's worth also checking another template just to make sure
Our marketing team have had to fix the emails we use for our purposes and has put together this short video explaining how we are fixing it, so hopefully this also helps you to understand the steps involved.
If any of this is unclear, please don't hesitate to contact Customer Success.