a system incident has impacted BidTheatre's platform and our services.
Background: Starting on the evening of Saturday 11th if April at 22 CET, our adserving cluster was able to only partially release collected statistics on ads bought and served to our data backend. This was not detected and alerted upon by our platform monitoring, and the issue was not detected until 11 CET Sunday 12th of April, after which an investigation into the problem ensued.
As of 13 CET Monday 13th of April, fixes have been put in production and operation is restored to normal. We are currently closely monitoring, and will follow up with more information if needed.
Cause: The problem was caused by a misconfiguration that limited the capacity of our database backend to receive data. Combined with recent changes to our database tables, we hit a capacity limitation that was not foreseen.
Effect: All missing data has been restored from a backup system. However, some campaigns have been paced poorly during the time of the incident, particularly during the morning of April 12th. We will reach out to you if your campaigns have been impacted.
Resolution: A number of measures have been taken to mitigate the risk of similar future incidents: 1) Application configuration has been altered to improve the throughput to our database backend. 2) Database tables have been reconfigured to improve the capacity to store data. 3) An alerting system is currently being implemented to notify our platform operations team if a similar incident would occur again.
Timeline: Incident started: Sat 11 Apr @ approx 22 CET Incident detected: Sun 12 Apr @ approx 11 CET Incident fully resolved: Mon 13 Apr @ approx 13 CET
Although we are doing everything we can to keep the BidTheatre platform robust and available 24/7, incidents unfortunately occur. Our job is to make them as few and far between as possible. We hope for the continued trust in delivering your marketing campaigns, thank you for your business!
Kind regards, Marcus Johansson, CEO
Posted Apr 13, 2020 - 15:34 CEST
Identified
The issue has been identified and a fix is being implemented.