All Systems Operational

About This Site

Welcome to BidTheatre's system health page. Here you get current information about any ongoing incidents or planned maintenance. Subscribe to get updates by email.

Adserving   Operational
90 days ago
100.0 % uptime
Today
Bidding   Operational
90 days ago
99.01 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Adserving Error Rate
Fetching
Bidding Error Rate
Fetching
Past Incidents
Jan 24, 2019

No incidents reported today.

Jan 23, 2019

No incidents reported.

Jan 22, 2019
Resolved - This incident has been resolved.
Jan 22, 11:51 CET
Update - We are continuing to investigate this issue.
Jan 22, 11:50 CET
Investigating - You may have noticed, that the BidTheatre DSP administration interface is currently unavailable. In addition to this all bidding is halted.

Our platform team is working with the highest effort to restore services, and we will keep you posted.

Please note that adserving is unaffected and fully operations.
Jan 22, 11:05 CET
Jan 21, 2019
Resolved - Dear client,

On Fri 18 Jan we had a severe system incident that affected BidTheatre services. This incident report details what occurred, and what measures have been taken to guard against similar futures incidents.

Summary:
On the afternoon of Friday the 18th, an intrusion was made on our database backend, consisting of a master / slave replica set. The replicated setup allows for fast failover, should something happen to the master database. However, the intrusion rendered both master and replica unusable, and we had to resort to off-site backups to restore the database service. All-in-all, this caused most services to be unavailable during Friday evening to mid-day Saturday.

Timeline:
Fri 18 12:14 - Monitoring indicates that standby replica database is unaccessible
Fri 18 15:00 - DB replica is established to have been intruded upon
Fri 18 16:00 - Intrusion is deemed to likely have been possible through a weakness in the Microsoft RDP protocol for remote access
Fri 18 17:30 - Clean system setup of slave is initiated
Fri 18 18:15 - Unplanned maintenance of master db is initiated to secure it from the same vulnerability as the master
Fri 18 18:20 - An incident issue to inform clients is created
Fri 18 18:55 - Master db is established to have been intruded upon. Clean system setup of server is initiated
Sat 19 09:50 - Backup restored on master db
Sat 19 18:00 - Full replication of database master - slave is restored

Impact:
Bidding service: Halted from Fri 18:15 to Sat 19 13:10
API service: Unavailable from Fri 18:15 to 10:55
Administration UI: Unavailable from Fri 18:15 to 10:55
Adserving service: Operational

Data loss from approx 16:14 (time of last backup) to 18:20 on Fri 18 (time for bidding off). Please consult with our platform team at operations@bidtheatre.com if you have experienced data loss, and we will try to remedy this to the best of our ability.

Measurements:
The intrusion was facilitated due to a weakness in the Microsoft RDP protocol used for remote management of servers. We have since the attack closed RDP access, and will resort to other ways to control the servers remotely.

Server patching with latest software updates has been upgraded to be done within maximum three days instead of weekly.

Comments:
We are deeply sorry for this incident, an outage of this sort is simply unacceptable. We have made it a first priority to employ state-of-the-art security measurements such as two factor authentication, encryption, anti-virus software, along with documented security policies and other procedures to enforce security. Nevertheless, the work to guard Internet services against malicious entities is an ongoing and never ending one.

We are committed towards offering an uninterrupted, always-on 24/7 service of programmatic advertising, and I sincerely hope that we will have your continued trust in doing so.

Best Regards
Marcus Johansson, Founder & CEO BidTheatre
Jan 21, 19:05 CET
Jan 20, 2019
Postmortem - Read details
Jan 21, 19:07 CET
Resolved - This incident has been resolved.
Jan 20, 15:04 CET
Monitoring - All systems are back online.
Jan 19, 13:22 CET
Identified - We are experiencing a system issue that affects bidding. Our platform team is working with the highest effort to restore services, and we will keep you posted.
Jan 18, 19:01 CET
Jan 17, 2019

No incidents reported.

Jan 16, 2019

No incidents reported.

Jan 15, 2019

No incidents reported.

Jan 14, 2019

No incidents reported.

Jan 13, 2019

No incidents reported.

Jan 12, 2019

No incidents reported.

Jan 11, 2019

No incidents reported.

Jan 10, 2019

No incidents reported.