Uptime November 2018

Uptime report for the past month:

api.rehtumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/345532

dashboard.rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346471

rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346472


Main Events

No events to report.


Notes

๐Ÿ—„๏ธ ๐Ÿ’พ Storage migration:

Changes in our storage infrastructure during the past month.

  • We are in the process of consolidating our storage and are now using Backblaze B2 as archive for older thumbnails.

  • Our main storage is now located in Amsterdam using Digital Ocean Spaces.

  • Images in older providers (Amazon and Dreamhost) are being moved at the moment.

  • We have been testing Minio for local cache and it is working great.

Uptime October 2018

Uptime report for the past month:

api.rehtumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/345532

dashboard.rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346471

rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346472


Main Events

No events to report.


Notes

๐Ÿ–ฅ๏ธ โœˆ๏ธ Servers migration:

Some changes in our infrastructure during the past month.

  • We moved our storage servers to Europe. This didnโ€™t have any direct impact on our users.

  • Server migration is almost completed.

Uptime September 2018

Uptime report for the past month:

api.rehtumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/345532

dashboard.rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346471

rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346472


Main Events

Some details on the main events of the past month.

๐Ÿงฏ๐Ÿ”ฅ dashboard.rethumb.com down for 4h 55m:
  • As reported in Uptime August 2018 we had some issues with the machine running dashboard.rethumb.com. These issues are now solved.

๐Ÿงฎ๐Ÿงช API Performance

  • The following charts show the maximum execution time for a new image request (without using the CDN and not hitting any local cache).

August 2018

September 2018

  • These values should be as low as possible but we can see a significant improvement from August to September as the values have less variation.

  • For now our primary goal is to avoid spikes as these occur when the system is under high load and our system is scaling up.

Uptime August 2018

Uptime report for the past month:

api.rehtumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/345532

dashboard.rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346471

rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346472


Main Events

Some details on the main events of the past month.

๐Ÿงฏ๐Ÿ”ฅ dashboard.rethumb.com down for 45min:
  • The machine running dashboard.rethumb.com had a drastic performance degradation and we had to move the application to other host.

  • This issue started in August but it was solved only in September.

  • This issue was similar to the one reported in Uptime February 2018.


Notes

Some changes in our infrastructure during the past month.

๐Ÿ–ฅ๏ธ โœˆ๏ธ Servers migration:

  • We moved our main entry point api.rethumb.com to servers in Europe.

  • The impact should be mininal and transparent as the entry point is still behind CloudFlare cache.

  • This also impacts our clients using the cname feature.

Uptime May 2018

Uptime report for the past month:

api.rehtumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/345532

dashboard.rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346471

rethumb.com
http://www.uptimedoctor.com/publicreport/vi083t3k/79788/1/346472


Main Events

May 15, 2018 - Object Store Incidents

We had anoter issue with our main object store provider. As usual the system uses the backup service but now we can do a fast switch and start using a new primary service without downtime.

Incident:

https://status.digitalocean.com/incidents/ql22cg0mzsj4

Our Twitter message:


Notes

  • We migrate all our queues from Beanstalk to RabbitMQ. In the future we would like to write a blog post with our experience and why the made the change.