Peter Marklund's Home

I have been doing an interesting project at work where we replicate (sync) data in realtime (with say 200 ms latency) from our CMS database (MongoDB) to file storage (AWS S3). The motivation for doing this is to achieve higher reliability/uptime and scalability than you would typically achieve with a REST API in front of a database like MongoDB or Postgres. For our use case a typical CDN setup with a 60 second TTL (Cloudwatch in our case) in front of the API didnt fulfill our requirements and didnt get us good cache hit rate. I think using something like Varnish/Fastly where you set a long (or even infinite) TTL and expire the cache when data changes and/or are able to have the CDN serve stale (i.e. using stale-while-revalidate, stale-if-error Cache-Control directives) could have solved some of this for us. But its hard to compete with S3 when it comes to reliability and scalability. I think this approach could be relevant to other projects and companies as well. You can find more details in these presentation slides.