I am hired as a developer, but am I?

My name is Joe Bloggs, and I am a senior .NET developer for 10 years. I love coding! It is year 2017, I got a new job in a global company using SDL Web, and my role is to be part of the team to implement our brand-new multi-country sites.

I learnt about Redis basics, then how SDL Web 8.5 uses it. I learnt about how to make Redis secure and resilient, so I started setting things up. It took a while and finally I got an instance running on DEV, QA and Pre-Production environments. A few additional servers, expensive!

Then I suddenly became the go-to point for all Redis related requests and issues, such as patching, tweaking Redis config, monitoring Redis data and hit rates etc. etc. Plus the servers hosting Redis need to apply regular hot-fixes / Service Packs. As a result, I have been so busy with daily operations of Redis, but all I want to do actually is to develop. Now it is Year 2018.

I see hope

I learn that my new company has already been hosting servers on a public cloud, and there is already an offshore team to manage the daily maintenance of these servers. I want to put my Redis instance to the cloud and let the offshore team manage that, too. By doing that I will have the joy of coding! So, before things go to Production, at which time I would be even busier with Redis operations, I need to change the situation. I do not want to quit my job, so I go back in time.

Redis Cloud Offerings

The first thing after I came off the time machine is to check and evaluate the existing Redis Cloud offerings back in July 2017. Then I found the following main ones

RedisLabs

At the time of writing RedisLabs is the official sponsor of Redis development. It has been providing the latest features of Redis. For instance it provided high availability Redis Cloud way before other cloud providers. At the moment Redis 4.0 has been in GA for 3 days, and RedisLabs already supports Redis Modules which is only available in Redis 4.0.

In additional to hosting in its own platform, RedisLabs provides Redis as a third-party service on other Cloud providers such as Azure, Google Cloud and Amazon AWS. The company also has a services team if you would want to have full control of your Redis data and want to set up Redis in your own datacentre.

Azure

Namely "Azure Redis Cache" - it offers 3 tiers (free, standard, and premium). Free tier for development/QA while standard and premium have high availability SLA, managed by Microsoft. Data size is up to 530GB

Google Cloud

Google Cloud does not have its own Redis service but partners with RedisLabs to provide such service.

Amazon AWS

Amazon has its Redis Service called "ElastiCache". It sounds just another Redis cloud service, but when I mentioned it to my boss (in absence of 2017's me), her eyes were brightened. She wanted me to find out more even after I said lots of good things about RedisLabs to her. Then I learnt it is the company's strategy to move everything to Amazon AWS.

Amazon ElastiCache

Security

Once you set up ElastiCache, by default it is not open to public. ElastiCache is designed to be only accessible from authorized Amazon EC2 instances, so if you do not do anything else, the cache cluster will not be accessible from anywhere. I really like this feature due to the numerous cases of open Redis just because nobody bothered to change the default security settings.

From here you can start granting accesses to the ElastiCache instance by configuring security groups, etc. which can be started by following the guide here, which is actually very straightforward. For more fine-grained access controls, read on AWS VPC and subnets.

Although ElastiCache can be accessed from outside AWS as described here,  Amazon has deliberately made it difficult. Without a lot more work, your ElastiCache instance is still accessible only from EC2 servers. This is good enough and desired for Test/Production environments but obviously not convenient to support local development. A simple solution is to set up a Redis instance on your local PC (and maybe on DEV server, too), while using ElastiCache instances for Test, Acceptance and Production environments. Only things to mind are

1. Use the same Redis version locally as ElastiCache`(3.2.8 at the moment)
2. For Windows development machines, use the Windows version maintained by Microsoft as Redis does not have official Windows support
3. ElastiCache supports all security features described in my previous post such as password protection, renaming commands, using different port, etc. Enable these in your local Redis as you do in ElastiCache

Availability & Scalability

When setting up an ElastiCache instance, you have the option to specify the number of read replicas and whether to support multi Availability Zones. If you have read replicas, the failover will automatically start if the primary node fails.

There is an additional option to enable "cluster" mode. When the cluster mode is not enabled, the read replicas can be freely added or removed but there can only be up to 5 of these replicas. If there is a demand of larger data set, the cluster mode can be enabled and with sharding, it can be scaled to support up to 3.55TiB of data. When cluster is enabled, as there are greater number of endpoints (both primary and read replicas), the load of both writing and reading can be spread across these endpoints.

Monitoring and Notifications

You can use AWS CloudWatch to monitor ElastiCache performance, such as Cache Hits, Misses, CPU Utilisation, Current Connections, etc. It is recommended that for certain metrics, a CloudWatch Alarm should be created in order for a timely correction action before performance problems occur.

You can also subscribe to certain important ElastiCache events and receive notifications through Amazon SNS (Simple Notification Service). For instance when a failover process is started, you can receive an email so you can fix the failed primary node in time.

Cost & Sizing

So far this is very promising! How to cost it out then? Like EC2 pricing it is based on regions and node types with different memory size, CPU cores and network performance. It also has options on on-demand and reserved node types. Its 3-year reserved nodes hourly cost is only 1/3 or less of its on-demand equivalent. There are additional cost of backing up (snapshots) but with free allowance of 1 snapshot, and some minor data transfer cost if your EC2 instance is not in the same availability zone as ElastiCache

During development and QA stage our dataset cannot be determined as we only have some sample data. So the thought process is roughly as follows:

• We chose a single node of cache.t2.small for Dev/QA purposes. 
• We provisioned 10GB production data maximum, and want to provide some degree of high availability, but a mere 10GB cannot justify data sharding
• We want to regularly back things up so these operations need additional memory. The application is also write-heavy, and according AWS recommendation, we need to double the size = 20G
• We chose 3 nodes (with 1 primary and 2 read replicas), and instance type is cache.m3.2xlarge
• We calculated the cost based on the above, got the budget approval, with the following conditions
  • After production data is ready and in operation, we would monitor the memory and CPU usage, and adjust the instance type accordingly
  • After the adjustment, we would change the payment model from on-demand to reserved to further save cost

I then sent the offshore company with the above request by raising a support ticket with them, and within a day, the request was completed. I installed Redis Desktop on one of the DEV EC2 servers, and it connects like a charm!

Now my mission has finished. We've got a production-ready Redis instance on Amazon AWS, operated by the offshore team. Time to travel back to Year 2018 and ....

Happy Coding!