AWS re:Invent 2019: Scaling Hotstar.com for 25 million concurrent viewers (CMY302)

0:00 Hi everyone. Thanks for coming. So my name is Gaurav. I'm a cloud architect at hotstar. And today I'm going to talk about an interesting story. How we scaled for more than 25 million users concurrent viewers on our platform during the recent ICC World Cup. And in my personal time, I also lead the AWS user group Mumbai and I'm AWS community hero from India.

0:32 Before we begin, quick show of hands. How many of you follow cricket or understand the game? That's cool. And anyone from New Zealand. Cool. So a bit about hotstar. We are the number one OTT in India. We do live streaming, watch streaming. We have sports, TV shows, news. And today I'm going to talk about the third point. That is one day more than 100 million unique users.

1:06 So this was during the recent ICC World cup wherein we had a game between India and New Zealand. On that day our platform had more than 100 million unique users. And at a given point we, we clogged almost 25.3 million which is a global record for concurrency for any kind of live streaming. The graph that you're looking at is the actual traffic pattern from that match.

1:39 The game was spread across two days. So usually it's a one day international. So it's played for 50 hours each side and usually gets completed in a day. But this was a rain affected match. So what happened was the match was postponed to the second day. That is why you see day one and day two. So New Zealand was batting first. And the interesting scene to see here is the ramp up, the initial ramp up that you see when the match started.

2:10 It's almost like 800k to 1 million users per minute. So that is the rate at which users were getting onboarded on the platform. And then the graph that you see during the New Zealand batting, you see the concurrency is almost about 10 million for most of the time. So this shows how resilient the platform is. It's not like one of the event wherein you have the concurrency of the platform going up for one or two minutes, but it's for a stretch duration.

2:40 It's almost like four hours, six hours. We were operating at about 10 million. And the 13.9 million number you see is the peak before it started raining. So we were at 13.9 million. New Zealand was batting and that's when rain started. And then we saw drop. The interesting thing here is so even when it was raining, right, there was no cricket being played.

3:10 It was just highlights and some commentators talking about the game. But still around 4 to 5 million people were stuck to the screen waiting for game to begin. And this is like one of the kind of event that we have seen wherein there is no cricket being played. But still the viewers were like so glued to their screens. They were waiting for rain to stop and the match to begin that there were like 4 to 5 million people almost for 4 or 5 hours waiting for game to start.

3:42 When finally the match was called off for the day, it got spilled over to the next day. So the initial spike on day two that you see is few overs that New Zealand had left. They came down to bat. I think it was just for 4 overs. And after that India came on to bat. The second inning started. The spikes that you see, the sudden ramp up and the dip is what can kill most of the platform.

4:10 We are Talking about adding one 1.2 million users every minute. So you can see how quick the spike goes up and comes down. The interesting thing about the downward spike or the drop is it is scary. You will say that, okay, only traffic coming to your platform is scary, but even a sudden drop of this scale can kill your backend or your applications. And then India continued playing and there were regular wickets in the match.

4:48 Then everyone lost hope that okay, India will not win. And we saw some traffic going away. Another thing I missed showing was the sudden dips that you see in between both the innings. If you are a cricket follower, those are strategic timeouts in IPL and drinks break in cricket, which usually happens in the 16th or the 32nd over. Then after the regular wickets were falling, what happened was Dhoni and I think Jadeja, these people picked up the game and started playing really well.

5:21 And that's where all the viewers started thinking that, okay, now India might win this game. And this is like one of the kind of spike we have seen in our platform wherein we were adding more than 1.3, 1.4 million users every minute. So within a short span of, you can say 10 to 12 minutes, our concurrency went from 13 million to 25 million. And that's like a global record. 25.3 is a peak when the traffic was at 25.3.

5:55 At that moment, Dhoni got out. And that's wherein all the viewers like lost their hopes because they knew, okay, now India cannot win. And unfortunately, like for Indians, New Zealand won that day. So we were also going through this phase of mixed feelings wherein you have created a global record, right? You have done something that no one else has done in the world, like streaming concurrently for 25 million users. But at the same Time your country lost the game.

6:26 So you also have to cope with that. You cannot celebrate that. Okay, you have made a world record. All right, let's talk about scale. So we spoke about the India New Zealand game. More than 25 million users. On an average, we do 1 million requests per second. And we consume more than 10 terabytes of bandwidth when we stream video.

6:57 To give a perspective into this number, this is almost 70, 75% of India's Internet bandwidth available. And when we do high concurrency matches like this, we almost consume 70% of the Internet bandwidth. And people will ask me like, why is 25.3 million a big number? So this will add some insight into it. So before we like created the global streaming record for concurrency, the biggest player in the market was YouTube.

7:34 It was in 2012 when YouTube created a global streaming record, there was a Red Bull event space jump when 8 million people were watching that live. Between 2012 to 2018, there were many events. There was royal wedding, there was Donald Trump inauguration, which I think was around 4.3, 4.7 million. There is Super bowl which usually clogs around 3 million. But none could break the record that YouTube had set until IPL 2018 happened.

8:07 And that time we created a record of 10.3 million. This year during IPL, our peak concurrency was 18.6 during the final between Mumbai and Chennai. And then in World cup we broke our own record and reached 25.3 million. So if you look at the closest competitor, it's almost like 3x of what they have.

8:40 So how do we prepare the platform to scale so much and what's the engineering effort that goes behind this? So we conduct a lot of game days and this game days are like very massive scale. We use C5.9X large machines. This has like 36 CPUs and 72GB of RAM to generate the load for 40 million 50 million concurrent users. We use upward of 3000 machines. C5.9X large, which has a total of like 108K CPUs, 216 TB of RAM and this load test are so massive, it often happens that our CDN partners or the cloud vendor that we work with, often because in a public cloud, you, your network is shared among all the customer.

9:29 So what happen if you load test a bit more than what it can handle? You can even impact other customers. So being mindful of that, we started seeing a lot of issues when we were doing load testing. Then we moved to a strategy wherein we geo distributed our load. So instead of generating load from A single region. We generate it from eight regions to to avoid an edge location or any availability or network disruption for other customers.

10:03 And during this load test we generate around 200Gbps of network out. So that is very huge. And in the past it has happened that during our load testing we have overwhelmed the edge location or pop location from where our traffic was flowing that it has caused an outage as well. Apart from that, we have an in house project called Hull which is an to go solution for all load testing things. It's a homegrown tool. There are Scala scripts, Python scripts which generate this load.

10:36 It also performs chaos engineering, performance testing, then tsunami testing and the concurrency graph that you saw in the first slide. The data from those matches or similar matches can be fed into this tool like the project Hulk and it can mimic the traffic pattern of any match. So let's say I want to replay the India New Zealand game when we saw there were like sudden spikes and sudden dips.

11:07 I can feed that data into this tool and it can mimic the same kind of traffic pattern at whatever concurrency I want. So if I want to load test for let's say 50 million or 60 million, we can do that using this. This is what the load generation infrastructure look like. Very simple, nothing fancy. Load generation grids, all C5NANU, large spread across multiple regions, goes over your Internet or CDN.

11:38 Then you have load balancers, auto scaling and EC dimensions. We use containers as well, self hosted K8s cluster. The other thing about this diagram, the load balancer piece, our traffic and the requirement of like per application. The number of tps, RPS it should support is so huge it doesn't fit in a single load balancer. So usually we use 5 to 10 load balancers pre warmed at their maximum limit.

12:10 And then we use either weighted routing, we sharp the load balancer basically to handle the scale. Single load balancer cannot handle the traffic requirements that we have. And another thing I miss pointing out in the graph that we saw earlier, the sudden dip from 25.3 million to the base traffic, I.e. 3 million 4 million is very scary in the sense you have to think from a developer perspective that when you are streaming video you are making less number of API calls, you are just requesting your video file metadata file and there will be like X number of API calls that you will make while viewing a video.

12:53 But when out of 25 million, let's say 90% of the users click your home button. Now you are landing on the homepage. And homepage has many APIs you have your content platform APIs, you have your master APIs, you have personalization engine, recommendation engine, because it's a streaming app, you have personalized content and your watch history or your favorite content or recommended content will be different than mine. So there is very limited scope to even cache those calls.

13:27 And most of this also comes to the backend. So in an event like where you have 25 million people on your platform and suddenly all of them come to the homepage, it's very scary because it can kill the backend. Like if you have a content management system which is showing all the clips and the trailers or the metadata for it. If that is not scaled up to handle a sudden surge of request when people move from watching video to homepage, your app can go down.

14:01 So that's the thing. And it's a reassurance time because we are streaming this match live. Cricket is being played in a stadium and you have to stream it live to all the users. This is not a VOD content wherein you can cast the content or apply some gimmicks just to improve the experience. But everything needs to happen at that moment. You cannot wait for your capacity or EC2 to scale up.

14:33 And then there are a bunch of other challenges as well. We'll look at it. So why is it like a race against time? Because we are looking at adding more than a million users every minute. And an application it takes roughly around a minute to boot up. Let's say you have to add more capacity into your backend. If you take an example of simple EC2 machine, the moment you request an EC2 machine, it will take some time to launch the server.

15:01 Then the auto scaling group will either attach it to a load balancer, then your health checks will pass though finally the instance is healthy and will start serving traffic. So no matter how much you optimize, it will still take a minute or so. And by that time, if your backend is not scaled up to handle that kind of load or you don't have buffer, you will add bottlenecks in your systems. Your new users that are coming to your platform won't be able to enjoy the game or have a good experience.

15:34 Even the reaction time is very less for us, usually around 90 seconds before we either make a scale up decision or take a decision to degrade a particular service. And push notification is so during a live match, if there are interesting moments wherein player is about to hit a century or someone is hitting three sixes, the marketing team usually sends out notifications to user base of 100, 150, 200 million users.

16:04 Even if we talk about conversion of 1 or 2%, we are still talking about adding 2 to 4 million users in a very short span of time. So if the marketing team is sending a push notification out and my backend is not ready to handle certain onboarding of 2 million, 4 million users, that can also like kill my platform or degrade the customer experience in that moment. And we also use fully baked amis.

16:36 Even in container, we use fully baked images. Container images. This is because we don't have times to use any configuration management tools like Chef Puppet or add any post boot up script because it adds delay to your scale up. We are talking about scaling from 400 to 4,000 servers, provisioning of those many EC2 instances and then they are server to become healthy pass the health check takes a time.

17:06 In all this, if you add configuration management it makes the matter even worse. And these are a few of the reasons why we don't use the traditional auto scaling that AWS provides. So we in the part where we operate we see a lot of insufficient capacity errors, the ICE errors. So this is usually when you request a large amount of compute capacity and a particular instance type is not available either in one availability zone or in the entire region.

17:43 So in a live match, when I want to scale, my traffic is like getting greased and I want to scale. If there is no servers, it's an issue for us. Right. And the other problems are so a single auto scaling group. This is way before EC2 fleet and AWS auto scaling came into picture. A single auto scaling group only supports one instance type. Like you can add one instance type in a launch configuration, attach it to a asg, but what happens if that particular instance type is out of capacity?

18:17 You have a live match going on. Now you want to scale up, but if you are limited by that capacity, either you will choke or like bottleneck your backend instances or it can even cause an outage. The third problem is the step size of asg. So this is an interesting one. And usually people who have worked at scale or you have a lot of traffic coming and scaling requirements, they will be able to relate with this.

18:48 So what happens is when you increase the desired capacity in an ASG from 20 to 100 AWS, like its internal algorithm, whatever the ASG has tries to add few servers and tries to balance it out in all the Availability zones. So it doesn't goes from let's say 20 to 100 directly it will first add 10 servers like 20 to 3030 to 40. This is very slow. If we are Talking about adding 10 or 50 servers, like a step size of 10 or 50 scaling from 400 to 4000, 5000 servers can even take half an hour in a live match.

19:27 That is simply not acceptable. Even if we engage with our TAMS account manager or the enterprise support and increase that limit, what happens is you are creating more problems. Because if you increase your step size to 100, 150 or whatever is the maximum limit. Now you're making more API calls. So these API calls are your describe EC2 instances, describe Stack, add to load balancer. So all the API actions that goes in behind the scene to launch a server and attach it to a load balancer consumes a control plane API, right?

20:04 And there are limits to that, like how many calls you can make in a minute. Some of them are hard limits and we have to operate within those boundaries. So simply increasing the step size to reduce the scale up time is not an option because it adds more complexity. If your API calls are throttled, your EC2 might not be able to read the described tags. If you have an application that reads something from your tags while booting up, that application might not even become healthy.

20:36 Because now when it tried to make a describe API call, it was throttled. And though you can add retries and back off, it makes the matter worse because it adds delay. When you want a server to be available in less than two minutes, you cannot have complexities around that. And next one is the same thing, retries and exponential backup. This happens with auto scaling group as well. So what happens is I'll explain this and the game of availability zones together.

21:11 So this is a reference to got because all of us at odds are like huge fans of Game of Thrones. What happens is when you scale up your order scaling group, let's say you increase your desired capacity from 50 to 200. Like I said, it tries to balance between Availability zone. Let's say 1 of the availability zone doesn't have enough capacity, so you increase the capacity in AZ1A. Let's take an example. For AP Southeast1, you have three availability zone, 1A, 1B and 1C.

21:45 You try to increase from 50 to 500. 1A successful. 1B is successful. In 1C you got ICE error, insufficient capacity. Now what will happen is since it tries to balance the capacity equally, even if there is no capacity available in 1C, it will still try in every attempt to launch servers there. And what happens is, if it unsuccessful, it increases the delay, the retries, it Increases your time.

22:16 So earlier for 1C it will check every 30 seconds. If it fails four times then it will try to attempt to launch every one minute or five minutes. And we have seen sometimes it almost goes to 25 minutes for auto scaling group to scale up. If sufficient capacity is not available in one of the easies and this is something that cannot be controlled by a customer because this is an inbuilt algorithm, you can remove that subnet to speed up the scaling.

22:49 But all the healthy instances which were launched or are already serving traffic from the particular az, if you remove that subnet those will also get terminated. And again like during a live match doing this operation becomes very critical. So how we do scaling is we pre warm the infrastructure before the match starts. So that gives us ample time to find out any issues or if there are any capacity related errors which we face while scaling.

23:23 And the normal auto scaling or the cloudwatch alarms that you can set up on top of ASG or your ELB usually is like people do CPU based scaling but for us CPU based scaling simply doesn't work. We do request based scaling or our platform concurrency. So application like the CPU of an application can be at 70%, 80% or even 85%. If that CPU usage is not hurting my customers or impacting an increase in latency, I should not be bothered to scale upon CPU like that will be a wrong metrics to scale upon.

24:05 So instead what we do is we find out what is the rated capacity of each applications, how many requests it can handle and basis that we take a decision, how many number of servers or how many pods we need to sustain or like handle a current platform concurrency. So platform concurrency is total number of active users watching video on your platform. So we have a ladder, we have a grid that at each concurrency ladder, let's say at 5 million 10 million, application A will have 10 servers, application B will have 20 servers.

24:41 So we have the entire matrix and then we have our own automated scaling. So whenever platform concurrency reaches a particular number, it automatically scales up all the application and for you can say backup option. We also have request count based scaling. So let's say the platform concurrency is at 5 million. And certainly an application due to some error or misconfiguration is getting more requests than expected.

25:13 So what happens in that case is the request scaling triggers and it will automatically rescale the app to the best possible value taking into account what is the current request rate that the application is handling. And all of this is automated. No human intervention. We also use EC2 spot flight because this is to overcome the challenges of single instance type. Because if you can use EC2 plot spot fleet, what you can do is you can diversify your instance type.

25:45 A single spot fleet can support up to 50 different instance type. So we use around six, seven of them spread across multiple availability zones. So that gives us more number of options to get the compute capacity that we require. And if you were in this room before my session there was a good talk on chaos engineering and these are a few of the ingredients that can cause an actual chaos at hotstar.

26:20 So one is push notification. We already spoke about it, how the marketing sends bulk notification to 150200 million users. And if my platform is not scaled up to handle that kind of search, it can bring an outage or like it can cause a disruption in one of the service. Tsunami traffic is the chart that we saw in the first slide. The sudden spike and the sudden drop. If you're not scaled up for such events or you're not expecting such a harsh spike, that can cause a bottleneck at any level, your cache level, your data store levels.

27:02 The next one is delayed scale up due to any reason like capacity not being available or let's say health checks are taking more time. If the scale up has even one minute or a two minute delay, it adds a lot of load on the existing servers that are serving the traffic. If let's say 100 servers are serving application A, but the need is 150 servers. If those 50 servers are not provisioned within a time of two or three minutes, then it adds more load on the hundred existing servers and they might even start throwing 5xx errors.

27:39 If like the backend is over capacity, we also have to deal with the bandwidth constraint both in terms of API and video bandwidth. Because this is something that you cannot auto scale. This is like physical infrastructure, right? CDN providers or the ISPs that the users are watching the match from, these are hardwired fiber optics or your Internet wires which cannot scale up.

28:14 Like we already saw that we consume more than 10 terabytes of bandwidth. If there is a need to use more of that, and if our partners or like any CDN provider is unable to use that thing or provide more bandwidth, then we are limited by that number. So what we have to do is then we have to think a different approach wherein you have to see what available bandwidth you have and how best you can accommodate or serve more users.

28:47 Increased latency adds a cascading effect throughout the app. So let's say you're one of the application content platform has an increased API latency. And there are other applications like recommendation engine or personalization engine which depends on input from content platform, they will also see a spike in their latency. And these applications are part of your homepage calls or your. When you open an app, these are the API calls that you trigger.

29:19 So what happens if your application startup time will increase? When user clicks on your app, it will take time to open, then they click a video or they move to a different page that will take time to load. So even a small increase somewhere in the system has a cascading effect throughout the application. And through all this game days and chaos engineering that we do to make up as platform resilient, these are few of the things that we usually look for.

29:55 What is the breaking point of each system? So you have to think of it from this way that when you open an app you need to understand the user journey wherein you have a homepage. What are the different API calls you will make if a user clicks on a login or a signup button? What are the API calls that it will make if you understand the user journey, your user journey of application or a website? You can then think from the chaos point of view.

30:23 What are the bottlenecks and what are the layers where problem can occur? What is the breaking point of each system? Because you will then get okay, application A can only support, let's say 50,000 requests per second. So if I'm reaching around 35,000 or 40,000, I'll try to either offload or put that application in panic, thereby reducing the load. Or have some buffers scale up in advance. Those are the tricks that you can apply if you know the breaking point of each system.

30:58 Death wave is a sudden spike that we saw in the first graph. This can really bring down your platform if you are not scaled up for it. And the other things that we try to find out is bottlenecks and choke points. This can be at any layer. This can be at ELB level, your RDS, DynamoDB, whether you have proper limit set, whether they are scaled up for Peak, because EC2 and all, you can scale on the fly, right?

31:26 You can increase the capacity. But think of elasticache rds, these are not that elastic. Like you cannot scale them up and down multiple times in three or four hours. And in few of these operations there is also downtime associated if it tries to switch the DNS endpoint, it's like very small downtime, but still it's very noticeable at our scale and then we also try to find out hidden patterns.

31:56 Because when you do a normal load test, let's say for 10 user, 10,000 users, you might not discover some of these issues. Some of the issues are only associated with scale. So unless you load test for 10 million, 20 million, few of the problems never occur. You will not see them in a normal load testing or any kind of performance test that you do. So some of these are like from the issues that we have faced in the past, we try to mimic them again.

32:28 And the failures can occur at any level, your network level and application level. So we try to look at breaking point of all single point of failure as well as if an application has a maximum rate rate limit which it can operate at. The other thing about chaos is we also look for things like, wherein an application is about to reach its rated capacity, we try to cut off or reduce the load.

33:00 We do this using panic mode, which I'll talk about more. And. Yeah, this one. So panic mode is something wherein because all this action has to be taken live in the while the game is going on, you don't have like 15 or 20 minutes to troubleshoot an issue, fix the problem. So what we do is we rely heavily on panic mode. What this does, is it?

33:35 So the goals are your P0 services should always be up. So for us the P Zeros are streaming video, advertisement, payment, subscriptions. So all these are critical services. So what we do is if there is a limited number of API calls or we are limited by API traffic that can go in, we turn off non critical services which are not essential in that moment. If 99% of our traffic, 99% of the users are watching cricket match, then there is, in that moment there is no need to show recommendation or personalization content because you know that okay, majority of your users are only going to watch cricket.

34:17 So you turn off those services temporary by temporarily and make room for the core services so that they can scale up more. And another thing that we do is graceful degradation. So what we do is at every ladder like at 10 million, 15 million, we have bunch of action. We gracefully degrade certain services to make room for critical services. And this is both client side and backend level.

34:47 So clients are intelligent enough to understand that okay, our service is in panic, so it doesn't send requests to them. Mostly we saw a panic JSON through which the client Android iOS application understand that, okay, certain application is now in panic mode. So they won't make a call and they won't even show error to a user because you don't want to spoil the customer experience. You want to degrade gracefully. Because without showing error message to your end users and turning off critical services, we do it in a orderly fashion at every 10 million, 15 million.

35:33 So I'll give you a practical example. Let's say you have a user logging system, right? And logging management system. You enter your username, password. Let's say your RDS which is now supporting that application is down for some X reason. You don't want to show user that okay database error or unable to connect to database or timeout error. Instead you allow you put the application into panic. You will now allow user to watch your content or like log into your platform without even using the login system.

36:06 So as a customer you come to the platform to watch cricket or any videos, right? You your experience should not be degraded in any way. Similarly, for payment application, if you have a subscription service and you want to due to like an error at maybe elasticache or database, you are not able to complete the transaction. Instead of showing a payment failure, you can do a batch reprocessing in the backend and allow the user to watch the content at that time.

36:42 You can like figure out in the backend if it was generally a case of error or like at your end or the payment gateway level or it was something at user level. But in the spirit of the game and in the interest of the customer who came to the platform, we apply this panic mode especially when we are serving to more than 20 million users to improve the customer experience.

37:16 And these are a few of the key takeaways. So always prepare for failures. Load testing and chaos engineering can help undiscover all those issues and help you prepare better understand your user journey. This is very important because once you know the flow of traffic request. Once you know that, okay, when a user clicks a home button or opens your app, what are the bunch of API calls that he will make? You can design your system, you can scale your application well in advance.

37:50 It will give you an insight into how much will tapping a home button cost me. Like in terms of capacity required and the pre warming required and the limits that needs to be scaled. It makes you more prepared for a tsunami or a search. And it's okay to degrade gracefully. Like I gave the example of the login system and the payment system. It's okay to degrade temporarily then showing error. Once your problem is fixed, you can re enable those application for the customers.

38:32 Cool. That's all I had for today.