Scalable Bulk Email Service
Recently, I worked on a feature where we had to send hundreds of thousands of email to our users in a very short amount of time. Sending email is not a complicated task by any means but doing it at scale certainly was.
We leveraged number of cloud services to build our system. They are:
In a nutshell, the application is divided into multiple 2 basic components.
A producer app(function) which is responsible for producing necessary data to send email and kicking off the emails.
A consumer app which sends the actual email from the data it received from producer and maintains the progress state of sending emails.
The communication from producer to consumer happens via Azure Service Bus.
Let’s look at simple flowcharts for producer and consumer:
In our architecture, we are using multiple separate function apps. The reason for doing this is because we could in theory scale each function according to the workload. In practice only the consumer app needed scaling.
The bulk of the complexity of the system is handled by Azure Service Bus. It is in-charge of the task queue, creating additional consumers instances according to our load and handling error conditions.
Our previous implementation for same feature, ran on a single instance of nodejs on a VM. Sending same number of emails took more than an hour to complete. Using appropriate could api’s and updating our architecture, we are able to send the same number of emails in less than 10 minutes(usually 5).
We are pretty happy with the result we got after the switch.
Final thoughts on Azure
It was my first stint using azure and I love it. Azure has a lot of small features that makes developers life easier. Here are a number of things I love about azure:
- VsCode extensions: The extensions does half of your job. There are tons of templates. Usually you just need to tweak the appropriate template code for your needs. This is a huge time saver.
- Documentation: The documentation in azure portal is great. It’s short, to the point and informative. The basic examples in documentation gets you started in no time.
- Local Development: All the cloud functions I used in azure were available locally. Its a boon for frontend developer like me who spends most of their time on developing software locally. Having local setup eliminated lots of configuration issues that might slow you down specially if you are new to a platform.
Note: I am not associated with Microsoft in any way. These are my thoughts as a happy user of azure platform.
Special thanks to Robus Gauli for mentoring me in azure. You rock.
