What is SQS and when we use it?
Amazon SQS is a web service that gives you access to message queues that store messages waiting to be processed.
You can use it to:
- Build a microservice architecture and use message queues to connect your microservices.
- Keep notifications, that need to be send
- Make applications more flexible and reliable
- Create work queues with each message as a task that process can complete
Managing AWS SQS through your .NET Solution project can be sometimes tricky thing to do. If you are in the situation where you:
- need to upload multiple items at once (for example: Send push notification to your users or update multiple item in your DB)
- are not sure how to handle the possible AWS Container fails
- are not confident whether there might be duplications of items.
You can easily follow the below steps in your .NET Core application and prevent it from failures.
How to handle Multiple Uploads?
There are couple of things you have to put in your application, to handle multiple uploads properly.
First: Ensure that you API endpoints support collection of items.
This step is crucial for the whole process, since if the endpoint does not support collection, you will not come accross one of the struggles above. This is the handling of multiple items at once. A perfect example of a situation when you will need this support is:
- When you want to send to all your clients that there will be Black Friday next Friday and all the items will be on half of their price. The endpoint might look like this:
SendMessageToAllClients(List<ClientModel> clients, Message message);
The ClientModel includes id and name of the client, and Message – the text you would like to send.
Second: Push the collection to AWS SQS.
You need to separate the collection into chunks of 10. AMAZON SQS currently supports maximum this number of items to be enqueued on a single action. Once you are ready with the chunks, you can send SendMessageBatch method using the AWS SDK for .NET Core. The final step is to open your AWS Console, navigate to SQS and review your queue.
Third: Create a task that will run on the background and pick up the enqueued items.
The creation of the task itself is not the matter of this blog, but the operations and endpoints that need to be called are. You can set the task to execute each number of seconds, depending on the needs of the application. On each of the executions, the task will call ReceiveMessage AWS Method and will perform certain amount of actions before sending the message to the client. As was the case with SendMessageBatch, the ReceiveMessage also supports up to 10 messages in one action. Once you finilize the operation, you can delete the message from SQS by calling DeleteMessage AWS Method.
How to handle Container Fails?
The answer to this question actually lies in the usage of SQS. In the previous section you noticed that we separated enqueue and dequeue operations from one another and there is no relationship between them.
Enqueue is responsible only to format the request and push it to the AWS SQS. If the service/container fails at this point the end user will be aware that the message has not been sent.
Dequeue is responsible only for picking up items from the AWS SQS and send them. If the service/container fails at this point, the dequeue operation will start once the service/container is up and running. The failure of the worker will not result in crashing the service.
This is how you handle the container/service fails. The popular sentiment here is: “Divide and Conquer”. By dividing the enqueue and dequeue operation, you are able to handle the fails separately without resulting of lost or corrupted data sent to the client.
How to handle multiple agents and the repetition of data sent?
If you recall from the How to handle Multiple Uploads, we mentioned that there are couple of things that you should perform prior to sending the message on the fly. You should set proper MessageVisibilityTimeout in your ReceiveMessage method parameters. This is extremely important, because this actually hides the received messages for predefined amount of time and no other service can pick them up for further processing. Choose wisely the number of seconds for Visibility Timeout, since your application might need to call third party server to perform the desired task. If you set too short timeout, you might come into situation, where your service is still processing the message, but the visibility timeout has passed, and another service or task has picked up the same message.
Last but not least, be sure that you have Deleted the message once it has been processed, in order not to encounter the situation from above.
Technologies Stack:
