Published on: 01.04.2019
I understand RESTful web services, or at least I think I do.
I agree that when you have huge teams and code base it makes sense to cut them in small independent pieces and connect them via queues and HTTP.
Collaboration on large software projects is hard and problems are increasing exponentially with a number of people added to the project.
The tradeoff is that the overall speed of your software will decrease (because of HTTP networks calls), but you will get software system that can be maintained and new features added without the need of understanding/changing/impacting whole system.
But I never found a use case for myself as somebody who is one man team working on his own projects.
Until one morning.
Architecture
I have a lot of (around 10) independent software programs that are running on daily (some even every hour) interval.
Most of them are doing some variation of web scraping, storing, analysis and reporting of results via email.
This was all fine until one morning I woke up and saw there where was no emails from my software.
I know that something was not right.
They all use yagmail for sending email, so I was thinking that there is some problem with that, because it is a single point of failure.
After an investigation, I found out that the problem was with Gmail itself it just stopped working, the next day it was fine, so they just had some issue that they need one day to resolve (I am not talking about Gmail web page, but with SMTP username/password authentication).
Why Gmail
Why do I use free Gmail for sending an email and not some more reliable service like SendGrid or Amazon SES?
That is a nice lecture in technical debt, in essence, what was a good idea for an initial requirement, as time progress and requirements or circumstances change, it is not so good idea anymore.
When I started with my first project in development as proof of concept Gmail was an excellent choice: easy to start and working fine.
As the project moved to deployment an additional projects where made it was easy to copy/paste the existing code than to refactor/redesign/rearchitect existing working solution.
REST solution
Emails did not work one day for me and after one day everything was back to normal.
I started to think about what can I do to avoid this problem in the future.
One solution would be to change from Gmail to something else. but here are a few issues that I do not like.
First issues
What if other solution (email provider) stops working in the future, I would again need to write new code for the third solution.
To fix this problem my idea is to use Gmail as primary providers for email sending if email sending fails I will just use a secondary email provider.
With this logic, I can add the third one also and so on, but I think that two are enough for the first version.
Second issues
Currently, I have around 10 apps (and this number will increase with new apps that I plan to do in future) that need email sending, each has a separate code base repository.
If I want to change something in email logic, even something simple ae username/password I need to do same change it in 10 different code bases.
One solution is to make one code base just for sending emails, this would solve the problem of the same changes in multiple apps.
But in order to work, I need to change the folder structure all my apps, update paths in the code bases, and I can use this only if all apps are in the same machine hosting.
If they are on separate machines it will not work.
REST to the rescue
After understanding all the difficulties, making RESTful web services just for email sending made total sense to me.
The only reason why it made sense to me is that I have a use case where REST is useful and look like the only solution.
The first version will just be adapter/facade around yagmail with REST API, but that is a story for another time.