Meet El Chapo - an open source & serverless URL shortener
If you work on a product that sends notifications to people and if those notifications may contain links, there is a high likelihood that you use a URL shortener so that the text in the link is formatted and easy to read.
Our product had a similar requirement. We send automated transactional WhatsApp notifications to folks who place orders on e-commerce stores and the messages look something like this
Pretty neat right? But here is how it looks without the URL shortened
Looking at the above images, it is safe to say that a URL shortener makes a ton of difference, so now let’s look into the attributes we expect this URL shortener to have.
- Separate micro service that is delinked from the rest of the product.
- Highly scalable since we don’t know what our message volume will grow to be.
- Not have the need to monitor and maintain once deployed.
- Use our own short domain name (bit.ly/something won’t work)
- Must be cheap (as bootstrappers we don’t have too many $$ to spend on stuff like this)
We looked into many 3rd party services that did this but their free plan did not include branded links and their paid plans cost around 30$ a month. We realised that we can build this in a day and it would cost us 1/10th of that.
Why serverless?
As you can probably tell, this is a service which will not have a sustained or predictable kind of traffic. The only time we need it is when a message is about to be sent, so provisioning a server that runs 24X7 for such a thing seems pointless when there are great serverless solutions out there.
We also had a strong reason to believe that if we used AWS Lambda, there is a good chance that we never even exceed the free tier for Lambda usage and this service would end up costing us close to 0$.
A while ago I was worried that deploying and running a Lambda function on AWS would force vendor lock in, but then I discovered something insanely cool called Zappa.
Zappa enables you to use a regular Flask/Django application and package it to run on AWS Lambda + API Gateway. This essentially means that if I ever wanted to move out of serverless, I could easily do that by running my application on a regular instance without having to change a thing in my application.
What’s even more cool is that deploying a Zappa application is as simple as creating a JSON file with some settings and then running the deploy command which does all of this under the hood.
And when the command finishes running, you get a URL where your application is now live.
Honestly, this is the least amount of work I’ve ever done in my life to deploy an application that scales almost infinitely.
What can El Chapo do?
You can find the source code for El Chapo here along with instructions on how to deploy it. Please feel free to fork it and create your own variations. As of now, it takes care of the following things.
- Create a new short URL by accepting a long URL and a short path.
- Redirect to the long URL when the short URL is fired.
- Call a webhook asynchronously when the short URL is fired. This can be very handy for tracking clicks and running other functions on click of the URL.
- Deploy to production with one single command.
What do you need to deploy El Chapo?
- An AWS account
- A short domain name that you own.
To be clear, you don’t require a domain name to test the application. You can do so with the default URL that Zappa creates for you and once you are done with this, you can then buy a short domain and route all requests coming to that domain to this URL in the DNS settings of your domain provider.
How does El Chapo work?
El Chapo uses AWS DynamoDB as the data store to save the following information:
- Short URL path
- Original URL
- Webhook URL.
DynamoDB is a very efficient and cheap key value data store that is capable of handling requests at the rate of even 20 million per second. It is also fully managed with built-in security, backup and restore, and in-memory caching for internet-scale applications.
We did not even end up deleting old shortened URLs from DynamoDB because it doesn’t cost us anything to keep it. However if you want to expire and delete old entries, you can do so easily since DynamoDB supports a TTL for entries in the database.
Now that the data store is established, the working of this application is quite straight forward. When a request is received to create a new short URL path, an entry is created in the database with the above 3 fields and when someone clicks on the short URL, they are redirected to the original URL and in parallel if a webhook is defined, it is called. That is pretty much it.
To make this process even simpler, dynamo DB has a python SDK that you can use to define a model in your database the same way you would do if you had access to an ORM.
Further documentation about the inner workings of the application are defined in the GitHub repo mentioned earlier.
Closing notes
Since I mentioned multiple times that this solution is very cheap to maintain, let me back up that claim with some numbers. In the month of June, we had 73K read requests and 914K write requests on DynamoDB. The cost for this came to be 1.16$
And as for Lamdba, we made 1.16 Million requests out of which about 1 million were under free tier, so the cost for this was 0.04$
Lastly, the API gateway charges came to be 3.73$
So the total cost for serving > 1mn requests using this setup cost us around $5 which is not zero but it is pretty close to zero considering our MRR and volume.