SaaS platform bundled with multi-domain solution

0x1 Introduction#

The reason I saw Jeff's post Demand: Bind ten thousand domains to a website and automatically generate HTTPS certificates reminded me of some memories, so I'll share my solution.
This solution has been put into production and running stably since 2019. However, unfortunately, I have decided to stop this project for some reasons. If anyone is interested, we can chat privately. I developed a food delivery platform that can support multiple users, and each user has their own independent backend and can have their own independent domain name. For example, if I have a store called AABBCC, the platform will assign a subdomain for me, such as AABBCC.example.com. Of course, this store can also have its own domain name, such as AABBCC.com, and these two domain names actually point to the same site.

0x2 Simple Considerations#

To implement the above solution, the first thing to consider is how to implement multi-user SaaS, because each user is an independent entity and the data needs to be stored separately. How to choose the database? For example, each user has a table in MySQL? Or each user establishes a database in MongoDB? Here I chose the latter, for the following reasons: although it is an SaaS system, each user has their own independent database, which is convenient for later maintenance. For example, if a user has special requirements, they can customize some functions. These functions can add certain special requirements to their unique version.

How to implement accessing domain names and redirecting to the corresponding website, there are two methods:

Reverse proxy
Routing

Here I prefer reverse proxy because each user has their own independent program, which does not affect each other. It will not cause other merchants to be affected if one system fails. Of course, there is no problem for 100 users. Because the program is backed by Node.js, the memory usage can be ignored.

I am more familiar with NGINX for reverse proxy. Here I use OpenResty, which is basically the same as NGINX, but it supports the Lua module. Here, the Lua module can be used for simple load balancing. For example, when the domain name AABBCC.example.com or AABBCC.com is accessed, I only need to find the corresponding backend address somewhere and forward the request there. Does it feel like a key-value relationship? Yes, it is Redis. Redis can help us do many things.

Since the domain name issue has been resolved, isn't SSL signing also important? Obviously, yes, such a service should have a green lock on the website. Here, I choose the famous Let's Encrypt, which provides free SSL certificates for 3 months. Of course, the issue of renewal needs to be considered...

0x3 Problem Solving#

The first problem is the installation of OpenResty. Here I chose Baota panel because it is a very old version. The key is to see if the Lua module is suitable. I couldn't find the one-click script I wrote before... It should be solvable for those who are good at hands-on.

Next, let's explain how to implement domain name binding. First, you need to have a wildcard domain name, such as *.example.com, which is used to host AABBCC.example.com. When accessing this domain name, it should automatically reverse proxy to the corresponding backend. According to the key-value principle mentioned earlier, perform a key-value query operation here. If it's okay, return the corresponding port and assign it to the downstream proxy. If not, return an error.

set $subdomain default;
access_by_lua '
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000)
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.say("failed to connect: ", err)
return
end
local host = ngx.var.host;
local res, err = red:get(host)
if type(err)=="nil" then
if res ~= ngx.null then
ngx.log(ngx.INFO, "[web.xxx.fr] Retrieved domain name corresponding to port => ", res)
ngx.var.subdomain = "http://127.0.0.1:"..res
else
ngx.log(ngx.INFO, "[web.xxx.fr] Port not found => ", res)
ngx.var.subdomain = "http://127.0.0.1:9998"
end
else
ngx.log(ngx.INFO, "[web.xxx.fr] Error message => ", err)
ngx.var.subdomain = "http://127.0.0.1:9998"
end
';

location / {
    proxy_pass $subdomain;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Origin "";
    add_header X-Cache $upstream_cache_status;
}

So far, the simple load balancing is done.

How to automatically enable the green lock? The project is written in Node.js. Although there is a library called GreenLock, it is not suitable for my project after testing... It should be fine for a small project, at least it was 4 years ago...

Here, I choose https://github.com/auto-ssl/lua-resty-auto-ssl to implement it. The installation process is omitted. Here, I choose the Redis adaptor so that all certificates will enter Redis and can be changed at any time. Although this thing has automatic renewal, it is not what I expected for automatic renewal. I want to achieve smart automatic renewal. So I wrote a scheduled task using Node.js.

If you want to automatically renew with lua-resty-auto-ssl, according to their source code, when renewing, you need to get something called a secret. So I wrote this code, which will always get the latest secret and send it to the server for renewal.

const getToken = () => {
    return new Promise((resolve, reject) => {
        httpRequest.get(`http://127.0.0.1:${HOOK_SERVER_PORT}/getSecret`)
            .then((response) => {
                resolve(response.data.trim());
            }).catch((e) => {
            reject(e);
        });
    });
}

The renewal command is very important!!! You need to configure the hook server and open port 8999. Here, it is also implemented by calling a script, aiming to open a '/getSecret' for the renewal command below.

export HOOK_SERVER_PORT=8999;
export HOOK_SECRET=obtained earlier;
/resty-auto-ssl path/dehydrated -c --accept-terms --no-lock -d domain --config xxxx -x --challenge http-01 --hook xxxx

Run a scheduled task when there is no one at night. If there is a need for renewal, you need to reload the server. Anyway, the script completes the scheduled task.

By the way, the renewal port is 8999, which can be modified according to preferences.

That's all for the requirements.

The certificates are stored in Redis, and the certificates can be replaced with your own certificates. It is easy to bind domain names as a customer, just CNAME to your server, and the server will handle it automatically. It's comfortable to deploy on a single machine.

0x4 Summary#

Actually, before choosing this solution, I also considered Caddy, but at that time, Caddy was not very stable, and there were often delays in renewal, so I gave up. I turned to the road of reading source code. But precisely because I saw such a friendly open-source project, I have gained a lot of improvement. This renewal solution has been mentioned by many people online, but no one has mentioned the self-service script hooking to the secret for active renewal. I will share it a little bit. If I have time, I will upload a demo because it's too long to explain...

Translation: