Cleverly Using /etc/hosts to Solve Cross-Cloud MongoDB Replica Set Public Traffic Waste

Background

It all started like this: as a bargain hunter, I have several (low-priced) servers from different cloud providers. I had already deployed a MongoDB replica set on Alibaba Cloud Guangzhou, and now I wanted to add a new node on Tencent Cloud Guangzhou. So, I followed the official MongoDB documentation and executed:

rs.add("xxx.xxx.xxx.xxx:27017")

At first, I thought I just needed to allow Tencent Cloud’s access in Alibaba Cloud’s firewall security group and vice versa. However, after messing around for a while, I found that the new node stayed in the STARTUP state and wouldn’t enter SECONDARY. I was very puzzled.

Later, I used telnet and found that whether on the Alibaba Cloud or Tencent Cloud machine, accessing its own public IP:27017 was not working. So, I allowed the machines to access themselves on port 27017 in their respective firewalls, and the new node successfully joined the replica set instantly.

Reflection

I checked the documentation and found that MongoDB replica set nodes also try to access themselves when joining and running. When I used rs.add, I used the public IP. Before the cloud security group rules were set, the machine could not connect to itself through the public IP, so it naturally couldn’t join the replica set smoothly.

My initial misunderstanding was that accessing one’s own IP is somewhat like accessing the loopback address 127.0.0.1. Because I had previously tested accessing public IP:80 in a home broadband intranet and it worked (though accessing it from other networks was blocked by the ISP on port 80), I later discovered this is usually because the router supports NAT Loopback (NAT Hairpinning). The intranet traffic is forwarded directly by the router without leaving the WAN port. So, with this stereotype that “accessing one’s own public IP is roughly equal to loopback,” I ignored the security group’s allow rules.

Just That?

But the problem is, if allowing oneself in the security group enables access, does it mean that the traffic of accessing oneself will also be counted as public network traffic? That would be a waste (after all, this Tencent Cloud machine has high bandwidth but limited traffic). Not good! Not good at all!

Is there a solution?

Yes! A common idea is to set up a VLAN and let them access each other through intranet addresses. This way, only the cross-cloud traffic is billed, and the traffic for accessing oneself won’t be billed.

But the price of cross-cloud VLANs is quite high. Then switching to SD-WAN or P2P VPN like Tailscale or n2n would work!

A Cleverer Solution

Considering that setting up Headscale (an open-source alternative to Tailscale) is still a bit troublesome, while flipping through the n2n documentation, I suddenly had a flash of inspiration!

If I replace the IP with a domain name for the nodes (MongoDB replica sets support domain names), and then resolve the domain name pointing to itself to 127.0.0.1, wouldn’t that avoid the public network? And it doesn’t require introducing other tools; it’s simpler and more stable.

So, I defined two domain names: mongo.aliyun-gz.internal and mongo.qcloud-gz.internal, pointing to Alibaba Cloud and Tencent Cloud respectively, and then modified /etc/hosts:

Modify /etc/hosts on the Alibaba Cloud machine:

127.0.0.1 mongo.aliyun-gz.internal        # Use loopback when accessing itself
QCLOUD_PUBLIC_IP mongo.qcloud-gz.internal # Use public network when accessing Tencent Cloud node

Modify /etc/hosts on the Tencent Cloud machine:

127.0.0.1 mongo.qcloud-gz.internal          # Use loopback when accessing itself
ALIYUN_PUBLIC_IP mongo.aliyun-gz.internal   # Use public network when accessing Alibaba Cloud node

Then change the MongoDB node host to the domain name and apply it. Problem solved!

Of course, this solution also has a drawback, which is the need to manually modify hosts. It would be better if there were a self-built DNS in the intranet, but my server architecture is still very simple for now. This will do.

This solution also applies to other clustered applications/middleware :-P