Hey there! In my first post here I’ll go through a DNS issue that haunted us for some years and explain how we dealt with it. I’m Gustavo by the way, the Android team’s tech lead.
A Domain Name System (DNS) is a hierarchical distributed database that primarily translates hostnames like “easytaxi.com” into IP addresses like “104.20.3.124” (IPv4) or “3ffe:1900:4545:3:200:f8ff:fe21:67cf” (IPv6), so we don’t have to remember all of these characters. DNS also works great as a scalability tool providing the first level of load balancing, retrieving different server IP addresses possibly based on the user’s current geolocation in order to reduce latency. Besides that, it can also be a major source of headache that can cost your business a lot of money by driving users away due to a poor experience.
In Easy Taxi it was very common from time to time to notice huge drops of active users (drivers and passengers) of a given country due to DNS-related issues. Our API couldn’t be reached because the apps simply couldn’t find its IP address, that is, they faced a lookup or name resolution problem.
Every solution to a problem sooner or later creates new problems to solve. While solving many issues of the Internet, there is a hidden cost when relying on DNS that many of us ignore: although it’s transparent, it’s not magical. Besides consuming valuable time to execute, DNS queries can and will fail from time to time either because of human mistakes or availability issues somewhere. Since this kind of problem happens on the client-side, you may be wondering how we can defend ourselves from it. First of all, there are smart DNS services clients can use, like Amazon’s Route 53 that provides a health-check feature. If your server goes down, its host’s IP address won’t be returned to the users until it gets online again. This is particularly useful if you have multiple server nodes.
The thing is, it isn’t enough to have a super-smart DNS service because your end-users will probably use something else, to begin with. Internet Service Providers (ISP) and cellphone network operators define the DNS services that are going to be used when your computer or mobile device first connects to them — normally they provide two options to favor redundancy. Our experiences show that they have high failure rates. There are faster, safer and more reliable third party options though. Want to see for yourself? There is a neat tool called namebench (Linux version) that allows one to do some DNS benchmarking. After you come up with an alternative service, all you need to do is configuring your internet connection to use it or even better, configure your router/switch to use it. But how to control the DNS services your users will use? Most of them will have a really hard time tweaking this, so asking them to do so is simply not acceptable.
On Android, we’re in control of the client side. We could even change the user’s connection settings but that’s way too intrusive, hence, not acceptable as well. The best approach we have found was to use a library called dnsjava to perform the name resolution using the specific DNS services we wanted. In case one service fails, the next will be used. As a last resort, we retrieve a static IP in case we’re trying to lookup our API’s address. Since we use OkHttp to perform HTTP connections, the integration with this lib was very simple to do, as I will demonstrate in the following code snippet.
import com.squareup.okhttp.Dns;
import org.xbill.DNS.Address;
import org.xbill.DNS.ExtendedResolver;
import org.xbill.DNS.Lookup;
import org.xbill.DNS.Resolver;
import org.xbill.DNS.SimpleResolver;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.Collections;
import java.util.List;
public class EasyDns implements Dns {
private static final String LIVE_API_HOST = "easytaxi.com.br";
private static final String LIVE_API_IP = "1.2.3.4";
private boolean mInitialized;
private InetAddress mLiveApiStaticIpAddress;
@Override
public List<InetAddress> lookup(String hostname) throws UnknownHostException {
// I'm initializing the DNS resolvers here to take advantage of this method being called in a background-thread managed by OkHttp
init();
try {
return Collections.singletonList(Address.getByName(hostname));
} catch (UnknownHostException e) {
// fallback to the API's static IP
if (LIVE_API_HOST.equals(hostname) && mLiveApiStaticIpAddress != null) {
return Collections.singletonList(mLiveApiStaticIpAddress);
} else {
throw e;
}
}
}
private void init() {
if (mInitialized) return; else mInitialized = true;
try {
mLiveApiStaticIpAddress = InetAddress.getByName(LIVE_API_IP);
} catch (UnknownHostException e) {
Log.w(“DNS”, "Couldn't initialize static IP address");
}
try {
// configure the resolvers, starting with the default ones (based on the current network connection)
Resolver defaultResolver = Lookup.getDefaultResolver();
// use Google's public DNS services
Resolver googleFirstResolver = new SimpleResolver("8.8.8.8");
Resolver googleSecondResolver = new SimpleResolver("8.8.4.4");
// also try using Amazon
Resolver amazonResolver = new SimpleResolver("205.251.198.30");
Lookup.setDefaultResolver(
new ExtendedResolver(new Resolver[]{
defaultResolver,
googleFirstResolver,
googleSecondResolver,
amazonResolver
})
);
} catch (UnknownHostException e) {
Log.w(“DNS”, "Couldn't initialize custom resolvers");
}
}
}It’s very simple to activate this configuration on OkHttp 2.6+, just call OkHttpClient.setDns passing an instance of the class above.
That’s it so far. I hope I can share some statistics in a future post. Let me know if it was useful to you!
References:
http://highscalability.com/blog/2013/9/16/the-hidden-dns-tax-cascading-timeouts-and-errors.html
http://www.howtogeek.com/167239/7-reasons-to-use-a-third-party-dns-service/
December 11, 2015
By Gustavo Steigert, Sofware Engineer
