A lesson in Analysis
Posted February 11, 2011on:
I thought I’d continue the theme of my last post “A lesson in Validation” with a lesson in analysis. This one is mostly focused on networking – one of my primary new focus areas but anyone can benefit from the process and lessons learned.
Recently, I was analyzing the results of some tests run against Yahoo’s Malaysia Front Page. The response times were incredibly high and on digging down, it soon became apparent why. The time taken to retrieve the static objects was more than 100 times what it should have been.
Like most websites, Yahoo! uses geographically distributed CDNs to serve static content. Malaysia gets served out of Singapore which is pretty darned close so there should be no reason for extra long hops. A large response time to retrieve a static object can mean one of two things: the object is not being cached or it is getting routed to the wrong location.
Sure enough, nslookup showed that the ip address being used to access all the static objects was located in the USA. It was therefore no surprise that it was taking so long. Satisfied that I had found a routing problem, I contacted the DNS team and they said … you guessed it. “It’s not our problem”. They ran some of their own tests and stated that Malaysia was getting routed correctly to Singapore, therefore the problem must be with my tests.
Dig(1) to the rescue
Since I was using a third-party tool, I contacted the vendor to see if they could help. The support engineer promptly ran dig and found that all the requests were being resolved (but didn’t care that they weren’t being resolved correctly!) Here is the output of dig:
<<>> DiG 9.3.4 <<>> l1.yimg.com @GPNKULDB01 A +notrace +recurse ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1442 ;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 2, ADDITIONAL: 2 ;; QUESTION SECTION: ;l1.yimg.com. IN A ;; ANSWER SECTION: l1.yimg.com. 3068 IN CNAME geoycs-l.gy1.b.yahoodns.net. geoycs-l.gy1.b.yahoodns.net. 3577 IN CNAME fo-anyycs-l.ay1.b.yahoodns.net. fo-anyycs-l.ay1.b.yahoodns.net. 210 IN A 126.96.36.199 fo-anyycs-l.ay1.b.yahoodns.net. 210 IN A 188.8.131.52 fo-anyycs-l.ay1.b.yahoodns.net. 210 IN A 184.108.40.206 fo-anyycs-l.ay1.b.yahoodns.net. 210 IN A 220.127.116.11 fo-anyycs-l.ay1.b.yahoodns.net. 210 IN A 18.104.22.168 fo-anyycs-l.ay1.b.yahoodns.net. 210 IN A 22.214.171.124 ;; AUTHORITY SECTION: ay1.b.yahoodns.net. 159379 IN NS yf1.yahoo.com. ay1.b.yahoodns.net. 159379 IN NS yf2.yahoo.com. ;; ADDITIONAL SECTION: yf2.yahoo.com. 1053 IN A 126.96.36.199 yf1.yahoo.com. 1053 IN A 188.8.131.52 ;; Query time: 0 msec ;; SERVER: 172.16.37.138#53(172.16.37.138) ;; WHEN: Wed Jan 12 11:15:42 2011 ;; MSG SIZE rcvd: 270
We now had the ip address of the DNS Resolver – 172.16.37.138. Where was this located? nslookup showed:
** server can't find 184.108.40.206.in-addr.arpa.: NXDOMAIN
No luck – this was a private IP address. Back I went to Tech Support: “Can you please let me know what the public IP address is where this private IP is NATed to?”
And here’s what I got: “I confirmed with our NOC team that the public IP Address of the DNS server located at “Kuala Lumpur, Malaysia – VNSL” site is a.b.c.d. Hope this is helpful for you.”
(I replaced the actual ip address above for security). I promptly did another nslookup which showed the host name with KaulaLumpur in it. So it seemed that there was no problem on the testing side after all. But not so fast … hostnames can be bogus!
Geo-Locating a Server
We had to find out where exactly this ip address was located in the world. So the ip address was plugged into http://whatismyipaddress.com/ip-lookup and it came back with:
The test server was actually located in Canada, while appearing to be from KaulaLampur! No wonder our DNS servers were routing the requests to the US.
What seemed to be a problem with our routing, turned out to be a problem of the tool’s routing !
Moral of the story: Don’t trust any tool and validate, validate, validate!