I got a problem with my GTM, where everytime I do DNS query to mail.google.com from user to GTM, GTM always answer this DNS query with 127.0.0.1. The strange thing is this 127.0.0.1 answer is only appear at some domain under google.com (mail.google.com, translate.google.com). If I do direct nslookup on GTM box from command line, I get the right answer for mail.google.com. But when I do nslookup from user to GTM, the result is 127.0.0.1.
This is a HA BIG-IP box with LTM and GTM module inside. OS that was running is 11.5.3 HF1. This system was deplyoed a month ago, and everything was running well until suddenly it can't answer DNS query to mail.google.com.
Does anybody ever face a problem like this?
When you do a lookup from the GTM itself, you're only sending the query to bind (named). In order for the GTM process itself, or DNS express, or DNS cache to handle the query, the packet needs to arrive on a tmm interface, so needs to be sent to it from another device.
The DNS profile associated with the listener, as well as feature configuration on the GTM, determine which features are considered when handling the query.
From the symptoms you describe, it may be that an erroneous record has entered the DNS cache, or a zone transfer has been pulled into DNS express.
You can check which handler processed the requests by looking at the GSLB listener / Statistics / View details, and then set the Profile you're viewing to DNS (as opposed to udp_gtm_dns), resetting the stats counters, making a bunch of failing queries, and then looking to see which counter incremented.