ISP DNS Hijacking breaks DirectAccess connectivity
Edit: Thanks to Richard Hicks for commenting on this post. It seems that my understanding of exactly why things broke the way they did may have been flawed. I leave the post here for reference however, since it does describe the symptoms and the solution that worked in this particular situation. Understand however that the explanation of why the solution worked may not be correct.
We recently began piloting Microsoft's DirectAccess (DA) for use connecting remote/mobile computers to our internal network over the internet. Yesterday I received a call from a user who had been using DirectAccess for several weeks without issue, but suddenly was not able to access internal resources. I'm documenting the issue here in hopes that it will help others experiencing similar issues with DirectAccess connections.
The first thing I tried was to remote control the computer using SCCM. When this connected successfully, I knew that the DA connection had at least established the tunnel and was working, even if it reported that it wasn't. After a bit of testing to identify what was working and what wasn't, I determined that when the computer tried to connect to the network location server it was able to resolve the host name to an IP address, but couldn't verify the identity of the server at the IP address given.
The network location server is used by DA to determine whether the computer is connected to the internal network, and thus whether DA needs to create a tunnel for corporate connectivity. DA attempts to establish an SSL connection to a web server on a host that can only be accessed internally. If the computer is on the internal network, the connection is successful and DA verifies the remote server's identity based on the SSL certificate. If the computer is connected to the public internet, the internal host name should not be resolved by DNS, and DA will then attempt to establish a secure tunnel to the internal network. An entry in the client's Name Resolution Policy Table (NRPT) prevents the client from connecting to that internal host over an established DA connection.
When the user's computer tried to resolve the internal host name of the NLS, instead of responding with "Non-existent domain" as it should, the ISP's DNS server returned an IP address of 198.105.254.63. The client then tried unsuccessfully to open an SSL connection to the server at that address to verify that it was on the internal network and did not need to establish a DA tunnel. At this point, even though DA was able to successfully open a connection to the internal network, it showed on the client side that corporate connectivity was not working, and the user was not able to connect to some internal resources.
This user was working from home, using an internet connection from their cable company, Mediacom. Like many big ISP's these days, Mediacom uses a form of deep packet inspection to return search results instead of a proper error when the address a user entered doesn't point to a valid web page. They also configure their DNS servers to respond with the IP of their "Search Guide" servers so they can provide search results even when the user enters key words or other data that isn't a valid address at all. By hijacking their users requests in this way, they can provide search results from their service instead of from the user's preferred search engine (or instead of returning a message that the page cannot be found.) They present this as a feature, a service to their users, but of course in reality it is designed to take a cut of money that google and others make selling ads in search results, by redirecting users to Mediacom's search results and ads instead.
To resolve this issue, the user had to get to one of the Mediacom search results pages. From a new browser window, I had the user attempt to browse to a web address that I knew didn't exist. As expected, this returned a Mediacom branded page listing search results similar to the address she entered. At the top right corner of this page was a Settings link, which opened a Preferences dialog with a single option inside labeled Search Guide Settings.
After turning off the Search Guide and saving changes, the user tried again to browse to a non-existant site and received a proper error message. Next, we tried to resolve the name of the internal NLS server and received the "Non-existant domain" response that was expected, instead of the IP of Mediacom's search server. By the time we were done testing these, the Direct Access Connectivity Assistant had already updated to show that corporate connectivity was working. The user's mail client was connected successfully to the server, and the intranet page which initially indicated a problem when it wouldn't display was now loading successfully.
Now the user is fully connected again and enjoying the simplicity of using DA. The only question that remains is why this DNS hijacking didn't affect this user previously, as Mediacom has been known for several years now to hijack DNS requests for non-existant domains and return the address of their search servers instead.
TL;DR: If an ISP hijacks DNS requests for non-existant domains, this can cause problems for DirectAccess connections when attempting to determine the client's network location.
The first thing I tried was to remote control the computer using SCCM. When this connected successfully, I knew that the DA connection had at least established the tunnel and was working, even if it reported that it wasn't. After a bit of testing to identify what was working and what wasn't, I determined that when the computer tried to connect to the network location server it was able to resolve the host name to an IP address, but couldn't verify the identity of the server at the IP address given.
The network location server is used by DA to determine whether the computer is connected to the internal network, and thus whether DA needs to create a tunnel for corporate connectivity. DA attempts to establish an SSL connection to a web server on a host that can only be accessed internally. If the computer is on the internal network, the connection is successful and DA verifies the remote server's identity based on the SSL certificate. If the computer is connected to the public internet, the internal host name should not be resolved by DNS, and DA will then attempt to establish a secure tunnel to the internal network. An entry in the client's Name Resolution Policy Table (NRPT) prevents the client from connecting to that internal host over an established DA connection.
When the user's computer tried to resolve the internal host name of the NLS, instead of responding with "Non-existent domain" as it should, the ISP's DNS server returned an IP address of 198.105.254.63. The client then tried unsuccessfully to open an SSL connection to the server at that address to verify that it was on the internal network and did not need to establish a DA tunnel. At this point, even though DA was able to successfully open a connection to the internal network, it showed on the client side that corporate connectivity was not working, and the user was not able to connect to some internal resources.
This user was working from home, using an internet connection from their cable company, Mediacom. Like many big ISP's these days, Mediacom uses a form of deep packet inspection to return search results instead of a proper error when the address a user entered doesn't point to a valid web page. They also configure their DNS servers to respond with the IP of their "Search Guide" servers so they can provide search results even when the user enters key words or other data that isn't a valid address at all. By hijacking their users requests in this way, they can provide search results from their service instead of from the user's preferred search engine (or instead of returning a message that the page cannot be found.) They present this as a feature, a service to their users, but of course in reality it is designed to take a cut of money that google and others make selling ads in search results, by redirecting users to Mediacom's search results and ads instead.
To resolve this issue, the user had to get to one of the Mediacom search results pages. From a new browser window, I had the user attempt to browse to a web address that I knew didn't exist. As expected, this returned a Mediacom branded page listing search results similar to the address she entered. At the top right corner of this page was a Settings link, which opened a Preferences dialog with a single option inside labeled Search Guide Settings.
After turning off the Search Guide and saving changes, the user tried again to browse to a non-existant site and received a proper error message. Next, we tried to resolve the name of the internal NLS server and received the "Non-existant domain" response that was expected, instead of the IP of Mediacom's search server. By the time we were done testing these, the Direct Access Connectivity Assistant had already updated to show that corporate connectivity was working. The user's mail client was connected successfully to the server, and the intranet page which initially indicated a problem when it wouldn't display was now loading successfully.
Now the user is fully connected again and enjoying the simplicity of using DA. The only question that remains is why this DNS hijacking didn't affect this user previously, as Mediacom has been known for several years now to hijack DNS requests for non-existant domains and return the address of their search servers instead.
TL;DR: If an ISP hijacks DNS requests for non-existant domains, this can cause problems for DirectAccess connections when attempting to determine the client's network location.
Comments
Post a Comment