AD Slow Authentication and prompting for credentials again and again (Active directory Troubleshooting – Part 2)
It’s a most common issue in a complicated Active Directory environment, before am going to discuss about the authentication issues, I would like to discuss about the Active Directory basics like Pass through authentication, AD secure channel, NTLM and Kerberos
Pass through authentication
If you are worked on multi Domain/Forest environment or environment designed with user forest and resource forest, an Exchange Server resource forest topology has two forests. One forest contains the all the user accounts for your organization. This forest is called the user forest (accounts forest). The other forest does not contain any user accounts. It only contains the Exchange Server and disabled user accounts, in simple you have one Active Directory forest where your user accounts live and another Active Directory forest where your application are lives (Exchange server, File server)
In the above scenario Domain controller receiving the request from Exchange/file server to verify the user access, this must pass the request to Domain controller in the user forest, we should have the trust between domain of the server (called the resource domain/forest) and the domain of the user account (called the account domain/forest)
User from Domain A try to access application on server from Domain B, application server in Domain B doesn’t have user detail, it will check the local Domain controller on Domain B through workstation secure channel, and Domain controller on Domain B check the Domain controller on Domain A through trusted domain secure channel and Domain A return back the authentication to Domain B it’s called the pass through authentication since the user authentication request been passed to user domain.
Secure Channel
I have discussed about the secure channel, what is secure channel? It’s a communication channel provides more secure communication path between the domain controller and the workstations or member servers. It can also be used to retrieve domain-specific information, handling NTLM authentication pass-through to the domain controller or from DC to DC for the same.
Two Forest or Domain connected through Forest trust / Domain trust, trust establishment is a shared secret (called a trust password) that domain controller use in the two domains for computing the session key that is used for protecting the secure channel traffic. By using this secure channel, the DC in the resource domain can pass logon requests securely to the DC in the account domain, in the same way that the server passed the logon request to the former DC. The secure channel between DCs in two domains that are connected via a trust relationship is called a trusted domain secure channel. In contrast, the secure channel between the member server and the DC in the resource domain is called a workstation secure channel
While adding a computer to domain, computer account has been created in Activity directory and password been generated for computer account, computer account password been changed every 30 day’s and stored in computer and domain controller, while power on the computer, Netlogon service on computer use the computer account password from the computer is authenticated against the password on the Domain Controller and establishes a secure channel with that DC, same way server creates a secure channel with that DC it get authenticated
For Domain controller, Netlogon service sets up secure channels with all the trusted domains (one Domain controller in each trusted domains) you can check this using Nltest command
To check the current secure channel with a particular Domain
nltest /sc_query:Domain Name
To reset secure channel to different Domain Controller (this will randomly select the Domain Controller)
nltest /SC_RESET: Domain Name
To reset secure channel to a particular Domain Controller
nltest /SC_RESET: Domain Name Domain Controller Name
You can change the Domain and Domain Controller Name as per your requirement
For remote server you can add
/server: server name
NTLM and Kerberos
NTLM and Kerberos are the protocols used for authentication, we all know NTLM is outdated and Kerberos is the new protocols used for authentication, Kerberos can impersonate a user when trusted, so no need to contact Domain controller every time in order to authenticate access to a resource, If the client is logged on to a domain, the browser never prompts the user for credentials; it simply uses the user’s default logon credentials.
We are not using NTLM any more? No still some areas where we have to use NTLM for the sake of compatibility, RPC over HTTP to connect to an Exchange mailbox. ISA for web proxy servers, let’s go in deep how it’s works
- Client opens a URL through browser
- Client browser sent a request to proxy server with integrated authentication credentials
- The proxy server needs to verify the user credentials, by sending the authentication request to the domain controller is has a secure channel
- That domain controller responds to the proxy server
- The proxy server answers the client with the requested internet page
Seems to be simple however for each web connection from a client, the proxy server needs to verify the user credentials by sending authentication request to the DC, this will increase the high volume of NTLM authentication
Think the similar scenario in multi Domain/Forest environment, user in one Forest and proxy server in other Forest, this will increase the NTLM Pass through authentication traffic
NTLM authentication handled by Netlogon service, passing NTLM authentication requests to a domain controller that can handle them, and receiving them on that domain controller to be handled, you can enable debug logging for the Net Logon service to see what happen on the proxy server / domain controller, like which user getting authenticated to which domain controller
We have number of threads which will authentication request, like number of concurrent NTLM authentications processed by the server, the defaults are typically 1 for this, meaning that there is one thread to hand off, receive and process these requests, we can re-configure this through MaxConcurrentApi
The MaxConcurrentApi thread can only deal with one authentication at a time, in normal scenario it is very quick. So the high volume of authentication transactions must be handled by one or two threads (by default) and this will be a bottleneck, which resulting the delay and the authentication request have to wait longer than a remote client can tolerate
Due to this delay, client browser would ask for a credential prompt rather than the web page he want to open, so this might be the issue for slow Authentication and prompted for credentials again and again
We can resolve this by increasing the MaxConcurrentApi value on proxy server, if it’s a multi Domain/Forest environment, then we needs to increase the MaxConcurrentApi value on resource forest Domain Controller, it depends on the configuration
- First we have to understand current issue
- Enable the Netlogon logs on proxy server and Domain Controller authenticating proxy server or the proxy server secure channeled Domain Controller
- Analysis the log for authentication failure and delay
- We can monitor the current secure channel traffic through the perfmon by adding the counters, Semaphore Waiters, Semaphore Holders, Semaphore Acquires, Semaphore Timeouts, Average Semaphore Hold Time
- Logon to Domain Controller authenticating proxy server
- Open perfmon, add counters – select Netlogon on performance object
- And select Average Semaphore Hold Time
- If you don’t find the counters, need to install http://support.microsoft.com/kb/928576 hot fix,
- This adds New performance counters for Windows Server 2003
If the Average Semaphore Hold Time is greater than normal, Average Semaphore Hold Time should normally be very quick. Longer hold times mean that a potential bottleneck is occurring, it will delay authentication process and the authentication request has to wait longer than a remote client can tolerate, it will slowdown the authentication and prompted for credentials to re-initiate the authentication process
To calculate MaxConcurrentApi for your environment
It depends on the outcome, you should add more servers to service the legacy authentication load or increase the MaxConcurrentApi registry value
Warning rules:
Average Semaphore Hold Time > 0.2 should be a yellow warning.
Average Semaphore Hold Time > 0 should be a red warning.
Semaphore waiters > 1 should be a yellow warning.
Semaphore waiters > 4 should be a red warning.
Any Semaphore Timeouts is a red warning.
Conclusion:
Slow Authentication might be due to many issues, like client to DC connectivity, network, subnet and site configuration, DNS SRV configuration, Profile, logon script and GPO, so you have to analysis properly to find the root cause, this is the one of the scenarios which I have faced many times in a complex environment, hope this will help you to understand the Active Directory Authentication and troubleshooting procedures see you soon in an another article.
Active directory Troubleshooting (Part1 – Diagnostics Logging)
Troubleshoot Active Directory Server Replication
Related posts:
Active directory Troubleshooting (Part1 – Diagnostics Logging)
How secure channel determine the Domain controller in cross-forest
DNS Entry of Domain Controller are Resolving to Incorrect value
Replsummary showing unknown for largest delta on AD replication checks
Planning safe Decommission of Domain Controller (Decommission of Active Directory site) Without Impacting Users
Hello
Thanks for posting this very useful information
Can I ask have you ever seen MaxConcurrentAPI issues causing Active Directory user account lockout issues?
for example we are getting random (or at lease they appear to be random) account lockout issues (normally in the mornings or when people come back from Lunch, and cannot unlock their workstation for example as account locked out).
when locking at basic lockout information it is always the user workstation/laptop that has locked them out (or at least a process running on it).
Therefore just wondered if a background process was trying to authenticate the user and kept failing due to timeouts as your post explains, then tried again and again, could this cause a lockout?
For example, with have NetApp NAS attached storage which offers up CIFs share, assuming the authentication protocol is NTLM (I have not check this yet) and there is a MaxConcurrentAPI issue (we has Windows Server 2012 R2) could repeated attempts to access a CIFS SHARE over NTLM when they NetApp device cannot get an answer from the DC about authenticating the user cause the users AD account to get locked out?
Thanks very much