Locked Out?

If you are in IT, you are more than likely aware of issues that can arise from account lockouts, especially on a service account in use by a critical application or infrastructure component.

I will dive into why lockouts occur, share troubleshooting steps, look at helpful tools, and guide you into interpreting logs so that the problem can be resolved as quickly as possible. A lockout can prevent you, an application, or the business from continuing work. It could also be an indicator of a possible Cybersecurity attack. So, how do you go about finding the source of the lockout?

Related: Visualize Account Lockout events with my AD Lockout Splunk Dashboards to graphically identify patterns. For investigating Group-related events, see my Group and Membership Changes post.

Bonus

As an added bonus, I have included information on how to look up when an account was modified, disabled, enabled, unlocked, password reset — and by whom. In later revisions, I also added some tips for Cybersecurity Defenders to look into.

Active Directory Accounts

Let’s look at what Active Directory is and how network logins are related.

Microsoft’s Active Directory (AD) is a service that governs how resources can be utilized by a collection of users, groups, and computers. Enterprises use AD to authenticate, authorize, secure, and audit access within a security boundary — a Domain — to file servers, computers, emails, and more. You are given a user account (often referred to as your “network login”) to access what has been made available to you. A Domain Controller (DC) is the server that contains a copy of the AD database and is responsible for the replication of said data between all other DCs within the Domain.

To secure the company network, Active Directory uses Group Policy Objects (GPOs) to define various user- and computer-related settings, including password policies and the Account Lockout Threshold. The latter controls when an account is locked after a set number of failed login attempts. If you mistype your password three times, for example, it would be locked for a specified time or until an administrator unlocks. This is an important security step to frustrate an unauthorized person from gaining access.

What Happens During a Lockout?

Behind the scenes, when an incorrect password was provided for an account, the Domain Controller that it authenticated to relays the request to the DC holding the “PDC Emulator” role. The PDC Emulator always holds the account’s most recent password, and so it will re-check the provided password against its own database. If it is still incorrect, the PDC Emulator increments the badPwdCount attribute of the account, and an invalid login is recorded to a Security Event Log. If the badPwdCount has met the Account Lockout Threshold, the DC will lock the account, record Event ID 4740 (more on that later) to its Security log, and notify the other Domain Controllers of the locked state. The key here is that every lockout is known by the PDC Emulator.

The PDC Emulator can be found through multiple ways, and my preferred one is via the command line:

How to find the PDC Emulator of a Domain

• Command Prompt: nltest /dclist:name (where name is the AD Domain name)
• PowerShell

# Get the PDC Emulator for the current AD domain
(Get-ADDomain).PDCEmulator
# Get the PDC Emulator for "name" (either AD Domain or Domain Controller)
(Get-ADDomain -Server name).PDCEmulator

Account Lockout Threshold

Here are two ways to quickly find the configured, Domain-wide threshold. A value of 0 means the account will never be locked. This setting can be from 0 to 999.

• Command Prompt: dsquery * -filter “(objectCategory=domain)” -attr lockoutThreshold
• PowerShell
  • Get-ADDomain -Server domain | Select -ExpandProperty distinguishedName | Get-ADObject -Property lockoutThreshold
  • Get-ADDefaultDomainPasswordPolicy -Server domain

Warning: If your company uses Fine-Grained Password Policies (FGPP) introduced with Windows Server 2008, the above commands may not reflect the password policy actually applied to individual accounts or Global Groups. FGPP allows an administrator to set different password policies for various groups, such as privileged administrator accounts that require more restrictive settings, than what is configured for the entire domain using GPO. FGPPs take precedence over the domain-wide GPO. Learn more

You may view a user’s FGPP in PowerShell with: Get-ADUserResultantPasswordPolicy username. However, your administrator may not have granted rights to view it if the result ends with the error, “Cannot find an object with identity: ‘CN=UserPasswordPolicy,CN=Password Settings Container,CN=System,DC=domain,DC=com”.

Account Lockout Status

You can use Active Directory Users and Computers (ADUC) to check on an account’s lockout status. However, for automation purposes, I prefer the command line:

To check lockout state:

• Command Prompt: net user username /Domain
  • If “Account active” is “No”, it is locked or disabled
•  PowerShell: Get-ADUser username -Properties LockedOut,AccountLockoutTime,badPwdCount

PowerShell: Lockout information for a particular account

To unlock:

• Command Prompt: net user username /Domain /Active:Yes
• PowerShell
  • Get-ADUser username | Unlock-ADAccount
  • Unlock-ADAccount username

Troubleshooting

When I used to be in Desktop Support with large companies, I came across lockout tickets all the time. As a Systems Engineer, I have more tools and access rights at my disposal to troubleshoot issues that were escalated to me. If after unlocking an account, the lockouts continue to occur, here are the general things I look at. Be sure to check both the local computer and any remote servers the account may have been used at.

General Areas

1. Has the password changed recently?
   • Has the password been mistyped? One way to find out is to open a command prompt as that account via RunAs. If there is no error, the password is correct
2. Is the account logged in elsewhere?
   • Look for disconnected or idle remote desktop/terminal server sessions
   • Check mobile devices, email clients in particular (Example: Kerberos Event ID 4771)
     • If your company uses Outlook Web Access (OWA), look for any mobile devices associated with the email account
       • Go to OWA > Options > Mobile Devices
     • Reboot the device as needed
   • Have the user check their personal computers and devices

TIP: You can write a PowerShell script to gather all disconnected remote desktop/terminal server sessions from a list of computer names

3. Have there been any successful logins recently?
   • Especially in the case of service accounts, having this info can help rule out the successful login sources from the list of potential lockout origins
4. Is there a time of day pattern?
   • Are lockouts happening only during business hours? If so, it may point to a device the user brings to work, such as a phone, tablet, or laptop
5. Saved WiFi password
   • If your company uses enterprise WiFi authentication, check the user device’s WiFi password. You may also try turning off its WiFi for a few hours and see if that changes anything
6. Drive mappings
   • Disconnect all mapped drives and log out (or reboot). Any drives mapped by GPO would automatically reconnect upon login
7. Scheduled tasks
   • Look for any tasks configured to run as the account
8. Windows Credential Manager
   • Take a look at the Web Passwords and the Windows Credentials sections
9. Web browsers, such as Internet Explorer
   • Check saved credentials
   • Delete cookies and temporary files
     • As a last resort, try deleting the browser history and all saved passwords
10. Saved passwords in applications, including third party software
    • With service accounts especially, be sure the applications are using the correct password
    • If lockouts are occurring for multiple users from the same server, there may be a culprit application
    • Any applications that automatically log in
11. Saved passwords on Windows Services
    • Go to services.msc and check any services that may be running under the affected account
12. Roaming profile may be corrupt
    • Have the account logged off from every computer so the roaming profile is fully synchronized to the server
    • Rename the roaming profile on the server. We will be creating a NEW one with the next step
    • As another user with admin rights, delete the roaming profile from ONE computer, then log on to it
      • A new roaming profile will be created on the computer
        • Log off so it synchronizes a fresh copy to the server
      • If the issue goes away, delete the roaming profiles on the other computers
      • Finally, manually copy any data from the old profile (on the server) back to the new one
13. Are an unusual number of other users also experiencing lockouts?
    • There may be a policy misconfiguration or the network is under attack
14. Run an Antivirus scan
    • Has there been a worm infection recently? Worms are self-replicating malware that can use your credentials to travel across the network
15. If all else fails, including going through the steps mentioned in the “Advanced Areas” section below, you may try renaming the account, but I generally advise against that

True story: In 2007, I was hunting a lockout for several days. I do not remember the details of how I found the MAC address of the culprit, but I asked the network team to check the switches to locate the port it came from. From the port, we traced to the connected computer, but oddly, there was no Windows profile for the affected account. As a result, that PC was initially dismissed, but lockouts kept coming from that address. Ultimately, I discovered that the account had a drive mapped under another user’s profile. That person likely mapped a network share to get something, but forgot to disconnect it when done.

Tip: If you get a hold of a MAC address, you can try to look up the manufacturer here, here, or here to help narrow down the hardware. The first 6 characters (example: AC:E2:D3) identify the manufacturer. In the example, that would be “Hewlett Packard”, my HP laptop.

Advanced Areas

1. Active Directory
   • Check the Account Lockout Threshold policy and see if it may be too restrictive, such as one that locks out after just 1 failed attempt
     • A Denial of Service (DoS) attack may take advantage of this to disrupt a business
   • Check replication performance between affected sites. There may be a delay in replicating password changes
   • Check the Security log with the Windows Event Viewer on Domain Controllers that have recorded Bad Password Counts, paying special attention to various Event IDs
     • A free tool from Microsoft can help with gathering that information. More details on that and Event IDs later
2. Enable Netlogon Debugging on the PDC Emulator for a few hours, if multiple users are affected, and look through the log
3. Use Netstat to look for any applications running on a server and disable any services or applications that may be trying to create a failed connection
4. Audit a suspect computer’s logons and processes
   • Should you have narrowed down the lockouts to come from a particular computer, and none of the general troubleshooting steps have helped, temporarily try logging some events locally
   • On the computer, with elevated administrator rights, run “gpedit.msc”
     • Go to Computer Configurations > Windows Settings > Security Settings > Local Policies > Audit Policy, and enable:
       1. Audit logon events: Success, Failure
       2. Audit process tracking: Success, Failure
     • Once the account locks again, check the Event Viewer’s Security log, paying attention to lockout-related Event IDs that I will describe later in this post
       • Some of the events can give you insight into what Caller Process or Source Network Address may have triggered the lockout
5. Hacking attempts
   • Hackers may be trying to guess someone’s password through various means, such as a brute force attack
   • A disgruntled employee may be locking his or her manager’s account on purpose
6. Penetration tests
   • An external company the security team has hired may be performing a penetration test with unexpected results

True story: A penetration test happened at one of my employers during the middle of the day that inadvertently locked out multiple accounts, prompting our team to hunt down a possible hack attempt. This brought to our attention that this type of Denial of Service attack could easily disrupt a business if someone got a hold of the account names or login format

7. Check the IIS Application Pool for any identities using the account
8. Consider writing a script to temporarily buy you more time to troubleshoot a business-critical service account, especially if your energy was waning after troubleshooting all night
   • Start with the “Account Lockout Status” section above to develop your script

True story: Recently, a service account used for backing up virtual machines, files, and more kept locking. I received an urgent Sunday evening conference call at 7:30pm for help as backups were failing. After 4.5 hours of troubleshooting, checking Splunk logs, and using other methods, we were unable to find the lockout source. It was now midnight and some were tired.

• Thankfully, I was writing a PowerShell script during the call — just in case we needed to automate — that would check whether an account was locked out and at what time it occurred. It would then give the option to unlock once or to continuously monitor and automatically unlock. Ultimately, we used it so backups could complete while we got some rest. (Do you see why I prefer the command line and automating things?)
• The next morning, we found that the password was typed incorrectly on one of the backup servers

PowerShell: Account Lockout Check/Automatic Unlock

Don’t forget: Wherever you had enabled additional debugging or auditing, remember to turn them off so that performance does not suffer or space runs out!

Gathering Account Activity

As illustrated earlier, troubleshooting lockouts can be a very time consuming process. Not only are lockouts frustrating for the user, application owner, or even the business, but the IT person tends to get the brunt of that even if it was not his or her fault. There are some tools you could use to help with troubleshooting, all of which essentially comb the Security section of the Windows Event Log. Here are some of my favorites.

Windows Event Viewer: Security Log

Free Tools

1. Microsoft Account Lockout Status and EventCombMT
   • This is Microsoft’s own utility
   • Lockoutstatus.exe: Displays the Bad Pwd Count, Last Bad Pwd date and time, when the password was last set, when the Lockout occurred, and which DC reported this data
   • EventCombMT
     • Can search through a list of Domain Controllers for specific lockout-related Event IDs associated with the account. It will save the output to a text file, showing successful and failed login attempts
       • Search for: 529, 644, 675, 676, 681, 4624, 4625, 4740, 4770, 4771, 4776
       • Note: see the “Interpreting Account Activity” section below for Event ID details
     • Depending on your environment, this can take several hours to complete
2. Netwrix Account Lockout Examiner
   • This is a popular, free tool that gives you quite a lot of details. It also performs a few, simple troubleshooting checks for you. However, I have tried the licensed version at two companies, and both times we ran into the same issue: it appears to have a memory leak that requires the server to be rebooted from time to time

Scripts

1. Event-based Triggered Tasks
   • Windows provides a free, built-in way to perform certain tasks when a specific Event ID is recorded to the Event Log
     • If you recall, I mentioned earlier that the PDC Emulator records all lockout events. Use this fact to have the Domain Controller send you an email every time a lockout event (ID 4740) has occurred. This is accomplished through an Event-based Triggered Task
   • When configured, you will see the trigger job listed in the DC’s Task Scheduler
   • Having all the lockout emails stored in one folder allows you to search for a specific name and quickly scan through the details
     • This saves you from having to search through multiple Domain Controllers’ Event Logs

Tip: You can control what the email contains by writing a PowerShell script that formats and provides exactly the details that you may be interested in, including:
Username, Domain, Caller Machine, Event ID, Lockout time, Failure reason, Logon type, Caller Process Name, Source Network Address, Source Port, and more

2. PowerShell
   • You may realize by now that I am a supporter of PowerShell scripting. I have written a few of my own to aid with troubleshooting lockouts. However, they are beyond the scope of this post, although I may share some in the future. Meanwhile, there are many, readily-available scripts at the PowerShell Gallery that you may want to look at.

Tool: Splunk ($)

Splunk is a VERY powerful, expensive tool that aggregates logs from multiple sources (such as systems, applications, network devices, and more) to allow you to search, monitor, and analyze a wealth of Big Data. It is a very useful SIEM (Security Information and Event Management) tool that can also be used to deconstruct a timeline of events, such as a breach in the network.

Splunk Search showing failed login attempts

I was first introduced to Splunk in late 2017 at one of my past employers, but did not have the opportunity to try it until this year. Knowing that the software aggregates Active Directory logs at my current employer, I wanted to see how Splunk Cloud could be used to help with lockout troubleshooting. Its administrator, however, was always too busy to help, and there were no good search queries I could find on the web at the time. Naturally, I decided to learn some of the Splunk Search Processing Language (SPL) on my own to produce the queries presented in this article.

Tip: See the “Interpreting Account Activity” section below on how to read the data that Splunk and other tools produce

One can turn SPL queries into beautiful Dashboards with Simple XML like the ones I created below to help visualize lockout data:

Tip: See how we used Splunk Dashboards to visualize lockout data and hunt for the causes in a MUCH shorter timeframe.

Event ID 4767 (Unlock) Note

The Splunk queries provided here currently include events where the queried user is the one who performed an unlock operation. I have not yet added logic to exclude them. This can skew results when looking for events where the queried user itself was unlocked by someone else. In other words, the queries currently include events where either the user was unlocked or the user performed the unlock on another account.

Search for Lockout-related Events

This query will locate any events within the last 14 days that contribute to an account’s lockout, including failed password attempts (and how the logon was used), which machine, IP address, and/or process spawned the event, and the actual lockout.

Lockout-related events

index="*" Account_Name=Michael.Yuen OR Account_Name=Michael.Yuen@*
earliest=-14d latest=now() source=WinEventLog:Security
(EventCode=4740 OR EventCode=4625 OR EventCode=644 OR EventCode=529 OR EventCode=675 OR EventCode=676 OR EventCode=681 OR EventCode=4771 OR EventCode=4770 OR EventCode=4768 OR EventCode=4776 OR EventCode=4777 OR EventCode=4725 OR EventCode=4723 OR EventCode=4724 OR EventCode=4767 OR EventCode=4800 OR EventCode=4801)
| eval Account0=mvindex(Account_Name,0) | eval Account1=mvindex(Account_Name,1)
| eval Account=case(EventCode==4624,Account1, EventCode==4625,Account1, EventCode==4648,Account1, EventCode==4722,Account1, EventCode==4723,Account1, EventCode==4724,Account1, EventCode==4725,Account1, EventCode==4738,Account1, EventCode==4740,Account1, EventCode==4767,Account1, EventCode==4768,Account0, EventCode==4769,Account0, EventCode==4771,Account0, EventCode==4770,Account0, EventCode==5140,Account0, EventCode==4778,Account0, EventCode==4779,Account0, EventCode==4800,Account0, EventCode==4801,Account0) | fillnull Value="-" Account
| eval ActionBy=case(EventCode==4725,Account0, EventCode==4722,src_user, EventCode==4767,src_user, EventCode==4723,src_user, EventCode==4724,src_user, EventCode==4738,src_user, EventCode==4794,src_user)
| eval Time=strftime(_time, "%m/%d/%y %H:%M:%S") | sort -_time
| eval Caller_Machine=if(Caller_Machine_Name!= NULL, Caller_Machine_Name, Caller_Computer_Name) | fillnull Value="-" Caller_Machine
| rex field=Process_Name "(?P<Process_Name>[^\\\]+)$" | fillnull Value="-" Process_Name
| rex field=Caller_Process_Name "(?P<Caller_Process_Name>[^\\\]+)$" | fillnull Value="-" Caller_Process_Name
| rename "Authentication_Package" as "Auth_Package", "Source_Network_Address" as "Src_Netw_Addr"
| replace "MICROSOFT_AUTHENTICATION_PACKAGE_V1_0" with "MsAuthPkgV1_0" in Auth_Package
| replace "Microsoft Unified Security Protocol Provider" with "MsUnifiedSecProt" in Auth_Package
| eval EventCode=case(EventCode==4740, "4740 Locked", EventCode==4625, "4625 Logon Failed", EventCode==644, "644 Locked", EventCode==529, "529 Logon Failed", EventCode==4768, "4768 Kerb TGT Req", EventCode==4771, "4771 Kerb Pre-Auth Failed", EventCode==4770, "4770 Kerb Svc Tkt Renewed", EventCode==4624, "4624 Logon OK", EventCode==4648, "4648 Logon Attempt Explicit Creds (ie. Task/RunAs)", EventCode==4767, "4767 Unlocked", EventCode==12294, "12294 Potential attack against Administrator", EventCode==4794, "4794 DSRM Admin PW Set Attempt", EventCode==4725, "4725 Disabled", EventCode==4722, "4722 Enabled", EventCode==4723, "4723 PW change attempt",EventCode==4724, "4724 PW reset attempt", EventCode==4738, "4738 Object changed", EventCode==4720, "4720 Created", EventCode==4726, "4726 Deleted", EventCode==4778, "4778 Session Reconnect", EventCode==4779, "4779 Session Disconnect", EventCode==4800, "4800 Wks Locked", EventCode==4801, "4801 Wks Unlocked", 1=1, EventCode)
| eval Logon_Type=case(Logon_Type==2, "2 Interactive", Logon_Type==3, "3 Network", Logon_Type==4, "4 Batch", Logon_Type==5, "5 Service", Logon_Type==7, "7 Unlock", Logon_Type==8, "8 NetworkClearText", Logon_Type==9, "9 NewCredentials", Logon_Type==10, "10 RemoteInteractive", Logon_Type==11, "11 CachedInteractive", 1=1, Logon_Type)
`comment(" | dedup Time, Account | dedup Account, Caller_Machine ")`
| table Time, Account, EventCode, ActionBy, Caller_Machine, Failure_Reason, Client_Address, Logon_Type, Logon_Process, Auth_Package, Caller_Process_Name, Process_Name, ComputerName, Workstation_Name, Src_Netw_Addr
`comment(" | stats count by Account, Caller_Machine, EventCode | sort -count ")`

Tips and Notes:

• Replace “index=*” with the appropriate index to search through. This can significantly shorten search time
  • You can find the list of available indexes by querying: “| eventcount summarize=false index=* index=_* | dedup index | fields index“
• The portion “OR Account_Name=Michael.Yuen@*” exists solely to gather information for Event ID 4770 (Kerberos). It can be removed if Event 4770 is not needed
• To summarize events as hit counts, uncomment the “stats count” line
  • Can more quickly aid in identifying which Caller_Machine/Events are most affected
• Commands are case-sensitive
• Replace “Account_name=” with “ComputerName=XYZ” [where “XYZ” is either a computer’s FQDN or “Name*” (wildcard)] to search for events logged by that machine
• “1=1, EventCode” means: Default = EventCode itself (to catch any value not matching any Case evaluations)
• “mvindex(Account_Name,0)” means: Get the first value of “Account_Name” (to select from multiple values)

Search for Non-Lockout-related Events

This query will help figure out where the account may have been used successfully within the last 14 days, including logons, logoffs, and which machines, IP addresses, and/or processes contributed to the event. Optionally, it will also show who unlocked the account.

Non-lockout related events

index="*" Account_Name=Michael.Yuen OR Account_Name=Michael.Yuen@*
earliest=-14d latest=now() source=WinEventLog:Security
NOT (EventCode=4740 OR EventCode=4625 OR EventCode=644 OR EventCode=529 OR EventCode=675 OR EventCode=676 OR EventCode=681 OR EventCode=4771 OR EventCode=4770 OR EventCode=4768 OR EventCode=4776 OR EventCode=4777) NOT EventCode=4767 NOT (EventCode=6272 OR EventCode=6273 OR EventCode=6274 OR EventCode=6275) NOT (EventCode=4627 OR EventCode=4780)
| eval Account0=mvindex(Account_Name,0) | eval Account1=mvindex(Account_Name,1)
| eval Account=case(EventCode==4624,Account1, EventCode==4625,Account1, EventCode==4648,Account1, EventCode==4722,Account1, EventCode==4723,Account1, EventCode==4724,Account1, EventCode==4725,Account1, EventCode==4738,Account1, EventCode==4740,Account1, EventCode==4767,Account1, EventCode==4768,Account0, EventCode==4769,Account0, EventCode==4771,Account0, EventCode==4770,Account0, EventCode==5140,Account0, EventCode==4778,Account0, EventCode==4779,Account0, EventCode==4800,Account0, EventCode==4801,Account0) | fillnull Value="-" Account
| eval ActionBy=case(EventCode==4725,Account0, EventCode==4722,src_user, EventCode==4767,src_user, EventCode==4723,src_user, EventCode==4724,src_user, EventCode==4738,src_user, EventCode==4794,src_user)
| eval Time=strftime(_time, "%m/%d/%y %H:%M:%S") | sort -_time
| eval Caller_Machine=if(Caller_Machine_Name!= NULL, Caller_Machine_Name, Caller_Computer_Name) | fillnull Value="-" Caller_Machine
| rex field=Process_Name "(?P<Process_Name>[^\\\]+)$" | fillnull Value="-" Process_Name
| rex field=Caller_Process_Name "(?P<Caller_Process_Name>[^\\\]+)$" | fillnull Value="-" Caller_Process_Name
| rename "Authentication_Package" as "Auth_Package", "Source_Network_Address" as "Src_Netw_Addr"
| replace "MICROSOFT_AUTHENTICATION_PACKAGE_V1_0" with "MsAuthPkgV1_0" in Auth_Package
| replace "Microsoft Unified Security Protocol Provider" with "MsUnifiedSecProt" in Auth_Package
| eval EventCode=case(EventCode==4740, "4740 Locked", EventCode==4625, "4625 Logon Failed", EventCode==644, "644 Locked", EventCode==529, "529 Logon Failed", EventCode==4768, "4768 Kerb TGT Req", EventCode==4771, "4771 Kerb Pre-Auth Failed", EventCode==4770, "4770 Kerb Svc Tkt Renewed", EventCode==4624, "4624 Logon OK", EventCode==4648, "4648 Logon Attempt Explicit Creds (ie. Task/RunAs)", EventCode==4767, "4767 Unlocked", EventCode==12294, "12294 Potential attack against Administrator", EventCode==4794, "4794 DSRM Admin PW Set Attempt", EventCode==4725, "4725 Disabled", EventCode==4722, "4722 Enabled", EventCode==4723, "4723 PW change attempt",EventCode==4724, "4724 PW reset attempt", EventCode==4738, "4738 Object changed", EventCode==4720, "4720 Created", EventCode==4726, "4726 Deleted", EventCode==4778, "4778 Session Reconnect", EventCode==4779, "4779 Session Disconnect", EventCode==4800, "4800 Wks Locked", EventCode==4801, "4801 Wks Unlocked", 1=1, EventCode)
| eval Logon_Type=case(Logon_Type==2, "2 Interactive", Logon_Type==3, "3 Network", Logon_Type==4, "4 Batch", Logon_Type==5, "5 Service", Logon_Type==7, "7 Unlock", Logon_Type==8, "8 NetworkClearText", Logon_Type==9, "9 NewCredentials", Logon_Type==10, "10 RemoteInteractive", Logon_Type==11, "11 CachedInteractive", 1=1, Logon_Type)
| table Time, Account, TaskCategory, EventCode, ActionBy, Logon_Type, Logon_Process, Auth_Package, Caller_Process_Name, Process_Name, ComputerName, Src_Netw_Addr, Network_Address

Note: Remove “NOT EventCode=4767” in the query above, if you want to also see who unlocked the account.

Search for ALL Events related to the account

This query combines lockout-related and non-lockout-related events from the last 14 days for a better look at the timeline leading up to a lockout.

All account-related events

index="*" Account_Name=Michael.Yuen OR Account_Name=Michael.Yuen@*
earliest=-14d latest=now() source=WinEventLog:Security
| eval Account0=mvindex(Account_Name,0) | eval Account1=mvindex(Account_Name,1)
| eval Account=case(EventCode==4624,Account1, EventCode==4625,Account1, EventCode==4648,Account1, EventCode==4722,Account1, EventCode==4723,Account1, EventCode==4724,Account1, EventCode==4725,Account1, EventCode==4738,Account1, EventCode==4740,Account1, EventCode==4767,Account1, EventCode==4768,Account0, EventCode==4769,Account0, EventCode==4771,Account0, EventCode==4770,Account0, EventCode==5140,Account0, EventCode==4778,Account0, EventCode==4779,Account0, EventCode==4800,Account0, EventCode==4801,Account0) | fillnull Value="-" Account
| eval ActionBy=case(EventCode==4725,Account0, EventCode==4722,src_user, EventCode==4767,src_user, EventCode==4723,src_user, EventCode==4724,src_user, EventCode==4738,src_user, EventCode==4794,src_user)
| eval Time=strftime(_time, "%m/%d/%y %H:%M:%S") | sort -_time
| eval Caller_Machine=if(Caller_Machine_Name!= NULL, Caller_Machine_Name, Caller_Computer_Name) | fillnull Value="-" Caller_Machine
| rex field=Process_Name "(?P<Process_Name>[^\\\]+)$" | fillnull Value="-" Process_Name
| rex field=Caller_Process_Name "(?P<Caller_Process_Name>[^\\\]+)$" | fillnull Value="-" Caller_Process_Name
| rename "Authentication_Package" as "Auth_Package", "Source_Network_Address" as "Src_Netw_Addr"
| replace "MICROSOFT_AUTHENTICATION_PACKAGE_V1_0" with "MsAuthPkgV1_0" in Auth_Package
| replace "Microsoft Unified Security Protocol Provider" with "MsUnifiedSecProt" in Auth_Package
| eval EventCode=case(EventCode==4740, "4740 Locked", EventCode==4625, "4625 Logon Failed", EventCode==644, "644 Locked", EventCode==529, "529 Logon Failed", EventCode==4768, "4768 Kerb TGT Req", EventCode==4771, "4771 Kerb Pre-Auth Failed", EventCode==4770, "4770 Kerb Svc Tkt Renewed", EventCode==4624, "4624 Logon OK", EventCode==4648, "4648 Logon Attempt Explicit Creds (ie. Task/RunAs)", EventCode==4767, "4767 Unlocked", EventCode==12294, "12294 Potential attack against Administrator", EventCode==4794, "4794 DSRM Admin PW Set Attempt", EventCode==4725, "4725 Disabled", EventCode==4722, "4722 Enabled", EventCode==4723, "4723 PW change attempt",EventCode==4724, "4724 PW reset attempt", EventCode==4738, "4738 Object changed", EventCode==4720, "4720 Created", EventCode==4726, "4726 Deleted", EventCode==4778, "4778 Session Reconnect", EventCode==4779, "4779 Session Disconnect", EventCode==4800, "4800 Wks Locked", EventCode==4801, "4801 Wks Unlocked", 1=1, EventCode)
| eval Logon_Type=case(Logon_Type==2, "2 Interactive", Logon_Type==3, "3 Network", Logon_Type==4, "4 Batch", Logon_Type==5, "5 Service", Logon_Type==7, "7 Unlock", Logon_Type==8, "8 NetworkClearText", Logon_Type==9, "9 NewCredentials", Logon_Type==10, "10 RemoteInteractive", Logon_Type==11, "11 CachedInteractive", 1=1, Logon_Type)
| table Time, Account, TaskCategory, EventCode, ActionBy, Caller_Machine, Failure_Reason, Client_Address, Logon_Type, Logon_Process, Auth_Package, Caller_Process_Name, Process_Name, ComputerName, Workstation_Name, Src_Netw_Addr, Network_Address

Search for Login Events related to the account

This query surfaces login-related events, excluding Kerberos Logon Process (to reduce noise) from the last 14 days. You DO want to look at Kerberos logons to get a complete picture but checking without them is a good start.

Login related events (without Kerberos)

index="*" Account_Name=Michael.Yuen OR Account_Name=Michael.Yuen@*
earliest=-14d latest=now() source=WinEventLog:Security
(EventCode=4625 OR EventCode=4740 OR EventCode=529 OR EventCode=644 OR EventCode=681 OR EventCode=4624 OR EventCode=4648 OR EventCode=4767 OR EventCode=4725 OR EventCode=4722 OR EventCode=4724 OR EventCode=4738 OR EventCode=4723 OR EventCode=4800 OR EventCode=4801 OR EventCode=4722 OR EventCode=4725)
NOT Logon_Process=Kerberos
| eval Account0=mvindex(Account_Name,0) | eval Account1=mvindex(Account_Name,1)
| eval Account=case(EventCode==4624,Account1, EventCode==4625,Account1, EventCode==4648,Account1, EventCode==4722,Account1, EventCode==4723,Account1, EventCode==4724,Account1, EventCode==4725,Account1, EventCode==4738,Account1, EventCode==4740,Account1, EventCode==4767,Account1, EventCode==4768,Account0, EventCode==4769,Account0, EventCode==4771,Account0, EventCode==4770,Account0, EventCode==5140,Account0) | fillnull Value="-" Account
| eval ActionBy=case(EventCode==4725,Account0, EventCode==4722,src_user, EventCode==4767,src_user, EventCode==4723,src_user, EventCode==4724,src_user, EventCode==4738,src_user)
| eval Time=strftime(_time, "%m/%d/%y %H:%M:%S") | sort -_time
| eval Caller_Machine=if(Caller_Machine_Name!= NULL, Caller_Machine_Name, Caller_Computer_Name) | fillnull Value="-" Caller_Machine
| rex field=Process_Name "(?P<Process_Name>[^\\\]+)$" | fillnull Value="-" Process_Name
| rex field=Caller_Process_Name "(?P<Caller_Process_Name>[^\\\]+)$" | fillnull Value="-" Caller_Process_Name
| rename "Authentication_Package" as "Auth_Package", "Source_Network_Address" as "Src_Netw_Addr"
| replace "MICROSOFT_AUTHENTICATION_PACKAGE_V1_0" with "MsAuthPkgV1_0" in Auth_Package
| replace "Microsoft Unified Security Protocol Provider" with "MsUnifiedSecProt" in Auth_Package
| eval EventCode=case(EventCode==4740, "4740 Locked", EventCode==4625, "4625 Logon Failed", EventCode==644, "644 Locked", EventCode==529, "529 Logon Failed", EventCode==4768, "4768 Kerb TGT Req", EventCode==4771, "4771 Kerb Pre-Auth Failed", EventCode==4770, "4770 Kerb Svc Ticket Renewed", EventCode==4624, "4624 Logon OK", EventCode==4648, "4648 Logon Attempt Explicit Creds (ie. Task/RunAs)", EventCode==4767, "4767 Unlocked", EventCode==12294, "12294 Potential attack against Administrator", EventCode==4725, "4725 Disabled", EventCode==4722, "4722 Enabled", EventCode==4723, "4723 PW change attempt",EventCode==4724, "4724 PW reset attempt", EventCode==4738, "4738 Object changed", EventCode==4720, "4720 Created", EventCode==4726, "4726 Deleted", 1=1, EventCode)
| eval Logon_Type=case(Logon_Type==2, "2 Interactive", Logon_Type==3, "3 Network", Logon_Type==4, "4 Batch", Logon_Type==5, "5 Service", Logon_Type==7, "7 Unlock", Logon_Type==8, "8 NetworkClearText", Logon_Type==9, "9 NewCredentials", Logon_Type==10, "10 RemoteInteractive", Logon_Type==11, "11 CachedInteractive", 1=1, Logon_Type)
| table Time, Account, TaskCategory, EventCode, ActionBy, Caller_Machine, Failure_Reason, Client_Address, Logon_Type, Logon_Process, Auth_Package, Caller_Process_Name, Process_Name, ComputerName, Workstation_Name, Src_Netw_Addr, Network_Address

Bonus: Search for Account Modifications — and by Whom

This query searches through the last 30 days for specific account modification events — and by whom, including: Disable, Enable, Unlock, Modify, Create, Delete, Password Reset

Account modifications and by whom

index="*" Account_Name=Michael.Yuen
earliest=-30d latest=now() source=WinEventLog:Security
(EventCode=4767 OR EventCode=4725 OR EventCode=4722 OR EventCode=4723 OR EventCode=4724 OR EventCode=4738 OR EventCode=4720 OR EventCode=4726)
| eval ActionBy=src_user | eval Account=mvindex(Security_ID,1)
| eval Time=strftime(_time, "%m/%d/%y %H:%M:%S") | sort -_time
| eval EventCode=case(EventCode==4767, "4767 Unlocked", EventCode==4725, "4725 Disabled", EventCode==4722, "4722 Enabled", EventCode==4723, "4723 PW change attempt", EventCode==4724, "4724 PW reset attempt", EventCode==4738, "4738 Object changed", EventCode==4720, "4720 Created", EventCode==4726, "4726 Deleted", 1=1, EventCode)
| table Time, Account, EventCode, ActionBy, ComputerName

Now that Splunk has produced some reports, how do you interpret them? This will be covered in the next section.

Tip: The Splunk results can be downloaded as CSV, XML, and JSON files for further reporting and analysis. They can also be printed as PDF, although at the time of writing, the orientation is locked in Portrait mode

Interpreting Account Activity

The amount of data that can be collected about an account’s activity can be very overwhelming. “Event ID 4740” — what is that? What does “Logon Type 3” mean? What about “Network Address” and “Caller Machine“? What if a lockout shows the source to be a Domain Controller? Does that mean an unauthorized user attempted to log in to the DC, and if so, why?

Windows Event IDs

Before we can interpret the Splunk results (or output provided by Microsoft’s EventCombMT and other tools), let’s look at the various Event IDs that are related to account activity.

Security Audit? See our Group and Membership Changes article for security-related event IDs.
Stuck? See additional notes on Event IDs 4769 and 4625 below.

Cybersecurity Defense
As an added bonus, I included Events (marked with “Security” under “Type” column, such as 4688, 4697) that can help identify possible persistence on a Windows system, one of the primary goals for an Attacker to establish after exploiting a Target. Also take a look at my “Cybersecurity Warning: Kerberos + KRBTGT” note in this article.

In the above table, I have separated the events by two types: Lockout and Info (a subsection of which is of interest for a Cybersecurity Defender). The former are general sources that could contribute to a lockout, and the latter are informational to help gain an understanding of what had occurred. The ones to pay particular attention to:

• 4740: Account has been locked
• 4625: There was a failed logon attempt
  • Tip: Also read the “Event ID 4625 (Account failed to log on) Note” for additional leads

Lockout-related events

Example Interpretation

When I troubleshoot an account, I look for a group of 4625 events that may have led to the 4740 lock event. The message within 4740 would tell you what ultimately caused the lock. Keep in mind that although 4625 reports a failed logon, it may not necessarily have been the culprit. It may have simply failed because the account was ALREADY locked as illustrated in the below 4625 sample.

Event ID 4740 – Locked

Event ID 4740: The account, “DOMAIN\MichaelYuen” was locked out by “Caller Computer Name“, “MyComputer1”. “DC01” logged this event. The lock likely came from “MyComputer1”. Start troubleshooting there.

If “Caller Computer Name” is blank/null or a Domain Controller, look at the “Process Information” and “Network Information” fields next:

• Process Information: the process on the computer that requested the logon
• Network Information: the remote computer that requested the logon

The next example provides more details on both.

Event ID 4625 – Logon Failure

Event ID 4625: “DC02” (from the Subject field) reported the logon failure for “Account Name: MichaelYuen” and cites “Failure Reason: Account locked out”. The account could not log in because it was ALREADY locked.

• The Caller Process, “lsass.exe” (Windows login service), on the remote computer with IP of 10.1.1.100 attempted the login. Perform a DNS/WINS lookup (example: “ping -a 10.1.1.100”) to find the name of that computer/server and start troubleshooting there.
  • “svchost.exe” is used to launch Windows services
• Under Network Information, the Workstation Name is “DC02”, a Domain Controller. It often also can be blank
  • In my experience, when the Caller Computer Name or Workstation Name are either blank or a DC, the request likely came from a non-Windows machine, such as a Linux/Unix server or an appliance based on those operating systems (ie. F5 Big-IP and Citrix Netscaler load balancers or a VMware Host)
• Logon Type is listed as “3”. That is another clue we can use to find out how the offending computer or process tried to log in
• Be sure to also take a look at the Logon Process and Authentication Package that will be discussed later

Event ID 4625 (Account failed to log on) Note

Over the years, I had found that many admins were confused about Event ID 4625 in particular. The confusion could come from not looking at the event’s message details. Thus, some admins may interpret all the hosts having logged 4625 ID as the source for the logon failures. That may NOT necessarily be true.

Event ID 4625 Task Categories

Look at the “TaskCategory” or “Failure Reason” messages associated with each Event ID 4625. You may find that the failure was NOT due to a bad password. Instead, account may have failed to log in BECAUSE the account was ALREADY locked.

If the Event ID 4625 message contains “TaskCategory=Logon” or “Failure Reason=Unknown user name or bad password”, those are the hosts that likely contributed to the lockout. A TaskCategory of “Account Lockout” or Failure Reason of “Account locked out” could simply indicate that the logon failure was the result of the account having ALREADY been locked.

Event ID 4769 (Kerberos Service Ticket Requested) Note

I did not get a chance to write more about this event, but it is worth querying for. It gets generated every time the KDC gets a Ticket Granting Service (TGS) ticket request (ie. attempt to map a drive to a file server) for both failures and successes. Read about it and the hex failure codes at Microsoft Learn. For example, code 0x17 is “KDC_ERR_KEY_EXPIRED” and indicates the user’s password had expired.

Tip: When all else fails, consider analyzing Kerberos events for any issues.

When you get stuck looking at the usual suspects, you should dive into the Kerberos events such as 4769 and 4771 — unfortunately, Kerberos is VERY chatty and will produce a boat load of data to sift through… but the lockout source(s) may be hidden in there. Visualizing the data with Splunk to look for patterns can be of tremendous help here. Also be sure to look at the “Event ID 4625 (Account failed to log on) Note” as another lead source.

Logon Types

Logon Type is the method an account tried to log in with. Let’s look at the various types.

In the example I had provided for Event ID 4625, the Logon Type was “3”. The logon was attempted over the network from a remote server — a VMware Host in that particular situation.

Lockout-related events

Logon Process, Authentication Package

When I created a Splunk Dashboard to visualize and analyze data, I noticed a large amount of Event ID 4624 (Successful logon) for a user. That led me to look into its “Detailed Authentication Information” section to see where the logins were coming from. Of note were Logon Process and Authentication Package that apply to Event IDs 4624 and 4625 (Failed logon).

The login process starts with credential collection that is passed on to the Local Security Authority (LSA) Server Service (LSASS) or Service Host (SvcHost). It subsequently works with a Logon Process (a Windows DLL, Dynamic Link Library) to handle the actual authentication attempt with an Authentication Package.

Event ID 4624/4625 – Logon Process, Authentication Package

As I understand it… LSASS or SvcHost initiates the logon, and Logon Process processes the authentication with an Authentication Package type.

Logon Processes

The Logon Process works with an Authentication Package to handle the actual authentication started by the LSASS or SvcHost process.

Authentication Packages

The Authentication Package is used by the Logon Process to handle the actual authentication attempt.

Kerberos is the default authentication package that replaced NTLM (NT LAN Manager) since Windows Server 2000. If Kerberos fails to authenticate the user, NTLM will be used instead.

Kerberos Authentication Process /ManageEngine

Final Thoughts

Looking for the suspect process, application, or computer can be a test of patience and perseverance, and having customers understandably vent their frustration in an effort to nudge you to find the source can be discouraging. Hopefully, the information and tools presented here will aid you towards a quick(er) resolution, and helped you understand how Active Directory tracks account activities. Furthermore, I encourage you to learn PowerShell, if you haven’t already, as it can be a very powerful tool to have in your IT arsenal.

Happy lockout hunting!

Related: Visualize Account Lockout events with my AD Lockout Splunk Dashboards to graphically identify patterns. For investigating Group-related events, see my Group and Membership Changes post.

Cybersecurity Warning: Kerberos + KRBTGT

Active Directory security is EXTREMELY important. One of the articles any domain administrator MUST read is one by Bryan Patton over at Quest. In it, he clearly describes what the special KRBTGT account is, what it is for, how it is used, and why its password should be changed regularly (but not too quickly). It is created with every AD domain.

Cybersecurity Defense: “Golden Ticket” is an attack type that gives a bad actor the ability to impersonate any user and talk to any Kerberos service in a domain without the need to authenticate to a domain controller or having a valid user account. The attack exploits the special domain user, KRBTGT, which is responsible for signing Kerberos TGTs using its own password’s NT Hash. Once a forged, signed ticket has been obtained, the Attacker can regain access even after getting kicked off the network.

Remediation would involve resetting the KRBTGT user’s password twice but do so with EXTREME caution! In fact, Bryan Patton’s article suggests doing the 2nd reset a week after the first one to give sufficient time for all DCs to have completed replication. Note: This should also be done after an administrator with access to KRBTGT has left the role or organization.

Additional References

• EventSentry: Event IDs
• Microsoft Docs: Windows Logon Scenarios / Audit Logon
• Randy Franklin Smith’s “Ultimate IT Security”: Logon/Logoff Events / Account Management Events
• Pearson’s Microsoft Press Store: Windows Logon
• Crowdstrike: NTLM vs Kerberos
• ManageEngine: Kerberos Authentication Ticket Request (Event ID 4768)
• Microsoft Learn: Kerberos Service Ticket Request (Event ID 4769)
• Sophos: Interesting Event IDs for Malware/General Investigation

Related Posts

• Active Directory
  • Visualize Account Lockouts with Splunk Dashboards
  • Group and Membership Changes
• Azure Active Directory
  • Audit Role Changes (ie. Global Administrator)
  • Successful Logins from Unknown Users

Credits:
– Featured Image by Warren Wong via Unsplash