Active Directory 서버의 NETLOGON 서비스가 멈췄을 때 발생하는 문제
Active Directory 서버의 NETLOGON 서비스가 시작되지 않으면, 우선 SQL 서버의 SPN 등록이 안되어 이후 SQL 서버로 접속하는 모든 서비스들이 장애를 겪게 됩니다.
우선, TFS 서버는 다음과 같은 이벤트를 남기며 서비스가 안됩니다.
Log Name: Application
Source: TFS Services
Date: 2016-10-21 오후 1:54:23
Event ID: 9005
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: testtfs.testad.com
Description:
TF53010: Team Foundation 구성 요소 또는 확장에서 다음 오류가 발생했습니다.:
날짜(UTC): 2016-10-21 오전 4:54:23
컴퓨터: testtfs
응용 프로그램 도메인: /LM/W3SVC/5/ROOT/tfs-1-131214991278339538
어셈블리: Microsoft.TeamFoundation.Framework.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319
서비스 호스트:
프로세스 정보:
프로세스 이름: w3wp
프로세스 ID: 3680
스레드 ID: 3984
계정 이름: TESTAD\tfsadmin
세부 메시지: TF30059: 웹 서비스를 초기화하는 동안 심각한 오류가 발생했습니다.
웹 요청 정보
URL: http://testtfs.testad.com:8083/tfs/DefaultCollection/VersionControl/v1.0/repository.asmx [메서드: POST]
사용자 에이전트: Team Foundation (devenv.exe, 14.102.25521.0, Enterprise, SKU:37)
헤더: 사용할 수 없음
경로: /tfs/DefaultCollection/VersionControl/v1.0/repository.asmx
로컬 요청: False
호스트 주소: 192.168.0.1
사용자: 사용할 수 없음 [인증 형식: 사용할 수 없음]
예외 메시지: TF53001: 관리자가 데이터베이스 작업을 취소했습니다.(DatabaseOperationCanceledException 형식)
Log Name: Application
Source: TFS Services
Date: 2016-10-21 오후 1:54:23
Event ID: 9001
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: testtfs.testad.com
Description:
TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 2016-10-21 오전 4:54:23
Machine: testtfs
Application Domain: /LM/W3SVC/5/ROOT/tfs-1-131214991278339538
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319
Service Host:
Process Details:
Process Name: w3wp
Process Id: 3680
Thread Id: 3984
Account name: TESTAD\tfsadmin
Detailed Message: Application Request Processing Started
Server Version = Microsoft.TeamFoundation.Framework.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Service Account = TESTAD\tfsadmin
Machine Name = testtfs
Exception Message: TF53001: The database operation was canceled by an administrator. (type DatabaseOperationCanceledException)
Exception Data Dictionary:
{421AC3F1-A306-4C9B-B3F6-5812F9121FC8} = True
Exception Stack Trace: at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(SqlException sqlException)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(Exception exception)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteReader()
at Microsoft.TeamFoundation.Framework.Server.ResourceManagementComponent.GetServiceVersion(String serviceName)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationResourceManagementService.ReadServiceVersionFromDatabase(ISqlConnectionInfo connectionInfo, Boolean handleNoResourceManagementSchema, String serviceName, Boolean& resourceManagementSchemaExists)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationResourceManagementService.GetServiceVersionRaw[TComponent](ISqlConnectionInfo connectionInfo, ServiceVersionEntry& serviceVersionEntry, IComponentCreator& componentCreator, Boolean handleNoResourceManagementSchema, Boolean throwExceptions, ComponentFactory& factory, Boolean& resourceManagementSchemaExists)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationResourceManagementService.CreateComponentRaw[TComponent](ISqlConnectionInfo connectionInfo, Int32 commandTimeout, Int32 deadlockPause, Int32 maxDeadlockRetries, Boolean handleNoResourceManagementSchema, Boolean verifyServiceVersion, Boolean throwExceptions, ITFLogger logger)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationDatabaseManagementService.GetConfigurationDatabaseBootstrap(ISqlConnectionInfo configDbConnectionInfo)
at Microsoft.TeamFoundation.Framework.Server.DeploymentServiceHost..ctor(HostProperties hostProperties, ISqlConnectionInfo connectionInfo, Boolean failOnInvalidConfiguration)
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationApplicationCore.ApplicationStart()
Inner Exception Details:
Exception Message: The target principal name is incorrect. Cannotgenerate SSPI context. (type SqlException)
SQL Exception Class: 11
SQL Exception Number: 0
SQL Exception Procedure: GenClientContext
SQL Exception Line Number: 0
SQL Exception Server: testdb
SQL Exception State: 0
SQL Error(s):
Exception Data Dictionary:
HelpLink.ProdName = Microsoft SQL Server
HelpLink.EvtSrc = MSSQLServer
HelpLink.EvtID = 0
HelpLink.BaseHelpUrl = http://go.microsoft.com/fwlink
HelpLink.LinkId = 20476
Exception Stack Trace: at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior)
Web Request Details
Url: http://testtfs.testad.com:8083/tfs/DefaultCollection/VersionControl/v1.0/repository.asmx [method: POST]
User Agent: Team Foundation (devenv.exe, 14.102.25521.0, Enterprise, SKU:37)
Headers: not available
Path: /tfs/DefaultCollection/VersionControl/v1.0/repository.asmx
Local Request: False
Host Address: 192.168.0.1
User: not available [authentication type: not available]
TFS 계정을 조회하는 명령어를 내려보면 이런 오류가 발생하는데,
C:\WINDOWS\system32>setspn -L TESTAD\tfsadmin
FindDomainForAccount: Call to DsGetDcNameWithAccountW failed with return value 0x0000054B
Could not find account tfsadmin
원래는 다음과 같은 출력 결과를 보여야 정상입니다.
D:\>setspn -L testad\tfsadmin
Registered ServicePrincipalNames for CN=tfsadmin,CN=Users,DC=testad,DC=com:
재미있는 것은 SQL 서버의 SPN을 Active Directory에서 조회해 보면 다음과 같이 잘 나옵니다.
C:\WINDOWS\system32>setspn -q MSSQLSvc/testdb.testad.com:1433
Checking domain DC=TESTAD,DC=com
CN=testdb,CN=Computers,DC=TESTAD,DC=com
MSSQLSvc/testdb.testad.com:1433
MSSQLSvc/testdb.testad.com
WSMAN/testdb.testad.com
WSMAN/testdb
TERMSRV/testdb.testad.com
TERMSRV/testdb
RestrictedKrbHost/testdb
HOST/testdb
RestrictedKrbHost/testdb.testad.com
HOST/testdb.testad.com
Existing SPN found!
그런데, DB 서버에서는 (AD의 NETLOGON 서비스가 멈춰 있으므로) 다음과 같은 오류가 발생합니다.
C:\WINDOWS\system32>setspn -q MSSQLSvc/testdb.testad.com:1433
Ldap Error(0x51 -- Server Down): ldap_connect
Failed to retrieve DN for domain "" : 0x00000051
Warning: No valid targets specified, reverting to current domain.
Ldap Error(0x51 -- Server Down): ldap_connect
Error occurred when searching for existing SPN: 0x00000051
Active Directory 서버에 함께 설치된 DNS의 관리 콘솔을 실행해도 오류가 발생하고,
The server TESTADSVR could not be contacted.
The error was:
The server is unavailable
"Active Directory Domain Services" 관리 콘솔을 실행해도,
Naming information cannot be located because:
The interface is unknown.
Contact your system administrator to verify that your domain is properly configured and is currently online.
"Group Policy Management Console" 관리 콘솔을 실행해도,
The specified domain controller could not be contacted. This affects the following domain in the console.
Domain: testad.com
The error was:
The specified domain controller could not be contacted.
"Active Directory Domains and Trusts" 관리 콘솔을 실행해도 역시 오류가 발생합니다.
You cannot modify domain or trust information because a Primary Domain Controller (PDC) emulator cannot be contacted. Please verify that the PDC emulator for the current domain and the network are both online and functioning properly.
The configuration information describing this enterprise is not available.
The interface is unknown.
다행히 dcdiag를 실행하면 NETLOGON 서비스가 멈춰있다고 알려주기는 합니다.
C:\WINDOWS\system32>dcdiag /fix
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = testadsvr
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\testadsvr
Starting test: Connectivity
......................... testadsvr passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\testadsvr
Starting test: Advertising
Fatal Error:DsGetDcName (testadsvr) call failed, error 1355
The Locator could not find the server.
......................... testadsvr failed test Advertising
Starting test: FrsEvent
......................... testadsvr passed test FrsEvent
Starting test: DFSREvent
......................... testadsvr passed test DFSREvent
Starting test: SysVolCheck
......................... testadsvr passed test SysVolCheck
Starting test: KccEvent
......................... testadsvr passed test KccEvent
Starting test: KnowsOfRoleHolders
......................... testadsvr passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... testadsvr passed test MachineAccount
Starting test: NCSecDesc
......................... testadsvr passed test NCSecDesc
Starting test: NetLogons
......................... testadsvr passed test NetLogons
Starting test: ObjectsReplicated
......................... testadsvr passed test ObjectsReplicated
Starting test: Replications
......................... testadsvr passed test Replications
Starting test: RidManager
......................... testadsvr passed test RidManager
Starting test: Services
DFSR Service is stopped on [testadsvr]
w32time Service is stopped on [testadsvr]
Invalid service startup type: NETLOGON on testadsvr, current value DEMAND_START, expected value AUTO_START
NETLOGON Service is stopped on [testadsvr]
......................... testadsvr failed test Services
Starting test: SystemLog
An error event occurred. EventID: 0xC00038D4
Time Generated: 10/21/2016 13:35:40
Event String:
The DFS Namespace service could not initialize the trusted domain information on this domain controller, but it will periodically retry the operation. The return code is in the record data.
......................... testadsvr failed test SystemLog
Starting test: VerifyReferences
......................... testadsvr passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running partition tests on : TESTAD
Starting test: CheckSDRefDom
......................... TESTAD passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... TESTAD passed test CrossRefValidation
Running enterprise tests on : testad.com
Starting test: LocatorCheck
Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1355
A Global Catalog Server could not be located - All GC's are down.
Warning: DcGetDcName(PDC_REQUIRED) call failed, error 1355
A Primary Domain Controller could not be located.
The server holding the PDC role is down.
Warning: DcGetDcName(TIME_SERVER) call failed, error 1355
A Time Server could not be located.
The server holding the PDC role is down.
Warning: DcGetDcName(GOOD_TIME_SERVER_PREFERRED) call failed, error 1355
A Good Time Server could not be located.
Warning: DcGetDcName(KDC_REQUIRED) call failed, error 1355
A KDC could not be located - All the KDCs are down.
......................... testad.com failed test LocatorCheck
Starting test: Intersite
......................... testad.com passed test Intersite
C:\WINDOWS\system32>
[이 글에 대해서 여러분들과 의견을 공유하고 싶습니다. 틀리거나 미흡한 부분 또는 의문 사항이 있으시면 언제든 댓글 남겨주십시오.]