Hello everyone. Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help.Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still ondomainlevel=0. I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistenciesbetween replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas. I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist. ipactl startStarting Directory ServiceFailed to read data from service file: Failed to get list of services to probe status!Configured hostname 'server.pop.domain.local' does not match any master server in LDAP:No master found because of error: no such entryShutting down
cat errors[26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password.[26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set.[26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers[26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled[26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled[26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled[26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled[26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled[26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled[26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled[26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled[26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled[26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled[26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled[26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled[26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled[26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled[26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled[26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled[26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled[26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled[26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled[26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled[26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled[26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled[26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled[26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled[26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled[26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled[26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled[26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled[26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2[26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up[26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match[26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize.[26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup![26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist[26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local)[26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped[26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds![26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests[26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests[26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests[26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1[26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate[26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins[26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop[26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped[26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects[26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup! [26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
We have a number of servers in different pops. When I say intermittent I mean it doesn't just happen on the same server again and again but rather on random servers each time. There is no pattern as far as which pop or time of day etc. I do ipactl status and see that dirsrv is STOPPED. ipactl restart doesn't help, I just get the below errormessage that ipa can't start without 389ds and to check journalctl. No matter what I've tried I never managed to fix the problem properly. I just blow the replica out and reinstall. I've sanitized the file. The servers are actually named something completely different than what's in logs below.
thank you and please let me know what other steps I should try.
From: Rob Crittenden rcritten@redhat.com To: pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org Sent: Thursday, December 28, 2017 2:26 PM Subject: Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup! [26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
pgb205 wrote:
We have a number of servers in different pops. When I say intermittent I mean it doesn't just happen on the same server again and again but rather on random servers each time. There is no pattern as far as which pop or time of day etc.
I do ipactl status and see that dirsrv is STOPPED. ipactl restart doesn't help, I just get the below error message that ipa can't start without 389ds and to check journalctl.
No matter what I've tried I never managed to fix the problem properly. I just blow the replica out and reinstall.
I've sanitized the file. The servers are actually named something completely different than what's in logs below.
thank you and please let me know what other steps I should try.
Like I said, this will blow up if the hostname is an unknown master so I'd start there. Check the list of masters and ensure the host is there (hostname -f)
If dirsrv is stopped you should look for a core or some indication of why it is stopped.
rob
*From:* Rob Crittenden rcritten@redhat.com *To:* pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org *Sent:* Thursday, December 28, 2017 2:26 PM *Subject:* Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server
startup!
[26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
mailto:freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org mailto:freeipa-users-leave@lists.fedorahosted.org
as far as hostname it's there on both failed replica with hostname -f commandbut also on the replica that it's connected to.on the neighbor replica I can ping failed replica by fqdnand it shows up in ipa-replica-manage list
From: Rob Crittenden rcritten@redhat.com To: pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org Sent: Tuesday, January 2, 2018 11:43 AM Subject: Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 wrote:
We have a number of servers in different pops. When I say intermittent I mean it doesn't just happen on the same server again and again but rather on random servers each time. There is no pattern as far as which pop or time of day etc.
I do ipactl status and see that dirsrv is STOPPED. ipactl restart doesn't help, I just get the below error message that ipa can't start without 389ds and to check journalctl.
No matter what I've tried I never managed to fix the problem properly. I just blow the replica out and reinstall.
I've sanitized the file. The servers are actually named something completely different than what's in logs below.
thank you and please let me know what other steps I should try.
Like I said, this will blow up if the hostname is an unknown master so I'd start there. Check the list of masters and ensure the host is there (hostname -f)
If dirsrv is stopped you should look for a core or some indication of why it is stopped.
rob
*From:* Rob Crittenden rcritten@redhat.com *To:* pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org *Sent:* Thursday, December 28, 2017 2:26 PM *Subject:* Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server
startup!
[26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
mailto:freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org mailto:freeipa-users-leave@lists.fedorahosted.org
I have also checked on the neighboring replica and can see the broken server in ldapsearch -b "cn=masters, cn=ipa, cn=etc, dc=domain,dc=local" -D cn="directory manager" -w <pass> "(objectclass=ipaReplTopoManagedServer)" output. so other servers are not losing the information. Just somehow broken replica loses its own hostname in this list.
From: Rob Crittenden rcritten@redhat.com To: pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org Sent: Thursday, December 28, 2017 2:26 PM Subject: Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup! [26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
pgb205 via FreeIPA-users wrote:
I have also checked on the neighboring replica and can see the broken server in
ldapsearch -b "cn=masters, cn=ipa, cn=etc, dc=domain,dc=local" -Dcn="directory manager" -w <pass> "(objectclass=ipaReplTopoManagedServer)"
output.
so other servers are not losing the information. Just somehow broken replica loses its own hostname in this list.
You might want to dig through the access log on that master to look for any changes to cn=masters.
You might also consider enabling the audit log to get more details if you find this but note that this logs EVERYTHING (including password changes) so be very careful with this log.
I don't think entries will disappear on their own. Why an entry can disappear only one one box is a bit of a mystery though.
rob
*From:* Rob Crittenden rcritten@redhat.com *To:* pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org *Sent:* Thursday, December 28, 2017 2:26 PM *Subject:* Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server
startup!
[26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
mailto:freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org mailto:freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Rob, thanks. This is what I've done with nsslapd-auditlog-logging-enabled:onI hope that will provide some answers should the entry disappear again.
From: Rob Crittenden rcritten@redhat.com To: pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org Sent: Wednesday, January 3, 2018 4:36 PM Subject: Re: [Freeipa-users] Re: Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
I have also checked on the neighboring replica and can see the broken server in
ldapsearch -b "cn=masters, cn=ipa, cn=etc, dc=domain,dc=local" -D cn="directory manager" -w <pass> "(objectclass=ipaReplTopoManagedServer)"
output.
so other servers are not losing the information. Just somehow broken replica loses its own hostname in this list.
You might want to dig through the access log on that master to look for any changes to cn=masters.
You might also consider enabling the audit log to get more details if you find this but note that this logs EVERYTHING (including password changes) so be very careful with this log.
I don't think entries will disappear on their own. Why an entry can disappear only one one box is a bit of a mystery though.
rob
*From:* Rob Crittenden rcritten@redhat.com *To:* pgb205 pgb205@yahoo.com; FreeIPA users list freeipa-users@lists.fedorahosted.org *Sent:* Thursday, December 28, 2017 2:26 PM *Subject:* Re: [Freeipa-users] Failed to read service file. Hostname does not match any master server in LDAP
pgb205 via FreeIPA-users wrote:
Hello everyone.
Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help. Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on domainlevel=0.
I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas.
I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.
ipactl start Starting Directory Service Failed to read data from service file: Failed to get list of services to probe status! Configured hostname 'server.pop.domain.local' does not match any master server in LDAP: No master found because of error: no such entry Shutting down
This isn't exactly a crash. In what context are you restarting it?
You said it is intermittent, does it ever start working again on its own?
Is this the correct hostname?
IPA uses the hostname to look in LDAP for the list of enabled services on a given host to know what to start.
rob
cat errors [26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password. [26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set. [26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers [26/Dec/2017:21:15:56.236652729 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.236921632 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237114079 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.237317678 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.237526365 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.237746660 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.237908539 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.238087338 +0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.238306056 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.238517868 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238724920 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.238889982 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239048124 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.239233534 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239402097 +0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.239767245 +0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.239997083 +0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.240177269 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [26/Dec/2017:21:15:56.240376177 +0000] SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [26/Dec/2017:21:15:56.240585031 +0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.240745192 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [26/Dec/2017:21:15:56.240897126 +0000] SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [26/Dec/2017:21:15:56.241075071 +0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled [26/Dec/2017:21:15:56.241245788 +0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241456256 +0000] SSL alert: TLS_AES_256_GCM_SHA384: enabled [26/Dec/2017:21:15:56.241617090 +0000] SSL alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241766851 +0000] SSL alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.241947040 +0000] SSL alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled [26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up [26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize. [26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server
startup!
[26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local) [26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped [26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! [26/Dec/2017:21:15:56.434117007 +0000] slapd started. Listening on All Interfaces port 389 for LDAP requests [26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests [26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests [26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1 [26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate [26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins [26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop [26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped [26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects [26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
mailto:freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org mailto:freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
freeipa-users@lists.fedorahosted.org
-
pgb205 -
Rob Crittenden