Hello,
We have a 3 node multi-master IPA setup. These are running on Red Hat Enterprise Linux Server release 7.7 (Maipo) and all are version:
Name : ipa-server Arch : x86_64 Version : 4.6.5 Release : 11.el7_7.3
Starting yesterday, we are getting the following messages approximately every 3 seconds from lidm01:
[19/Nov/2019:15:48:24.221023177 -0500] - ERR - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:24.222186079 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389): Missing data encountered. If the error persists the replica must be reinitialized. [19/Nov/2019:15:48:25.229931823 -0500] - ERR - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:25.231001956 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389): Missing data encountered. If the error persists the replica must be reinitialized.
There are no error messages on lidm02 or lidm03. I ran the command 'ipa-replica-manage re-initalize --from lidm01.idm.domain.edu’ on both lidm02 and lidm03. When that did not change anything I ran the command 'ipa-replica-manage re-initalize --from lidm02.idm.domain.edu’ on lidm01. This did not change anything either. I have run restart-dirsrv on all 3 idm servers as well.
Also, here are the ipa-replica-manage list-ruv output on the 3 servers:
Lidm01: Replica Update Vectors: lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9 lidm03.idm.domain.edu:389: 6
Lidm02: Replica Update Vectors: lidm02.idm.domain.edu:389: 8 lidm01.idm.domain.edu:389: 4 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm02.idm.domain.edu:389: 9 lidm01.idm.domain.edu:389: 7 lidm03.idm.domain.edu:389: 6
Lidm03: Replica Update Vectors: lidm03.idm.domain.edu:389: 5 lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 Certificate Server Replica Update Vectors: lidm03.idm.domain.edu:389: 6 lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9
check_ipa_consistency shows the following:
+--------------------+----------+----------+----------+-------+ | FreeIPA servers: | lidm01 | lidm02 | lidm03 | STATE | +--------------------+----------+----------+----------+-------+ | Active Users | 8 | 8 | 8 | OK | | Stage Users | 0 | 0 | 0 | OK | | Preserved Users | 0 | 0 | 0 | OK | | Hosts | 93 | 93 | 93 | OK | | Services | 13 | 13 | 13 | OK | | User Groups | 89 | 89 | 89 | OK | | Host Groups | 15 | 15 | 15 | OK | | Netgroups | 14 | 14 | 14 | OK | | HBAC Rules | 41 | 41 | 41 | OK | | SUDO Rules | 28 | 28 | 28 | OK | | DNS Zones | 0 | 0 | 0 | OK | | Certificates | 27 | 27 | 27 | OK | | LDAP Conflicts | 0 | 0 | 0 | OK | | Ghost Replicas | 0 | 0 | 0 | OK | | Anonymous BIND | ON | ON | ON | OK | | Microsoft ADTrust | True | True | True | OK | | Replication Status | lidm02 0 | lidm03 0 | lidm02 0 | OK | | | lidm03 0 | lidm01 0 | lidm01 0 | | +--------------------+----------+----------+----------+-------+
At this point I’m not sure what to do next as most references to problems like this that I could find were solved with the re-initialize of the replication agreements. I appreciate any help anyone can provide.
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On 11/19/19 10:04 PM, Jones, Bob (rwj5d) via FreeIPA-users wrote:
Hello,
We have a 3 node multi-master IPA setup. These are running on Red Hat Enterprise Linux Server release 7.7 (Maipo) and all are version:
Name : ipa-server Arch : x86_64 Version : 4.6.5 Release : 11.el7_7.3
Starting yesterday, we are getting the following messages approximately every 3 seconds from lidm01:
[19/Nov/2019:15:48:24.221023177 -0500] - ERR - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:24.222186079 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389): Missing data encountered. If the error persists the replica must be reinitialized. [19/Nov/2019:15:48:25.229931823 -0500] - ERR - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:25.231001956 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389): Missing data encountered. If the error persists the replica must be reinitialized.
Hi,
there are 2 different suffixes managed by IPA when it is deployed with an embedded CA: - dc=domain,dc=com (replace with your actual domain name): this one stores the user / host / service entries and most of IPA objects - o=ipaca: this one is used by PKI server to store the certificate requests, the certificates, etc... Each suffix has his own replication agreements with other servers.
The command ipa-replica-manage re-initialize applies only to the dc=domain,dc=com suffix. To re-initialize the o=ipaca suffix, the command ipa-csreplica-manage re-initialize must be used.
Hope this clarifies, flo
There are no error messages on lidm02 or lidm03. I ran the command 'ipa-replica-manage re-initalize --from lidm01.idm.domain.edu’ on both lidm02 and lidm03. When that did not change anything I ran the command 'ipa-replica-manage re-initalize --from lidm02.idm.domain.edu’ on lidm01. This did not change anything either. I have run restart-dirsrv on all 3 idm servers as well.
Also, here are the ipa-replica-manage list-ruv output on the 3 servers:
Lidm01: Replica Update Vectors: lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9 lidm03.idm.domain.edu:389: 6
Lidm02: Replica Update Vectors: lidm02.idm.domain.edu:389: 8 lidm01.idm.domain.edu:389: 4 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm02.idm.domain.edu:389: 9 lidm01.idm.domain.edu:389: 7 lidm03.idm.domain.edu:389: 6
Lidm03: Replica Update Vectors: lidm03.idm.domain.edu:389: 5 lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 Certificate Server Replica Update Vectors: lidm03.idm.domain.edu:389: 6 lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9
check_ipa_consistency shows the following:
+--------------------+----------+----------+----------+-------+ | FreeIPA servers: | lidm01 | lidm02 | lidm03 | STATE | +--------------------+----------+----------+----------+-------+ | Active Users | 8 | 8 | 8 | OK | | Stage Users | 0 | 0 | 0 | OK | | Preserved Users | 0 | 0 | 0 | OK | | Hosts | 93 | 93 | 93 | OK | | Services | 13 | 13 | 13 | OK | | User Groups | 89 | 89 | 89 | OK | | Host Groups | 15 | 15 | 15 | OK | | Netgroups | 14 | 14 | 14 | OK | | HBAC Rules | 41 | 41 | 41 | OK | | SUDO Rules | 28 | 28 | 28 | OK | | DNS Zones | 0 | 0 | 0 | OK | | Certificates | 27 | 27 | 27 | OK | | LDAP Conflicts | 0 | 0 | 0 | OK | | Ghost Replicas | 0 | 0 | 0 | OK | | Anonymous BIND | ON | ON | ON | OK | | Microsoft ADTrust | True | True | True | OK | | Replication Status | lidm02 0 | lidm03 0 | lidm02 0 | OK | | | lidm03 0 | lidm01 0 | lidm01 0 | | +--------------------+----------+----------+----------+-------+
At this point I’m not sure what to do next as most references to problems like this that I could find were solved with the re-initialize of the replication agreements. I appreciate any help anyone can provide.
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thank you for the help Flo. Doing the ipa-csreplica-manage re-initialize corrected the issue I was seeing.
Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Nov 20, 2019, at 6:54 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 11/19/19 10:04 PM, Jones, Bob (rwj5d) via FreeIPA-users wrote:
Hello, We have a 3 node multi-master IPA setup. These are running on Red Hat Enterprise Linux Server release 7.7 (Maipo) and all are version: Name : ipa-server Arch : x86_64 Version : 4.6.5 Release : 11.el7_7.3 Starting yesterday, we are getting the following messages approximately every 3 seconds from lidm01: [19/Nov/2019:15:48:24.221023177 -0500] - ERR - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:24.222186079 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389): Missing data encountered. If the error persists the replica must be reinitialized. [19/Nov/2019:15:48:25.229931823 -0500] - ERR - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:25.231001956 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389): Missing data encountered. If the error persists the replica must be reinitialized.
Hi,
there are 2 different suffixes managed by IPA when it is deployed with an embedded CA:
- dc=domain,dc=com (replace with your actual domain name): this one stores the user / host / service entries and most of IPA objects
- o=ipaca: this one is used by PKI server to store the certificate requests, the certificates, etc...
Each suffix has his own replication agreements with other servers.
The command ipa-replica-manage re-initialize applies only to the dc=domain,dc=com suffix. To re-initialize the o=ipaca suffix, the command ipa-csreplica-manage re-initialize must be used.
Hope this clarifies, flo
There are no error messages on lidm02 or lidm03. I ran the command 'ipa-replica-manage re-initalize --from lidm01.idm.domain.edu’ on both lidm02 and lidm03. When that did not change anything I ran the command 'ipa-replica-manage re-initalize --from lidm02.idm.domain.edu’ on lidm01. This did not change anything either. I have run restart-dirsrv on all 3 idm servers as well. Also, here are the ipa-replica-manage list-ruv output on the 3 servers: Lidm01: Replica Update Vectors: lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9 lidm03.idm.domain.edu:389: 6 Lidm02: Replica Update Vectors: lidm02.idm.domain.edu:389: 8 lidm01.idm.domain.edu:389: 4 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm02.idm.domain.edu:389: 9 lidm01.idm.domain.edu:389: 7 lidm03.idm.domain.edu:389: 6 Lidm03: Replica Update Vectors: lidm03.idm.domain.edu:389: 5 lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 Certificate Server Replica Update Vectors: lidm03.idm.domain.edu:389: 6 lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9 check_ipa_consistency shows the following: +--------------------+----------+----------+----------+-------+ | FreeIPA servers: | lidm01 | lidm02 | lidm03 | STATE | +--------------------+----------+----------+----------+-------+ | Active Users | 8 | 8 | 8 | OK | | Stage Users | 0 | 0 | 0 | OK | | Preserved Users | 0 | 0 | 0 | OK | | Hosts | 93 | 93 | 93 | OK | | Services | 13 | 13 | 13 | OK | | User Groups | 89 | 89 | 89 | OK | | Host Groups | 15 | 15 | 15 | OK | | Netgroups | 14 | 14 | 14 | OK | | HBAC Rules | 41 | 41 | 41 | OK | | SUDO Rules | 28 | 28 | 28 | OK | | DNS Zones | 0 | 0 | 0 | OK | | Certificates | 27 | 27 | 27 | OK | | LDAP Conflicts | 0 | 0 | 0 | OK | | Ghost Replicas | 0 | 0 | 0 | OK | | Anonymous BIND | ON | ON | ON | OK | | Microsoft ADTrust | True | True | True | OK | | Replication Status | lidm02 0 | lidm03 0 | lidm02 0 | OK | | | lidm03 0 | lidm01 0 | lidm01 0 | | +--------------------+----------+----------+----------+-------+ At this point I’m not sure what to do next as most references to problems like this that I could find were solved with the re-initialize of the replication agreements. I appreciate any help anyone can provide. Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 11/20/19 10:16 PM, Jones, Bob (rwj5d) via FreeIPA-users wrote:
Thank you for the help Flo. Doing the ipa-csreplica-manage re-initialize corrected the issue I was seeing.
Glad I was able to help, and thanks for the update. It's good to get confirmation that the issue was solved with the suggestion. flo
Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Nov 20, 2019, at 6:54 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 11/19/19 10:04 PM, Jones, Bob (rwj5d) via FreeIPA-users wrote:
Hello, We have a 3 node multi-master IPA setup. These are running on Red Hat Enterprise Linux Server release 7.7 (Maipo) and all are version: Name : ipa-server Arch : x86_64 Version : 4.6.5 Release : 11.el7_7.3 Starting yesterday, we are getting the following messages approximately every 3 seconds from lidm01: [19/Nov/2019:15:48:24.221023177 -0500] - ERR - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:24.222186079 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm02.idm.domain.edu" (lidm02:389): Missing data encountered. If the error persists the replica must be reinitialized. [19/Nov/2019:15:48:25.229931823 -0500] - ERR - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389) - clcache_load_buffer - Can't locate CSN 5dd2adc0000100070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [19/Nov/2019:15:48:25.231001956 -0500] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caTolidm03.idm.domain.edu" (lidm03:389): Missing data encountered. If the error persists the replica must be reinitialized.
Hi,
there are 2 different suffixes managed by IPA when it is deployed with an embedded CA:
- dc=domain,dc=com (replace with your actual domain name): this one stores the user / host / service entries and most of IPA objects
- o=ipaca: this one is used by PKI server to store the certificate requests, the certificates, etc...
Each suffix has his own replication agreements with other servers.
The command ipa-replica-manage re-initialize applies only to the dc=domain,dc=com suffix. To re-initialize the o=ipaca suffix, the command ipa-csreplica-manage re-initialize must be used.
Hope this clarifies, flo
There are no error messages on lidm02 or lidm03. I ran the command 'ipa-replica-manage re-initalize --from lidm01.idm.domain.edu’ on both lidm02 and lidm03. When that did not change anything I ran the command 'ipa-replica-manage re-initalize --from lidm02.idm.domain.edu’ on lidm01. This did not change anything either. I have run restart-dirsrv on all 3 idm servers as well. Also, here are the ipa-replica-manage list-ruv output on the 3 servers: Lidm01: Replica Update Vectors: lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9 lidm03.idm.domain.edu:389: 6 Lidm02: Replica Update Vectors: lidm02.idm.domain.edu:389: 8 lidm01.idm.domain.edu:389: 4 lidm03.idm.domain.edu:389: 5 Certificate Server Replica Update Vectors: lidm02.idm.domain.edu:389: 9 lidm01.idm.domain.edu:389: 7 lidm03.idm.domain.edu:389: 6 Lidm03: Replica Update Vectors: lidm03.idm.domain.edu:389: 5 lidm01.idm.domain.edu:389: 4 lidm02.idm.domain.edu:389: 8 Certificate Server Replica Update Vectors: lidm03.idm.domain.edu:389: 6 lidm01.idm.domain.edu:389: 7 lidm02.idm.domain.edu:389: 9 check_ipa_consistency shows the following: +--------------------+----------+----------+----------+-------+ | FreeIPA servers: | lidm01 | lidm02 | lidm03 | STATE | +--------------------+----------+----------+----------+-------+ | Active Users | 8 | 8 | 8 | OK | | Stage Users | 0 | 0 | 0 | OK | | Preserved Users | 0 | 0 | 0 | OK | | Hosts | 93 | 93 | 93 | OK | | Services | 13 | 13 | 13 | OK | | User Groups | 89 | 89 | 89 | OK | | Host Groups | 15 | 15 | 15 | OK | | Netgroups | 14 | 14 | 14 | OK | | HBAC Rules | 41 | 41 | 41 | OK | | SUDO Rules | 28 | 28 | 28 | OK | | DNS Zones | 0 | 0 | 0 | OK | | Certificates | 27 | 27 | 27 | OK | | LDAP Conflicts | 0 | 0 | 0 | OK | | Ghost Replicas | 0 | 0 | 0 | OK | | Anonymous BIND | ON | ON | ON | OK | | Microsoft ADTrust | True | True | True | OK | | Replication Status | lidm02 0 | lidm03 0 | lidm02 0 | OK | | | lidm03 0 | lidm01 0 | lidm01 0 | | +--------------------+----------+----------+----------+-------+ At this point I’m not sure what to do next as most references to problems like this that I could find were solved with the re-initialize of the replication agreements. I appreciate any help anyone can provide. Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Jones, Bob (rwj5d)