Hey,
Replication isn't working, at least not automatically. If I do a ipa-replica-manage re-initialize then everything is present on the replica.
I've looked through all the logs, but I couldn't find anything that hints me what could be wrong.
Today I created a new replica. The installation went OK. No error. But also that replica does not receive updates.
The IPA master (three at the moment) are running Centos7.
[root@rotte ~]# rpm -qa 'ipa*' ipa-server-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-dns-4.5.4-10.el7.centos.4.4.noarch ipa-client-common-4.5.4-10.el7.centos.4.4.noarch ipa-server-common-4.5.4-10.el7.centos.4.4.noarch ipa-client-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-trust-ad-4.5.4-10.el7.centos.4.4.x86_64 ipa-common-4.5.4-10.el7.centos.4.4.noarch
[root@rotte ~]# ipa-replica-manage -v list rotte.ghs.nl iparep3.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00 linge.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00
rotte is the main master (doing CA), linge and iparep3 are the replicas.
I know that it may be hard to tell me what is wrong, without further information, but I would like to know what information I need to look for.
Any help is greatly appreciated.
On 2/18/19 9:00 AM, Kees Bakker via FreeIPA-users wrote:
Hey,
Replication isn't working, at least not automatically. If I do a ipa-replica-manage re-initialize then everything is present on the replica.
I've looked through all the logs, but I couldn't find anything that hints me what could be wrong.
Today I created a new replica. The installation went OK. No error. But also that replica does not receive updates.
The IPA master (three at the moment) are running Centos7.
[root@rotte ~]# rpm -qa 'ipa*' ipa-server-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-dns-4.5.4-10.el7.centos.4.4.noarch ipa-client-common-4.5.4-10.el7.centos.4.4.noarch ipa-server-common-4.5.4-10.el7.centos.4.4.noarch ipa-client-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-trust-ad-4.5.4-10.el7.centos.4.4.x86_64 ipa-common-4.5.4-10.el7.centos.4.4.noarch
[root@rotte ~]# ipa-replica-manage -v list rotte.ghs.nl iparep3.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00 linge.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00
rotte is the main master (doing CA), linge and iparep3 are the replicas.
I know that it may be hard to tell me what is wrong, without further information, but I would like to know what information I need to look for.
Any help is greatly appreciated.
Hi,
please find more info in the wiki: https://www.freeipa.org/page/Troubleshooting/Directory_Server
If you add an entry on rotte, does this entry get replicated to the other servers? and is the reverse true? The "last update status" seems to indicate that everything is working well.
flo
On 18-02-19 10:06, Florence Blanc-Renaud wrote:
On 2/18/19 9:00 AM, Kees Bakker via FreeIPA-users wrote:
Hey,
Replication isn't working, at least not automatically. If I do a ipa-replica-manage re-initialize then everything is present on the replica.
I've looked through all the logs, but I couldn't find anything that hints me what could be wrong.
Today I created a new replica. The installation went OK. No error. But also that replica does not receive updates.
The IPA master (three at the moment) are running Centos7.
[root@rotte ~]# rpm -qa 'ipa*' ipa-server-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-dns-4.5.4-10.el7.centos.4.4.noarch ipa-client-common-4.5.4-10.el7.centos.4.4.noarch ipa-server-common-4.5.4-10.el7.centos.4.4.noarch ipa-client-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-trust-ad-4.5.4-10.el7.centos.4.4.x86_64 ipa-common-4.5.4-10.el7.centos.4.4.noarch
[root@rotte ~]# ipa-replica-manage -v list rotte.ghs.nl iparep3.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00 linge.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00
rotte is the main master (doing CA), linge and iparep3 are the replicas.
I know that it may be hard to tell me what is wrong, without further information, but I would like to know what information I need to look for.
Any help is greatly appreciated.
Hi,
please find more info in the wiki: https://www.freeipa.org/page/Troubleshooting/Directory_Server
If you add an entry on rotte, does this entry get replicated to the other servers? and is the reverse true? The "last update status" seems to indicate that everything is working well.
Hi Flo,
Hmm, that's funny. I did not try to create a user on the other two, because I was trying to do everything on my first master (rotte). The funny part is, that now a new user on linge is replicated correctly to the other two. Why haven't I tested this before? And also a new user on iparep3 is correctly replicated to the other two. Then I added a new user on rotte, which is now correctly replicated. All seems to be alright. I'm puzzled.
The logs did not reveal anything suspicious, replication simply did not work. New users were created on rotte, and also new DNS entries were created (our DHCP server updates DNS entries). But nothing was replicated.
Still, there is one added user (test01) on rotte which was not replicated to linge nor to iparep3. I did a re-initialize on linge and made user test01 become present on linge. That user is still not present on iparep3.
BTW. There is a problem on rotte with numSubordinates in cn=users,cn=accounts,$SUFFIX. The number is one too high. We have 81 users. Have a look at the output of cipa [2] (which just looks at numSubordinates I believe).
[root@rotte ~]# cipa +--------------------+-----------+---------+---------+-------+ | FreeIPA servers: | rotte | linge | iparep3 | STATE | +--------------------+-----------+---------+---------+-------+ | Active Users | 82 | 81 | 80 | FAIL |
How this happened? I think this may have happened when a user was added on two systems (rotte and linge) when there was an old IPA master in between, but that server was switched off. As a result there were errors on rotte saying it could not delete a tombstone, something like this
[14/Jan/2019:16:29:01.225643460 +0100] - ERR - NSMMReplicationPlugin - _delete_tombstone - Unable to delete tombstone nsuniqueid=c0a66e04-125a11e9-bb6698e2-54354ddc,cn=bmot,cn=groups,cn=accounts,$SUFFIX, uniqueid c0a66e04-125a11e9-bb6698e2-54354ddc: Operations error.
I followed this webpage [1] to delete that manually. A ldapdelete command failed because of a linked entry. Maybe that caused a failure to update numSubordinates.
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/htm... [2] https://github.com/peterpakos/checkipaconsistency
On 2/18/19 11:41 AM, Kees Bakker via FreeIPA-users wrote:
On 18-02-19 10:06, Florence Blanc-Renaud wrote:
On 2/18/19 9:00 AM, Kees Bakker via FreeIPA-users wrote:
Hey,
Replication isn't working, at least not automatically. If I do a ipa-replica-manage re-initialize then everything is present on the replica.
I've looked through all the logs, but I couldn't find anything that hints me what could be wrong.
Today I created a new replica. The installation went OK. No error. But also that replica does not receive updates.
The IPA master (three at the moment) are running Centos7.
[root@rotte ~]# rpm -qa 'ipa*' ipa-server-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-dns-4.5.4-10.el7.centos.4.4.noarch ipa-client-common-4.5.4-10.el7.centos.4.4.noarch ipa-server-common-4.5.4-10.el7.centos.4.4.noarch ipa-client-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-trust-ad-4.5.4-10.el7.centos.4.4.x86_64 ipa-common-4.5.4-10.el7.centos.4.4.noarch
[root@rotte ~]# ipa-replica-manage -v list rotte.ghs.nl iparep3.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00 linge.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00
rotte is the main master (doing CA), linge and iparep3 are the replicas.
I know that it may be hard to tell me what is wrong, without further information, but I would like to know what information I need to look for.
Any help is greatly appreciated.
Hi,
please find more info in the wiki: https://www.freeipa.org/page/Troubleshooting/Directory_Server
If you add an entry on rotte, does this entry get replicated to the other servers? and is the reverse true? The "last update status" seems to indicate that everything is working well.
Hi Flo,
Hmm, that's funny. I did not try to create a user on the other two, because I was trying to do everything on my first master (rotte). The funny part is, that now a new user on linge is replicated correctly to the other two. Why haven't I tested this before? And also a new user on iparep3 is correctly replicated to the other two. Then I added a new user on rotte, which is now correctly replicated. All seems to be alright. I'm puzzled.
The logs did not reveal anything suspicious, replication simply did not work. New users were created on rotte, and also new DNS entries were created (our DHCP server updates DNS entries). But nothing was replicated.
Still, there is one added user (test01) on rotte which was not replicated to linge nor to iparep3. I did a re-initialize on linge and made user test01 become present on linge. That user is still not present on iparep3.
Did you also run re-initialize from rotte to iparep3? If no, the difference may be caused by replication conflicts in iparep3. The following doc explains how to list them: [1], and how to repair. Note that re-initialize from rotte to iparep3 would also solve the issue.
BTW. There is a problem on rotte with numSubordinates in cn=users,cn=accounts,$SUFFIX. The number is one too high. We have 81 users. Have a look at the output of cipa [2] (which just looks at numSubordinates I believe).
This issue may be linked to issue 7839 [1], please have a look at this comment specifically [2]
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/ht... [2] https://pagure.io/freeipa/issue/7839 [3] https://pagure.io/freeipa/issue/7839#comment-550188
[root@rotte ~]# cipa +--------------------+-----------+---------+---------+-------+ | FreeIPA servers: | rotte | linge | iparep3 | STATE | +--------------------+-----------+---------+---------+-------+ | Active Users | 82 | 81 | 80 | FAIL |
How this happened? I think this may have happened when a user was added on two systems (rotte and linge) when there was an old IPA master in between, but that server was switched off. As a result there were errors on rotte saying it could not delete a tombstone, something like this
[14/Jan/2019:16:29:01.225643460 +0100] - ERR - NSMMReplicationPlugin - _delete_tombstone - Unable to delete tombstone nsuniqueid=c0a66e04-125a11e9-bb6698e2-54354ddc,cn=bmot,cn=groups,cn=accounts,$SUFFIX, uniqueid c0a66e04-125a11e9-bb6698e2-54354ddc: Operations error.
I followed this webpage [1] to delete that manually. A ldapdelete command failed because of a linked entry. Maybe that caused a failure to update numSubordinates.
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/htm... [2] https://github.com/peterpakos/checkipaconsistency _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 18-02-19 21:17, Florence Blanc-Renaud wrote:
On 2/18/19 11:41 AM, Kees Bakker via FreeIPA-users wrote:
On 18-02-19 10:06, Florence Blanc-Renaud wrote:
On 2/18/19 9:00 AM, Kees Bakker via FreeIPA-users wrote:
Hey,
Replication isn't working, at least not automatically. If I do a ipa-replica-manage re-initialize then everything is present on the replica.
I've looked through all the logs, but I couldn't find anything that hints me what could be wrong.
Today I created a new replica. The installation went OK. No error. But also that replica does not receive updates.
The IPA master (three at the moment) are running Centos7.
[root@rotte ~]# rpm -qa 'ipa*' ipa-server-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-dns-4.5.4-10.el7.centos.4.4.noarch ipa-client-common-4.5.4-10.el7.centos.4.4.noarch ipa-server-common-4.5.4-10.el7.centos.4.4.noarch ipa-client-4.5.4-10.el7.centos.4.4.x86_64 ipa-server-trust-ad-4.5.4-10.el7.centos.4.4.x86_64 ipa-common-4.5.4-10.el7.centos.4.4.noarch
[root@rotte ~]# ipa-replica-manage -v list rotte.ghs.nl iparep3.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00 linge.ghs.nl: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2019-02-18 07:50:56+00:00
rotte is the main master (doing CA), linge and iparep3 are the replicas.
I know that it may be hard to tell me what is wrong, without further information, but I would like to know what information I need to look for.
Any help is greatly appreciated.
Hi,
please find more info in the wiki: https://www.freeipa.org/page/Troubleshooting/Directory_Server
If you add an entry on rotte, does this entry get replicated to the other servers? and is the reverse true? The "last update status" seems to indicate that everything is working well.
Hi Flo,
Hmm, that's funny. I did not try to create a user on the other two, because I was trying to do everything on my first master (rotte). The funny part is, that now a new user on linge is replicated correctly to the other two. Why haven't I tested this before? And also a new user on iparep3 is correctly replicated to the other two. Then I added a new user on rotte, which is now correctly replicated. All seems to be alright. I'm puzzled.
The logs did not reveal anything suspicious, replication simply did not work. New users were created on rotte, and also new DNS entries were created (our DHCP server updates DNS entries). But nothing was replicated.
Still, there is one added user (test01) on rotte which was not replicated to linge nor to iparep3. I did a re-initialize on linge and made user test01 become present on linge. That user is still not present on iparep3.
Did you also run re-initialize from rotte to iparep3? If no, the difference may be caused by replication conflicts in iparep3. The following doc explains how to list them: [1], and how to repair. Note that re-initialize from rotte to iparep3 would also solve the issue.
I did not, because I was hoping to be able to analyse the situation. But now I did.
Before re-initializing I listed the nsds5ReplConflict entries. And there were. Huh? I'm guessing these were copied from the first master (rotte), because there were never any modifications/additions on iparep3. After re-initialize there are no replConflicts anymore. And I also see DNS entries coming in being replicated from rotte. That's looking OK now. BTW. This could be a great addition for cipa, to count these nsds5ReplConflict entries.
BTW. There is a problem on rotte with numSubordinates in cn=users,cn=accounts,$SUFFIX. The number is one too high. We have 81 users. Have a look at the output of cipa [2] (which just looks at numSubordinates I believe).
This issue may be linked to issue 7839 [1], please have a look at this comment specifically [2]
Yes, this really looks the same. Thanks for noticing, respect! Now that I looked at issue #7839, I am really scared to read "The problem is that numsubordinates was incorrectly computed and then stored into the DB and will stay like this unless you reinitialize." Not that I know what reinitialize exactly means here, but I fear the worst.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/ht... [2] https://pagure.io/freeipa/issue/7839 [3] https://pagure.io/freeipa/issue/7839#comment-550188
[root@rotte ~]# cipa +--------------------+-----------+---------+---------+-------+ | FreeIPA servers: | rotte | linge | iparep3 | STATE | +--------------------+-----------+---------+---------+-------+ | Active Users | 82 | 81 | 80 | FAIL |
How this happened? I think this may have happened when a user was added on two systems (rotte and linge) when there was an old IPA master in between, but that server was switched off. As a result there were errors on rotte saying it could not delete a tombstone, something like this
[14/Jan/2019:16:29:01.225643460 +0100] - ERR - NSMMReplicationPlugin - _delete_tombstone - Unable to delete tombstone nsuniqueid=c0a66e04-125a11e9-bb6698e2-54354ddc,cn=bmot,cn=groups,cn=accounts,$SUFFIX, uniqueid c0a66e04-125a11e9-bb6698e2-54354ddc: Operations error.
I followed this webpage [1] to delete that manually. A ldapdelete command failed because of a linked entry. Maybe that caused a failure to update numSubordinates.
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/htm... [2] https://github.com/peterpakos/checkipaconsistency _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org