Hello,
we are using freeipa in a 4way multi master replication setup. Servers ipa14,ipa15 and ipa34,ipa35 on CentOS Linux release 7.3.1611 (Core) with version ipa-server-common-4.4.0-14.el7.centos.7.noarch.
We have an issue where one of the servers log a missing CSN. It happens even after ipa replication reinitialized. We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we see it in those files listed on ipa15 and the error is reported there. Please see attached file with logs.
How can we fix this?
-- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
Hello,
we are using freeipa in a 4way multi master replication setup. Servers ipa14,ipa15 and ipa34,ipa35 on CentOS Linux release 7.3.1611 (Core) with version ipa-server-common-4.4.0-14.el7.centos.7.noarch.
We have an issue where one of the servers log a missing CSN. It happens even after ipa replication reinitialized. We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we see it in those files listed on ipa15 and the error is reported there. Please see attached file with logs.
the missing csn is from Nov,13,2017 - so it is not unlikely it was trimmed. But in some RUV there seems to be a reference to it, and replication uses to position it in the changelog.
How can we fix this?
we first should get a full picture of the replicaids and RUVs on all servers, could you do on all servers the following search: ldapsearch .... -o ldif-wrap=no -D "cn=directory manager" .... -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
That should help in deciding what to do.
There is also on option to kick an agreement to ingnore a missing change:
do the following change on the failing replication agreement, but it would be better to have the data first:
ldapmodify .... dn: <agmt> replace: nsds5ReplicaIgnoreMissingChange nsds5ReplicaIgnoreMissingChange: once
-- *Sándor Juhász* System Administrator *ChemAxon**Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
here are the results:
~~~~~~~~
[root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 4 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad07160000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad0711c003a000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa14 ~]#
~~~~~~~~
[root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 15 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad071c20000000f0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad071d20021000b0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa15 ~]#
~~~~~~~~
[root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 11 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072120003000b0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa34 ~]#
~~~~~~~~
[root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 7 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad07248001800070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072490010000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa35 ~]#
~~~~~~~~
-- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
Hello,
we are using freeipa in a 4way multi master replication setup. Servers ipa14,ipa15 and ipa34,ipa35 on CentOS Linux release 7.3.1611 (Core) with version ipa-server-common-4.4.0-14.el7.centos.7.noarch.
We have an issue where one of the servers log a missing CSN. It happens even after ipa replication reinitialized. We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we see it in those files listed on ipa15 and the error is reported there. Please see attached file with logs.
the missing csn is from Nov,13,2017 - so it is not unlikely it was trimmed. But in some RUV there seems to be a reference to it, and replication uses to position it in the changelog.
How can we fix this?
we first should get a full picture of the replicaids and RUVs on all servers, could you do on all servers the following search: ldapsearch .... -o ldif-wrap=no -D "cn=directory manager" .... -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
That should help in deciding what to do.
There is also on option to kick an agreement to ingnore a missing change:
do the following change on the failing replication agreement, but it would be better to have the data first:
ldapmodify .... dn: <agmt> replace: nsds5ReplicaIgnoreMissingChange nsds5ReplicaIgnoreMissingChange: once
-- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Hi, when extracting the relevant data, we see:
[root@ipa14 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 ~~~~~~~~ [root@ipa15 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 ~~~~~~~~ [root@ipa34 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 ~~~~~~~~ [root@ipa35 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
this indicates that all replicas are in sync for replicaid 8 and 12, but for rid 16, ipa 34 and ipa 35 have no data and for rid 6 they have older data. I cannot say what has happened, but I think you need reinit 34 and 35 from either 14 or 15
On 04/13/2018 11:13 AM, Sandor Juhasz wrote:
here are the results:
[root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 4 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad07160000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad0711c003a000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa14 ~]#[root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 15 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad071c20000000f0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad071d20021000b0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa15 ~]#
[root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 11 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072120003000b0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa34 ~]#[root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 7 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad07248001800070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072490010000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa35 ~]#
-- *Sándor Juhász* System Administrator *ChemAxon**Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964 On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> wrote: On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote: > Hello, > > we are using freeipa in a 4way multi master replication setup. > Servers ipa14,ipa15 and ipa34,ipa35 on > CentOS Linux release 7.3.1611 (Core) with version > ipa-server-common-4.4.0-14.el7.centos.7.noarch. > > We have an issue where one of the servers log a missing CSN. It > happens even after > ipa replication reinitialized. > We are guessing that CSN 5a0a27d9000000060000 only exists on > ipa35, but we see it in those files listed on ipa15 and the error > is reported there. > Please see attached file with logs. the missing csn is from Nov,13,2017 - so it is not unlikely it was trimmed. But in some RUV there seems to be a reference to it, and replication uses to position it in the changelog. > > How can we fix this? we first should get a full picture of the replicaids and RUVs on all servers, could you do on all servers the following search: ldapsearch .... -o ldif-wrap=no -D "cn=directory manager" .... -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv That should help in deciding what to do. There is also on option to kick an agreement to ingnore a missing change: do the following change on the failing replication agreement, but it would be better to have the data first: ldapmodify .... dn: <agmt> replace: nsds5ReplicaIgnoreMissingChange nsds5ReplicaIgnoreMissingChange: once > > -- > *Sándor Juhász* > System Administrator > *ChemAxon**Ltd*. > Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, > H-1031 > Cell: +36704258964 > > > _______________________________________________ > FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> -- Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>
re-initialize was not sufficient. We needed to trash ipa34 and ipa35 completely and install new replica. We had some hassle with hosts file and permission denied on the ipa-replica-install step, in the end it worked out. -- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
On Fri, Apr 13, 2018 at 11:56 AM Ludwig Krispenz lkrispen@redhat.com wrote:
Hi, when extracting the relevant data, we see:
[root@ipa14 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
[root@ipa15 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000[root@ipa34 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
[root@ipa35 ~] dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 this indicates that all replicas are in sync for replicaid 8 and 12, but for rid 16, ipa 34 and ipa 35 have no data and for rid 6 they have older data. I cannot say what has happened, but I think you need reinit 34 and 35 from either 14 or 15 On 04/13/2018 11:13 AM, Sandor Juhasz wrote: here are the results:[root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 4 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad07160000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad0711c003a000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 6 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa14 ~]#
[root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 15 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad071c20000000f0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad071d20021000b0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 16 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 5a0da16d000200100000 nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5ad07153000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa15 ~]#[root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv #
# replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 11 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072120003000b0000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad06adb000900070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000
# replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 12 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2 [root@ipa34 ~]#
[root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=config> with scope subtree # filter: objectclass=nsds5replica # requesting: nsds5replicaid nsds50ruv # # replica, dc\3Dcxn, mapping tree, config dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config nsds5replicaid: 7 nsds50ruv: {replicageneration} 58987d9e000000040000 nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 5ad07248001800070000 nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 5ad071af002d00040000 nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 5ad072490010000b0000 nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 5ad06e1a0004000f0000 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds5replicaid: 8 nsds50ruv: {replicageneration} 58987e19000000060000 nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 589adeca000000080000 nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 5a0a27d9000000060000 nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 59d74c4e0004000c0000 # search result search: 2 result: 0 Success # numResponses: 3 # numEntries: 2 [root@ipa35 ~]#-- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
Hello,
we are using freeipa in a 4way multi master replication setup. Servers ipa14,ipa15 and ipa34,ipa35 on CentOS Linux release 7.3.1611 (Core) with version ipa-server-common-4.4.0-14.el7.centos.7.noarch.
We have an issue where one of the servers log a missing CSN. It happens even after ipa replication reinitialized. We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we see it in those files listed on ipa15 and the error is reported there. Please see attached file with logs.
the missing csn is from Nov,13,2017 - so it is not unlikely it was trimmed. But in some RUV there seems to be a reference to it, and replication uses to position it in the changelog.
How can we fix this?
we first should get a full picture of the replicaids and RUVs on all servers, could you do on all servers the following search: ldapsearch .... -o ldif-wrap=no -D "cn=directory manager" .... -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
That should help in deciding what to do.
There is also on option to kick an agreement to ingnore a missing change:
do the following change on the failing replication agreement, but it would be better to have the data first:
ldapmodify .... dn: <agmt> replace: nsds5ReplicaIgnoreMissingChange nsds5ReplicaIgnoreMissingChange: once
-- *Sándor Juhász* System Administrator *ChemAxon* *Ltd*. Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031 Cell: +36704258964
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
-- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
freeipa-users@lists.fedorahosted.org
-
Ludwig Krispenz -
Sandor Juhasz