Hi there, this is ipa-server-4.4.0-12.0.1 with 389-ds-base-1.3.5.10-11 and suddenly daily backup has started to fail with messages:
2019-01-28T04:10:04Z INFO Backing up ipaca in REALM-COM to LDIF 2019-01-28T04:10:04Z INFO Waiting for LDIF to finish 2019-01-28T04:10:05Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_backup.py", line 300, in run self.db2ldif(instance, 'ipaca', online=options.online) File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_backup.py", line 425, in db2ldif shutil.move(ldiffile, os.path.join(self.dir, ldifname)) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: 2019-01-28T04:10:05Z DEBUG The ipa-backup command failed, exception: IOError: [Errno 2] No such file or directory: u'/var/ lib/dirsrv/slapd-REALM-COM/ldif/REALM-COM-ipaca.ldif' 2019-01-28T04:10:05Z ERROR [Errno 2] No such file or directory: u'/var/lib/dirsrv/slapd-REALM-COM/ldif/REALM-COM-ipaca.ldif' 2019-01-28T04:10:05Z ERROR The ipa-backup command failed. See /var/log/ipabackup.log for more information
And service start fails with messages:
[02/Feb/2019:22:47:37.889779410 -0800] 389-Directory/1.3.5.10 B2016.309.1527 starting up [02/Feb/2019:22:47:37.906422534 -0800] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [02/Feb/2019:22:47:37.921288555 -0800] WARNING: userRoot: entry cache size 10485760 B is less than db size 16932864 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.921943984 -0800] WARNING: ipaca: entry cache size 10485760 B is less than db size 1757741056 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.922701343 -0800] WARNING: changelog: entry cache size 2097152 B is less than db size 82935808 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.925215059 -0800] Detected Disorderly Shutdown last time Directory Server was running, recovering database. [02/Feb/2019:22:47:37.926177620 -0800] libdb: BDB1546 unable to join the environment
thanks in advance for any help, Zarko
On 2/3/19 8:08 AM, Zarko D via FreeIPA-users wrote:
Hi there, this is ipa-server-4.4.0-12.0.1 with 389-ds-base-1.3.5.10-11 and suddenly daily backup has started to fail with messages:
2019-01-28T04:10:04Z INFO Backing up ipaca in REALM-COM to LDIF 2019-01-28T04:10:04Z INFO Waiting for LDIF to finish 2019-01-28T04:10:05Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_backup.py", line 300, in run self.db2ldif(instance, 'ipaca', online=options.online) File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_backup.py", line 425, in db2ldif shutil.move(ldiffile, os.path.join(self.dir, ldifname)) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: 2019-01-28T04:10:05Z DEBUG The ipa-backup command failed, exception: IOError: [Errno 2] No such file or directory: u'/var/ lib/dirsrv/slapd-REALM-COM/ldif/REALM-COM-ipaca.ldif' 2019-01-28T04:10:05Z ERROR [Errno 2] No such file or directory: u'/var/lib/dirsrv/slapd-REALM-COM/ldif/REALM-COM-ipaca.ldif' 2019-01-28T04:10:05Z ERROR The ipa-backup command failed. See /var/log/ipabackup.log for more information
And service start fails with messages:
[02/Feb/2019:22:47:37.889779410 -0800] 389-Directory/1.3.5.10 B2016.309.1527 starting up [02/Feb/2019:22:47:37.906422534 -0800] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [02/Feb/2019:22:47:37.921288555 -0800] WARNING: userRoot: entry cache size 10485760 B is less than db size 16932864 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.921943984 -0800] WARNING: ipaca: entry cache size 10485760 B is less than db size 1757741056 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.922701343 -0800] WARNING: changelog: entry cache size 2097152 B is less than db size 82935808 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.925215059 -0800] Detected Disorderly Shutdown last time Directory Server was running, recovering database. [02/Feb/2019:22:47:37.926177620 -0800] libdb: BDB1546 unable to join the environment
thanks in advance for any help, Zarko
Hi, You may get more help from 389-users mailing list, which I CC'ed. flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi,
The IPA message are from Jan 28th (failing ipa backup ) while the restart failure is from Feb 2nd. Nothing in the ds error logs from Jan28th ?
The first message "Detected Disorderly Shutdown" means that DS stopped abruptly (crash, assert,..). So at restart it runs a recovery of the database. Usually it works fine but here the recovery failed "libdb: BDB1546 unable to join the environment".
You may check if there is a captured DS core file. Also would you provide ls -lR /var/lib/dirsrv/slapd-<instance>/db
best regards thierry
On 02/06/2019 10:43 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 2/3/19 8:08 AM, Zarko D via FreeIPA-users wrote:
Hi there, this is ipa-server-4.4.0-12.0.1 with 389-ds-base-1.3.5.10-11 and suddenly daily backup has started to fail with messages:
2019-01-28T04:10:04Z INFO Backing up ipaca in REALM-COM to LDIF 2019-01-28T04:10:04Z INFO Waiting for LDIF to finish 2019-01-28T04:10:05Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_backup.py", line 300, in run self.db2ldif(instance, 'ipaca', online=options.online) File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_backup.py", line 425, in db2ldif shutil.move(ldiffile, os.path.join(self.dir, ldifname)) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: 2019-01-28T04:10:05Z DEBUG The ipa-backup command failed, exception: IOError: [Errno 2] No such file or directory: u'/var/ lib/dirsrv/slapd-REALM-COM/ldif/REALM-COM-ipaca.ldif' 2019-01-28T04:10:05Z ERROR [Errno 2] No such file or directory: u'/var/lib/dirsrv/slapd-REALM-COM/ldif/REALM-COM-ipaca.ldif' 2019-01-28T04:10:05Z ERROR The ipa-backup command failed. See /var/log/ipabackup.log for more information
And service start fails with messages:
[02/Feb/2019:22:47:37.889779410 -0800] 389-Directory/1.3.5.10 B2016.309.1527 starting up [02/Feb/2019:22:47:37.906422534 -0800] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match [02/Feb/2019:22:47:37.921288555 -0800] WARNING: userRoot: entry cache size 10485760 B is less than db size 16932864 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.921943984 -0800] WARNING: ipaca: entry cache size 10485760 B is less than db size 1757741056 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.922701343 -0800] WARNING: changelog: entry cache size 2097152 B is less than db size 82935808 B; We recommend to increase the entry cache size nsslapd-cachememsize. [02/Feb/2019:22:47:37.925215059 -0800] Detected Disorderly Shutdown last time Directory Server was running, recovering database. [02/Feb/2019:22:47:37.926177620 -0800] libdb: BDB1546 unable to join the environment
thanks in advance for any help, Zarko
Hi, You may get more help from 389-users mailing list, which I CC'ed. flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thanks Thierry, IPA backup had failed much before, unfortunate not able to restore those logs. But I did some progress, by trying to restore different daily backups. And I found one, that was restored "successfully", and 389ds has started after that. But new problem is that replica from another master permanently fails ("Error (16) : Incremental update connection error. Backing off, will retry update later.") and my work around is having this cron job: ipa-replica-manage -v re-initialize --from=<another-master> The result of this is: 1. DB can reinit with latest data, this is good 2. Any action (add user/host) fails with errors like "IPA Error 4203: DatabaseError". Fine, at least I have that server are read-only in my infrastructure.
On the top of this in Oct 2018, certmonger had failed to renew certs, and "back-in" time proposed resolution didn't help. This is obstacle to add new server, I tried promoting client into server, but replica prepare fails (domain level 0), so I am not sure what we can do!? Maybe built new IPA (latest version), same domain name and realm, and manually migrate from crippled infrastructure.
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
thierry bordaz -
Zarko D