Hello,
I want to move my IPA master to new hardware, but IPA does not want to start on that new hardware.
/var/log/krb5kdc.log shows: krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
And then of course the rest of FreeIPA is not working either.
I've basically copied the whole disk using rsync, and tweaked some things like ifcfg and fstab.
The rsync command needs --numeric-ids, but other than that nothing else is needed, I think. rsync -ai -x --delete --numeric-ids oldmaster:/oldroot/ /croot/
Also force a relabeling for SELINUX touch /croot/.autorelabel
It boots alright, but IPA isn't started properly.
Can someone shed some light on this? Does krb5kdc depend on its hardware? Is there documentation how to move an IPA master to other hardware?
You’re going to need to provide some basic errors in the logs. Otherwise people are just going to be left to guess at a eleventy different things that could go wrong and you’ll spend tons of time trying to chase them all down. It’s a bad use of everyone’s time, including yours.
On Dec 17, 2018, at 7:40 PM, Kees Bakker via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
Hello,
I want to move my IPA master to new hardware, but IPA does not want to start on that new hardware.
/var/log/krb5kdc.log shows: krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
And then of course the rest of FreeIPA is not working either.
I've basically copied the whole disk using rsync, and tweaked some things like ifcfg and fstab.
The rsync command needs --numeric-ids, but other than that nothing else is needed, I think. rsync -ai -x --delete --numeric-ids oldmaster:/oldroot/ /croot/
Also force a relabeling for SELINUX touch /croot/.autorelabel
It boots alright, but IPA isn't started properly.
Can someone shed some light on this? Does krb5kdc depend on its hardware? Is there documentation how to move an IPA master to other hardware? -- Kees _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Sure I understand that, but this error in /var/log/krb5kdc.log is basically all I have. krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
The system is Centos 7. OK, here are some lines from /var/log/messages, if that helps.
Dec 17 13:43:01 alblas named-pkcs11[9684]: LDAP error: Invalid credentials: bind to LDAP server failed Dec 17 13:43:01 alblas named-pkcs11[9684]: couldn't establish connection in LDAP connection pool: permission denied Dec 17 13:43:01 alblas named-pkcs11[9684]: dynamic database 'ipa' configuration failed: permission denied Dec 17 13:43:01 alblas named-pkcs11[9684]: loading configuration: permission denied Dec 17 13:43:01 alblas named-pkcs11[9684]: exiting (due to fatal error) Dec 17 13:43:01 alblas systemd: named-pkcs11.service: control process exited, code=exited status=1 Dec 17 13:43:01 alblas systemd: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11. Dec 17 13:43:01 alblas systemd: Unit named-pkcs11.service entered failed state. Dec 17 13:43:01 alblas systemd: named-pkcs11.service failed. Dec 17 13:43:01 alblas systemd: Reached target Host and Network Name Lookups. Dec 17 13:43:01 alblas systemd: Starting Host and Network Name Lookups. Dec 17 13:43:01 alblas ipactl: Failed to start named Service Dec 17 13:43:01 alblas ipactl: Shutting down Dec 17 13:43:01 alblas systemd: Stopping Kerberos 5 KDC... Dec 17 13:43:01 alblas systemd: Stopped Kerberos 5 KDC. Dec 17 13:43:01 alblas systemd: Stopping Kerberos 5 Password-changing and Administration... Dec 17 13:43:01 alblas systemd: kadmin.service: main process exited, code=exited, status=2/INVALIDARGUMENT Dec 17 13:43:01 alblas systemd: Stopped Kerberos 5 Password-changing and Administration. Dec 17 13:43:01 alblas systemd: Unit kadmin.service entered failed state. Dec 17 13:43:01 alblas systemd: kadmin.service failed. Dec 17 13:43:01 alblas systemd: Stopping 389 Directory Server GHS-NL.... Dec 17 13:43:01 alblas ns-slapd: [17/Dec/2018:13:43:01.678884473 +0100] - INFO - op_thread_cleanup - slapd shutting down - signaling operation threads - op stack size 6 max work q size 4 max work q stack size 4 Dec 17 13:43:01 alblas ns-slapd: [17/Dec/2018:13:43:01.737634634 +0100] - INFO - slapd_daemon - slapd shutting down - waiting for 18 threads to terminate Dec 17 13:43:01 alblas ns-slapd: [17/Dec/2018:13:43:01.775014892 +0100] - INFO - slapd_daemon - slapd shutting down - closing down internal subsystems and plugins Dec 17 13:43:05 alblas ns-slapd: [17/Dec/2018:13:43:05.190616894 +0100] - INFO - dblayer_pre_close - Waiting for 4 database threads to stop Dec 17 13:43:06 alblas ns-slapd: [17/Dec/2018:13:43:06.295603458 +0100] - INFO - dblayer_pre_close - All database threads now stopped Dec 17 13:43:06 alblas ns-slapd: [17/Dec/2018:13:43:06.388499718 +0100] - INFO - ldbm_back_instance_set_destructor - Set of instances destroyed Dec 17 13:43:06 alblas ns-slapd: [17/Dec/2018:13:43:06.415985937 +0100] - INFO - connection_post_shutdown_cleanup - slapd shutting down - freed 4 work q stack objects - freed 6 op stack objects Dec 17 13:43:06 alblas ns-slapd: [17/Dec/2018:13:43:06.449122641 +0100] - INFO - main - slapd stopped. Dec 17 13:43:07 alblas systemd: Stopped 389 Directory Server GHS-NL.. Dec 17 13:43:07 alblas ipactl: Hint: You can use --ignore-service-failure option for forced start in case that a non-critical service failed Dec 17 13:43:07 alblas ipactl: Aborting ipactl
Is there a sequence of systemctl commands I can try to eliminate which service is actually the problem?
On 17-12-18 13:42, Brian Topping wrote:
You’re going to need to provide some basic errors in the logs. Otherwise people are just going to be left to guess at a eleventy different things that could go wrong and you’ll spend tons of time trying to chase them all down. It’s a bad use of everyone’s time, including yours.
On Dec 17, 2018, at 7:40 PM, Kees Bakker via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
Hello,
I want to move my IPA master to new hardware, but IPA does not want to start on that new hardware.
/var/log/krb5kdc.log shows: krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
And then of course the rest of FreeIPA is not working either.
I've basically copied the whole disk using rsync, and tweaked some things like ifcfg and fstab.
The rsync command needs --numeric-ids, but other than that nothing else is needed, I think. rsync -ai -x --delete --numeric-ids oldmaster:/oldroot/ /croot/
Also force a relabeling for SELINUX touch /croot/.autorelabel
It boots alright, but IPA isn't started properly.
Can someone shed some light on this? Does krb5kdc depend on its hardware? Is there documentation how to move an IPA master to other hardware? -- Kees _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Kees Bakker via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
Sure I understand that, but this error in /var/log/krb5kdc.log is basically all I have. krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
What are the permissions on your stash file? Does a checksum match the old replica?
Thanks, --Robbie
On 17-12-18 20:44, Robbie Harwood wrote:
Kees Bakker via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
Sure I understand that, but this error in /var/log/krb5kdc.log is basically all I have. krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
What are the permissions on your stash file? Does a checksum match the old replica?
Where do I find the stash file?
I've copied everything with rsync from the old machine. That should have made an exact copy. Well, except for the selinux attributes, which hopefully recovered with the .autorelabel. But I'm not 100% sure about that.
From other discussions in the past about this krb5kdc error I get the impression that the stash file may be stored in LDAP (i.e. dirsrv). If that is true, then I need to concentrate on why dirsrv isn't started properly.
Kees Bakker keesb@ghs.com writes:
On 17-12-18 20:44, Robbie Harwood wrote:
Kees Bakker via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
Sure I understand that, but this error in /var/log/krb5kdc.log is basically all I have. krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
What are the permissions on your stash file? Does a checksum match the old replica?
Where do I find the stash file?
I've copied everything with rsync from the old machine. That should have made an exact copy. Well, except for the selinux attributes, which hopefully recovered with the .autorelabel. But I'm not 100% sure about that.
From other discussions in the past about this krb5kdc error I get the impression that the stash file may be stored in LDAP (i.e. dirsrv). If that is true, then I need to concentrate on why dirsrv isn't started properly.
Oh right, I'm on the freeipa list. Sorry about that.
If dirsrv isn't starting, you need to look at that first. LDAP needs to be working in order for freeipa to bring up the KDC.
If dirsrv isn't starting, it's likely that krb5 is having some problem connecting. I'd look at dirsrv's logs to see if anything seems amiss.
Thanks, --Robbie
On 18-12-18 19:18, Robbie Harwood wrote:
Kees Bakker keesb@ghs.com writes:
On 17-12-18 20:44, Robbie Harwood wrote:
Kees Bakker via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
Sure I understand that, but this error in /var/log/krb5kdc.log is basically all I have. krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
What are the permissions on your stash file? Does a checksum match the old replica?
Where do I find the stash file?
I've copied everything with rsync from the old machine. That should have made an exact copy. Well, except for the selinux attributes, which hopefully recovered with the .autorelabel. But I'm not 100% sure about that.
From other discussions in the past about this krb5kdc error I get the impression that the stash file may be stored in LDAP (i.e. dirsrv). If that is true, then I need to concentrate on why dirsrv isn't started properly.
Oh right, I'm on the freeipa list. Sorry about that.
If dirsrv isn't starting, you need to look at that first. LDAP needs to be working in order for freeipa to bring up the KDC.
If dirsrv isn't starting, it's likely that krb5 is having some problem connecting. I'd look at dirsrv's logs to see if anything seems amiss.
dirsrv is starting, but eventually it dies. KDC starts, I think, but there something funny here. The log on the new hardware.
Dec 17 13:45:11 alblas systemd: Starting Kerberos 5 KDC... Dec 17 13:45:11 alblas systemd: Started Kerberos 5 KDC.
But when I diff with the old machine there is always a message PID not found. This is from the old machine.
Dec 17 13:05:01 alblas systemd: Starting Kerberos 5 KDC... Dec 17 13:05:02 alblas systemd: PID file /var/run/krb5kdc.pid not readable (yet?) after start. Dec 17 13:05:02 alblas systemd: Started Kerberos 5 KDC.
I don't understand how /var/run/krb5kdc.pid can be there at startup. Now maybe systemd thinks that KDC has started (wild guess). After that named starts but quickly fails due to dirsrv problems.
Dec 17 13:43:01 alblas named-pkcs11[9684]: loading DynDB instance 'ipa' driver '/usr/lib64/bind/ldap.so' Dec 17 13:43:01 alblas named-pkcs11[9684]: bind-dyndb-ldap version 11.1 compiled at 13:38:22 Aug 23 2017, compiler 4.8.5 20150623 (Red Hat 4.8.5-16) Dec 17 13:43:01 alblas named-pkcs11[9684]: LDAP error: Invalid credentials: bind to LDAP server failed Dec 17 13:43:01 alblas named-pkcs11[9684]: couldn't establish connection in LDAP connection pool: permission denied
Anyway, I want to give it a new attempt. The reason is that I discovered a clock issue. The new hardware had the clock 1 hour in the future. Perhaps KDC and/or dirsrv refuses to start when its files have a timestamp in the future.
If this all fails I intend to follow Flo's advice to setup a replica and move CA renewal and CRL master to the new replica.
On 12/17/18 1:40 PM, Kees Bakker via FreeIPA-users wrote:
Hello,
I want to move my IPA master to new hardware, but IPA does not want to start on that new hardware.
/var/log/krb5kdc.log shows: krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
And then of course the rest of FreeIPA is not working either.
I've basically copied the whole disk using rsync, and tweaked some things like ifcfg and fstab.
The rsync command needs --numeric-ids, but other than that nothing else is needed, I think. rsync -ai -x --delete --numeric-ids oldmaster:/oldroot/ /croot/
Also force a relabeling for SELINUX touch /croot/.autorelabel
It boots alright, but IPA isn't started properly.
Can someone shed some light on this? Does krb5kdc depend on its hardware? Is there documentation how to move an IPA master to other hardware?
Hi,
you can have a look at the ipa-backup / ipa-restore commands [1]. The limitations are that you need to restore on a server with the same IPA version and with the same hostname.
If you have a spare machine you can also use replication, and create a replica of your current master with all the needed services (CA, KRA, DNS if needed). If you really need to keep the same hostname, then you will need a spare machine: 1. create serverB as a replica of serverA on your spare machine. Do not forget to promote serverB as CA renewal master and CRL master [2]. 2. decommission serverA with (on serverA) ipa-server-install --uninstall and (on serverB) ipa-replica-manage del serverA --clean 3. provision your new hardware with hostname=serverA, install serverA as a replica of serverB. I would advise to keep serverB as it will provide redundancy.
This wiki [3] also explains the preferred paths depending on your situation. HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... [2] https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master [3] https://www.freeipa.org/page/Backup_and_Restore
On 18-12-18 17:50, Florence Blanc-Renaud wrote:
On 12/17/18 1:40 PM, Kees Bakker via FreeIPA-users wrote:
Hello,
I want to move my IPA master to new hardware, but IPA does not want to start on that new hardware.
/var/log/krb5kdc.log shows: krb5kdc: Server error - while fetching master key K/M for realm GHS.NL
And then of course the rest of FreeIPA is not working either.
I've basically copied the whole disk using rsync, and tweaked some things like ifcfg and fstab.
The rsync command needs --numeric-ids, but other than that nothing else is needed, I think. rsync -ai -x --delete --numeric-ids oldmaster:/oldroot/ /croot/
Also force a relabeling for SELINUX touch /croot/.autorelabel
It boots alright, but IPA isn't started properly.
Can someone shed some light on this? Does krb5kdc depend on its hardware? Is there documentation how to move an IPA master to other hardware?
Hi,
you can have a look at the ipa-backup / ipa-restore commands [1]. The limitations are that you need to restore on a server with the same IPA version and with the same hostname.
Yes, I looked at that document. However, I was hoping to just do a "simple" file system copy. Well, it turned out to not be so simple.
If you have a spare machine you can also use replication, and create a replica of your current master with all the needed services (CA, KRA, DNS if needed). If you really need to keep the same hostname, then you will need a spare machine:
- create serverB as a replica of serverA on your spare machine. Do not forget to promote serverB as CA renewal master and CRL master [2].
- decommission serverA with (on serverA) ipa-server-install --uninstall and (on serverB) ipa-replica-manage del serverA --clean
- provision your new hardware with hostname=serverA, install serverA as a replica of serverB.
I would advise to keep serverB as it will provide redundancy.
This wiki [3] also explains the preferred paths depending on your situation.
I have read that document too. First I want to give it another try. If it fails again I will follow advice described above.
Thanks for your help.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... [2] https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master [3] https://www.freeipa.org/page/Backup_and_Restore
On 19-12-18 12:06, Kees Bakker via FreeIPA-users wrote:
On 18-12-18 17:50, Florence Blanc-Renaud wrote: [...]
If you have a spare machine you can also use replication, and create a replica of your current master with all the needed services (CA, KRA, DNS if needed). If you really need to keep the same hostname, then you will need a spare machine:
- create serverB as a replica of serverA on your spare machine. Do not forget to promote serverB as CA renewal master and CRL master [2].
- decommission serverA with (on serverA) ipa-server-install --uninstall and (on serverB) ipa-replica-manage del serverA --clean
- provision your new hardware with hostname=serverA, install serverA as a replica of serverB.
I would advise to keep serverB as it will provide redundancy.
This wiki [3] also explains the preferred paths depending on your situation.
I have read that document too. First I want to give it another try. If it fails again I will follow advice described above.
Thanks for your help.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... [2] https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master [3] https://www.freeipa.org/page/Backup_and_Restore
Just to let you know, I have given up with my "rsync" procedure. I am now following the steps above. Well, except step3, because I didn't want to add even more hardware in the process (the "spare machine" mentioned above).
Step 1 is completed. Promotion of CA renewal and CRL master is done.
I have a remaining question. What do I do with all the IPA clients that point to serverA? At some point I want to execute step 2, and shut off that system. I briefly looked at the files in /etc and found these (alblas is my serverA):
/etc/sssd/sssd.conf:ipa_server = _srv_, alblas.ghs.nl /etc/ipa/default.conf:server = alblas.ghs.nl /etc/ipa/default.conf:xmlrpc_uri = https://alblas.ghs.nl/ipa/xml /etc/ntp.conf:server alblas.ghs.nl /etc/ldap/ldap.conf:URI ldaps://alblas.ghs.nl
Do I have to visit each client and modify these files? Anything else?
On 12/20/18 11:51 AM, Kees Bakker via FreeIPA-users wrote:
On 19-12-18 12:06, Kees Bakker via FreeIPA-users wrote:
On 18-12-18 17:50, Florence Blanc-Renaud wrote: [...]
If you have a spare machine you can also use replication, and create a replica of your current master with all the needed services (CA, KRA, DNS if needed). If you really need to keep the same hostname, then you will need a spare machine:
- create serverB as a replica of serverA on your spare machine. Do not forget to promote serverB as CA renewal master and CRL master [2].
- decommission serverA with (on serverA) ipa-server-install --uninstall and (on serverB) ipa-replica-manage del serverA --clean
- provision your new hardware with hostname=serverA, install serverA as a replica of serverB.
I would advise to keep serverB as it will provide redundancy.
This wiki [3] also explains the preferred paths depending on your situation.
I have read that document too. First I want to give it another try. If it fails again I will follow advice described above.
Thanks for your help.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... [2] https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master [3] https://www.freeipa.org/page/Backup_and_Restore
Just to let you know, I have given up with my "rsync" procedure. I am now following the steps above. Well, except step3, because I didn't want to add even more hardware in the process (the "spare machine" mentioned above).
Step 1 is completed. Promotion of CA renewal and CRL master is done.
I have a remaining question. What do I do with all the IPA clients that point to serverA? At some point I want to execute step 2, and shut off that system. I briefly looked at the files in /etc and found these (alblas is my serverA):
If your applications are using DNS to find the server, they will be fine. But some have hardcoded values and need to be reconfigured.
/etc/sssd/sssd.conf:ipa_server = _srv_, alblas.ghs.nl
Please have a look at the man page sssd-ipa(5), especially the SERVICE DISCOVERY section. _srv_ means that service discovery will be used to find a server, and if no servers can be discovered using DNS, alblas will be used instead.
/etc/ipa/default.conf:server = alblas.ghs.nl /etc/ipa/default.conf:xmlrpc_uri = https://alblas.ghs.nl/ipa/xml
default.conf is used for all the ipa * commands. By defaut, the command will start with the configured xmlrpc_uri but if it fails, it will fall back to the _ldap._tcp. servers found in the DNS. So if you replace alblas with the new servre hostname you will speed up the command.
/etc/ntp.conf:server alblas.ghs.nl /etc/ldap/ldap.conf:URI ldaps://alblas.ghs.nl
The URI is used as default if none is provided to ldapsearch.
Do I have to visit each client and modify these files? Anything else?
Before completely removing your initial server, perform ipactl stop on the initial server and check that the clients are still working: # id $USER # kinit $USER # ipa user-find # host `hostname` # ipa cert-find 1
HTH, flo
freeipa-users@lists.fedorahosted.org
-
Brian Topping -
Florence Blanc-Renaud -
Kees Bakker -
Robbie Harwood