Hi Team, We have 2 IPA servers in Mater-Master setup are we facing the below issue on these servers.
Isuue1: Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “*loging failed due to an unknown reason*” error while we log in to the UI
1. First, the IPA console was not working as httpd service was stopped, httpd was not starting as HTTP certificate is expired. Added *NSSEnforceValidCerts off* line in nss.conf to start the service.
2. After the change IPA console was loading we are not able to login to the console as pki-tomcatd service was not running, [root@ipa1 ca]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 97233 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background process Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug 31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
# getcert list Request ID '20180412150739': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxxx.COM subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Issue2:
On the IPA2 server, we are unable to login with the admin user credentials without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error *"The password you entered is incorrect."*
# [root@ipa2 log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l ● ipa-otpd.socket - ipa-otpd socket Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled) Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; 1h 31min ago Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd socket. Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket entered failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket [Unit] Description=ipa-otpd socket
[Socket] ListenStream=/var/run/krb5kdc/DEFAULT.socket RemoveOnStop=true SocketMode=0600 Accept=true
[Install] WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as the changes made on IPA2 is not reflecting on IPA1
We the below errors as well.
IPA1 Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxxx.xxxx.COM Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
IPA2 # tailf krb5kdc.log Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM, Additional pre-authentication required Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11 Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/ xxx.xxxx.COM@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11 Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Regards Nikita S
On 11/13/19 8:41 AM, Nikita Deeksha wrote:
Hi Florence, Please find the below getcert complete list. I had changed the time to*24 October 2019 *where all the certificates were valid, subsystem cert was renewed in the month of September 2019.
Hi,
according to your getcert output, 24 Oct 2019 should be ok.
*[root@ipa1 nikita.d]# getcert list* Number of certificates and requests being tracked: 8. Request ID '20171024201539': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxx.COM http://xxx.xxx.COM subject: CN=IPA RA,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2021-09-06 03:26:08 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20171024201553': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=CA Audit,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2021-09-06 03:26:28 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201554': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=OCSP Subsystem,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2021-09-06 03:25:30 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201555': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=CA Subsystem,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2021-09-06 03:25:48 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201556': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2037-10-24 20:15:28 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201557': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=ipa1.xxx.xxx.com http://ipa1.xxx.xxx.com,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2021-09-06 03:26:18 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201637': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=ipa1.xxx.xxxx.com http://ipa1.xxx.xxxx.com,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2021-09-28 03:25:29 UTC principal name: krbtgt/CORP.ENDURANCE.COM@CORP.ENDURANCE.COM mailto:CORP.ENDURANCE.COM@CORP.ENDURANCE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20180412150739': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN=ipa1.corp.endurance.com http://ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN=ipa1.corp.endurance.com http://ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM subject: CN=ipa1.xxx.xxxx.com http://ipa1.xxx.xxxx.com,O=xxx.xxxx.COM http://xxx.xxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/CORP.ENDURANCE.COM@CORP.ENDURANCE.COM mailto:CORP.ENDURANCE.COM@CORP.ENDURANCE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
*Subsystem cert*
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'subsystemCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 57 (0x39) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=xxx.xxxx.COM http://xxx.xxxx.COM" Validity: Not Before: Tue Sep 17 03:25:48 2019 Not After : Mon Sep 06 03:25:48 2021 Subject: "CN=CA Subsystem,O=xxx.xxxx.COM http://xxx.xxxx.COM" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: c2:e8:9c:3f:ad:99:88:90:53:56:a1:3e:eb:33:1d:ab: 91:c7:1e:3b:46:09:f2:53:cf:ca:aa:07:9f:05:ed:e7: d1:45:68:1b:60:71:92:c2:c8:d4:88:e3:d0:3a:49:d9: 95:56:6c:ba:87:0b:61:4f:b3:2b:10:e6:27:5c:3b:22: 90:16:e1:6f:f8:44:cf:df:37:8a:f1:28:cb:c0:e0:f3: f8:2d:c8:a4:95:dd:76:78:03:a1:b9:ea:63:3b:38:1a: 7a:db:b6:43:90:c5:ee:b6:44:a5:3b:e2:64:b8:94:4d: 02:4c:0c:cf:30:40:cf:9c:64:15:69:dc:b3:d6:8b:c0: ff:09:58:97:40:a7:3f:4a:37:96:cd:62:fa:b1:03:53: c6:75:a1:25:1d:0e:8e:fa:6b:98:a7:89:07:aa:00:10: df:d3:e5:c2:ea:af:9a:f1:9c:d0:ac:9f:63:2d:5d:b4: 00:5e:ad:63:c1:1b:87:62:4a:5b:c0:3e:b8:37:e6:80: 29:b6:06:47:4d:d5:1d:65:a0:3f:53:12:6d:ba:77:5d: 85:ae:74:cd:f8:87:4f:9c:95:97:93:7a:89:8d:49:35: 6e:c6:f1:34:9f:d2:42:a4:01:a9:79:20:c7:20:66:e6: bd:31:b9:f8:ef:4b:bb:b7:59:4d:ef:d4:1d:2e:78:9d Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: 62:dd:fb:9c:2d:47:73:1a:a4:17:76:47:fb:9f:63:ef: bf:be:db:a5
Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.xxx.xxxx.com/ca/ocsp"
Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Key Encipherment Data Encipherment
Name: Extended Key Usage TLS Web Server Authentication Certificate TLS Web Client Authentication Certificate
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 1f:d3:47:c9:bf:67:23:8d:57:99:af:b1:58:74:76:59: 44:1e:d1:a1:a6:4d:21:8f:92:8e:c2:a4:7f:11:8f:f0: a0:85:14:e9:5e:28:4e:82:5e:5a:ad:5e:50:c3:d0:12: 98:9e:ab:c3:bf:ed:c2:db:b1:b5:50:88:d3:5e:65:29: 6c:94:bf:4c:a7:22:7a:66:58:05:90:bb:c8:33:6c:d0: 5d:4a:d3:85:55:d3:47:35:09:e6:a0:f9:fe:37:91:a4: 7e:a0:86:45:a0:41:a7:1f:04:8b:28:dc:a8:4b:0b:28: 76:92:8f:5a:71:9f:ca:b5:c0:b7:4a:23:ea:78:7e:27: a0:03:ba:9d:ee:24:cc:4b:5c:ba:7c:5e:b9:7b:17:14: 6d:f0:f2:a0:67:2a:4b:9f:9d:91:17:d0:f3:3d:91:fe: 7f:b9:03:54:06:b2:dd:fa:b2:f0:ff:c3:64:93:11:4e: 89:18:52:30:c3:54:18:36:32:ae:c1:f7:d2:87:3d:5e: 93:5c:76:e0:73:f6:fb:23:d4:dd:13:3e:5a:67:46:40: e2:64:a4:58:b3:70:fd:9c:15:3d:bc:76:d9:da:e1:12: ba:6f:22:ce:61:dd:18:d1:6c:fa:81:ed:f3:6c:35:6b: 45:dc:61:59:06:7d:ef:8d:1f:5f:ba:0a:35:67:25:12 Fingerprint (SHA-256):
79:19:D3:40:03:55:51:69:57:1D:54:46:16:36:F3:BD:DE:1B:18:2D:6D:D3:22:9B:3A:2E:BF:FD:74:E3:50:87 Fingerprint (SHA1): D6:BE:5D:6D:91:24:E1:A4:AE:4C:C1:4A:70:4B:92:F6:3B:6D:4E:A8
Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User
In https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... they are asking to *Check the subsystemCert cert-pki-ca* where we check if**the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag /internal=/…)
[root@ipa1 nikita.d]# *certutil -K -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/alias/pwdfile.txt* certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS Certificate DB:Server-Cert cert-pki-ca < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS Certificate DB:auditSigningCert cert-pki-ca < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS Certificate DB:ocspSigningCert cert-pki-ca
This command shows that the keys can be read using the password from /etc/pki/pki-tomcat/alias/pwdfile.txt so you're good.
What you need to check next is that the entry uid=pkidbuser,ou=people,o=ipaca contains the latest version of the certificate (both the usercertificate and description attributes must be up-to-date).
I also noted from your output that the certificate for the Directory Server is not tracked by certmonger. Did you replace it with a cert provided by an external ca? If not, then you need to track the Server-Cert from /etc/dirsrv/slapd-<instance>, and make sure it also gets renewed:
$ getcert start-tracking -d /etc/dirsrv/slapd-<instance> -n Server-Cert -c IPA -p /etc/dirsrv/slapd-<instance>/pwdfile.txt -C "/usr/libexec/ipa/certmonger/restart_dirsrv <instance>"
HTH, flo
[root@ipa1 nikita.d]# [root@ipa1 nikita.d]# *sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'NSS Certificate DB:Server-Cert cert-pki-ca'* certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Incorrect password/PIN entered. certutil: could not authenticate to token NSS Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect. [root@ipa1 nikita.d]# *sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'* certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Incorrect password/PIN entered. certutil: could not authenticate to token NSS Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
Regards Nikita S
On Tue, Nov 12, 2019 at 8:04 PM Florence Blanc-Renaud <flo@redhat.com mailto:flo@redhat.com> wrote:
On 11/8/19 4:33 PM, Nikita Deeksha via FreeIPA-users wrote: > Alexander/Florence, > > While we were trying to renew the certificate the httpd cert, we went > back in time where httpd cert was vaild and then tried to renew the > cert, but in *Step7: Test CA operation* is failing with the below error. > Can you please help us with this? > > https://access.redhat.com/solutions/3357261 > > [root@ipa1 ~]# curl --cacert /etc/ipa/ca.crt -v > https://`hostname`:8443/ca/ee/ca/getCertChain > * About to connect() to ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> > <http://ipa1.corp.endurance.com> port 8443 (#0) > * Trying 172.27.152.34... > * Connected to ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> <http://ipa1.corp.endurance.com> > (172.27.152.34) port 8443 (#0) > * Initializing NSS with certpath: sql:/etc/pki/nssdb > * CAfile: /etc/ipa/ca.crt > CApath: none > * NSS: client certificate not found (nickname not specified) > * SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA > * Server certificate: > * subject: CN=ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> > <http://ipa1.corp.endurance.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM> > * start date: Sep 17 03:26:18 2019 GMT > * expire date: Sep 06 03:26:18 2021 GMT > * common name: ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> <http://ipa1.corp.endurance.com> > * issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM> > > GET /ca/ee/ca/getCertChain HTTP/1.1 > > User-Agent: curl/7.29.0 > > Host: ipa1.corp.endurance.com:8443 <http://ipa1.corp.endurance.com:8443> <http://ipa1.corp.endurance.com:8443> > > Accept: */* > > > < HTTP/1.1 500 Internal Server Error > < Server: Apache-Coyote/1.1 > < Content-Type: text/html;charset=utf-8 > < Content-Language: en > < Content-Length: 2208 > < Date: Thu, 24 Oct 2019 16:31:10 GMT > < Connection: close > < > <html><head><title>Apache Tomcat/7.0.76 - Error > report</title><style><!--H1 > {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} > H2 > {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} > H3 > {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} > BODY > {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B > {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} > P > {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A > {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> > </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR > size="1" noshade="noshade"><p><b>type</b> Exception > report</p><p><b>message</b> <u>Subsystem > unavailable</u></p><p><b>description</b> <u>The server encountered an > internal error that prevented it from fulfilling this > request.</u></p><p><b>exception</b> > <pre>javax.ws.rs <http://javax.ws.rs>.ServiceUnavailableException: Subsystem unavailable Subsystem unavailable means that the CA subsystem is not running. > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145) > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500) > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) > org.apache.tomcat.util.net <http://org.apache.tomcat.util.net>.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > java.lang.Thread.run(Thread.java:748) > </pre></p><p><b>note</b> <u>The ful* Closing connection 0 > l stack trace of the root cause is available in the Apache Tomcat/7.0.76 > logs.</u></p><HR size="1" noshade="noshade"><h3>Apache > Tomcat/7.0.76</h3></body></html>[root@ipa1 ~]# > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > /var/log/pki/pki-tomcat/ca/debug shows this error > > > Could not connect to LDAP server host ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> > <http://ipa1.corp.endurance.com> port 636 Error > netscape.ldap.LDAPException: Authentication failed (49) This happens when the subsystem cert is not accepted as valid authentication from Dogtag to the ldap server. > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) > at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > at > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > at > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > at > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > at > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > at > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > at > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Internal Database Error encountered: Could not connect to LDAP server > host ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> <http://ipa1.corp.endurance.com> port 636 > Error netscape.ldap.LDAPException: Authentication failed (49) > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) > at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > at > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > at > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > at > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > at > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > at > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > at > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [24/Oct/2019:16:29:50][localhost-startStop-1]: CMS.start(): shutdown server > [24/Oct/2019:16:29:50][localhost-startStop-1]: CMSEngine.shutdown() > > > ------------------------------------------------------------------------------------------- > sybsystemCert is still valid. Which date in the past did you use? If the subsystemcert's Not Before is after this date, the cert is not valid yet and authentication will fail. Could you post the full output of getcert list? If you have multiple certificates expired, you need to carefully select a date in the past where all the certs are in their validity window (ie after "Not before" and before "Not after"). HTH, flo > > Request ID '20171024201555': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM> > subject: CN=CA Subsystem,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> <http://CORP.ENDURANCE.COM> > expires: 2021-09-06 03:25:48 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > track: yes > auto-renew: yes > ------------------------------------------------------------- > [root@ipa1 ~]# cat /etc/pki/pki-tomcat/ca/CS.cfg| grep subsystem.cert > ca.cert.subsystem.certusage=SSLClient > ca.subsystem.cert=MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS > ca.subsystem.certreq=MIICeTCCAWECAQAwNDEbMBkGA1UECgwSQ09SUC5FTkRVUkFOQ0UuQ09NMRUwEwYDVQQDDAxDQSBTdWJzeXN0ZW0wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDC6Jw/rZmIkFNWoT7rMx2rkcceO0YJ8lPPyqoHnwXt59FFaBtgcZLCyNSI49A6SdmVVmy6hwthT7MrEOYnXDsikBbhb/hEz983ivEoy8Dg8/gtyKSV3XZ4A6G56mM7OBp627ZDkMXutkSlO+JkuJRNAkwMzzBAz5xkFWncs9aLwP8JWJdApz9KN5bNYvqxA1PGdaElHQ6O+muYp4kHqgAQ39PlwuqvmvGc0KyfYy1dtABerWPBG4diSlvAPrg35oAptgZHTdUdZaA/UxJtunddha50zfiHT5yVl5N6iY1JNW7G8TSf0kKkAal5IMcgZua9Mbn470u7t1lN79QdLnidAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEABl3uxfntEZZFeoB1eIjd774MSxK2itaj6B1VyXV0qxcp/SHF1sfJnUHjDpNIB0J/Bpk/sc/+Y2Gxar3zcwyORjF+ojCBc8lP6QZ7mB9H3m9XEuA+H1qmgGz+h2FhnQy5/EmgJnTAn8fNpZCQwQmSZLtnbmd++yK9y+YXCAvaHx+Wsz5c/GhyNxoHGADcmir+iFVltO3jpQ06ThVBb0bD4D6ZzaimkM1vnAqseKdqXvLJI1/ZfIVj6QDcdnnTryt24zOZoxzeOcc2X5HKOtR/5pXilmCS/7zpujXTd1iZi8nfRkpurUdMgC7FLK+B3Dq03G61RPJUwJZvsKlXbwBghg== > [root@ipa1 ~]# cat /tmp/subsystemCert\ cert-pki-ca.pem > MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS > > I had a few queries on this > > 1. Is there any way we take the backup of the current setup and create a > new IPA instance without changing any parameter in the client servers > 2. If we upgrade the IPA version we can fix this issue? > > > Regards > Nikita S > > > > On Fri, Nov 8, 2019 at 4:29 PM Nikita Deeksha <nikita.d@endurance.com <mailto:nikita.d@endurance.com> > <mailto:nikita.d@endurance.com <mailto:nikita.d@endurance.com>>> wrote: > > Thanks, Florence and Alexander. > > @Alexander Bokovoy <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>> , > > While we were debugging in one of a blog we came across a scenario > where NSS DB had got corrupted during a certificate renewal, so > wanted to eliminate that issue. > > We see all the private key. > > [root@ipa1 nikita.d]# certutil -K -d /etc/pki/pki-tomcat/alias -f > /tmp/pwdfile.txt > certutil: Checking token "NSS Certificate DB" in slot "NSS User > Private Key and Certificate Services" > < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS > Certificate DB:subsystemCert cert-pki-ca > < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) > < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS > Certificate DB:Server-Cert cert-pki-ca > < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS > Certificate DB:auditSigningCert cert-pki-ca > < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d > caSigningCert cert-pki-ca > < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS > Certificate DB:ocspSigningCert cert-pki-ca > > We will try renewing the HTTP cert in our environment, will let you > know once the cert is updated successfully. > > https://access.redhat.com/solutions/3357261 > > Regards > Nikita S > > On Fri, Nov 8, 2019 at 3:42 PM Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com> > <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote: > > On 11/7/19 11:16 AM, Nikita Deeksha via FreeIPA-users wrote: > > Thanks for the update Alexander will check this and get back > to you, > > wanted to check on another thing as well. > > > > Can you please help us to understand this error that we see > for the cert > > in pki > > > > [root@ipa1 nikita.d]# for i in $(certutil -d > /etc/pki/pki-tomcat/alias > > -L | grep cert-pki-ca | awk'{print $1}');do certutil -K -d > > /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i > cert-pki-ca";done > > > > certutil: Checking token "NSS Certificate DB" in slot "NSS > User Private > > Key and Certificate Services" > > > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: > Unrecognized > > Object Identifier. > > > > certutil: Checking token "NSS Certificate DB" in slot "NSS > User Private > > Key and Certificate Services" > > > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: > Unrecognized > > Object Identifier. > > > > certutil: Checking token "NSS Certificate DB" in slot "NSS > User Private > > Key and Certificate Services" > > > > < 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d > caSigningCert > > cert-pki-ca > > > > certutil: Checking token "NSS Certificate DB" in slot "NSS > User Private > > Key and Certificate Services" > > > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: > Unrecognized > > Object Identifier. > > > > certutil: Checking token "NSS Certificate DB" in slot "NSS > User Private > > Key and Certificate Services" > > > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: > Unrecognized > > Object Identifier. > > > Hi, > > If you use "certutil -K -d /etc/pki/pki-tomcat/alias -f > /tmp/pwdfile.txt" (without the -n alias option), you will see > all the > keys present in the database and notice that some of them have a > prefixed nickname, for instance: > 'NSS Certificate DB:Server-Cert cert-pki-ca' > instead of 'Server-Cert cert-pki-ca'. > You need to provide this prefixed name with -K -n nickname. > > HTH, > flo > > > > > > These are the cert which is present /etc/pki/pki-tomcat/alias > > > > [root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias > > > > Certificate Nickname > Trust > > Attributes > > > > SSL,S/MIME,JAR/XPI > > > > COMODO CA BUNDLE > CT,C,C > > ocspSigningCert cert-pki-ca > u,u,u > > auditSigningCert cert-pki-ca > u,u,Pu > > caSigningCert cert-pki-ca > CTu,Cu,Cu > > subsystemCert cert-pki-ca > u,u,u > > Server-Cert cert-pki-ca > u,u,u > > > > > > > > Regards > > > > Nikita S > > > > > > On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy > <abokovoy@redhat.com <mailto:abokovoy@redhat.com> <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>> > > <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com> <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>>> wrote: > > > > On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users > wrote: > > >2. > > > > > >Status SUBMITTING means the renewal is not yet > completed. It will not > > >complete until you get Dogtag working. > > > > > >But now the status says CA_UNEACHABLE > > > > > >Request ID '20180412150739': > > >status: CA_UNREACHABLE > > >ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml > failed > > request, will > > >retry: -504 (libcurl failed to execute the HTTP POST > transaction, > > >explaining: Peer's Certificate has expired.). > > > > This is exactly an issue with expired HTTP certificate. > > I guess you'd need to roll back time to when the > certificate was valid > > (before 2019-10-25) and restart certmonger. > > > > See discussion in this thread: > > > https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html > > > > In newer RHEL version (RHEL 7.7) there is a special tool, > ipa-cert-fix, > > that can help with fixing these issues. However, since > you are on the > > version before it, you need to do manual renewal. > > > > >stuck: no > > >key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='CN= > > >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com> > <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM> > > <http://CORP.ENDURANCE.COM>',token='NSS Certificate > > >DB',pinfile='/etc/httpd/alias/pwdfile.txt' > > >certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='CN= > > >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com> > <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM> > > <http://CORP.ENDURANCE.COM>',token='NSS Certificate DB' > > >CA: IPA > > >issuer: CN=Certificate Authority,O=xxx.xxxx.COM <http://xxx.xxxx.COM> > <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > > >subject: CN=ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> > <http://ipa1.xxxx.xxxxx.com> > > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> > <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > > >expires: 2019-10-25 20:16:38 UTC > > >principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> > <mailto:xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM>> > > <mailto:xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> > <mailto:xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM>>> > > >key usage: > > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >eku: id-kp-serverAuth,id-pkinit-KPKdc > > >pre-save command: > > >post-save command: > /usr/libexec/ipa/certmonger/restart_httpd > > >track: yes > > > > > > > > > > > > > > >*Issue2:* > > > > > >We are getting this alert while we log in to UI in > httpd error logs > > > > > > > > >#ipa: INFO: 401 Unauthorized: kinit: Preauthentication > failed > > while getting > > >initial credentials > > > > > >and PKINIT was disabled > > > > > >[root@ipa2 httpd]# ipa-pkinit-manage status > > >PKINIT is disabled > > > > > >While I tried to enable this > > > > > >[root@ipa2 httpd]# ipa-pkinit-manage enable > > >Configuring Kerberos KDC (krb5kdc) > > > [1/1]: installing X509 Certificate for PKINIT > > > > > >the process was getting stuck, so I had to terminate it > manually. > > After > > >trying to enable, I'm getting "Login failed due to an > unknown reason." > > >error in web UI when I try to login > > > > > >*Error in httpd:* > > > > > >[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] mod_wsgi (pid=24416): > > Exception occurred processing WSGI > > >script '/usr/share/ipa/wsgi.py'. > > >[Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] Traceback (most recent > > call last): > > >[Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > "/usr/share/ipa/wsgi.py", line 59, in application > > >[Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] return > > api.Backend.wsgi_dispatch(environ, > > >start_response) > > >[Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > > 267, in > > >__call__ > > >[Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] return > > self.route(environ, start_response) > > >[Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > > 279, in > > >route > > >[Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] return app(environ, > > start_response) > > >[Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > > 937, in > > >__call__ > > >[Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] > > self.kinit(user_principal, password, ipa_ccache_name) > > >[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > > 973, in > > >kinit > > >[Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] > > pkinit_anchors=[paths.KDC_CERT, > > >paths.KDC_CA_BUNDLE_PEM], > > >[Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > > >"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line > > 127, in > > >kinit_armor > > >[Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] run(args, env=env, > > raiseonerr=True, capture_error=True) > > >[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] File > > > >"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, > > in run > > >[Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] raise > > CalledProcessError(p.returncode, arg_string, > > >str(output)) > > >[Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] > [remote > > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0> > <http://172.27.10.113:0>] CalledProcessError: > > Command '/usr/bin/kinit -n -c > > >/var/run/ipa/ccaches/armor_24416 -X > > >X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X > > > >X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' > returned > > >non-zero exit status 1 > > > > > >And when I try to list the certificates using *getcert > list,* > > there is a > > >new cert which was added > > > > > >Request ID '20191106100258': > > >status: CA_UNREACHABLE > > >ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml > failed > > request, will > > >retry: 907 (RPC failed at server. cannot connect to ' > > >https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': > [SSL: > > >CERTIFICATE_VERIFY_FAILED] certificate verify failed > (_ssl.c:618)). > > >stuck: no > > >key pair storage: > type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > > >certificate: > type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > > >CA: IPA > > >issuer: > > >subject: > > >expires: unknown > > >pre-save command: > > >post-save command: > /usr/libexec/ipa/certmonger/renew_kdc_cert > > >track: yes > > >auto-renew: yes > > > > > > > > >Regards > > >Nikita S > > > > > >When > > >On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy > > <abokovoy@redhat.com <mailto:abokovoy@redhat.com> <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>> > <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com> <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>>> > > >wrote: > > > > > >> On ti, 05 marras 2019, Nikita Deeksha via > FreeIPA-users wrote: > > >> >Hi Team, > > >> >We have 2 IPA servers in Mater-Master setup are we > facing the > > below issue > > >> >on these servers. > > >> > > > >> >Isuue1: > > >> >Our httpd certificate has expired because of which > our IPA1 UI > > wasn't > > >> >working, we are getting “*loging failed due to an > unknown > > reason*” error > > >> >while we log in to the UI > > >> > > > >> > > > >> >1. First, the IPA console was not working as httpd > service was > > stopped, > > >> >httpd was not starting as HTTP certificate is > expired. Added > > >> >*NSSEnforceValidCerts > > >> >off* line in nss.conf to start the service. > > >> > > > >> >2. After the change IPA console was loading we are > not able to > > login to > > >> the > > >> >console as pki-tomcatd service was not running, > > >> >[root@ipa1 ca]# ipactl status > > >> >Directory Service: RUNNING > > >> >krb5kdc Service: RUNNING > > >> >kadmin Service: RUNNING > > >> >httpd Service: RUNNING > > >> >ipa-custodia Service: RUNNING > > >> >ntpd Service: RUNNING > > >> >pki-tomcatd Service: STOPPED > > >> >ipa-otpd Service: RUNNING > > >> > > > >> ># systemctl status pki-tomcatd@pki-tomcat.service -l > > >> >● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server > pki-tomcat > > >> > Loaded: loaded > (/lib/systemd/system/pki-tomcatd@.service; > > enabled; > > >> >vendor preset: disabled) > > >> > Active: active (running) since Tue 2019-11-05 > 10:16:50 GMT; > > 31min ago > > >> > Process: 97068 ExecStartPre=/usr/bin/pkidaemon > start %i > > (code=exited, > > >> >status=0/SUCCESS) > > >> > Main PID: 97233 (java) > > >> > CGroup: > > >> > > > >/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service > > >> > └─97233 > /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > > >> >-DRESTEASY_LIB=/usr/share/java/resteasy-base > > >> >-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath > > >> > > >> > > > >/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > > >> >-Dcatalina.base=/var/lib/pki/pki-tomcat > > -Dcatalina.home=/usr/share/tomcat > > >> >-Djava.endorsed.dirs= > -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > > >> > > >> > > > >-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > > >> > >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > > >> >-Djava.security.manager > > >> > > > >-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > > >> >org.apache.catalina.startup.Bootstrap start > > >> > > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: WARNING: Exception > > >> >processing realm > com.netscape.cms.tomcat.ProxyRealm@1896e072 > > background > > >> >process > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: > > >> >javax.ws.rs <http://javax.ws.rs> <http://javax.ws.rs> > <http://javax.ws.rs>.ServiceUnavailableException: > > Subsystem unavailable > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > > >com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > >> > > > >org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> > <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > > server[97233]: at > > >> >java.lang.Thread.run(Thread.java:748) > > >> > > > >> > > > >> >This service wasn’t starting with this error > > >> > > > >> ># less /var/log/pki/pki-tomcat/ca/debug > > >> >31/Oct/2019:13:24:23][localhost-startStop-1]: > > >> >SSLClientCertificateSelectionCB: desired cert found > in list: > > subsystemCert > > >> >cert-pki-ca > > >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: > > >> >SSLClientCertificateSelectionCB: returning: > subsystemCert > > cert-pki-ca > > >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL > handshake > > happened > > >> >Could not connect to LDAP server host > ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com> > > <http://ipa1.xxx.xxxx.com> port 636 Error > > >> >netscape.ldap.LDAPException: Authentication failed (49) > > >> > > >> Authentication failed means the RA agent certificate > dogtag uses to > > >> authenticate to LDAP server is not the same as the > one mentioned > > in the > > >> LDAP entry for RA agent. > > >> > > >> I think there was some procedure to fix it but I > don't have > > links handy. > > >> Also, you did not specify what versions of FreeIPA > you run. > > >> > > >> > > >> >at > > >> > > >> > > > >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) > > >> > at > > >> > > >> > > > >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) > > >> > at > > >> > > >> > > > >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) > > >> > at > > > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) > > >> > at > > >> > > > >com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > > >> > at > > >> > > > >com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > > >> > at > > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) > > >> > at > com.netscape.certsrv.apps.CMS.init(CMS.java:189) > > >> > at > com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > > >> > at > > >> > > >> > > > >com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > > >> > at > > javax.servlet.GenericServlet.init(GenericServlet.java:158) > > >> > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > >> > at > > >> > > >> > > > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > >> > at > > >> > > >> > > > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > at > java.lang.reflect.Method.invoke(Method.java:498) > > >> > at > > >> > > > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > > >> > at > > >> > > > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > > >> > at > java.security.AccessController.doPrivileged(Native > > Method) > > >> > at > > javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > > >> > at > > >> > > > >org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > > >> > at > > >> > > >> > > > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > > >> > at > > >> > > >> > > > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > > >> > at > > >> > > > >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > > >> > at > > >> > > > >org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > > >> > at > > >> > > > >org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > > >> > at > java.security.AccessController.doPrivileged(Native > > Method) > > >> > at > > >> > > > >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > > >> > at > > >> > > > >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > > >> > at > > >> > > >> > > > >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > > >> > at > > >> > > >> > > > >org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > > >> > at > > >> > > > >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > >> > at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > >> > at > > >> > > >> > > > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > >> > at > > >> > > >> > > > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > >> > at java.lang.Thread.run(Thread.java:748) > > >> >Internal Database Error encountered: Could not > connect to LDAP > > server host > > >> >ipa1.xxx.xxx.com <http://ipa1.xxx.xxx.com> <http://ipa1.xxx.xxx.com> > <http://ipa1.xxx.xxx.com> port 636 Error > > netscape.ldap.LDAPException: > > >> Authentication > > >> >failed (49) > > >> > at > > > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) > > >> > at > > >> > > > >com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > > >> > at > > >> > > > >com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > > >> > at > > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) > > >> > at > com.netscape.certsrv.apps.CMS.init(CMS.java:189) > > >> > at > com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > > >> > at > > >> > > >> > > > >com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > > >> > at > > javax.servlet.GenericServlet.init(GenericServlet.java:158) > > >> > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > >> > at > > >> > > >> > > > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > >> > at > > >> > > >> > > > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > at > java.lang.reflect.Method.invoke(Method.java:498) > > >> > at > > >> > > > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > > >> > at > > >> > > > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > > >> > at > java.security.AccessController.doPrivileged(Native > > Method) > > >> > at > > javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > > >> > at > > >> > > > >org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > > >> > at > > >> > > >> > > > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > > >> > at > > >> > > >> > > > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > > >> > at > > >> > > > >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > > >> > at > > >> > > > >org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > > >> > at > > >> > > > >org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > > >> > at > > >> > > >> > > > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > > >> > at > java.security.AccessController.doPrivileged(Native > > Method) > > >> > at > > >> > > > >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > > >> > at > > >> > > > >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > > >> > at > > >> > > >> > > > >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > > >> > at > > >> > > >> > > > >org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > > >> > at > > >> > > > >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > >> > at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > >> > at > > >> > > >> > > > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > >> > at > > >> > > >> > > > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > >> > at java.lang.Thread.run(Thread.java:748) > > >> > > > >> ># getcert list > > >> >Request ID '20180412150739': > > >> >status: SUBMITTING > > >> >stuck: no > > >> >key pair storage: > > type=NSSDB,location='/etc/httpd/alias',nickname='CN= > > >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> <http://ipa1.xxxx.xxxxx.com> > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > > <http://xxx.xxxx.COM>',token='NSS Certificate > > >> >DB',pinfile='/etc/httpd/alias/pwdfile.txt' > > >> >certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='CN= > > >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> <http://ipa1.xxxx.xxxxx.com> > > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM> > > <http://xxx.xxxxx.COM>',token='NSS Certificate DB' > > >> >CA: IPA > > >> >issuer: CN=Certificate Authority,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM> > > <http://xxx.xxxxx.COM> > > >> >subject: CN=ipa1.xxxx.xxxx.com <http://ipa1.xxxx.xxxx.com> > <http://ipa1.xxxx.xxxx.com> > > <http://ipa1.xxxx.xxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM> <http://xxx.xxxxx.COM> > > >> >expires: 2019-10-25 20:16:38 UTC > > >> >principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>> > > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>>> > > >> >key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> >eku: id-kp-serverAuth,id-pkinit-KPKdc > > >> >pre-save command: > > >> >post-save command: > /usr/libexec/ipa/certmonger/restart_httpd > > >> >track: yes > > >> >auto-renew: yes > > >> > > >> Status SUBMITTING means the renewal is not yet > completed. It > > will not > > >> complete until you get Dogtag working. > > >> > > >> > > > >> >Issue2: > > >> > > > >> >On the IPA2 server, we are unable to login with the > admin user > > credentials > > >> >without OTP, but when an AD user is trying to login > with 2FA (i.e, > > >> >password and OTP) we are getting this error *"The > password you > > entered is > > >> >incorrect."* > > >> > > >> AD users cannot use multifactor authentication > defined in IPA. > > >> > > >> > > >> ># [root@ipa2 log]# ipactl status > > >> >Directory Service: RUNNING > > >> >krb5kdc Service: RUNNING > > >> >kadmin Service: RUNNING > > >> >httpd Service: RUNNING > > >> >ipa-custodia Service: RUNNING > > >> >ntpd Service: RUNNING > > >> >ipa-otpd Service: STOPPED > > >> >ipa: INFO: The ipactl command was successful > > >> > > > >> ># systemctl status ipa-otpd.socket -l > > >> >● ipa-otpd.socket - ipa-otpd socket > > >> > Loaded: loaded > (/usr/lib/systemd/system/ipa-otpd.socket; > > disabled; > > >> >vendor preset: disabled) > > >> > Active: failed (Result: resources) since Tue > 2019-11-05 > > 08:19:04 GMT; > > >> 1h > > >> >31min ago > > >> > Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) > > >> > Accepted: 2; Connected: 0 > > >> > > > >> >Nov 05 07:42:53 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > systemd[1]: Listening on ipa-otpd > > >> socket. > > >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > systemd[1]: ipa-otpd.socket failed to > > >> >queue service startup job (Maybe the service file is > missing or > > not a > > >> >template unit?): Resource temporarily unavailable > > >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > systemd[1]: Unit ipa-otpd.socket > > >> entered > > >> >failed state. > > >> > > > >> ># cat /usr/lib/systemd/system/ipa-otpd.socket > > >> >[Unit] > > >> >Description=ipa-otpd socket > > >> > > > >> >[Socket] > > >> >ListenStream=/var/run/krb5kdc/DEFAULT.socket > > >> >RemoveOnStop=true > > >> >SocketMode=0600 > > >> >Accept=true > > >> > > > >> >[Install] > > >> >WantedBy=krb5kdc.service > > >> > > > >> > > > >> > > > >> >We see that data replication is broken between the 2 IPA > > servers, as the > > >> >changes made on IPA2 is not reflecting on IPA1 > > >> This is most likely because your LDAP server > certificate expired as > > >> well. > > >> > > >> > > >> >We the below errors as well. > > >> > > > >> >IPA1 > > >> >Nov 05 10:09:23 ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> > <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com> > > krb5kdc[28021](info): TGS_REQ (8 etypes > > >> >{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime > 1572948563, > > etypes > > >> >{rep=18 tkt=18 ses=18}, > ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> > <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM>> > > <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> > <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM>>> for ldap/ > > >> >ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>>> > > >> >Nov 05 10:14:24 ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> > <http://ipa1.corp.endurance.com> > > <http://ipa1.corp.endurance.com> krb5kdc[28021](info): > TGS_REQ (8 > > >> >etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: > authtime > > 1572948863, > > >> >etypes {rep=18 tkt=18 ses=18}, > > ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> > <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM>> > > <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> > <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM>>> for > > >> >ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>>> > > >> > > >> These aren't errors. They are normal operations: > ldap/ipa1 > > service (LDAP > > >> server on IPA1) asked for a Kerberos service ticket > to LDAP > > service on > > >> IPA2 and was granted it. This is just as it should be for > > replication. > > >> > > >> > > > >> >IPA2 > > >> ># tailf krb5kdc.log > > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > krb5kdc[2451](info): AS_REQ (8 etypes > > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: > ldap/ > > >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> > > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>>> for > > krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>>>, > > >> >Additional pre-authentication required > > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > krb5kdc[2451](info): closing down fd > > >> 11 > > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > krb5kdc[2451](info): AS_REQ (8 etypes > > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime > 1572947965, > > etypes > > >> >{rep=18 tkt=18 ses=18}, > ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> > > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>>> for krbtgt/ > > >> >xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> > <mailto:xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM>> > <mailto:xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> > <mailto:xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM>>> > > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > krb5kdc[2451](info): closing down fd > > >> 11 > > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > krb5kdc[2451](info): TGS_REQ (8 etypes > > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime > 1572947965, > > etypes > > >> >{rep=18 tkt=18 ses=18}, > ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>>> for ldap/ > > >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> > > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>>> > > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> > <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > > krb5kdc[2451](info): closing down fd > > >> 11 > > >> > > >> Same here. LDAP server on IPA2 operated against > itself here. > > >> > > >> -- > > >> / Alexander Bokovoy > > >> Sr. Principal Software Engineer > > >> Security / Identity Management Engineering > > >> Red Hat Limited, Finland > > >> > > >> > > > > > > -- > > / Alexander Bokovoy > > Sr. Principal Software Engineer > > Security / Identity Management Engineering > > Red Hat Limited, Finland > > > > > > _______________________________________________ > > FreeIPA-users mailing list -- > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >
On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Hi Team, We have 2 IPA servers in Mater-Master setup are we facing the below issue on these servers.
Isuue1: Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “*loging failed due to an unknown reason*” error while we log in to the UI
- First, the IPA console was not working as httpd service was stopped,
httpd was not starting as HTTP certificate is expired. Added *NSSEnforceValidCerts off* line in nss.conf to start the service.
- After the change IPA console was loading we are not able to login to the
console as pki-tomcatd service was not running, [root@ipa1 ca]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 97233 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background process Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug 31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Authentication failed means the RA agent certificate dogtag uses to authenticate to LDAP server is not the same as the one mentioned in the LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy. Also, you did not specify what versions of FreeIPA you run.
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
# getcert list Request ID '20180412150739': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxxx.COM subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
Issue2:
On the IPA2 server, we are unable to login with the admin user credentials without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error *"The password you entered is incorrect."*
AD users cannot use multifactor authentication defined in IPA.
# [root@ipa2 log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l ● ipa-otpd.socket - ipa-otpd socket Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled) Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; 1h 31min ago Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd socket. Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket entered failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket [Unit] Description=ipa-otpd socket
[Socket] ListenStream=/var/run/krb5kdc/DEFAULT.socket RemoveOnStop=true SocketMode=0600 Accept=true
[Install] WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as the changes made on IPA2 is not reflecting on IPA1
This is most likely because your LDAP server certificate expired as well.
We the below errors as well.
IPA1 Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxxx.xxxx.COM Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
These aren't errors. They are normal operations: ldap/ipa1 service (LDAP server on IPA1) asked for a Kerberos service ticket to LDAP service on IPA2 and was granted it. This is just as it should be for replication.
IPA2 # tailf krb5kdc.log Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM, Additional pre-authentication required Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11 Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/ xxx.xxxx.COM@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11 Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Same here. LDAP server on IPA2 operated against itself here.
Hi Alexander,
*Issue1:*
Authentication failed means the RA agent certificate dogtag uses to authenticate to LDAP server is not the same as the one mentioned in the LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy. Also, you did not specify what versions of FreeIPA you run.
1. We are using IPA version 4.5 version
[root@ipa1 ~]$ ipa --version VERSION: 4.5.0, API_VERSION: 2.228
Can you help us fix this issue?
2.
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
But now the status says CA_UNEACHABLE
Request ID '20180412150739': status: CA_UNREACHABLE ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.). stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes
*Issue2:*
We are getting this alert while we log in to UI in httpd error logs
#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while getting initial credentials
and PKINIT was disabled
[root@ipa2 httpd]# ipa-pkinit-manage status PKINIT is disabled
While I tried to enable this
[root@ipa2 httpd]# ipa-pkinit-manage enable Configuring Kerberos KDC (krb5kdc) [1/1]: installing X509 Certificate for PKINIT
the process was getting stuck, so I had to terminate it manually. After trying to enable, I'm getting "Login failed due to an unknown reason." error in web UI when I try to login
*Error in httpd:*
[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote 172.27.10.113:0] mod_wsgi (pid=24416): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote 172.27.10.113:0] Traceback (most recent call last): [Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/share/ipa/wsgi.py", line 59, in application [Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return api.Backend.wsgi_dispatch(environ, start_response) [Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in __call__ [Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return self.route(environ, start_response) [Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in route [Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return app(environ, start_response) [Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in __call__ [Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote 172.27.10.113:0] self.kinit(user_principal, password, ipa_ccache_name) [Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in kinit [Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote 172.27.10.113:0] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in kinit_armor [Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote 172.27.10.113:0] run(args, env=env, raiseonerr=True, capture_error=True) [Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run [Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote 172.27.10.113:0] raise CalledProcessError(p.returncode, arg_string, str(output)) [Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote 172.27.10.113:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_24416 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1
And when I try to list the certificates using *getcert list,* there is a new cert which was added
Request ID '20191106100258': status: CA_UNREACHABLE ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to ' https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Regards Nikita S
When On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy abokovoy@redhat.com wrote:
On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Hi Team, We have 2 IPA servers in Mater-Master setup are we facing the below issue on these servers.
Isuue1: Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “*loging failed due to an unknown reason*” error while we log in to the UI
- First, the IPA console was not working as httpd service was stopped,
httpd was not starting as HTTP certificate is expired. Added *NSSEnforceValidCerts off* line in nss.conf to start the service.
- After the change IPA console was loading we are not able to login to
the
console as pki-tomcatd service was not running, [root@ipa1 ca]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 97233 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background process Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug 31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Authentication failed means the RA agent certificate dogtag uses to authenticate to LDAP server is not the same as the one mentioned in the LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy. Also, you did not specify what versions of FreeIPA you run.
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException:
Authentication
failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
# getcert list Request ID '20180412150739': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxxx.COM subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
Issue2:
On the IPA2 server, we are unable to login with the admin user credentials without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error *"The password you entered is incorrect."*
AD users cannot use multifactor authentication defined in IPA.
# [root@ipa2 log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l ● ipa-otpd.socket - ipa-otpd socket Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled) Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT;
1h
31min ago Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd
socket.
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket
entered
failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket [Unit] Description=ipa-otpd socket
[Socket] ListenStream=/var/run/krb5kdc/DEFAULT.socket RemoveOnStop=true SocketMode=0600 Accept=true
[Install] WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as the changes made on IPA2 is not reflecting on IPA1
This is most likely because your LDAP server certificate expired as well.
We the below errors as well.
IPA1 Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxxx.xxxx.COM Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
These aren't errors. They are normal operations: ldap/ipa1 service (LDAP server on IPA1) asked for a Kerberos service ticket to LDAP service on IPA2 and was granted it. This is just as it should be for replication.
IPA2 # tailf krb5kdc.log Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM, Additional pre-authentication required Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/ xxx.xxxx.COM@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd
11
Same here. LDAP server on IPA2 operated against itself here.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
But now the status says CA_UNEACHABLE
Request ID '20180412150739': status: CA_UNREACHABLE ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.).
This is exactly an issue with expired HTTP certificate. I guess you'd need to roll back time to when the certificate was valid (before 2019-10-25) and restart certmonger.
See discussion in this thread: https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html
In newer RHEL version (RHEL 7.7) there is a special tool, ipa-cert-fix, that can help with fixing these issues. However, since you are on the version before it, you need to do manual renewal.
stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes
*Issue2:*
We are getting this alert while we log in to UI in httpd error logs
#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while getting initial credentials
and PKINIT was disabled
[root@ipa2 httpd]# ipa-pkinit-manage status PKINIT is disabled
While I tried to enable this
[root@ipa2 httpd]# ipa-pkinit-manage enable Configuring Kerberos KDC (krb5kdc) [1/1]: installing X509 Certificate for PKINIT
the process was getting stuck, so I had to terminate it manually. After trying to enable, I'm getting "Login failed due to an unknown reason." error in web UI when I try to login
*Error in httpd:*
[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote 172.27.10.113:0] mod_wsgi (pid=24416): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote 172.27.10.113:0] Traceback (most recent call last): [Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/share/ipa/wsgi.py", line 59, in application [Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return api.Backend.wsgi_dispatch(environ, start_response) [Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in __call__ [Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return self.route(environ, start_response) [Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in route [Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return app(environ, start_response) [Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in __call__ [Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote 172.27.10.113:0] self.kinit(user_principal, password, ipa_ccache_name) [Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in kinit [Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote 172.27.10.113:0] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in kinit_armor [Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote 172.27.10.113:0] run(args, env=env, raiseonerr=True, capture_error=True) [Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run [Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote 172.27.10.113:0] raise CalledProcessError(p.returncode, arg_string, str(output)) [Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote 172.27.10.113:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_24416 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1
And when I try to list the certificates using *getcert list,* there is a new cert which was added
Request ID '20191106100258': status: CA_UNREACHABLE ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to ' https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Regards Nikita S
When On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy abokovoy@redhat.com wrote:
On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Hi Team, We have 2 IPA servers in Mater-Master setup are we facing the below issue on these servers.
Isuue1: Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “*loging failed due to an unknown reason*” error while we log in to the UI
- First, the IPA console was not working as httpd service was stopped,
httpd was not starting as HTTP certificate is expired. Added *NSSEnforceValidCerts off* line in nss.conf to start the service.
- After the change IPA console was loading we are not able to login to
the
console as pki-tomcatd service was not running, [root@ipa1 ca]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 97233 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background process Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug 31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Authentication failed means the RA agent certificate dogtag uses to authenticate to LDAP server is not the same as the one mentioned in the LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy. Also, you did not specify what versions of FreeIPA you run.
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException:
Authentication
failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
# getcert list Request ID '20180412150739': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxxx.COM subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
Issue2:
On the IPA2 server, we are unable to login with the admin user credentials without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error *"The password you entered is incorrect."*
AD users cannot use multifactor authentication defined in IPA.
# [root@ipa2 log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l ● ipa-otpd.socket - ipa-otpd socket Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled) Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT;
1h
31min ago Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd
socket.
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket
entered
failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket [Unit] Description=ipa-otpd socket
[Socket] ListenStream=/var/run/krb5kdc/DEFAULT.socket RemoveOnStop=true SocketMode=0600 Accept=true
[Install] WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as the changes made on IPA2 is not reflecting on IPA1
This is most likely because your LDAP server certificate expired as well.
We the below errors as well.
IPA1 Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxxx.xxxx.COM Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
These aren't errors. They are normal operations: ldap/ipa1 service (LDAP server on IPA1) asked for a Kerberos service ticket to LDAP service on IPA2 and was granted it. This is just as it should be for replication.
IPA2 # tailf krb5kdc.log Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM, Additional pre-authentication required Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/ xxx.xxxx.COM@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd
11
Same here. LDAP server on IPA2 operated against itself here.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
Thanks for the update Alexander will check this and get back to you, wanted to check on another thing as well.
Can you please help us to understand this error that we see for the cert in pki
[root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias -L | grep cert-pki-ca | awk '{print $1}');do certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
These are the cert which is present /etc/pki/pki-tomcat/alias
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
COMODO CA BUNDLE CT,C,C ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u
Regards
Nikita S
On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy abokovoy@redhat.com wrote:
On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
But now the status says CA_UNEACHABLE
Request ID '20180412150739': status: CA_UNREACHABLE ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request,
will
retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.).
This is exactly an issue with expired HTTP certificate. I guess you'd need to roll back time to when the certificate was valid (before 2019-10-25) and restart certmonger.
See discussion in this thread: https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html
In newer RHEL version (RHEL 7.7) there is a special tool, ipa-cert-fix, that can help with fixing these issues. However, since you are on the version before it, you need to do manual renewal.
stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes
*Issue2:*
We are getting this alert while we log in to UI in httpd error logs
#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while
getting
initial credentials
and PKINIT was disabled
[root@ipa2 httpd]# ipa-pkinit-manage status PKINIT is disabled
While I tried to enable this
[root@ipa2 httpd]# ipa-pkinit-manage enable Configuring Kerberos KDC (krb5kdc) [1/1]: installing X509 Certificate for PKINIT
the process was getting stuck, so I had to terminate it manually. After trying to enable, I'm getting "Login failed due to an unknown reason." error in web UI when I try to login
*Error in httpd:*
[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote 172.27.10.113:0] mod_wsgi (pid=24416): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote 172.27.10.113:0] Traceback (most recent call last): [Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/share/ipa/wsgi.py", line 59, in application [Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return api.Backend.wsgi_dispatch(environ, start_response) [Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in __call__ [Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return self.route(environ, start_response) [Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in route [Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return app(environ, start_response) [Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in __call__ [Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote 172.27.10.113:0] self.kinit(user_principal, password,
ipa_ccache_name)
[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in kinit [Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote 172.27.10.113:0] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in kinit_armor [Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote 172.27.10.113:0] run(args, env=env, raiseonerr=True,
capture_error=True)
[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run [Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote 172.27.10.113:0] raise CalledProcessError(p.returncode, arg_string, str(output)) [Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote 172.27.10.113:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_24416 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1
And when I try to list the certificates using *getcert list,* there is a new cert which was added
Request ID '20191106100258': status: CA_UNREACHABLE ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request,
will
retry: 907 (RPC failed at server. cannot connect to ' https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Regards Nikita S
When On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy abokovoy@redhat.com wrote:
On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Hi Team, We have 2 IPA servers in Mater-Master setup are we facing the below
issue
on these servers.
Isuue1: Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “*loging failed due to an unknown reason*”
error
while we log in to the UI
- First, the IPA console was not working as httpd service was stopped,
httpd was not starting as HTTP certificate is expired. Added *NSSEnforceValidCerts off* line in nss.conf to start the service.
- After the change IPA console was loading we are not able to login to
the
console as pki-tomcatd service was not running, [root@ipa1 ca]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min
ago
Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 97233 (java) CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
└─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
-DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/var/lib/pki/pki-tomcat
-Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072
background
process Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug 31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list:
subsystemCert
cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Authentication failed means the RA agent certificate dogtag uses to authenticate to LDAP server is not the same as the one mentioned in the LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy. Also, you did not specify what versions of FreeIPA you run.
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at
com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Internal Database Error encountered: Could not connect to LDAP server
host
ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException:
Authentication
failed (49) at
com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
# getcert list Request ID '20180412150739': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxxx.COM subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
Issue2:
On the IPA2 server, we are unable to login with the admin user
credentials
without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error *"The password you entered
is
incorrect."*
AD users cannot use multifactor authentication defined in IPA.
# [root@ipa2 log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l ● ipa-otpd.socket - ipa-otpd socket Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled) Active: failed (Result: resources) since Tue 2019-11-05 08:19:04
GMT;
1h
31min ago Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd
socket.
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed
to
queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket
entered
failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket [Unit] Description=ipa-otpd socket
[Socket] ListenStream=/var/run/krb5kdc/DEFAULT.socket RemoveOnStop=true SocketMode=0600 Accept=true
[Install] WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as
the
changes made on IPA2 is not reflecting on IPA1
This is most likely because your LDAP server certificate expired as well.
We the below errors as well.
IPA1 Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for
ldap/
ipa2.xxxx.xxxx.com@xxxx.xxxx.COM Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ
(8
etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM
for
ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
These aren't errors. They are normal operations: ldap/ipa1 service (LDAP server on IPA1) asked for a Kerberos service ticket to LDAP service on IPA2 and was granted it. This is just as it should be for replication.
IPA2 # tailf krb5kdc.log Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM
,
Additional pre-authentication required Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down
fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for
krbtgt/
xxx.xxxx.COM@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down
fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for
ldap/
ipa2.xxxx.xxxx.com@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down
fd
11
Same here. LDAP server on IPA2 operated against itself here.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On to, 07 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Thanks for the update Alexander will check this and get back to you, wanted to check on another thing as well.
Can you please help us to understand this error that we see for the cert in pki
[root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias -L | grep cert-pki-ca | awk '{print $1}');do certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done
I'm not sure what you are trying to achieve with this?
Using
certutil -K -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/alias/pwdfile.txt
will show you all information already for all the certs present. Your version ignores full names of the certificates (they have spaces in the names) and therefore produces error output, as it should, for invalid input.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
These are the cert which is present /etc/pki/pki-tomcat/alias
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
COMODO CA BUNDLE CT,C,C ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u
The 'u' in the trust column means there are keys for these certificates in the database.
Regards
Nikita S
On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy abokovoy@redhat.com wrote:
On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
But now the status says CA_UNEACHABLE
Request ID '20180412150739': status: CA_UNREACHABLE ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request,
will
retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.).
This is exactly an issue with expired HTTP certificate. I guess you'd need to roll back time to when the certificate was valid (before 2019-10-25) and restart certmonger.
See discussion in this thread: https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html
In newer RHEL version (RHEL 7.7) there is a special tool, ipa-cert-fix, that can help with fixing these issues. However, since you are on the version before it, you need to do manual renewal.
stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxx.xxxx.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes
*Issue2:*
We are getting this alert while we log in to UI in httpd error logs
#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while
getting
initial credentials
and PKINIT was disabled
[root@ipa2 httpd]# ipa-pkinit-manage status PKINIT is disabled
While I tried to enable this
[root@ipa2 httpd]# ipa-pkinit-manage enable Configuring Kerberos KDC (krb5kdc) [1/1]: installing X509 Certificate for PKINIT
the process was getting stuck, so I had to terminate it manually. After trying to enable, I'm getting "Login failed due to an unknown reason." error in web UI when I try to login
*Error in httpd:*
[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote 172.27.10.113:0] mod_wsgi (pid=24416): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote 172.27.10.113:0] Traceback (most recent call last): [Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/share/ipa/wsgi.py", line 59, in application [Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return api.Backend.wsgi_dispatch(environ, start_response) [Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in __call__ [Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return self.route(environ, start_response) [Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in route [Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote 172.27.10.113:0] return app(environ, start_response) [Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in __call__ [Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote 172.27.10.113:0] self.kinit(user_principal, password,
ipa_ccache_name)
[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in kinit [Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote 172.27.10.113:0] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in kinit_armor [Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote 172.27.10.113:0] run(args, env=env, raiseonerr=True,
capture_error=True)
[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote 172.27.10.113:0] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run [Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote 172.27.10.113:0] raise CalledProcessError(p.returncode, arg_string, str(output)) [Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote 172.27.10.113:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_24416 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1
And when I try to list the certificates using *getcert list,* there is a new cert which was added
Request ID '20191106100258': status: CA_UNREACHABLE ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request,
will
retry: 907 (RPC failed at server. cannot connect to ' https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: subject: expires: unknown pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Regards Nikita S
When On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy abokovoy@redhat.com wrote:
On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote:
Hi Team, We have 2 IPA servers in Mater-Master setup are we facing the below
issue
on these servers.
Isuue1: Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “*loging failed due to an unknown reason*”
error
while we log in to the UI
- First, the IPA console was not working as httpd service was stopped,
httpd was not starting as HTTP certificate is expired. Added *NSSEnforceValidCerts off* line in nss.conf to start the service.
- After the change IPA console was loading we are not able to login to
the
console as pki-tomcatd service was not running, [root@ipa1 ca]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min
ago
Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 97233 (java) CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
└─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
-DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/var/lib/pki/pki-tomcat
-Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072
background
process Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug 31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list:
subsystemCert
cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Authentication failed means the RA agent certificate dogtag uses to authenticate to LDAP server is not the same as the one mentioned in the LDAP entry for RA agent.
I think there was some procedure to fix it but I don't have links handy. Also, you did not specify what versions of FreeIPA you run.
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at
com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Internal Database Error encountered: Could not connect to LDAP server
host
ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException:
Authentication
failed (49) at
com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676)
at
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
# getcert list Request ID '20180412150739': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxxx.COM subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
Status SUBMITTING means the renewal is not yet completed. It will not complete until you get Dogtag working.
Issue2:
On the IPA2 server, we are unable to login with the admin user
credentials
without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error *"The password you entered
is
incorrect."*
AD users cannot use multifactor authentication defined in IPA.
# [root@ipa2 log]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l ● ipa-otpd.socket - ipa-otpd socket Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled) Active: failed (Result: resources) since Tue 2019-11-05 08:19:04
GMT;
1h
31min ago Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd
socket.
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed
to
queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket
entered
failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket [Unit] Description=ipa-otpd socket
[Socket] ListenStream=/var/run/krb5kdc/DEFAULT.socket RemoveOnStop=true SocketMode=0600 Accept=true
[Install] WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as
the
changes made on IPA2 is not reflecting on IPA1
This is most likely because your LDAP server certificate expired as well.
We the below errors as well.
IPA1 Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for
ldap/
ipa2.xxxx.xxxx.com@xxxx.xxxx.COM Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ
(8
etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM
for
ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
These aren't errors. They are normal operations: ldap/ipa1 service (LDAP server on IPA1) asked for a Kerberos service ticket to LDAP service on IPA2 and was granted it. This is just as it should be for replication.
IPA2 # tailf krb5kdc.log Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM
,
Additional pre-authentication required Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down
fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for
krbtgt/
xxx.xxxx.COM@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down
fd
11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for
ldap/
ipa2.xxxx.xxxx.com@xxx.xxxx.COM Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down
fd
11
Same here. LDAP server on IPA2 operated against itself here.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On 11/7/19 11:16 AM, Nikita Deeksha via FreeIPA-users wrote:
Thanks for the update Alexander will check this and get back to you, wanted to check on another thing as well.
Can you please help us to understand this error that we see for the cert in pki
[root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias -L | grep cert-pki-ca | awk'{print $1}');do certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
If you use "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt" (without the -n alias option), you will see all the keys present in the database and notice that some of them have a prefixed nickname, for instance: 'NSS Certificate DB:Server-Cert cert-pki-ca' instead of 'Server-Cert cert-pki-ca'. You need to provide this prefixed name with -K -n nickname.
HTH, flo
These are the cert which is present /etc/pki/pki-tomcat/alias
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
COMODO CA BUNDLE CT,C,C ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u
Regards
Nikita S
On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy <abokovoy@redhat.com mailto:abokovoy@redhat.com> wrote:
On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote: >2. > >Status SUBMITTING means the renewal is not yet completed. It will not >complete until you get Dogtag working. > >But now the status says CA_UNEACHABLE > >Request ID '20180412150739': >status: CA_UNREACHABLE >ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request, will >retry: -504 (libcurl failed to execute the HTTP POST transaction, >explaining: Peer's Certificate has expired.). This is exactly an issue with expired HTTP certificate. I guess you'd need to roll back time to when the certificate was valid (before 2019-10-25) and restart certmonger. See discussion in this thread: https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html In newer RHEL version (RHEL 7.7) there is a special tool, ipa-cert-fix, that can help with fixing these issues. However, since you are on the version before it, you need to do manual renewal. >stuck: no >key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM>',token='NSS Certificate >DB',pinfile='/etc/httpd/alias/pwdfile.txt' >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM>',token='NSS Certificate DB' >CA: IPA >issuer: CN=Certificate Authority,O=xxx.xxxx.COM <http://xxx.xxxx.COM> >subject: CN=ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> >expires: 2019-10-25 20:16:38 UTC >principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> >key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >eku: id-kp-serverAuth,id-pkinit-KPKdc >pre-save command: >post-save command: /usr/libexec/ipa/certmonger/restart_httpd >track: yes > > > > >*Issue2:* > >We are getting this alert while we log in to UI in httpd error logs > > >#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while getting >initial credentials > >and PKINIT was disabled > >[root@ipa2 httpd]# ipa-pkinit-manage status >PKINIT is disabled > >While I tried to enable this > >[root@ipa2 httpd]# ipa-pkinit-manage enable >Configuring Kerberos KDC (krb5kdc) > [1/1]: installing X509 Certificate for PKINIT > >the process was getting stuck, so I had to terminate it manually. After >trying to enable, I'm getting "Login failed due to an unknown reason." >error in web UI when I try to login > >*Error in httpd:* > >[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] mod_wsgi (pid=24416): Exception occurred processing WSGI >script '/usr/share/ipa/wsgi.py'. >[Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] Traceback (most recent call last): >[Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File "/usr/share/ipa/wsgi.py", line 59, in application >[Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return api.Backend.wsgi_dispatch(environ, >start_response) >[Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in >__call__ >[Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return self.route(environ, start_response) >[Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in >route >[Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return app(environ, start_response) >[Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in >__call__ >[Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] self.kinit(user_principal, password, ipa_ccache_name) >[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in >kinit >[Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] pkinit_anchors=[paths.KDC_CERT, >paths.KDC_CA_BUNDLE_PEM], >[Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in >kinit_armor >[Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] run(args, env=env, raiseonerr=True, capture_error=True) >[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run >[Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] raise CalledProcessError(p.returncode, arg_string, >str(output)) >[Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] CalledProcessError: Command '/usr/bin/kinit -n -c >/var/run/ipa/ccaches/armor_24416 -X >X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X >X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned >non-zero exit status 1 > >And when I try to list the certificates using *getcert list,* there is a >new cert which was added > >Request ID '20191106100258': >status: CA_UNREACHABLE >ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request, will >retry: 907 (RPC failed at server. cannot connect to ' >https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: >CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). >stuck: no >key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' >certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' >CA: IPA >issuer: >subject: >expires: unknown >pre-save command: >post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert >track: yes >auto-renew: yes > > >Regards >Nikita S > >When >On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy <abokovoy@redhat.com <mailto:abokovoy@redhat.com>> >wrote: > >> On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote: >> >Hi Team, >> >We have 2 IPA servers in Mater-Master setup are we facing the below issue >> >on these servers. >> > >> >Isuue1: >> >Our httpd certificate has expired because of which our IPA1 UI wasn't >> >working, we are getting “*loging failed due to an unknown reason*” error >> >while we log in to the UI >> > >> > >> >1. First, the IPA console was not working as httpd service was stopped, >> >httpd was not starting as HTTP certificate is expired. Added >> >*NSSEnforceValidCerts >> >off* line in nss.conf to start the service. >> > >> >2. After the change IPA console was loading we are not able to login to >> the >> >console as pki-tomcatd service was not running, >> >[root@ipa1 ca]# ipactl status >> >Directory Service: RUNNING >> >krb5kdc Service: RUNNING >> >kadmin Service: RUNNING >> >httpd Service: RUNNING >> >ipa-custodia Service: RUNNING >> >ntpd Service: RUNNING >> >pki-tomcatd Service: STOPPED >> >ipa-otpd Service: RUNNING >> > >> ># systemctl status pki-tomcatd@pki-tomcat.service -l >> >● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat >> > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; >> >vendor preset: disabled) >> > Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago >> > Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, >> >status=0/SUCCESS) >> > Main PID: 97233 (java) >> > CGroup: >> >/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service >> > └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java >> >-DRESTEASY_LIB=/usr/share/java/resteasy-base >> >-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath >> >> >/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar >> >-Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat >> >-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp >> >> >-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties >> >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> >-Djava.security.manager >> >-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy >> >org.apache.catalina.startup.Bootstrap start >> > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: WARNING: Exception >> >processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background >> >process >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: >> >javax.ws.rs <http://javax.ws.rs>.ServiceUnavailableException: Subsystem unavailable >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >> >org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >> >org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >> >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >> >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >> >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >> >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> server[97233]: at >> >java.lang.Thread.run(Thread.java:748) >> > >> > >> >This service wasn’t starting with this error >> > >> ># less /var/log/pki/pki-tomcat/ca/debug >> >31/Oct/2019:13:24:23][localhost-startStop-1]: >> >SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert >> >cert-pki-ca >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: >> >SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened >> >Could not connect to LDAP server host ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> port 636 Error >> >netscape.ldap.LDAPException: Authentication failed (49) >> >> Authentication failed means the RA agent certificate dogtag uses to >> authenticate to LDAP server is not the same as the one mentioned in the >> LDAP entry for RA agent. >> >> I think there was some procedure to fix it but I don't have links handy. >> Also, you did not specify what versions of FreeIPA you run. >> >> >> >at >> >> >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) >> > at >> >> >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) >> > at >> >> >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) >> > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) >> > at >> >com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) >> > at >> >com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) >> > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) >> > at >> >> >com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >> > at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> >> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> >> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >> >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> > at >> >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> > at >> >org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> > at >> >> >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> > at >> >> >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> > at >> >> >org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) >> > at >> >> >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) >> > at >> >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) >> > at >> >> >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) >> > at >> >> >org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) >> > at >> >org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> > at >> >> >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >> >org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> > at >> >> >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> > at >> >> >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at >> >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> > at >> >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> > at >> >> >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> > at >> >> >org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> > at >> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > at >> >> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> >> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at java.lang.Thread.run(Thread.java:748) >> >Internal Database Error encountered: Could not connect to LDAP server host >> >ipa1.xxx.xxx.com <http://ipa1.xxx.xxx.com> port 636 Error netscape.ldap.LDAPException: >> Authentication >> >failed (49) >> > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) >> > at >> >com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) >> > at >> >com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) >> > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) >> > at >> >> >com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >> > at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> >> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> >> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >> >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> > at >> >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> > at >> >org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> > at >> >> >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> > at >> >> >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> > at >> >> >org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) >> > at >> >> >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) >> > at >> >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) >> > at >> >> >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) >> > at >> >> >org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) >> > at >> >org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> > at >> >> >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >> >org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> > at >> >> >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> > at >> >> >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at >> >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> > at >> >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> > at >> >> >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> > at >> >> >org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> > at >> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > at >> >> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> >> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at java.lang.Thread.run(Thread.java:748) >> > >> ># getcert list >> >Request ID '20180412150739': >> >status: SUBMITTING >> >stuck: no >> >key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM>',token='NSS Certificate >> >DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM>',token='NSS Certificate DB' >> >CA: IPA >> >issuer: CN=Certificate Authority,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> >> >subject: CN=ipa1.xxxx.xxxx.com <http://ipa1.xxxx.xxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> >> >expires: 2019-10-25 20:16:38 UTC >> >principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> >> >key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> >eku: id-kp-serverAuth,id-pkinit-KPKdc >> >pre-save command: >> >post-save command: /usr/libexec/ipa/certmonger/restart_httpd >> >track: yes >> >auto-renew: yes >> >> Status SUBMITTING means the renewal is not yet completed. It will not >> complete until you get Dogtag working. >> >> > >> >Issue2: >> > >> >On the IPA2 server, we are unable to login with the admin user credentials >> >without OTP, but when an AD user is trying to login with 2FA (i.e, >> >password and OTP) we are getting this error *"The password you entered is >> >incorrect."* >> >> AD users cannot use multifactor authentication defined in IPA. >> >> >> ># [root@ipa2 log]# ipactl status >> >Directory Service: RUNNING >> >krb5kdc Service: RUNNING >> >kadmin Service: RUNNING >> >httpd Service: RUNNING >> >ipa-custodia Service: RUNNING >> >ntpd Service: RUNNING >> >ipa-otpd Service: STOPPED >> >ipa: INFO: The ipactl command was successful >> > >> ># systemctl status ipa-otpd.socket -l >> >● ipa-otpd.socket - ipa-otpd socket >> > Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; >> >vendor preset: disabled) >> > Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; >> 1h >> >31min ago >> > Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) >> > Accepted: 2; Connected: 0 >> > >> >Nov 05 07:42:53 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: Listening on ipa-otpd >> socket. >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: ipa-otpd.socket failed to >> >queue service startup job (Maybe the service file is missing or not a >> >template unit?): Resource temporarily unavailable >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: Unit ipa-otpd.socket >> entered >> >failed state. >> > >> ># cat /usr/lib/systemd/system/ipa-otpd.socket >> >[Unit] >> >Description=ipa-otpd socket >> > >> >[Socket] >> >ListenStream=/var/run/krb5kdc/DEFAULT.socket >> >RemoveOnStop=true >> >SocketMode=0600 >> >Accept=true >> > >> >[Install] >> >WantedBy=krb5kdc.service >> > >> > >> > >> >We see that data replication is broken between the 2 IPA servers, as the >> >changes made on IPA2 is not reflecting on IPA1 >> This is most likely because your LDAP server certificate expired as >> well. >> >> >> >We the below errors as well. >> > >> >IPA1 >> >Nov 05 10:09:23 ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> krb5kdc[28021](info): TGS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> for ldap/ >> >ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> >> >Nov 05 10:14:24 ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> krb5kdc[28021](info): TGS_REQ (8 >> >etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, >> >etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> for >> >ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> >> >> These aren't errors. They are normal operations: ldap/ipa1 service (LDAP >> server on IPA1) asked for a Kerberos service ticket to LDAP service on >> IPA2 and was granted it. This is just as it should be for replication. >> >> > >> >IPA2 >> ># tailf krb5kdc.log >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): AS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>, >> >Additional pre-authentication required >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): AS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> for krbtgt/ >> >xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): TGS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> for ldap/ >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >> Same here. LDAP server on IPA2 operated against itself here. >> >> -- >> / Alexander Bokovoy >> Sr. Principal Software Engineer >> Security / Identity Management Engineering >> Red Hat Limited, Finland >> >> -- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thanks, Florence and Alexander.
@Alexander Bokovoy abokovoy@redhat.com ,
While we were debugging in one of a blog we came across a scenario where NSS DB had got corrupted during a certificate renewal, so wanted to eliminate that issue.
We see all the private key.
[root@ipa1 nikita.d]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS Certificate DB:Server-Cert cert-pki-ca < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS Certificate DB:auditSigningCert cert-pki-ca < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS Certificate DB:ocspSigningCert cert-pki-ca
We will try renewing the HTTP cert in our environment, will let you know once the cert is updated successfully.
https://access.redhat.com/solutions/3357261
Regards Nikita S
On Fri, Nov 8, 2019 at 3:42 PM Florence Blanc-Renaud flo@redhat.com wrote:
On 11/7/19 11:16 AM, Nikita Deeksha via FreeIPA-users wrote:
Thanks for the update Alexander will check this and get back to you, wanted to check on another thing as well.
Can you please help us to understand this error that we see for the cert in pki
[root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias -L | grep cert-pki-ca | awk'{print $1}');do certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
If you use "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt" (without the -n alias option), you will see all the keys present in the database and notice that some of them have a prefixed nickname, for instance: 'NSS Certificate DB:Server-Cert cert-pki-ca' instead of 'Server-Cert cert-pki-ca'. You need to provide this prefixed name with -K -n nickname.
HTH, flo
These are the cert which is present /etc/pki/pki-tomcat/alias
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
COMODO CA BUNDLE CT,C,C ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u
Regards
Nikita S
On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy <abokovoy@redhat.com mailto:abokovoy@redhat.com> wrote:
On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote: >2. > >Status SUBMITTING means the renewal is not yet completed. It will
not
>complete until you get Dogtag working. > >But now the status says CA_UNEACHABLE > >Request ID '20180412150739': >status: CA_UNREACHABLE >ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request, will >retry: -504 (libcurl failed to execute the HTTP POST transaction, >explaining: Peer's Certificate has expired.). This is exactly an issue with expired HTTP certificate. I guess you'd need to roll back time to when the certificate was
valid
(before 2019-10-25) and restart certmonger. See discussion in this thread:
https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html
In newer RHEL version (RHEL 7.7) there is a special tool,
ipa-cert-fix,
that can help with fixing these issues. However, since you are on the version before it, you need to do manual renewal. >stuck: no >key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=
>ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM>',token='NSS Certificate >DB',pinfile='/etc/httpd/alias/pwdfile.txt' >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM>',token='NSS Certificate DB' >CA: IPA >issuer: CN=Certificate Authority,O=xxx.xxxx.COM <
>subject: CN=ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> >expires: 2019-10-25 20:16:38 UTC >principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> >key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >eku: id-kp-serverAuth,id-pkinit-KPKdc >pre-save command: >post-save command: /usr/libexec/ipa/certmonger/restart_httpd >track: yes > > > > >*Issue2:* > >We are getting this alert while we log in to UI in httpd error logs > > >#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while getting >initial credentials > >and PKINIT was disabled > >[root@ipa2 httpd]# ipa-pkinit-manage status >PKINIT is disabled > >While I tried to enable this > >[root@ipa2 httpd]# ipa-pkinit-manage enable >Configuring Kerberos KDC (krb5kdc) > [1/1]: installing X509 Certificate for PKINIT > >the process was getting stuck, so I had to terminate it manually. After >trying to enable, I'm getting "Login failed due to an unknown
reason."
>error in web UI when I try to login > >*Error in httpd:* > >[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] mod_wsgi (pid=24416): Exception occurred processing WSGI >script '/usr/share/ipa/wsgi.py'. >[Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] Traceback (most recent call last): >[Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File "/usr/share/ipa/wsgi.py", line 59, in application >[Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return api.Backend.wsgi_dispatch(environ, >start_response) >[Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in >__call__ >[Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return self.route(environ, start_response) >[Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in >route >[Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return app(environ, start_response) >[Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in >__call__ >[Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] self.kinit(user_principal, password, ipa_ccache_name) >[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in >kinit >[Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] pkinit_anchors=[paths.KDC_CERT, >paths.KDC_CA_BUNDLE_PEM], >[Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in >kinit_armor >[Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] run(args, env=env, raiseonerr=True, capture_error=True) >[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run >[Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] raise CalledProcessError(p.returncode, arg_string, >str(output)) >[Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] CalledProcessError: Command '/usr/bin/kinit -n -c >/var/run/ipa/ccaches/armor_24416 -X >X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X >X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
returned
>non-zero exit status 1 > >And when I try to list the certificates using *getcert list,* there is a >new cert which was added > >Request ID '20191106100258': >status: CA_UNREACHABLE >ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request, will >retry: 907 (RPC failed at server. cannot connect to ' >https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: >CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). >stuck: no >key pair storage:
type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' >CA: IPA >issuer: >subject: >expires: unknown >pre-save command: >post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert >track: yes >auto-renew: yes > > >Regards >Nikita S > >When >On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy <abokovoy@redhat.com <mailto:abokovoy@redhat.com>> >wrote: > >> On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote: >> >Hi Team, >> >We have 2 IPA servers in Mater-Master setup are we facing the below issue >> >on these servers. >> > >> >Isuue1: >> >Our httpd certificate has expired because of which our IPA1 UI wasn't >> >working, we are getting “*loging failed due to an unknown reason*” error >> >while we log in to the UI >> > >> > >> >1. First, the IPA console was not working as httpd service was stopped, >> >httpd was not starting as HTTP certificate is expired. Added >> >*NSSEnforceValidCerts >> >off* line in nss.conf to start the service. >> > >> >2. After the change IPA console was loading we are not able to login to >> the >> >console as pki-tomcatd service was not running, >> >[root@ipa1 ca]# ipactl status >> >Directory Service: RUNNING >> >krb5kdc Service: RUNNING >> >kadmin Service: RUNNING >> >httpd Service: RUNNING >> >ipa-custodia Service: RUNNING >> >ntpd Service: RUNNING >> >pki-tomcatd Service: STOPPED >> >ipa-otpd Service: RUNNING >> > >> ># systemctl status pki-tomcatd@pki-tomcat.service -l >> >● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat >> > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; >> >vendor preset: disabled) >> > Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago >> > Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, >> >status=0/SUCCESS) >> > Main PID: 97233 (java) >> > CGroup: >>
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service >> > └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java >> >-DRESTEASY_LIB=/usr/share/java/resteasy-base >> >-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath >> >>
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar >> >-Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat >> >-Djava.endorsed.dirs=
-Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
>> >>
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> >-Djava.security.manager >>
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy >> >org.apache.catalina.startup.Bootstrap start >> > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception >> >processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background >> >process >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: >> >javax.ws.rs http://javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >java.lang.Thread.run(Thread.java:748) >> > >> > >> >This service wasn’t starting with this error >> > >> ># less /var/log/pki/pki-tomcat/ca/debug >> >31/Oct/2019:13:24:23][localhost-startStop-1]: >> >SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert >> >cert-pki-ca >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: >> >SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened >> >Could not connect to LDAP server host ipa1.xxx.xxxx.com http://ipa1.xxx.xxxx.com port 636 Error >> >netscape.ldap.LDAPException: Authentication failed (49) >> >> Authentication failed means the RA agent certificate dogtag uses
to
>> authenticate to LDAP server is not the same as the one mentioned in the >> LDAP entry for RA agent. >> >> I think there was some procedure to fix it but I don't have links handy. >> Also, you did not specify what versions of FreeIPA you run. >> >> >> >at >> >>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) >> > at >> >>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) >> > at >> >>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) >> > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) >> > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) >> > at >> >>
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >> > at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> >>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> >>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> > at >>
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> > at >> >>
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) >> > at >> >>
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) >> > at >>
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) >> > at >> >>
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) >> > at >> >>
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) >> > at >>
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> > at >> >>
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >>
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at >>
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> > at >>
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> > at >> >>
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> > at >> >>
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> > at >>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at >> >>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> >>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at java.lang.Thread.run(Thread.java:748) >> >Internal Database Error encountered: Could not connect to LDAP server host >> >ipa1.xxx.xxx.com http://ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException: >> Authentication >> >failed (49) >> > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) >> > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) >> > at >> >>
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >> > at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> >>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> >>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> > at >>
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> > at >> >>
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) >> > at >> >>
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) >> > at >>
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) >> > at >> >>
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) >> > at >> >>
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) >> > at >>
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> > at >> >>
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >>
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at >>
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> > at >>
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> > at >> >>
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> > at >> >>
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> > at >>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at >> >>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> >>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at java.lang.Thread.run(Thread.java:748) >> > >> ># getcert list >> >Request ID '20180412150739': >> >status: SUBMITTING >> >stuck: no >> >key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >> >ipa1.xxxx.xxxxx.com http://ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM http://xxx.xxxx.COM',token='NSS Certificate >> >DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> >certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=
>> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM>',token='NSS Certificate DB' >> >CA: IPA >> >issuer: CN=Certificate Authority,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> >> >subject: CN=ipa1.xxxx.xxxx.com <http://ipa1.xxxx.xxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> >> >expires: 2019-10-25 20:16:38 UTC >> >principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> >> >key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> >eku: id-kp-serverAuth,id-pkinit-KPKdc >> >pre-save command: >> >post-save command: /usr/libexec/ipa/certmonger/restart_httpd >> >track: yes >> >auto-renew: yes >> >> Status SUBMITTING means the renewal is not yet completed. It will not >> complete until you get Dogtag working. >> >> > >> >Issue2: >> > >> >On the IPA2 server, we are unable to login with the admin user credentials >> >without OTP, but when an AD user is trying to login with 2FA
(i.e,
>> >password and OTP) we are getting this error *"The password you entered is >> >incorrect."* >> >> AD users cannot use multifactor authentication defined in IPA. >> >> >> ># [root@ipa2 log]# ipactl status >> >Directory Service: RUNNING >> >krb5kdc Service: RUNNING >> >kadmin Service: RUNNING >> >httpd Service: RUNNING >> >ipa-custodia Service: RUNNING >> >ntpd Service: RUNNING >> >ipa-otpd Service: STOPPED >> >ipa: INFO: The ipactl command was successful >> > >> ># systemctl status ipa-otpd.socket -l >> >● ipa-otpd.socket - ipa-otpd socket >> > Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; >> >vendor preset: disabled) >> > Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; >> 1h >> >31min ago >> > Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) >> > Accepted: 2; Connected: 0 >> > >> >Nov 05 07:42:53 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: Listening on ipa-otpd >> socket. >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: ipa-otpd.socket failed to >> >queue service startup job (Maybe the service file is missing or not a >> >template unit?): Resource temporarily unavailable >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: Unit ipa-otpd.socket >> entered >> >failed state. >> > >> ># cat /usr/lib/systemd/system/ipa-otpd.socket >> >[Unit] >> >Description=ipa-otpd socket >> > >> >[Socket] >> >ListenStream=/var/run/krb5kdc/DEFAULT.socket >> >RemoveOnStop=true >> >SocketMode=0600 >> >Accept=true >> > >> >[Install] >> >WantedBy=krb5kdc.service >> > >> > >> > >> >We see that data replication is broken between the 2 IPA servers, as the >> >changes made on IPA2 is not reflecting on IPA1 >> This is most likely because your LDAP server certificate expired
as
>> well. >> >> >> >We the below errors as well. >> > >> >IPA1 >> >Nov 05 10:09:23 ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> krb5kdc[28021](info): TGS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> for ldap/ >> >ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> >> >Nov 05 10:14:24 ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> krb5kdc[28021](info): TGS_REQ (8 >> >etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, >> >etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> for >> >ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> >> >> These aren't errors. They are normal operations: ldap/ipa1 service (LDAP >> server on IPA1) asked for a Kerberos service ticket to LDAP service on >> IPA2 and was granted it. This is just as it should be for replication. >> >> > >> >IPA2 >> ># tailf krb5kdc.log >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): AS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:
xxxx.xxxx.COM@xxxx.xxxx.COM>,
>> >Additional pre-authentication required >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): AS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> for krbtgt/ >> >xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): TGS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> for ldap/ >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >> Same here. LDAP server on IPA2 operated against itself here. >> >> -- >> / Alexander Bokovoy >> Sr. Principal Software Engineer >> Security / Identity Management Engineering >> Red Hat Limited, Finland >> >> -- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Alexander/Florence,
While we were trying to renew the certificate the httpd cert, we went back in time where httpd cert was vaild and then tried to renew the cert, but in *Step7: Test CA operation* is failing with the below error. Can you please help us with this?
https://access.redhat.com/solutions/3357261
[root@ipa1 ~]# curl --cacert /etc/ipa/ca.crt -v https://%60hostname%60:8443/ca/ee/ca/getCertChain
* About to connect() to ipa1.corp.endurance.com port 8443 (#0) * Trying 172.27.152.34... * Connected to ipa1.corp.endurance.com (172.27.152.34) port 8443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/ipa/ca.crt CApath: none * NSS: client certificate not found (nickname not specified) * SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA * Server certificate: * subject: CN=ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM * start date: Sep 17 03:26:18 2019 GMT * expire date: Sep 06 03:26:18 2021 GMT * common name: ipa1.corp.endurance.com * issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM
GET /ca/ee/ca/getCertChain HTTP/1.1 User-Agent: curl/7.29.0 Host: ipa1.corp.endurance.com:8443 Accept: */*
< HTTP/1.1 500 Internal Server Error < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 2208 < Date: Thu, 24 Oct 2019 16:31:10 GMT < Connection: close < <html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Thread.java:748) </pre></p><p><b>note</b> <u>The ful* Closing connection 0 l stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>[root@ipa1 ~]#
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /var/log/pki/pki-tomcat/ca/debug shows this error
Could not connect to LDAP server host ipa1.corp.endurance.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.corp.endurance.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [24/Oct/2019:16:29:50][localhost-startStop-1]: CMS.start(): shutdown server [24/Oct/2019:16:29:50][localhost-startStop-1]: CMSEngine.shutdown()
------------------------------------------------------------------------------------------- sybsystemCert is still valid.
Request ID '20171024201555': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM subject: CN=CA Subsystem,O=CORP.ENDURANCE.COM expires: 2021-09-06 03:25:48 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes ------------------------------------------------------------- [root@ipa1 ~]# cat /etc/pki/pki-tomcat/ca/CS.cfg| grep subsystem.cert ca.cert.subsystem.certusage=SSLClient ca.subsystem.cert=MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS ca.subsystem.certreq=MIICeTCCAWECAQAwNDEbMBkGA1UECgwSQ09SUC5FTkRVUkFOQ0UuQ09NMRUwEwYDVQQDDAxDQSBTdWJzeXN0ZW0wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDC6Jw/rZmIkFNWoT7rMx2rkcceO0YJ8lPPyqoHnwXt59FFaBtgcZLCyNSI49A6SdmVVmy6hwthT7MrEOYnXDsikBbhb/hEz983ivEoy8Dg8/gtyKSV3XZ4A6G56mM7OBp627ZDkMXutkSlO+JkuJRNAkwMzzBAz5xkFWncs9aLwP8JWJdApz9KN5bNYvqxA1PGdaElHQ6O+muYp4kHqgAQ39PlwuqvmvGc0KyfYy1dtABerWPBG4diSlvAPrg35oAptgZHTdUdZaA/UxJtunddha50zfiHT5yVl5N6iY1JNW7G8TSf0kKkAal5IMcgZua9Mbn470u7t1lN79QdLnidAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEABl3uxfntEZZFeoB1eIjd774MSxK2itaj6B1VyXV0qxcp/SHF1sfJnUHjDpNIB0J/Bpk/sc/+Y2Gxar3zcwyORjF+ojCBc8lP6QZ7mB9H3m9XEuA+H1qmgGz+h2FhnQy5/EmgJnTAn8fNpZCQwQmSZLtnbmd++yK9y+YXCAvaHx+Wsz5c/GhyNxoHGADcmir+iFVltO3jpQ06ThVBb0bD4D6ZzaimkM1vnAqseKdqXvLJI1/ZfIVj6QDcdnnTryt24zOZoxzeOcc2X5HKOtR/5pXilmCS/7zpujXTd1iZi8nfRkpurUdMgC7FLK+B3Dq03G61RPJUwJZvsKlXbwBghg== [root@ipa1 ~]# cat /tmp/subsystemCert\ cert-pki-ca.pem MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS
I had a few queries on this
1. Is there any way we take the backup of the current setup and create a new IPA instance without changing any parameter in the client servers 2. If we upgrade the IPA version we can fix this issue?
Regards Nikita S
On Fri, Nov 8, 2019 at 4:29 PM Nikita Deeksha nikita.d@endurance.com wrote:
Thanks, Florence and Alexander.
@Alexander Bokovoy abokovoy@redhat.com ,
While we were debugging in one of a blog we came across a scenario where NSS DB had got corrupted during a certificate renewal, so wanted to eliminate that issue.
We see all the private key.
[root@ipa1 nikita.d]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS Certificate DB:Server-Cert cert-pki-ca < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS Certificate DB:auditSigningCert cert-pki-ca < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS Certificate DB:ocspSigningCert cert-pki-ca
We will try renewing the HTTP cert in our environment, will let you know once the cert is updated successfully.
https://access.redhat.com/solutions/3357261
Regards Nikita S
On Fri, Nov 8, 2019 at 3:42 PM Florence Blanc-Renaud flo@redhat.com wrote:
On 11/7/19 11:16 AM, Nikita Deeksha via FreeIPA-users wrote:
Thanks for the update Alexander will check this and get back to you, wanted to check on another thing as well.
Can you please help us to understand this error that we see for the
cert
in pki
[root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias -L | grep cert-pki-ca | awk'{print $1}');do certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
Unrecognized
Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
Unrecognized
Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
Unrecognized
Object Identifier.
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
Unrecognized
Object Identifier.
Hi,
If you use "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt" (without the -n alias option), you will see all the keys present in the database and notice that some of them have a prefixed nickname, for instance: 'NSS Certificate DB:Server-Cert cert-pki-ca' instead of 'Server-Cert cert-pki-ca'. You need to provide this prefixed name with -K -n nickname.
HTH, flo
These are the cert which is present /etc/pki/pki-tomcat/alias
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
COMODO CA BUNDLE CT,C,C ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u
Regards
Nikita S
On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy <abokovoy@redhat.com mailto:abokovoy@redhat.com> wrote:
On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote: >2. > >Status SUBMITTING means the renewal is not yet completed. It will
not
>complete until you get Dogtag working. > >But now the status says CA_UNEACHABLE > >Request ID '20180412150739': >status: CA_UNREACHABLE >ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed request, will >retry: -504 (libcurl failed to execute the HTTP POST transaction, >explaining: Peer's Certificate has expired.). This is exactly an issue with expired HTTP certificate. I guess you'd need to roll back time to when the certificate was
valid
(before 2019-10-25) and restart certmonger. See discussion in this thread:
https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html
In newer RHEL version (RHEL 7.7) there is a special tool,
ipa-cert-fix,
that can help with fixing these issues. However, since you are on
the
version before it, you need to do manual renewal. >stuck: no >key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=
>ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM>',token='NSS Certificate >DB',pinfile='/etc/httpd/alias/pwdfile.txt' >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM>',token='NSS Certificate DB' >CA: IPA >issuer: CN=Certificate Authority,O=xxx.xxxx.COM <
>subject: CN=ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> >expires: 2019-10-25 20:16:38 UTC >principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> >key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >eku: id-kp-serverAuth,id-pkinit-KPKdc >pre-save command: >post-save command: /usr/libexec/ipa/certmonger/restart_httpd >track: yes > > > > >*Issue2:* > >We are getting this alert while we log in to UI in httpd error
logs
> > >#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed while getting >initial credentials > >and PKINIT was disabled > >[root@ipa2 httpd]# ipa-pkinit-manage status >PKINIT is disabled > >While I tried to enable this > >[root@ipa2 httpd]# ipa-pkinit-manage enable >Configuring Kerberos KDC (krb5kdc) > [1/1]: installing X509 Certificate for PKINIT > >the process was getting stuck, so I had to terminate it manually. After >trying to enable, I'm getting "Login failed due to an unknown
reason."
>error in web UI when I try to login > >*Error in httpd:* > >[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] mod_wsgi (pid=24416): Exception occurred processing WSGI >script '/usr/share/ipa/wsgi.py'. >[Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] Traceback (most recent call last): >[Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File "/usr/share/ipa/wsgi.py", line 59, in application >[Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return api.Backend.wsgi_dispatch(environ, >start_response) >[Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in >__call__ >[Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return self.route(environ, start_response) >[Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in >route >[Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] return app(environ, start_response) >[Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in >__call__ >[Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] self.kinit(user_principal, password, ipa_ccache_name) >[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in >kinit >[Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] pkinit_anchors=[paths.KDC_CERT, >paths.KDC_CA_BUNDLE_PEM], >[Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in >kinit_armor >[Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] run(args, env=env, raiseonerr=True, capture_error=True) >[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] File >"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run >[Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] raise CalledProcessError(p.returncode, arg_string, >str(output)) >[Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote >172.27.10.113:0 <http://172.27.10.113:0>] CalledProcessError: Command '/usr/bin/kinit -n -c >/var/run/ipa/ccaches/armor_24416 -X >X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X >X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
returned
>non-zero exit status 1 > >And when I try to list the certificates using *getcert list,* there is a >new cert which was added > >Request ID '20191106100258': >status: CA_UNREACHABLE >ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed request, will >retry: 907 (RPC failed at server. cannot connect to ' >https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: >CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:618)).
>stuck: no >key pair storage:
type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' >CA: IPA >issuer: >subject: >expires: unknown >pre-save command: >post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert >track: yes >auto-renew: yes > > >Regards >Nikita S > >When >On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy <abokovoy@redhat.com <mailto:abokovoy@redhat.com>> >wrote: > >> On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote: >> >Hi Team, >> >We have 2 IPA servers in Mater-Master setup are we facing the below issue >> >on these servers. >> > >> >Isuue1: >> >Our httpd certificate has expired because of which our IPA1 UI wasn't >> >working, we are getting “*loging failed due to an unknown reason*” error >> >while we log in to the UI >> > >> > >> >1. First, the IPA console was not working as httpd service was stopped, >> >httpd was not starting as HTTP certificate is expired. Added >> >*NSSEnforceValidCerts >> >off* line in nss.conf to start the service. >> > >> >2. After the change IPA console was loading we are not able to login to >> the >> >console as pki-tomcatd service was not running, >> >[root@ipa1 ca]# ipactl status >> >Directory Service: RUNNING >> >krb5kdc Service: RUNNING >> >kadmin Service: RUNNING >> >httpd Service: RUNNING >> >ipa-custodia Service: RUNNING >> >ntpd Service: RUNNING >> >pki-tomcatd Service: STOPPED >> >ipa-otpd Service: RUNNING >> > >> ># systemctl status pki-tomcatd@pki-tomcat.service -l >> >● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server
pki-tomcat
>> > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; >> >vendor preset: disabled) >> > Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago >> > Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, >> >status=0/SUCCESS) >> > Main PID: 97233 (java) >> > CGroup: >>
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service >> > └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java >> >-DRESTEASY_LIB=/usr/share/java/resteasy-base >> >-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath >> >>
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar >> >-Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat >> >-Djava.endorsed.dirs=
-Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
>> >>
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> >-Djava.security.manager >>
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy >> >org.apache.catalina.startup.Bootstrap start >> > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception >> >processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background >> >process >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: >> >javax.ws.rs http://javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com server[97233]: at >> >java.lang.Thread.run(Thread.java:748) >> > >> > >> >This service wasn’t starting with this error >> > >> ># less /var/log/pki/pki-tomcat/ca/debug >> >31/Oct/2019:13:24:23][localhost-startStop-1]: >> >SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert >> >cert-pki-ca >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: >> >SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened >> >Could not connect to LDAP server host ipa1.xxx.xxxx.com http://ipa1.xxx.xxxx.com port 636 Error >> >netscape.ldap.LDAPException: Authentication failed (49) >> >> Authentication failed means the RA agent certificate dogtag
uses to
>> authenticate to LDAP server is not the same as the one mentioned in the >> LDAP entry for RA agent. >> >> I think there was some procedure to fix it but I don't have links handy. >> Also, you did not specify what versions of FreeIPA you run. >> >> >> >at >> >>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) >> > at >> >>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) >> > at >> >>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) >> > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) >> > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) >> > at >> >>
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >> > at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> >>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> >>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> > at >>
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> > at >> >>
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) >> > at >> >>
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) >> > at >>
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) >> > at >> >>
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) >> > at >> >>
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) >> > at >>
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> > at >> >>
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >>
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at >>
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> > at >>
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> > at >> >>
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> > at >> >>
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> > at >>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at >> >>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> >>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at java.lang.Thread.run(Thread.java:748) >> >Internal Database Error encountered: Could not connect to LDAP server host >> >ipa1.xxx.xxx.com http://ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException: >> Authentication >> >failed (49) >> > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) >> > at >>
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) >> > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) >> > at >> >>
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) >> > at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> >>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> >>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> > at >>
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> > at >>
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> > at >> >>
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> > at >> >>
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) >> > at >> >>
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) >> > at >>
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) >> > at >> >>
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) >> > at >> >>
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) >> > at >>
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) >> > at >> >>
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >>
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> > at >> >>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at >>
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> > at >>
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> > at >> >>
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> > at >> >>
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> > at >>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at >> >>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> > at >> >>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> > at java.lang.Thread.run(Thread.java:748) >> > >> ># getcert list >> >Request ID '20180412150739': >> >status: SUBMITTING >> >stuck: no >> >key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= >> >ipa1.xxxx.xxxxx.com http://ipa1.xxxx.xxxxx.com,O=
xxx.xxxx.COM
<http://xxx.xxxx.COM>',token='NSS Certificate >> >DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> >certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=
>> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM>',token='NSS Certificate DB' >> >CA: IPA >> >issuer: CN=Certificate Authority,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> >> >subject: CN=ipa1.xxxx.xxxx.com <http://ipa1.xxxx.xxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> >> >expires: 2019-10-25 20:16:38 UTC >> >principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> >> >key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> >eku: id-kp-serverAuth,id-pkinit-KPKdc >> >pre-save command: >> >post-save command: /usr/libexec/ipa/certmonger/restart_httpd >> >track: yes >> >auto-renew: yes >> >> Status SUBMITTING means the renewal is not yet completed. It will not >> complete until you get Dogtag working. >> >> > >> >Issue2: >> > >> >On the IPA2 server, we are unable to login with the admin user credentials >> >without OTP, but when an AD user is trying to login with 2FA
(i.e,
>> >password and OTP) we are getting this error *"The password you entered is >> >incorrect."* >> >> AD users cannot use multifactor authentication defined in IPA. >> >> >> ># [root@ipa2 log]# ipactl status >> >Directory Service: RUNNING >> >krb5kdc Service: RUNNING >> >kadmin Service: RUNNING >> >httpd Service: RUNNING >> >ipa-custodia Service: RUNNING >> >ntpd Service: RUNNING >> >ipa-otpd Service: STOPPED >> >ipa: INFO: The ipactl command was successful >> > >> ># systemctl status ipa-otpd.socket -l >> >● ipa-otpd.socket - ipa-otpd socket >> > Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; >> >vendor preset: disabled) >> > Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; >> 1h >> >31min ago >> > Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) >> > Accepted: 2; Connected: 0 >> > >> >Nov 05 07:42:53 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: Listening on ipa-otpd >> socket. >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: ipa-otpd.socket failed to >> >queue service startup job (Maybe the service file is missing or not a >> >template unit?): Resource temporarily unavailable >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> systemd[1]: Unit ipa-otpd.socket >> entered >> >failed state. >> > >> ># cat /usr/lib/systemd/system/ipa-otpd.socket >> >[Unit] >> >Description=ipa-otpd socket >> > >> >[Socket] >> >ListenStream=/var/run/krb5kdc/DEFAULT.socket >> >RemoveOnStop=true >> >SocketMode=0600 >> >Accept=true >> > >> >[Install] >> >WantedBy=krb5kdc.service >> > >> > >> > >> >We see that data replication is broken between the 2 IPA servers, as the >> >changes made on IPA2 is not reflecting on IPA1 >> This is most likely because your LDAP server certificate
expired as
>> well. >> >> >> >We the below errors as well. >> > >> >IPA1 >> >Nov 05 10:09:23 ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> krb5kdc[28021](info): TGS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes >> >{rep=18 tkt=18 ses=18}, ldap/
ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM
<mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> for ldap/ >> >ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> >> >Nov 05 10:14:24 ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> krb5kdc[28021](info): TGS_REQ (8 >> >etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, >> >etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> for >> >ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> >> >> These aren't errors. They are normal operations: ldap/ipa1 service (LDAP >> server on IPA1) asked for a Kerberos service ticket to LDAP service on >> IPA2 and was granted it. This is just as it should be for replication. >> >> > >> >IPA2 >> ># tailf krb5kdc.log >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): AS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:
xxxx.xxxx.COM@xxxx.xxxx.COM>,
>> >Additional pre-authentication required >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): AS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> for krbtgt/ >> >xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): TGS_REQ (8 etypes >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> for ldap/ >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> krb5kdc[2451](info): closing down fd >> 11 >> >> Same here. LDAP server on IPA2 operated against itself here. >> >> -- >> / Alexander Bokovoy >> Sr. Principal Software Engineer >> Security / Identity Management Engineering >> Red Hat Limited, Finland >> >> -- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 11/8/19 4:33 PM, Nikita Deeksha via FreeIPA-users wrote:
Alexander/Florence,
While we were trying to renew the certificate the httpd cert, we went back in time where httpd cert was vaild and then tried to renew the cert, but in *Step7: Test CA operation* is failing with the below error. Can you please help us with this?
https://access.redhat.com/solutions/3357261
[root@ipa1 ~]# curl --cacert /etc/ipa/ca.crt -v https://%60hostname%60:8443/ca/ee/ca/getCertChain
- About to connect() to ipa1.corp.endurance.com
http://ipa1.corp.endurance.com port 8443 (#0)
- Trying 172.27.152.34...
- Connected to ipa1.corp.endurance.com http://ipa1.corp.endurance.com
(172.27.152.34) port 8443 (#0)
- Initializing NSS with certpath: sql:/etc/pki/nssdb
- CAfile: /etc/ipa/ca.crt
CApath: none
- NSS: client certificate not found (nickname not specified)
- SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
- Server certificate:
- subject: CN=ipa1.corp.endurance.com
http://ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM
- start date: Sep 17 03:26:18 2019 GMT
- expire date: Sep 06 03:26:18 2021 GMT
- common name: ipa1.corp.endurance.com http://ipa1.corp.endurance.com
- issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM
GET /ca/ee/ca/getCertChain HTTP/1.1 User-Agent: curl/7.29.0 Host: ipa1.corp.endurance.com:8443 http://ipa1.corp.endurance.com:8443 Accept: */*
< HTTP/1.1 500 Internal Server Error < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 2208 < Date: Thu, 24 Oct 2019 16:31:10 GMT < Connection: close <
<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Subsystem unavailable means that the CA subsystem is not running.
com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Thread.java:748) </pre></p><p><b>note</b> <u>The ful* Closing connection 0 l stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>[root@ipa1 ~]#
/var/log/pki/pki-tomcat/ca/debug shows this error
Could not connect to LDAP server host ipa1.corp.endurance.com http://ipa1.corp.endurance.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
This happens when the subsystem cert is not accepted as valid authentication from Dogtag to the ldap server.
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.corp.endurance.com http://ipa1.corp.endurance.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [24/Oct/2019:16:29:50][localhost-startStop-1]: CMS.start(): shutdown server [24/Oct/2019:16:29:50][localhost-startStop-1]: CMSEngine.shutdown()
sybsystemCert is still valid.
Which date in the past did you use? If the subsystemcert's Not Before is after this date, the cert is not valid yet and authentication will fail.
Could you post the full output of getcert list? If you have multiple certificates expired, you need to carefully select a date in the past where all the certs are in their validity window (ie after "Not before" and before "Not after").
HTH, flo
Request ID '20171024201555': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM subject: CN=CA Subsystem,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM expires: 2021-09-06 03:25:48 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes
[root@ipa1 ~]# cat /etc/pki/pki-tomcat/ca/CS.cfg| grep subsystem.cert ca.cert.subsystem.certusage=SSLClient ca.subsystem.cert=MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS ca.subsystem.certreq=MIICeTCCAWECAQAwNDEbMBkGA1UECgwSQ09SUC5FTkRVUkFOQ0UuQ09NMRUwEwYDVQQDDAxDQSBTdWJzeXN0ZW0wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDC6Jw/rZmIkFNWoT7rMx2rkcceO0YJ8lPPyqoHnwXt59FFaBtgcZLCyNSI49A6SdmVVmy6hwthT7MrEOYnXDsikBbhb/hEz983ivEoy8Dg8/gtyKSV3XZ4A6G56mM7OBp627ZDkMXutkSlO+JkuJRNAkwMzzBAz5xkFWncs9aLwP8JWJdApz9KN5bNYvqxA1PGdaElHQ6O+muYp4kHqgAQ39PlwuqvmvGc0KyfYy1dtABerWPBG4diSlvAPrg35oAptgZHTdUdZaA/UxJtunddha50zfiHT5yVl5N6iY1JNW7G8TSf0kKkAal5IMcgZua9Mbn470u7t1lN79QdLnidAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEABl3uxfntEZZFeoB1eIjd774MSxK2itaj6B1VyXV0qxcp/SHF1sfJnUHjDpNIB0J/Bpk/sc/+Y2Gxar3zcwyORjF+ojCBc8lP6QZ7mB9H3m9XEuA+H1qmgGz+h2FhnQy5/EmgJnTAn8fNpZCQwQmSZLtnbmd++yK9y+YXCAvaHx+Wsz5c/GhyNxoHGADcmir+iFVltO3jpQ06ThVBb0bD4D6ZzaimkM1vnAqseKdqXvLJI1/ZfIVj6QDcdnnTryt24zOZoxzeOcc2X5HKOtR/5pXilmCS/7zpujXTd1iZi8nfRkpurUdMgC7FLK+B3Dq03G61RPJUwJZvsKlXbwBghg== [root@ipa1 ~]# cat /tmp/subsystemCert\ cert-pki-ca.pem MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS
I had a few queries on this
- Is there any way we take the backup of the current setup and create a
new IPA instance without changing any parameter in the client servers 2. If we upgrade the IPA version we can fix this issue?
Regards Nikita S
On Fri, Nov 8, 2019 at 4:29 PM Nikita Deeksha <nikita.d@endurance.com mailto:nikita.d@endurance.com> wrote:
Thanks, Florence and Alexander. @Alexander Bokovoy <mailto:abokovoy@redhat.com> , While we were debugging in one of a blog we came across a scenario where NSS DB had got corrupted during a certificate renewal, so wanted to eliminate that issue. We see all the private key. [root@ipa1 nikita.d]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS Certificate DB:Server-Cert cert-pki-ca < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS Certificate DB:auditSigningCert cert-pki-ca < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS Certificate DB:ocspSigningCert cert-pki-ca We will try renewing the HTTP cert in our environment, will let you know once the cert is updated successfully. https://access.redhat.com/solutions/3357261 Regards Nikita S On Fri, Nov 8, 2019 at 3:42 PM Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: On 11/7/19 11:16 AM, Nikita Deeksha via FreeIPA-users wrote: > Thanks for the update Alexander will check this and get back to you, > wanted to check on another thing as well. > > Can you please help us to understand this error that we see for the cert > in pki > > [root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias > -L | grep cert-pki-ca | awk'{print $1}');do certutil -K -d > /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > < 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert > cert-pki-ca > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > Hi, If you use "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt" (without the -n alias option), you will see all the keys present in the database and notice that some of them have a prefixed nickname, for instance: 'NSS Certificate DB:Server-Cert cert-pki-ca' instead of 'Server-Cert cert-pki-ca'. You need to provide this prefixed name with -K -n nickname. HTH, flo > > These are the cert which is present /etc/pki/pki-tomcat/alias > > [root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias > > Certificate Nickname Trust > Attributes > > SSL,S/MIME,JAR/XPI > > COMODO CA BUNDLE CT,C,C > ocspSigningCert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > subsystemCert cert-pki-ca u,u,u > Server-Cert cert-pki-ca u,u,u > > > > Regards > > Nikita S > > > On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy <abokovoy@redhat.com <mailto:abokovoy@redhat.com> > <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>> wrote: > > On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote: > >2. > > > >Status SUBMITTING means the renewal is not yet completed. It will not > >complete until you get Dogtag working. > > > >But now the status says CA_UNEACHABLE > > > >Request ID '20180412150739': > >status: CA_UNREACHABLE > >ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed > request, will > >retry: -504 (libcurl failed to execute the HTTP POST transaction, > >explaining: Peer's Certificate has expired.). > > This is exactly an issue with expired HTTP certificate. > I guess you'd need to roll back time to when the certificate was valid > (before 2019-10-25) and restart certmonger. > > See discussion in this thread: > https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html > > In newer RHEL version (RHEL 7.7) there is a special tool, ipa-cert-fix, > that can help with fixing these issues. However, since you are on the > version before it, you need to do manual renewal. > > >stuck: no > >key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM>',token='NSS Certificate > >DB',pinfile='/etc/httpd/alias/pwdfile.txt' > >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM>',token='NSS Certificate DB' > >CA: IPA > >issuer: CN=Certificate Authority,O=xxx.xxxx.COM <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > >subject: CN=ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > >expires: 2019-10-25 20:16:38 UTC > >principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> > <mailto:xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM>> > >key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >eku: id-kp-serverAuth,id-pkinit-KPKdc > >pre-save command: > >post-save command: /usr/libexec/ipa/certmonger/restart_httpd > >track: yes > > > > > > > > > >*Issue2:* > > > >We are getting this alert while we log in to UI in httpd error logs > > > > > >#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed > while getting > >initial credentials > > > >and PKINIT was disabled > > > >[root@ipa2 httpd]# ipa-pkinit-manage status > >PKINIT is disabled > > > >While I tried to enable this > > > >[root@ipa2 httpd]# ipa-pkinit-manage enable > >Configuring Kerberos KDC (krb5kdc) > > [1/1]: installing X509 Certificate for PKINIT > > > >the process was getting stuck, so I had to terminate it manually. > After > >trying to enable, I'm getting "Login failed due to an unknown reason." > >error in web UI when I try to login > > > >*Error in httpd:* > > > >[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] mod_wsgi (pid=24416): > Exception occurred processing WSGI > >script '/usr/share/ipa/wsgi.py'. > >[Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] Traceback (most recent > call last): > >[Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > "/usr/share/ipa/wsgi.py", line 59, in application > >[Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] return > api.Backend.wsgi_dispatch(environ, > >start_response) > >[Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 267, in > >__call__ > >[Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] return > self.route(environ, start_response) > >[Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 279, in > >route > >[Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] return app(environ, > start_response) > >[Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 937, in > >__call__ > >[Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] > self.kinit(user_principal, password, ipa_ccache_name) > >[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 973, in > >kinit > >[Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] > pkinit_anchors=[paths.KDC_CERT, > >paths.KDC_CA_BUNDLE_PEM], > >[Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line > 127, in > >kinit_armor > >[Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] run(args, env=env, > raiseonerr=True, capture_error=True) > >[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, > in run > >[Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] raise > CalledProcessError(p.returncode, arg_string, > >str(output)) > >[Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] CalledProcessError: > Command '/usr/bin/kinit -n -c > >/var/run/ipa/ccaches/armor_24416 -X > >X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X > >X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned > >non-zero exit status 1 > > > >And when I try to list the certificates using *getcert list,* > there is a > >new cert which was added > > > >Request ID '20191106100258': > >status: CA_UNREACHABLE > >ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed > request, will > >retry: 907 (RPC failed at server. cannot connect to ' > >https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: > >CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). > >stuck: no > >key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > >certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > >CA: IPA > >issuer: > >subject: > >expires: unknown > >pre-save command: > >post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert > >track: yes > >auto-renew: yes > > > > > >Regards > >Nikita S > > > >When > >On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy > <abokovoy@redhat.com <mailto:abokovoy@redhat.com> <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>> > >wrote: > > > >> On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote: > >> >Hi Team, > >> >We have 2 IPA servers in Mater-Master setup are we facing the > below issue > >> >on these servers. > >> > > >> >Isuue1: > >> >Our httpd certificate has expired because of which our IPA1 UI > wasn't > >> >working, we are getting “*loging failed due to an unknown > reason*” error > >> >while we log in to the UI > >> > > >> > > >> >1. First, the IPA console was not working as httpd service was > stopped, > >> >httpd was not starting as HTTP certificate is expired. Added > >> >*NSSEnforceValidCerts > >> >off* line in nss.conf to start the service. > >> > > >> >2. After the change IPA console was loading we are not able to > login to > >> the > >> >console as pki-tomcatd service was not running, > >> >[root@ipa1 ca]# ipactl status > >> >Directory Service: RUNNING > >> >krb5kdc Service: RUNNING > >> >kadmin Service: RUNNING > >> >httpd Service: RUNNING > >> >ipa-custodia Service: RUNNING > >> >ntpd Service: RUNNING > >> >pki-tomcatd Service: STOPPED > >> >ipa-otpd Service: RUNNING > >> > > >> ># systemctl status pki-tomcatd@pki-tomcat.service -l > >> >● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat > >> > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; > enabled; > >> >vendor preset: disabled) > >> > Active: active (running) since Tue 2019-11-05 10:16:50 GMT; > 31min ago > >> > Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i > (code=exited, > >> >status=0/SUCCESS) > >> > Main PID: 97233 (java) > >> > CGroup: > >> > >/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service > >> > └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > >> >-DRESTEASY_LIB=/usr/share/java/resteasy-base > >> >-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath > >> > >> > >/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > >> >-Dcatalina.base=/var/lib/pki/pki-tomcat > -Dcatalina.home=/usr/share/tomcat > >> >-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > >> > >> > >-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > >> >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > >> >-Djava.security.manager > >> > >-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > >> >org.apache.catalina.startup.Bootstrap start > >> > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: WARNING: Exception > >> >processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 > background > >> >process > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: > >> >javax.ws.rs <http://javax.ws.rs> <http://javax.ws.rs>.ServiceUnavailableException: > Subsystem unavailable > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >> > >org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >> > >org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >> > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >> > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >> > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> > >> > >org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com <http://ipa1.xxx.xxxxx.com> <http://ipa1.xxx.xxxxx.com> > server[97233]: at > >> >java.lang.Thread.run(Thread.java:748) > >> > > >> > > >> >This service wasn’t starting with this error > >> > > >> ># less /var/log/pki/pki-tomcat/ca/debug > >> >31/Oct/2019:13:24:23][localhost-startStop-1]: > >> >SSLClientCertificateSelectionCB: desired cert found in list: > subsystemCert > >> >cert-pki-ca > >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: > >> >SSLClientCertificateSelectionCB: returning: subsystemCert > cert-pki-ca > >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake > happened > >> >Could not connect to LDAP server host ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> > <http://ipa1.xxx.xxxx.com> port 636 Error > >> >netscape.ldap.LDAPException: Authentication failed (49) > >> > >> Authentication failed means the RA agent certificate dogtag uses to > >> authenticate to LDAP server is not the same as the one mentioned > in the > >> LDAP entry for RA agent. > >> > >> I think there was some procedure to fix it but I don't have > links handy. > >> Also, you did not specify what versions of FreeIPA you run. > >> > >> > >> >at > >> > >> > >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) > >> > at > >> > >> > >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) > >> > at > >> > >> > >com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) > >> > at > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) > >> > at > >> > >com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > >> > at > >> > >com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > >> > at > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) > >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) > >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > >> > at > >> > >> > >com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > >> > at > javax.servlet.GenericServlet.init(GenericServlet.java:158) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> > at > >> > >> > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > at > >> > >> > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > at > >> > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > >> > at > >> > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at > javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > >> > at > >> > >org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > >> > at > >> > >> > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > >> > at > >> > >> > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > >> > at > >> > >> > >org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > >> > at > >> > >> > >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > >> > at > >> > >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > >> > at > >> > >> > >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > >> > at > >> > >> > >org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > >> > at > >> > >org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > >> > at > >> > >> > >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > >> > at > >> > >org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > >> > at > >> > >> > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > >> > at > >> > >> > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at > >> > >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > >> > at > >> > >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > >> > at > >> > >> > >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > >> > at > >> > >> > >org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > >> > at > >> > >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> > at > >> > >> > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> > at > >> > >> > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> > at java.lang.Thread.run(Thread.java:748) > >> >Internal Database Error encountered: Could not connect to LDAP > server host > >> >ipa1.xxx.xxx.com <http://ipa1.xxx.xxx.com> <http://ipa1.xxx.xxx.com> port 636 Error > netscape.ldap.LDAPException: > >> Authentication > >> >failed (49) > >> > at > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) > >> > at > >> > >com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > >> > at > >> > >com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > >> > at > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) > >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) > >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > >> > at > >> > >> > >com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > >> > at > javax.servlet.GenericServlet.init(GenericServlet.java:158) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> > at > >> > >> > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > at > >> > >> > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > at > >> > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > >> > at > >> > >org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at > javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > >> > at > >> > >org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > >> > at > >> > >> > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > >> > at > >> > >> > >org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > >> > at > >> > >> > >org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > >> > at > >> > >> > >org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > >> > at > >> > >org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > >> > at > >> > >> > >org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > >> > at > >> > >> > >org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > >> > at > >> > >org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > >> > at > >> > >> > >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > >> > at > >> > >org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > >> > at > >> > >> > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > >> > at > >> > >> > >org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at > >> > >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > >> > at > >> > >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > >> > at > >> > >> > >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > >> > at > >> > >> > >org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > >> > at > >> > >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> > at > >> > >> > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> > at > >> > >> > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> > at java.lang.Thread.run(Thread.java:748) > >> > > >> ># getcert list > >> >Request ID '20180412150739': > >> >status: SUBMITTING > >> >stuck: no > >> >key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> > <http://xxx.xxxx.COM>',token='NSS Certificate > >> >DB',pinfile='/etc/httpd/alias/pwdfile.txt' > >> >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM>',token='NSS Certificate DB' > >> >CA: IPA > >> >issuer: CN=Certificate Authority,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM> > >> >subject: CN=ipa1.xxxx.xxxx.com <http://ipa1.xxxx.xxxx.com> > <http://ipa1.xxxx.xxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> <http://xxx.xxxxx.COM> > >> >expires: 2019-10-25 20:16:38 UTC > >> >principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>> > >> >key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> >eku: id-kp-serverAuth,id-pkinit-KPKdc > >> >pre-save command: > >> >post-save command: /usr/libexec/ipa/certmonger/restart_httpd > >> >track: yes > >> >auto-renew: yes > >> > >> Status SUBMITTING means the renewal is not yet completed. It > will not > >> complete until you get Dogtag working. > >> > >> > > >> >Issue2: > >> > > >> >On the IPA2 server, we are unable to login with the admin user > credentials > >> >without OTP, but when an AD user is trying to login with 2FA (i.e, > >> >password and OTP) we are getting this error *"The password you > entered is > >> >incorrect."* > >> > >> AD users cannot use multifactor authentication defined in IPA. > >> > >> > >> ># [root@ipa2 log]# ipactl status > >> >Directory Service: RUNNING > >> >krb5kdc Service: RUNNING > >> >kadmin Service: RUNNING > >> >httpd Service: RUNNING > >> >ipa-custodia Service: RUNNING > >> >ntpd Service: RUNNING > >> >ipa-otpd Service: STOPPED > >> >ipa: INFO: The ipactl command was successful > >> > > >> ># systemctl status ipa-otpd.socket -l > >> >● ipa-otpd.socket - ipa-otpd socket > >> > Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; > disabled; > >> >vendor preset: disabled) > >> > Active: failed (Result: resources) since Tue 2019-11-05 > 08:19:04 GMT; > >> 1h > >> >31min ago > >> > Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) > >> > Accepted: 2; Connected: 0 > >> > > >> >Nov 05 07:42:53 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > systemd[1]: Listening on ipa-otpd > >> socket. > >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > systemd[1]: ipa-otpd.socket failed to > >> >queue service startup job (Maybe the service file is missing or > not a > >> >template unit?): Resource temporarily unavailable > >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > systemd[1]: Unit ipa-otpd.socket > >> entered > >> >failed state. > >> > > >> ># cat /usr/lib/systemd/system/ipa-otpd.socket > >> >[Unit] > >> >Description=ipa-otpd socket > >> > > >> >[Socket] > >> >ListenStream=/var/run/krb5kdc/DEFAULT.socket > >> >RemoveOnStop=true > >> >SocketMode=0600 > >> >Accept=true > >> > > >> >[Install] > >> >WantedBy=krb5kdc.service > >> > > >> > > >> > > >> >We see that data replication is broken between the 2 IPA > servers, as the > >> >changes made on IPA2 is not reflecting on IPA1 > >> This is most likely because your LDAP server certificate expired as > >> well. > >> > >> > >> >We the below errors as well. > >> > > >> >IPA1 > >> >Nov 05 10:09:23 ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com> > krb5kdc[28021](info): TGS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, > etypes > >> >{rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> > <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM>> for ldap/ > >> >ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > >> >Nov 05 10:14:24 ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> > <http://ipa1.corp.endurance.com> krb5kdc[28021](info): TGS_REQ (8 > >> >etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime > 1572948863, > >> >etypes {rep=18 tkt=18 ses=18}, > ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> > <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM>> for > >> >ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > >> > >> These aren't errors. They are normal operations: ldap/ipa1 > service (LDAP > >> server on IPA1) asked for a Kerberos service ticket to LDAP > service on > >> IPA2 and was granted it. This is just as it should be for > replication. > >> > >> > > >> >IPA2 > >> ># tailf krb5kdc.log > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): AS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ > >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> for > krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>>, > >> >Additional pre-authentication required > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): closing down fd > >> 11 > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): AS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, > etypes > >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> for krbtgt/ > >> >xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> <mailto:xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM>> > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): closing down fd > >> 11 > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): TGS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, > etypes > >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> for ldap/ > >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): closing down fd > >> 11 > >> > >> Same here. LDAP server on IPA2 operated against itself here. > >> > >> -- > >> / Alexander Bokovoy > >> Sr. Principal Software Engineer > >> Security / Identity Management Engineering > >> Red Hat Limited, Finland > >> > >> > > > -- > / Alexander Bokovoy > Sr. Principal Software Engineer > Security / Identity Management Engineering > Red Hat Limited, Finland > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence, Please find the below getcert complete list. I had changed the time to* 24 October 2019 *where all the certificates were valid, subsystem cert was renewed in the month of September 2019.
*[root@ipa1 nikita.d]# getcert list* Number of certificates and requests being tracked: 8. Request ID '20171024201539': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxx.COM subject: CN=IPA RA,O=xxx.xxxx.COM expires: 2021-09-06 03:26:08 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20171024201553': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=CA Audit,O=xxx.xxxx.COM expires: 2021-09-06 03:26:28 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201554': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=OCSP Subsystem,O=xxx.xxxx.COM expires: 2021-09-06 03:25:30 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201555': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=CA Subsystem,O=xxx.xxxx.COM expires: 2021-09-06 03:25:48 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201556': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=Certificate Authority,O=xxx.xxxx.COM expires: 2037-10-24 20:15:28 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201557': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxx.xxx.com,O=xxx.xxxx.COM expires: 2021-09-06 03:26:18 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20171024201637': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxx.xxxx.com,O=xxx.xxxx.COM expires: 2021-09-28 03:25:29 UTC principal name: krbtgt/CORP.ENDURANCE.COM@CORP.ENDURANCE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20180412150739': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=xxx.xxxx.COM subject: CN=ipa1.xxx.xxxx.com,O=xxx.xxxx.COM expires: 2019-10-25 20:16:38 UTC principal name: krbtgt/CORP.ENDURANCE.COM@CORP.ENDURANCE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
*Subsystem cert*
[root@ipa1 nikita.d]# certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'subsystemCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 57 (0x39) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=xxx.xxxx.COM" Validity: Not Before: Tue Sep 17 03:25:48 2019 Not After : Mon Sep 06 03:25:48 2021 Subject: "CN=CA Subsystem,O=xxx.xxxx.COM" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: c2:e8:9c:3f:ad:99:88:90:53:56:a1:3e:eb:33:1d:ab: 91:c7:1e:3b:46:09:f2:53:cf:ca:aa:07:9f:05:ed:e7: d1:45:68:1b:60:71:92:c2:c8:d4:88:e3:d0:3a:49:d9: 95:56:6c:ba:87:0b:61:4f:b3:2b:10:e6:27:5c:3b:22: 90:16:e1:6f:f8:44:cf:df:37:8a:f1:28:cb:c0:e0:f3: f8:2d:c8:a4:95:dd:76:78:03:a1:b9:ea:63:3b:38:1a: 7a:db:b6:43:90:c5:ee:b6:44:a5:3b:e2:64:b8:94:4d: 02:4c:0c:cf:30:40:cf:9c:64:15:69:dc:b3:d6:8b:c0: ff:09:58:97:40:a7:3f:4a:37:96:cd:62:fa:b1:03:53: c6:75:a1:25:1d:0e:8e:fa:6b:98:a7:89:07:aa:00:10: df:d3:e5:c2:ea:af:9a:f1:9c:d0:ac:9f:63:2d:5d:b4: 00:5e:ad:63:c1:1b:87:62:4a:5b:c0:3e:b8:37:e6:80: 29:b6:06:47:4d:d5:1d:65:a0:3f:53:12:6d:ba:77:5d: 85:ae:74:cd:f8:87:4f:9c:95:97:93:7a:89:8d:49:35: 6e:c6:f1:34:9f:d2:42:a4:01:a9:79:20:c7:20:66:e6: bd:31:b9:f8:ef:4b:bb:b7:59:4d:ef:d4:1d:2e:78:9d Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: 62:dd:fb:9c:2d:47:73:1a:a4:17:76:47:fb:9f:63:ef: bf:be:db:a5
Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.xxx.xxxx.com/ca/ocsp"
Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Key Encipherment Data Encipherment
Name: Extended Key Usage TLS Web Server Authentication Certificate TLS Web Client Authentication Certificate
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 1f:d3:47:c9:bf:67:23:8d:57:99:af:b1:58:74:76:59: 44:1e:d1:a1:a6:4d:21:8f:92:8e:c2:a4:7f:11:8f:f0: a0:85:14:e9:5e:28:4e:82:5e:5a:ad:5e:50:c3:d0:12: 98:9e:ab:c3:bf:ed:c2:db:b1:b5:50:88:d3:5e:65:29: 6c:94:bf:4c:a7:22:7a:66:58:05:90:bb:c8:33:6c:d0: 5d:4a:d3:85:55:d3:47:35:09:e6:a0:f9:fe:37:91:a4: 7e:a0:86:45:a0:41:a7:1f:04:8b:28:dc:a8:4b:0b:28: 76:92:8f:5a:71:9f:ca:b5:c0:b7:4a:23:ea:78:7e:27: a0:03:ba:9d:ee:24:cc:4b:5c:ba:7c:5e:b9:7b:17:14: 6d:f0:f2:a0:67:2a:4b:9f:9d:91:17:d0:f3:3d:91:fe: 7f:b9:03:54:06:b2:dd:fa:b2:f0:ff:c3:64:93:11:4e: 89:18:52:30:c3:54:18:36:32:ae:c1:f7:d2:87:3d:5e: 93:5c:76:e0:73:f6:fb:23:d4:dd:13:3e:5a:67:46:40: e2:64:a4:58:b3:70:fd:9c:15:3d:bc:76:d9:da:e1:12: ba:6f:22:ce:61:dd:18:d1:6c:fa:81:ed:f3:6c:35:6b: 45:dc:61:59:06:7d:ef:8d:1f:5f:ba:0a:35:67:25:12 Fingerprint (SHA-256):
79:19:D3:40:03:55:51:69:57:1D:54:46:16:36:F3:BD:DE:1B:18:2D:6D:D3:22:9B:3A:2E:BF:FD:74:E3:50:87 Fingerprint (SHA1): D6:BE:5D:6D:91:24:E1:A4:AE:4C:C1:4A:70:4B:92:F6:3B:6D:4E:A8
Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User
In https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat... doc they are asking to *Check the subsystemCert cert-pki-ca* where we check if the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag *internal=*…)
[root@ipa1 nikita.d]# *certutil -K -d /etc/pki/pki-tomcat/alias -f /etc/pki/pki-tomcat/alias/pwdfile.txt* certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS Certificate DB:Server-Cert cert-pki-ca < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS Certificate DB:auditSigningCert cert-pki-ca < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS Certificate DB:ocspSigningCert cert-pki-ca [root@ipa1 nikita.d]# [root@ipa1 nikita.d]# *sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'NSS Certificate DB:Server-Cert cert-pki-ca'* certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Incorrect password/PIN entered. certutil: could not authenticate to token NSS Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect. [root@ipa1 nikita.d]# *sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'* certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Incorrect password/PIN entered. certutil: could not authenticate to token NSS Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
Regards Nikita S
On Tue, Nov 12, 2019 at 8:04 PM Florence Blanc-Renaud flo@redhat.com wrote:
On 11/8/19 4:33 PM, Nikita Deeksha via FreeIPA-users wrote:
Alexander/Florence,
While we were trying to renew the certificate the httpd cert, we went back in time where httpd cert was vaild and then tried to renew the cert, but in *Step7: Test CA operation* is failing with the below error. Can you please help us with this?
https://access.redhat.com/solutions/3357261
[root@ipa1 ~]# curl --cacert /etc/ipa/ca.crt -v https://%60hostname%60:8443/ca/ee/ca/getCertChain
- About to connect() to ipa1.corp.endurance.com
http://ipa1.corp.endurance.com port 8443 (#0)
- Trying 172.27.152.34...
- Connected to ipa1.corp.endurance.com http://ipa1.corp.endurance.com
(172.27.152.34) port 8443 (#0)
- Initializing NSS with certpath: sql:/etc/pki/nssdb
- CAfile: /etc/ipa/ca.crt CApath: none
- NSS: client certificate not found (nickname not specified)
- SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
- Server certificate:
- subject: CN=ipa1.corp.endurance.com
http://ipa1.corp.endurance.com,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM
- start date: Sep 17 03:26:18 2019 GMT
- expire date: Sep 06 03:26:18 2021 GMT
- common name: ipa1.corp.endurance.com http://ipa1.corp.endurance.com
- issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM
GET /ca/ee/ca/getCertChain HTTP/1.1 User-Agent: curl/7.29.0 Host: ipa1.corp.endurance.com:8443 <
http://ipa1.corp.endurance.com:8443%3E
Accept: */*
< HTTP/1.1 500 Internal Server Error < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 2208 < Date: Thu, 24 Oct 2019 16:31:10 GMT < Connection: close <
<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Subsystem unavailable means that the CA subsystem is not running.
com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
org.apache.tomcat.util.net
.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748) </pre></p><p><b>note</b> <u>The ful* Closing connection 0 l stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>[root@ipa1 ~]#
/var/log/pki/pki-tomcat/ca/debug shows this error
Could not connect to LDAP server host ipa1.corp.endurance.com http://ipa1.corp.endurance.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
This happens when the subsystem cert is not accepted as valid authentication from Dogtag to the ldap server.
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host ipa1.corp.endurance.com http://ipa1.corp.endurance.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) at
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572) at com.netscape.certsrv.apps.CMS.init(CMS.java:189) at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) at
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) [24/Oct/2019:16:29:50][localhost-startStop-1]: CMS.start(): shutdown
server
[24/Oct/2019:16:29:50][localhost-startStop-1]: CMSEngine.shutdown()
sybsystemCert is still valid.
Which date in the past did you use? If the subsystemcert's Not Before is after this date, the cert is not valid yet and authentication will fail.
Could you post the full output of getcert list? If you have multiple certificates expired, you need to carefully select a date in the past where all the certs are in their validity window (ie after "Not before" and before "Not after").
HTH, flo
Request ID '20171024201555': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=CORP.ENDURANCE.COM http://CORP.ENDURANCE.COM subject: CN=CA Subsystem,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM
expires: 2021-09-06 03:25:48 UTC key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes
[root@ipa1 ~]# cat /etc/pki/pki-tomcat/ca/CS.cfg| grep subsystem.cert ca.cert.subsystem.certusage=SSLClient
ca.subsystem.cert=MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS
ca.subsystem.certreq=MIICeTCCAWECAQAwNDEbMBkGA1UECgwSQ09SUC5FTkRVUkFOQ0UuQ09NMRUwEwYDVQQDDAxDQSBTdWJzeXN0ZW0wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDC6Jw/rZmIkFNWoT7rMx2rkcceO0YJ8lPPyqoHnwXt59FFaBtgcZLCyNSI49A6SdmVVmy6hwthT7MrEOYnXDsikBbhb/hEz983ivEoy8Dg8/gtyKSV3XZ4A6G56mM7OBp627ZDkMXutkSlO+JkuJRNAkwMzzBAz5xkFWncs9aLwP8JWJdApz9KN5bNYvqxA1PGdaElHQ6O+muYp4kHqgAQ39PlwuqvmvGc0KyfYy1dtABerWPBG4diSlvAPrg35oAptgZHTdUdZaA/UxJtunddha50zfiHT5yVl5N6iY1JNW7G8TSf0kKkAal5IMcgZua9Mbn470u7t1lN79QdLnidAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEABl3uxfntEZZFeoB1eIjd774MSxK2itaj6B1VyXV0qxcp/SHF1sfJnUHjDpNIB0J/Bpk/sc/+Y2Gxar3zcwyORjF+ojCBc8lP6QZ7mB9H3m9XEuA+H1qmgGz+h2FhnQy5/EmgJnTAn8fNpZCQwQmSZLtnbmd++yK9y+YXCAvaHx+Wsz5c/GhyNxoHGADcmir+iFVltO3jpQ06ThVBb0bD4D6ZzaimkM1vnAqseKdqXvLJI1/ZfIVj6QDcdnnTryt24zOZoxzeOcc2X5HKOtR/5pXilmCS/7zpujXTd1iZi8nfRkpurUdMgC7FLK+B3Dq03G61RPJUwJZvsKlXbwBghg==
[root@ipa1 ~]# cat /tmp/subsystemCert\ cert-pki-ca.pem
MIIDhjCCAm6gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA9MRswGQYDVQQKDBJDT1JQLkVORFVSQU5DRS5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA5MTcwMzI1NDhaFw0yMTA5MDYwMzI1NDhaMDQxGzAZBgNVBAoMEkNPUlAuRU5EVVJBTkNFLkNPTTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwuicP62ZiJBTVqE+6zMdq5HHHjtGCfJTz8qqB58F7efRRWgbYHGSwsjUiOPQOknZlVZsuocLYU+zKxDmJ1w7IpAW4W/4RM/fN4rxKMvA4PP4Lcikld12eAOhuepjOzgaetu2Q5DF7rZEpTviZLiUTQJMDM8wQM+cZBVp3LPWi8D/CViXQKc/SjeWzWL6sQNTxnWhJR0OjvprmKeJB6oAEN/T5cLqr5rxnNCsn2MtXbQAXq1jwRuHYkpbwD64N+aAKbYGR03VHWWgP1MSbbp3XYWudM34h0+clZeTeomNSTVuxvE0n9JCpAGpeSDHIGbmvTG5+O9Lu7dZTe/UHS54nQIDAQABo4GZMIGWMB8GA1UdIwQYMBaAFGLd+5wtR3MapBd2R/ufY++/vtulMEQGCCsGAQUFBwEBBDgwNjA0BggrBgEFBQcwAYYoaHR0cDovL2lwYS1jYS5jb3JwLmVuZHVyYW5jZS5jb20vY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAf00fJv2cjjVeZr7FYdHZZRB7RoaZNIY+SjsKkfxGP8KCFFOleKE6CXlqtXlDD0BKYnqvDv+3C27G1UIjTXmUpbJS/TKciemZYBZC7yDNs0F1K04VV00c1Ceag+f43kaR+oIZFoEGnHwSLKNyoSwsodpKPWnGfyrXAt0oj6nh+J6ADup3uJMxLXLp8Xrl7FxRt8PKgZypLn52RF9DzPZH+f7kDVAay3fqy8P/DZJMRTokYUjDDVBg2Mq7B99KHPV6TXHbgc/b7I9TdEz5aZ0ZA4mSkWLNw/ZwVPbx22drhErpvIs5h3RjRbPqB7fNsNWtF3GFZBn3vjR9fugo1ZyUS
I had a few queries on this
- Is there any way we take the backup of the current setup and create a
new IPA instance without changing any parameter in the client servers 2. If we upgrade the IPA version we can fix this issue?
Regards Nikita S
On Fri, Nov 8, 2019 at 4:29 PM Nikita Deeksha <nikita.d@endurance.com mailto:nikita.d@endurance.com> wrote:
Thanks, Florence and Alexander. @Alexander Bokovoy <mailto:abokovoy@redhat.com> , While we were debugging in one of a blog we came across a scenario where NSS DB had got corrupted during a certificate renewal, so wanted to eliminate that issue. We see all the private key. [root@ipa1 nikita.d]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa b8763cec560cf751cdafa5b0006af9405bdfabf0 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa c7d8b4bf5f7d60de906444744a3b512c801676d2 (orphan) < 2> rsa 49ddc4aff2ae7a59ed5c3a939217d006d059fea2 NSS Certificate DB:Server-Cert cert-pki-ca < 3> rsa 12717b4e7fec1f408c947015b069e94838198947 NSS Certificate DB:auditSigningCert cert-pki-ca < 4> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert cert-pki-ca < 5> rsa 079bf91860780244f89ee9509853ed3e975ca11d NSS Certificate DB:ocspSigningCert cert-pki-ca We will try renewing the HTTP cert in our environment, will let you know once the cert is updated successfully. https://access.redhat.com/solutions/3357261 Regards Nikita S On Fri, Nov 8, 2019 at 3:42 PM Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: On 11/7/19 11:16 AM, Nikita Deeksha via FreeIPA-users wrote: > Thanks for the update Alexander will check this and get back to you, > wanted to check on another thing as well. > > Can you please help us to understand this error that we see for the cert > in pki > > [root@ipa1 nikita.d]# for i in $(certutil -d /etc/pki/pki-tomcat/alias > -L | grep cert-pki-ca | awk'{print $1}');do certutil -K -d > /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n "$i cert-pki-ca";done > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > < 0> rsa 249cfa8ef238a902bd45ce397eda0a8ce8dda01d caSigningCert > cert-pki-ca > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > > certutil: Checking token "NSS Certificate DB" in slot "NSS User Private > Key and Certificate Services" > > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized > Object Identifier. > Hi, If you use "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt" (without the -n alias option), you will see all the keys present in the database and notice that some of them have a prefixed nickname, for instance: 'NSS Certificate DB:Server-Cert cert-pki-ca' instead of 'Server-Cert cert-pki-ca'. You need to provide this prefixed name with -K -n nickname. HTH, flo > > These are the cert which is present /etc/pki/pki-tomcat/alias > > [root@ipa1 nikita.d]# certutil -L -d
/etc/pki/pki-tomcat/alias
> > Certificate Nickname Trust > Attributes > > SSL,S/MIME,JAR/XPI > > COMODO CA BUNDLE CT,C,C > ocspSigningCert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > subsystemCert cert-pki-ca u,u,u > Server-Cert cert-pki-ca u,u,u > > > > Regards > > Nikita S > > > On Wed, Nov 6, 2019 at 9:52 PM Alexander Bokovoy <abokovoy@redhat.com <mailto:abokovoy@redhat.com> > <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>>
wrote:
> > On ke, 06 marras 2019, Nikita Deeksha via FreeIPA-users wrote: > >2. > > > >Status SUBMITTING means the renewal is not yet completed. It will not > >complete until you get Dogtag working. > > > >But now the status says CA_UNEACHABLE > > > >Request ID '20180412150739': > >status: CA_UNREACHABLE > >ca-error: Server at https://ipa1.xxx.xxxx.com/ipa/xml failed > request, will > >retry: -504 (libcurl failed to execute the HTTP POST transaction, > >explaining: Peer's Certificate has expired.). > > This is exactly an issue with expired HTTP certificate. > I guess you'd need to roll back time to when the certificate was valid > (before 2019-10-25) and restart certmonger. > > See discussion in this thread: >
https://www.redhat.com/archives/freeipa-users/2016-July/msg00270.html
> > In newer RHEL version (RHEL 7.7) there is a special tool, ipa-cert-fix, > that can help with fixing these issues. However, since you are on the > version before it, you need to do manual renewal. > > >stuck: no > >key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM>',token='NSS Certificate > >DB',pinfile='/etc/httpd/alias/pwdfile.txt' > >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com>,O=CORP.ENDURANCE.COM <http://CORP.ENDURANCE.COM> > <http://CORP.ENDURANCE.COM>',token='NSS Certificate DB' > >CA: IPA > >issuer: CN=Certificate Authority,O=xxx.xxxx.COM <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > >subject: CN=ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxx.COM <http://xxx.xxxx.COM> <http://xxx.xxxx.COM> > >expires: 2019-10-25 20:16:38 UTC > >principal name: krbtgt/xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM> > <mailto:xxx.xxxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxxx.COM@xxx.xxxx.COM>> > >key usage: >
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >eku: id-kp-serverAuth,id-pkinit-KPKdc > >pre-save command: > >post-save command: /usr/libexec/ipa/certmonger/restart_httpd > >track: yes > > > > > > > > > >*Issue2:* > > > >We are getting this alert while we log in to UI in httpd error logs > > > > > >#ipa: INFO: 401 Unauthorized: kinit: Preauthentication failed > while getting > >initial credentials > > > >and PKINIT was disabled > > > >[root@ipa2 httpd]# ipa-pkinit-manage status > >PKINIT is disabled > > > >While I tried to enable this > > > >[root@ipa2 httpd]# ipa-pkinit-manage enable > >Configuring Kerberos KDC (krb5kdc) > > [1/1]: installing X509 Certificate for PKINIT > > > >the process was getting stuck, so I had to terminate it manually. > After > >trying to enable, I'm getting "Login failed due to an unknown reason." > >error in web UI when I try to login > > > >*Error in httpd:* > > > >[Wed Nov 06 10:13:37.465318 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] mod_wsgi (pid=24416): > Exception occurred processing WSGI > >script '/usr/share/ipa/wsgi.py'. > >[Wed Nov 06 10:13:37.465433 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] Traceback (most recent > call last): > >[Wed Nov 06 10:13:37.468706 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > "/usr/share/ipa/wsgi.py", line 59, in application > >[Wed Nov 06 10:13:37.470083 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] return > api.Backend.wsgi_dispatch(environ, > >start_response) > >[Wed Nov 06 10:13:37.470146 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 267, in > >__call__ > >[Wed Nov 06 10:13:37.473376 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] return > self.route(environ, start_response) > >[Wed Nov 06 10:13:37.473437 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 279, in > >route > >[Wed Nov 06 10:13:37.473462 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] return app(environ, > start_response) > >[Wed Nov 06 10:13:37.473475 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 937, in > >__call__ > >[Wed Nov 06 10:13:37.476093 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] > self.kinit(user_principal, password, ipa_ccache_name) > >[Wed Nov 06 10:13:37.478843 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line > 973, in > >kinit > >[Wed Nov 06 10:13:37.478864 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] > pkinit_anchors=[paths.KDC_CERT, > >paths.KDC_CA_BUNDLE_PEM], > >[Wed Nov 06 10:13:37.478878 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py",
line
> 127, in > >kinit_armor > >[Wed Nov 06 10:13:37.484351 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] run(args, env=env, > raiseonerr=True, capture_error=True) > >[Wed Nov 06 10:13:37.484381 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] File > >"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
562,
> in run > >[Wed Nov 06 10:13:37.487470 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] raise > CalledProcessError(p.returncode, arg_string, > >str(output)) > >[Wed Nov 06 10:13:37.488932 2019] [:error] [pid 24416] [remote > >172.27.10.113:0 <http://172.27.10.113:0> <http://172.27.10.113:0>] CalledProcessError: > Command '/usr/bin/kinit -n -c > >/var/run/ipa/ccaches/armor_24416 -X > >X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X > >X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned > >non-zero exit status 1 > > > >And when I try to list the certificates using *getcert list,* > there is a > >new cert which was added > > > >Request ID '20191106100258': > >status: CA_UNREACHABLE > >ca-error: Server at https://ipa2.xxx.xxxxx.com/ipa/xml failed > request, will > >retry: 907 (RPC failed at server. cannot connect to ' > >https://ipa1.xxx.xxxxx.com:443/ca/rest/account/login': [SSL: > >CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)). > >stuck: no > >key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > >certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > >CA: IPA > >issuer: > >subject: > >expires: unknown > >pre-save command: > >post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert > >track: yes > >auto-renew: yes > > > > > >Regards > >Nikita S > > > >When > >On Wed, Nov 6, 2019 at 6:39 PM Alexander Bokovoy > <abokovoy@redhat.com <mailto:abokovoy@redhat.com> <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>> > >wrote: > > > >> On ti, 05 marras 2019, Nikita Deeksha via FreeIPA-users wrote: > >> >Hi Team, > >> >We have 2 IPA servers in Mater-Master setup are we facing the > below issue > >> >on these servers. > >> > > >> >Isuue1: > >> >Our httpd certificate has expired because of which our IPA1 UI > wasn't > >> >working, we are getting “*loging failed due to an unknown > reason*” error > >> >while we log in to the UI > >> > > >> > > >> >1. First, the IPA console was not working as httpd service was > stopped, > >> >httpd was not starting as HTTP certificate is expired. Added > >> >*NSSEnforceValidCerts > >> >off* line in nss.conf to start the service. > >> > > >> >2. After the change IPA console was loading we are not able to > login to > >> the > >> >console as pki-tomcatd service was not running, > >> >[root@ipa1 ca]# ipactl status > >> >Directory Service: RUNNING > >> >krb5kdc Service: RUNNING > >> >kadmin Service: RUNNING > >> >httpd Service: RUNNING > >> >ipa-custodia Service: RUNNING > >> >ntpd Service: RUNNING > >> >pki-tomcatd Service: STOPPED > >> >ipa-otpd Service: RUNNING > >> > > >> ># systemctl status pki-tomcatd@pki-tomcat.service -l > >> >● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat > >> > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; > enabled; > >> >vendor preset: disabled) > >> > Active: active (running) since Tue 2019-11-05 10:16:50 GMT; > 31min ago > >> > Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i > (code=exited, > >> >status=0/SUCCESS) > >> > Main PID: 97233 (java) > >> > CGroup: > >> >
/system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service > >> > └─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > >> >-DRESTEASY_LIB=/usr/share/java/resteasy-base > >> >-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath > >> > >> >
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > >> >-Dcatalina.base=/var/lib/pki/pki-tomcat > -Dcatalina.home=/usr/share/tomcat > >> >-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > >> > >> >
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > >>
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > >> >-Djava.security.manager > >> >
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > >> >org.apache.catalina.startup.Bootstrap start > >> > > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: WARNING: Exception > >> >processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 > background > >> >process > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: > >> >javax.ws.rs http://javax.ws.rs http://javax.ws.rs.ServiceUnavailableException: > Subsystem unavailable > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> >
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> > >> >
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> > >> >
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> > >> >
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> > >> >
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> > >> >
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> > >> >
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > >> >Nov 05 10:47:57 ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com http://ipa1.xxx.xxxxx.com > server[97233]: at > >> >java.lang.Thread.run(Thread.java:748) > >> > > >> > > >> >This service wasn’t starting with this error > >> > > >> ># less /var/log/pki/pki-tomcat/ca/debug > >> >31/Oct/2019:13:24:23][localhost-startStop-1]: > >> >SSLClientCertificateSelectionCB: desired cert found in list: > subsystemCert > >> >cert-pki-ca > >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: > >> >SSLClientCertificateSelectionCB: returning: subsystemCert > cert-pki-ca > >> >[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake > happened > >> >Could not connect to LDAP server host ipa1.xxx.xxxx.com http://ipa1.xxx.xxxx.com > http://ipa1.xxx.xxxx.com port 636 Error > >> >netscape.ldap.LDAPException: Authentication failed
(49)
> >> > >> Authentication failed means the RA agent certificate dogtag uses to > >> authenticate to LDAP server is not the same as the one mentioned > in the > >> LDAP entry for RA agent. > >> > >> I think there was some procedure to fix it but I don't have > links handy. > >> Also, you did not specify what versions of FreeIPA you run. > >> > >> > >> >at > >> > >> >
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) > >> > at > >> > >> >
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) > >> > at > >> > >> >
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) > >> > at > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) > >> > at > >> >
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > >> > at > >> >
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > >> > at >
com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) > >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > >> > at > >> > >> >
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > >> > at > javax.servlet.GenericServlet.init(GenericServlet.java:158) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> > at > >> > >> >
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > at > >> > >> >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > at > >> >
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > >> > at > >> >
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at >
javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> >> > at > >> >
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > >> > at > >> > >> >
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > >> > at > >> > >> >
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > >> > at > >> > >> >
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > >> > at > >> > >> >
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > >> > at > >> >
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > >> > at > >> > >> >
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > >> > at > >> > >> >
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > >> > at > >> >
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > >> > at > >> > >> >
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > >> > at > >> >
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > >> > at > >> > >> >
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > >> > at > >> > >> >
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at > >> >
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > >> > at > >> >
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > >> > at > >> > >> >
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > >> > at > >> > >> >
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > >> > at > >> >
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> > at > >> > >> >
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> > at > >> > >> >
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> > at java.lang.Thread.run(Thread.java:748) > >> >Internal Database Error encountered: Could not connect to LDAP > server host > >> >ipa1.xxx.xxx.com http://ipa1.xxx.xxx.com http://ipa1.xxx.xxx.com port 636 Error > netscape.ldap.LDAPException: > >> Authentication > >> >failed (49) > >> > at > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) > >> > at > >> >
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176) > >> > at > >> >
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082) > >> > at >
com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> >> > at com.netscape.certsrv.apps.CMS.init(CMS.java:189) > >> > at com.netscape.certsrv.apps.CMS.start(CMS.java:1631) > >> > at > >> > >> >
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117) > >> > at > javax.servlet.GenericServlet.init(GenericServlet.java:158) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> > at > >> > >> >
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > at > >> > >> >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > at > >> >
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > >> > at > >> >
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at >
javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> >> > at > >> >
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > >> > at > >> > >> >
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > >> > at > >> > >> >
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > >> > at > >> > >> >
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257) > >> > at > >> > >> >
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) > >> > at > >> >
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) > >> > at > >> > >> >
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) > >> > at > >> > >> >
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) > >> > at > >> >
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > >> > at > >> > >> >
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > >> > at > >> >
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > >> > at > >> > >> >
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > >> > at > >> > >> >
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > >> > at java.security.AccessController.doPrivileged(Native > Method) > >> > at > >> >
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > >> > at > >> >
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > >> > at > >> > >> >
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > >> > at > >> > >> >
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > >> > at > >> >
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> > at > >> > >> >
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >> > at > >> > >> >
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >> > at java.lang.Thread.run(Thread.java:748) > >> > > >> ># getcert list > >> >Request ID '20180412150739': > >> >status: SUBMITTING > >> >stuck: no > >> >key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >> >ipa1.xxxx.xxxxx.com http://ipa1.xxxx.xxxxx.com http://ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM <http://xxx.xxxx.COM
> <http://xxx.xxxx.COM>',token='NSS Certificate > >> >DB',pinfile='/etc/httpd/alias/pwdfile.txt' > >> >certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN= > >> >ipa1.xxxx.xxxxx.com <http://ipa1.xxxx.xxxxx.com> > <http://ipa1.xxxx.xxxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM>',token='NSS Certificate DB' > >> >CA: IPA > >> >issuer: CN=Certificate Authority,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> > <http://xxx.xxxxx.COM> > >> >subject: CN=ipa1.xxxx.xxxx.com <http://ipa1.xxxx.xxxx.com> > <http://ipa1.xxxx.xxxx.com>,O=xxx.xxxxx.COM <http://xxx.xxxxx.COM> <http://xxx.xxxxx.COM> > >> >expires: 2019-10-25 20:16:38 UTC > >> >principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> > <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>> > >> >key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> >eku: id-kp-serverAuth,id-pkinit-KPKdc > >> >pre-save command: > >> >post-save command: /usr/libexec/ipa/certmonger/restart_httpd > >> >track: yes > >> >auto-renew: yes > >> > >> Status SUBMITTING means the renewal is not yet completed. It > will not > >> complete until you get Dogtag working. > >> > >> > > >> >Issue2: > >> > > >> >On the IPA2 server, we are unable to login with the admin user > credentials > >> >without OTP, but when an AD user is trying to login with 2FA (i.e, > >> >password and OTP) we are getting this error *"The password you > entered is > >> >incorrect."* > >> > >> AD users cannot use multifactor authentication defined in IPA. > >> > >> > >> ># [root@ipa2 log]# ipactl status > >> >Directory Service: RUNNING > >> >krb5kdc Service: RUNNING > >> >kadmin Service: RUNNING > >> >httpd Service: RUNNING > >> >ipa-custodia Service: RUNNING > >> >ntpd Service: RUNNING > >> >ipa-otpd Service: STOPPED > >> >ipa: INFO: The ipactl command was successful > >> > > >> ># systemctl status ipa-otpd.socket -l > >> >● ipa-otpd.socket - ipa-otpd socket > >> > Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; > disabled; > >> >vendor preset: disabled) > >> > Active: failed (Result: resources) since Tue 2019-11-05 > 08:19:04 GMT; > >> 1h > >> >31min ago > >> > Listen: /var/run/krb5kdc/DEFAULT.socket (Stream) > >> > Accepted: 2; Connected: 0 > >> > > >> >Nov 05 07:42:53 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > systemd[1]: Listening on ipa-otpd > >> socket. > >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > systemd[1]: ipa-otpd.socket failed to > >> >queue service startup job (Maybe the service file is missing or > not a > >> >template unit?): Resource temporarily unavailable > >> >Nov 05 08:19:04 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > systemd[1]: Unit ipa-otpd.socket > >> entered > >> >failed state. > >> > > >> ># cat /usr/lib/systemd/system/ipa-otpd.socket > >> >[Unit] > >> >Description=ipa-otpd socket > >> > > >> >[Socket] > >> >ListenStream=/var/run/krb5kdc/DEFAULT.socket > >> >RemoveOnStop=true > >> >SocketMode=0600 > >> >Accept=true > >> > > >> >[Install] > >> >WantedBy=krb5kdc.service > >> > > >> > > >> > > >> >We see that data replication is broken between the 2
IPA
> servers, as the > >> >changes made on IPA2 is not reflecting on IPA1 > >> This is most likely because your LDAP server certificate expired as > >> well. > >> > >> > >> >We the below errors as well. > >> > > >> >IPA1 > >> >Nov 05 10:09:23 ipa1.xxx.xxxx.com <http://ipa1.xxx.xxxx.com> <http://ipa1.xxx.xxxx.com> > krb5kdc[28021](info): TGS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, > etypes > >> >{rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM> > <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM <mailto:ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM>> for ldap/ > >> >ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > >> >Nov 05 10:14:24 ipa1.corp.endurance.com <http://ipa1.corp.endurance.com> > <http://ipa1.corp.endurance.com> krb5kdc[28021](info): TGS_REQ (8 > >> >etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime > 1572948863, > >> >etypes {rep=18 tkt=18 ses=18}, > ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM> > <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM <mailto:ipa1.xxxx.xxx.com@xxxx.xxxx.COM>> for > >> >ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> > >> > >> These aren't errors. They are normal operations: ldap/ipa1 > service (LDAP > >> server on IPA1) asked for a Kerberos service ticket to LDAP > service on > >> IPA2 and was granted it. This is just as it should be
for
> replication. > >> > >> > > >> >IPA2 > >> ># tailf krb5kdc.log > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): AS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ > >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> for > krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM> <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM <mailto:xxxx.xxxx.COM@xxxx.xxxx.COM>>, > >> >Additional pre-authentication required > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): closing down fd > >> 11 > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): AS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, > etypes > >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> for krbtgt/ > >> >xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM> <mailto:xxx.xxxx.COM@xxx.xxxx.COM <mailto:xxx.xxxx.COM@xxx.xxxx.COM>> > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): closing down fd > >> 11 > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): TGS_REQ (8 etypes > >> >{18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, > etypes > >> >{rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxxx.xxxx.COM>> for ldap/ > >> >ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM> > <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM <mailto:ipa2.xxxx.xxxx.com@xxx.xxxx.COM>> > >> >Nov 05 09:59:25 ipa2.xxxx.xxxx.com <http://ipa2.xxxx.xxxx.com> <http://ipa2.xxxx.xxxx.com> > krb5kdc[2451](info): closing down fd > >> 11 > >> > >> Same here. LDAP server on IPA2 operated against itself here. > >> > >> -- > >> / Alexander Bokovoy > >> Sr. Principal Software Engineer > >> Security / Identity Management Engineering > >> Red Hat Limited, Finland > >> > >> > > > -- > / Alexander Bokovoy > Sr. Principal Software Engineer > Security / Identity Management Engineering > Red Hat Limited, Finland > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
>
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org