https://bugzilla.redhat.com/show_bug.cgi?id=842346
Bug ID: 842346 QA Contact: extras-qa@fedoraproject.org Severity: unspecified Version: rawhide Priority: unspecified CC: akurtako@redhat.com, ivan.afonichev@gmail.com, java-sig-commits@lists.fedoraproject.org, kdaniel@redhat.com Assignee: ivan.afonichev@gmail.com Summary: Properly migrate tomcat to systemd Regression: --- Story Points: --- Classification: Fedora OS: Unspecified Reporter: johannbg@gmail.com Type: Bug Documentation: --- Hardware: Unspecified Mount Type: --- Status: NEW Component: tomcat Product: Fedora
Description of problem:
Let's try and properly migrate tomcat to native systemd units the tomcat.service is just an ugly hack.
I propose that we split tomcat startup into 4 units...
1 tomcat.service 2 tomcat-security.service 3 tomcat@.service ( For multiple instances ) 4.tomcat-security@.service ( For multiple instance )
Have all the required environmental parameters in one configuration file only ( /etc/tomcat/tomcat.conf )
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce: 1. 2. 3.
Actual results:
Expected results:
Additional info:
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #1 from Jóhann B. Guðmundsson johannbg@gmail.com --- I mean The tomcat service being shipped is just an ugly hack
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #2 from Ivan Afonichev ivan.afonichev@gmail.com --- What do you mean as tomcat-security.service ?
Michal Vyskocil from SUSE team have made some commitments on tomcat systemd support... I'am going to merge them and package 7.0.29 release.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #3 from Jóhann B. Guðmundsson johannbg@gmail.com --- (In reply to comment #2)
What do you mean as tomcat-security.service ?
elif [ "$1" = "start-security" ]; then ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \ -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ -Djava.security.manager \ -Djava.security.policy=="${CATALINA_BASE}/conf/catalina.policy" \ -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \ -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \ org.apache.catalina.startup.Bootstrap start \ >> ${CATALINA_BASE}/logs/catalina.out 2>&1 & if [ ! -z "$CATALINA_PID" ]; then echo $! > $CATALINA_PID
Michal Vyskocil from SUSE team have made some commitments on tomcat systemd support... I'am going to merge them and package 7.0.29 release.
Got a link to those commitments so I can review it?
There are being made and have been made some changes to enhance multiple instance support via templates which he might not have caught up to yet unless "frozcat" has been assisting overseeing his change set
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #4 from Ivan Afonichev ivan.afonichev@gmail.com --- " First of all, for service type forking, you have to specify PIDFile=, unless systemctl status reports the invalid state, when daemon is running. However, even if you use the PIDFile, you will get state failed on systemctl stop because jvm ends with code 143. Because of that, I've used a different approach to yours - please consider if that's usable for you or not
I've removed a lot of bloat, which is not needed under systemd - like you don't need to source /etc/tomcat/tomcat.conf as EnvironmentFile=/etc/tomcat/tomcat.conf do the same for you. Or the ugly part with nohup su tomcat -c, for which you have to call parseArguments and export the environment for /usr/sbin/tomcat is useless, becase of User=/Group=
For that reason I've simplified the structure of scripts and removed all crap needed for sysvinit. You can take a look at Java:packages/tomcat [1] - the most important files are tomcat-7.0-tomcat-sysd [2], tomcat-7.0.service [3] and tomcat-7.0-jsvc.service [4].
For jsvc, I've merged the separate script into one, to reuse most of the parts. But this part should get an another care - I would say -jsvc.service can be Type=forking with appropriate PIDFile=, but I did not want to make the differences between those two.
As an alternative, tomcat.service started with CapabilityBoundingSet=CAP_NET_BIND_SERVICE can bind to port 80 and run as tomcat user, but this will need to be checked first. But in case it's true, this can make the -jsvc service and package obsoleted.
[1] https://build.opensuse.org/package/files?package=tomcat&project=Java%3Ap... [2] https://build.opensuse.org/package/view_file?file=tomcat-7.0-tomcat-sysd&... [3] https://build.opensuse.org/package/view_file?file=tomcat-7.0.service&pac... [4] https://build.opensuse.org/package/view_file?file=tomcat-7.0-jsvc.service&am... "
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #5 from Jóhann B. Guðmundsson johannbg@gmail.com --- as suspected this is no better than we currently have...
I'm going to see if I cant come up with something better then this and we want to use separated tomcat config files along with systemd template files for multiple instances.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
Paul P Komkoff Jr i@stingr.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |i@stingr.net
--- Comment #7 from Paul P Komkoff Jr i@stingr.net --- Ok, it's fairly simple to do this: [Service] Type=simple Environment="NAME=%I" ExecStart=/usr/sbin/tomcat start-new ExecStop=/usr/sbin/tomcat stop SuccessExitStatus=143 User=tomcat Group=tomcat
and then if [ "$1" = "start-new" ]; then exec ${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \ -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \ -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \ org.apache.catalina.startup.Bootstrap start
also fixing stop elif [ "$1" = "stop" ]; then exec ${JAVACMD} $JAVA_OPTS \ -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ org.apache.catalina.startup.Bootstrap stop
the real problem is that everything is in the server.xml, and in the rest of the /etc/tomcat as well I'm not very familiar with tomcat and to me it looks like duplicating the entire /etc/tomcat and then /usr/share/tomcat as well (in which frankly there's only one non-symlink)
alternatively, systemd fs namespaces instead of symlinks, map different /etc/tomcatX to /usr/share/tomcat/conf
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #8 from Jóhann B. Guðmundsson johannbg@gmail.com --- That's not properly migrating this...
" ExecStart=/usr/sbin/tomcat start-new <--- ExecStop=/usr/sbin/tomcat stop <--- "
The scripts that start and stop the tomcat instance ( like catalina.sh ) should be migrated to systemd...
Calling the old initscripts or otherwise using "scripts" is not properly migrating this to native systemd unit start where the unit(s) would obsolete/replace the function the script
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #9 from Paul P Komkoff Jr i@stingr.net --- Well, the thing is there's a lot of shell magic that sets up environment variables and common pattern to solve it is to drop a small wrapper for it. "start-new" is just a hack, I've replaced it with start now. In fact, this is my current setup: [Service] Type=simple EnvironmentFile=/etc/tomcat/tomcat.conf EnvironmentFile=-/etc/sysconfig/tomcat@%I Environment="NAME=%I" ExecStart=/usr/sbin/tomcat start ExecStop=/usr/sbin/tomcat stop SuccessExitStatus=143 User=tomcat Group=tomcat
tomcat.conf is severely reduced, if you don't drop sysconfig/tomcat@NAME it means that catalina.base is /var/lib/tomcats/NAME (while .home is still /usr/share/tomcat, so binaries and libraries are from there)
execstop is needed because this thing shuts down through the message on shutdown port, but then we still have systemd kill whatever is accidentally left.
In the end I think the best option is still to have a wrapper script but it's much better than what it was before I started.
I'll post the scripts and stuff shortly.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #10 from Paul P Komkoff Jr i@stingr.net --- I have posted a git tree with changes to Ivan. There's still jsvc question, which I'll try solving in the next pull.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
Ade Lee alee@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |alee@redhat.com, | |dknox@redhat.com, | |nkinder@redhat.com
--- Comment #11 from Ade Lee alee@redhat.com --- The changes that are suggested here (and which presumably have been checked into f20) break Dogtag installation and as a result IPA.
When a dogtag instance is created, we create a clone of the tomcat service as described in the description below (from /usr/sbin/tomcat-sysd)
# This script provides systemd activation of the tomcat service # To create clones of this service: # 1) SERVICE_NAME must be defined before calling this script # 2) Create /etc/sysconfig/${SERVICE_NAME} from /etc/sysconfig/tomcat # to override tomcat defaults
# SERVICE_NAME is a required value only if the service name is # different from 'tomcat' #
In the file /etc/sysconfig/${SERVICE_NAME}, we override a bunch of defaults including CATALINA_BASE, CATALINA_TMPDIR, TOMCAT_LOG, JAVA_OPTS etc.
I would strongly prefer not to change this for Fedora 19/18. The changes you are envisioning will break all existing IPA and dogtag installs out there, unless you leave in place the existing scripts for backward compatibility.
I am open to changing how we configure things on F20 - but I need a clear explanation of exactly how things need to be changed.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #12 from Jóhann B. Guðmundsson johannbg@gmail.com --- afaik my proposed changes have not been implemented...
https://bugzilla.redhat.com/show_bug.cgi?id=842346
Ivan Afonichev ivan.afonichev@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Version|19 |20
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #13 from Ade Lee alee@redhat.com --- OK great -- so I am trying out tomcat-7.0.42-3 on fedora 20.
Not sure what has been implemented and what has not. As I understand it, I need to do the following:
If my instance name is tomcat28: 1. Create a file /etc/sysconfig/tomcat@tomcat28 with my overrides. Not that in this script, I do override CATALINA_BASE.
2. From my init scripts, call /usr/sbin/tomcat start tomcat28
When I do (2), I see that the init script attempts to run
systemctl start tomcat@tomcat28.service
The config file in step 1 appears to be sourced and I can see some of my overrides. One of the overrides though, is TOMCAT_USER, which appears not to be working as I get errors like:
Sep 10 04:10:24 localhost server: java.io.FileNotFoundException: /var/lib/pki/tomcat35/conf/server.xml (Permission denied)
Secondly, if I select a service name like pk-tomcat28 - then it appears that the name is decomposed into pki/tomcat28. This will definitely not work for us. The default instance name we use is pki-tomcat.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #14 from Ivan Afonichev ivan.afonichev@gmail.com --- User/Group is set by systemd and should be overriden in .service as described https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add... It is recomended way according to https://fedoraproject.org/wiki/Packaging:Systemd#EnvironmentFiles_and_suppor...
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #15 from Paul P Komkoff Jr i@stingr.net --- Do you have a specific need to put tomcat in /var/lib/pki/tomcat35? Because how it's used by default is - if you don't create any file in sysconfig and just start an instance tomcat@name is it'll set catalina_base to /var/lib/tomcats/name. However if you override you can just CATALINA_BASE=/var/lib/pki/tomcat35. Now, configs. As we run separate instances and they need separate configs, you need conf directory and in it server.xml and some other configs, look at /usr/libexec/tomcat/{preamble,server} to see how it loads them from /etc/tomcat vs. CATALINA_BASE/conf.
Ah, you also need to create conf,logs,webapps,work in your instance CATALINA_BASE
I understand this is poorly documented, but I guess we can fix it together; I was trying to solve an immediate problem of running this thing and didn't care that much about documentation. I guess we can add some tools like tomcat-create-instance that will do all the directory creation/file copying and also make it work if you just start a named service - to create missing directories.
Johann: I've implemented your proposed changes. So tomcat is not a forking service - it's a simple service now. We still need to have ExecStop because we're stopping it properly, by telling tomcat to shutdown instead of just killing it.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #16 from Ivan Afonichev ivan.afonichev@gmail.com --- systemd instance names seems to be escaped. '-' is used to be able to use '/' in .service filename which is restricted. '-' is converted to '/' '\x2d' is converted to '-'
http://www.freedesktop.org/software/systemd/man/systemd.unit.html "Basically, given a path, "/" is replaced by "-", and all unprintable characters and the "-" are replaced by C-style "\x20" escapes."
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #17 from Ade Lee alee@redhat.com --- OK, after a lot of experimentation, I've figured out the best and simplest option for me, which is to simply modify my own unit files (the unit files for pki-tomcat@.service) to be similar to what you currently have. This way, I will not have to create any new unit or config files.
For reference, my /lib/systemd/system/pki-tomcatd@.service file will look like:
[Unit] Description=PKI Tomcat Server %i After=pki-tomcatd.target syslog.target network.target PartOf=pki-tomcatd.target
[Service] Type=simple EnvironmentFile=/etc/tomcat/tomcat.conf Environment="NAME=%i" EnvironmentFile=-/etc/sysconfig/%i ExecStart=/usr/libexec/tomcat/server start ExecStop=/usr/libexec/tomcat/server stop SuccessExitStatus=143 User=pkiuser Group=pkiuser
[Install] WantedBy=multi-user.target
A couple changes you might want to consider: 1. Adding an identifier %i to the description, so you know which unit file has been executed.
2. Using %i instead of %I for the NAME variable. This will allow names with a dash in them to work correctly. Names as a path with relevant substitutions make no sense for instance names.
One more thing. When I use this unit file (or even the tomcat@.service file you propose), all my startup output ends up going to /var/log/messages instead of catalina.out. How do I fix that?
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #18 from Ade Lee alee@redhat.com --- Any details on how to fix where the server output is going? Right now, its going into /var/log/messages instead of catalina.out.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #19 from Paul P Komkoff Jr i@stingr.net --- I didn't break it by accident - I thought that having just > catalina.out isn't really manageable, as you can't really rotate it, so I directed all output to journal so you can do systemctl status tomcat@foo and see it. Or journalctl -u tomcat@foo.
If you want logging to a specific file here, it might be overall better way to set it up through a syslog.
If you really want it to be old-school though, we could change /usr/libexec/tomcat/server to allow it to be configured.
(but I like the journal approach better)
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #20 from Ade Lee alee@redhat.com --- We can leave it as journal for now, although I reserve the right to ask for it to be changed later.
I did notice a typo in the /usr/libexec/tomcat/server
-Djava.security.policy=="${CATALINA_BASE}/conf/catalina.policy""
Notice the ==. Should be =. With ==, the policy is not actually loaded. You can see this by appending -Djava.security.debug=all to the JAVA_OPTS.
With =, the policies actually load.
Notwithstanding, though, when you try to use the security manager, you get the following exception:
access: access denied ("java.util.PropertyPermission" "java.util.logging.config.class" "read")
Not sure why that is - considering that the rule appears to be in the policy. This could be a separate bug, but it did not occur for previous versions of tomcat.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #21 from Ade Lee alee@redhat.com --- Any comments on the security manager issues above? We'd like to get these resolved before f20 release if possible.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #22 from Paul P Komkoff Jr i@stingr.net --- Sorry for the delay, I'm trying to deal with too many things at once; it'll be nice though if Ivan will help with at least the type you've found :)
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #23 from Ivan Afonichev ivan.afonichev@gmail.com --- Sorry for delay I thought you was loading your custom security manager. "-Djava.security.policy==" could be found in tomcat6 too. http://pkgs.fedoraproject.org/cgit/tomcat6.git/tree/tomcat6-6.0.wrapper?h=f1...
Merging Paul's changes I saw "==", but when I find it in old scripts I decided that it is correct.
I'll check upstrem scripts, how it is loaded there.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #24 from Ivan Afonichev ivan.afonichev@gmail.com --- I've checked upstrem catalina.sh and there is also -Djava.security.policy=="$CATALINA_BASE/conf/catalina.policy" \
You can find docs about loading security policies here http://docs.oracle.com/javase/7/docs/technotes/guides/security/PolicyFiles.h... " (note the double equals) then just the specified policy file will be used; all the ones indicated in the security properties file will be ignored. "
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #25 from Ade Lee alee@redhat.com --- Using the double equals is fine, but the line is still incorrect because the entry is supposed to be a URL. From the link you pointed to above:
java -Djava.security.manager -Djava.security.policy==someURL SomeApp
As the code currently stands, a MalFormedURLException (no protocol) is thrown when the security manager attempts to load the policy.
If the following is used in /usr/libexec/tomcat/server instead, then the policy is actually loaded correctly:
if [ "${SECURITY_MANAGER}" = "true" ] ; then OPTIONS="${OPTIONS} \ -Djava.security.manager \ -Djava.security.policy==file:${CATALINA_BASE}/conf/catalina.policy" fi
Note the removal of the quotes and the addition of file:
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #26 from Ade Lee alee@redhat.com --- Any update on this?
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #27 from Ivan Afonichev ivan.afonichev@gmail.com --- Upstream catalina.sh has -Djava.security.policy=="$CATALINA_BASE"/conf/catalina.policy \
I've searched the Internet and didn't find any bugreports for this issue in catalina.sh
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #28 from Ade Lee alee@redhat.com --- Ivan,
Have you tried to run a basic tomcat app with security manager enabled? My guess is that you will get policy related exceptions, and if you turn on -Djava.security.debug=all (or maybe even if you do not), you will see the MalFormedURLException I mention above.
I'm not sure whats going on upstream, but in this environment, security manager just does not work with this init script.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #29 from Ade Lee alee@redhat.com --- Ivan,
It turns out that the file: part is not necessary. The following also works:
-Djava.security.policy==${CATALINA_BASE}/conf/catalina.policy"
compared to what is currently there:
-Djava.security.policy=="${CATALINA_BASE}/conf/catalina.policy""
You can easily see this bug. You dont even need a webapp. Simply enable security manager on the default tomcat instance and restart the instance.
I did this by adding the following to /etc/sysconfig/tomcat
SECURITY_MANAGER="true" JAVA_OPTS="-Djava.security.debug=all"
and then restarting tomcat : systemctl start tomcat.service
The tomcat service FAILS TO COME UP and there is an error in the log indicating a Malformed URL for the policy.
So, upstream is right -- what is wrong is the invocation in this script.
This is a clear and simply reproducible bug. Its also going to cause all kinds of problems for IPA and Dogtag users when they upgrade to Fedora 20. We need a fix for this ASAP.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #30 from Ivan Afonichev ivan.afonichev@gmail.com --- Oh Thanks a lot for finding real root cause. I'll fix it within 7.0.47 update. Current ETA is this weekend.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #31 from Ivan Afonichev ivan.afonichev@gmail.com --- I've found the issue. In upstream's catalina.sh tomcat is running via eval which seems to remove quotation marks.
https://bugzilla.redhat.com/show_bug.cgi?id=842346
Fedora Update System updates@fedoraproject.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |MODIFIED
https://bugzilla.redhat.com/show_bug.cgi?id=842346
--- Comment #32 from Fedora Update System updates@fedoraproject.org --- tomcat-7.0.47-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/tomcat-7.0.47-1.fc20
https://bugzilla.redhat.com/show_bug.cgi?id=842346
Fedora Update System updates@fedoraproject.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |ON_QA
--- Comment #33 from Fedora Update System updates@fedoraproject.org --- Package tomcat-7.0.47-1.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing tomcat-7.0.47-1.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-20684/tomcat-7.0.47-1.fc... then log in and leave karma (feedback).
https://bugzilla.redhat.com/show_bug.cgi?id=842346
Fedora Update System updates@fedoraproject.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |CLOSED Fixed In Version| |tomcat-7.0.47-1.fc20 Resolution|--- |ERRATA Last Closed| |2013-11-10 02:20:34
--- Comment #34 from Fedora Update System updates@fedoraproject.org --- tomcat-7.0.47-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
java-sig-commits@lists.fedoraproject.org
-
Red Hat Bugzilla