Hi!
Am I still the only one on the planet using Pg9.2 as the backend? It constantly grows the number of open files, the only workaroung I have found, that I restart the RHQ servers once in every day - the two server restart are shifted 12 hours. The number of open files ranges between approximately 6K and 12K after and before restart.
Yes, I use postgres plugin - but I have tried without it, and there's no difference at all. It is clear for me thet the server itself causes the problem.
Should I really use another version of Pg? Or a different Database? What's the recommendation?
Regards,
Attila
p.s. It is a fresh install... Debian 7, as the previous one, Pg is from Pg repo, RHQ is 4.9
Le 05/11/2013 22:21, Attila Heidrich a écrit :
Hi!
Am I still the only one on the planet using Pg9.2 as the backend? It constantly grows the number of open files, the only workaroung I have found, that I restart the RHQ servers once in every day - the two server restart are shifted 12 hours. The number of open files ranges between approximately 6K and 12K after and before restart.
Yes, I use postgres plugin - but I have tried without it, and there's no difference at all. It is clear for me thet the server itself causes the problem.
Should I really use another version of Pg? Or a different Database? What's the recommendation?
Regards,
Attila
p.s. It is a fresh install... Debian 7, as the previous one, Pg is from Pg repo, RHQ is 4.9
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Hi Attila,
Please read my previous reply to the same topic.
https://lists.fedorahosted.org/pipermail/rhq-users/2013-September/000911.htm...
I use Pg9.2 (for development) and have no problem of this kind. But there are many parameters which can interfere (load, sysconfig parameters,... etc).
Most likely you hit Bug 1009640 - "JDBC connections leaked during baseline calculations". We can help you rebuild RHQ 4.9 with the necessary patch, or build the master branch.
One good way to determine if the version of Postgres is the problem is to install Pg 9.1 on your system. As you're already pulling the database server binaries from Pg Debian repository, this should not be too difficult.
Regards, Thomas
Thanks a lot, and you are absolutely right with this patch thing...
I guess there is a complete build with this patch already applied, isn't there? It looks very time consuming to build an app like this, is it really necessary? Stg similar that it was to build a new Linux kernel on my 386SX/256M back in 1985. :)
Anyway, I have found several pages with build instruction, please help me which is the recommended one to follow, and how can I select the correct brach to build with the proper patch!
Regards,
Attila
2013/11/7 Thomas Segismont tsegismo@redhat.com
Le 05/11/2013 22:21, Attila Heidrich a écrit :
Hi!
Am I still the only one on the planet using Pg9.2 as the backend? It constantly grows the number of open files, the only workaroung I have found, that I restart the RHQ servers once in every day - the two server restart are shifted 12 hours. The number of open files ranges between approximately 6K and 12K after and before restart.
Yes, I use postgres plugin - but I have tried without it, and there's no difference at all. It is clear for me thet the server itself causes the problem.
Should I really use another version of Pg? Or a different Database? What's the recommendation?
Regards,
Attila
p.s. It is a fresh install... Debian 7, as the previous one, Pg is from Pg repo, RHQ is 4.9
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Hi Attila,
Please read my previous reply to the same topic.
https://lists.fedorahosted.org/pipermail/rhq-users/2013- September/000911.html
I use Pg9.2 (for development) and have no problem of this kind. But there are many parameters which can interfere (load, sysconfig parameters,... etc).
Most likely you hit Bug 1009640 - "JDBC connections leaked during baseline calculations". We can help you rebuild RHQ 4.9 with the necessary patch, or build the master branch.
One good way to determine if the version of Postgres is the problem is to install Pg 9.1 on your system. As you're already pulling the database server binaries from Pg Debian repository, this should not be too difficult.
Regards, Thomas _______________________________________________ rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
The problem is not caused by pg 9.2, but rhq 4.9. Is it possible to downgrade RHQ with the same database content? I try to build a new version, but if I fail to finish today, it seems easier to downgrade, but I do not really want to loose sll the measurements collected until now.
Attila
Le 08/11/2013 09:55, Attila Heidrich a écrit :
The problem is not caused by pg 9.2, but rhq 4.9.
It could be something we rely on which has changed between 9.1 and 9.2. It's unlikely but possible. If you can dump your data and re-import it in a 9.1 server you'll know tomorrow if Postgres version has something to do with your problem.
Is it possible to downgrade RHQ with the same database content? I try to
No, it isn't.
build a new version, but if I fail to finish today, it seems easier to downgrade, but I do not really want to loose sll the measurements collected until now.
Everything you need to know to build RHQ is in the wiki: https://docs.jboss.org/author/display/RHQ/Building+RHQ
If anything is missing, feel free to ask, here or in #rhq@freenode
Attila
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Cheers
The build really took soma time, but the problem isn't that, but the result:
[INFO] Creating ../../../../dev-container/rhq-server/bin/rhq-storage.properties [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] RHQ Enterprise Server Application Server Container FAILURE [43.065s] [INFO] RHQ Server JAR Integration Tests .................. SKIPPED [INFO] RHQ Code Coverage ................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 49.295s [INFO] Finished at: Fri Nov 08 13:47:07 CET 2013 [INFO] Final Memory: 66M/247M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "enterprise" could not be activated because it does not exist. [ERROR] Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory)
While: root@rhq-test:~/projects/rhq# ls -l dev-container/rhq-server/bin/rhq-storage.properties -rw-r--r-- 1 root root 5429 Nov 8 13:47 dev-container/rhq-server/bin/rhq-storage.properties
Attila
2013/11/8 Thomas Segismont tsegismo@redhat.com
Le 08/11/2013 09:55, Attila Heidrich a écrit :
The problem is not caused by pg 9.2, but rhq 4.9.
It could be something we rely on which has changed between 9.1 and 9.2. It's unlikely but possible. If you can dump your data and re-import it in a 9.1 server you'll know tomorrow if Postgres version has something to do with your problem.
Is it possible to downgrade RHQ with the same database content? I try to
No, it isn't.
build a new version, but if I fail to finish today, it seems easier to
downgrade, but I do not really want to loose sll the measurements collected until now.
Everything you need to know to build RHQ is in the wiki: https://docs.jboss.org/author/display/RHQ/Building+RHQ
If anything is missing, feel free to ask, here or in #rhq@freenode
Attila
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Cheers
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Le 08/11/2013 14:20, Attila Heidrich a écrit :
The build really took soma time, but the problem isn't that, but the result:
[INFO] Creating ../../../../dev-container/rhq-server/bin/rhq-storage.properties [INFO]
[INFO] Reactor Summary: [INFO] [INFO] RHQ Enterprise Server Application Server Container FAILURE [43.065s] [INFO] RHQ Server JAR Integration Tests .................. SKIPPED [INFO] RHQ Code Coverage ................................. SKIPPED [INFO]
[INFO] BUILD FAILURE [INFO]
[INFO] Total time: 49.295s [INFO] Finished at: Fri Nov 08 13:47:07 CET 2013 [INFO] Final Memory: 66M/247M [INFO]
[WARNING] The requested profile "enterprise" could not be activated because it does not exist. [ERROR] Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory)
While: root@rhq-test:~/projects/rhq# ls -l dev-container/rhq-server/bin/rhq-storage.properties -rw-r--r-- 1 root root 5429 Nov 8 13:47 dev-container/rhq-server/bin/rhq-storage.properties
Attila
Attila,
What did you set in your Maven settings file for the rhq.rootDir and rhq.containerDir ?
Try with: <rhq.rootDir><your_home>/projects/rhq</rhq.rootDir> and rhq.containerDir unset.
One other thing: you're building a dev-container. This is useful for RHQ hacking but what you need is a distribution build (dev profile deactivated).
Thomas
I haven't set anything till now... now I have set the recommended values... results are the same after
mvn -Penterprise,dev -Ddbsetup -DskipTests install -rf :rhq-enterprise-server-appserver
or what's recommended after changing the settings.xml?
A
2013/11/8 Thomas Segismont tsegismo@redhat.com
Le 08/11/2013 14:20, Attila Heidrich a écrit :
The build really took soma time, but the problem isn't that, but the
result:
[INFO] Creating ../../../../dev-container/rhq-server/bin/rhq-storage.properties [INFO]
[INFO] Reactor Summary: [INFO] [INFO] RHQ Enterprise Server Application Server Container FAILURE [43.065s] [INFO] RHQ Server JAR Integration Tests .................. SKIPPED [INFO] RHQ Code Coverage ................................. SKIPPED [INFO]
[INFO] BUILD FAILURE [INFO]
[INFO] Total time: 49.295s [INFO] Finished at: Fri Nov 08 13:47:07 CET 2013 [INFO] Final Memory: 66M/247M [INFO]
[WARNING] The requested profile "enterprise" could not be activated because it does not exist. [ERROR] Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory)
While: root@rhq-test:~/projects/rhq# ls -l dev-container/rhq-server/bin/rhq-storage.properties -rw-r--r-- 1 root root 5429 Nov 8 13:47 dev-container/rhq-server/bin/rhq-storage.properties
Attila
Attila,
What did you set in your Maven settings file for the rhq.rootDir and rhq.containerDir ?
Try with: <rhq.rootDir><your_home>/projects/rhq</rhq.rootDir> and rhq.containerDir unset.
One other thing: you're building a dev-container. This is useful for RHQ hacking but what you need is a distribution build (dev profile deactivated).
Thomas
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
anyway... why is there the profile "enterprise" in the building sample, since there's nothing like that in the xml!
A
2013/11/8 Attila Heidrich attila.heidrich@gmail.com
I haven't set anything till now... now I have set the recommended values... results are the same after
mvn -Penterprise,dev -Ddbsetup -DskipTests install -rf :rhq-enterprise-server-appserver
or what's recommended after changing the settings.xml?
A
2013/11/8 Thomas Segismont tsegismo@redhat.com
Le 08/11/2013 14:20, Attila Heidrich a écrit :
The build really took soma time, but the problem isn't that, but the
result:
[INFO] Creating ../../../../dev-container/rhq-server/bin/rhq-storage.properties [INFO]
[INFO] Reactor Summary: [INFO] [INFO] RHQ Enterprise Server Application Server Container FAILURE [43.065s] [INFO] RHQ Server JAR Integration Tests .................. SKIPPED [INFO] RHQ Code Coverage ................................. SKIPPED [INFO]
[INFO] BUILD FAILURE [INFO]
[INFO] Total time: 49.295s [INFO] Finished at: Fri Nov 08 13:47:07 CET 2013 [INFO] Final Memory: 66M/247M [INFO]
[WARNING] The requested profile "enterprise" could not be activated because it does not exist. [ERROR] Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory)
While: root@rhq-test:~/projects/rhq# ls -l dev-container/rhq-server/bin/rhq-storage.properties -rw-r--r-- 1 root root 5429 Nov 8 13:47 dev-container/rhq-server/bin/rhq-storage.properties
Attila
Attila,
What did you set in your Maven settings file for the rhq.rootDir and rhq.containerDir ?
Try with: <rhq.rootDir><your_home>/projects/rhq</rhq.rootDir> and rhq.containerDir unset.
One other thing: you're building a dev-container. This is useful for RHQ hacking but what you need is a distribution build (dev profile deactivated).
Thomas
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
OK, the clean build was succesfull, what's next? How can I install this new build?
A
2013/11/8 Attila Heidrich attila.heidrich@gmail.com
anyway... why is there the profile "enterprise" in the building sample, since there's nothing like that in the xml!
A
2013/11/8 Attila Heidrich attila.heidrich@gmail.com
I haven't set anything till now... now I have set the recommended values... results are the same after
mvn -Penterprise,dev -Ddbsetup -DskipTests install -rf :rhq-enterprise-server-appserver
or what's recommended after changing the settings.xml?
A
2013/11/8 Thomas Segismont tsegismo@redhat.com
Le 08/11/2013 14:20, Attila Heidrich a écrit :
The build really took soma time, but the problem isn't that, but the
result:
[INFO] Creating ../../../../dev-container/rhq-server/bin/rhq-storage.properties [INFO]
[INFO] Reactor Summary: [INFO] [INFO] RHQ Enterprise Server Application Server Container FAILURE [43.065s] [INFO] RHQ Server JAR Integration Tests .................. SKIPPED [INFO] RHQ Code Coverage ................................. SKIPPED [INFO]
[INFO] BUILD FAILURE [INFO]
[INFO] Total time: 49.295s [INFO] Finished at: Fri Nov 08 13:47:07 CET 2013 [INFO] Final Memory: 66M/247M [INFO]
[WARNING] The requested profile "enterprise" could not be activated because it does not exist. [ERROR] Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute (storage-setup) on project rhq-enterprise-server-appserver: Execution storage-setup of goal org.codehaus.gmaven:groovy-maven-plugin:2.0-SNAPSHOT:execute failed: java.io.FileNotFoundException: ../../../../dev-container/rhq-server/bin/rhq-storage.properties (No such file or directory)
While: root@rhq-test:~/projects/rhq# ls -l dev-container/rhq-server/bin/rhq-storage.properties -rw-r--r-- 1 root root 5429 Nov 8 13:47 dev-container/rhq-server/bin/rhq-storage.properties
Attila
Attila,
What did you set in your Maven settings file for the rhq.rootDir and rhq.containerDir ?
Try with: <rhq.rootDir><your_home>/projects/rhq</rhq.rootDir> and rhq.containerDir unset.
One other thing: you're building a dev-container. This is useful for RHQ hacking but what you need is a distribution build (dev profile deactivated).
Thomas
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Le 08/11/2013 16:11, Attila Heidrich a écrit :
OK, the clean build was succesfull, what's next? How can I install this new build?
Great!
So you did a dev build, right? I saw you set the dev profile... The dev profile does not give a distribution zip, but rather creates a development system in the "dev-container" directory.
From your ~/projects/rhq directory, try:
mvn -Penterprise,-dev -DskipTests install -pl :rhq-enterprise-server-appserver
You should get this artifact: ~/projects/rhq/modules/enterprise/server/appserver/target/rhq-server-4.10.0-SNAPSHOT.zip
As you already installed RHQ4.9 you should know what to do with it ;)
Seriously, if you have any question, keep on posting.
Thomas
Le 08/11/2013 16:35, Thomas Segismont a écrit :
Le 08/11/2013 16:11, Attila Heidrich a écrit :
OK, the clean build was succesfull, what's next? How can I install this new build?
Great!
So you did a dev build, right? I saw you set the dev profile... The dev profile does not give a distribution zip, but rather creates a development system in the "dev-container" directory.
From your ~/projects/rhq directory, try:
mvn -Penterprise,-dev -DskipTests install -pl :rhq-enterprise-server-appserver
You should get this artifact: ~/projects/rhq/modules/enterprise/server/appserver/target/rhq-server-4.10.0-SNAPSHOT.zip
As you already installed RHQ4.9 you should know what to do with it ;)
Seriously, if you have any question, keep on posting.
Thomas
Attila,
Note that the GUI module which will be included in the distribution zip was built while you had the dev profile activated. So depending on the gwt.* elements you set in your Maven settings file, the GUI might not work with some browsers.
See https://docs.jboss.org/author/display/RHQ/Advanced+Build+Notes#AdvancedBuild...
Thomas
Thanks again helping so much and so quick!
Installation seems OK, but the followings: 00:26:26,113 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] App server has been successfully asked to reload its configuration 00:26:26,155 ERROR [org.jboss.remoting.remote.connection] JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 38dcf88e to localhost/127.0.0.1:6999 00:26:27,253 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] Installing RHQ EAR subsystem
"rhqctl install" stops everything at the end. It doesn't really seems to be an error, so I have restarted it again, but I have got errors. Only one for storage node (which actually isn't even an error): INFO [main] 2013-11-10 00:27:44,600 CassandraDaemon.java (line 402) Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
but server.log has more than 6K lines with lots of exceptions and traces. I just don't know whet to look for in it, will you please take a look? https://dl.dropboxusercontent.com/u/15307875/server.log
root@rhq-test:/opt/rhq-server-4.10.0-SNAPSHOT# jps 806 AgentMain 644 jboss-modules.jar 423 CassandraDaemon 1637 Jps
The server is available on port 7080, but ther's no agent data at all, while Partition events show AGENT CONNECT as last event.
In Administration http://127.0.0.1:7082/coregui/#Administration/Storage nodes/Cluster alerts I can see: Unable to render storage node alert view: Unfortunately, there are no associated resources for the available storage nodes. Check if the agents are running on the machines where the storage nodes are deployed.
Regards,
Attila
2013/11/8 Thomas Segismont tsegismo@redhat.com
Le 08/11/2013 16:11, Attila Heidrich a écrit :
OK, the clean build was succesfull, what's next? How can I install this
new build?
Great!
So you did a dev build, right? I saw you set the dev profile... The dev profile does not give a distribution zip, but rather creates a development system in the "dev-container" directory.
From your ~/projects/rhq directory, try:
mvn -Penterprise,-dev -DskipTests install -pl :rhq-enterprise-server- appserver
You should get this artifact: ~/projects/rhq/modules/enterprise/server/appserver/ target/rhq-server-4.10.0-SNAPSHOT.zip
As you already installed RHQ4.9 you should know what to do with it ;)
Seriously, if you have any question, keep on posting.
Thomas _______________________________________________ rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Installation seems OK, but the followings: 00:26:26,113 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] App server has been successfully asked to reload its configuration 00:26:26,155 ERROR [org.jboss.remoting.remote.connection] JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 38dcf88e to localhost/ 127.0.0.1:6999 00:26:27,253 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] Installing RHQ EAR subsystem
That error is not really an error :) Its to be expected. What is happening is after the installer successfully sends all the remote commands to the app server to configure it to run RHQ, it asks the app server to reload its internal components (that is, the app server internals will do something like a shutdown, reload its config, and start back up again - thus picking up all changes the installer did.) Since some of the config changes the installer asked to make are not hot-loaded at runtime, we have to ask the app server to essentially recycle its internals so the changes are picked up. When this happens, the remote connection is disconnected and sometimes you get that error message (though I can't say I remember seeing that in my latest builds, but maybe I just haven't looked for it).
"rhqctl install" stops everything at the end. It doesn't really seems to be an error, so I have restarted it again
Yes, this "feature" of stopping after the install was something that people explicitly asked for. We were asked to ensure "install" doesn't also do a "start" - as apparently, some customers don't expect an install to leave the software running once the install is done. So, now, when you install, the components are shutdown at the end.
If you want to keep the components started after the install, just pass in "--start" on the command line when doing the install - e.g. "rhqctl install --start".
On Nov 9, 2013, at 7:14 PM, John Mazzitelli mazz@redhat.com wrote:
Installation seems OK, but the followings: 00:26:26,113 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] App server has been successfully asked to reload its configuration 00:26:26,155 ERROR [org.jboss.remoting.remote.connection] JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 38dcf88e to localhost/ 127.0.0.1:6999 00:26:27,253 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] Installing RHQ EAR subsystem
That error is not really an error :) Its to be expected. What is happening is after the installer successfully sends all the remote commands to the app server to configure it to run RHQ, it asks the app server to reload its internal components (that is, the app server internals will do something like a shutdown, reload its config, and start back up again - thus picking up all changes the installer did.) Since some of the config changes the installer asked to make are not hot-loaded at runtime, we have to ask the app server to essentially recycle its internals so the changes are picked up. When this happens, the remote connection is disconnected and sometimes you get that error message (though I can't say I remember seeing that in my latest builds, but maybe I just haven't looked for it).
"rhqctl install" stops everything at the end. It doesn't really seems to be an error, so I have restarted it again
Yes, this "feature" of stopping after the install was something that people explicitly asked for. We were asked to ensure "install" doesn't also do a "start" - as apparently, some customers don't expect an install to leave the software running once the install is done. So, now, when you install, the components are shutdown at the end.
If you want to keep the components started after the install, just pass in "--start" on the command line when doing the install - e.g. "rhqctl install --start".
I was opposed to changing the behavior of rhqctl install and was quick to voice my objection. I have found that I actually prefer the new behavior now :)
Thanks for the comments!
My problem with the new build is different. All apps are running, but server interface shows the active agent, but the autodiscovery queue is empty.
There are exceptions in server.log and agent.log, like:
agent.log:
2013-11-11 12:48:56,383 INFO [ResourceDiscoveryComponent.invoker.daemon-147] (org.rhq.plugins.apache.ApacheServerDiscoveryComponent)- SNMPConf directive not found. Skipping SNMP configuration. 2013-11-11 12:48:56,384 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Discovered new platform with 8 child server(s). 2013-11-11 12:48:56,384 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [server] inventory report to Server... 2013-11-11 12:48:56,849 WARN [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Exception caught while executing server discovery scan. [Warning] javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not perform addBatch ... ... Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into RHQ_STORAGE_NODE (ADDRESS, CQL_PORT, CTIME, ERROR_MSG, RESOURCE_OP_HIST_ID, MAINTENANCE_PENDING, MTIME, OPERATION_MODE, RESOURCE_ID, ID) values (' rhq-test.xxxxxxx.hu', 9142, 1384170536686, NULL, NULL, '0', 1384170536686, 'INSTALLED', 10332, 10176) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.postgresql.ds.jdbc23.AbstractJdbc23PooledConnection$StatementHandler.invoke(AbstractJdbc23PooledConnection.java:453) at com.sun.proxy.$Proxy106.executeBatch(Unknown Source) at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:708) at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1077) at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:110)
rhq=# select * from rhq_storage_node; id | address | cql_port | operation_mode | ctime | mtime | error_msg | maintenance_pending | resource_id | resource_op_hist_id ------+--------------------+----------+----------------+---------------+---------------+-----------+---------------------+-------------+--------------------- 1001 | rhq-test.xxxxxx.hu | 9142 | INSTALLED | 1384039582998 | 1384039582998 | | f | |
(1 row)
Attila
2013/11/10 John Mazzitelli mazz@redhat.com
Installation seems OK, but the followings: 00:26:26,113 INFO
[org.rhq.enterprise.server.installer.InstallerServiceImpl]
App server has been successfully asked to reload its configuration 00:26:26,155 ERROR [org.jboss.remoting.remote.connection] JBREM000200:
Remote
connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 38dcf88e to localhost/ 127.0.0.1:6999 00:26:27,253 INFO
[org.rhq.enterprise.server.installer.InstallerServiceImpl]
Installing RHQ EAR subsystem
That error is not really an error :) Its to be expected. What is happening is after the installer successfully sends all the remote commands to the app server to configure it to run RHQ, it asks the app server to reload its internal components (that is, the app server internals will do something like a shutdown, reload its config, and start back up again - thus picking up all changes the installer did.) Since some of the config changes the installer asked to make are not hot-loaded at runtime, we have to ask the app server to essentially recycle its internals so the changes are picked up. When this happens, the remote connection is disconnected and sometimes you get that error message (though I can't say I remember seeing that in my latest builds, but maybe I just haven't looked for it).
"rhqctl install" stops everything at the end. It doesn't really seems to
be
an error, so I have restarted it again
Yes, this "feature" of stopping after the install was something that people explicitly asked for. We were asked to ensure "install" doesn't also do a "start" - as apparently, some customers don't expect an install to leave the software running once the install is done. So, now, when you install, the components are shutdown at the end.
If you want to keep the components started after the install, just pass in "--start" on the command line when doing the install - e.g. "rhqctl install --start". _______________________________________________ rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Hi Attila,
Can you copy the full exception trace to pastebin?
Thanks
Le 11/11/2013 13:08, Attila Heidrich a écrit :
Thanks for the comments!
My problem with the new build is different. All apps are running, but server interface shows the active agent, but the autodiscovery queue is empty.
There are exceptions in server.log and agent.log, like:
agent.log:
2013-11-11 12:48:56,383 INFO [ResourceDiscoveryComponent.invoker.daemon-147] (org.rhq.plugins.apache.ApacheServerDiscoveryComponent)- SNMPConf directive not found. Skipping SNMP configuration. 2013-11-11 12:48:56,384 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Discovered new platform with 8 child server(s). 2013-11-11 12:48:56,384 INFO [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [server] inventory report to Server... 2013-11-11 12:48:56,849 WARN [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Exception caught while executing server discovery scan. [Warning] javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not perform addBatch ... ... Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into RHQ_STORAGE_NODE (ADDRESS, CQL_PORT, CTIME, ERROR_MSG, RESOURCE_OP_HIST_ID, MAINTENANCE_PENDING, MTIME, OPERATION_MODE, RESOURCE_ID, ID) values ('rhq-test.xxxxxxx.hu http://rhq-test.xxxxxxx.hu', 9142, 1384170536686, NULL, NULL, '0', 1384170536686, 'INSTALLED', 10332, 10176) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.postgresql.ds.jdbc23.AbstractJdbc23PooledConnection$StatementHandler.invoke(AbstractJdbc23PooledConnection.java:453) at com.sun.proxy.$Proxy106.executeBatch(Unknown Source) at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:708) at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1077) at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:110)
rhq=# select * from rhq_storage_node; id | address | cql_port | operation_mode | ctime | mtime | error_msg | maintenance_pending | resource_id | resource_op_hist_id ------+--------------------+----------+----------------+---------------+---------------+-----------+---------------------+-------------+--------------------- 1001 | rhq-test.xxxxxx.hu http://rhq-test.xxxxxx.hu | 9142 | INSTALLED | 1384039582998 | 1384039582998 | | f | | (1 row)
Attila
2013/11/10 John Mazzitelli <mazz@redhat.com mailto:mazz@redhat.com>
> Installation seems OK, but the followings: > 00:26:26,113 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] > App server has been successfully asked to reload its configuration > 00:26:26,155 ERROR [org.jboss.remoting.remote.connection] JBREM000200: Remote > connection failed: java.io.IOException: JBREM000202: Abrupt close on > Remoting connection 38dcf88e to localhost/ 127.0.0.1:6999 <http://127.0.0.1:6999> > 00:26:27,253 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] > Installing RHQ EAR subsystem That error is not really an error :) Its to be expected. What is happening is after the installer successfully sends all the remote commands to the app server to configure it to run RHQ, it asks the app server to reload its internal components (that is, the app server internals will do something like a shutdown, reload its config, and start back up again - thus picking up all changes the installer did.) Since some of the config changes the installer asked to make are not hot-loaded at runtime, we have to ask the app server to essentially recycle its internals so the changes are picked up. When this happens, the remote connection is disconnected and sometimes you get that error message (though I can't say I remember seeing that in my latest builds, but maybe I just haven't looked for it). > "rhqctl install" stops everything at the end. It doesn't really seems to be > an error, so I have restarted it again Yes, this "feature" of stopping after the install was something that people explicitly asked for. We were asked to ensure "install" doesn't also do a "start" - as apparently, some customers don't expect an install to leave the software running once the install is done. So, now, when you install, the components are shutdown at the end. If you want to keep the components started after the install, just pass in "--start" on the command line when doing the install - e.g. "rhqctl install --start". _______________________________________________ rhq-users mailing list rhq-users@lists.fedorahosted.org <mailto:rhq-users@lists.fedorahosted.org> https://lists.fedorahosted.org/mailman/listinfo/rhq-users
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Hi!
I hope a long tail of agent.log will do! I am pretty new using pastebin, hope you will see this: http://pastebin.com/aqQufGMz
Currently I can see only this on the web interface / Admin / storage nodes / this server "There is no configuration available for this node. Is the agent running on the rhq-test.aensys.hu?"
The logs are very new since I have reinstalled the whole thing to check if it can be reproduced.
Anyway, I use this command to delete the old inatsllation: rm -rf rhq-server-4.10.0-SNAPSHOT rhq-agent/ rhq-data/ ~/.java/.userPrefs/rhq*
With the recent stable versions there were some storage data in /var/lib/rhq*, but I couldn't see such now.
Attila
after I have deleted the only line from the RHQ_STORAGE_NODE table by hand, the node came up, and so does the agent!
rhq=> select * from RHQ_STORAGE_NODE; id | address | cql_port | operation_mode | ctime | mtime | error_msg | maintenance_pending | resource_id | resource_op_hist_id ------+--------------------+----------+----------------+---------------+---------------+-----------+---------------------+-------------+--------------------- 1001 | rhq-test.aensys.hu | 9142 | INSTALLED | 1384248842490 | 1384248842491 | | f | |
(1 row) rhq=> delete from RHQ_STORAGE_NODE where id=1001; DELETE 1
In no time there is the storage node running, and also there are the resources in the discovery queue.
Attila
2013/11/12 Attila Heidrich attila.heidrich@gmail.com
Hi!
I hope a long tail of agent.log will do! I am pretty new using pastebin, hope you will see this: http://pastebin.com/aqQufGMz
Currently I can see only this on the web interface / Admin / storage nodes / this server "There is no configuration available for this node. Is the agent running on the rhq-test.aensys.hu?"
The logs are very new since I have reinstalled the whole thing to check if it can be reproduced.
Anyway, I use this command to delete the old inatsllation: rm -rf rhq-server-4.10.0-SNAPSHOT rhq-agent/ rhq-data/ ~/.java/.userPrefs/rhq*
With the recent stable versions there were some storage data in /var/lib/rhq*, but I couldn't see such now.
Attila
That's good news. But how come you had a storage node entity already stored in the database?
Le 12/11/2013 14:28, Attila Heidrich a écrit :
after I have deleted the only line from the RHQ_STORAGE_NODE table by hand, the node came up, and so does the agent!
rhq=> select * from RHQ_STORAGE_NODE; id | address | cql_port | operation_mode | ctime | mtime | error_msg | maintenance_pending | resource_id | resource_op_hist_id ------+--------------------+----------+----------------+---------------+---------------+-----------+---------------------+-------------+--------------------- 1001 | rhq-test.aensys.hu http://rhq-test.aensys.hu | 9142 | INSTALLED | 1384248842490 | 1384248842491 | | f | | (1 row) rhq=> delete from RHQ_STORAGE_NODE where id=1001; DELETE 1
In no time there is the storage node running, and also there are the resources in the discovery queue.
Attila
2013/11/12 Attila Heidrich <attila.heidrich@gmail.com mailto:attila.heidrich@gmail.com>
Hi! I hope a long tail of agent.log will do! I am pretty new using pastebin, hope you will see this: http://pastebin.com/aqQufGMz Currently I can see only this on the web interface / Admin / storage nodes / this server "There is no configuration available for this node. Is the agent running on the rhq-test.aensys.hu <http://rhq-test.aensys.hu>?" The logs are very new since I have reinstalled the whole thing to check if it can be reproduced. Anyway, I use this command to delete the old inatsllation: rm -rf rhq-server-4.10.0-SNAPSHOT rhq-agent/ rhq-data/ ~/.java/.userPrefs/rhq* With the recent stable versions there were some storage data in /var/lib/rhq*, but I couldn't see such now. Attila
rhq-users mailing list rhq-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-users
Good question, the database was completely empty before the installation, I have dropped and created again just before starting "rhqctl install"
The log fragment of psql: grep -i rhq_storage /var/log/postgresql/postgresql-9.2-main.log ... 2013-11-12 10:33:54 CET ERROR: table "rhq_storage_node" does not exist 2013-11-12 10:33:54 CET STATEMENT: DROP TABLE RHQ_STORAGE_NODE 2013-11-12 10:33:54 CET ERROR: sequence "rhq_storage_node_id_seq" does not exist 2013-11-12 10:33:54 CET STATEMENT: DROP SEQUENCE RHQ_STORAGE_NODE_ID_SEQ 2013-11-12 10:39:41 CET ERROR: duplicate key value violates unique constraint "rhq_storage_node_unique" ...
in those 6 minutes something has happened...
This is the very first "insert into RHQ_STORAGE_NODE ..." statement in the log, and already breaks the uniqueness!
Attila
2013/11/12 Thomas Segismont tsegismo@redhat.com
That's good news. But how come you had a storage node entity already stored in the database?
Bad news again...
unfortunately the host OS has crashed in the evening, this morning I restarted all, but the RHQ in the VM cannot be started again.
server log looks like this:
13:31:46,837 INFO [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker]] (http-/0.0.0.0:7080-4) ServerInvokerServlet: invokerObjectNameQuery=jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* 13:31:46,838 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] (http-/0.0.0.0:7080-4) JBWEB000235: Allocate exception for servlet ServerInvokerServlet: javax.servlet.ServletException: Could not find the remoting servlet invoker: jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* - need to wait for remoting to be initialized later at org.rhq.enterprise.communications.servlet.ServerInvokerServlet.getInvokerFromInvokerName(ServerInvokerServlet.java:78) [classes:] at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.init(ServerInvokerServlet.java:78) [jboss-remoting-2.5.4.SP4.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45] at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_45] at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) [rt.jar:1.7.0_45] at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:155) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:108) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1188) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:950) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:151) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
root@rhq-test:/opt/rhq-server-4.10.0-SNAPSHOT# bin/rhqctl status 13:34:51,011 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1 RHQ Storage Node (pid 3165 ) IS running RHQ Server (pid 3337 ) IS running JBossAS Java VM child process (pid 3337 ) IS running RHQ Agent (pid 3485 ) IS running
The browser shows an empty page only - after login.
I tried to restart postgres, and / or RHQ several times. Postgres is available, can be accessed with the same user/pass it worked yestarday.
Attila
2013/11/12 Attila Heidrich attila.heidrich@gmail.com
Good question, the database was completely empty before the installation, I have dropped and created again just before starting "rhqctl install"
The log fragment of psql: grep -i rhq_storage /var/log/postgresql/postgresql-9.2-main.log ... 2013-11-12 10:33:54 CET ERROR: table "rhq_storage_node" does not exist 2013-11-12 10:33:54 CET STATEMENT: DROP TABLE RHQ_STORAGE_NODE 2013-11-12 10:33:54 CET ERROR: sequence "rhq_storage_node_id_seq" does not exist 2013-11-12 10:33:54 CET STATEMENT: DROP SEQUENCE RHQ_STORAGE_NODE_ID_SEQ 2013-11-12 10:39:41 CET ERROR: duplicate key value violates unique constraint "rhq_storage_node_unique" ...
in those 6 minutes something has happened...
This is the very first "insert into RHQ_STORAGE_NODE ..." statement in the log, and already breaks the uniqueness!
Attila
2013/11/12 Thomas Segismont tsegismo@redhat.com
That's good news. But how come you had a storage node entity already stored in the database?
rhq-users@lists.stg.fedorahosted.org
-
Attila Heidrich -
John Mazzitelli -
John Sanda -
Thomas Segismont