I am seeing the following error in the server log when an agent trys to connect to the server. The server had been running for quite some time (1-2 weeks) and jconsole showed it out of heap memory. I have restarted the server with the HeapDumpOnOutOfMemoryError java option to capture anything if it happens again. Anybody seen this before?
2011-01-07 13:50:37,001 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1][3.0.0(59e9341)] would like to connect to this server 2011-01-07 13:50:41,350 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1] has connected to this server at Fri Jan 07 13:50:41 EST 2011 2011-01-07 13:50:41,351 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception java.lang.OutOfMemoryError: Java heap space
This repeats over and over every few minutes.
Bala Nair SeaChange International
never saw that before but the server definitely loads data when an agent connects (mainly its the agent condition cache stuff).
Definitely get a heap dump and see what's using all the heap. That would help pinpoint where we'd have to look.
On 01/07/2011 02:57 PM, Bala Nair wrote:
I am seeing the following error in the server log when an agent trys to connect to the server. The server had been running for quite some time (1-2 weeks) and jconsole showed it out of heap memory. I have restarted the server with the HeapDumpOnOutOfMemoryError java option to capture anything if it happens again. Anybody seen this before?
2011-01-07 13:50:37,001 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1][3.0.0(59e9341)] would like to connect to this server 2011-01-07 13:50:41,350 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1] has connected to this server at Fri Jan 07 13:50:41 EST 2011 2011-01-07 13:50:41,351 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception java.lang.OutOfMemoryError: Java heap space
This repeats over and over every few minutes.
Bala Nair SeaChange International
rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
One place where I've seen large amounts of heap get consumed is when using the Content subsystem to install a very large cumulative patch or EAR or WAR. I think we recently converted most (possibly all) places that access package bits to using streams rather than copying the all of the bits into memory. However, there may still be a few places that copy into memory. Are you using RHQ at all for installing/deploying large packages?
On 01/07/2011 02:57 PM, Bala Nair wrote:
I am seeing the following error in the server log when an agent trys to connect to the server. The server had been running for quite some time (1-2 weeks) and jconsole showed it out of heap memory. I have restarted the server with the HeapDumpOnOutOfMemoryError java option to capture anything if it happens again. Anybody seen this before?
2011-01-07 13:50:37,001 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1][3.0.0(59e9341)] would like to connect to this server 2011-01-07 13:50:41,350 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1] has connected to this server at Fri Jan 07 13:50:41 EST 2011 2011-01-07 13:50:41,351 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception java.lang.OutOfMemoryError: Java heap space
This repeats over and over every few minutes.
Bala Nair SeaChange International
rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
I recently hit this same error with CPU pegged at 99%. I could not find a root cause. I have used the content system but only to deploy some small shell scripts.
java.lang.OutOfMemoryError: Java heap space
2010-12-31 13:30:07,383 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [prv1lxws0002.provid.healthconnected.mb.ca][3.0.0(59e9341)] would like to connect to this server
2010-12-31 13:30:09,411 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [reg1lxws0005.manitoba-ehealth.ca] has connected to this server at Fri Dec 31 13:30:09 CST 2010
2010-12-31 13:30:09,411 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss- remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception
java.lang.OutOfMemoryError: Java heap space
Al Amyot | Consultant Sierra Systems
(eHealth): 204-926-4257 (T): 204-942-2575 (F): 204-942-2047 444 St. Mary Avenue, Suite 1050 Winnipeg, MB R3C 3T1
Management Consulting | System Integration | Managed Services website: www.SierraSystems.com
-----Original Message----- From: rhq-users-bounces@lists.fedorahosted.org [mailto:rhq-users-bounces@lists.fedorahosted.org] On Behalf Of Ian Springer Sent: Friday, January 07, 2011 2:52 PM To: rhq-users@lists.fedorahosted.org Subject: Re: out of memory error
One place where I've seen large amounts of heap get consumed is when using the Content subsystem to install a very large cumulative patch or EAR or WAR. I think we recently converted most (possibly all) places that access package bits to using streams rather than copying the all of the bits into memory. However, there may still be a few places that copy into memory. Are you using RHQ at all for installing/deploying large packages?
On 01/07/2011 02:57 PM, Bala Nair wrote:
I am seeing the following error in the server log when an agent trys to connect to the server. The server had been running for quite some time (1-2 weeks) and jconsole showed it out of heap memory. I have restarted the server with the HeapDumpOnOutOfMemoryError java option to capture anything if it happens again. Anybody seen this before?
2011-01-07 13:50:37,001 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1][3.0.0(59e9341)] would like to connect to this server 2011-01-07 13:50:41,350 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1] has connected to this server at Fri Jan 07 13:50:41 EST
2011 2011-01-07 13:50:41,351 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jbos s-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception java.lang.OutOfMemoryError: Java heap space
This repeats over and over every few minutes.
Bala Nair SeaChange International
rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
-- Ian Springer Principal Software Developer JBoss Operations Network Red Hat ian.springer@redhat.com
_______________________________________________ rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
This email and/or any documents in this transmission is intended for the addressee(s) only and may contain legally privileged or confidential information. Any unauthorized use, disclosure, distribution, copying or dissemination is strictly prohibited. If you receive this transmission in error, please notify the sender immediately and return the original.
Ce courriel et tout document dans cette transmission est destiné à la personne ou aux personnes à qui il est adressé. Il peut contenir des informations privilégiées ou confidentielles. Toute utilisation, divulgation, distribution, copie, ou diffusion non autorisée est strictement défendue. Si vous n'êtes pas le destinataire de ce message, veuillez en informer l'expéditeur immédiatement et lui remettre l'original.
We are not using RHQ for deployment, just monitoring. We are running clustered with 2 servers serving 10 agents.
On 1/7/11 3:52 PM, Ian Springer wrote:
One place where I've seen large amounts of heap get consumed is when using the Content subsystem to install a very large cumulative patch or EAR or WAR. I think we recently converted most (possibly all) places that access package bits to using streams rather than copying the all of the bits into memory. However, there may still be a few places that copy into memory. Are you using RHQ at all for installing/deploying large packages?
On 01/07/2011 02:57 PM, Bala Nair wrote:
I am seeing the following error in the server log when an agent trys to connect to the server. The server had been running for quite some time (1-2 weeks) and jconsole showed it out of heap memory. I have restarted the server with the HeapDumpOnOutOfMemoryError java option to capture anything if it happens again. Anybody seen this before?
2011-01-07 13:50:37,001 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1][3.0.0(59e9341)] would like to connect to this server 2011-01-07 13:50:41,350 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1] has connected to this server at Fri Jan 07 13:50:41 EST 2011 2011-01-07 13:50:41,351 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception java.lang.OutOfMemoryError: Java heap space
This repeats over and over every few minutes.
Bala Nair SeaChange International
rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
Ok, that at least rules out that possibility. Your setup does not sound too extreme, so there could be a heap leak somewhere. Once we have a heap dump, we should be able to figure out if that's the case. Once you have one, let us know, and we'll figure out some way for you to get it to us.
On 01/07/2011 05:10 PM, Bala Nair wrote:
We are not using RHQ for deployment, just monitoring. We are running clustered with 2 servers serving 10 agents.
On 1/7/11 3:52 PM, Ian Springer wrote:
One place where I've seen large amounts of heap get consumed is when using the Content subsystem to install a very large cumulative patch or EAR or WAR. I think we recently converted most (possibly all) places that access package bits to using streams rather than copying the all of the bits into memory. However, there may still be a few places that copy into memory. Are you using RHQ at all for installing/deploying large packages?
On 01/07/2011 02:57 PM, Bala Nair wrote:
I am seeing the following error in the server log when an agent trys to connect to the server. The server had been running for quite some time (1-2 weeks) and jconsole showed it out of heap memory. I have restarted the server with the HeapDumpOnOutOfMemoryError java option to capture anything if it happens again. Anybody seen this before?
2011-01-07 13:50:37,001 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1][3.0.0(59e9341)] would like to connect to this server 2011-01-07 13:50:41,350 INFO [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent [aphilLabMMC1] has connected to this server at Fri Jan 07 13:50:41 EST 2011 2011-01-07 13:50:41,351 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception java.lang.OutOfMemoryError: Java heap space
This repeats over and over every few minutes.
Bala Nair SeaChange International
rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
rhq-users mailing list rhq-users@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-users
Hi,
are you using call time data?
If so, this may come in too frequently.
Heiko
rhq-users@lists.stg.fedorahosted.org
-
Al Amyot -
Bala Nair -
Heiko W.Rupp -
Ian Springer -
John Mazzitelli