JVM consomme tout le CPU, la plupart des threads sont BLOQUÉS. Bug JVM?
La nuit dernière, un serveur (JBoss 5.1 GA, Java(TM) SE Runtime Environment (build 1.6.0_20-b02),
Java HotSpot(TM) 64-Bit Server VM, exécuté dans une machine virtuelle Linux sous VMWare) a soudainement commencé à utiliser 100% du CPU. L'application est assez typique J2EE applications d'entreprise exécution de la Couture, rien de particulier à ce sujet. La charge a été extrêmement faible à l'époque.
J'ai réussi à obtenir un thread dump avant que le processus a été tué et redémarré (l'application doit être disponible). JStack avait une exception dans son de détection de blocage; en dehors de cela la plupart des discussions ont été BLOQUÉ.
Est-ce une JVM bug? Je n'ai pas vu avant. Après un redémarrage, tout était ok, comme dans les quelques mois depuis que la demande a été utilisé sur le serveur (avec mises à jour périodiques).
Merci pour toutes les suggestions.
Légèrement modifié thread dump (afin de rentrer dans le post de limite de taille) est ci-dessous. J'ai enlevé quelques identique threads (à partir des différents pools de threads à l'intérieur de JBoss/Couture) et n'ont laissé que les lignes du haut de la pile des traces.
EDIT: plein de thread dump à http://pastie.org/1083984
[~]$ jstack -F 13553
Attaching to process ID 13553, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.3-b02
Deadlock Detection:
java.lang.NullPointerException
at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426)
at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137)
at sun.jvm.hotspot.oops.Oop.isA(Oop.java:92)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Can't print deadlocks:null
Thread 26208: (state = BLOCKED)
Thread 25250: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 25249: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 21240: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame)
Thread 21140: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame)
Thread 21139: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame)
Thread 21138: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame)
Thread 19380: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 19377: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 19361: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
Thread 19343: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 19317: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 18995: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
Thread 18986: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
Thread 17659: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 17658: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 17653: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 17631: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 14977: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 14796: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 14609: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 14437: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 14248: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 14232: (state = BLOCKED)
- java.util.Arrays.copyOf(char[], int) @bci=1, line=2882 (Compiled frame)
- java.lang.AbstractStringBuilder.expandCapacity(int) @bci=33, line=100 (Compiled frame)
Thread 14040: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384 (Interpreted frame)
Thread 13858: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame)
Thread 13815: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame)
Thread 13814: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
Thread 13813: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame)
Thread 13812: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.utilities.AssertionFailure: range check
at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
at sun.jvm.hotspot.runtime.RegisterMap.<init>(RegisterMap.java:109)
at sun.jvm.hotspot.runtime.x86.X86RegisterMap.<init>(X86RegisterMap.java:39)
at sun.jvm.hotspot.runtime.x86.X86RegisterMap.clone(X86RegisterMap.java:43)
at sun.jvm.hotspot.runtime.VFrame.<init>(VFrame.java:37)
at sun.jvm.hotspot.runtime.JavaVFrame.<init>(JavaVFrame.java:45)
at sun.jvm.hotspot.runtime.CompiledVFrame.<init>(CompiledVFrame.java:43)
at sun.jvm.hotspot.runtime.VFrame.newVFrame(VFrame.java:77)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:129)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:231)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Thread 13811: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
Thread 13810: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
Thread 13809: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71, line=866 (Interpreted frame)
Thread 13808: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71, line=866 (Interpreted frame)
Thread 13807: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.jboss.bootstrap.AbstractServerImpl$LifeThread.run() @bci=11, line=866 (Interpreted frame)
Thread 13806: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 13805: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 13804: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame)
Thread 13803: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame)
Thread 13802: (state = BLOCKED)
- java.util.Arrays.copyOfRange(char[], int, int) @bci=40, line=3209 (Compiled frame)
- java.lang.String.<init>(char[], int, int) @bci=65, line=216 (Compiled frame)
- java.lang.StringBuffer.toString() @bci=13, line=585 (Compiled frame)
- org.apache.log4j.PatternLayout.format(org.apache.log4j.spi.LoggingEvent) @bci=68, line=506 (Compiled frame)
Thread 13784: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame)
- org.jboss.el.util.ReferenceCache$ReferenceQueueRunner.run() @bci=1, line=159 (Interpreted frame)
Thread 13753: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.jboss.ejb3.cache.simple.SimpleStatefulCache$SessionTimeoutTask.block() @bci=11, line=210 (Interpreted frame)
Thread 13752: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run() @bci=31, line=167 (Interpreted frame)
Thread 13749: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.jboss.remoting.transport.socket.ServerThread.run() @bci=209, line=284 (Interpreted frame)
Thread 13734: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame)
Thread 13701: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
Thread 13700: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.jboss.console.plugins.AOPLister$RefreshPoller.run() @bci=41, line=898 (Interpreted frame)
Thread 13699: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run() @bci=22, line=1590 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 13698: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
Thread 13678: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=198 (Interpreted frame)
Thread 13554: (state = BLOCKED)
Thread 13560: (state = BLOCKED)
Thread 13559: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame)
Thread 13558: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
source d'informationauteur wishihadabettername
Vous devez vous connecter pour publier un commentaire.
Une possibilité est que vous avez manqué de mémoire. Si vous exécutez hors de l'espace du tas de la JVM va commencer à brûler, le CPU à un rythme extraordinaire de faire frénétique de collecte des ordures.
Une autre possibilité est un disque ou d'une erreur de RAM. Ils sont très rares, surtout si vous utilisez de la RAM ECC, SAS ou SCSI qui contiennent la correction des erreurs de transfert de données. Mais ça peut arriver!
J'ai lu sur certains vraiment étrange bugs vu par IBM et Sun noyau d'ingénieurs faisant de soutien pour les entreprises de grande taille qu'ils ne peuvent le blâme sur des trucs comme ça.
Cela ne vous aidera pas beaucoup, d'autres que de vous laisser savoir que vous ne peut jamais découvrir la vraie raison de ce qui s'est passé et il ne peut jamais se reproduire pendant des années. 🙂