콘텐츠로 이동

2022 11 06

2022-11-06

ThreadDump 떠보기

  • 참고: https://d2.naver.com/helloworld/10963
  • Java & Thread
    • 수십~수백개의 쓰레드를 사용하여 처리
    • 두 개 이상의 쓰레드가 같은 자원 경합시 데드락 발생할 수도
      • Log 기록하는 것도 공유자원 접근과 비슷
    • Thread Dump를 통해 각 쓰레드의 상태를 알 수 있음
  • 쓰레드 동기화
    • Java에서는 Monitor를 활용해 쓰레드 동기화 진행
    • 모든 Java 객체가 하나의 Monitor를 가짐 => Monitor는 하나의 쓰레드만 소유함
      • 어떤 쓰레드가 소유한 Monitor를 다른 쓰레드가 획득하려면 이미 들고 있는 쓰레드가 Monitor 놔줄때까지 Wait Queue 대기!
  • 스레드 상태
    • java.lang.Thread 클래스엔 State 변수가 있음
      • Enum으로 정의되어 있으며...
        • NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING
  • 스레드 종류
    • Daemon Thread
      • 다른 Non-daemon Thread가 없다면 동작 중지
      • 사용자가 직접 스레드 생성 안하더라도, Java 어플리케이션이 기본적으로 여러개의 쓰레드 생성
        • 대부분이 Daemon Thread => GC, JMX 등의 처리를 하기 위함
    • Non-daemon Thread

  • 스레드 덤프 획득해보기 - jstack 

    [DEV][15:23:26][ubuntu@ip-172-31-34-52 ~/deploy] 
$ jstack 10034
2022-11-06 15:23:30
Full thread dump OpenJDK 64-Bit Server VM (11.0.16+8-post-Ubuntu-0ubuntu118.04 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fca780b9df0, length=36, elements={
0x00007fcaa80b8800, 0x00007fcaa80ba800, 0x00007fcaa80c3800, 0x00007fcaa80c5800,
0x00007fcaa80c7800, 0x00007fcaa80c9800, 0x00007fcaa80cb800, 0x00007fcaa810b000,
0x00007fcaa861a000, 0x00007fcaa8b4e000, 0x00007fcaa8b4f800, 0x00007fcaa98db800,
0x00007fcaa9389000, 0x00007fcaa976c800, 0x00007fcaa97c6000, 0x00007fcaa9cdc000,
0x00007fcaa9cc8800, 0x00007fcaa9ff2000, 0x00007fcaa9e84800, 0x00007fcaaa7cd800,
0x00007fcaaa7cf000, 0x00007fcaaa7d1000, 0x00007fcaaa7d2800, 0x00007fcaaa7d4800,
0x00007fcaaa7d6800, 0x00007fcaaa7d8800, 0x00007fcaaa7da800, 0x00007fcaaa7dc800,
0x00007fcaaa7de800, 0x00007fcaaad6b000, 0x00007fcaaad78800, 0x00007fcaa98f4800,
0x00007fcaa8016800, 0x00007fca741e5000, 0x00007fca742f1800, 0x00007fca780ba000
}

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=13.40ms elapsed=59481.70s tid=0x00007fcaa80b8800 nid=0x2735 waiting on condition  [0x00007fca8f9b4000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.16/Native Method)
    at java.lang.ref.Reference.processPendingReferences(java.base@11.0.16/Reference.java:241)
    at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.16/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.83ms elapsed=59481.70s tid=0x00007fcaa80ba800 nid=0x2736 in Object.wait()  [0x00007fca8f8b3000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.16/Native Method)

    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:155)

    - waiting to re-lock in wait() <0x00000000eb603468> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:176)
    at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.16/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.41ms elapsed=59481.69s tid=0x00007fcaa80c3800 nid=0x2737 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=92.43ms elapsed=59481.69s tid=0x00007fcaa80c5800 nid=0x2738 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=23540.37ms elapsed=59481.69s tid=0x00007fcaa80c7800 nid=0x2739 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=5071.85ms elapsed=59481.69s tid=0x00007fcaa80c9800 nid=0x273a waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=35.52ms elapsed=59481.69s tid=0x00007fcaa80cb800 nid=0x273b runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #9 daemon prio=8 os_prio=0 cpu=80.76ms elapsed=59481.65s tid=0x00007fcaa810b000 nid=0x273d in Object.wait()  [0x00007fca8ee9d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.16/Native Method)

    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:155)

    - waiting to re-lock in wait() <0x00000000eb603c30> (a java.lang.ref.ReferenceQueue$Lock)
    at jdk.internal.ref.CleanerImpl.run(java.base@11.0.16/CleanerImpl.java:148)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)
    at jdk.internal.misc.InnocuousThread.run(java.base@11.0.16/InnocuousThread.java:161)

"java-sdk-http-connection-reaper" #11 daemon prio=5 os_prio=0 cpu=64.41ms elapsed=59476.43s tid=0x00007fcaa861a000 nid=0x273e waiting on condition  [0x00007fca8dd46000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(java.base@11.0.16/Native Method)
    at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:188)

"AsyncAppender-Worker-ASYNC_SLACK" #13 daemon prio=5 os_prio=0 cpu=0.53ms elapsed=59473.46s tid=0x00007fcaa8b4e000 nid=0x2740 waiting on condition  [0x00007fca8c714000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ec350fe0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.16/ArrayBlockingQueue.java:417)
    at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)

"AsyncAppender-Worker-DEV_ASYNC_SLACK" #14 daemon prio=5 os_prio=0 cpu=0.14ms elapsed=59473.46s tid=0x00007fcaa8b4f800 nid=0x2741 waiting on condition  [0x00007fca8c613000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ec351238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.16/ArrayBlockingQueue.java:417)
    at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)

"mysql-cj-abandoned-connection-cleanup" #16 daemon prio=5 os_prio=0 cpu=1116.03ms elapsed=59465.12s tid=0x00007fcaa98db800 nid=0x2742 in Object.wait()  [0x00007fca8d842000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.16/Native Method)

    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:155)

    - waiting to re-lock in wait() <0x00000000ece232a8> (a java.lang.ref.ReferenceQueue$Lock)
    at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:91)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"Catalina-utility-1" #17 prio=1 os_prio=0 cpu=1119.25ms elapsed=59464.61s tid=0x00007fcaa9389000 nid=0x2743 waiting on condition  [0x00007fca8c112000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ecdcac18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1177)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"Catalina-utility-2" #18 prio=1 os_prio=0 cpu=4854.06ms elapsed=59464.60s tid=0x00007fcaa976c800 nid=0x2744 waiting on condition  [0x00007fca6edfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ecdcac18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"container-0" #19 prio=5 os_prio=0 cpu=199.99ms elapsed=59464.49s tid=0x00007fcaa97c6000 nid=0x2745 waiting on condition  [0x00007fca6ecfd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(java.base@11.0.16/Native Method)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:566)
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run(TomcatWebServer.java:197)

"HikariPool-1 housekeeper" #20 daemon prio=5 os_prio=0 cpu=501.79ms elapsed=59463.78s tid=0x00007fcaa9cdc000 nid=0x2746 waiting on condition  [0x00007fca6e78a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ecf68a08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"lettuce-eventExecutorLoop-1-1" #23 daemon prio=5 os_prio=0 cpu=21.11ms elapsed=59455.36s tid=0x00007fcaa9cc8800 nid=0x2749 waiting on condition  [0x00007fca6e689000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000eda87210> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123)
    at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.16/LinkedBlockingQueue.java:458)
    at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:257)
    at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:64)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"lettuce-timer-3-1" #22 daemon prio=5 os_prio=0 cpu=19515.77ms elapsed=59455.36s tid=0x00007fcaa9ff2000 nid=0x274a waiting on condition  [0x00007fca6cd1a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(java.base@11.0.16/Native Method)
    at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"lettuce-timer-5-1" #24 daemon prio=5 os_prio=0 cpu=18590.19ms elapsed=59451.43s tid=0x00007fcaa9e84800 nid=0x274b waiting on condition  [0x00007fca6bdd5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(java.base@11.0.16/Native Method)
    at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-1" #25 daemon prio=5 os_prio=0 cpu=265.55ms elapsed=59450.65s tid=0x00007fcaaa7cd800 nid=0x274c waiting on condition  [0x00007fca6bcd4000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-2" #26 daemon prio=5 os_prio=0 cpu=49.61ms elapsed=59450.65s tid=0x00007fcaaa7cf000 nid=0x274d waiting on condition  [0x00007fca6bbd3000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-3" #27 daemon prio=5 os_prio=0 cpu=38.55ms elapsed=59450.65s tid=0x00007fcaaa7d1000 nid=0x274e waiting on condition  [0x00007fca6bad2000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-4" #28 daemon prio=5 os_prio=0 cpu=68.49ms elapsed=59450.64s tid=0x00007fcaaa7d2800 nid=0x274f waiting on condition  [0x00007fca6b790000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-5" #29 daemon prio=5 os_prio=0 cpu=61.23ms elapsed=59450.64s tid=0x00007fcaaa7d4800 nid=0x2750 waiting on condition  [0x00007fca6b68f000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-6" #30 daemon prio=5 os_prio=0 cpu=62.78ms elapsed=59450.64s tid=0x00007fcaaa7d6800 nid=0x2751 waiting on condition  [0x00007fca6b58e000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-7" #31 daemon prio=5 os_prio=0 cpu=80.69ms elapsed=59450.64s tid=0x00007fcaaa7d8800 nid=0x2752 waiting on condition  [0x00007fca6b48d000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-8" #32 daemon prio=5 os_prio=0 cpu=40.85ms elapsed=59450.64s tid=0x00007fcaaa7da800 nid=0x2753 waiting on condition  [0x00007fca6b38c000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-9" #33 daemon prio=5 os_prio=0 cpu=41.47ms elapsed=59450.64s tid=0x00007fcaaa7dc800 nid=0x2754 waiting on condition  [0x00007fca6b28b000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-exec-10" #34 daemon prio=5 os_prio=0 cpu=65.06ms elapsed=59450.64s tid=0x00007fcaaa7de800 nid=0x2755 waiting on condition  [0x00007fca6b18a000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee47b4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:146)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-Poller" #35 daemon prio=5 os_prio=0 cpu=2640.82ms elapsed=59450.63s tid=0x00007fcaaad6b000 nid=0x2756 runnable  [0x00007fca6b089000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124)

    - locked <0x00000000ee47c7d8> (a sun.nio.ch.Util$2)
    - locked <0x00000000ee47c780> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:136)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:807)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"http-nio-8080-Acceptor" #36 daemon prio=5 os_prio=0 cpu=35.73ms elapsed=59450.61s tid=0x00007fcaaad78800 nid=0x2757 runnable  [0x00007fca6af88000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.ServerSocketChannelImpl.accept0(java.base@11.0.16/Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.16/ServerSocketChannelImpl.java:533)
    at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@11.0.16/ServerSocketChannelImpl.java:285)
    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:546)
    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:79)
    at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"scheduling-1" #37 prio=5 os_prio=0 cpu=173.76ms elapsed=59449.62s tid=0x00007fcaa98f4800 nid=0x2758 waiting on condition  [0x00007fca6ac87000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee3814f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"DestroyJavaVM" #38 prio=5 os_prio=0 cpu=13616.05ms elapsed=59449.55s tid=0x00007fcaa8016800 nid=0x2733 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"logback-1" #51 daemon prio=5 os_prio=0 cpu=13.39ms elapsed=54616.86s tid=0x00007fca741e5000 nid=0x27ff waiting on condition  [0x00007fca6a786000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee761560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"logback-2" #310 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=19410.42s tid=0x00007fca742f1800 nid=0x2eb2 waiting on condition  [0x00007fca6a584000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)

    - parking to wait for  <0x00000000ee761560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

"Attach Listener" #497 daemon prio=9 os_prio=0 cpu=0.24ms elapsed=0.11s tid=0x00007fca780ba000 nid=0x3220 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=8855.96ms elapsed=59481.70s tid=0x00007fcaa80b5800 nid=0x2734 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=37230.87ms elapsed=59481.65s tid=0x00007fcaa8108000 nid=0x273c waiting on condition  

JNI global refs: 30, weak refs: 0

    • 해당 프로세스에 대응되는 쓰레드들이 돌아가고 있음
    • Java Runtime (GC 등) 관련 쓰레드?
      • Reference Handler
      • Finalizer
      • Compiler Thread
      • Sweeper thread
      • Common-Cleaner
    • Spring Tomcat 관련 쓰레드
      • http-nio-8080-exec-1 ~ http-nio-8080-exec-10 // min-spare 스레드가 10개인가보군
    • Spring Database Connection 쓰레드
      • HikariPool-1 housekeeper
  • 스레드 덤프 문제상황
    • 락을 획득하지 못하는 경우 (Blocked)
    • 데드락 상태인 경우
    • socketReadThread가 메시지 수신을 위해 계속 대기하는 경우
    • 스레드가 계속 WAIT 인 경우
    • 스레드 리소스를 정상적으로 정리를 못하는 경우

SpringBoot가 다중 유저 요청을 처리하는 방식

  • 참고: https://sihyung92.oopy.io/spring/18
  • 과정
    1. SpringBoot는 내장 서블릿 컨테이너인 tomcat을 사용한다
    2. Tomcat은 다중 요청을 처리하기 위해, 부팅할 때 쓰레드의 컬렉션인 Thread Pool을 생성
    3. 유저 요청 들어오면 Thread Pool에서 하나씩 Thread 할당. 해당 Thread에서 스프링부트에서 작성한 Dispatcher Servlet 거쳐 유저 요청 처리
    4. 작업 모두 수행 완료시, 쓰레드풀로 반환됨
  • BIO Connector & NIO Connector
    • [Connector]
      1. 소켓 연결 수립
      2. 데이터 패킷 획득해 HttpServletRequest 객체로 변환
      3. Servlet 객체에 전달 - Acceptor에서 while 문으로 대기하며 port listen을 통해 Socket Connection 얻음 - Socket Connection으로 데이터 획득 => 패킷 파싱 후 HttpServletRequest 객체 생성 - Servlet Container에 해당 요청객체 전달 => Servlet Container에서 알맞은 MVC 흐름 뚝딱
    • [BIO Connector]
      • 블록킹 방식으로 Connection <-> Thread 연결해서 처리해주면...
      • 동시 사용되는 쓰레드 수가 동시 접속하는 사용자 수가 될 것
        • idle 상태의 쓰레드 많아졌어
        • 효율적인 관리 위해 NIO 등장
    • [NIO Connector]
      • Poller라는 별도의 쓰레드가 커넥션 처리
        • Socket을 캐시로 들고있다가 Socket에서 data 처리가 가능한 순간에만 thread 할당하는 방식 => idle 최소화
      • Java Nio Selector를 통해 data가 처리 가능할 때만 Thread를 사용하기에 idle로 낭비되는 Thread가 줄어듦
      • 셀렉터를 통해 채널을 관리하기에 작업큐 사이즈와 관계없이 추가로 커넥션 거절하지 않고 받을 수 있음

Spring Boot + Prometheus + Grafana