您的位置：首页 > 其它

JVM最多能创建多少个线程: unable to create new native thread

2017-11-08 08:52 573 查看
转载自：http://www.rigongyizu.com/jvm-max-threads/

有应用报出这样的异常“java.lang.OutOfMemoryError: unable to create new native thread”。甚至机器上执行shell命令也会报”-bash: fork: Resource temporarily unavailable”异常。机器上的其他应用如hadoop也会受影响：

1
2013
-
08
-
21
20
:
15
:
48
,
496
FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
2
java.lang.OutOfMemoryError: unable to create
new
native
thread
3
at java.lang.Thread.start0(Native Method)
4
at java.lang.Thread.start(Thread.java:
640
)
5
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
6
ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:
524
)
7
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
8
ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:
456
)
9
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:
128
)
10
at org.apache.hadoop.yarn.event.AsyncDispatcher$
1
.run(AsyncDispatcher.java:
77
)
11
at java.lang.Thread.run(Thread.java:
662
)
12
2013
-
08
-
21
 
20
:
15
:
48
,
497
 
INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
一看以为内存不够导致无法创建新的线程，但是观察机器上的内存还有空闲，猜测是哪个地方对线程创建有限制。

首先需要排除操作系统对线程创建数的限制，参考：《JVM中可生成的最大Thread数量》一文，设置操作系统可以支持创建10万个线程：

1
echo
"100000"
 
> /proc/sys/kernel/threads-max
2
echo
"100000"
 
> /proc/sys/kernel/pid_max （默认
32768
）
3
echo
"200000"
 
> /proc/sys/vm/max_map_count （默认
65530
）
4
ulimit -u unlimited (设置max user processes的值)
当前测试环境为：

1
[admin@bufer108081.tbc ~]$
uname
 
-a
2
Linux bufer108081.tbc 2.6.32-220.23.2.ali927.el5.x86_64
#1 SMP Mon Jan 28 14:57:06 CST 2013 x86_64 x86_64 x86_64 GNU/Linux
3
[admin@bufer108081.tbc ~]$
cat
 
/etc/redhat-release
4
Red Hat Enterprise Linux Server release 5.7 (Tikanga)
5
[admin@bufer108081.tbc ~]$java -version
6
java version
"1.7.0_51"
7
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
8
OpenJDK (Alibaba) 64-Bit Server VM (build 24.45-b08-internal, mixed mode)
9
[admin@bufer108081.tbc ~]$
ulimit
 
-a
10
core
file
 
size(blocks, -c) 0
11
data seg size (kbytes, -d) unlimited
12
scheduling priority (-e) 0
13
file
 
size (blocks, -f) unlimited
14
pending signals (-i) 387068
15
max locked memory (kbytes, -l) 64
16
max memory size (kbytes, -m) unlimited
17
open
 
files(-n) 131072
18
pipe size(512 bytes, -p) 8
19
POSIX message queues (bytes, -q) 819200
20
real-
time
 
priority(-r) 0
21
stack size(kbytes, -s) 10240
22
cpu
time
 
(seconds, -t) unlimited
23
max user processes(-u) unlimited
24
virtual memory(kbytes, -
v
) unlimited
25
file
 
locks(-x) unlimited
26
[admin@bufer108081.tbc ~/dev/baoniu]$
free
 
-g
27
 
total used
free
 
sharedbuffers cached
28
Mem:47 31 1503 25
29
-/+ buffers/cache:3 44
30
Swap:000
测试程序见本文最后面。测试结果：突破了网上所说的32000个线程数，成功创建了 10万个线程。 （由于/proc/sys/kernel/pid_max默认为32768，所以网上很多测试程序测试JVM只能创建32000个线程。）

1
[admin@bufer108081.tbc ~/dev/baoniu]$java -Xss128k MaxThreadsMain
2
The stack size specified is too small, Specify at least 228k
3
Error: Could not create the Java Virtual Machine.
4
Error: A fatal exception has occurred. Program will
exit
.
5
[admin@bufer108081.tbc ~/dev/baoniu]$java -Xss228k MaxThreadsMain
6
4,000 threads: Time to create 4,000 threads was 0.846 seconds
7
8,000 threads: Time to create 4,000 threads was 2.425 seconds
8
12,000 threads: Time to create 4,000 threads was 4.813 seconds
9
16,000 threads: Time to create 4,000 threads was 7.229 seconds
10
20,000 threads: Time to create 4,000 threads was 10.443 seconds
11
24,000 threads: Time to create 4,000 threads was 14.480 seconds
12
28,000 threads: Time to create 4,000 threads was 19.709 seconds
13
32,000 threads: Time to create 4,000 threads was 24.742 seconds
14
36,000 threads: Time to create 4,000 threads was 31.181 seconds
15
40,000 threads: Time to create 4,000 threads was 36.629 seconds
16
44,000 threads: Time to create 4,000 threads was 42.796 seconds
17
48,000 threads: Time to create 4,000 threads was 48.659 seconds
18
52,000 threads: Time to create 4,000 threads was 55.030 seconds
19
56,000 threads: Time to create 4,000 threads was 60.130 seconds
20
60,000 threads: Time to create 4,000 threads was 67.419 seconds
21
64,000 threads: Time to create 4,000 threads was 73.507 seconds
22
68,000 threads: Time to create 4,000 threads was 79.416 seconds
23
72,000 threads: Time to create 4,000 threads was 85.261 seconds
24
76,000 threads: Time to create 4,000 threads was 92.201 seconds
25
80,000 threads: Time to create 4,000 threads was 98.087 seconds
26
84,000 threads: Time to create 4,000 threads was 108.263 seconds
27
88,000 threads: Time to create 4,000 threads was 114.840 seconds
28
92,000 threads: Time to create 4,000 threads was 121.841 seconds
29
96,000 threads: Time to create 4,000 threads was 127.714 seconds
30
After creating 99,410 threads, java.lang.OutOfMemoryError: unable to createnew native thread
31
at java.lang.Thread.start0(Native Method)
32
at java.lang.Thread.start(Thread.java:713)
33
at MaxThreadsMain.addThread(MaxThreadsMain.java:43)
34
at MaxThreadsMain.main(MaxThreadsMain.java:13)
创建9W多个线程后，进程占用内存：VIRT=40.5g RES=4.7g，用free -g查看系统还有9G的空闲(free)内存。

JVM最多能启动的线程数参照公式：

1
(MaxProcessMemory- JVMMemory – ReservedOsMemory) / (ThreadStackSize) = Number of threads
MaxProcessMemory : 进程的最大寻址空间

JVMMemory : JVM内存

ReservedOsMemory : 保留的操作系统内存，如Native heap，JNI之类，一般100多M

ThreadStackSize : 线程栈的大小，jvm启动时由Xss指定

MaxProcessMemory：如32位的linux默认每个进程最多申请3G的地址空间，64位的操作系统可以支持到46位（64TB）的物理地址空间和47位（128T）的进程虚拟地址空间（linux 64位CPU内存限制）。

JVM内存：由Heap区和Perm区组成。通过-Xms和-Xmx可以指定heap区大小，通过-XX:PermSize和-XX:MaxPermSize指定perm区的大小(默认从32MB 到64MB，和JVM版本有关)。

线程栈ThreadStackSize：


Java程序中，每个线程都有自己的Stack Space。这个Stack Space的空间是独立分配的，与-Xmx和-Xms指定的堆大小无关。Stack Space用来做方法的递归调用时压入Stack Frame。所以当递归调用太深的时候，就有可能耗尽Stack Space，爆出StackOverflow的错误。对于32位JVM，缺省值为256KB，对于64位JVM，缺省值为512KB。最大值根据平台和特定机器配置的不同而不同。如果超过最大值，那么将报告java/lang/OutOfMemoryError消息。


可见，减少Xss指定的线程栈大小能够启动更多的线程，但是线程总数也受到系统空闲内存和操作系统的限制。

总结下影响Java线程数量的因素：

Java虚拟机本身：-Xms，-Xmx，-Xss；

系统限制： /proc/sys/kernel/pid_max， /proc/sys/kernel/thread-max， max_user_process（ulimit -u）， /proc/sys/vm/max_map_count。

ps: 最后发现是这台机器上有个应用代码问题创建了过多的线程，达到系统限制，而影响了YARN和其他应用。一般来说，单机线程数过多可以考虑使用线程池或者更多的服务器。

附测试程序：

1
import
 
java.util.ArrayList;
2
import
 
java.util.List;
3
4
public
 
class
 
MaxThreadsMain {
5
6
public
 
static
 
final
 
int
 
BATCH_SIZE =
4000
;
7
8
public
 
static
 
void
 
main(String... args)
throws
 
InterruptedException {
9
List<Thread> threads =
new
 
ArrayList<Thread>();
10
try
 
{
11
for
 
(
int
 
i =
0
; i <=
100
 
*
1000
; i += BATCH_SIZE) {
12
long
 
start = System.currentTimeMillis();
13
addThread(threads, BATCH_SIZE);
14
long
 
end = System.currentTimeMillis();
15
Thread.sleep(
1000
);
16
long
 
delay = end - start;
17
System.out.printf(
"%,d threads: Time to create %,d threads was %.3f seconds %n"
, threads.size(), BATCH_SIZE, delay / 1e3);
18
}
19
}
catch
 
(Throwable e) {
20
System.err.printf(
"After creating %,d threads, "
, threads.size());
21
e.printStackTrace();
22
}
23
24
}
25
26
private
 
static
 
void
 
addThread(List<Thread> threads,
int
 
num) {
27
for
 
(
int
 
i =
0
; i < num; i++) {
28
Thread t =
new
 
Thread(
new
 
Runnable() {
29
@Override
30
public
 
void
 
run() {
31
try
 
{
32
while
 
(!Thread.interrupted()) {
33
Thread.sleep(
1000
);
34
}
35
}
catch
 
(InterruptedException ignored) {
36
//
37
}
38
}
39
});
40
t.setDaemon(
true
);
41
t.setPriority(Thread.MIN_PRIORITY);
42
threads.add(t);
43
t.start();
44
}
45
}
46
}


附：

[html] view plain copy print?

MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:
 Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html 
$ Ulimit -U

1024

It Explains 1000 Sessions Limitation ;-)

Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:

# Cat /Etc/Security/Limits.Conf

Mysql SoftNofile10240

Mysql HardNofile40960

Mysql SoftNproc 10240

Mysql HardNproc 40960

Fixed My Issue! :-))

MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections: Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html 
$ Ulimit -U
1024
It Explains 1000 Sessions Limitation ;-)

Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:

# Cat /Etc/Security/Limits.Conf
Mysql 	Soft 	Nofile 	10240
Mysql 	HardNofile 	40960
Mysql 	Soft 	Nproc 	10240
Mysql 	HardNproc 	40960
Fixed My Issue! :-))


                                            

                                        

                        
                        内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 点击这里给我发消息 
                    

                    

                    

                        

                         标签： 
                                                

                        

                    


                

            


            

                
相关文章推荐

                

                

                    
                

            

        

        

            

            
            

                

                    
新的分享

                    

                        
                    

                

            


            

                

                    
章节导航