curl用postfields传输大量信息时提示out of memory可能的解决办法
2013-12-27 18:07
519 查看
转载自:http://devcs.blogspot.com/2008/12/curl-out-of-memory-on-xen-instance-use.html
cURL, a part of everyone's favorite UNIX tool subset, got me into a bit of trouble recently, while trying to post a relatively large file, following a common 'just curl it' - logic (so commonplace that a lot of major projects simply incorporate curlin' as a
part of standard deploy procedure).
The case was posting 8Gb file on 16Gb Xen instance. While this worked quite nice on real box, on virtual box curl said hello with :
out of memory
Now that seemed quite bizzare. Figurng out that the process actually gets ENOMEM, it was logical to look at curl code and figure out what's going on.
And there it was, power-of-two allocator in the file read loop :
static ParameterError file2memory(char **bufp, size_t *size, FILE *file)
...
char *newbuf;
char *buffer = NULL;
size_t alloc = 512;
size_t nused = 0;
size_t nread;
do {
if(!buffer || (alloc == nused)) {
if(alloc+1 > ((size_t)-1)/2) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
alloc *= 2;
if((newbuf = realloc(buffer, alloc+1)) == NULL) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
buffer = newbuf;
}
}
Whoa :) - now apparently someone didn't expect some geniuses will try po post XX Gb files withcurl - so it's the abusers that are to blame. Stop abusing curl and do your own posts !
However, if you don't have the time to change your app, and still want to post files of the size (N,2N) Gb on a 2N Gb box, a simple hack of given form should do it :
if (alloc < ALLOC_THRESHOLD)
alloc *= 2;
else
alloc = alloc + ALLOC_THRESHOLD;
(Where ALLOC_THRESHOLD would usually be 1Gb)
This should make allocation linear, rather than exponential, once the allocated memory passes given threshold.
Now - what does is all has to do with XEN, you might ask?.
Couple of things, actually. First off, such environment (local or any virtualized cloud platform offering xen instances) usually provide user with something like effective 2^N - penalty memory space (say 15Gb instead of 16Gb) - and that's where the impact of
power of two allocator becomes apparent much sooner. Also - memory allocation policies are quite stricter and enomems are dispatched much earlier, oom killer is fast on the trigger, etc :) - so that's why the curl ooms immediately, rather than trying to make
that darn realloc() after all.
Moral of the story - don't abuse standard unix tools !
Be nice to curl - do not POST binary data larger than 50% of effective RAM.
Keep it safe !
Posted by Alek at 12:23
PM
cURL, a part of everyone's favorite UNIX tool subset, got me into a bit of trouble recently, while trying to post a relatively large file, following a common 'just curl it' - logic (so commonplace that a lot of major projects simply incorporate curlin' as a
part of standard deploy procedure).
The case was posting 8Gb file on 16Gb Xen instance. While this worked quite nice on real box, on virtual box curl said hello with :
out of memory
Now that seemed quite bizzare. Figurng out that the process actually gets ENOMEM, it was logical to look at curl code and figure out what's going on.
And there it was, power-of-two allocator in the file read loop :
static ParameterError file2memory(char **bufp, size_t *size, FILE *file)
...
char *newbuf;
char *buffer = NULL;
size_t alloc = 512;
size_t nused = 0;
size_t nread;
do {
if(!buffer || (alloc == nused)) {
if(alloc+1 > ((size_t)-1)/2) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
alloc *= 2;
if((newbuf = realloc(buffer, alloc+1)) == NULL) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
buffer = newbuf;
}
}
Whoa :) - now apparently someone didn't expect some geniuses will try po post XX Gb files withcurl - so it's the abusers that are to blame. Stop abusing curl and do your own posts !
However, if you don't have the time to change your app, and still want to post files of the size (N,2N) Gb on a 2N Gb box, a simple hack of given form should do it :
if (alloc < ALLOC_THRESHOLD)
alloc *= 2;
else
alloc = alloc + ALLOC_THRESHOLD;
(Where ALLOC_THRESHOLD would usually be 1Gb)
This should make allocation linear, rather than exponential, once the allocated memory passes given threshold.
Now - what does is all has to do with XEN, you might ask?.
Couple of things, actually. First off, such environment (local or any virtualized cloud platform offering xen instances) usually provide user with something like effective 2^N - penalty memory space (say 15Gb instead of 16Gb) - and that's where the impact of
power of two allocator becomes apparent much sooner. Also - memory allocation policies are quite stricter and enomems are dispatched much earlier, oom killer is fast on the trigger, etc :) - so that's why the curl ooms immediately, rather than trying to make
that darn realloc() after all.
Moral of the story - don't abuse standard unix tools !
Be nice to curl - do not POST binary data larger than 50% of effective RAM.
Keep it safe !
Posted by Alek at 12:23
PM
相关文章推荐
- 【转】Oracle创建数据库时提示out of memory错误解决办法
- Oracle创建数据库时提示out of memory错误解决办法
- [Java] java.lang.OutOfMemoryError: PermGen space的一种可能解决办法
- 修改桌面堆栈大小,解决运行大量程序时出现”Out of Memory”(内存不足)错误信息的问题
- MyEclipse6.5通过Jetty跑Web应用时提示OutOfMemoryError: PermGen space的解决办法
- 在myeclipse中启动Tomcat时,提示"ava.lang.OutOfMemoryError: Java heap space",解决办法。
- 修改桌面堆栈大小,解决运行大量程序时出现”Out of Memory”(内存不足)错误信息的问题
- .net发生类型为 System.OutOfMemoryException 的异常解决办法
- java.lang.OutOfMemoryError: Java heap space的解决办法
- 关于setImageURI out of memory的一些解决办法
- 黄聪:wordpress中PHP运行错最有效解决办法Fatal error: Out of memory (allocated 6029312)(转)
- Java中OutOfMemoryError(内存溢出)的三种情况及解决办法
- OutOfMemory解决办法
- Java中OutOfMemoryError(内存溢出)的三种情况及解决办法
- 结束 adb.exe 进程后仍然提示“adb server is out of date.” 解决办法
- 解决办法:matlab Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap
- java.lang.OutOfMemoryError: Java heap space 解决办法
- 用Bitmap加载图片资源时,报错java.lang.OutOfMemoryError: bitmap size exceeds VM budget的解决办法
- java.lang.OutOfMemoryError 问题解决办法
- Tomcat内存溢出(java.lang.OutOfMemoryError: PermGen space)的解决办法