mongoDB文件太大查错纪录
2014-02-27 13:57
337 查看
日志系统,突然从24号之后的都断层了,交易看不见。查了一下问题是MongoDB把硬盘撑爆了,看了下情况:
-bash-3.2$ du -h 82M ./log 3.1G ./db/journal 4.0K ./db/ciflogs/_tmp 4.0G ./db/ciflogs 4.0K ./db/local/_tmp 1.1G ./db/local 4.0K ./db/_tmp 8.1G ./db 8.2G .
去google了两把,又去官网看了下,发现官网FAQ中有一段回答:(注意加粗部分)
Why are the files in my data directory larger than the data in my database?
Preallocated data files.In the data directory, MongoDB preallocates data files to a particular size, in part to prevent file system fragmentation. MongoDB names the first data file <databasename>.0, the next <databasename>.1, etc. The first file mongodallocates is 64 megabytes, the next 128 megabytes, and so on, up to 2 gigabytes, at which point all subsequent files are 2 gigabytes. The data files include files with allocated space but that hold no data. mongod may allocate a 1 gigabyte data file that may be 90% empty. For most larger databases, unused allocated space is small compared to the database.
On Unix-like systems, mongod preallocates an additional data file and initializes the disk space to 0. Preallocating data files in the background prevents significant delays when a new database file is next allocated.
You can disable preallocation with the noprealloc run time option. However noprealloc is not intended for use in production environments: only use noprealloc for testing and with small data sets where you frequently drop databases.
On Linux systems you can use hdparm to get an idea of how costly allocation might be:
time hdparm --fallocate $((1024*1024)) testfile
The oplog.
If this mongod is a member of a replica set, the data directory includes the oplog.rs file, which is a preallocated capped collection in the local database. The default allocation is approximately 5% of disk space on 64-bit installations, seeOplog Sizing for more information. In most cases, you should not need to resize the oplog. However, if you do, seeChange the Size of the Oplog.
The journal.
The data directory contains the journal files, which store write operations on disk prior to MongoDB applying them to databases. See Journaling Mechanics.
Empty records.
MongoDB maintains lists of empty records in data files when deleting documents and collections. MongoDB can reuse this space, but will never return this space to the operating system.
To de-fragment allocated storage, use compact, which de-fragments allocated space. By de-fragmenting storage, MongoDB can effectively use the allocated space. compact requires up to 2 gigabytes of extra disk space to run. Do not use compact if you are critically low on disk space.
Important
compact only removes fragmentation from MongoDB data files and does not return any disk space to the operating system.
http://docs.mongodb.org/manual/faq/storage/
然后在Journaling Mechanics页面又有详细对于Journa的介绍:
Journal Files
With journaling enabled, MongoDB creates a journal directory within the directory defined by dbpath, which is /data/db by default. The journal directory holds journal files, which contain write-ahead redo logs. The directory also holds a last-sequence-number file. A clean shutdown removes all the files in the journal directory.Journal files are append-only files and have file names prefixed with j._. When a journal file holds 1 gigabyte of data, MongoDB creates a new journal file. Once MongoDB applies all the write operations in the journal files, it deletes these files. Unless you write many bytes of data per-second, the journal directory should contain only two or three journal files.
To limit the size of each journal file to 128 megabytes, use the smallfiles run time option when starting mongod.
To speed the frequent sequential writes that occur to the current journal file, you can ensure that the journal directory is on a different filesystem.
Important
If you place the journal on a different filesystem from your data files you cannot use a filesystem snapshot alone to capture valid backups of a dbpath directory. In this case, use fsyncLock() to ensure that database files are consistent before the snapshot and fsyncUnlock() once the snapshot is complete.
Note
Depending on your filesystem, you might experience a preallocation lag the first time you start a mongod instance with journaling enabled.
MongoDB may preallocate journal files if the mongod process determines that it is more efficient to preallocate journal files than create new journal files as needed. The amount of time required to pre-allocate lag might last several minutes, during which you will not be able to connect to the database. This is a one-time preallocation and does not occur with future invocations.
http://docs.mongodb.org/manual/core/journaling/
文中可以看出,journa最多只有3个文件。也就是最大只会占用3G硬盘,而且停止之后会自动删除。启动时使用-smallfiles则会让mongo的journa最大128M。
另外删除的纪录不会立刻释放硬盘,但会在下次写入的时候重新利用。
OK,停一下Mongo,删掉journa,用smallfiles参数。另外删掉一些太早的日志纪录。
相关文章推荐
- MongoDB中的数据导出为excel CSV 文件
- MongoDB学习以及集群搭建的实践全纪录
- Mongodb源码分析--内存文件映射(MMAP)
- mongodb 批量更新 数组的键操作的文件
- 通过nodejs将文件上传到mongodb
- MongoDb gridfs-ngnix文件存储方案 - 图片
- 基于 MongoDB 及 Spring Boot 的文件服务器的实现
- python写入文件到mongoDB
- MongoDB GridFS 分布式文件存储系统
- mongodb固定集合(Capped Collection)和大文件管理(GridFS)
- MongoDB数据文件备份与恢复
- MongoDB数据库的文件备份恢复以及文件导入导出
- 远程从Mongodb 数据库中 导出数据为Excel 文件
- 监管文件显示 MongoDB 将获得 1 亿美金新资金
- MongoDB3.4配置文件参数选项
- python mongodb 设置密码前一篇ok,csv文件存入mongodb
- MongoDB3.4配置文件参数选项
- MongoDB 通过配置文件启动
- 【转载】把文件二进制数据存入mongodb
- 详解log4j2(下) - Async/MongoDB/Flume Appender 按日志级别区分文件输出