spark core 2.0 StorageMemoryPool
2017-01-13 17:20
148 查看
StorageMemoryPool是MemoryPool的子类,MemoryPool参见MemoryPool 源代码分析, 构造参数多了一个MemoryMode,poolName根据MemoryMode生成。
获取内存方法,获取N字节的内存来缓存数据块,如果必要,会回收一些数据块的内存。
返回:是否成功获得N字节的内存成功。
这个方法首先计算需要回收的内存空间。
/** * Performs bookkeeping for managing an adjustable-size pool of memory that is used for storage * (caching). * * @param lock a [[MemoryManager]] instance to synchronize on * @param memoryMode the type of memory tracked by this pool (on- or off-heap) */ private[memory] class StorageMemoryPool( lock: Object, memoryMode: MemoryMode ) extends MemoryPool(lock) with Logging {
private[this] val poolName: String = memoryMode match { case MemoryMode.ON_HEAP => "on-heap storage" case MemoryMode.OFF_HEAP => "off-heap storage" }这里定义了_memoryUsed变量,并且实现了父类的虚函数。
@GuardedBy("lock") private[this] var _memoryUsed: Long = 0L
override def memoryUsed: Long = lock.synchronized { _memoryUsed }
获取内存方法,获取N字节的内存来缓存数据块,如果必要,会回收一些数据块的内存。
返回:是否成功获得N字节的内存成功。
这个方法首先计算需要回收的内存空间。
/** * Acquire N bytes of memory to cache the given block, evicting existing ones if necessary. * * @return whether all N bytes were successfully granted. */ def acquireMemory(blockId: BlockId, numBytes: Long): Boolean = lock.synchronized { val numBytesToFree = math.max(0, numBytes - memoryFree) acquireMemory(blockId, numBytes, numBytesToFree) }
/** * Acquire N bytes of storage memory for the given block, evicting existing ones if necessary. * * @param blockId the ID of the block we are acquiring storage memory for * @param numBytesToAcquire the size of this block * @param numBytesToFree the amount of space to be freed through evicting blocks * @return whether all N bytes were successfully granted. */ def acquireMemory( blockId: BlockId, numBytesToAcquire: Long, numBytesToFree: Long): Boolean = lock.synchronized { assert(numBytesToAcquire >= 0) assert(numBytesToFree >= 0) assert(memoryUsed <= poolSize) if (numBytesToFree > 0) { memoryStore.evictBlocksToFreeSpace(Some(blockId), numBytesToFree, memoryMode) } // NOTE: If the memory store evicts blocks, then those evictions will synchronously call // back into this StorageMemoryPool in order to free memory. Therefore, these variables // should have been updated. val enoughMemory = numBytesToAcquire <= memoryFree if (enoughMemory) { _memoryUsed += numBytesToAcquire } enoughMemory }释放指定大小的内存。
def releaseMemory(size: Long): Unit = lock.synchronized { if (size > _memoryUsed) { logWarning(s"Attempted to release $size bytes of storage " + s"memory when we only have ${_memoryUsed} bytes") _memoryUsed = 0 } else { _memoryUsed -= size } } def releaseAllMemory(): Unit = lock.synchronized { _memoryUsed = 0 }
/** * Free space to shrink the size of this storage memory pool by `spaceToFree` bytes. * Note: this method doesn't actually reduce the pool size but relies on the caller to do so. * * @return number of bytes to be removed from the pool's capacity. */ def freeSpaceToShrinkPool(spaceToFree: Long): Long = lock.synchronized { val spaceFreedByReleasingUnusedMemory = math.min(spaceToFree, memoryFree) val remainingSpaceToFree = spaceToFree - spaceFreedByReleasingUnusedMemory if (remainingSpaceToFree > 0) { // If reclaiming free memory did not adequately shrink the pool, begin evicting blocks: val spaceFreedByEviction = memoryStore.evictBlocksToFreeSpace(None, remainingSpaceToFree, memoryMode) // When a block is released, BlockManager.dropFromMemory() calls releaseMemory(), so we do // not need to decrement _memoryUsed here. However, we do need to decrement the pool size. spaceFreedByReleasingUnusedMemory + spaceFreedByEviction } else { spaceFreedByReleasingUnusedMemory } }
相关文章推荐
- spark core 2.0 ExecutionMemoryPool
- spark core 2.0 UnifiedMemoryManager
- spark 2.1 MemoryPool, StoragememoryPool
- spark core 2.0 MemoryBlock Soruce Code Analysis
- SD Memory Card V2.0 协议栈代码
- spark core 2.0 CoarseGrainedSchedulerBackend SchedulerBackend ExecutorAllocationClient 源代码解析
- 第十篇:Spark SQL 源码分析之 In-Memory Columnar Storage源码分析之 query
- ORA-00371: not enough shared pool memory, should be atleast 62198988 bytes
- smart memory pool :内存池--智能内存池 新鲜大出炉啦
- iTron3学习笔记(一) System Calls of Memory Pool Management Functions
- SQLOS's memory manager and SQL Server's Buffer Pool
- is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.6 GB of 40 GB virtual memory used
- 重删之删除-Memory Efficient Sanitization of a Deduplicated Storage System
- find the total amount of memory consumed (including AWE) by the buffer pool
- Tachyon在Spark中的作用(Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks 论文阅读翻译)
- Mysql cannot allocate memory for the buffer pool 解决方法
- 安装mysql 报Fatal error: cannot allocate memory for the buffer pool
- How to Replace a Device in a ZFS Storage Pool
- ACE shared memory mmap memory pool
- linux-kernel-map,linux-memory-management,Linux-storage-stack三幅很好的图