HBase BucketAllocatorException 例外剖析

近日，觀察到HBase集群出現如下WARN日志：

2020-04-18 16:17:03,081 WARN [regionserver/xxx-BucketCacheWriter-1] bucket.BucketCache:Failed allocation for 604acc82edd349ca906939af14464bcb_175674734;
org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocatorException: Allocation too big size=1114202; adjust BucketCache sizes hbase.bucketcache.bucket.sizes to accomodate if size seems reasonable and you want it cached.

大概意思是說：由于block塊太大（size=1114202）導致BucketAllocator無法為其分配空間，如果想要被快取并且覺得這樣做合理，可以調整引數hbase.bucketcache.bucket.sizes，

默認情況下，HBase BucketCache 能夠快取block的最大值為512KB，即hbase.bucketcache.bucket.sizes=5120,9216,17408,33792,41984,50176,58368,66560,99328,132096,197632,263168,394240,525312，默認14種size標簽，如果想要快取更大的block塊，我們可以調整引數為 hbase.bucketcache.bucket.sizes=5120,9216,17408,33792,41984,50176,58368,66560,99328,132096,197632,263168,394240,525312,1049600,2098176，此時最大容許2MB的block，

下面我們簡單看一下對應源代碼，涉及相關類為：
/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java
BucketAllocator主要實作對bucket的組織管理，為block分配記憶體空間，

/**
   * Allocate a block with specified size. Return the offset
   * @param blockSize size of block
   * @throws BucketAllocatorException
   * @throws CacheFullException
   * @return the offset in the IOEngine
   */
  public synchronized long allocateBlock(int blockSize) throws CacheFullException,
      BucketAllocatorException {
    assert blockSize > 0;
    BucketSizeInfo bsi = roundUpToBucketSizeInfo(blockSize);
    if (bsi == null) {
      throw new BucketAllocatorException("Allocation too big size=" + blockSize +
        "; adjust BucketCache sizes " + BlockCacheFactory.BUCKET_CACHE_BUCKETS_KEY +
        " to accomodate if size seems reasonable and you want it cached.");
    }
    long offset = bsi.allocateBlock();

    // Ask caller to free up space and try again!
    if (offset < 0)
      throw new CacheFullException(blockSize, bsi.sizeIndex());
    usedSize += bucketSizes[bsi.sizeIndex()];
    return offset;
  }

在呼叫roundUpToBucketSizeInfo()方法后，回傳結果如果為null則拋出BucketAllocatorException例外，看一下roundUpToBucketSizeInfo()方法：

/**
 * Round up the given block size to bucket size, and get the corresponding
 * BucketSizeInfo
*/
public BucketSizeInfo roundUpToBucketSizeInfo(int blockSize) {
	for (int i = 0; i < bucketSizes.length; ++i)
	  if (blockSize <= bucketSizes[i])
		return bucketSizeInfos[i];
	return null;
}

該方法將傳入的blockSize與陣列bucketSizes從索引0開始取值進行比較，一旦小于bucketSize[i]，則為該block分配bucketSizeInfos[i]大小的空間存放該block，

我們看一下陣列bucketSizes的初始化程序：

private static final int DEFAULT_BUCKET_SIZES[] = { 4 * 1024 + 1024, 8 * 1024 + 1024,
  16 * 1024 + 1024, 32 * 1024 + 1024, 40 * 1024 + 1024, 48 * 1024 + 1024,
  56 * 1024 + 1024, 64 * 1024 + 1024, 96 * 1024 + 1024, 128 * 1024 + 1024,
  192 * 1024 + 1024, 256 * 1024 + 1024, 384 * 1024 + 1024,
  512 * 1024 + 1024 };

private final int[] bucketSizes;
BucketAllocator(long availableSpace, int[] bucketSizes)
  throws BucketAllocatorException {
  this.bucketSizes = bucketSizes == null ? DEFAULT_BUCKET_SIZES : bucketSizes;
  Arrays.sort(this.bucketSizes);
  ...
}

可以看到，如果bucketSizes == null默認取陣列DEFAULT_BUCKET_SIZES的值，并對該陣列進行排序，這一步的排序為上一步的回圈比較奠定了基礎，

而陣列DEFAULT_BUCKET_SIZES的值，也就是引數hbase.bucketcache.bucket.sizes的默認值，

轉載請注明出處！歡迎關注本人微信公眾號【HBase作業筆記】

標籤：大數據

上一篇：Hadoop偽分布安裝搭建

下一篇：安裝 Hadoop 2.9.1 on Windows 10 64 bit (最全步驟整理)