当前位置: 代码迷 >> 综合 >> hbase 写入性能
  详细解决方案

hbase 写入性能

热度:66   发布时间:2023-12-21 16:24:09.0

 

hbase单表的平均写入QPS大概在5000-7000左右。

测试代码:

 

 

private static void doMigrate(HTable inTable, HTable outTable, Date beginTime, Date endTime) throws ParseException,IOException {Scan scan = new Scan();scan.setCacheBlocks(false);scan.setTimeRange(beginTime.getTime(), endTime.getTime());scan.setCaching(100);ResultScanner scanner = inTable.getScanner(scan);Iterator<Result> res = scanner.iterator();int count = 0;List<Put> puts = new ArrayList<Put>();long begin = System.currentTimeMillis();while (res.hasNext()) {try {count++;Result result = res.next();KeyValue[] allKvs = result.raw();byte[] rowkey = result.getRow();if (count % 200 == 0) {outTable.put(puts);puts.clear();}if (count % 10000 == 0) {long end = System.currentTimeMillis();System.out.println("put " + (count - 10000) + " to " + count + " cost:" + (end - begin));begin = System.currentTimeMillis();}Put put = new Put(rowkey);for (KeyValue kv : allKvs) {put.add(kv);}puts.add(put);System.out.println("put " + count + ",rowkey=" + Bytes.toString(rowkey));} catch (Exception e) {System.out.println("error!!!");System.out.println(e.getMessage());e.printStackTrace();}}

 参考:提升HBase写性能