当前位置: 代码迷 >> 综合 >> SparkSQL读写kudu小记
  详细解决方案

SparkSQL读写kudu小记

热度:64   发布时间:2023-10-25 12:21:40.0

在操作之前首先要有相应的依赖

<dependency><groupId>org.apache.kudu</groupId><artifactId>kudu-spark2_2.11</artifactId><version>1.8.0</version><!--<scope>compile</scope>-->
</dependency>

SparkSQL读取Kudu(Java语言描述)

public static Dataset<Row> getDatasetFromKudu(String tableName, SparkSession sparkSession) {SQLContext sqlContext = sparkSession.sqlContext();Dataset<Row> load = sqlContext.read().format("org.apache.kudu.spark.kudu").option("kudu.master", "192.168.1.126:7051,192.168.1.207:7051,192.168.1.208:7051").option("kudu.table", tableName).load();return load;
}

SparkSQL将DataFrame存入Kudu(Scala语言描述)

def saveData2Kudu(tableName : String, sparkSession: SparkSession, dataset: DataFrame) = {val kuduContext = new KuduContext("192.168.1.126:7051,192.168.1.207:7051,192.168.1.208:7051",sparkSession.sparkContext)kuduContext.upsertRows(dataset,tableName)true
}