AWSGlue中的GoogleBigQueryConnector是否允许编写自定义查询?
创始人
2024-09-25 17:02:56
0

AWS Glue中使用Google BigQuery Connector的过程中,确实能够编写自定义查询。下面是一个基本的编写自定义查询的示例:

import com.amazonaws.services.glue.util.AWSGlueArgParser
import com.amazonaws.services.glue.DynamicFrame

import com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration
import com.google.cloud.hadoop.io.bigquery.BigQueryStrings
import com.google.cloud.hadoop.io.bigquery.BigQueryUtils

import org.apache.hadoop.mapreduce.Job

import org.apache.spark.SparkContext
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions._

object CustomQuery {
  def main(sysArgs: Array[String]) {
  
    // Glue job-specific args.
    val glueArgs = Map("JOB_NAME" -> "custom-query")
    
    // Google BigQuery related args.
    val projectId = "my-big-query-project"
    val bucketName = "my-bucket"
    val datasetName = "my-dataset"
    val tableName = "my-table"
    val customQuery = "SELECT COUNT(*) FROM `my-dataset.my-table`"
  
    // Use AWSGlueArgParser to parse args.
    val args = AWSGlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray)
  
    // Set Spark context.
    val sparkContext = new SparkContext()
    val sparkSession = SparkSession
      .builder()
      .appName(args("JOB_NAME"))
      .getOrCreate()
  
    // Initialize the BigQuery Configuration.
    val bigQueryConf = BigQueryConfiguration.configureBigQueryOutput(projectId)
  
    // Set the Google Cloud storage bucket to export results to.
    bigQueryConf.set(BigQueryConfiguration.OUTPUT_FORMAT_OPTION, BigQueryStrings.CSV)
    bigQueryConf.set(BigQueryConfiguration.GCS_BUCKET_OPTION, s"gs://$bucketName")
  
    // Set up the Job, OutputFormat and Table to write to.
    val job = Job.getInstance(sparkContext.hadoopConfiguration)
    BigQueryConfiguration.configureBigQueryOutput(job, projectId, customQuery, datasetName, tableName)
  
    // Load DataFrame from customQuery.
    val df = sparkSession.read
      .format("

相关内容

热门资讯

透视辅助!hhpoker透视码... 透视辅助!hhpoker透视码(透视)hhpoker辅助靠谱吗-原来真的有挂(哔哩哔哩)1、金币登录...
透视线上!德州局hhpoker... 透视线上!德州局hhpoker(透视)hhpoker真的有透视吗-确实真的是有挂(哔哩哔哩)1、用户...
透视模拟器!werplan辅助... 透视模拟器!werplan辅助软件(透视)佛手在线是不是有挂-果然是真的挂(哔哩哔哩)1、在佛手在线...
最终!aa poker辅助包(... 最终!aa poker辅助包(透视)xpoker辅助怎么用-都是存在有挂(哔哩哔哩)1、进入游戏-大...
黑科技辅助挂!aapoker怎... 黑科技辅助挂!aapoker怎么控制牌(透视)德州局hhpoker-竟然是真的挂(哔哩哔哩)1、进入...
据通报!wepoker养号规律... 据通报!wepoker养号规律(透视)wepoker私人局可以透视-一贯有挂(哔哩哔哩)1、wepo...
透视透视挂!智星菠萝辅助(透视... 透视透视挂!智星菠萝辅助(透视)智星德州可以透视吗-确实真的有挂(哔哩哔哩)1、上手简单,内置详细流...
透视神器!德普之星辅助工具如何... 透视神器!德普之星辅助工具如何打开(透视)德普之星透视-原来真的是有挂(哔哩哔哩)1、进入到德普之星...
为切实保障!wpk模拟器(透视... 为切实保障!wpk模拟器(透视)wpk俱乐部是真的吗-本来真的是有挂(哔哩哔哩)1、玩家可以在wpk...
目前!wpk俱乐部有没有辅助(... 目前!wpk俱乐部有没有辅助(透视)wpk辅助器是真的吗-好像是真的挂(哔哩哔哩)进入游戏-大厅左侧...