Concurrently错误表示试图同时执行多个查询。这可能会导致资源问题和查询错误。为了解决这个问题,您可以在AWS设置中更改“max_concurrency”值或通过分批执行查询来降低并发性。下面是一些示例代码,演示如何在AWS Glue中更改“max_concurrency”:
import boto3
client = boto3.client('glue')
response = client.put_workflow_run_properties(
Name='MyWorkflow',
RunId='jr_…',
RunProperties={
'max_concurrency': '1'
}
)
您还可以使用以下代码演示如何使用AWS SDK将查询分批执行:
import boto3
client = boto3.client('athena')
database_name = 'my_database'
table_name = 'my_table'
batch_size = 1000 # number of rows per batch
query = f"SELECT * FROM {table_name}"
response_iterator = client.get_paginator('start_query_execution').paginate(
QueryString=query,
QueryExecutionContext={
'Database': database_name
},
ResultConfiguration={
'OutputLocation': 's3://my-bucket/my-folder/'
}
)
for response in response_iterator:
query_execution_id = response['QueryExecutionId']
result_iterator = client.get_paginator('get_query_results').paginate(
QueryExecutionId=query_execution_id,
MaxResults=batch_size
)
for result in result_iterator:
# iterate over the result rows and process them as needed
使用以上示例代码可以帮助您在AWS中解决Concurrently错误。