当使用AWS Glue的write_dynamic_frame_from_options方法时出现模式异常,可能是由于动态框架中的模式与目标数据源的模式不匹配。
解决方法如下:
dynamic_frame = ...
data_frame = dynamic_frame.toDF()
data_frame.printSchema()
mapped_dynamic_frame = dynamic_frame.apply_mapping([
("source_field", "target_type", "target_field"),
...
])
dynamic_frame = ...
data_frame = dynamic_frame.toDF()
transformed_data_frame = data_frame.select("field1", "field2").withColumn("new_field", ...)
dynamic_frame = ...
glue_context.write_dynamic_frame.from_options(
frame = dynamic_frame,
connection_type = "s3",
connection_options = {"path": "s3://bucket/path", "mode": "overwrite", "partitionKeys": []},
format = "parquet",
format_options = {"parquet.block.size": "128m", "parquet.compression": "snappy"},
transformation_ctx = "transformed_data"
)
通过以上方法,你应该能够解决AWS Glue write_dynamic_frame_from_options遇到模式异常的问题。请根据你的具体情况选择合适的解决方法。