ApacheBeamSplittableDoFnStreamingAPI
创始人
2024-09-05 12:00:24
0

Apache Beam 提供了 Splittable DoFn Streaming API,它允许用户创建可拆分的并行处理函数,并在流式数据处理步骤中使用它们。

以下是一个使用 Splittable DoFn 的示例代码:

import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.splittabledofn.OffsetRange;
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
import org.apache.beam.sdk.transforms.splittabledofn.SplitResult;

public class MyDoFn extends DoFn {
  @ProcessElement
  public void processElement(ProcessContext c, RestrictionTracker tracker) {
    OffsetRange range = tracker.currentRestriction();
    for (long i = range.getFrom(); tracker.tryClaim(i) && i < range.getTo(); ++i) {
      // Process input element with index i.
      c.output("Processed element " + i);
    }
  }

  @GetInitialRestriction
  public OffsetRange getInitialRestriction(String element) {
    // Split the global range into subranges and assign one to each worker.
    return new OffsetRange(0L, 100L);
  }

  @SplitRestriction
  public SplitResult splitRestriction(
      String element, OffsetRange restriction) {
    // Split the given range into sub-ranges that can be processed independently.
    long midpoint = (restriction.getFrom() + restriction.getTo()) / 2;
    return SplitResult.of(
        new OffsetRange(restriction.getFrom(), midpoint),
        new OffsetRange(midpoint, restriction.getTo()));
  }

  @NewTracker
  public RestrictionTracker newTracker(OffsetRange range) {
    // Return a new RestrictionTracker suitable for processing a subrange of the input
    // based on the given initial restriction.
    return new MyRestrictionTracker(range);
  }
}

此代码定义了一个可拆分的 DoFn,它从输入字符串流中处理元素,并输出处理后的字符串流。

我们可以使用此 DoFn 在 Apache Beam 流水线中进行流式处理,如下所示:

PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);

p.apply("ReadFromSource", TextIO.read().from("inputFile.txt"))
    .apply(ParDo.of(new MyDoFn()))
    .apply("WriteToSink", TextIO.write().to("outputFile.txt"));

p.run();

此代码从输入文件中读取字符串流

相关内容

热门资讯

透视了解!wepoker作弊视... 透视了解!wepoker作弊视频,wepoker俱乐部辅助-都是有辅助脚本(哔哩哔哩)1、完成辅助器...
透视有挂!aapoker如何设... 透视有挂!aapoker如何设置胜率,aapoker免费透视脚本-都是有辅助插件(哔哩哔哩)1、游戏...
透视揭露!wepoker开辅助... 透视揭露!wepoker开辅助能查到吗,wepoker有插件吗-一贯一直总是有辅助攻略(哔哩哔哩)1...
透视分享!wpk辅助插件,wp... 透视分享!wpk辅助插件,wpk有那种辅助吗-原来一直总是有辅助工具(哔哩哔哩)1、超多福利:超高返...
透视科普!wepoker免费脚... 透视科普!wepoker免费脚本咨询,wepoker辅助透视-其实存在有辅助插件(哔哩哔哩)1、用户...
透视解迷!大菠萝789辅助器下... 透视解迷!大菠萝789辅助器下载,cloudpoker作弊-真是存在有辅助攻略(哔哩哔哩)1、大菠萝...
透视科普!wepoker辅助器... 您好,这款游戏可以开挂的,确实是有挂的,需要了解加去威信【485275054】很多玩家在这款游戏中打...
透视必备!wepoker插件辅... 透视必备!wepoker插件辅助,wepoker有透视底牌吗-真是有辅助教程(哔哩哔哩)1、让任何用...
透视必备!pokerworld... 透视必备!pokerworld修改器,悦扑克脚本-本来真的有辅助工具(哔哩哔哩)1、pokerwor...
透视教你!wejoker免费脚... 透视教你!wejoker免费脚本,wepoker插件程序激活码-原来一直总是有辅助插件(哔哩哔哩)1...