What & why
The aggregate folder udf/table/agg/ has 20 functions (Count, Sum*, Avg*,
Percentile*, …) but no bitwise aggregates (BIT_AND / BIT_OR) or boolean
aggregate (BOOL_AND — true only if every value is true). Aggregates are a step
up from scalar functions: you implement an Accumulator and, crucially, a
correct distributed merge. This is the issue that teaches you how distributed
aggregation actually works.
聚合函数目录已有 20 个,但缺位运算聚合 BIT_AND/BIT_OR 和布尔聚合 BOOL_AND
(全为真才为真)。聚合函数比标量函数稍难——要实现 Accumulator 并写对分布式 merge。
做完这题,你就真正理解了分布式聚合是怎么回事。
The task
bit_and(x) / bit_or(x) → bitwise AND/OR over a column of integers
bool_and(b) → whether a column of booleans is all true
Where to look
udf/table/agg/Count.java is the simplest UDAF template (it has the inner
Accumulator class). Implement the five methods:
createAccumulator / accumulate / merge / resetAccumulator / getValue.
- The
Accumulator must implement Serializable (it gets serialized across the cluster).
- Register in
BuildInSqlFunctionTable.java.
- Build multi-row test data; ideally exercise a multi-partition
merge.
Done when
What & why
The aggregate folder
udf/table/agg/has 20 functions (Count,Sum*,Avg*,Percentile*, …) but no bitwise aggregates (BIT_AND/BIT_OR) or booleanaggregate (
BOOL_AND— true only if every value is true). Aggregates are a stepup from scalar functions: you implement an
Accumulatorand, crucially, acorrect distributed
merge. This is the issue that teaches you how distributedaggregation actually works.
聚合函数目录已有 20 个,但缺位运算聚合
BIT_AND/BIT_OR和布尔聚合BOOL_AND(全为真才为真)。聚合函数比标量函数稍难——要实现
Accumulator并写对分布式merge。做完这题,你就真正理解了分布式聚合是怎么回事。
The task
bit_and(x)/bit_or(x)→ bitwise AND/OR over a column of integersbool_and(b)→ whether a column of booleans is all trueWhere to look
udf/table/agg/Count.javais the simplest UDAF template (it has the innerAccumulatorclass). Implement the five methods:createAccumulator / accumulate / merge / resetAccumulator / getValue.Accumulatormustimplement Serializable(it gets serialized across the cluster).BuildInSqlFunctionTable.java.merge.Done when
mergelogic (this is what makes distributed results correct)