[BUG] Make return type inferrable for TRANSFORM and other lambda functions

[yuancu]

Query Information

PPL Command/Query:

source=people | eval array = array(1, -2, 3), result = transform(array, x -> x + 2) | fields result

Expected Result:
[3,0,5] of type INTEGER ARRAY

Actual Result:
[3,0,5] of type ANY ARRAY

Dataset Information

Dataset/Schema Type

• OpenTelemetry (OTEL)
• Simple Schema for Observability (SS4O)
• Open Cybersecurity Schema Framework (OCSF)
• Custom (details below)

Index Mapping
Mapping not relevant

Bug Description

Before #4892, the types are injected at runtime:

sql/core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/TransformFunctionImpl.java

Line 183 in d19ba53

transferLambdaOutputToTargetType(lambdaFunction.apply(candidate), returnType));

sql/core/src/main/java/org/opensearch/sql/expression/function/CollectionUDF/TransformFunctionImpl.java

Lines 190 to 191 in d19ba53

	org.apache.calcite.linq4j.function.Function2 lambdaFunction =
	(org.apache.calcite.linq4j.function.Function2) args[1];

However, the standard SQL requires the return type to be inferrable before executing the query. Currently, the argument is of type ANY, making the result also typed ANY. Therefore, the result array is an ANY array. For this case, it is indeed inferrable -- the array is an INTEGER ARRAY, meaning that x should be of type INTEGER instead of ANY.

Impact:
doctest in collections.md

Environment Information

OpenSearch Version: 3.4 (with PR #4892 )

Screenshots

File "/Users/yuanchu/src/sql/doctest/../docs/user/ppl/functions/collection.md", line 167, in /Users/yuanchu/src/sql/doctest/../docs/user/ppl/functions/collection.md
Failed example:
    ppl_cmd.process('source=people | eval array = array(1, -2, 3), result = transform(array, x -> x + 2) | fields result | head 1')
Expected:
    fetched rows / total rows = 1/1
    +---------+
    | result  |
    |---------|
    | [3,0,5] |
    +---------+
Got:
    fetched rows / total rows = 1/1
    +---------------+
    | result        |
    |---------------|
    | [3.0,0.0,5.0] |
    +---------------+

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#4839 - Support mvfind eval function [merged]
#4856 - Support mvmap eval function [merged]

👤 Suggested Assignees

• ahkcs

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!