[Bug]: ExplainAnalyzeToken panics when undefined field has null value

[jnory]

Prerequisites

• I have searched the existing issues
• I have read the documentation

Go Client Version

v8.19.0

Go Version

1.23.12, 1.24.7

Elasticsearch Version

8.15.2

Current Behavior

We've just noticed that `UnmarshalJSON` panics when an undefined field has a null value.

For example, when we use `nori_tokenizer` with explain=true, `ExplainAnalyzeToken` fails to unmarshal the response.

Here's an example of commands to confirm that `nori-tokenizer` contains the problematic fields:
(Launch ES with Docker and install nori-tokenizer)

% docker run --rm -p '127.0.0.1:9200:9200' -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.15.2 /bin/bash -c "bin/elasticsearch-plugin install -b analysis-nori && bin/elasticsearch"


(Query the analyzer - please note the null values in the response)

% curl -s -H 'Content-Type: application/json' http://localhost:9200/_analyze -XPOST -d '{"tokenizer": "nori_tokenizer", "text": "안녕하세요.", "explain": true}'| jq '.detail.tokenizer.tokens[0]'


Response:

{
  "token": "안녕",
  "start_offset": 0,
  "end_offset": 2,
  "type": "word",
  "position": 0,
  "bytes": "[ec 95 88 eb 85 95]",
  "leftPOS": "NNG(General Noun)",
  "morphemes": null,  // <- here
  "posType": "MORPHEME",
  "positionLength": 1,
  "reading": null,  // <- here
  "rightPOS": "NNG(General Noun)",
  "termFrequency": 1
}


It seems that the `default` clause of the switch statement may not handle the null value.
https://github.com/elastic/go-elasticsearch/blob/21d8bc238310758a5eb7d94a663a21911563bad8/typedapi/types/explainanalyzetoken.go#L188-L201

Especially:

s.ExplainAnalyzeToken[key] = *raw  // panics if raw is nil

Expected Behavior

Successfully unmarshal the response without panic.

Steps to Reproduce

package main

import (
	"encoding/json"
	"fmt"

	"github.com/elastic/go-elasticsearch/v8/typedapi/types"
)

func main() {
	example := `{
	  "token": "안녕",
	  "start_offset": 0,
	  "end_offset": 2,
	  "type": "word",
	  "position": 0,
	  "bytes": "[ec 95 88 eb 85 95]",
	  "leftPOS": "NNG(General Noun)",
	  "morphemes": null,
	  "posType": "MORPHEME",
	  "positionLength": 1,
	  "reading": null,
	  "rightPOS": "NNG(General Noun)",
	  "termFrequency": 1
	}`

	var token types.ExplainAnalyzeToken
	err := json.Unmarshal([]byte(example), &token)
	if err != nil {
		panic(err)
	}

	fmt.Println(token)
}

Environment Details

- OS: linux, macOS
- Architecture: arm64

Relevant log output

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xcb388]

goroutine 1 [running]:
github.com/elastic/go-elasticsearch/v8/typedapi/types.(*ExplainAnalyzeToken).UnmarshalJSON(0x4000078000, {0x4000076580, 0x145, 0x160})
        /go/pkg/mod/github.com/elastic/go-elasticsearch/v8@v8.19.0/typedapi/types/explainanalyzetoken.go:198 +0x388
encoding/json.(*decodeState).object(0x400007a000, {0xdfb00?, 0x4000078000?, 0xbf048?})
        /usr/local/go/src/encoding/json/decode.go:604 +0x5c0
encoding/json.(*decodeState).value(0x400007a000, {0xdfb00?, 0x4000078000?, 0xbd020?})
        /usr/local/go/src/encoding/json/decode.go:374 +0x40
encoding/json.(*decodeState).unmarshal(0x400007a000, {0xdfb00?, 0x4000078000?})
        /usr/local/go/src/encoding/json/decode.go:181 +0x100
encoding/json.Unmarshal({0x4000076580, 0x145, 0x160}, {0xdfb00, 0x4000078000})
        /usr/local/go/src/encoding/json/decode.go:108 +0xe4
main.main()
        /path/to/explain_analyze_token/main.go:28 +0x4c
exit status 2

[MattDevy]

Hi @jnory , thank you for reporting this bug! I will check it out and get back to you