From 4aedfaec6d1e7381354563ba174a77de2838eb08 Mon Sep 17 00:00:00 2001 From: lcawl Date: Tue, 18 Feb 2025 11:45:47 -0800 Subject: [PATCH 1/4] Add Elasticsearch release notes --- docs/en/install-upgrade/breaking.asciidoc | 16 +- .../release-notes-elasticsearch.asciidoc | 1155 ++++++++++++++++- .../release-notes-kibana.asciidoc | 10 +- .../release-notes-logstash.asciidoc | 2 +- .../release-notes-security.asciidoc | 2 +- .../release-notes/release-notes.asciidoc | 8 +- 6 files changed, 1172 insertions(+), 21 deletions(-) diff --git a/docs/en/install-upgrade/breaking.asciidoc b/docs/en/install-upgrade/breaking.asciidoc index d8a82e4c0..1e5456129 100644 --- a/docs/en/install-upgrade/breaking.asciidoc +++ b/docs/en/install-upgrade/breaking.asciidoc @@ -5,14 +5,14 @@ Before you upgrade, you must review the breaking changes for each product you use and make the necessary changes so your code is compatible with {version}. // tag::breaking-changes-links[] -** {apm-guide-ref}/apm-breaking.html[APM breaking changes] -** {beats-ref}/breaking-changes.html[{beats} breaking changes] -** {ref}/breaking-changes.html[{es} migration guide] -** {security-guide}/release-notes.html[{elastic-sec} release notes] -** {enterprise-search-ref}/changelog.html[{ents} release notes] -** {fleet-guide}/release-notes.html[{fleet} and {agent} release notes] -** {kibana-ref}/release-notes.html[{kib} release notes] -** {logstash-ref}/breaking-changes.html[{ls} breaking changes] +// ** {apm-guide-ref}/apm-breaking.html[APM breaking changes] +// ** {beats-ref}/breaking-changes.html[{beats} breaking changes] +** <> +** <> +// ** {enterprise-search-ref}/changelog.html[{ents} release notes] +// ** {fleet-guide}/release-notes.html[{fleet} and {agent} release notes] +** <> +** <> // end::breaking-changes-links[] // tag::breaking-changes-admon[] diff --git a/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc index 55548e966..1af6bcf80 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc @@ -1,8 +1,1159 @@ -[[release-notes-elasticsearch-9.0.0-beta1]] +[[release-notes-elasticsearch-9.0.0]] = {es} version 9.0.0-beta1 ++++ {es} ++++ -coming::[9.0.0-beta1] +[discrete] +[[breaking-changes-9.0]] +== Breaking changes + +The following changes in {es} 9.0 might affect your applications +and prevent them from operating normally. +Before upgrading to 9.0, review these changes and take the described steps +to mitigate the impact. + +[discrete] +[[breaking_90_aggregations_changes]] +=== Aggregations changes + +[[remove_date_histogram_boolean_support]] +.Remove date histogram boolean support +[%collapsible] +==== +*Details* + +Elasticsearch no longer allows running Date Histogram aggregations over boolean fields. Instead, use Terms aggregation for boolean fields. + +*Impact* + +We expect the impact to be minimal, as this never produced good results, and has been deprecated for years. +==== + +[discrete] +[[breaking_90_analysis_changes]] +=== Analysis changes + +[[snowball_stemmers_have_been_upgraded]] +.Snowball stemmers have been upgraded +[%collapsible] +==== +*Details* + +Lucene 10 ships with an upgrade of its Snowball stemmers. For details see https://github.com/apache/lucene/issues/13209. Users using Snowball stemmers that are experiencing changes in search behaviour on existing data are advised to reindex. + +*Impact* + +The upgrade should generally provide improved stemming results. Small changes in token analysis can lead to mismatches with previously index data, so existing indices using Snowball stemmers as part of their analysis chain should be reindexed. +==== + +[[german2_snowball_stemmer_an_alias_for_german_stemmer]] +.The "german2" snowball stemmer is now an alias for the "german" stemmer +[%collapsible] +==== +*Details* + +Lucene 10 has merged the improved "german2" snowball language stemmer with the "german" stemmer. For Elasticsearch, "german2" is now a deprecated alias for "german". This may results in slightly different tokens being generated for terms with umlaut substitution (like "ue" for "ΓΌ" etc...) + +*Impact* + +Replace usages of "german2" with "german" in analysis configuration. Old indices that use the "german" stemmer should be reindexed if possible. +==== + +[[persian_analyzer_has_stemmer_by_default]] +.The 'persian' analyzer has stemmer by default +[%collapsible] +==== +*Details* + +Lucene 10 has added a final stemming step to its PersianAnalyzer that Elasticsearch exposes as 'persian' analyzer. Existing indices will keep the old non-stemming behaviour while new indices will see the updated behaviour with added stemming. Users that wish to maintain the non-stemming behaviour need to define their own analyzer as outlined in https://www.elastic.co/guide/en/elasticsearch/reference/8.15/analysis-lang-analyzer.html#persian-analyzer. Users that wish to use the new stemming behaviour for existing indices will have to reindex their data. + +*Impact* + +Indexing with the 'persian' analyzer will produce slightly different tokens. Users should check if this impacts their search results. If they wish to maintain the legacy non-stemming behaviour they can define their own analyzer equivalent as explained in https://www.elastic.co/guide/en/elasticsearch/reference/8.15/analysis-lang-analyzer.html#persian-analyzer. +==== + +[[korean_dictionary_for_nori_has_been_updated]] +.The Korean dictionary for Nori has been updated +[%collapsible] +==== +*Details* + +Lucene 10 ships with an updated Korean dictionary (mecab-ko-dic-2.1.1). For details see https://github.com/apache/lucene/issues/11452. Users experiencing changes in search behaviour on existing data are advised to reindex. + +*Impact* + +The change is small and should generally provide better analysis results. Existing indices for full-text use cases should be reindexed though. +==== + +[discrete] +[[breaking_90_cluster_and_node_setting_changes]] +=== Cluster and node setting changes + +[[configuring_bind_dn_in_an_ldap_or_active_directory_ad_realm_without_corresponding_bind_password_will_prevent_node_from_starting]] +.Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password will prevent node from starting +[%collapsible] +==== +*Details* + +For LDAP or AD authentication realms, setting a bind DN (via the `xpack.security.authc.realms.ldap.*.bind_dn` or `xpack.security.authc.realms.active_directory.*.bind_dn` realm settings) without a bind password is a misconfiguration that may prevent successful authentication to the node. Nodes will fail to start if a bind DN is specified without a password. + +*Impact* + +If you have a bind DN configured for an LDAP or AD authentication realm, set a bind password for {ref}/ldap-realm.html#ldap-realm-configuration[LDAP] or {ref}/active-directory-realm.html#ad-realm-configuration[Active Directory]. Configuring a bind DN without a password prevents the misconfigured node from starting. +==== + +[[deprecated_tracing_apm_settings_got_removed]] +.Deprecated tracing.apm.* settings got removed. +[%collapsible] +==== +*Details* + +Deprecated `tracing.apm.*` settings got removed, use respective `telemetry.*` / `telemetry.tracing.*` settings instead. + +*Impact* + +9.x nodes will refuse to start if any such setting (including secret settings) is still present. +==== + +[[limit_bytesizeunit_to_2_decimals]] +.Limit `ByteSizeUnit` to 2 decimals +[%collapsible] +==== +*Details* + +In the past, byte values like `1.25 mb` were allowed but deprecated. Now, values with up to two decimal places are allowed, unless the unit is bytes, in which case no decimals are allowed. Values with too many decimal places result in an error. + +*Impact* + +Values with more than two decimal places, like `0.123 mb` will be rejected as an error, where in the past, they'd be accepted with a deprecation warning. +==== + +[[minimum_shard_balancer_threshold_1_0]] +.Minimum shard balancer threshold is now 1.0 +[%collapsible] +==== +*Details* + +Earlier versions of {es} accepted any non-negative value for `cluster.routing.allocation.balance.threshold`, but values smaller than `1.0` do not make sense and have been ignored since version 8.6.1. From 9.0.0 these nonsensical values are now forbidden. + +*Impact* + +Do not set `cluster.routing.allocation.balance.threshold` to a value less than `1.0`. +==== + +[[remove_tlsv1_1_from_default_protocols]] +.Remove TLSv1.1 from default protocols +[%collapsible] +==== +*Details* + +TLSv1.1 is no longer enabled by default. Prior to version 9.0, Elasticsearch would attempt to enable TLSv1.1 if the JDK supported it. In most cases, including all cases where Elasticsearch 8 was running with the bundled JDK, the JDK would not support TLSv1.1, so that protocol would not be available in Elasticsearch. However, if Elasticsearch was running on an old JDK or a JDK that have been reconfigured to support TLSv1.1, then the protocol would automatically be available within Elasticsearch. As of Elasticsearch 9.0, this is no longer true. If you wish to enable TLSv1.1 then you must enable it within the JDK and also enable it within Elasticsearch by using the `ssl.supported_protocols` setting. + +*Impact* + +Most users will not be impacted. If your Elastisearch 8 cluster was using a custom JDK and you relied on TLSv1.1, then you will need to explicitly enable TLSv1.1 within Elasticsearch (as well as enabling it within your JDK) +==== + +[[remove_client_type_setting]] +.Remove `client.type` setting +[%collapsible] +==== +*Details* + +The node setting `client.type` has been ignored since the node client was removed in 8.0. The setting is now removed. + +*Impact* + +Remove the `client.type` setting from `elasticsearch.yml` +==== + +[[remove_cluster_routing_allocation_disk_watermark_enable_for_single_data_node_setting]] +.Remove `cluster.routing.allocation.disk.watermark.enable_for_single_data_node` setting +[%collapsible] +==== +*Details* + +Prior to 7.8, whenever a cluster had only a single data node, the watermarks would not be respected. In order to change this in 7.8+ in a backwards compatible way, we introduced the `cluster.routing.allocation.disk.watermark.enable_for_single_data_node` node setting. The setting was deprecated in 7.14 and was made to accept only true in 8.0 + +*Impact* + +No known end user impact +==== + +[[remove_deprecated_xpack_searchable_snapshot_allocate_on_rolling_restart_setting]] +.Remove deprecated `xpack.searchable.snapshot.allocate_on_rolling_restart` setting +[%collapsible] +==== +*Details* + +The `xpack.searchable.snapshot.allocate_on_rolling_restart` setting was created as an escape-hatch just in case relying on the `cluster.routing.allocation.enable=primaries` setting for allocating searchable snapshots during rolling restarts had some unintended side-effects. It has been deprecated since 8.2.0. + +*Impact* + +Remove `xpack.searchable.snapshot.allocate_on_rolling_restart` from your settings if present. +==== + +[[remove_unsupported_legacy_value_for_discovery_type]] +.Remove unsupported legacy value for `discovery.type` +[%collapsible] +==== +*Details* + +Earlier versions of {es} had a `discovery.type` setting which permitted values that referred to legacy discovery types. From v9.0.0 onwards, the only supported values for this setting are `multi-node` (the default) and `single-node`. + +*Impact* + +Remove any value for `discovery.type` from your `elasticsearch.yml` configuration file. +==== + +[discrete] +[[breaking_90_index_setting_changes]] +=== Index setting changes + +[[remove_ability_to_read_frozen_indices]] +.Remove the ability to read frozen indices +[%collapsible] +==== +*Details* + +The ability to read frozen indices has been removed. (Frozen indices are no longer useful due to improvements in heap memory usage. The ability to freeze indices was removed in 8.0.) + +*Impact* + +Users must unfreeze any frozen indices before upgrading. +==== + +[discrete] +[[breaking_90_ingest_changes]] +=== Ingest changes + +[[remove_ecs_option_on_user_agent_processor]] +.Remove `ecs` option on `user_agent` processor +[%collapsible] +==== +*Details* + +The `user_agent` ingest processor no longer accepts the `ecs` option. (It was previously deprecated and ignored.) + +*Impact* + +Users should stop using the `ecs` option when creating instances of the `user_agent` ingest processor. The option will be removed from existing processors stored in the cluster state on upgrade. +==== + +[[remove_ignored_fallback_option_on_geoip_processor]] +.Remove ignored fallback option on GeoIP processor +[%collapsible] +==== +*Details* + +The option fallback_to_default_databases on the geoip ingest processor has been removed. (It was deprecated and ignored since 8.0.0.) + +*Impact* + +Customers should stop remove the noop fallback_to_default_databases option on any geoip ingest processors. +==== + +[discrete] +[[breaking_90_logging_changes]] +=== Logging changes + +[[deprecation_logging_value_change_for_data_stream_dataset_event_dataset]] +.Deprecation logging value change for "data_stream.dataset" and "event.dataset" +[%collapsible] +==== +*Details* + +This change modifies the "data_stream.dataset" and "event.dataset" value for deprecation logging +to use the value `elasticsearch.deprecation` instead of `deprecation.elasticsearch`. This is now +consistent with other values where the name of the service is the first part of the key. + +*Impact* + +If you are directly consuming deprecation logs for "data_stream.dataset" and "event.dataset" and filtering on +this value, you will need to update your filters to use `elasticsearch.deprecation` instead of +`deprecation.elasticsearch`. +==== + +[discrete] +[[breaking_90_logs_changes]] +=== Logs changes + +[[conditionally_enable_logsdb_by_default]] +.Conditionally enable logsdb by default +[%collapsible] +==== +*Details* + +Logsdb will be enabled by default for data streams matching with logs-*-* pattern. +If upgrading from 8.x to 9.x and data streams matching with log-*-* do exist, +then Logsdb will not be enabled by default. + +*Impact* + +Logsdb reduce storage footprint in Elasticsearch for logs, but there are side effects +to be taken into account that are described in the Logsdb docs: +https://www.elastic.co/guide/en/elasticsearch/reference/current/logs-data-stream.html#upgrade-to-logsdb-notes +==== + +[discrete] +[[breaking_90_mapping_changes]] +=== Mapping changes + +[[remove_support_for_type_fields_copy_to_boost_in_metadata_field_definition]] +.Remove support for type, fields, copy_to and boost in metadata field definition +[%collapsible] +==== +*Details* + +The type, fields, copy_to and boost parameters are no longer supported in metadata field definition starting with version 9. + +*Impact* + +Users providing type, fields, copy_to or boost as part of metadata field definition should remove them from their mappings. +==== + +[[turn_source_meta_fieldmappers_mode_attribute_into_no_op]] +.Turn `_source` meta fieldmapper's mode attribute into a no-op +[%collapsible] +==== +*Details* + +The `mode` mapping attribute of `_source` metadata field mapper has been turned into a no-op. Instead the `index.mapping.source.mode` index setting should be used to configure source mode. + +*Impact* + +Configuring the `mode` attribute for the `_source` meta field mapper will have no effect on indices created with Elasticsearch 9.0.0 or later. Note that `_source.mode` configured on indices before upgrading to 9.0.0 or later will remain efficive after upgrading. +==== + +[discrete] +[[breaking_90_packaging_changes]] +=== Packaging changes + +[[disable_machine_learning_on_macos_x86_64]] +.Disable machine learning on macOS x86_64 +[%collapsible] +==== +*Details* + +The machine learning plugin is permanently disabled on macOS x86_64. For the last three years Apple has been selling hardware based on the arm64 architecture, and support will increasingly focus on this architecture in the future. Changes to upstream dependencies of Elastic's machine learning functionality have made it unviable for Elastic to continue to build machine learning on macOS x86_64. + +*Impact* + +To continue to use machine learning functionality on macOS please switch to an arm64 machine (Apple silicon). Alternatively, it will still be possible to run Elasticsearch with machine learning enabled in a Docker container on macOS x86_64. +==== + +[discrete] +[[breaking_90_rest_api_changes]] +=== REST API changes + +[[apply_more_strict_parsing_of_actions_in_bulk_api]] +.Apply more strict parsing of actions in bulk API +[%collapsible] +==== +*Details* + +Previously, the following classes of malformed input were deprecated but not rejected in the action lines of the a bulk request: missing closing brace; additional keys after the action (which were ignored); additional data after the closing brace (which was ignored). They will now be considered errors and rejected. + +*Impact* + +Users must provide well-formed input when using the bulk API. (They can request REST API compatibility with v8 to get the previous behaviour back as an interim measure.) +==== + +[[change_most_elasticsearch_timeouts_to_429_response_instead_of_5xx]] +.Change most Elasticsearch timeouts to 429 response instead of 5xx +[%collapsible] +==== +*Details* + +When a timeout occurs in most REST requests, whether via a per-request timeout, or a system default, the request would return a 5xx response code. The response code from those APIs when a timeout occurs is now 429. + +*Impact* + +Adjust any code relying on retrying on 5xx responses for timeouts to look for a 429 response code and inspect the response to determine whether a timeout occured. +==== + +[[error_json_structure_has_changed_when_detailed_errors_are_disabled]] +.Error JSON structure has changed when detailed errors are disabled +[%collapsible] +==== +*Details* + +This change modifies the JSON format of error messages returned to REST clients +when detailed messages are turned off. +Previously, JSON returned when an exception occurred, and `http.detailed_errors.enabled: false` was set, +just consisted of a single `"error"` text field with some basic information. +Setting `http.detailed_errors.enabled: true` (the default) changed this field +to an object with more detailed information. +With this change, non-detailed errors now have the same structure as detailed errors. `"error"` will now always +be an object with, at a minimum, a `"type"` and `"reason"` field. Additional fields are included when detailed +errors are enabled. +To use the previous structure for non-detailed errors, use the v8 REST API. + +*Impact* + +If you have set `http.detailed_errors.enabled: false` (the default is `true`) +the structure of JSON when any exceptions occur now matches the structure when +detailed errors are enabled. +To use the previous structure for non-detailed errors, use the v8 REST API. +==== + +[[remove_any_references_to_org_elasticsearch_core_restapiversion_v_7]] +.Remove any references to org.elasticsearch.core.RestApiVersion#V_7 +[%collapsible] +==== +*Details* + +This PR removes all references to V_7 in the Rest API. V7 features marked for deprecation have been removed. + +*Impact* + +This change is breaking for any external plugins/clients that rely on the V_7 enum or deprecated version 7 functionality +==== + +[[remove_cluster_state_from_cluster_reroute_response]] +.Remove cluster state from `/_cluster/reroute` response +[%collapsible] +==== +*Details* + +The `POST /_cluster/reroute` API no longer returns the cluster state in its response. The `?metric` query parameter to this API now has no effect and its use will be forbidden in a future version. + +*Impact* + +Cease usage of the `?metric` query parameter when calling the `POST /_cluster/reroute` API. +==== + +[[remove_deprecated_local_attribute_from_alias_apis]] +.Remove deprecated local attribute from alias APIs +[%collapsible] +==== +*Details* + +The following APIs no longer accept the `?local` query parameter: `GET /_alias`, `GET /_aliases`, `GET /_alias/{name}`, `HEAD /_alias/{name}`, `GET /{index}/_alias`, `HEAD /{index}/_alias`, `GET /{index}/_alias/{name}`, `HEAD /{index}/_alias/{name}`, `GET /_cat/aliases`, and `GET /_cat/aliases/{alias}`. This parameter has been deprecated and ignored since version 8.12. + +*Impact* + +Cease usage of the `?local` query parameter when calling the listed APIs. +==== + +[[remove_legacy_params_from_range_query]] +.Remove legacy params from range query +[%collapsible] +==== +*Details* + +The deprecated range query parameters `to`, `from`, `include_lower`, and `include_upper` are no longer supported. + +*Impact* + +Users should use `lt`, `lte`, `gt`, and `gte` query parameters instead. +==== + +[[remove_old_knn_search_tech_preview_api_in_v9]] +.Remove old `_knn_search` tech preview API in v9 +[%collapsible] +==== +*Details* + +The original, tech-preview api for vector search, `_knn_search`, has been removed in v9. For all vector search operations, you should utilize the `_search` endpoint. + +*Impact* + +The `_knn_search` API is now inaccessible without providing a compatible-with flag for v8. +==== + +[[remove_support_for_deprecated_force_source_highlighting_parameter]] +.Remove support for deprecated `force_source` highlighting parameter +[%collapsible] +==== +*Details* + +The deprecated highlighting `force_source` parameter is no longer supported. + +*Impact* + +Users should remove usages of the `force_source` parameter from their search requests. +==== + +[[remove_unfreeze_rest_endpoint]] +.Remove unfreeze REST endpoint +[%collapsible] +==== +*Details* + +The `/{index}/_unfreeze` REST endpoint is no longer supported. This API was deprecated, and the corresponding `/{index}/_freeze` endpoint was removed in 8.0. + +*Impact* + +None, since it is not possible to have a frozen index in a version which is readable by Elasticsearch 9.0 +==== + +[[removing_support_for_types_field_in_watcher_search]] +.Removing support for types field in watcher search +[%collapsible] +==== +*Details* + +Previously, setting the `input.search.request.types` field in the payload when creating a watcher to an empty array was allowed, although it resulted in a deprecation warning and had no effect (and any value other than an empty array would result in an error). Now, support for this field is entirely removed, and the empty array will also result in an error. + +*Impact* + +Users should stop setting this field (which did not have any effect anyway). +==== + +[[restrict_connector_apis_to_manage_monitor_connector_privileges]] +.Restrict Connector APIs to manage/monitor_connector privileges +[%collapsible] +==== +*Details* + +Connector APIs now enforce the manage_connector and monitor_connector privileges (introduced in 8.15), replacing the previous reliance on index-level permissions for .elastic-connectors and .elastic-connectors-sync-jobs in API calls. + +*Impact* + +Connector APIs now require manage_connector and monitor_connector privileges +==== + +[[set_allow_partial_search_results_true_by_default]] +.Set allow_partial_search_results=true by default +[%collapsible] +==== +*Details* + +Before this change, in case of shard failures, EQL queries always returned an error. With this change, they will keep running and will return partial results. + +*Impact* + +EQL queries that would previously fail due to shard failures, will now succeed and return partial results. The previous defaults can be restored by setting `xpack.eql.default_allow_partial_results` cluster setting to `false` or setting with `allow_partial_search_results` to `false` in the query request. +==== + +[discrete] +[[breaking_90_search_changes1]] +=== Search changes + +[[adjust_random_score_default_field_to_seq_no_field]] +.Adjust `random_score` default field to `_seq_no` field +[%collapsible] +==== +*Details* + +When providing a 'seed' parameter to a 'random_score' function in the 'function_score' query but NOT providing a 'field', the default 'field' is switched from '_id' to '_seq_no'. + +*Impact* + +The random scoring and ordering may change when providing a 'seed' and not providing a 'field' to a 'random_score' function. +==== + +[[change_semantic_text_to_act_like_normal_text_field]] +.Change semantic text to act like a normal text field +[%collapsible] +==== +*Details* + +The previous semantic_text format used a complex subfield structure in _source to store the embeddings. This complicated interactions/integrations with semantic_text fields and _source in general. This new semantic_text format treats it as a normal text field, where the field's value in _source is the value assigned by the user. + +*Impact* + +Users who parsed the subfield structure of the previous semantic_text format in _source will need to update their parsing logic. The new format does not directly expose the chunks and embeddings generated from the input text. The new format will be applied to all new indices, any existing indices will continue to use the previous format. +==== + +[discrete] +[[breaking_90_transform_changes]] +=== Transform changes + +[[remove_data_frame_transforms_roles]] +.Remove `data_frame_transforms` roles +[%collapsible] +==== +*Details* + +`data_frame_transforms_admin` and `data_frame_transforms_user` were deprecated in Elasticsearch 7 and are being removed in Elasticsearch 9. `data_frame_transforms_admin` is now `transform_admin`. `data_frame_transforms_user` is now `transform_user`. Users must call the `_update` API to replace the permissions on the Transform before the Transform can be started. + +*Impact* + +Transforms created with either the `data_frame_transforms_admin` or the `data_frame_transforms_user` role will fail to start. The Transform will remain in a `stopped` state, and its health will be red while displaying permission failures. +==== + +The following pull requests relate to breaking changes: + +Aggregations:: +* Remove date histogram boolean support {es-pull}118484[#118484] + +Allocation:: +* Increase minimum threshold in shard balancer {es-pull}115831[#115831] +* Remove `cluster.routing.allocation.disk.watermark.enable_for_single_data_node` setting {es-pull}114207[#114207] +* Remove cluster state from `/_cluster/reroute` response {es-pull}114231[#114231] (issue: {es-issue}88978[#88978]) + +Analysis:: +* Snowball stemmers have been upgraded {es-pull}114146[#114146] +* The 'german2' stemmer is now an alias for the 'german' snowball stemmer {es-pull}113614[#113614] +* The 'persian' analyzer has stemmer by default {es-pull}113482[#113482] (issue: {es-issue}113050[#113050]) +* The Korean dictionary for Nori has been updated {es-pull}114124[#114124] + +Authentication:: +* Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password +will prevent node from starting {es-pull}118366[#118366] + +Cluster Coordination:: +* Remove unsupported legacy value for `discovery.type` {es-pull}112903[#112903] + +EQL:: +* Set allow_partial_search_results=true by default {es-pull}120267[#120267] + +Extract&Transform:: +* Restrict Connector APIs to manage/monitor_connector privileges {es-pull}119863[#119863] + +Highlighting:: +* Remove support for deprecated `force_source` highlighting parameter {es-pull}116943[#116943] + +Indices APIs:: +* Apply more strict parsing of actions in bulk API {es-pull}115923[#115923] +* Remove deprecated local attribute from alias APIs {es-pull}115393[#115393] +* Remove the ability to read frozen indices {es-pull}120108[#120108] +* Remove unfreeze REST endpoint {es-pull}119227[#119227] + +Infra/Core:: +* Change Elasticsearch timeouts to 429 response instead of 5xx {es-pull}116026[#116026] +* Limit `ByteSizeUnit` to 2 decimals {es-pull}120142[#120142] +* Remove `client.type` setting {es-pull}118192[#118192] (issue: {es-issue}104574[#104574]) +* Remove any references to org.elasticsearch.core.RestApiVersion#V_7 {es-pull}118103[#118103] + +Infra/Logging:: +* Change `deprecation.elasticsearch` keyword to `elasticsearch.deprecation` {es-pull}117933[#117933] (issue: {es-issue}83251[#83251]) + +Infra/Metrics:: +* Deprecated tracing.apm.* settings got removed. {es-pull}119926[#119926] + +Infra/REST API:: +* Output a consistent format when generating error json {es-pull}90529[#90529] (issue: {es-issue}89387[#89387]) + +Ingest Node:: +* Remove `ecs` option on `user_agent` processor {es-pull}116077[#116077] +* Remove ignored fallback option on GeoIP processor {es-pull}116112[#116112] + +Logs:: +* Conditionally enable logsdb by default for data streams matching with logs-*-* pattern. {es-pull}121049[#121049] (issue: {es-issue}106489[#106489]) + +Machine Learning:: +* Disable machine learning on macOS x86_64 {es-pull}104125[#104125] + +Mapping:: +* Remove support for type, fields, `copy_to` and boost in metadata field definition {es-pull}118825[#118825] +* Turn `_source` meta fieldmapper's mode attribute into a no-op {es-pull}119072[#119072] (issue: {es-issue}118596[#118596]) + +Search:: +* Adjust `random_score` default field to `_seq_no` field {es-pull}118671[#118671] +* Change Semantic Text To Act Like A Normal Text Field {es-pull}120813[#120813] +* Remove legacy params from range query {es-pull}116970[#116970] + +Snapshot/Restore:: +* Remove deprecated `xpack.searchable.snapshot.allocate_on_rolling_restart` setting {es-pull}114202[#114202] + +TLS:: +* Remove TLSv1.1 from default protocols {es-pull}121731[#121731] + +Transform:: +* Remove `data_frame_transforms` roles {es-pull}117519[#117519] + +Vector Search:: +* Remove old `_knn_search` tech preview API in v9 {es-pull}118104[#118104] + +Watcher:: +* Removing support for types field in watcher search {es-pull}120748[#120748] + +[discrete] +[[deprecated-9.0]] +== Deprecations + +The following functionality has been deprecated in {es} 9.0 +and will be removed in a future version. +While this won't have an immediate impact on your applications, +we strongly encourage you to take the described steps to update your code +after upgrading to 9.0. + +To find out if you are using any deprecated functionality, +enable {ref}/logging.html#deprecation-logging[deprecation logging]. + +[discrete] +[[deprecations_90_authorization]] +=== Authorization deprecations + +[[deprecate_certificate_based_remote_cluster_security_model]] +.Deprecate certificate based remote cluster security model +[%collapsible] +==== +*Details* + +-| {ref}/remote-clusters-cert.html[Certificate-based remote cluster security model] is deprecated and will be removed in a future major version. Users are encouraged to {ref}/remote-clusters-migrate.html[migrate remote clusters from certificate to API key authentication]. The {ref}/remote-clusters-api-key.html[API key-based security model] is preferred way to configure remote clusters, as it allows to follow security best practices when setting up remote cluster connections and defining fine-grained access control. + +*Impact* + +-| If you have configured remote clusters with certificate-based security model, you should {ref}/remote-clusters-migrate.html[migrate remote clusters from certificate to API key authentication]. Configuring a remote cluster using {ref}/remote-clusters-cert.html[certificate authentication], generates a warning in the deprecation logs. +==== + +[discrete] +[[deprecations_90_es_ql]] +=== ES|QL deprecations + +[[drop_support_for_brackets_from_metadata_syntax]] +.Drop support for brackets from METADATA syntax +[%collapsible] +==== +*Details* + +Please describe the details of this change for the release notes. You can use asciidoc. + +*Impact* + +Please describe the impact of this change to users +==== + +[discrete] +[[deprecations_90_rest_api]] +=== REST API deprecations + +[[document_type_deprecated_on_simulate_pipeline_api]] +.Document `_type` deprecated on simulate pipeline API +[%collapsible] +==== +*Details* + +Passing a document with a `_type` property is deprecated in the `/_ingest/pipeline/{id}/_simulate` and `/_ingest/pipeline/_simulate` APIs. + +*Impact* + +Users should already have stopped using mapping types, which were deprecated in {es} 7. This deprecation warning will fire if they specify mapping types on documents pass to the simulate pipeline API. +==== + +[[inference_api_deprecate_elser_service]] +.Deprecate elser service +[%collapsible] +==== +*Details* + +The `elser` service of the inference API will be removed in an upcoming release. Please use the elasticsearch service instead. + +*Impact* + +In the current version there is no impact. In a future version, users of the `elser` service will no longer be able to use it, and will be required to use the `elasticsearch` service to access elser through the inference API. +==== + +[discrete] +[[deprecations_90_rollup]] +=== Rollup deprecations + +[[emit_deprecation_warning_when_executing_one_of_rollup_apis]] +.Emit deprecation warning when executing one of the rollup APIs +[%collapsible] +==== +*Details* + +Rollup is already deprecated since 8.11.0 via documentation and since 8.15.0 it is no longer possible to create new rollup jobs in clusters without rollup usage. This change updates the rollup APIs to emit a deprecation warning. + +*Impact* + +Returning a deprecation warning when using one of the rollup APIs. +==== + +[discrete] +[[deprecations_90_transform]] +=== Transform deprecations + +[[removing_index_alias_creation_for_deprecated_transforms_notification_index]] +.Removing index alias creation for deprecated transforms notification index +[%collapsible] +==== +*Details* + +As part of the migration from 7.x to 8.x, the `.data-frame-notifications-1` index was deprecated and replaced with the `.transform-notifications-000002` index. The index is no longer created by default, all writes are directed to the new index, and any clusters with the deprecated index will have an alias created to ensure that reads are still retrieving data that was written to the index before the migration to 8.x. This change removes the alias from the deprecated index in 9.x. Any clusters with the alias present will retain it, but it will not be created on new clusters. + +*Impact* + +No known end user impact. +==== + +The following pull requests relate to deprecations: + +ES|QL:: +* Drop support for brackets from METADATA syntax {es-pull}119846[#119846] (issue: {es-issue}115401[#115401]) + +Ingest Node:: +* Fix `_type` deprecation on simulate pipeline API {es-pull}116259[#116259] + +Machine Learning:: +* Removing index alias creation for deprecated transforms notification index {es-pull}117583[#117583] +* [Inference API] Deprecate elser service {es-pull}113216[#113216] + +Rollup:: +* Emit deprecation warning when executing one of the rollup APIs {es-pull}113131[#113131] + +Security:: +* Deprecate certificate based remote cluster security model {es-pull}120806[#120806] + +[[bug-9.0.0]] +[float] +== Bug fixes + +Aggregations:: +* Handle with `illegalArgumentExceptions` negative values in HDR percentile aggregations {es-pull}116174[#116174] (issue: {es-issue}115777[#115777]) + +Analysis:: +* Analyze API to return 400 for wrong custom analyzer {es-pull}121568[#121568] (issue: {es-issue}121443[#121443]) + +CAT APIs:: +* Fix cat_component_templates documentation {es-pull}120487[#120487] + +CRUD:: +* Preserve thread context when waiting for segment generation in RTG {es-pull}114623[#114623] +* Preserve thread context when waiting for segment generation in RTG {es-pull}117148[#117148] + +Data streams:: +* Avoid updating settings version in `MetadataMigrateToDataStreamService` when settings have not changed {es-pull}118704[#118704] +* Block-writes cannot be added after read-only {es-pull}119007[#119007] (issue: {es-issue}119002[#119002]) +* Ensure removal of index blocks does not leave key with null value {es-pull}122246[#122246] +* Match dot prefix of migrated DS backing index with the source index {es-pull}120042[#120042] +* Refresh source index before reindexing data stream index {es-pull}120752[#120752] (issue: {es-issue}120314[#120314]) +* `ReindexDataStreamIndex` bug in assertion caused by reference equality {es-pull}121325[#121325] + +Downsampling:: +* Copy metrics and `default_metric` properties when downsampling `aggregate_metric_double` {es-pull}121727[#121727] (issues: {es-issue}119696[#119696], {es-issue}96076[#96076]) + +EQL:: +* Fix JOIN command validation (not supported) {es-pull}122011[#122011] + +ES|QL:: +* Allow the data type of `null` in filters {es-pull}118324[#118324] (issue: {es-issue}116351[#116351]) +* Correct line and column numbers of missing named parameters {es-pull}120852[#120852] +* Drop null columns in text formats {es-pull}117643[#117643] (issue: {es-issue}116848[#116848]) +* Fix ENRICH validation for use of wildcards {es-pull}121911[#121911] +* Fix ROUND() with unsigned longs throwing in some edge cases {es-pull}119536[#119536] +* Fix TDigestState.read CB leaks {es-pull}114303[#114303] (issue: {es-issue}114194[#114194]) +* Fix TopN row size estimate {es-pull}119476[#119476] (issue: {es-issue}106956[#106956]) +* Fix `AbstractShapeGeometryFieldMapperTests` {es-pull}119265[#119265] (issue: {es-issue}119201[#119201]) +* Fix a bug in TOP {es-pull}121552[#121552] +* Fix async stop sometimes not properly collecting result {es-pull}121843[#121843] (issue: {es-issue}121249[#121249]) +* Fix attribute set equals {es-pull}118823[#118823] +* Fix double lookup failure on ESQL {es-pull}115616[#115616] (issue: {es-issue}111398[#111398]) +* Fix queries with document level security on lookup indexes {es-pull}120617[#120617] (issue: {es-issue}120509[#120509]) +* Fix writing for LOOKUP status {es-pull}119296[#119296] (issue: {es-issue}119086[#119086]) +* Limit memory usage of `fold` {es-pull}118602[#118602] +* Limit size of query {es-pull}117898[#117898] +* Remove redundant sorts from execution plan {es-pull}121156[#121156] +* Revert unwanted ES|QL lexer changes from PR #120354 {es-pull}120538[#120538] + +Health:: +* Do not recommend increasing `max_shards_per_node` {es-pull}120458[#120458] + +Indices APIs:: +* Add `?master_timeout` to `POST /_ilm/migrate_to_data_tiers` {es-pull}120883[#120883] +* Fix broken yaml test `30_create_from` {es-pull}120662[#120662] +* Include hidden indices in `DeprecationInfoAction` {es-pull}118035[#118035] (issue: {es-issue}118020[#118020]) + +Inference:: +* [Inference API] Put back legacy EIS URL setting {es-pull}121207[#121207] + +Infra/Core:: +* Epoch Millis Rounding Down and Not Up 2 {es-pull}118353[#118353] +* Wrap jackson exception on malformed json string {es-pull}114445[#114445] (issue: {es-issue}114142[#114142]) + +Infra/Logging:: +* Move `SlowLogFieldProvider` instantiation to node construction {es-pull}117949[#117949] + +Infra/Metrics:: +* Make `randomInstantBetween` always return value in range [minInstant, `maxInstant]` {es-pull}114177[#114177] + +Infra/Plugins:: +* Remove unnecessary entitlement {es-pull}120959[#120959] +* Restrict agent entitlements to the system classloader unnamed module {es-pull}120546[#120546] + +Infra/REST API:: +* Fixed a `NullPointerException` in `_capabilities` API when the `path` parameter is null. {es-pull}113413[#113413] (issue: {es-issue}113413[#113413]) + +Infra/Scripting:: +* Register mustache size limit setting {es-pull}119291[#119291] + +Infra/Settings:: +* Don't allow secure settings in YML config (109115) {es-pull}115779[#115779] (issue: {es-issue}109115[#109115]) + +Ingest Node:: +* Add warning headers for ingest pipelines containing special characters {es-pull}114837[#114837] (issue: {es-issue}104411[#104411]) +* Fix geoip databases index access after system feature migration {es-pull}121196[#121196] + +Logs:: +* Always check if index mode is logsdb {es-pull}116922[#116922] + +Machine Learning:: +* Change format for Unified Chat {es-pull}121396[#121396] +* Fix get all inference endponts not returning multiple endpoints sharing model deployment {es-pull}121821[#121821] +* Fixing bedrock event executor terminated cache issue {es-pull}118177[#118177] (issue: {es-issue}117916[#117916]) +* Fixing bug setting index when parsing Google Vertex AI results {es-pull}117287[#117287] +* Updating Inference Update API documentation to have the correct PUT method {es-pull}121048[#121048] +* Wait for up to 2 seconds for yellow status before starting search {es-pull}115938[#115938] (issues: {es-issue}107777[#107777], {es-issue}105955[#105955], {es-issue}107815[#107815], {es-issue}112191[#112191]) +* [Inference API] Fix unique ID message for inference ID matches trained model ID {es-pull}119543[#119543] (issue: {es-issue}111312[#111312]) + +Mapping:: +* Enable New Semantic Text Format Only On Newly Created Indices {es-pull}121556[#121556] +* Fix propagation of dynamic mapping parameter when applying `copy_to` {es-pull}121109[#121109] (issue: {es-issue}113049[#113049]) +* Fix realtime get of nested fields with synthetic source {es-pull}119575[#119575] (issue: {es-issue}119553[#119553]) +* Merge field mappers when updating mappings with [subobjects:false] {es-pull}120370[#120370] (issue: {es-issue}120216[#120216]) +* Tweak `copy_to` handling in synthetic `_source` to account for nested objects {es-pull}120974[#120974] (issue: {es-issue}120831[#120831]) + +Network:: +* Remove ChunkedToXContentBuilder {es-pull}119310[#119310] (issue: {es-issue}118647[#118647]) + +Search:: +* Catch and handle disconnect exceptions in search {es-pull}115836[#115836] +* Fix leak in `DfsQueryPhase` and introduce search disconnect stress test {es-pull}116060[#116060] (issue: {es-issue}115056[#115056]) +* Fix/QueryBuilderBWCIT_muted_test {es-pull}117831[#117831] +* In this pr, a 400 error is returned when _source / _seq_no / _feature / _nested_path / _field_names is requested, rather a 5xx {es-pull}117229[#117229] +* Inconsistency in the _analyzer api when the index is not included {es-pull}115930[#115930] +* Remove duplicate code in ESIntegTestCase {es-pull}120799[#120799] +* SearchStatesIt failures reported by CI {es-pull}117618[#117618] (issues: {es-issue}116617[#116617], {es-issue}116618[#116618]) +* Skip fetching _inference_fields field in legacy semantic_text format {es-pull}121720[#121720] +* Test/107515 restore template with match only text mapper it fail {es-pull}120392[#120392] (issue: {es-issue}107515[#107515]) +* Updated Date Range to Follow Documentation When Assuming Missing Values {es-pull}112258[#112258] (issue: {es-issue}111484[#111484]) +* `CrossClusterIT` `testCancel` failure {es-pull}117750[#117750] (issue: {es-issue}108061[#108061]) +* `SearchServiceTests.testParseSourceValidation` failure {es-pull}117963[#117963] + +Snapshot/Restore:: +* Add undeclared Azure settings, modify test to exercise them {es-pull}118634[#118634] +* Fork post-snapshot-delete cleanup off master thread {es-pull}122047[#122047] +* Retry throttled snapshot deletions {es-pull}113237[#113237] +* Use the system index descriptor in the snapshot blob cache cleanup task {es-pull}120937[#120937] (issue: {es-issue}120518[#120518]) + +Store:: +* Do not capture `ClusterChangedEvent` in `IndicesStore` call to #onClusterStateShardsClosed {es-pull}120193[#120193] + +Vector Search:: +* Apply default k for knn query eagerly {es-pull}118774[#118774] +* Fix `bbq_hnsw` merge file cleanup on random IO exceptions {es-pull}119691[#119691] (issue: {es-issue}119392[#119392]) + +Watcher:: +* Watcher history index has too many indexed fields - {es-pull}117701[#117701] (issue: {es-issue}71479[#71479]) + +[[enhancement-9.0.0]] +[float] +== Enhancements + +Allocation:: +* Add a not-master state for desired balance {es-pull}116904[#116904] +* Only publish desired balance gauges on master {es-pull}115383[#115383] +* Reset relocation/allocation failure counter on node join/shutdown {es-pull}119968[#119968] + +Authentication:: +* Allow `SSHA-256` for API key credential hash {es-pull}120997[#120997] + +Authorization:: +* Allow kibana_system user to manage .reindexed-v8-internal.alerts indices {es-pull}118959[#118959] +* Grant necessary Kibana application privileges to `reporting_user` role {es-pull}118058[#118058] +* Make reserved built-in roles queryable {es-pull}117581[#117581] +* [Security Solution] Add `create_index` to `kibana_system` role for index/DS `.logs-endpoint.action.responses-*` {es-pull}115241[#115241] +* [Security Solution] allows `kibana_system` user to manage .reindexed-v8-* Security Solution indices {es-pull}119054[#119054] + +CCS:: +* Resolve/cluster allows querying for cluster info only (no index expression required) {es-pull}119898[#119898] + +CRUD:: +* Remove INDEX_REFRESH_BLOCK after index becomes searchable {es-pull}120807[#120807] +* Suppress merge-on-recovery for older indices {es-pull}113462[#113462] + +Cluster Coordination:: +* Include `clusterApplyListener` in long cluster apply warnings {es-pull}120087[#120087] + +Data streams:: +* Add action to create index from a source index {es-pull}118890[#118890] +* Add index and reindex request settings to speed up reindex {es-pull}119780[#119780] +* Add rest endpoint for `create_from_source_index` {es-pull}119250[#119250] +* Add sanity check to `ReindexDatastreamIndexAction` {es-pull}120231[#120231] +* Adding a migration reindex cancel API {es-pull}118291[#118291] +* Adding get migration reindex status {es-pull}118267[#118267] +* Consistent mapping for OTel log and event bodies {es-pull}120547[#120547] +* Filter deprecated settings when making dest index {es-pull}120163[#120163] +* Ignore closed indices for reindex {es-pull}120244[#120244] +* Improve how reindex data stream index action handles api blocks {es-pull}120084[#120084] +* Initial work on `ReindexDatastreamIndexAction` {es-pull}116996[#116996] +* Make `requests_per_second` configurable to throttle reindexing {es-pull}120207[#120207] +* Optimized index sorting for OTel logs {es-pull}119504[#119504] +* Report Deprecated Indices That Are Flagged To Ignore Migration Reindex As A Warning {es-pull}120629[#120629] +* Update data stream deprecations warnings to new format and filter searchable snapshots from response {es-pull}118562[#118562] + +Distributed:: +* Make various alias retrieval APIs wait for cluster to unblock {es-pull}117230[#117230] +* Metrics for incremental bulk splits {es-pull}116765[#116765] +* Use Azure blob batch API to delete blobs in batches {es-pull}114566[#114566] + +EQL:: +* Add support for partial shard results {es-pull}116388[#116388] +* Optional named arguments for function in map {es-pull}118619[#118619] + +ES|QL:: +* Add ES|QL cross-cluster query telemetry collection {es-pull}119474[#119474] +* Add a `LicenseAware` interface for licensed Nodes {es-pull}118931[#118931] (issue: {es-issue}117405[#117405]) +* Add a `PostAnalysisAware,` distribute verification {es-pull}119798[#119798] +* Add a standard deviation aggregating function: STD_DEV {es-pull}116531[#116531] +* Add cluster level reduction {es-pull}117731[#117731] +* Add nulls support to Categorize {es-pull}117655[#117655] +* Async search responses have CCS metadata while searches are running {es-pull}117265[#117265] +* Check for early termination in Driver {es-pull}118188[#118188] +* Do not serialize `EsIndex` in plan {es-pull}119580[#119580] +* ESQL - Remove restrictions for disjunctions in full text functions {es-pull}118544[#118544] +* ESQL - enabling scoring with METADATA `_score` {es-pull}113120[#113120] +* ESQL Add esql hash function {es-pull}117989[#117989] +* ESQL Support IN operator for Date nanos {es-pull}119772[#119772] (issue: {es-issue}118578[#118578]) +* ESQL: CATEGORIZE as a `BlockHash` {es-pull}114317[#114317] +* ESQL: Enterprise license enforcement for CCS {es-pull}118102[#118102] +* ES|QL: Partial result on demand for async queries {es-pull}118122[#118122] +* Enable KQL function as a tech preview {es-pull}119730[#119730] +* Enable LOOKUP JOIN in non-snapshot builds {es-pull}121193[#121193] (issue: {es-issue}121185[#121185]) +* Enable node-level reduction by default {es-pull}119621[#119621] +* Enable physical plan verification {es-pull}118114[#118114] +* Ensure cluster string could be quoted {es-pull}120355[#120355] +* Esql - Support date nanos in date extract function {es-pull}120727[#120727] (issue: {es-issue}110000[#110000]) +* Esql - support date nanos in date format function {es-pull}120143[#120143] (issue: {es-issue}109994[#109994]) +* Esql Support date nanos on date diff function {es-pull}120645[#120645] (issue: {es-issue}109999[#109999]) +* Esql bucket function for date nanos {es-pull}118474[#118474] (issue: {es-issue}118031[#118031]) +* Esql compare nanos and millis {es-pull}118027[#118027] (issue: {es-issue}116281[#116281]) +* Esql implicit casting for date nanos {es-pull}118697[#118697] (issue: {es-issue}118476[#118476]) +* Extend `TranslationAware` to all pushable expressions {es-pull}120192[#120192] +* Hash functions {es-pull}118938[#118938] +* Implement a `MetricsAware` interface {es-pull}121074[#121074] +* LOOKUP JOIN using field-caps for field mapping {es-pull}117246[#117246] +* Lookup join on multiple join fields not yet supported {es-pull}118858[#118858] +* Move scoring in ES|QL out of snapshot {es-pull}120354[#120354] +* Optimize ST_EXTENT_AGG for `geo_shape` and `cartesian_shape` {es-pull}119889[#119889] +* Push down filter passed lookup join {es-pull}118410[#118410] +* Resume Driver on cancelled or early finished {es-pull}120020[#120020] +* Rewrite TO_UPPER/TO_LOWER comparisons {es-pull}118870[#118870] (issue: {es-issue}118304[#118304]) +* ST_EXTENT_AGG optimize envelope extraction from doc-values for cartesian_shape {es-pull}118802[#118802] +* Smarter field caps with subscribable listener {es-pull}116755[#116755] +* Support partial sort fields in TopN pushdown {es-pull}116043[#116043] (issue: {es-issue}114515[#114515]) +* Support some stats on aggregate_metric_double {es-pull}120343[#120343] (issue: {es-issue}110649[#110649]) +* Take named parameters for identifier and pattern out of snapshot {es-pull}121850[#121850] +* Term query for ES|QL {es-pull}117359[#117359] +* Update grammar to rely on `indexPattern` instead of identifier in join target {es-pull}120494[#120494] +* `_score` should not be a reserved attribute in ES|QL {es-pull}118435[#118435] (issue: {es-issue}118460[#118460]) + +Engine:: +* Defer unpromotable shard refreshes until index refresh blocks are cleared {es-pull}120642[#120642] +* POC mark read-only {es-pull}119743[#119743] + +Experiences:: +* Integrate IBM watsonx to Inference API for re-ranking task {es-pull}117176[#117176] + +Geo:: +* Optimize indexing points with index and doc values set to true {es-pull}120271[#120271] + +Health:: +* Increase `replica_unassigned_buffer_time` default from 3s to 5s {es-pull}112834[#112834] + +ILM+SLM:: +* Add a `replicate_for` option to the ILM `searchable_snapshot` action {es-pull}119003[#119003] + +Indices APIs:: +* Add `remove_index_block` arg to `_create_from` api {es-pull}120548[#120548] +* Remove index blocks by default in `create_from` {es-pull}120643[#120643] +* Run `TransportGetComponentTemplateAction` on local node {es-pull}116868[#116868] +* Run `TransportGetComposableIndexTemplate` on local node {es-pull}119830[#119830] +* Run `TransportGetIndexTemplateAction` on local node {es-pull}119837[#119837] +* introduce new categories for deprecated resources in deprecation API {es-pull}120505[#120505] + +Inference:: +* Add version prefix to Inference Service API path {es-pull}117095[#117095] +* Remove Elastic Inference Service feature flag and deprecated setting {es-pull}120842[#120842] +* Update sparse text embeddings API route for Inference Service {es-pull}118025[#118025] +* [Elastic Inference Service] Add ElasticInferenceService Unified ChatCompletions Integration {es-pull}118871[#118871] + +Infra/CLI:: +* Strengthen encryption for elasticsearch-keystore tool to AES 256 {es-pull}119749[#119749] + +Infra/Circuit Breakers:: +* Add link to Circuit Breaker "Data too large" exception message {es-pull}113561[#113561] + +Infra/Core:: +* Add support for specifying reindexing script for system index migration {es-pull}119001[#119001] +* Change default Docker image to be based on UBI minimal instead of Ubuntu {es-pull}116739[#116739] + +Infra/Metrics:: +* Add `ensureGreen` test method for use with `adminClient` {es-pull}113425[#113425] + +Infra/REST API:: +* A new query parameter `?include_source_on_error` was added for create / index, update and bulk REST APIs to control +if to include the document source in the error response in case of parsing errors. The default value is `true`. {es-pull}120725[#120725] + +Infra/Scripting:: +* Add a `mustache.max_output_size_bytes` setting to limit the length of results from mustache scripts {es-pull}114002[#114002] + +Infra/Settings:: +* Introduce `IndexSettingDeprecatedInV8AndRemovedInV9` Setting property {es-pull}120334[#120334] +* Run `TransportClusterGetSettingsAction` on local node {es-pull}119831[#119831] + +Ingest Node:: +* Optimize `IngestCtxMap` construction {es-pull}120833[#120833] +* Optimize `IngestDocMetadata` `isAvailable` {es-pull}120753[#120753] +* Optimize `IngestDocument` `FieldPath` allocation {es-pull}120573[#120573] +* Optimize some per-document hot paths in the geoip processor {es-pull}120824[#120824] +* Returning ignored fields in the simulate ingest API {es-pull}117214[#117214] +* Run `GetPipelineTransportAction` on local node {es-pull}120445[#120445] +* Run `TransportGetEnrichPolicyAction` on local node {es-pull}121124[#121124] +* Run template simulation actions on local node {es-pull}120038[#120038] + +License:: +* Bump `TrialLicenseVersion` to allow starting new trial on 9.0 {es-pull}120198[#120198] + +Logs:: +* Add LogsDB option to route on sort fields {es-pull}116687[#116687] +* Add a new index setting to skip recovery source when synthetic source is enabled {es-pull}114618[#114618] +* Configure index sorting through index settings for logsdb {es-pull}118968[#118968] (issue: {es-issue}118686[#118686]) +* Optimize loading mappings when determining synthetic source usage and whether host.name can be sorted on. {es-pull}120055[#120055] + +Machine Learning:: +* Add DeBERTa-V2/V3 tokenizer {es-pull}111852[#111852] +* Add Inference Unified API for chat completions for OpenAI {es-pull}117589[#117589] +* Add Jina AI API to do inference for Embedding and Rerank models {es-pull}118652[#118652] +* Add enterprise license check for Inference API actions {es-pull}119893[#119893] +* Adding chunking settings to `IbmWatsonxService` {es-pull}114914[#114914] +* Adding default endpoint for Elastic Rerank {es-pull}117939[#117939] +* Adding endpoint creation validation for all task types to remaining services {es-pull}115020[#115020] +* Check for presence of error object when validating streaming responses from integrations in the inference API {es-pull}118375[#118375] +* Ignore failures from renormalizing buckets in read-only index {es-pull}118674[#118674] +* Inference duration and error metrics {es-pull}115876[#115876] +* Migrate stream to core error parsing {es-pull}120722[#120722] +* Remove all mentions of eis and gateway and deprecate flags that do {es-pull}116692[#116692] +* Remove deprecated sort from reindex operation within dataframe analytics procedure {es-pull}117606[#117606] +* Retry on `ClusterBlockException` on transform destination index {es-pull}118194[#118194] +* [Inference API] fix spell words: covertToString to convertToString {es-pull}119922[#119922] + +Mapping:: +* Add Optional Source Filtering to Source Loaders {es-pull}113827[#113827] + +Network:: +* Allow http unsafe buffers by default {es-pull}116115[#116115] +* Http stream activity tracker and exceptions handling {es-pull}119564[#119564] +* Remove HTTP content copies {es-pull}117303[#117303] +* `ConnectTransportException` returns retryable BAD_GATEWAY {es-pull}118681[#118681] (issue: {es-issue}118320[#118320]) + +Ranking:: +* Set default reranker for text similarity reranker to Elastic reranker {es-pull}120551[#120551] + +Recovery:: +* Allow archive and searchable snapshots indices in N-2 version {es-pull}118941[#118941] +* Trigger merges after recovery {es-pull}113102[#113102] + +Reindex:: +* Change Reindexing metrics unit from millis to seconds {es-pull}115721[#115721] + +Search:: +* Add match support for `semantic_text` fields {es-pull}117839[#117839] +* Add support for `sparse_vector` queries against `semantic_text` fields {es-pull}118617[#118617] +* Add support for knn vector queries on `semantic_text` fields {es-pull}119011[#119011] +* Adding linear retriever to support weighted sums of sub-retrievers {es-pull}120222[#120222] +* Address and remove any references of RestApiVersion version 7 {es-pull}117572[#117572] +* Feat: add a user-configurable timeout parameter to the `_resolve/cluster` API {es-pull}120542[#120542] +* Make semantic text part of the text family {es-pull}119792[#119792] +* Only aggregations require at least one shard request {es-pull}115314[#115314] +* Prevent data nodes from sending stack traces to coordinator when `error_trace=false` {es-pull}118266[#118266] +* Propagate status codes from shard failures appropriately {es-pull}118016[#118016] (issue: {es-issue}118482[#118482]) + +Security:: +* Add refresh `.security` index call between security migrations {es-pull}114879[#114879] + +Snapshot/Restore:: +* Add IMDSv2 support to `repository-s3` {es-pull}117748[#117748] (issue: {es-issue}105135[#105135]) +* Expose operation and request counts separately in repository stats {es-pull}117530[#117530] (issue: {es-issue}104443[#104443]) +* Retry `S3BlobContainer#getRegister` on all exceptions {es-pull}114813[#114813] +* Retry internally when CAS upload is throttled [GCS] {es-pull}120250[#120250] (issue: {es-issue}116546[#116546]) +* Track shard snapshot progress during node shutdown {es-pull}112567[#112567] + +Suggesters:: +* Extensible Completion Postings Formats {es-pull}111494[#111494] + +TSDB:: +* Increase field limit for OTel metrics to 10 000 {es-pull}120591[#120591] + +Transform:: +* Add support for `extended_stats` {es-pull}120340[#120340] +* Auto-migrate `max_page_search_size` {es-pull}119348[#119348] +* Create upgrade mode {es-pull}117858[#117858] +* Wait while index is blocked {es-pull}119542[#119542] +* [Deprecation] Add `transform_ids` to outdated index {es-pull}120821[#120821] + +Vector Search:: +* Even better(er) binary quantization {es-pull}117994[#117994] +* Speed up bit compared with floats or bytes script operations {es-pull}117199[#117199] + +[[feature-9.0.0]] +[float] +== New features + +CRUD:: +* Metrics for indexing failures due to version conflicts {es-pull}119067[#119067] + +ES|QL:: +* ESQL - Add Match function options {es-pull}120360[#120360] +* ESQL - Allow full text functions disjunctions for non-full text functions {es-pull}120291[#120291] +* ESQL: Enable async get to support formatting {es-pull}111104[#111104] (issue: {es-issue}110926[#110926]) +* Expand type compatibility for match function and operator {es-pull}117555[#117555] +* ST_EXTENT aggregation {es-pull}117451[#117451] (issue: {es-issue}104659[#104659]) +* Support ST_ENVELOPE and related (ST_XMIN, ST_XMAX, ST_YMIN, ST_YMAX) functions {es-pull}116964[#116964] (issue: {es-issue}104875[#104875]) + +Extract&Transform:: +* [Connector API] Support hard deletes with new URL param in delete endpoint {es-pull}120200[#120200] +* [Connector API] Support soft-deletes of connectors {es-pull}118669[#118669] +* [Connector APIs] Enforce index prefix for managed connectors {es-pull}117778[#117778] + +Highlighting:: +* Add Highlighter for Semantic Text Fields {es-pull}118064[#118064] + +Infra/Core:: +* Infrastructure for assuming cluster features in the next major version {es-pull}118143[#118143] + +Machine Learning:: +* ES|QL categorize with multiple groupings {es-pull}118173[#118173] +* Support mTLS for the Elastic Inference Service integration inside the inference API {es-pull}119679[#119679] +* [Inference API] Add node-local rate limiting for the inference API {es-pull}120400[#120400] + +Mapping:: +* Add option to store `sparse_vector` outside `_source` {es-pull}117917[#117917] + +Ranking:: +* Add a generic `rescorer` retriever based on the search request's rescore functionality {es-pull}118585[#118585] (issue: {es-issue}118327[#118327]) + +Relevance:: +* Add Multi-Field Support for Semantic Text Fields {es-pull}120128[#120128] + +Vector Search:: +* Add new experimental `rank_vectors` mapping for late-interaction second order ranking {es-pull}118804[#118804] +* KNN vector rescoring for quantized vectors {es-pull}116663[#116663] +* Mark bbq indices as GA and add rolling upgrade integration tests {es-pull}121105[#121105] + +[[upgrade-9.0.0]] +[float] +== Upgrades + +Infra/Core:: +* Bump major version for feature migration system indices {es-pull}117243[#117243] +* Update ASM 9.7 -> 9.7.1 to support JDK 24 {es-pull}118094[#118094] + +Machine Learning:: +* Automatically rollover legacy .ml-anomalies indices {es-pull}120913[#120913] +* Automatically rollover legacy ml indices {es-pull}120405[#120405] +* Change the auditor to write via an alias {es-pull}120064[#120064] + +Search:: +* Upgrade to Lucene 10 {es-pull}114741[#114741] +* Upgrade to Lucene 10.1.0 {es-pull}119308[#119308] + diff --git a/docs/en/install-upgrade/release-notes/release-notes-kibana.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-kibana.asciidoc index 837f5e746..c0a31f2ad 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-kibana.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-kibana.asciidoc @@ -1,4 +1,4 @@ -[[release-notes-kibana-9.0.0-beta1]] +[[release-notes-kibana-9.0.0]] = {kib} version 9.0.0-beta1 ++++ {kib} @@ -195,7 +195,7 @@ Logs Explorer has been removed. Instead, you can use Discover, that was improved **Elastic Security solution** -For the Elastic Security 9.0.0 release information, refer to <>. +For the Elastic Security 9.0.0 release information, refer to <>. **Data ingestion and Fleet** @@ -368,7 +368,7 @@ The `visualization:colorMapping` advanced setting for TSVB and Visualize charts Data ingestion and Fleet:: * New setting allowing automatic deletion of unenrolled agents in Fleet settings ({kibana-pull}195544[#195544]). Elastic Security solution:: -For the Elastic Security 9.0.0 release information, refer to <>. +For the Elastic Security 9.0.0 release information, refer to <>. Kibana security:: * New interface for Automatic Import CEL generation flow ({kibana-pull}206491[#206491]). @@ -390,7 +390,7 @@ Data ingestion and Fleet:: * Updates max supported package version ({kibana-pull}196675[#196675]). //* Remove old bundled.yaml from oas, fixed tags ({kibana-pull}194788[#194788]). Elastic Security solution:: -For the Elastic Security 9.0.0 release information, refer to <>. +For the Elastic Security 9.0.0 release information, refer to <>. Kibana security:: * Updates `js-yaml` to `4.1.0` ({kibana-pull}190678[#190678]). Machine Learning:: @@ -414,6 +414,6 @@ Elastic Observability solution:: * Fixes chat on the Alerts page ({kibana-pull}197126[#197126]). * Fixes an error that could prevent the Observability Infrastructure Inventory view from loading after an upgrade due to missing versioning on inventory_view_saved_object ({kibana-pull}207007[#207007]). Elastic Security solution:: -For the Elastic Security 9.0.0 release information, refer to <>. +For the Elastic Security 9.0.0 release information, refer to <>. Platform:: * Fixes several interface inconsistencies on the Space creation and settings pages ({kibana-pull}197303[#197303]). \ No newline at end of file diff --git a/docs/en/install-upgrade/release-notes/release-notes-logstash.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-logstash.asciidoc index 7e99bf088..14ce0eaae 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-logstash.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-logstash.asciidoc @@ -1,4 +1,4 @@ -[[release-notes-logstash-9.0.0-beta1]] +[[release-notes-logstash-9.0.0]] = {ls} version 9.0.0-beta1 ++++ {ls} diff --git a/docs/en/install-upgrade/release-notes/release-notes-security.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-security.asciidoc index cc42b8be6..e4c913270 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-security.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-security.asciidoc @@ -1,4 +1,4 @@ -[[release-notes-security-9.0.0-beta1]] +[[release-notes-security-9.0.0]] = {elastic-sec} version 9.0.0-beta1 ++++ {elastic-sec} diff --git a/docs/en/install-upgrade/release-notes/release-notes.asciidoc b/docs/en/install-upgrade/release-notes/release-notes.asciidoc index e015e2b52..9b9a1bc66 100644 --- a/docs/en/install-upgrade/release-notes/release-notes.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes.asciidoc @@ -3,8 +3,8 @@ This section summarizes the changes in the {stack} releases. -* <> -* <> -* <> -* <> +* <> +* <> +* <> +* <> From b71b08d00b9ffbeb9e3838bdb7ce83a8ab7b5a77 Mon Sep 17 00:00:00 2001 From: lcawl Date: Tue, 18 Feb 2025 12:17:04 -0800 Subject: [PATCH 2/4] Fix hard-coded links --- .../release-notes/release-notes-elasticsearch.asciidoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc index 1af6bcf80..5b4ad4cf1 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc @@ -59,10 +59,10 @@ Replace usages of "german2" with "german" in analysis configuration. Old indices [%collapsible] ==== *Details* + -Lucene 10 has added a final stemming step to its PersianAnalyzer that Elasticsearch exposes as 'persian' analyzer. Existing indices will keep the old non-stemming behaviour while new indices will see the updated behaviour with added stemming. Users that wish to maintain the non-stemming behaviour need to define their own analyzer as outlined in https://www.elastic.co/guide/en/elasticsearch/reference/8.15/analysis-lang-analyzer.html#persian-analyzer. Users that wish to use the new stemming behaviour for existing indices will have to reindex their data. +Lucene 10 has added a final stemming step to its PersianAnalyzer that Elasticsearch exposes as 'persian' analyzer. Existing indices will keep the old non-stemming behaviour while new indices will see the updated behaviour with added stemming. Users that wish to maintain the non-stemming behaviour need to define their own analyzer as outlined in {ref}analysis-lang-analyzer.html#persian-analyzer[Persion analyzer]. Users that wish to use the new stemming behaviour for existing indices will have to reindex their data. *Impact* + -Indexing with the 'persian' analyzer will produce slightly different tokens. Users should check if this impacts their search results. If they wish to maintain the legacy non-stemming behaviour they can define their own analyzer equivalent as explained in https://www.elastic.co/guide/en/elasticsearch/reference/8.15/analysis-lang-analyzer.html#persian-analyzer. +Indexing with the 'persian' analyzer will produce slightly different tokens. Users should check if this impacts their search results. If they wish to maintain the legacy non-stemming behaviour they can define their own analyzer equivalent as explained in {ref}/analysis-lang-analyzer.html#persian-analyzer[Persian analyzer]. ==== [[korean_dictionary_for_nori_has_been_updated]] @@ -254,8 +254,7 @@ then Logsdb will not be enabled by default. *Impact* + Logsdb reduce storage footprint in Elasticsearch for logs, but there are side effects -to be taken into account that are described in the Logsdb docs: -https://www.elastic.co/guide/en/elasticsearch/reference/current/logs-data-stream.html#upgrade-to-logsdb-notes +to be taken into account that are described in the {ref}/logs-data-stream.html[Logsdb docs]. ==== [discrete] From 9b7715d3939dc1e368ed10fae346a31f48f8aae9 Mon Sep 17 00:00:00 2001 From: lcawl Date: Tue, 18 Feb 2025 13:06:54 -0800 Subject: [PATCH 3/4] Fix broken link --- .../release-notes/release-notes-elasticsearch.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc index 5b4ad4cf1..ab62f8b27 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-elasticsearch.asciidoc @@ -59,7 +59,7 @@ Replace usages of "german2" with "german" in analysis configuration. Old indices [%collapsible] ==== *Details* + -Lucene 10 has added a final stemming step to its PersianAnalyzer that Elasticsearch exposes as 'persian' analyzer. Existing indices will keep the old non-stemming behaviour while new indices will see the updated behaviour with added stemming. Users that wish to maintain the non-stemming behaviour need to define their own analyzer as outlined in {ref}analysis-lang-analyzer.html#persian-analyzer[Persion analyzer]. Users that wish to use the new stemming behaviour for existing indices will have to reindex their data. +Lucene 10 has added a final stemming step to its PersianAnalyzer that Elasticsearch exposes as 'persian' analyzer. Existing indices will keep the old non-stemming behaviour while new indices will see the updated behaviour with added stemming. Users that wish to maintain the non-stemming behaviour need to define their own analyzer as outlined in {ref}/analysis-lang-analyzer.html#persian-analyzer[Persion analyzer]. Users that wish to use the new stemming behaviour for existing indices will have to reindex their data. *Impact* + Indexing with the 'persian' analyzer will produce slightly different tokens. Users should check if this impacts their search results. If they wish to maintain the legacy non-stemming behaviour they can define their own analyzer equivalent as explained in {ref}/analysis-lang-analyzer.html#persian-analyzer[Persian analyzer]. From 5a49e66415716a2b6a53a3b52f32f54e19719dac Mon Sep 17 00:00:00 2001 From: lcawl Date: Tue, 18 Feb 2025 13:32:04 -0800 Subject: [PATCH 4/4] Fix release note links in Fleet --- .../release-notes/release-notes-fleet-agent.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/install-upgrade/release-notes/release-notes-fleet-agent.asciidoc b/docs/en/install-upgrade/release-notes/release-notes-fleet-agent.asciidoc index 0234a9c49..f1ab6f0e2 100644 --- a/docs/en/install-upgrade/release-notes/release-notes-fleet-agent.asciidoc +++ b/docs/en/install-upgrade/release-notes/release-notes-fleet-agent.asciidoc @@ -21,8 +21,8 @@ This section summarizes the changes in each release. Also see: -* {kibana-ref}/release-notes.html[{kib} release notes] -* {beats-ref}/release-notes.html[{beats} release notes] +* <> +* <> // begin 9.0.0-beta1 relnotes