Unable to use API key authentication to connect to Elasticsearch

[smriti2901]

Currently we use Basic Authentication to create document logs in elastic which works fine with NLog.Targets.ElasticSearch v7.7.0. On switching to API Key authentication I keep getting Unauthorized error on authentication. Our ElasticSearch is on version 8.7.0.

Here is the exception stack trace
Exception: Elasticsearch.Net.ElasticsearchClientException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401 from: POST /_bulk ---> Elasticsearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized. at System.Net.HttpWebRequest.GetResponse() at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) --- End of inner exception stack trace --- at Elasticsearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IElasticsearchResponse response) at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)`

The API Key is working with curl request to fetch the document based on elastic uri and the index. I am passing the same API Key in the Authorization header. So permission shouldn't be the problem. Here is the API Key information and you can see privileges are set correctly.

{ "api_keys": [ { "username": "*****", "realm": "cloud-saml", "name": "*****", "creation": 1693470140150, "invalidated": false, "role_descriptors": { "appuser": { "applications": [], "transient_metadata": { "enabled": true }, "run_as": [], "cluster": [], "indices": [ { "privileges": [ "write", "create_index", "create_doc", "create", "all" ], "allow_restricted_indices": false, "names": [ "logs-<index1>-default-*", "logs-<index2>-*" ] } ], "metadata": {} } }, "id": "APIKEYID", "metadata": {} } ] }

Here is my NLog.Config

<?xml version="1.0" encoding="utf-8"?>

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	  xsi:schemaLocation="http://www.nlog-project.org/schemas/NLog.xsd NLog.xsd"
      autoReload="true"
      throwExceptions="false"
      internalLogLevel="Error" internalLogFile="${basedir}/logs/error.log">

	<variable name="defaultLayout" value="${longdate} | ${level:uppercase=true} | ${logger} | ${message}${onexception:${newline}${exception:format=tostring}}"/>
	<extensions>
		<add assembly="NLog.Web" />
		<add assembly="NLog.Targets.ElasticSearch" />
		<add assembly="Elastic.Apm.NLog"/>
		<add assembly="Elastic.CommonSchema.NLog"/>
	</extensions>
	<targets>
		<target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">
			<target xsi:type="ElasticSearch"
					name="elastic"
					enableJsonLayout="true"
					opCodeCreate="true"
					includeDefaultFields="true"
					includeAllProperties="true"
					documentType="">
				<layout xsi:type="EcsLayout">
					<metadata name="data_stream.dataset" layout="${appsetting:Log.DataStream.Dataset}" />
					<metadata name="application.title" layout="${appsetting:Log.Application.Title}" />
				</layout>
			</target>
		</target>
	</targets>

	<rules>
		<logger name="*" minlevel="Trace" writeTo="elastic" />
	</rules>
</nlog>

And my helper class to initialize the ElasticSearchTarget

public static class ElasticSearchTargetSetup
    {
        public static bool SetupElasticTarget()
        {
            bool isElasticActive = false;

            var elasticTargetWrapper = LogManager.LogFactory.Configuration.AllTargets.SingleOrDefault(x => x is BufferingTargetWrapper) as BufferingTargetWrapper;

            if (elasticTargetWrapper != null && elasticTargetWrapper.WrappedTarget is ElasticSearchTarget)
            {
                var elasticTarget = elasticTargetWrapper.WrappedTarget as ElasticSearchTarget;
                elasticTarget.Index = ConfigurationManager.AppSettings["Log.Elastic.Index"];
                elasticTarget.CloudId = ConfigurationManager.AppSettings["Log.Elastic.CloudId"];
                elasticTarget.Uri = ConfigurationManager.AppSettings["Log.Elastic.Url"];
                elasticTarget.ApiKeyId = ConfigurationManager.AppSettings["Log.Elastic.ApiKeyId"];
                elasticTarget.ApiKey = ConfigurationManager.AppSettings["Log.Elastic.ApiKey"];
            }            

            return isElasticActive;
        }
    }

Here is my web.config to read appsettings

<appSettings>
	<add key="Log.Application.Title" value="My App 1" />
	<add key="Log.DataStream.Dataset" value="my data stream" />
	<add key="Log.Elastic.CloudId" value="my-cluster:<Base64string>" />
	<add key="Log.Elastic.Index" value="logs-<index1>-default" />
	<add key="Log.Elastic.Url" value="https://my-elastic-cloud:9243" />
	<add key="Log.Elastic.ApiKeyId" value="APIKEYID"/>
	<add key="Log.Elastic.ApiKey" value="APIKEY"/>
</appSettings>

Has anyone faced this issue or were able to use the API Key authentication successfully with NLog.Targets.ElasticSearch 7.7.0?

[snakefoot]

Notice that ElasticSearch have now released their own official NLog Target for ElasticSearch:

• https://www.nuget.org/packages/Elastic.NLog.Targets/#readme-body-tab
• https://github.com/elastic/ecs-dotnet/tree/main/src/Elastic.NLog.Targets