add ECS support (#138)

This PR maps `header.*` to structured ECS fields `http`, `url`, `user_agent` and `host`

Author: kaisecheng

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 3.4.0
+ - Add ECS support, mapping Http header to ECS compatible fields [#137](https://github.com/logstash-plugins/logstash-input-http/pull/137)
+
 ## 3.3.7
  - Feat: improved error handling/logging/unwraping [#133](https://github.com/logstash-plugins/logstash-input-http/pull/133)
 

VERSION

@@ -1 +1 @@
-3.3.7
+3.4.0

docs/index.asciidoc

@@ -32,6 +32,28 @@ This input can also be used to receive webhook requests to integrate with other
 and applications. By taking advantage of the vast plugin ecosystem available in Logstash
 you can trigger actionable events right from your application.
 
+[id="plugins-{type}s-{plugin}-ecs_metadata"]
+==== Event Metadata and the Elastic Common Schema (ECS)
+In addition to decoding the events, this input will add HTTP headers containing connection information to each event.
+When ECS compatibility is disabled, the headers are stored in the `headers` field, which has the potential to create confusion and schema conflicts downstream.
+When ECS is enabled, we can ensure a pipeline maintains access to this metadata throughout the event's lifecycle without polluting the top-level namespace.
+
+Here’s how ECS compatibility mode affects output.
+[cols="<l,<l,e,<e"]
+|=======================================================================
+| ECS disabled | ECS v1 | Availability | Description
+
+| [host] | [host][ip] | Always | Host IP address
+| [headers] | [@metadata][input][http][request][headers] | Always | Complete HTTP headers
+| [headers][http_version] | [http][version] | Always | HTTP version
+| [headers][http_user_agent] | [user_agent][original] | Always | client user agent
+| [headers][http_host] | [url][domain] and [url][port] | Always | host domain and port
+| [headers][request_method] | [http][method] | Always | HTTP method
+| [headers][request_path] | [url][path] | Always | Query path
+| [headers][content_length] | [http][request][body][bytes] | Always | Request content length
+| [headers][content_type] | [http][request][mime_type] | Always | Request mime type
+|=======================================================================
+
 ==== Blocking Behavior
 
 The HTTP protocol doesn't deal well with long running requests. This plugin will either return 
@@ -70,6 +92,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 |Setting |Input type|Required
 | <<plugins-{type}s-{plugin}-additional_codecs>> |<<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-cipher_suites>> |<<array,array>>|No
+| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
 | <<plugins-{type}s-{plugin}-host>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-keystore>> |<<path,path>>|No
 | <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|No
@@ -115,6 +138,72 @@ and no codec for the request's content-type is found
 
 The list of ciphers suite to use, listed by priorities.
 
+[id="plugins-{type}s-{plugin}-ecs_compatibility"]
+===== `ecs_compatibility`
+
+* Value type is <<string,string>>
+* Supported values are:
+** `disabled`: unstructured connection metadata added at root level
+** `v1`: headers added under `[@metadata][http][header]`. Some are copied to structured ECS fields `http`, `url`, `user_agent` and `host`
+
+Controls this plugin's compatibility with the
+{ecs-ref}[Elastic Common Schema (ECS)].
+See <<plugins-{type}s-{plugin}-ecs_metadata>> for detailed information.
+
+Example output:
+
+**Sample output: ECS disabled**
+[source,text]
+-----
+{
+    "@version" => "1",
+    "headers" => {
+           "request_path" => "/twitter/tweet/1",
+            "http_accept" => "*/*",
+           "http_version" => "HTTP/1.1",
+         "request_method" => "PUT",
+              "http_host" => "localhost:8080",
+        "http_user_agent" => "curl/7.64.1",
+         "content_length" => "5",
+           "content_type" => "application/x-www-form-urlencoded"
+    },
+    "@timestamp" => 2021-05-28T19:27:28.609Z,
+    "host" => "127.0.0.1",
+    "message" => "hello"
+}
+-----
+
+**Sample output: ECS enabled**
+[source,text]
+-----
+{
+    "@version" => "1",
+    "user_agent" => {
+        "original" => "curl/7.64.1"
+    },
+    "http" => {
+        "method" => "PUT",
+        "request" => {
+            "mime_type" => "application/x-www-form-urlencoded",
+            "body" => {
+                "bytes" => "5"
+            }
+        },
+        "version" => "HTTP/1.1"
+    },
+    "url" => {
+          "port" => "8080",
+        "domain" => "snmp1",
+          "path" => "/twitter/tweet/1"
+    },
+    "@timestamp" => 2021-05-28T23:32:38.222Z,
+    "host" => {
+        "ip" => "127.0.0.1"
+    },
+    "message" => "hello",
+}
+-----
+
 [id="plugins-{type}s-{plugin}-host"]
 ===== `host` 
 
@@ -209,15 +298,17 @@ specify a custom set of response headers
 ===== `remote_host_target_field` 
 
   * Value type is <<string,string>>
-  * Default value is `"host"`
+  * Default value is `"host"` when ECS is disabled
+  * Default value is `[host][ip]` when ECS is enabled
 
 specify a target field for the client host of the http request
 
 [id="plugins-{type}s-{plugin}-request_headers_target_field"]
 ===== `request_headers_target_field` 
 
   * Value type is <<string,string>>
-  * Default value is `"headers"`
+  * Default value is `"headers"` when ECS is disabled
+  * Default value is `[@metadata][http][header]` when ECS is enabled
 
 specify target field for the client host of the http request
 

lib/logstash/inputs/http.rb

@@ -3,6 +3,7 @@
 require "logstash/namespace"
 require "stud/interval"
 require "logstash-input-http_jars"
+require "logstash/plugin_mixins/ecs_compatibility_support"
 
 # Using this input you can receive single or multiline events over http(s).
 # Applications can send a HTTP POST request with a body to the endpoint started by this
@@ -25,6 +26,7 @@
 # format]
 #
 class LogStash::Inputs::Http < LogStash::Inputs::Base
+  include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled, :v1, :v8 => :v1)
   require "logstash/inputs/http/tls"
 
   java_import "io.netty.handler.codec.http.HttpUtil"
@@ -104,10 +106,10 @@ class LogStash::Inputs::Http < LogStash::Inputs::Base
   config :response_headers, :validate => :hash, :default => { 'Content-Type' => 'text/plain' }
 
   # target field for the client host of the http request
-  config :remote_host_target_field, :validate => :string, :default => "host"
+  config :remote_host_target_field, :validate => :string
 
   # target field for the client host of the http request
-  config :request_headers_target_field, :validate => :string, :default => "headers"
+  config :request_headers_target_field, :validate => :string
 
   config :threads, :validate => :number, :required => false, :default => ::LogStash::Config::CpuCoreStrategy.maximum
 
@@ -130,7 +132,7 @@ def register
 
     validate_ssl_settings!
 
-    if @user && @password then
+    if @user && @password
       token = Base64.strict_encode64("#{@user}:#{@password.value}")
       @auth_token = "Basic #{token}"
     end
@@ -144,6 +146,9 @@ def register
     require "logstash/inputs/http/message_handler"
     message_handler = MessageHandler.new(self, @codec, @codecs, @auth_token)
     @http_server = create_http_server(message_handler)
+
+    @remote_host_target_field ||= ecs_select[disabled: "host", v1: "[host][ip]"]
+    @request_headers_target_field ||= ecs_select[disabled: "headers", v1: "[@metadata][input][http][request][headers]"]
   end # def register
 
   def run(queue)
@@ -177,12 +182,50 @@ def decode_body(headers, remote_address, body, default_codec, additional_codecs)
   end
 
   def push_decoded_event(headers, remote_address, event)
+    add_ecs_fields(headers, event)
     event.set(@request_headers_target_field, headers)
     event.set(@remote_host_target_field, remote_address)
     decorate(event)
     @queue << event
   end
 
+  def add_ecs_fields(headers, event)
+    return if ecs_compatibility == :disabled
+
+    http_version = headers.get("http_version")
+    event.set("[http][version]", http_version) if http_version
+
+    http_user_agent = headers.get("http_user_agent")
+    event.set("[user_agent][original]", http_user_agent) if http_user_agent
+
+    http_host = headers.get("http_host")
+    domain, port = self.class.get_domain_port(http_host)
+    event.set("[url][domain]", domain) if domain
+    event.set("[url][port]", port) if port
+
+    request_method = headers.get("request_method")
+    event.set("[http][method]", request_method) if request_method
+
+    request_path = headers.get("request_path")
+    event.set("[url][path]", request_path) if request_path
+
+    content_length = headers.get("content_length")
+    event.set("[http][request][body][bytes]", content_length) if content_length
+
+    content_type = headers.get("content_type")
+    event.set("[http][request][mime_type]", content_type) if content_type
+  end
+
+  # match the domain and port in either IPV4, "127.0.0.1:8080", or IPV6, "[2001:db8::8a2e:370:7334]:8080", style
+  # return [domain, port]
+  def self.get_domain_port(http_host)
+    if /^(([^:]+)|\[(.*)\])\:([\d]+)$/ =~ http_host
+      ["#{$2 || $3}", $4.to_i]
+    else
+      [http_host, nil]
+    end
+  end
+
   def validate_ssl_settings!
     if !@ssl
       @logger.warn("SSL Certificate will not be used") if @ssl_certificate

logstash-input-http.gemspec

@@ -23,6 +23,7 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
   s.add_runtime_dependency 'logstash-codec-plain'
   s.add_runtime_dependency 'jar-dependencies', '~> 0.3', '>= 0.3.4'
+  s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.2'
 
   s.add_development_dependency 'logstash-devutils'
   s.add_development_dependency 'logstash-codec-json'