From 0ac1eae15c31ea9b43d0ff2071a6fa71ef4c4277 Mon Sep 17 00:00:00 2001 From: chemamartinez Date: Wed, 10 Dec 2025 08:52:13 +0100 Subject: [PATCH 1/3] Add daily refresh option to TI Misp --- packages/ti_misp/_dev/build/docs/README.md | 28 ++++++++- packages/ti_misp/changelog.yml | 8 +++ .../test/system/test-daily-refresh-config.yml | 15 +++++ .../agent/stream/httpjson.yml.hbs | 59 +++++++++++++++++++ .../elasticsearch/ingest_pipeline/default.yml | 21 ++++++- .../threat_attributes/manifest.yml | 11 +++- .../threat_attributes/sample_event.json | 23 +++++--- packages/ti_misp/docs/README.md | 28 ++++++++- packages/ti_misp/manifest.yml | 2 +- 9 files changed, 181 insertions(+), 14 deletions(-) create mode 100644 packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml diff --git a/packages/ti_misp/_dev/build/docs/README.md b/packages/ti_misp/_dev/build/docs/README.md index a49dc8c9f7a..97ff058ecb6 100644 --- a/packages/ti_misp/_dev/build/docs/README.md +++ b/packages/ti_misp/_dev/build/docs/README.md @@ -23,8 +23,34 @@ This data stream uses the `/attributes/restSearch` API endpoint which returns mo #### Expiration of Indicators of Compromise (IOCs) The ingested IOCs expire after certain duration which is indicated by the `decayed` field. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named `logs-ti_misp_latest.dest_threat_attributes-*` which only contains active and unexpired IOCs. The latest destination index also has an alias named `logs-ti_misp_latest.threat_attributes`. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for `Threat Attributes` datastream are also pointing to the latest destination indices containing active IoCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source datastream `.ds-logs-ti_misp.threat_attributes-*` indices. +#### Daily Refresh Mode +By default, the integration uses incremental updates, only fetching attributes that have been modified since the last poll (tracked via an internal cursor). However, MISP's decay scores are dynamic and decrease over time, which means an attribute's decay status may change without the attribute itself being modified. In such cases, incremental updates would not capture the updated decay state. + +To address this, users can enable the `Enable Daily Refresh` toggle. When enabled, the integration will: +1. **Perform a daily full refresh**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP. +2. **Set 24-hour expiration**: Attributes ingested during a daily refresh will have their `decayed_at` set to 24 hours after ingestion, ensuring they expire before the next refresh cycle. +3. **Update decay states**: The next daily refresh will re-ingest attributes with their current decay scores from MISP, removing any that have since been marked as decayed. + +This approach ensures that: +- The destination indices stay aligned with MISP's current view of valid indicators +- Attributes that become decayed in MISP are automatically removed in the next refresh cycle +- No stale indicators remain in the destination indices beyond 24 hours + +**Note**: Daily refreshes will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refresh period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion are not affected by the 24-hour expiration and will be removed immediately. + +#### IOC Expiration Duration +The `IOC Expiration Duration` parameter controls when ingested IOCs are marked as expired when **Daily Refresh is disabled**. This setting applies to all ingested attributes that are not decayed, not just orphaned IOCs. The expiration date for each attribute is calculated as `max(last_seen, timestamp) + IOC Expiration Duration`, which defaults to 90 days. + +**Note**: When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. This ensures attributes are refreshed with current decay scores from MISP in the next daily cycle. + +When Daily Refresh is disabled, this setting serves as a fail-safe expiration mechanism that works independently of MISP's decay models. Even if MISP does not mark an attribute as decayed, Elastic will expire the attribute after the configured duration. + #### Handling Orphaned IOCs -Some IOCs may never get decayed/expired and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. To avoid any false positives from such orphaned IOCs, users are allowed to configure `IOC Expiration Duration` parameter while setting up the integration. This parameter deletes all data inside the destination indices `logs-ti_misp_latest.dest_threat_attributes-*` after this specified duration is reached, defaults to `90d` after attribute's `max(last_seen, timestamp)`. Note that `IOC Expiration Duration` parameter only exists to add a fail-safe default expiration in case IOCs never expire. +Some IOCs may never get decayed/expired by MISP's decay models and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. + +When `Enable Daily Refresh` is **disabled**, the `IOC Expiration Duration` parameter ensures these orphaned IOCs are eventually removed from destination indices after the specified duration from the attribute's `max(last_seen, timestamp)`. + +When `Enable Daily Refresh` is **enabled**, orphaned IOCs are handled automatically by the 24-hour expiration cycle. Each daily refresh re-ingests all attributes with their current decay state from MISP, ensuring the destination indices remain aligned with MISP's view of valid indicators. #### ILM Policy To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_misp.threat_attributes-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date. diff --git a/packages/ti_misp/changelog.yml b/packages/ti_misp/changelog.yml index 8a01f26cd2f..58c47ffc496 100644 --- a/packages/ti_misp/changelog.yml +++ b/packages/ti_misp/changelog.yml @@ -1,4 +1,12 @@ # newer versions go on top +- version: "1.40.0" + changes: + - description: Clarify documentation about the data retention policy. + type: enhancement + link: https://github.com/elastic/integrations/pull/99999 + - description: Add option to refresh ingested indicators daily. + type: enhancement + link: https://github.com/elastic/integrations/pull/99999 - version: "1.39.0" changes: - description: Prevent updating fleet health status to degraded when pagination completes. diff --git a/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml b/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml new file mode 100644 index 00000000000..f3c55ba7ccf --- /dev/null +++ b/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml @@ -0,0 +1,15 @@ +input: httpjson +service: misp +vars: ~ +data_stream: + vars: + preserve_original_event: true + url: http://{{Hostname}}:{{Port}} + api_token: test + interval: 1s + initial_interval: 10s + enable_request_tracer: true + daily_refresh: true + ioc_expiration_duration: 5d +assert: + hit_count: 10 diff --git a/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs b/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs index 54b362f7f93..3fc1f0cf6ff 100644 --- a/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs +++ b/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs @@ -46,6 +46,39 @@ request.transforms: - set: target: body.returnFormat value: json +{{#if daily_refresh}} +# Daily refresh mode: every 24 hours, reset cursor to re-fetch all attributes from initial_interval. +# When a daily refresh occurs, the timestamp is reset to initial_interval and last_daily_refresh is updated. +- set: + target: body.timestamp + value: |- + [[- $dailyRefreshDuration := (parseDuration "24h") -]] + [[- $initialInterval := (parseDuration "-{{initial_interval}}") -]] + [[- $initialTimestamp := (now $initialInterval).Unix -]] + [[- $isDailyRefresh := true -]] + [[- if index .cursor "last_daily_refresh" -]] + [[- $lastRefresh := (parseDate .cursor.last_daily_refresh "RFC3339") -]] + [[- $nextRefresh := 0 -]] + [[- with $lastRefresh -]] + [[- $nextRefresh = .Add $dailyRefreshDuration -]] + [[- end -]] + [[- with $nextRefresh -]] + [[- if .Before now -]] + [[- $isDailyRefresh = true -]] + [[- else -]] + [[- $isDailyRefresh = false -]] + [[- end -]] + [[- end -]] + [[- end -]] + [[- if $isDailyRefresh -]] + [[- $initialTimestamp -]] + [[- else if index .cursor "timestamp" -]] + [[- .cursor.timestamp -]] + [[- else -]] + [[- $initialTimestamp -]] + [[- end -]] + default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]' +{{else}} - set: target: body.timestamp value: >- @@ -55,6 +88,7 @@ request.transforms: [[- .last_response.url.params.Get "timestamp" -]] [[- end -]] default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]' +{{/if}} - set: # Ignored by MISP, set as a workaround to make it available in response.pagination. target: url.params.timestamp @@ -87,10 +121,35 @@ response.pagination: cursor: timestamp: value: '[[.last_event.timestamp]]' +{{#if daily_refresh}} + last_daily_refresh: + # Track when the last daily refresh started. Updated when 24h has passed since previous refresh. + value: |- + [[- $dailyRefreshDuration := (parseDuration "24h") -]] + [[- if index .cursor "last_daily_refresh" -]] + [[- $lastRefresh := (parseDate .cursor.last_daily_refresh "RFC3339") -]] + [[- $nextRefresh := 0 -]] + [[- with $lastRefresh -]] + [[- $nextRefresh = .Add $dailyRefreshDuration -]] + [[- end -]] + [[- with $nextRefresh -]] + [[- if .Before now -]] + [[- formatDate now "RFC3339" -]] + [[- else -]] + [[- $.cursor.last_daily_refresh -]] + [[- end -]] + [[- end -]] + [[- else -]] + [[- formatDate now "RFC3339" -]] + [[- end -]] +{{/if}} tags: {{#if preserve_original_event}} - preserve_original_event {{/if}} +{{#if daily_refresh}} + - daily_refresh +{{/if}} {{#each tags as |tag i|}} - {{tag}} {{/each}} diff --git a/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml b/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml index ce76e268e56..9491565783b 100644 --- a/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml +++ b/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml @@ -148,10 +148,11 @@ processors: # Add default decayed_at to expire all IOCs after duration `ioc_expiration_duration` from `_tmp_max_time`. # If user-provided value of `ioc_expiration_duration` is not in d, h, or m, default to 90d. # Set `decayed` as true if decayed_at is before ingest time i.e., already expired during ingestion. +# This script is skipped when daily_refresh tag is present. - script: lang: painless tag: script-default-decayed_at - if: (ctx.misp?.attribute?.decayed_at == null && ctx._conf?.ioc_expiration_duration != null && ctx._conf.ioc_expiration_duration != '') + if: (ctx.misp?.attribute?.decayed_at == null && ctx._conf?.ioc_expiration_duration != null && ctx._conf.ioc_expiration_duration != '' && !(ctx.tags instanceof List && ctx.tags.contains('daily_refresh'))) description: > Add default decayed_at source: > @@ -195,6 +196,24 @@ processors: - append: field: error.message value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' +# Script for daily refresh mode: set decayed_at to 24 hours after ingestion. +# When daily_refresh tag is present, all non-decayed attributes will expire 24 hours after ingestion, +# allowing the next daily refresh cycle to re-ingest them with updated decay scores from MISP. +# Attributes already marked as decayed by MISP (script-misp_decayed) are not affected. + - script: + lang: painless + tag: script-daily_refresh-decayed_at + if: (ctx.misp?.attribute?.decayed != true && ctx.misp?.attribute?.decayed_at == null && ctx.tags instanceof List && ctx.tags.contains('daily_refresh')) + description: > + Set decayed_at to 24 hours after ingestion for daily refresh mode + source: | + ZonedDateTime _tmp_event_ingested = ZonedDateTime.parse(ctx._tmp.event_ingested); + ctx.misp.attribute.decayed_at = _tmp_event_ingested.plusHours(24L); + ctx.tags.remove(ctx.tags.indexOf('daily_refresh')); + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' - date: if: ctx.misp?.event?.timestamp != null field: misp.event.timestamp diff --git a/packages/ti_misp/data_stream/threat_attributes/manifest.yml b/packages/ti_misp/data_stream/threat_attributes/manifest.yml index dace298bdd1..e7d0c433fe3 100644 --- a/packages/ti_misp/data_stream/threat_attributes/manifest.yml +++ b/packages/ti_misp/data_stream/threat_attributes/manifest.yml @@ -35,6 +35,15 @@ streams: show_user: true default: 120h description: How far back to look for indicators the first time the agent is started. Supported units for this parameter are h/m/s. + - name: daily_refresh + type: bool + title: Enable Daily Refresh + multi: false + required: false + show_user: true + default: false + description: >- + When enabled, the integration performs a daily full refresh of all attributes from the MISP API (every 24 hours), ignoring the cursor and re-fetching from Initial Interval. This ensures decay scores are updated and attributes ingested during a daily refresh will expire 24 hours after ingestion, allowing the next refresh cycle to update their decay state from MISP. - name: ioc_expiration_duration type: text title: IOC Expiration Duration @@ -43,7 +52,7 @@ streams: show_user: true default: "90d" description: >- - Enforces all IOCs to expire after this duration. This setting is required to avoid "orphaned" IOCs that never expire. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g 10d) + Enforces all IOCs to expire after this duration. This setting applies to ALL ingested attributes (not just orphaned IOCs) and serves as a fail-safe expiration for "orphaned" IOCs that never expire. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g 10d). When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. - name: http_client_timeout type: text title: HTTP Client Timeout diff --git a/packages/ti_misp/data_stream/threat_attributes/sample_event.json b/packages/ti_misp/data_stream/threat_attributes/sample_event.json index 28b47f1648d..e46d411f1e2 100644 --- a/packages/ti_misp/data_stream/threat_attributes/sample_event.json +++ b/packages/ti_misp/data_stream/threat_attributes/sample_event.json @@ -1,34 +1,35 @@ { "@timestamp": "2014-10-03T07:14:05.000Z", "agent": { - "ephemeral_id": "98efca5d-4e4c-4bab-b557-dccd2aa01ed0", - "id": "b20dde43-9229-4544-be2f-fc8d8a4f5450", - "name": "elastic-agent-78638", + "ephemeral_id": "f755233a-3146-44ed-8822-7de70dab3b39", + "id": "7b07b3d9-db6c-4091-a7c4-bbbd22fc8bfe", + "name": "elastic-agent-44381", "type": "filebeat", - "version": "8.19.4" + "version": "9.2.1" }, "data_stream": { "dataset": "ti_misp.threat_attributes", - "namespace": "20988", + "namespace": "23477", "type": "logs" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "b20dde43-9229-4544-be2f-fc8d8a4f5450", + "id": "7b07b3d9-db6c-4091-a7c4-bbbd22fc8bfe", "snapshot": false, - "version": "8.19.4" + "version": "9.2.1" }, "event": { "agent_id_status": "verified", "category": [ "threat" ], - "created": "2025-12-03T07:52:43.898Z", + "created": "2025-12-03T17:32:54.494Z", "dataset": "ti_misp.threat_attributes", - "ingested": "2025-12-03T07:52:46Z", + "ingested": "2025-12-03T17:32:57Z", "kind": "enrichment", + "module": "ti_misp", "original": "{\"Event\":{\"distribution\":\"3\",\"id\":\"1\",\"info\":\"OSINT ShellShock scanning IPs from OpenDNS\",\"org_id\":\"1\",\"orgc_id\":\"2\",\"uuid\":\"542e4c9c-cadc-4f8f-bb11-6d13950d210b\"},\"category\":\"External analysis\",\"comment\":\"\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"1\",\"first_seen\":null,\"id\":\"1\",\"last_seen\":null,\"object_id\":\"0\",\"object_relation\":null,\"sharing_group_id\":\"0\",\"timestamp\":\"1412320445\",\"to_ids\":false,\"type\":\"link\",\"uuid\":\"542e4cbd-ee78-4a57-bfb8-1fda950d210b\",\"value\":\"http://labs.opendns.com/2014/10/02/opendns-and-bash/\"}", "type": [ "indicator" @@ -37,6 +38,9 @@ "input": { "type": "httpjson" }, + "labels": { + "is_ioc_transform_source": "true" + }, "misp": { "attribute": { "category": "External analysis", @@ -72,6 +76,7 @@ ], "threat": { "feed": { + "dashboard_id": "ti_misp-56ed8040-6c7d-11ec-9bce-f7a4dc94c294", "name": "MISP" }, "indicator": { diff --git a/packages/ti_misp/docs/README.md b/packages/ti_misp/docs/README.md index 670850cfa2e..bf55b9fb594 100644 --- a/packages/ti_misp/docs/README.md +++ b/packages/ti_misp/docs/README.md @@ -262,8 +262,34 @@ This data stream uses the `/attributes/restSearch` API endpoint which returns mo #### Expiration of Indicators of Compromise (IOCs) The ingested IOCs expire after certain duration which is indicated by the `decayed` field. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named `logs-ti_misp_latest.dest_threat_attributes-*` which only contains active and unexpired IOCs. The latest destination index also has an alias named `logs-ti_misp_latest.threat_attributes`. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for `Threat Attributes` datastream are also pointing to the latest destination indices containing active IoCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source datastream `.ds-logs-ti_misp.threat_attributes-*` indices. +#### Daily Refresh Mode +By default, the integration uses incremental updates, only fetching attributes that have been modified since the last poll (tracked via an internal cursor). However, MISP's decay scores are dynamic and decrease over time, which means an attribute's decay status may change without the attribute itself being modified. In such cases, incremental updates would not capture the updated decay state. + +To address this, users can enable the `Enable Daily Refresh` toggle. When enabled, the integration will: +1. **Perform a daily full refresh**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP. +2. **Set 24-hour expiration**: Attributes ingested during a daily refresh will have their `decayed_at` set to 24 hours after ingestion, ensuring they expire before the next refresh cycle. +3. **Update decay states**: The next daily refresh will re-ingest attributes with their current decay scores from MISP, removing any that have since been marked as decayed. + +This approach ensures that: +- The destination indices stay aligned with MISP's current view of valid indicators +- Attributes that become decayed in MISP are automatically removed in the next refresh cycle +- No stale indicators remain in the destination indices beyond 24 hours + +**Note**: Daily refreshes will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refresh period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion are not affected by the 24-hour expiration and will be removed immediately. + +#### IOC Expiration Duration +The `IOC Expiration Duration` parameter controls when ingested IOCs are marked as expired when **Daily Refresh is disabled**. This setting applies to all ingested attributes that are not decayed, not just orphaned IOCs. The expiration date for each attribute is calculated as `max(last_seen, timestamp) + IOC Expiration Duration`, which defaults to 90 days. + +**Note**: When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. This ensures attributes are refreshed with current decay scores from MISP in the next daily cycle. + +When Daily Refresh is disabled, this setting serves as a fail-safe expiration mechanism that works independently of MISP's decay models. Even if MISP does not mark an attribute as decayed, Elastic will expire the attribute after the configured duration. + #### Handling Orphaned IOCs -Some IOCs may never get decayed/expired and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. To avoid any false positives from such orphaned IOCs, users are allowed to configure `IOC Expiration Duration` parameter while setting up the integration. This parameter deletes all data inside the destination indices `logs-ti_misp_latest.dest_threat_attributes-*` after this specified duration is reached, defaults to `90d` after attribute's `max(last_seen, timestamp)`. Note that `IOC Expiration Duration` parameter only exists to add a fail-safe default expiration in case IOCs never expire. +Some IOCs may never get decayed/expired by MISP's decay models and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. + +When `Enable Daily Refresh` is **disabled**, the `IOC Expiration Duration` parameter ensures these orphaned IOCs are eventually removed from destination indices after the specified duration from the attribute's `max(last_seen, timestamp)`. + +When `Enable Daily Refresh` is **enabled**, orphaned IOCs are handled automatically by the 24-hour expiration cycle. Each daily refresh re-ingests all attributes with their current decay state from MISP, ensuring the destination indices remain aligned with MISP's view of valid indicators. #### ILM Policy To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_misp.threat_attributes-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date. diff --git a/packages/ti_misp/manifest.yml b/packages/ti_misp/manifest.yml index 8507d18de32..e29a4c19ef0 100644 --- a/packages/ti_misp/manifest.yml +++ b/packages/ti_misp/manifest.yml @@ -1,6 +1,6 @@ name: ti_misp title: MISP -version: "1.39.0" +version: "1.40.0" description: Ingest threat intelligence indicators from MISP platform with Elastic Agent. type: integration format_version: "3.0.2" From a6a046a517766becd6ce12f3af86f83e038f34db Mon Sep 17 00:00:00 2001 From: chemamartinez Date: Wed, 10 Dec 2025 09:17:26 +0100 Subject: [PATCH 2/3] Update changelog --- packages/ti_misp/changelog.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/ti_misp/changelog.yml b/packages/ti_misp/changelog.yml index 58c47ffc496..ce4c489b310 100644 --- a/packages/ti_misp/changelog.yml +++ b/packages/ti_misp/changelog.yml @@ -3,10 +3,10 @@ changes: - description: Clarify documentation about the data retention policy. type: enhancement - link: https://github.com/elastic/integrations/pull/99999 + link: https://github.com/elastic/integrations/pull/16491 - description: Add option to refresh ingested indicators daily. type: enhancement - link: https://github.com/elastic/integrations/pull/99999 + link: https://github.com/elastic/integrations/pull/16491 - version: "1.39.0" changes: - description: Prevent updating fleet health status to degraded when pagination completes. From 2371dcad7ac3d2123a144bd27eb0ccd05802e663 Mon Sep 17 00:00:00 2001 From: chemamartinez Date: Tue, 23 Dec 2025 17:21:23 +0100 Subject: [PATCH 3/3] Avoid forcing a 24h expiration time when daily refetch is enabled --- packages/ti_misp/_dev/build/docs/README.md | 27 +++------- packages/ti_misp/changelog.yml | 5 +- .../data_stream/threat/sample_event.json | 22 ++++---- ...nfig.yml => test-daily-refetch-config.yml} | 2 +- .../agent/stream/httpjson.yml.hbs | 54 +++++++++---------- .../elasticsearch/ingest_pipeline/default.yml | 21 +------- .../threat_attributes/manifest.yml | 8 +-- .../threat_attributes/sample_event.json | 14 ++--- packages/ti_misp/docs/README.md | 49 +++++++---------- 9 files changed, 79 insertions(+), 123 deletions(-) rename packages/ti_misp/data_stream/threat_attributes/_dev/test/system/{test-daily-refresh-config.yml => test-daily-refetch-config.yml} (92%) diff --git a/packages/ti_misp/_dev/build/docs/README.md b/packages/ti_misp/_dev/build/docs/README.md index 97ff058ecb6..4f6a2da3afb 100644 --- a/packages/ti_misp/_dev/build/docs/README.md +++ b/packages/ti_misp/_dev/build/docs/README.md @@ -23,34 +23,21 @@ This data stream uses the `/attributes/restSearch` API endpoint which returns mo #### Expiration of Indicators of Compromise (IOCs) The ingested IOCs expire after certain duration which is indicated by the `decayed` field. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named `logs-ti_misp_latest.dest_threat_attributes-*` which only contains active and unexpired IOCs. The latest destination index also has an alias named `logs-ti_misp_latest.threat_attributes`. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for `Threat Attributes` datastream are also pointing to the latest destination indices containing active IoCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source datastream `.ds-logs-ti_misp.threat_attributes-*` indices. -#### Daily Refresh Mode +#### Daily Refetch Mode By default, the integration uses incremental updates, only fetching attributes that have been modified since the last poll (tracked via an internal cursor). However, MISP's decay scores are dynamic and decrease over time, which means an attribute's decay status may change without the attribute itself being modified. In such cases, incremental updates would not capture the updated decay state. -To address this, users can enable the `Enable Daily Refresh` toggle. When enabled, the integration will: -1. **Perform a daily full refresh**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP. -2. **Set 24-hour expiration**: Attributes ingested during a daily refresh will have their `decayed_at` set to 24 hours after ingestion, ensuring they expire before the next refresh cycle. -3. **Update decay states**: The next daily refresh will re-ingest attributes with their current decay scores from MISP, removing any that have since been marked as decayed. +To address this, users can enable the `Enable Daily Refetch` toggle. When enabled, the integration will: +1. **Perform a daily full refetch**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP. +2. **Update decay states**: Thanks to the re-ingest of all attributes with their current decay scores from MISP, it removes any that have since been marked as decayed from destination indices. This approach ensures that: - The destination indices stay aligned with MISP's current view of valid indicators -- Attributes that become decayed in MISP are automatically removed in the next refresh cycle -- No stale indicators remain in the destination indices beyond 24 hours +- Attributes that become decayed in MISP are automatically removed in the next refetch cycle from destination indices -**Note**: Daily refreshes will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refresh period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion are not affected by the 24-hour expiration and will be removed immediately. - -#### IOC Expiration Duration -The `IOC Expiration Duration` parameter controls when ingested IOCs are marked as expired when **Daily Refresh is disabled**. This setting applies to all ingested attributes that are not decayed, not just orphaned IOCs. The expiration date for each attribute is calculated as `max(last_seen, timestamp) + IOC Expiration Duration`, which defaults to 90 days. - -**Note**: When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. This ensures attributes are refreshed with current decay scores from MISP in the next daily cycle. - -When Daily Refresh is disabled, this setting serves as a fail-safe expiration mechanism that works independently of MISP's decay models. Even if MISP does not mark an attribute as decayed, Elastic will expire the attribute after the configured duration. +**Note**: This mode will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refetch period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion will be removed immediately. #### Handling Orphaned IOCs -Some IOCs may never get decayed/expired by MISP's decay models and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. - -When `Enable Daily Refresh` is **disabled**, the `IOC Expiration Duration` parameter ensures these orphaned IOCs are eventually removed from destination indices after the specified duration from the attribute's `max(last_seen, timestamp)`. - -When `Enable Daily Refresh` is **enabled**, orphaned IOCs are handled automatically by the 24-hour expiration cycle. Each daily refresh re-ingests all attributes with their current decay state from MISP, ensuring the destination indices remain aligned with MISP's view of valid indicators. +Some IOCs may never get decayed/expired and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. To avoid any false positives from such orphaned IOCs, users are allowed to configure `IOC Expiration Duration` parameter while setting up the integration. This parameter deletes all data inside the destination indices `logs-ti_misp_latest.dest_threat_attributes-*` after this specified duration is reached, defaults to `90d` after attribute's `max(last_seen, timestamp)`. Note that `IOC Expiration Duration` parameter only exists to add a fail-safe default expiration in case IOCs never expire. #### ILM Policy To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_misp.threat_attributes-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date. diff --git a/packages/ti_misp/changelog.yml b/packages/ti_misp/changelog.yml index 9b1f26ff019..75262037e35 100644 --- a/packages/ti_misp/changelog.yml +++ b/packages/ti_misp/changelog.yml @@ -1,10 +1,7 @@ # newer versions go on top - version: "1.41.0" changes: - - description: Clarify documentation about the data retention policy. - type: enhancement - link: https://github.com/elastic/integrations/pull/16491 - - description: Add option to refresh ingested indicators daily. + - description: Add option to refetch ingested indicators daily. type: enhancement link: https://github.com/elastic/integrations/pull/16491 - version: "1.40.0" diff --git a/packages/ti_misp/data_stream/threat/sample_event.json b/packages/ti_misp/data_stream/threat/sample_event.json index 090567f0a1e..b1ca264eef4 100644 --- a/packages/ti_misp/data_stream/threat/sample_event.json +++ b/packages/ti_misp/data_stream/threat/sample_event.json @@ -1,34 +1,35 @@ { "@timestamp": "2021-05-21T10:22:12.000Z", "agent": { - "ephemeral_id": "4ec820f2-c626-43cc-b3db-568e6ad9b30a", - "id": "3faf71dc-932a-4b95-a008-0d898b8d33bb", - "name": "elastic-agent-17637", + "ephemeral_id": "8b246fe9-01d2-4eca-9fbf-c44c7ab815a2", + "id": "24203e73-bb5e-4816-9ac3-b8e3b0d7174b", + "name": "elastic-agent-22844", "type": "filebeat", - "version": "8.19.4" + "version": "9.2.1" }, "data_stream": { "dataset": "ti_misp.threat", - "namespace": "95126", + "namespace": "41648", "type": "logs" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "3faf71dc-932a-4b95-a008-0d898b8d33bb", + "id": "24203e73-bb5e-4816-9ac3-b8e3b0d7174b", "snapshot": false, - "version": "8.19.4" + "version": "9.2.1" }, "event": { "agent_id_status": "verified", "category": [ "threat" ], - "created": "2025-12-03T07:51:45.031Z", + "created": "2025-12-23T16:15:30.344Z", "dataset": "ti_misp.threat", - "ingested": "2025-12-03T07:51:48Z", + "ingested": "2025-12-23T16:15:33Z", "kind": "enrichment", + "module": "ti_misp", "original": "{\"Event\":{\"Attribute\":{\"Galaxy\":[],\"ShadowAttribute\":[],\"category\":\"Payload delivery\",\"comment\":\"filename content for test event 3\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"3633\",\"first_seen\":null,\"id\":\"266263\",\"last_seen\":null,\"object_id\":\"0\",\"object_relation\":null,\"sharing_group_id\":\"0\",\"timestamp\":\"1621589229\",\"to_ids\":false,\"type\":\"filename\",\"uuid\":\"3b322e1a-1dd8-490c-ab96-12e1bc3ee6a3\",\"value\":\"thetestfile.txt\"},\"EventReport\":[],\"Galaxy\":[],\"Object\":{\"Attribute\":{\"Galaxy\":[],\"ShadowAttribute\":[],\"category\":\"Payload delivery\",\"comment\":\"\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"3633\",\"first_seen\":null,\"id\":\"266265\",\"last_seen\":null,\"object_id\":\"18207\",\"object_relation\":\"sha256\",\"sharing_group_id\":\"0\",\"timestamp\":\"1621589548\",\"to_ids\":true,\"type\":\"sha256\",\"uuid\":\"657c5f2b-9d68-4ff7-a9ad-ab9e6a6c953e\",\"value\":\"f33c27745f2bd87344be790465ef984a972fd539dc83bd4f61d4242c607ef1ee\"},\"ObjectReference\":[],\"comment\":\"File object for event 3\",\"deleted\":false,\"description\":\"File object describing a file with meta-information\",\"distribution\":\"5\",\"event_id\":\"3633\",\"first_seen\":null,\"id\":\"18207\",\"last_seen\":null,\"meta-category\":\"file\",\"name\":\"file\",\"sharing_group_id\":\"0\",\"template_uuid\":\"688c46fb-5edb-40a3-8273-1af7923e2215\",\"template_version\":\"22\",\"timestamp\":\"1621589548\",\"uuid\":\"42a88ad4-6834-46a9-a18b-aff9e078a4ea\"},\"Org\":{\"id\":\"1\",\"local\":true,\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"Orgc\":{\"id\":\"1\",\"local\":true,\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"RelatedEvent\":[{\"Event\":{\"Org\":{\"id\":\"1\",\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"Orgc\":{\"id\":\"1\",\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"analysis\":\"0\",\"date\":\"2021-05-21\",\"distribution\":\"1\",\"id\":\"3631\",\"info\":\"Test event 1 just atrributes\",\"org_id\":\"1\",\"orgc_id\":\"1\",\"published\":false,\"threat_level_id\":\"1\",\"timestamp\":\"1621588162\",\"uuid\":\"8ca56ae9-3747-4172-93d2-808da1a4eaf3\"}}],\"ShadowAttribute\":[],\"analysis\":\"0\",\"attribute_count\":\"6\",\"date\":\"2021-05-21\",\"disable_correlation\":false,\"distribution\":\"1\",\"event_creator_email\":\"admin@admin.test\",\"extends_uuid\":\"\",\"id\":\"3633\",\"info\":\"Test event 3 objects and attributes\",\"locked\":false,\"org_id\":\"1\",\"orgc_id\":\"1\",\"proposal_email_lock\":false,\"publish_timestamp\":\"0\",\"published\":false,\"sharing_group_id\":\"0\",\"threat_level_id\":\"1\",\"timestamp\":\"1621592532\",\"uuid\":\"4edb20c7-8175-484d-bdcd-fce6872c1ef3\"}}", "type": [ "indicator" @@ -74,7 +75,7 @@ }, "event": { "attribute_count": 6, - "date": "2021-05-21", + "date": "2021-05-21T00:00:00.000Z", "disable_correlation": false, "distribution": 1, "extends_uuid": "", @@ -119,6 +120,7 @@ ], "threat": { "feed": { + "dashboard_id": "ti_misp-56ed8040-6c7d-11ec-9bce-f7a4dc94c294", "name": "MISP" }, "indicator": { diff --git a/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml b/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refetch-config.yml similarity index 92% rename from packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml rename to packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refetch-config.yml index f3c55ba7ccf..13aadf80c52 100644 --- a/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refresh-config.yml +++ b/packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-daily-refetch-config.yml @@ -9,7 +9,7 @@ data_stream: interval: 1s initial_interval: 10s enable_request_tracer: true - daily_refresh: true + daily_refetch: true ioc_expiration_duration: 5d assert: hit_count: 10 diff --git a/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs b/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs index 3fc1f0cf6ff..983179cf982 100644 --- a/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs +++ b/packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs @@ -46,31 +46,31 @@ request.transforms: - set: target: body.returnFormat value: json -{{#if daily_refresh}} -# Daily refresh mode: every 24 hours, reset cursor to re-fetch all attributes from initial_interval. -# When a daily refresh occurs, the timestamp is reset to initial_interval and last_daily_refresh is updated. +{{#if daily_refetch}} +# Daily refetch mode: every 24 hours, reset cursor to re-fetch all attributes from initial_interval. +# When a daily refetch occurs, the timestamp is reset to initial_interval and last_daily_refetch is updated. - set: target: body.timestamp value: |- - [[- $dailyRefreshDuration := (parseDuration "24h") -]] + [[- $dailyRefetchDuration := (parseDuration "24h") -]] [[- $initialInterval := (parseDuration "-{{initial_interval}}") -]] [[- $initialTimestamp := (now $initialInterval).Unix -]] - [[- $isDailyRefresh := true -]] - [[- if index .cursor "last_daily_refresh" -]] - [[- $lastRefresh := (parseDate .cursor.last_daily_refresh "RFC3339") -]] - [[- $nextRefresh := 0 -]] - [[- with $lastRefresh -]] - [[- $nextRefresh = .Add $dailyRefreshDuration -]] + [[- $isDailyRefetch := true -]] + [[- if index .cursor "last_daily_refetch" -]] + [[- $lastRefetch := (parseDate .cursor.last_daily_refetch "RFC3339") -]] + [[- $nextRefetch := 0 -]] + [[- with $lastRefetch -]] + [[- $nextRefetch = .Add $dailyRefetchDuration -]] [[- end -]] - [[- with $nextRefresh -]] + [[- with $nextRefetch -]] [[- if .Before now -]] - [[- $isDailyRefresh = true -]] + [[- $isDailyRefetch = true -]] [[- else -]] - [[- $isDailyRefresh = false -]] + [[- $isDailyRefetch = false -]] [[- end -]] [[- end -]] [[- end -]] - [[- if $isDailyRefresh -]] + [[- if $isDailyRefetch -]] [[- $initialTimestamp -]] [[- else if index .cursor "timestamp" -]] [[- .cursor.timestamp -]] @@ -121,22 +121,22 @@ response.pagination: cursor: timestamp: value: '[[.last_event.timestamp]]' -{{#if daily_refresh}} - last_daily_refresh: - # Track when the last daily refresh started. Updated when 24h has passed since previous refresh. +{{#if daily_refetch}} + last_daily_refetch: + # Track when the last daily refetch started. Updated when 24h has passed since previous refetch. value: |- - [[- $dailyRefreshDuration := (parseDuration "24h") -]] - [[- if index .cursor "last_daily_refresh" -]] - [[- $lastRefresh := (parseDate .cursor.last_daily_refresh "RFC3339") -]] - [[- $nextRefresh := 0 -]] - [[- with $lastRefresh -]] - [[- $nextRefresh = .Add $dailyRefreshDuration -]] + [[- $dailyRefetchDuration := (parseDuration "24h") -]] + [[- if index .cursor "last_daily_refetch" -]] + [[- $lastRefetch := (parseDate .cursor.last_daily_refetch "RFC3339") -]] + [[- $nextRefetch := 0 -]] + [[- with $lastRefetch -]] + [[- $nextRefetch = .Add $dailyRefetchDuration -]] [[- end -]] - [[- with $nextRefresh -]] + [[- with $nextRefetch -]] [[- if .Before now -]] [[- formatDate now "RFC3339" -]] [[- else -]] - [[- $.cursor.last_daily_refresh -]] + [[- $.cursor.last_daily_refetch -]] [[- end -]] [[- end -]] [[- else -]] @@ -147,8 +147,8 @@ tags: {{#if preserve_original_event}} - preserve_original_event {{/if}} -{{#if daily_refresh}} - - daily_refresh +{{#if daily_refetch}} + - daily_refetch {{/if}} {{#each tags as |tag i|}} - {{tag}} diff --git a/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml b/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml index 9491565783b..ce76e268e56 100644 --- a/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml +++ b/packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml @@ -148,11 +148,10 @@ processors: # Add default decayed_at to expire all IOCs after duration `ioc_expiration_duration` from `_tmp_max_time`. # If user-provided value of `ioc_expiration_duration` is not in d, h, or m, default to 90d. # Set `decayed` as true if decayed_at is before ingest time i.e., already expired during ingestion. -# This script is skipped when daily_refresh tag is present. - script: lang: painless tag: script-default-decayed_at - if: (ctx.misp?.attribute?.decayed_at == null && ctx._conf?.ioc_expiration_duration != null && ctx._conf.ioc_expiration_duration != '' && !(ctx.tags instanceof List && ctx.tags.contains('daily_refresh'))) + if: (ctx.misp?.attribute?.decayed_at == null && ctx._conf?.ioc_expiration_duration != null && ctx._conf.ioc_expiration_duration != '') description: > Add default decayed_at source: > @@ -196,24 +195,6 @@ processors: - append: field: error.message value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' -# Script for daily refresh mode: set decayed_at to 24 hours after ingestion. -# When daily_refresh tag is present, all non-decayed attributes will expire 24 hours after ingestion, -# allowing the next daily refresh cycle to re-ingest them with updated decay scores from MISP. -# Attributes already marked as decayed by MISP (script-misp_decayed) are not affected. - - script: - lang: painless - tag: script-daily_refresh-decayed_at - if: (ctx.misp?.attribute?.decayed != true && ctx.misp?.attribute?.decayed_at == null && ctx.tags instanceof List && ctx.tags.contains('daily_refresh')) - description: > - Set decayed_at to 24 hours after ingestion for daily refresh mode - source: | - ZonedDateTime _tmp_event_ingested = ZonedDateTime.parse(ctx._tmp.event_ingested); - ctx.misp.attribute.decayed_at = _tmp_event_ingested.plusHours(24L); - ctx.tags.remove(ctx.tags.indexOf('daily_refresh')); - on_failure: - - append: - field: error.message - value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' - date: if: ctx.misp?.event?.timestamp != null field: misp.event.timestamp diff --git a/packages/ti_misp/data_stream/threat_attributes/manifest.yml b/packages/ti_misp/data_stream/threat_attributes/manifest.yml index e7d0c433fe3..fa6e8498aaf 100644 --- a/packages/ti_misp/data_stream/threat_attributes/manifest.yml +++ b/packages/ti_misp/data_stream/threat_attributes/manifest.yml @@ -35,15 +35,15 @@ streams: show_user: true default: 120h description: How far back to look for indicators the first time the agent is started. Supported units for this parameter are h/m/s. - - name: daily_refresh + - name: daily_refetch type: bool - title: Enable Daily Refresh + title: Enable Daily Refetch multi: false required: false show_user: true default: false description: >- - When enabled, the integration performs a daily full refresh of all attributes from the MISP API (every 24 hours), ignoring the cursor and re-fetching from Initial Interval. This ensures decay scores are updated and attributes ingested during a daily refresh will expire 24 hours after ingestion, allowing the next refresh cycle to update their decay state from MISP. + When enabled, the integration performs a daily full refetch of all attributes from the MISP API (every 24 hours), ignoring the cursor and re-fetching from Initial Interval. This ensures decay scores are updated and attributes marked as decayed by MISP's decay models are removed from destination indices. - name: ioc_expiration_duration type: text title: IOC Expiration Duration @@ -52,7 +52,7 @@ streams: show_user: true default: "90d" description: >- - Enforces all IOCs to expire after this duration. This setting applies to ALL ingested attributes (not just orphaned IOCs) and serves as a fail-safe expiration for "orphaned" IOCs that never expire. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g 10d). When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. + Enforces all IOCs to expire after this duration. This setting applies to ALL ingested attributes (not just orphaned IOCs) and serves as a fail-safe expiration for "orphaned" IOCs that never expire. Use [Elasticsearch time units](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#time-units) in days, hours, or minutes (e.g 10d). - name: http_client_timeout type: text title: HTTP Client Timeout diff --git a/packages/ti_misp/data_stream/threat_attributes/sample_event.json b/packages/ti_misp/data_stream/threat_attributes/sample_event.json index e46d411f1e2..a3f27005687 100644 --- a/packages/ti_misp/data_stream/threat_attributes/sample_event.json +++ b/packages/ti_misp/data_stream/threat_attributes/sample_event.json @@ -1,22 +1,22 @@ { "@timestamp": "2014-10-03T07:14:05.000Z", "agent": { - "ephemeral_id": "f755233a-3146-44ed-8822-7de70dab3b39", - "id": "7b07b3d9-db6c-4091-a7c4-bbbd22fc8bfe", - "name": "elastic-agent-44381", + "ephemeral_id": "8840bb4d-14f5-474c-a674-58b87780cf68", + "id": "6b0fdd15-672d-4d68-8efe-da26fe41c131", + "name": "elastic-agent-27049", "type": "filebeat", "version": "9.2.1" }, "data_stream": { "dataset": "ti_misp.threat_attributes", - "namespace": "23477", + "namespace": "90086", "type": "logs" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "7b07b3d9-db6c-4091-a7c4-bbbd22fc8bfe", + "id": "6b0fdd15-672d-4d68-8efe-da26fe41c131", "snapshot": false, "version": "9.2.1" }, @@ -25,9 +25,9 @@ "category": [ "threat" ], - "created": "2025-12-03T17:32:54.494Z", + "created": "2025-12-23T16:17:09.559Z", "dataset": "ti_misp.threat_attributes", - "ingested": "2025-12-03T17:32:57Z", + "ingested": "2025-12-23T16:17:12Z", "kind": "enrichment", "module": "ti_misp", "original": "{\"Event\":{\"distribution\":\"3\",\"id\":\"1\",\"info\":\"OSINT ShellShock scanning IPs from OpenDNS\",\"org_id\":\"1\",\"orgc_id\":\"2\",\"uuid\":\"542e4c9c-cadc-4f8f-bb11-6d13950d210b\"},\"category\":\"External analysis\",\"comment\":\"\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"1\",\"first_seen\":null,\"id\":\"1\",\"last_seen\":null,\"object_id\":\"0\",\"object_relation\":null,\"sharing_group_id\":\"0\",\"timestamp\":\"1412320445\",\"to_ids\":false,\"type\":\"link\",\"uuid\":\"542e4cbd-ee78-4a57-bfb8-1fda950d210b\",\"value\":\"http://labs.opendns.com/2014/10/02/opendns-and-bash/\"}", diff --git a/packages/ti_misp/docs/README.md b/packages/ti_misp/docs/README.md index bf55b9fb594..df00f4c15a5 100644 --- a/packages/ti_misp/docs/README.md +++ b/packages/ti_misp/docs/README.md @@ -114,34 +114,35 @@ An example event for `threat` looks as following: { "@timestamp": "2021-05-21T10:22:12.000Z", "agent": { - "ephemeral_id": "4ec820f2-c626-43cc-b3db-568e6ad9b30a", - "id": "3faf71dc-932a-4b95-a008-0d898b8d33bb", - "name": "elastic-agent-17637", + "ephemeral_id": "8b246fe9-01d2-4eca-9fbf-c44c7ab815a2", + "id": "24203e73-bb5e-4816-9ac3-b8e3b0d7174b", + "name": "elastic-agent-22844", "type": "filebeat", - "version": "8.19.4" + "version": "9.2.1" }, "data_stream": { "dataset": "ti_misp.threat", - "namespace": "95126", + "namespace": "41648", "type": "logs" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "3faf71dc-932a-4b95-a008-0d898b8d33bb", + "id": "24203e73-bb5e-4816-9ac3-b8e3b0d7174b", "snapshot": false, - "version": "8.19.4" + "version": "9.2.1" }, "event": { "agent_id_status": "verified", "category": [ "threat" ], - "created": "2025-12-03T07:51:45.031Z", + "created": "2025-12-23T16:15:30.344Z", "dataset": "ti_misp.threat", - "ingested": "2025-12-03T07:51:48Z", + "ingested": "2025-12-23T16:15:33Z", "kind": "enrichment", + "module": "ti_misp", "original": "{\"Event\":{\"Attribute\":{\"Galaxy\":[],\"ShadowAttribute\":[],\"category\":\"Payload delivery\",\"comment\":\"filename content for test event 3\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"3633\",\"first_seen\":null,\"id\":\"266263\",\"last_seen\":null,\"object_id\":\"0\",\"object_relation\":null,\"sharing_group_id\":\"0\",\"timestamp\":\"1621589229\",\"to_ids\":false,\"type\":\"filename\",\"uuid\":\"3b322e1a-1dd8-490c-ab96-12e1bc3ee6a3\",\"value\":\"thetestfile.txt\"},\"EventReport\":[],\"Galaxy\":[],\"Object\":{\"Attribute\":{\"Galaxy\":[],\"ShadowAttribute\":[],\"category\":\"Payload delivery\",\"comment\":\"\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"3633\",\"first_seen\":null,\"id\":\"266265\",\"last_seen\":null,\"object_id\":\"18207\",\"object_relation\":\"sha256\",\"sharing_group_id\":\"0\",\"timestamp\":\"1621589548\",\"to_ids\":true,\"type\":\"sha256\",\"uuid\":\"657c5f2b-9d68-4ff7-a9ad-ab9e6a6c953e\",\"value\":\"f33c27745f2bd87344be790465ef984a972fd539dc83bd4f61d4242c607ef1ee\"},\"ObjectReference\":[],\"comment\":\"File object for event 3\",\"deleted\":false,\"description\":\"File object describing a file with meta-information\",\"distribution\":\"5\",\"event_id\":\"3633\",\"first_seen\":null,\"id\":\"18207\",\"last_seen\":null,\"meta-category\":\"file\",\"name\":\"file\",\"sharing_group_id\":\"0\",\"template_uuid\":\"688c46fb-5edb-40a3-8273-1af7923e2215\",\"template_version\":\"22\",\"timestamp\":\"1621589548\",\"uuid\":\"42a88ad4-6834-46a9-a18b-aff9e078a4ea\"},\"Org\":{\"id\":\"1\",\"local\":true,\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"Orgc\":{\"id\":\"1\",\"local\":true,\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"RelatedEvent\":[{\"Event\":{\"Org\":{\"id\":\"1\",\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"Orgc\":{\"id\":\"1\",\"name\":\"ORGNAME\",\"uuid\":\"78acad2d-cc2d-4785-94d6-b428a0070488\"},\"analysis\":\"0\",\"date\":\"2021-05-21\",\"distribution\":\"1\",\"id\":\"3631\",\"info\":\"Test event 1 just atrributes\",\"org_id\":\"1\",\"orgc_id\":\"1\",\"published\":false,\"threat_level_id\":\"1\",\"timestamp\":\"1621588162\",\"uuid\":\"8ca56ae9-3747-4172-93d2-808da1a4eaf3\"}}],\"ShadowAttribute\":[],\"analysis\":\"0\",\"attribute_count\":\"6\",\"date\":\"2021-05-21\",\"disable_correlation\":false,\"distribution\":\"1\",\"event_creator_email\":\"admin@admin.test\",\"extends_uuid\":\"\",\"id\":\"3633\",\"info\":\"Test event 3 objects and attributes\",\"locked\":false,\"org_id\":\"1\",\"orgc_id\":\"1\",\"proposal_email_lock\":false,\"publish_timestamp\":\"0\",\"published\":false,\"sharing_group_id\":\"0\",\"threat_level_id\":\"1\",\"timestamp\":\"1621592532\",\"uuid\":\"4edb20c7-8175-484d-bdcd-fce6872c1ef3\"}}", "type": [ "indicator" @@ -187,7 +188,7 @@ An example event for `threat` looks as following: }, "event": { "attribute_count": 6, - "date": "2021-05-21", + "date": "2021-05-21T00:00:00.000Z", "disable_correlation": false, "distribution": 1, "extends_uuid": "", @@ -232,6 +233,7 @@ An example event for `threat` looks as following: ], "threat": { "feed": { + "dashboard_id": "ti_misp-56ed8040-6c7d-11ec-9bce-f7a4dc94c294", "name": "MISP" }, "indicator": { @@ -262,34 +264,21 @@ This data stream uses the `/attributes/restSearch` API endpoint which returns mo #### Expiration of Indicators of Compromise (IOCs) The ingested IOCs expire after certain duration which is indicated by the `decayed` field. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates destination indices named `logs-ti_misp_latest.dest_threat_attributes-*` which only contains active and unexpired IOCs. The latest destination index also has an alias named `logs-ti_misp_latest.threat_attributes`. When querying for active indicators or setting up indicator match rules, only use the latest destination indices or the alias to avoid false positives from expired IOCs. Dashboards for `Threat Attributes` datastream are also pointing to the latest destination indices containing active IoCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source datastream `.ds-logs-ti_misp.threat_attributes-*` indices. -#### Daily Refresh Mode +#### Daily Refetch Mode By default, the integration uses incremental updates, only fetching attributes that have been modified since the last poll (tracked via an internal cursor). However, MISP's decay scores are dynamic and decrease over time, which means an attribute's decay status may change without the attribute itself being modified. In such cases, incremental updates would not capture the updated decay state. -To address this, users can enable the `Enable Daily Refresh` toggle. When enabled, the integration will: -1. **Perform a daily full refresh**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP. -2. **Set 24-hour expiration**: Attributes ingested during a daily refresh will have their `decayed_at` set to 24 hours after ingestion, ensuring they expire before the next refresh cycle. -3. **Update decay states**: The next daily refresh will re-ingest attributes with their current decay scores from MISP, removing any that have since been marked as decayed. +To address this, users can enable the `Enable Daily Refetch` toggle. When enabled, the integration will: +1. **Perform a daily full refetch**: Every 24 hours, the cursor is reset and all attributes from the configured `Initial Interval` are re-fetched from MISP. +2. **Update decay states**: Thanks to the re-ingest of all attributes with their current decay scores from MISP, it removes any that have since been marked as decayed from destination indices. This approach ensures that: - The destination indices stay aligned with MISP's current view of valid indicators -- Attributes that become decayed in MISP are automatically removed in the next refresh cycle -- No stale indicators remain in the destination indices beyond 24 hours +- Attributes that become decayed in MISP are automatically removed in the next refetch cycle from destination indices -**Note**: Daily refreshes will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refresh period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion are not affected by the 24-hour expiration and will be removed immediately. - -#### IOC Expiration Duration -The `IOC Expiration Duration` parameter controls when ingested IOCs are marked as expired when **Daily Refresh is disabled**. This setting applies to all ingested attributes that are not decayed, not just orphaned IOCs. The expiration date for each attribute is calculated as `max(last_seen, timestamp) + IOC Expiration Duration`, which defaults to 90 days. - -**Note**: When `Enable Daily Refresh` is enabled, this setting is ignored and all non-decayed attributes will expire 24 hours after ingestion instead. This ensures attributes are refreshed with current decay scores from MISP in the next daily cycle. - -When Daily Refresh is disabled, this setting serves as a fail-safe expiration mechanism that works independently of MISP's decay models. Even if MISP does not mark an attribute as decayed, Elastic will expire the attribute after the configured duration. +**Note**: This mode will re-ingest all attributes within the `Initial Interval` window, which may result in higher data volume during the refetch period. The transform handles deduplication via unique keys. Attributes already marked as decayed by MISP's decay models during ingestion will be removed immediately. #### Handling Orphaned IOCs -Some IOCs may never get decayed/expired by MISP's decay models and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. - -When `Enable Daily Refresh` is **disabled**, the `IOC Expiration Duration` parameter ensures these orphaned IOCs are eventually removed from destination indices after the specified duration from the attribute's `max(last_seen, timestamp)`. - -When `Enable Daily Refresh` is **enabled**, orphaned IOCs are handled automatically by the 24-hour expiration cycle. Each daily refresh re-ingests all attributes with their current decay state from MISP, ensuring the destination indices remain aligned with MISP's view of valid indicators. +Some IOCs may never get decayed/expired and will continue to stay in the latest destination indices `logs-ti_misp_latest.dest_threat_attributes-*`. To avoid any false positives from such orphaned IOCs, users are allowed to configure `IOC Expiration Duration` parameter while setting up the integration. This parameter deletes all data inside the destination indices `logs-ti_misp_latest.dest_threat_attributes-*` after this specified duration is reached, defaults to `90d` after attribute's `max(last_seen, timestamp)`. Note that `IOC Expiration Duration` parameter only exists to add a fail-safe default expiration in case IOCs never expire. #### ILM Policy To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_misp.threat_attributes-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date.