Skip to content

Exceptions using the adx argument to export ETL into Azure Data Explorer #61

@suvamM

Description

@suvamM

I have the following scenario: we have a tool dumping ETW events into an ETL file. I want to ingest this into Kusto. I was experimenting with the adx option, but am facing some issues (and didn't find the documentation to unblock me):

  1. What should be my table schema in Kusto? I used a hack, using the Azure Data Explorer one-click ingestion to upload an ETL file to extract the schema. I tried the following schema: EventName: string, Time: real, ProcessName: string, Rest: string, data: string. When I try ingestion using the following command line:
RealTimeKql.exe etl "D:\rEvents.etl" adx --adxcluster=<cluster> --adxdatabase=<db> --adxtable=<tableName>

I see some progress, and then an index out of bounds exception:

Welcome to Real-Time KQL!
Current Batch Count: 5
Current Batch Count: 7
[0]Kusto.Ingest.Exceptions.IngestClientException: An error occurred for source: 'DataReader'. Error: 'Index was outside the bounds of the array.'
  1. When I try to debug by providing the exact command line in Visual Studio, I get the following error:
Welcome to Real-Time KQL!
System.Exception: Unexpected TDH status 1168
   at Tx.Windows.EtwTdhEventInfo.ReadTdhMetadata(EtwNativeEvent& e)
   at Tx.Windows.EtwTdhEventInfo..ctor(EtwNativeEvent& e)
   at Tx.Windows.EtwTdhDeserializer.Deserialize(EtwNativeEvent& e)
   at Tx.Windows.EtwTdhEvent.Materialize()
   at Tx.Windows.EtwTdhEvent.get_Keys()
   at RealTimeKqlLibrary.AdxOutput.OutputAction(IDictionary`2 obj) in D:\repos\KqlTools\Source\RealTimeKqlLibrary\Output\AdxOutput.cs:line 111
Stopping RealTimeKql...

Could you provide some pointers on the Kusto ingestion of ETL files?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions