Implement Strobes Analysis/ Filtering For RITA v2 #91
Conversation
…g log (for testing)
…ct wraps a specific DB. Soon, there will be another object introduced called either OutputDB or RITADB which does wrap a single database
… to DBManager since the named stuttered as rita.RITADBManager)
…auto flush collection before issuing the remove command to Mongo
| [[constraint]] | ||
| name = "github.com/activecm/rita" | ||
| version = "^1.0.3" | ||
| version = "^2.0.0" |
There was a problem hiding this comment.
This should have been updated earlier, but it the change was needed now as we have to get the parsetypes.Freq object out of RITA to get the needed MongoDB indexes for the freqConn collection
| coll.Database.Session.Close() | ||
|
|
||
| outDB, err := rita.NewOutputDB(conf.GetOutputConfig().GetRITAConfig()) | ||
| outDB, err := rita.NewDBManager(conf.GetOutputConfig().GetRITAConfig(), 100, 1*time.Second) |
There was a problem hiding this comment.
Had to add some dummy values to make the constructor happy.
| //freqConn collection / Strobes analysis in RITA | ||
| type Strobe interface { | ||
| GetConnectionLimit() int | ||
| } |
There was a problem hiding this comment.
Generally had to touch a few files to implement the ConnectionLimit config option for strobes as in RITA.
Changed:
converter/config/config.go
converter/config/yaml/output.go
converter/integrationtest/config.go
converter/etc/config.yaml
runtime/etc/converter/converter.yaml
| //into the respective database. The data is batched up in buffers | ||
| //before being sent to MongoDB. The buffers are flushed when | ||
| //they are full or after a deadline passes for the individual buffer. | ||
| type batchRITAConnDateWriter struct { |
There was a problem hiding this comment.
The batch writer has to be activated with a cli flag.
It sends each connection to a timestamp based database and marks them ready for import with ipfix-rita closes. It does not do any database rotation. It has been updated to support strobes via the rita.DB class.
Largely references to the AutoFlushCollection directly tied to the conn collection have been replaced with calls to rita.DB
| //connect to the db | ||
| var err error | ||
| outColl, err := r.db.NewRITAOutputConnection(endTimeStr) | ||
| outDB, err := r.db.NewRitaDB(endTimeStr, autoFlushAsyncErrChan, autoFlushOnFatal) |
There was a problem hiding this comment.
NewRITADB encapsulates the removed logic.
| @@ -0,0 +1,106 @@ | |||
| package freqconn_test | |||
There was a problem hiding this comment.
These main_test files set up the integration tests.
To run the integration tests use sudo $(which go) test -v ./.... Its unfortunate that docker requires root privileges to spin up a container, but the ability to write tests against a Mongo database was very valuable while developing this new feature. Note you do not need to use sudo if your user is in the docker group.
| "time" | ||
|
|
||
| "github.com/activecm/ipfix-rita/converter/config" | ||
| "github.com/activecm/ipfix-rita/converter/logging" |
There was a problem hiding this comment.
The streaming session writer is similar to the batch one in that, largely, the changes revolve around switching AutoFlushCollections to rita.DB objects.
| return nil, errors.Wrapf(err, "failed to start auto flusher for collection XXX-%s.%s", newTime.Format(s.timeFormatString), rita.RitaConnInputCollection) | ||
| return nil, errors.Wrapf(err, "failed to create new rita database") | ||
| } | ||
| err = s.ritaDBManager.EnsureMetaDBRecordExists(newColl.Database.Name) |
There was a problem hiding this comment.
This logic has been encapsulated in NewRitaDB
| return sessions | ||
| } | ||
|
|
||
| func TestStrobes(t *testing.T) { |
There was a problem hiding this comment.
This integration test proves that the new code correctly implements strobes.
|
|
||
| env := fixtures.Get(integrationtest.EnvironmentFixture.Key).(environment.Environment) | ||
| strobeLimit := env.GetOutputConfig().GetRITAConfig().GetStrobe().GetConnectionLimit() | ||
| expectedConnCount := strobeLimit + 5 |
There was a problem hiding this comment.
Essentially, we push through the ConnectionLimit + 5 connections with the same source and destination through the streaming session writer and ensure that the conn collection is empty and that the freqconn collection contains a valid record for the data.
|
Running AI-Hunter v2.1.0 I can see the freqConn in the mongo database for a given dataset if I'm running mongo from the command line, but I can't see to get Strobes to show up in AI-Hunter. If you could track down why that's happening I think we should be good to continue with this |
Implements #85.
This PR implements Strobes analysis in IPFIX-RITA. A new configuration option has been introduced to control the Strobes ConnectionLimit as in RITA.
I have not tested the code on a live installation yet. Though I imagine one could set up a beacon script, set the strobe threshold really low and watch the resulting freqConn collection.
However, tests have been written for each new addition to the code base, including integration tests which ensure the code is running as desired.
To run the tests without the integration tests use
go test -v --short ./....To run the tests with integration tests use
sudo $(which go) test -v ./.... The sudo invocation is needed to start MongoDB containers. If your user is a member of the docker group, sudo is not needed.The large amount of changes are due to the fact that the session writers previously dealt directly with AutoFlushCollection objects tied directly to the conn collection of each database. In order to cleanly implement Strobes analysis, we needed to add a layer of indirection which bound the freqconn and conn collections under one database object. Additionally several constants were moved between packages, creating a large number of simple changes.
I understand this is a large amount of code to go through at once. Please contact me if you would like assistance.
The changes are as follows:
Updated RITA Dependency to v2.0.0
Added Strobes ConnectionLimit setting to config as in RITA
Added rita.DB class to encapsulate the logic needed to set up a new RITA database and insert new data into that database.
This class provides an InsertConnRecord() method which uses a ConnCounter to decide what to do with the record.
Added freqconn.ConnCounter class to count connection pairs and call upon a ConnCountNotifier (namely StrobesNotifier) when a connection count
meets or exceeds a given threshold
Added freqconn.StrobesNotifier class to carry out the MongoDB logic needed to implement strobes analysis
Renamed Incorrectly Named OutputDB class to DBManager. Changed the class to hand out rita.DB objects instead of AutoFlushCollections
Updated Batch Session Writer To Use New rita.DB Objects Instead of buffered.AutoFlushCollection Objects
Updated Streaming (Rotating) Session Writer To Use New rita.DB Objects Instead of buffered.AutoFlushCollection Objects
Updated check_config.go to match new RITADBManager constructor
Added comments
Created rita.constants package to hold collection names/ version numbers used in both the rita and freqconn packages