NIFI-15480: Added ability to drop FlowFiles selectively using a Predi… #10780
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduced comprehensive REST endpoints for managing Connectors through the NiFi web interface and API clients. Key capabilities: - Full lifecycle management (create, read, update, delete) - Start/stop control with run status endpoints - Multi-step configuration with property groups supporting string literals, asset references, and secret references - Asynchronous configuration verification with polling support - Access to connector's managed flow and process group status - Search functionality within connector's encapsulated flow Cluster support: - Request replication across cluster nodes - Response merging for dynamic property allowable values that may differ between nodes - Consistent configuration step and property descriptor handling Infrastructure: - Audit logging for all connector configuration changes - Authorization integration with NiFi's policy framework - UI extension support for connector-specific interfaces - Frontend updates for connector access policy management
… ConfigurationStep are unique (apache#10626) * NIFI-15322: Huge refactoring of Connectors to require that <Step Name, Property Name> tuple is unique. Requiring that is simple, but now that we enforce that, we can greatly simplify much of the code. * NIFI-15322: Added StepConfiguration to simplify API rather than using Map<String, ConnectorValueReference> in some places * NIFI-15322: Now that we enforce unique property names per step, introduced StepConfigurationContext to further reduce API complexity Signed-off-by: Kevin Doran <kdoran@apache.org>
…he#10631) NIFI-15322: Adapting to new connector configuration step model.
…r; updated mock framework to support secrets; validation cleanup (apache#10638)
…derSecretProvider to provide the Parameter Provider as the associated Authorizable (apache#10644)
apache#10654) * NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be used in system tests; created system tests for Connectors; bug fixes found running tests; updated Secret/SecretReference to support fully qualified names * NIFI-15352: Addressed review feedback
apache#10645) * NIFI-15343: Adding an endpoint to return available secrets to the connector configuration wizard. * NIFI-15343: Adding providerId to SecretDTO. * NIFI-15343: Skipping ParameterProviders that are invalid or validating when fetching secrets. * NIFI-15343: Adding fully qualified name to SecretDTO. - Fixing rebase issue. This closes apache#10645
- Support multiple asset references - Clean up of assets in StandardConnectorDAO after apply and discard - Add endpoint for listing assets for a given connector - Add endpoint for discarding the working config of a connector - Add endpoint for retrieving asset content - Add system test for connectors that uploads an asset Signed-off-by: Kevin Doran <kdoran@apache.org> * Fix issues after rebasing, enable ClusteredConnectorAssetsIT
apache#10667) * NIFI-15361: Allowing configuration step documentation to be returned independent of any connector instances.
…onStep,Property) tuple (apache#10669)
apache#10665) * NIFI-15367: Ensure that Connectors' implicit parameter contexts are not registered with ParameterContextManager; added system test to verify; fixed bug in ConnectorAssetsIT where it did not wait for connector to return to its 'STOPPED' state before attempting to delete * NIFI-15367: Bug fixes; review feedback
…eate process groups if they have a connector ID. (apache#10673)
…ache#10683) * NIFI-15353: Adding support for rendering Connector Documentation. * NIFI-15353: Addressing review feedback. * NIFI-15353: Using mergeMap instead of switchMap to allow for concurrent doc loading. This closes apache#10683
apache#10732) * NIFI-15429: Adding an optional query parameter for specifying which process group should be returned within the Connectors flow. * NIFI-15429: Including managed process group id in the ConnectorDTO. * NIFI-15429: Addressing review feedback. * NIFI-15429: Updating path for fetching a Connector flow to require the desired Process Group ID. * NIFI-15429: Fixing failing unit test.
apache#10733) * NIFI-15430: Ensure that if we fail to initialize a Connector, we create a GhostConnector instead and ensure that we also proivde the extensionMissing flag on ConnectorNode * NIFI-15430: Added extensionMissing flag to Connector DTO * NIFI-15430: If unable to load initial flow of a Connector, make ghosted connector instead
apache#10730) * NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a Connector; added some system tests to verify existing behavior; fixed existing issue with IT * NIFI-15427: Fixed Exception message in case unable to purge FlowFiles due to Connector state * NIFI-15427: Ensured thread safety of state transitions when draining/purging FlowFiles * NIFI-15427: Addressed review feedback
apache#10736) * NIFI-15433: If connector validation throws an Exception keep trying until it completes (with a timeout between retries); some bug fixes * NIFI-15433: Addressed review feedback * NIFI-15433: Fixed failing unit tests
apache#10749) * NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug to ensure that if we fail to apply an update to a Connector that we stop it. * NIFI-15445: Addressed review feedback * NIFI-15445: Fixed checkstyle
* NIFI-15440: Implementation of ConnectorActions * NIFI-15440: Addressed review feedback * NIFI-15440: Addressed issues that occurred after rebase * NIFI-15440: Addressed review feedback
apache#10742) * NIFI-15439: Ensure that Process Groups are accessible from the appropriate endpoint but not accessible elsewhere * NIFI-15439: FlowAnalyzingRegistryClient should use an explicit Connector ID of null when retrieving Process Group from FlowManager
…a Ghost Connector is created (apache#10775) This closes apache#10775
apache#10750) * NIFI-15446: When invoking ConnectorMethod, make sure to serialize/deserialize the arguments and return value * NIFI-15446: Addressed review feedback
…cate Added system tests to verify selective dropping of flowfiles from a Connector. Also made significant updates so that diagnostic bundles are captured when a system test fails and diagnostic bundle also contains details of each Connection's FlowFileQueue. This was necessary in aiding the debugging of the new feature but is also something that has been missing for some time.
Contributor
Author
|
Closing in favor of #10802 after rebase |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…cate
Added system tests to verify selective dropping of flowfiles from a Connector. Also made significant updates so that diagnostic bundles are captured when a system test fails and diagnostic bundle also contains details of each Connection's FlowFileQueue. This was necessary in aiding the debugging of the new feature but is also something that has been missing for some time.
Summary
NIFI-00000
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation