Flink

Send DataSet jobs to an Apache Flink cluster.

Metadata

Property	Value
Scheme	`flink`
Support Level	Preview
Labels	transformation,bigdata
Version	4.20.0 (CEQ 3.36.0)

Maven Dependency

<dependency>
    <groupId>org.apache.camel.quarkus</groupId>
    <artifactId>camel-quarkus-flink</artifactId>
    <version>4.20.0 (CEQ 3.36.0)</version>
</dependency>

Endpoint Properties

Name	Type	Required	Default	Description
`endpointType`	enum	✓		Type of the endpoint (dataset, datastream).
`collect`	boolean		`true`	Indicates if results should be collected or counted.
`dataSet`	object			DataSet to compute against.
`dataSetCallback`	object			Function performing action against a DataSet.
`dataStream`	object			DataStream to compute against.
`dataStreamCallback`	object			Function performing action against a DataStream.
`checkpointingMode`	enum			Checkpointing mode: EXACTLY_ONCE (default) or AT_LEAST_ONCE. EXACTLY_ONCE provides stronger guarantees but may have higher overhead.
`checkpointInterval`	integer			Interval in milliseconds between checkpoints. Enables checkpointing when set. Recommended for streaming jobs to ensure fault tolerance.
`checkpointTimeout`	integer			Timeout in milliseconds for checkpoints. Checkpoints that take longer will be aborted.
`executionMode`	enum			Execution mode for the Flink job. Options: STREAMING (default), BATCH, AUTOMATIC. BATCH mode is recommended for bounded streams (batch processing).
`jobName`	string			Name for the Flink job. Useful for identification in the Flink UI and logs.
`lazyStartProducer`	boolean		`false`	Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.
`maxParallelism`	integer			Maximum parallelism for the Flink job. Defines the upper bound for dynamic scaling and the number of key groups for stateful operators.
`minPauseBetweenCheckpoints`	integer			Minimum pause in milliseconds between consecutive checkpoints. Helps prevent checkpoint storms under heavy load.
`parallelism`	integer			Parallelism for the Flink job. If not set, uses the default parallelism of the execution environment.