pulsar.source.allowKeySharedOutOfOrderDelivery |
false |
Boolean |
If enabled, it will relax the ordering requirement, allowing the broker to send out-of-order messages in case of failures. This will make it faster for new consumers to join without being stalled by an existing slow consumer.
In this case, a single consumer will still receive all the keys, but they may be coming in different orders. |
pulsar.source.autoCommitCursorInterval |
5000 |
Long |
This option is used only when the user disables the checkpoint and uses Exclusive or Failover subscription. We would automatically commit the cursor using the given period (in ms). |
pulsar.source.enableAutoAcknowledgeMessage |
false |
Boolean |
Flink commits the consuming position with pulsar transactions on checkpoint. However, if you have disabled the Flink checkpoint or disabled transaction for your Pulsar cluster, ensure that you have set this option to true.
The source would use pulsar client's internal mechanism and commit cursor in a given interval. |
pulsar.source.enableMetrics |
true |
Boolean |
The metrics from Pulsar Consumer are only exposed if you enable this option.You should set the pulsar.client.statsIntervalSeconds to a positive value if you enable this option. |
pulsar.source.enableSchemaEvolution |
false |
Boolean |
If you enable this option and use PulsarSourceBuilder.setDeserializationSchema(Schema), we would consume and deserialize the message by using Pulsar's Schema interface with extra schema evolution check. |
pulsar.source.fetchOneMessageTime |
(none) |
Integer |
The time (in ms) for fetching one message from Pulsar. If time exceed and no message returned from Pulsar. We would consider there is no record at the current topic partition and stop fetching until next switch.
It's not configured by default. We will use the remaining time in pulsar.source.maxFetchTime by default, which may cause a long wait in small message rates. Add this option in source builder avoiding waiting too long. |
pulsar.source.maxFetchRecords |
100 |
Integer |
The maximum number of records to fetch to wait when polling. A longer time increases throughput but also latency. A fetch batch might be finished earlier because of pulsar.source.maxFetchTime. |
pulsar.source.maxFetchTime |
10000 |
Long |
The maximum time (in ms) to wait when fetching records. A longer time increases throughput but also latency. A fetch batch might be finished earlier because of pulsar.source.maxFetchRecords. |
pulsar.source.partitionDiscoveryIntervalMs |
300000 |
Long |
The interval (in ms) for the Pulsar source to discover the new partitions. A non-positive value disables the partition discovery. |
pulsar.source.resetSubscriptionCursor |
false |
Boolean |
The StartCursor in connector is used to create the initial subscription. Enable this option will reset the start cursor in subscription by using StartCursor everytime you start the application without the checkpoint. |
pulsar.source.verifyInitialOffsets |
WARN_ON_MISMATCH |
Enum |
Upon (re)starting the source, check whether the expected message can be read. If failure is enabled, the application fails. Otherwise, it logs a warning. A possible solution is to adjust the retention settings in Pulsar or ignoring the check result.
Possible values:- "FAIL_ON_MISMATCH": Fail the consuming from Pulsar when we don't find the related cursor.
- "WARN_ON_MISMATCH": Print a warn message and start consuming from the valid offset.
|