We found that when record.offset is not equal to the offset we are expecting, kafka will set the variable inPlaceAssignment to false. When inPlaceAssignment is false, data will not be verified. This patch adds the missing CRC verification.
Reviewers: Jason Gustafson <jason@confluent.io>
This fix is already in 2.1+ branches, but did not get into older branches.
Should be cherry-picked all the way to 0.10.2.
Reviewers: Matthias J. Sax <mjsax@apache.org>
Pricing for m3.xlarge: On-Demand is at $0.266. Reserved is at about $0.16 (40% discount). And Spot is at $0.0627 (76% discount relative to On-Demand, or 60% discount relative to Reserved). Insignificant fluctuation in the past 3 months.
Ran on branch builder and works as expected -- each worker is created using spot instances (https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1982/console)
This can be safely backported to 0.10.2 (tested using https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1983/)
Author: Max Zheng <maxzheng.os@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5707 from maxzheng/minor-switch@trunk
(cherry picked from commit 50ec82940d)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Various converters (AvroConverter and JsonConverter) produce a
SchemaAndValue consisting of a logical schema type and a java.util.Date.
This is a fix for SchemaProjector to properly handle the Date.
Author: Robert Yokota <rayokota@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5736 from rayokota/KAFKA-7476
(cherry picked from commit 3edd8e7333)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
In some tests, the check monitoring the JMX tool log output doesn’t quite wait long enough before failing. Increasing the timeout from 10 to 20 seconds.
## Fixes race condition in TestVerifiableProducer sanity test:
The test starts a producer, waits for at least 5 acks, and then
logs in to the worker to grep for the producer process to figure
out what version it is running.
The problem was that the producer was set up to produce 1000 messages
at a rate of 1000 msgs/s and then exit. This means it will have a
typical runtime slightly above 1 second.
Logging in to the vagrant instance might take longer than that thus
resulting in the process grep to fail, failing the test.
This commit doesn't really fix the issue - a proper fix would be to tell
the producer to stick around until explicitly killed - but it increases
the chances of the test passing, at the expense of a slightly longer
runtime.
## Improves error reporting when is_version() fails
Author: Magnus Edenhill <magnus@edenhill.se>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2765 from edenhill/trunk
ZooKeeper listener for change notifications should be created before loading the ACL cache to avoid timing window if acls are modified when broker is starting up.
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@confluent.io>
Maven Central dropped support for all versions but
TLS 1.2, so dependency resolution fails if Gradle
builds run with JDK 7. 2.0 and trunk require JDK 8,
but every other version is affected. Gradle 4.8.1
fixes the issue by enabling TLS 1.2 by default even
when JDK 7 is used.
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
https://issues.apache.org/jira/browse/KAFKA-7058
* Summary of testing strategy: Added new unit test
Author: Gunnar Morling <gunnar.morling@googlemail.com>
Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5225 from gunnarmorling/KAFKA-7058
(cherry picked from commit be846d833c)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
This is a cherry-pick PR from #5207
1. add the committed offsets to checkpointable offset map.
2. add the restoration integration test for the source KTable case.
Now that we support re-initializing state stores, we need to clear the segments when the store is closed so that they can be re-opened.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ted Yu <yuzhihong@gmail.com>
Closes#4324 from dguy/kafka-6360
This could be backported to older branches to reduce the extra log warning messages there, too.
Running Connect system tests in this branch builder job: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1773/
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5151 from rhauch/kafka-7009
(cherry picked from commit 7a1f555676)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
This is a change to improve resource cleanup for sink tasks and source tasks. Now `Task.stop()` is called from both `WorkerSinkTask.close()` and `WorkerSourceTask.close()`.
It is called from `WorkerXXXTask.close()` since this method is called in the `finally` block of `WorkerTask.run()`, and Connect developers use `stop()` to clean up resources.
Author: Robert Yokota <rayokota@gmail.com>
Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5020 from rayokota/K6566-improve-connect-resource-cleanup
Debian installer packages are no longer available for Java 7.
Also upgrade AMI to latest ubuntu/trusty 14 amd64 as the older
one is no longer available.
Note that this only changes the JDK used to build and run
the system tests. We still have Jenkins jobs that compile
and run the JUnit tests with Java 7 so that we don't use
features that are only available in newer Java versions.
This is a backport of #4925 to the 0.10.2 branch.
Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.
The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.
Several debug statements were added to record when the context is called by the connector.
This can be backported to older releases, since this bug goes back to 0.10 or 0.9.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#4716 from rhauch/kafka-6661
(cherry picked from commit e7ef719a5b)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
This will allow us to trace leaked instances back to the job,
so that we can figure out what happened and fix the leak.
Reviewers: Ismael Juma <ismael@juma.me.uk>
add check to KafkaApis, add unit test specific to follower fetch
developed with @mimaison
backport to 0.10.2 of #4433
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>
The previous timeout was 10 seconds, but system test failures have occurred when Zookeeper has started after about 11 seconds. Increasing the timeout to 30 seconds, since most of the time this extra time will not be required, and when it is it will prevent a failed system test.
In addition to merging to `trunk`, please backport to the `0.11.x` and `0.10.2.x` branches.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3774 from rhauch/MINOR-Increase-timeout-of-zookeeper-service-in-system-tests
(cherry picked from commit 75070bdb5d)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
This fixes PlainSaslServerTest from the last commit which uses a class not present in 0.10.2. Also cherry-picked commit 9934d28 to 0.10.2 since that is required for multiple mechanism support (SaslMultiMechanismConsumerTest was failing with the changes without this).
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#3773 from rajinisivaram/MINOR-0.10.2-buildfix
Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with.
This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks.
This change also backports the fix for KAFKA-4942, which was minimal that caused the new tests to fail.
**This is for the `0.10.2` branch; see #3662 for the equivalent and already-approved PR for `trunk` and #3672 for the equivalent and already-approved PR for the `0.11.0` branch.**
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3717 from rhauch/kafka-5731-0.10.2
- Cherry picked from #3432
- Minor checkstyle fixes to get to build
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3623 from enothereska/KAFKA-5571-0.10.2
This allows a build system to set the correct commit ID when .git/HEAD would be wrong
if there are local commits for build purposes.
Author: Max Zheng <mzheng@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3606 from maxzheng/read-commit@trunk
(cherry picked from commit 613971f8d4)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
lordcheng10
Guozhang Wang
John Roesler
Matthias J. Sax
Gardner Vickers
Max Zheng
Robert Yokota
Randall Hauch
Jason Gustafson
Magnus Edenhill
Rajini Sivaram
Matthias J. Sax
Anna Povzner
Ismael Juma
Gunnar Morling
Damian Guy
Jarek Rudzinski
Maytee Chinavanichkit
Ewen Cheslack-Postava
Edoardo Comar
Arjun Satish
Konstantine Karantasis
Manikumar Reddy
Eno Thereska
Paolo Patierno
Max Zheng
Hooman Broujerdi
Colin P. Mccabe