Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Gwen Shapira <cshapi@gmail.com>
Closes#151 from ijuma/kafka-2411-remove-usage-of-blocking-channel
Minor update to point to testing tutorial, and install the correct version of vagrant-hostmanager
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Gwen Shapira
Closes#187 from granders/minor-testing-readme-update
No Jira ticket created, as the Contributing Code Changes doc says it's not necessary for javadoc typo fixes.
Author: Magnus Reftel <magnus.reftel@skatteetaten.no>
Reviewers: Gwen Shapira
Closes#186 from magnusr/feature/its
The sleep() in KafkaConsumer's poll blocked any pending IO from being completed and created a performance bottleneck. It was intended to implement the fetch backoff behavior, but that was a misunderstanding of the setting "retry.backoff.ms" which should only affect failed fetches.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ewen Cheslack-Postava, Gwen Shapira
Closes#180 from hachikuji/KAFKA-2486
… via --consumer.config command line parameter
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ewen Cheslack-Postava, Onur Karaman
Closes#174 from lindong28/KAFKA-2485
The Converter class now translates directly between byte[] and Copycat's data
API instead of requiring an intermediate runtime type like Avro's GenericRecord
or Jackson's JsonNode.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira
Closes#172 from ewencp/kafka-2475-unified-serializer-converter and squashes the following commits:
566c52f [Ewen Cheslack-Postava] Checkstyle fixes
320d0df [Ewen Cheslack-Postava] Restrict offset format.
85797e7 [Ewen Cheslack-Postava] Add StringConverter for using Copycat with raw strings.
698d65c [Ewen Cheslack-Postava] Move and update outdated comment about handing of types for BYTES type in Copycat.
4bed051 [Ewen Cheslack-Postava] KAFKA-2475: Make Copycat only have a Converter class instead of Serializer, Deserializer, and Converter.
This patch makes it possible to publish kafkatest (system test package) to pypi and use it as a library in other projects by:
- including necessary static resources with the package
- renaming the version to conform w/PEP 440, since python packaging tools reject the current version name
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava, Gwen Shapira
Closes#173 from granders/minor-kafkatest-add-manifest
As we handle exceptions or invalid states in Kafka server by shutting it down, there is no reason to use exit() and not halt() in shutdown itself.
Author: asingh <asingh@cloudera.com>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang
Closes#167 from SinghAsDev/KAFKA-2468
Tests standalone mode by running separate source and sink connectors, catting
data into the source file, and validating the output in the sink file. Restarts
the service to verify that clean restarts will result in tasks resuming where
they left off.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Geoff Andreson, Gwen Shapira
Closes#150 from ewencp/kafka-2377-copycat-system-test
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Guozhang Wang
Closes#166 from ewencp/kafka-2467-fix-console-consumer-behavior-regressions and squashes the following commits:
8b5e30c [Ewen Cheslack-Postava] KAFKA-2466: Fix ConsoleConsumer exit process for new consumer to avoid ConcurrentModificationException.
c6abe38 [Ewen Cheslack-Postava] Fix missing parameter in test.
a6961ee [Ewen Cheslack-Postava] KAFKA-2467: Fix changes to behavior in ConsoleConsumer: properly parse consumer.config option, handle exceptions during message processing, and print number of processed messages to stderr.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Edward Ribeiro, Onur Karaman, Ismael Juma, Guozhang Wang
Closes#139 from hachikuji/KAFKA-2388 and squashes the following commits:
377c67e [Jason Gustafson] KAFKA-2388; refactor KafkaConsumer subscribe API
This PR adds StopReplica request and response as it is required by ijuma for KAFKA-2411. Migration of core module is addressed a separate PR (#141).
ijuma Could you review it? gwenshap Could you take a look as well?
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#170 from dajac/KAFKA-2072-part-1
Added MirrorMaker service and a few corresponding sanity checks, as well as necessary config template files. A few additional updates to accomodate the change in wait_until from ducktape0.2.0->0.3.0
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava, Gwen Shapira
Closes#148 from granders/KAFKA-2439 and squashes the following commits:
c7c3ebd [Geoff Anderson] MirrorMaker now can run as multi-node service. Added kill -9 to various clean_node methods.
1e806f2 [Geoff Anderson] Various cleanups per review.
1b4b049 [Geoff Anderson] Added MirrorMaker service and a few corresponding sanity checks, as well as necessary config template files. A few additional updates to accomodate the change in wait_until from ducktape0.2.0->0.3.0
The default is typically `1m` for 64-bit machines and the Scala compiler sometimes needs more than this.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Manikumar Reddy, Gwen Shapira
Closes#157 from ijuma/kafka-2457-stackoverflowerror-during-builds
This extends the original patch done by GZ to provide Console access to both the new and old consumer API's. The code follows a pattern similar to that already used in ConsoleProducer.
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Jason Gustafson, Ismael Juma, Guozhang Wang
Closes#144 from benstopford/KAFKA-2015 and squashes the following commits:
5058a7b [Ben Stopford] Patch for KAFKA-2015: removed unused imports
6e08bf4 [Ben Stopford] Patch for KAFKA-2015: fixed formatting error
739457c [Ben Stopford] Patch for KAFKA-2015: switched to blocking poll + typo + fixed to match style guide
883a626 [Ben Stopford] Patch for KAFKA-2015: incorporating comments to date.
0660629 [Ben Stopford] Merge remote-tracking branch 'upstream/trunk' into KAFKA-2015
e102ff6 [Ben Stopford] KAFKA-2015 - ported patch from jira in and altered to match exising ConsoleProducer template
This bug was introduced while committing KAFKA-2205. Basically, the path for topic overrides was renamed to "topic" from "topics". However, this causes existing topic config overrides to break because they will not be read from ZK anymore since the path is different.
https://reviews.apache.org/r/34554/
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Joel Koshy
Closes#152 from auradkar/2446
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Guozhang
Closes#140 from ewencp/kafka-1782-followup and squashes the following commits:
fe36bd1 [Ewen Cheslack-Postava] Fix missing annotation of Before and After in ControllerFailoverTest.
1dcaf39 [Ewen Cheslack-Postava] KAFKA-1782: Follow up - add missing @Test annotations.
This is an initial patch implementing the basics of Copycat for KIP-26.
The intent here is to start a review of the key pieces of the core API and get a reasonably functional, baseline, non-distributed implementation of Copycat in place to get things rolling. The current patch has a number of known issues that need to be addressed before a final version:
* Some build-related issues. Specifically, requires some locally-installed dependencies (see below), ignores checkstyle for the runtime data library because it's lifted from Avro currently and likely won't last in its current form, and some Gradle task dependencies aren't quite right because I haven't gotten rid of the dependency on `core` (which should now be an easy patch since new consumer groups are in a much better state).
* This patch currently depends on some Confluent trunk code because I prototyped with our Avro serializers w/ schema-registry support. We need to figure out what we want to provide as an example built-in set of serializers. Unlike core Kafka where we could ignore the issue, providing only ByteArray or String serializers, this is pretty central to how Copycat works.
* This patch uses a hacked up version of Avro as its runtime data format. Not sure if we want to go through the entire API discussion just to get some basic code committed, so I filed KAFKA-2367 to handle that separately. The core connector APIs and the runtime data APIs are entirely orthogonal.
* This patch needs some updates to get aligned with recent new consumer changes (specifically, I'm aware of the ConcurrentModificationException issue on exit). More generally, the new consumer is in flux but Copycat depends on it, so there are likely to be some negative interactions.
* The layout feels a bit awkward to me right now because I ported it from a Maven layout. We don't have nearly the same level of granularity in Kafka currently (core and clients, plus the mostly ignored examples, log4j-appender, and a couple of contribs). We might want to reorganize, although keeping data+api separate from runtime and connector plugins is useful for minimizing dependencies.
* There are a variety of other things (e.g., I'm not happy with the exception hierarchy/how they are currently handled, TopicPartition doesn't really need to be duplicated unless we want Copycat entirely isolated from the Kafka APIs, etc), but I expect those we'll cover in the review.
Before commenting on the patch, it's probably worth reviewing https://issues.apache.org/jira/browse/KAFKA-2365 and https://issues.apache.org/jira/browse/KAFKA-2366 to get an idea of what I had in mind for a) what we ultimately want with all the Copycat patches and b) what we aim to cover in this initial patch. My hope is that we can use a WIP patch (after the current obvious deficiencies are addressed) while recognizing that we want to make iterative progress with a bunch of subsequent PRs.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma, Gwen Shapira
Closes#99 from ewencp/copycat and squashes the following commits:
a3a47a6 [Ewen Cheslack-Postava] Simplify Copycat exceptions, make them a subclass of KafkaException.
8c108b0 [Ewen Cheslack-Postava] Rename Coordinator to Herder to avoid confusion with the consumer coordinator.
7bf8075 [Ewen Cheslack-Postava] Make Copycat CLI speific to standalone mode, clean up some config and get rid of config storage in standalone mode.
656a003 [Ewen Cheslack-Postava] Clarify and expand the explanation of the Copycat Coordinator interface.
c0e5fdc [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat
0fa7a36 [Ewen Cheslack-Postava] Mark Copycat classes as unstable and reduce visibility of some classes where possible.
d55d31e [Ewen Cheslack-Postava] Reorganize Copycat code to put it all under one top-level directory.
b29cb2c [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat
d713a21 [Ewen Cheslack-Postava] Address Gwen's review comments.
6787a85 [Ewen Cheslack-Postava] Make Converter generic to match serializers since some serialization formats do not require a base class of Object; update many other classes to have generic key and value class type parameters to match this change.
b194c73 [Ewen Cheslack-Postava] Split Copycat converter option into two options for key and value.
0b5a1a0 [Ewen Cheslack-Postava] Normalize naming to use partition for both source and Kafka, adjusting naming in CopycatRecord classes to clearly differentiate.
e345142 [Ewen Cheslack-Postava] Remove Copycat reflection utils, use existing Utils and ConfigDef functionality from clients package.
be5c387 [Ewen Cheslack-Postava] Minor cleanup
122423e [Ewen Cheslack-Postava] Style cleanup
6ba87de [Ewen Cheslack-Postava] Remove most of the Avro-based mock runtime data API, only preserving enough schema functionality to support basic primitive types for an initial patch.
4674d13 [Ewen Cheslack-Postava] Address review comments, clean up some code styling.
25b5739 [Ewen Cheslack-Postava] Fix sink task offset commit concurrency issue by moving it to the worker thread and waking up the consumer to ensure it exits promptly.
0aefe21 [Ewen Cheslack-Postava] Add log4j settings for Copycat.
220e42d [Ewen Cheslack-Postava] Replace Avro serializer with JSON serializer.
1243a7c [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat
5a618c6 [Ewen Cheslack-Postava] Remove offset serializers, instead reusing the existing serializers and removing schema projection support.
e849e10 [Ewen Cheslack-Postava] Remove duplicated TopicPartition implementation.
dec1379 [Ewen Cheslack-Postava] Switch to using new consumer coordinator instead of manually assigning partitions. Remove dependency of copycat-runtime on core.
4a9b4f3 [Ewen Cheslack-Postava] Add some helpful Copycat-specific build and test targets that cover all Copycat packages.
31cd1ca [Ewen Cheslack-Postava] Add CLI tools for Copycat.
e14942c [Ewen Cheslack-Postava] Add Copycat file connector.
0233456 [Ewen Cheslack-Postava] Add copycat-avro and copycat-runtime
11981d2 [Ewen Cheslack-Postava] Add copycat-data and copycat-api
Added base_box variable to Vagrantfile. This makes it possible to override the base box in Vagrantfile.local.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Guozhang Wang
Closes#137 from granders/minor-expose-vagrant-box and squashes the following commits:
44936f7 [Geoff Anderson] Added base_box variable to Vagrantfile. This makes it possible to override the base box in Vagrantfile.local.
This is a follow up patch for KAFKA-2406. Further test to verify if this change alone is enough to solve the problem or not.
Author: Jiangjie Qin <becket.qin@gmail.com>
Author: Jiangjie Qin <jqin@jqin-ld1.linkedin.biz>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#114 from becketqin/KAFKA-2406
If there is a single commit in the PR, then it's never listed.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang
Closes#136 from ijuma/kafka-2430-optional-listing-commits and squashes the following commits:
64f1aec [Ismael Juma] Listing of PR commits in commit message should be optional
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang
Closes#135 from ewencp/kafka-1782-junit3-misusage and squashes the following commits:
0ae6258 [Ewen Cheslack-Postava] KAFKA-1782: Junit3 Misusage
This also marks the consumer as unstable to show an example of using these annotations.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira
Closes#133 from ewencp/stability-annotations and squashes the following commits:
09c15c3 [Ewen Cheslack-Postava] KAFKA-2429: Add annotations to mark classes as stable/unstable
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma, Ewen Cheslack-Postava and Guozhang Wang
Closes#131 from granthenke/minor-string and squashes the following commits:
3c6250d [Grant Henke] MINOR: Fix hard coded strings in ProduceResponse
Author: flavio junqueira <fpj@apache.org>
Reviewers: Ismael Juma, Guozhang Wang
Closes#102 from fpj/2300 and squashes the following commits:
7bd2edb [flavio junqueira] KAFKA-2300: Removed unnecessary s" occurrences.
aa6ec90 [flavio junqueira] KAFKA-2300: Wrapped all occurences of sendRequestToBrokers with try/catch and fixed string typo.
f1261b1 [flavio junqueira] Fixed some style issues.
9b6390a [flavio junqueira] Updated package name and removed unnecessary imports.
dbd1bf3 [flavio junqueira] KAFKA-2300: Error in controller log when broker tries to rejoin cluster
console consumer writes to System.out, while (some) log4j loggers operate in other threads.
This occasionally led to funky interleaved output which disrupted parsing of consumed messages by ConsoleConsumerService, leading to spurious test failures.
This fix directs log output to a separate file.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava
Closes#123 from granders/KAFKA-2408 and squashes the following commits:
247b0e0 [Geoff Anderson] Updated line counting to use wc -l
66d6f4f [Geoff Anderson] lower -> uperrcase constants
e67f554 [Geoff Anderson] Changed incorrect license header
af67e01 [Geoff Anderson] Merged in upstream trunk
8f89044 [Geoff Anderson] Added another lifecycle check. Wait for log file to exist before exmaning contents.
521a84b [Geoff Anderson] Updated console consumer to directo log output directly to file rather than stdout
Hardcode the key/value serializer to ByteArraySerializer according to Jun’s comments.
Author: Jiangjie Qin <jqin@jqin-ld1.linkedin.biz>
Reviewers: Guozhang Wang
Closes#120 from becketqin/KAFKA-1997 and squashes the following commits:
7f2e5a6 [Jiangjie Qin] KAFKA-1997: Follow-up patch, hardcode key/value serializer in mirror maker to byte serializer.
Ben Stopford
Ismael Juma
Ashish Singh
Joel Koshy
Jiangjie Qin
Parth Brahmbhatt
Geoff Anderson
Stevo Slavic
Magnus Reftel
Dong Lin
Jason Gustafson
Ewen Cheslack-Postava
David Jacot
Gwen Shapira
Aditya Auradkar
Mayuresh Gharat
Manikumar Reddy
Sriharsha Chintalapani
Sriharsha Chintalapani
Alexander Pakulov
Grant Henke
flavio junqueira