src-kafka

Commit Graph

Author	SHA1	Message	Date
Gwen Shapira	7915396ee8	KAFKA-2203: Getting Java8 to relax about javadoc and let our build pass This patch is different than the one attached to the JIRA - I'm applying the new javadoc rules to all subprojects while the one in the JIRA applies only to "clients". We need this since Copycat has the same issues. Author: Gwen Shapira <cshapi@gmail.com> Reviewers: Ismael Juma, Guozhang Wang Closes #147 from gwenshap/KAFKA-2203	9 years ago
Manikumar reddy O	4016011457	KAFKA-2425; Copy latest docs to kafka repo docs/ directory This PR copies the latest kafka docs to kafka repo docs directory. Here I have copied 0.8.3/ directory contents from svn website repo to kafka/docs repository. Some questions: This PR contains generated javadocs also. Do we need to copy javadocs here? Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Gwen Shapira, Ismael Juma Closes #171 from omkreddy/KAFKA-2425-MOVE-DOCS-TO-KAFKA-REPO	9 years ago
Guozhang Wang	263c10ab7c	KIP-28: Add a processor client for Kafka Streaming This work has been contributed by Jesse Anderson, Randall Hauch, Yasuhiro Matsuda and Guozhang Wang. The detailed design can be found in https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client. Author: Guozhang Wang <wangguoz@gmail.com> Author: Yasuhiro Matsuda <yasuhiro.matsuda@gmail.com> Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Author: ymatsuda <yasuhiro.matsuda@gmail.com> Author: Randall Hauch <rhauch@gmail.com> Author: Jesse Anderson <jesse@smokinghand.com> Author: Ismael Juma <ismael@juma.me.uk> Author: Jesse Anderson <eljefe6a@gmail.com> Reviewers: Ismael Juma, Randall Hauch, Edward Ribeiro, Gwen Shapira, Jun Rao, Jay Kreps, Yasuhiro Matsuda, Guozhang Wang Closes #130 from guozhangwang/streaming	9 years ago
Stevo Slavic	17f85fcdef	KAFKA-2492; Upgraded zkclient dependency from 0.5 to 0.6 Author: Stevo Slavic <sslavic@gmail.com> Reviewers: Ismael Juma, Ewen Cheslack-Postava, Flavio Junqueira Closes #184 from sslavic/feature/KAFKA-2492	9 years ago
Ewen Cheslack-Postava	492bfdfa87	KAFKA-2367; Add Copycat runtime data API. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Gwen Shapira, Jay Kreps Closes #163 from ewencp/kafka-2367-copycat-runtime-data-api	9 years ago
Ismael Juma	6acd37720d	KAFKA-2457; Fix how the argument is passed to `compileScala` Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Gwen Shapira Closes #159 from ijuma/kafka-2457-fix	9 years ago
Ismael Juma	3285a9c890	KAFKA-2457; StackOverflowError during builds The default is typically `1m` for 64-bit machines and the Scala compiler sometimes needs more than this. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Manikumar Reddy, Gwen Shapira Closes #157 from ijuma/kafka-2457-stackoverflowerror-during-builds	9 years ago
Manikumar Reddy	9a6b573349	KAFKA-1901; Move Kafka version to be generated in code by build (instead of in manifest); reviewed by Ismael Juma, Joel Koshy, Jason Rosenberg	9 years ago
Sriharsha Chintalapani	9e2c683f55	kafka-1690; Add SSL support to Kafka Broker, Producer and Consumer; patched by Sriharsha Chintalapani; reviewed Rajini Sivaram, Joel Koshy, Michael Herstine, Ismael Juma, Dong Lin, Jiangjie Qin and Jun Rao	9 years ago
Sriharsha Chintalapani	786867c2e1	KAFKA-2438; add maxParallelForks to build.gradle to speedup tests. Author: Sriharsha Chintalapani <harsha@hortonworks.com> Reviewers: Gwen Shapira Closes #143 from harshach/KAFKA-2438	9 years ago
Aditya Auradkar	bbb7d97ade	KAFKA-2084; Add per-client-id byte-rate metrics and quota manager; reviewed by Joel Koshy, Dong Lin, Jun Rao and Edward Ribeiro	9 years ago
Ewen Cheslack-Postava	f6acfb0891	KAFKA-2366; Initial patch for Copycat This is an initial patch implementing the basics of Copycat for KIP-26. The intent here is to start a review of the key pieces of the core API and get a reasonably functional, baseline, non-distributed implementation of Copycat in place to get things rolling. The current patch has a number of known issues that need to be addressed before a final version: * Some build-related issues. Specifically, requires some locally-installed dependencies (see below), ignores checkstyle for the runtime data library because it's lifted from Avro currently and likely won't last in its current form, and some Gradle task dependencies aren't quite right because I haven't gotten rid of the dependency on `core` (which should now be an easy patch since new consumer groups are in a much better state). * This patch currently depends on some Confluent trunk code because I prototyped with our Avro serializers w/ schema-registry support. We need to figure out what we want to provide as an example built-in set of serializers. Unlike core Kafka where we could ignore the issue, providing only ByteArray or String serializers, this is pretty central to how Copycat works. * This patch uses a hacked up version of Avro as its runtime data format. Not sure if we want to go through the entire API discussion just to get some basic code committed, so I filed KAFKA-2367 to handle that separately. The core connector APIs and the runtime data APIs are entirely orthogonal. * This patch needs some updates to get aligned with recent new consumer changes (specifically, I'm aware of the ConcurrentModificationException issue on exit). More generally, the new consumer is in flux but Copycat depends on it, so there are likely to be some negative interactions. * The layout feels a bit awkward to me right now because I ported it from a Maven layout. We don't have nearly the same level of granularity in Kafka currently (core and clients, plus the mostly ignored examples, log4j-appender, and a couple of contribs). We might want to reorganize, although keeping data+api separate from runtime and connector plugins is useful for minimizing dependencies. * There are a variety of other things (e.g., I'm not happy with the exception hierarchy/how they are currently handled, TopicPartition doesn't really need to be duplicated unless we want Copycat entirely isolated from the Kafka APIs, etc), but I expect those we'll cover in the review. Before commenting on the patch, it's probably worth reviewing https://issues.apache.org/jira/browse/KAFKA-2365 and https://issues.apache.org/jira/browse/KAFKA-2366 to get an idea of what I had in mind for a) what we ultimately want with all the Copycat patches and b) what we aim to cover in this initial patch. My hope is that we can use a WIP patch (after the current obvious deficiencies are addressed) while recognizing that we want to make iterative progress with a bunch of subsequent PRs. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Ismael Juma, Gwen Shapira Closes #99 from ewencp/copycat and squashes the following commits: `a3a47a6` [Ewen Cheslack-Postava] Simplify Copycat exceptions, make them a subclass of KafkaException. `8c108b0` [Ewen Cheslack-Postava] Rename Coordinator to Herder to avoid confusion with the consumer coordinator. `7bf8075` [Ewen Cheslack-Postava] Make Copycat CLI speific to standalone mode, clean up some config and get rid of config storage in standalone mode. `656a003` [Ewen Cheslack-Postava] Clarify and expand the explanation of the Copycat Coordinator interface. `c0e5fdc` [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat `0fa7a36` [Ewen Cheslack-Postava] Mark Copycat classes as unstable and reduce visibility of some classes where possible. `d55d31e` [Ewen Cheslack-Postava] Reorganize Copycat code to put it all under one top-level directory. `b29cb2c` [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat `d713a21` [Ewen Cheslack-Postava] Address Gwen's review comments. `6787a85` [Ewen Cheslack-Postava] Make Converter generic to match serializers since some serialization formats do not require a base class of Object; update many other classes to have generic key and value class type parameters to match this change. `b194c73` [Ewen Cheslack-Postava] Split Copycat converter option into two options for key and value. `0b5a1a0` [Ewen Cheslack-Postava] Normalize naming to use partition for both source and Kafka, adjusting naming in CopycatRecord classes to clearly differentiate. `e345142` [Ewen Cheslack-Postava] Remove Copycat reflection utils, use existing Utils and ConfigDef functionality from clients package. `be5c387` [Ewen Cheslack-Postava] Minor cleanup `122423e` [Ewen Cheslack-Postava] Style cleanup `6ba87de` [Ewen Cheslack-Postava] Remove most of the Avro-based mock runtime data API, only preserving enough schema functionality to support basic primitive types for an initial patch. `4674d13` [Ewen Cheslack-Postava] Address review comments, clean up some code styling. `25b5739` [Ewen Cheslack-Postava] Fix sink task offset commit concurrency issue by moving it to the worker thread and waking up the consumer to ensure it exits promptly. `0aefe21` [Ewen Cheslack-Postava] Add log4j settings for Copycat. `220e42d` [Ewen Cheslack-Postava] Replace Avro serializer with JSON serializer. `1243a7c` [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat `5a618c6` [Ewen Cheslack-Postava] Remove offset serializers, instead reusing the existing serializers and removing schema projection support. `e849e10` [Ewen Cheslack-Postava] Remove duplicated TopicPartition implementation. `dec1379` [Ewen Cheslack-Postava] Switch to using new consumer coordinator instead of manually assigning partitions. Remove dependency of copycat-runtime on core. `4a9b4f3` [Ewen Cheslack-Postava] Add some helpful Copycat-specific build and test targets that cover all Copycat packages. `31cd1ca` [Ewen Cheslack-Postava] Add CLI tools for Copycat. `e14942c` [Ewen Cheslack-Postava] Add Copycat file connector. `0233456` [Ewen Cheslack-Postava] Add copycat-avro and copycat-runtime `11981d2` [Ewen Cheslack-Postava] Add copycat-data and copycat-api	9 years ago
Geoff Anderson	e43c9aff92	KAFKA-2276; KIP-25 initial patch Initial patch for KIP-25 Note that to install ducktape, do not use pip to install ducktape. Instead: ``` $ git clone gitgithub.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` Author: Geoff Anderson <geoff@confluent.io> Author: Geoff <granders@gmail.com> Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen, Gwen, Jun, Guozhang Closes #70 from granders/KAFKA-2276 and squashes the following commits: `a62fb6c` [Geoff Anderson] fixed checkstyle errors `a70f0f8` [Geoff Anderson] Merged in upstream trunk. `8b62019` [Geoff Anderson] Merged in upstream trunk. `47b7b64` [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j. `a9e6a14` [Geoff Anderson] Merged in upstream changes `d18db7b` [Geoff Anderson] fixed :rat errors (needed to add licenses) `321fdf8` [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task `795fc75` [Geoff Anderson] Merged in changes from upstream trunk. `1d93f06` [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316 `2ea4e29` [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer `0eb6fdc` [Geoff Anderson] Merged in system-tests `69dd7be` [Geoff Anderson] Merged in trunk `4034dd6` [Geoff Anderson] Merged in upstream trunk `ede6450` [Geoff] Merge pull request #4 from confluentinc/move_muckrake `7751545` [Geoff Anderson] Corrected license headers `e6d532f` [Geoff Anderson] java 7 -> java 6 `8c61e2d` [Geoff Anderson] Reverted jdk back to 6 `f14c507` [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup. `98b7253` [Geoff Anderson] Updated consumer tests to pre-populate kafka logs `e6a41f1` [Geoff Anderson] removed stray println `b15b24f` [Geoff Anderson] leftover KafkaBenchmark in super call `0f75187` [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test `f469f84` [Geoff Anderson] Tweaked readme, added example Vagrantfile.local `3d73857` [Geoff Anderson] Merged downstream changes `42dcdb1` [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast `7f7c3e0` [Geoff Anderson] Updated setup.py for kafkatest `c60125c` [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency `4f476fe` [Geoff Anderson] Moved aws scripts to vagrant directory `5af88fc` [Geoff Anderson] Updated README to include aws quickstart `e5edf03` [Geoff Anderson] Updated example aws Vagrantfile.local `96533c3` [Geoff] Update aws-access-keys-commands `25a413d` [Geoff] Update aws-example-Vagrantfile.local `884b20e` [Geoff Anderson] Moved a bunch of files to kafkatest directory `fc7c81c` [Geoff Anderson] added setup.py `632be12` [Geoff] Merge pull request #3 from confluentinc/verbose-client `51a94fd` [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0. `a80a428` [Geoff Anderson] Added shell program for VerifiableProducer. `d586fb0` [Geoff Anderson] Updated comments to reflect that throttler is not message-specific `6842ed1` [Geoff Anderson] left out a file from last commit `1228eef` [Geoff Anderson] Renamed throttler `9100417` [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. `0a5de8e` [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. `475423b` [Geoff Anderson] Convert class to string before adding to json object. `bc009f2` [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients) `c0526fe` [Geoff Anderson] Updates per review comments. `8b4b1f2` [Geoff Anderson] Minor updates to VerboseProducer `2777712` [Geoff Anderson] Added some metadata to producer output. `da94b8c` [Geoff Anderson] Added number of messages option. `07cd1c6` [Geoff Anderson] Added simple producer which prints status of produced messages to stdout. `a278988` [Geoff Anderson] fixed typos `f1914c3` [Liquan Pei] Merge pull request #2 from confluentinc/system_tests `81e4156` [Liquan Pei] Bootstrap Kafka system tests	9 years ago
Ismael Juma	4389a11f68	KAFKA-2321; Introduce CONTRIBUTING.md Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Guozhang Closes #97 from ijuma/kafka-2321 and squashes the following commits: `4834464` [Ismael Juma] KAFKA-2321; Introduce CONTRIBUTING.md	9 years ago
Ismael Juma	fd3a3cd595	KAFKA-2348; Drop support for Scala 2.9 `testAll` passed locally. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Harsha, Ewen, Guozhang Closes #87 from ijuma/kafka-2348-drop-support-for-scala-2.9 and squashes the following commits: `cf9796a` [Ismael Juma] KAFKA-2348; Drop support for Scala 2.9	9 years ago
Ismael Juma	15cba9f00d	KAFKA-2324; Update to Scala 2.11.7 Author: Ismael Juma <ismael@juma.me.uk> Closes #82 from ijuma/kafka-2324 and squashes the following commits: `d71bf5c` [Ismael Juma] KAFKA-2324; Update to Scala 2.11.7	9 years ago
Ismael Juma	458ebeb04b	kafka-2323; Simplify ScalaTest dependency versions; patched by Ismael Juma; reviewed by Jun Rao	9 years ago
Joel Koshy	4aba4bc1df	Trivial commit - explicitly exclude build/** from rat check	9 years ago
Sriharsha Chintalapani	7df39e0394	KAFKA-2316: Drop java 1.6 support; patched by Sriharsha Chintalapani reviewed by Ismael Juma and Gwen Shapira	9 years ago
Joel Koshy	4204f4a06b	Trivial commit - explicitly exclude build/rat-report.xml from rat check	9 years ago
Ashish Singh	2d96da05a0	kafka-2132; Move Log4J appender to a separate module; patched by Ashish Singh; reviewed by Gwen Shapira, Aditya Auradkar and Jun Rao	10 years ago
Ewen Cheslack-Postava	fd612a2d50	kafka-2248; Use Apache Rat to enforce copyright headers; patched by Ewen Cheslack-Postava; reviewed by Gwen Shapira, Joel Joshy and Jun Rao	10 years ago
Ewen Cheslack-Postava	3d8494bca3	KAFKA-2199 Make signing artifacts optional and disabled by default for SNAPSHOTs and allow remote Maven repository configuration from the command line.	10 years ago
Ismael Juma	3836c57865	kafka-2185; Update to Gradle 2.4; patched by Ismael Juma; reviewed by Jun Rao	10 years ago
Ismael Juma	c60f325014	kafka-2189; Snappy compression of message batches less efficient in 0.8.2.1; patched by Ismael Juma; reviewed by Jun Rao	10 years ago
Parth Brahmbhatt	41ba26273b	kafka-2169; Upgrade to zkclient-0.5; patched by Parth Brahmbhatt; reviewed by Jun Rao	10 years ago
Martin Lemanski	22b3db47e7	kafka-1940; Initial checkout and build failing; patched by Martin Lemanski; reviewed by Jun Rao	10 years ago
Neha Narkhede	622e707554	KAFKA-2034 sourceCompatibility not set in Kafka build.gradle; reviewed by Neha Narkhede and Ewen Cheslack-Postava	10 years ago
Ismael Juma	66c6f9b1c4	kafka-2039; Update Scala to 2.10.5 and 2.11.6; patched by Ismael Juma; reviewed by Jun Rao	10 years ago
Jay Kreps	1c6d5bbac6	KAFKA-1915: Add checkstyle for java code.	10 years ago
Jay Kreps	0699ff2ce6	KAFKA-1760: New consumer.	10 years ago
Manikumar Reddy	11ec9bf5a5	KAFKA-1861; Publishing kafka-client:test in order to utilize the helper utils in TestUtils; patched by Manikumar Reddy; reviewed by Jun Rao	10 years ago
jaikiran pai	0b171dee1f	KAFKA-1885 Upgrade junit dependency in core to 4.6 version to allow running individual test methods via gradle command line; reviewed by Neha Narkhede	10 years ago
Jun Rao	5d648cfdb1	KAFKA-1876; pom file for scala 2.11 should reference a specific version; patched by Jun Rao; reviewed by Joe Stein and Sriharsha Chintalapani	10 years ago
Manikumar Reddy	688e38ce45	KAFKA-1723; make the metrics name in new producer more standard; patched by Manikumar Reddy; reviewed by Jay Kreps and Jun Rao	10 years ago
Jun Rao	50b734690a	kafka-1797; (follow-up patch) add the serializer/deserializer api to the new java client; patched by Jun Rao; reviewed by Jay Kreps	10 years ago
Guozhang Wang	72601f783e	KAFKA-1624; bump up default scala version to 2.11.4 to compile with java 8; reviewed by Joe Stein, Gwen Shapira and Joel Koshy	10 years ago
Vladimir Tretyakov	457744a820	kafka-1481; Stop using dashes AND underscores as separators in MBean names; patched by Vladimir Tretyakov; reviewed by Joel Koshy and Jun Rao	10 years ago
Jun Rao	6f4b8a1011	kafka-17693; javadoc should only include client facing packages; patched by Jun Rao; reviewed by Neha Narkhede	10 years ago
Ewen Cheslack-Postava	26ad54c109	kafka-1721; Snappy compressor is not thread safe; patched by Ewen Cheslack-Postava; reviewed by Jun Rao	10 years ago
Jun Rao	2d322a7bd1	kafka-1717; remove netty dependency through ZK 3.4.x; patched by Jun Rao; reviewed by Sriharsha Chintalapani and Neha Narkhede	10 years ago
Ivan Lyutov	5f1ad107b7	kafka-1671; uploaded archives are missing for Scala version 2.11; patched by Ivan Lyutov; reviewed by Jun Rao	10 years ago
Joe Stein	084566b837	kafka-1645; some more jars in our src release; patched by Joe Stein; reviewed by Jun Rao	10 years ago
Joe Stein	d2d1ef357b	KAFKA-1490 remove gradlew initial setup output from source distribution patch by Ivan Lyutov reviewed by Joe Stein	10 years ago
Ivan Lyutov	6d7057566f	kafka-1622; project shouldn't require signing to build; patched by Ivan Lyutov; reviewed by Jun Rao	10 years ago
Roger Hoover	f5ab8e1780	kafka-1369; snappy version update 1.1.x; patched by Roger Hoover; reviewed by Jun Rao	10 years ago
Jun Rao	8d0ab1de70	kafka-1619; perf dir can be removed; patched by Jun Rao; reviewed by Guozhang Wang, Neha Narkhede and Stevo Slavic	10 years ago
Joe Stein	3abe3403a1	KAFKA-1502 source jar is empty patch by Jun Rao reviewed by Joe Stein	10 years ago
Joe Stein	2cdc83e85b	KAFKA-1485 Upgrade to Zookeeper 3.4.6 and create shim for ZKCLI so system tests can run patch by Gwen Shapira reviewed by Joe Stein and Jun Rao	10 years ago
Joe Stein	48cc2dc0ea	Revert KAFKA-1485 Upgrade to Zookeeper 3.4.6 since system tests are failing	10 years ago

1 2 3 4

167 Commits (a6799f4e14ac68a5915bce50f37343bec45c988a)