src-kafka

Commit Graph

Author	SHA1	Message	Date
flavio junqueira	21443f214f	KAFKA-2641; Upgrade path for ZK authentication This pull request adds a configuration parameter and a migration tool. It is also based on pull request #303, which should go in first. Author: flavio junqueira <fpj@apache.org> Author: Flavio Junqueira <fpj@apache.org> Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com> Closes #313 from fpj/KAFKA-2641	9 years ago
Parth Brahmbhatt	5764e54de1	KAFKA-2212: Authorizer CLI implementation. Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com> Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com> Closes #230 from Parth-Brahmbhatt/KAFKA-2212	9 years ago
Manikumar reddy O	c7bc5366dd	KAFKA-2514: change default JVM options in kafka-run-class.sh. GC is set to G1 collector. Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com> Closes #218 from omkreddy/KAFKA-2514	9 years ago
Guozhang Wang	263c10ab7c	KIP-28: Add a processor client for Kafka Streaming This work has been contributed by Jesse Anderson, Randall Hauch, Yasuhiro Matsuda and Guozhang Wang. The detailed design can be found in https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client. Author: Guozhang Wang <wangguoz@gmail.com> Author: Yasuhiro Matsuda <yasuhiro.matsuda@gmail.com> Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Author: ymatsuda <yasuhiro.matsuda@gmail.com> Author: Randall Hauch <rhauch@gmail.com> Author: Jesse Anderson <jesse@smokinghand.com> Author: Ismael Juma <ismael@juma.me.uk> Author: Jesse Anderson <eljefe6a@gmail.com> Reviewers: Ismael Juma, Randall Hauch, Edward Ribeiro, Gwen Shapira, Jun Rao, Jay Kreps, Yasuhiro Matsuda, Guozhang Wang Closes #130 from guozhangwang/streaming	9 years ago
Ewen Cheslack-Postava	48b4d6938d	KAFKA-2373: Add Kafka-backed offset storage for Copycat. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Gwen Shapira, James Cheng Closes #202 from ewencp/kafka-2373-copycat-distributed-offset	9 years ago
Ewen Cheslack-Postava	492bfdfa87	KAFKA-2367; Add Copycat runtime data API. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Gwen Shapira, Jay Kreps Closes #163 from ewencp/kafka-2367-copycat-runtime-data-api	9 years ago
Ewen Cheslack-Postava	f6acfb0891	KAFKA-2366; Initial patch for Copycat This is an initial patch implementing the basics of Copycat for KIP-26. The intent here is to start a review of the key pieces of the core API and get a reasonably functional, baseline, non-distributed implementation of Copycat in place to get things rolling. The current patch has a number of known issues that need to be addressed before a final version: * Some build-related issues. Specifically, requires some locally-installed dependencies (see below), ignores checkstyle for the runtime data library because it's lifted from Avro currently and likely won't last in its current form, and some Gradle task dependencies aren't quite right because I haven't gotten rid of the dependency on `core` (which should now be an easy patch since new consumer groups are in a much better state). * This patch currently depends on some Confluent trunk code because I prototyped with our Avro serializers w/ schema-registry support. We need to figure out what we want to provide as an example built-in set of serializers. Unlike core Kafka where we could ignore the issue, providing only ByteArray or String serializers, this is pretty central to how Copycat works. * This patch uses a hacked up version of Avro as its runtime data format. Not sure if we want to go through the entire API discussion just to get some basic code committed, so I filed KAFKA-2367 to handle that separately. The core connector APIs and the runtime data APIs are entirely orthogonal. * This patch needs some updates to get aligned with recent new consumer changes (specifically, I'm aware of the ConcurrentModificationException issue on exit). More generally, the new consumer is in flux but Copycat depends on it, so there are likely to be some negative interactions. * The layout feels a bit awkward to me right now because I ported it from a Maven layout. We don't have nearly the same level of granularity in Kafka currently (core and clients, plus the mostly ignored examples, log4j-appender, and a couple of contribs). We might want to reorganize, although keeping data+api separate from runtime and connector plugins is useful for minimizing dependencies. * There are a variety of other things (e.g., I'm not happy with the exception hierarchy/how they are currently handled, TopicPartition doesn't really need to be duplicated unless we want Copycat entirely isolated from the Kafka APIs, etc), but I expect those we'll cover in the review. Before commenting on the patch, it's probably worth reviewing https://issues.apache.org/jira/browse/KAFKA-2365 and https://issues.apache.org/jira/browse/KAFKA-2366 to get an idea of what I had in mind for a) what we ultimately want with all the Copycat patches and b) what we aim to cover in this initial patch. My hope is that we can use a WIP patch (after the current obvious deficiencies are addressed) while recognizing that we want to make iterative progress with a bunch of subsequent PRs. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Ismael Juma, Gwen Shapira Closes #99 from ewencp/copycat and squashes the following commits: `a3a47a6` [Ewen Cheslack-Postava] Simplify Copycat exceptions, make them a subclass of KafkaException. `8c108b0` [Ewen Cheslack-Postava] Rename Coordinator to Herder to avoid confusion with the consumer coordinator. `7bf8075` [Ewen Cheslack-Postava] Make Copycat CLI speific to standalone mode, clean up some config and get rid of config storage in standalone mode. `656a003` [Ewen Cheslack-Postava] Clarify and expand the explanation of the Copycat Coordinator interface. `c0e5fdc` [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat `0fa7a36` [Ewen Cheslack-Postava] Mark Copycat classes as unstable and reduce visibility of some classes where possible. `d55d31e` [Ewen Cheslack-Postava] Reorganize Copycat code to put it all under one top-level directory. `b29cb2c` [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat `d713a21` [Ewen Cheslack-Postava] Address Gwen's review comments. `6787a85` [Ewen Cheslack-Postava] Make Converter generic to match serializers since some serialization formats do not require a base class of Object; update many other classes to have generic key and value class type parameters to match this change. `b194c73` [Ewen Cheslack-Postava] Split Copycat converter option into two options for key and value. `0b5a1a0` [Ewen Cheslack-Postava] Normalize naming to use partition for both source and Kafka, adjusting naming in CopycatRecord classes to clearly differentiate. `e345142` [Ewen Cheslack-Postava] Remove Copycat reflection utils, use existing Utils and ConfigDef functionality from clients package. `be5c387` [Ewen Cheslack-Postava] Minor cleanup `122423e` [Ewen Cheslack-Postava] Style cleanup `6ba87de` [Ewen Cheslack-Postava] Remove most of the Avro-based mock runtime data API, only preserving enough schema functionality to support basic primitive types for an initial patch. `4674d13` [Ewen Cheslack-Postava] Address review comments, clean up some code styling. `25b5739` [Ewen Cheslack-Postava] Fix sink task offset commit concurrency issue by moving it to the worker thread and waking up the consumer to ensure it exits promptly. `0aefe21` [Ewen Cheslack-Postava] Add log4j settings for Copycat. `220e42d` [Ewen Cheslack-Postava] Replace Avro serializer with JSON serializer. `1243a7c` [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat `5a618c6` [Ewen Cheslack-Postava] Remove offset serializers, instead reusing the existing serializers and removing schema projection support. `e849e10` [Ewen Cheslack-Postava] Remove duplicated TopicPartition implementation. `dec1379` [Ewen Cheslack-Postava] Switch to using new consumer coordinator instead of manually assigning partitions. Remove dependency of copycat-runtime on core. `4a9b4f3` [Ewen Cheslack-Postava] Add some helpful Copycat-specific build and test targets that cover all Copycat packages. `31cd1ca` [Ewen Cheslack-Postava] Add CLI tools for Copycat. `e14942c` [Ewen Cheslack-Postava] Add Copycat file connector. `0233456` [Ewen Cheslack-Postava] Add copycat-avro and copycat-runtime `11981d2` [Ewen Cheslack-Postava] Add copycat-data and copycat-api	9 years ago
Grant Henke	fc40016cf3	KAFKA-2407: Only create log directory when it will be used Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes #115 from granthenke/log-fix and squashes the following commits: `de36138` [Grant Henke] Small comment fix `49a8dd4` [Grant Henke] KAFKA-2407: Only create log directory when it will be used	9 years ago
Aditya Auradkar	a56a79055d	kafka-2205; Generalize TopicConfigManager to handle multiple entity configs; patched by Aditya Auradkar; reviewed Jun Rao	9 years ago
Geoff Anderson	e43c9aff92	KAFKA-2276; KIP-25 initial patch Initial patch for KIP-25 Note that to install ducktape, do not use pip to install ducktape. Instead: ``` $ git clone gitgithub.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` Author: Geoff Anderson <geoff@confluent.io> Author: Geoff <granders@gmail.com> Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen, Gwen, Jun, Guozhang Closes #70 from granders/KAFKA-2276 and squashes the following commits: `a62fb6c` [Geoff Anderson] fixed checkstyle errors `a70f0f8` [Geoff Anderson] Merged in upstream trunk. `8b62019` [Geoff Anderson] Merged in upstream trunk. `47b7b64` [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j. `a9e6a14` [Geoff Anderson] Merged in upstream changes `d18db7b` [Geoff Anderson] fixed :rat errors (needed to add licenses) `321fdf8` [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task `795fc75` [Geoff Anderson] Merged in changes from upstream trunk. `1d93f06` [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316 `2ea4e29` [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer `0eb6fdc` [Geoff Anderson] Merged in system-tests `69dd7be` [Geoff Anderson] Merged in trunk `4034dd6` [Geoff Anderson] Merged in upstream trunk `ede6450` [Geoff] Merge pull request #4 from confluentinc/move_muckrake `7751545` [Geoff Anderson] Corrected license headers `e6d532f` [Geoff Anderson] java 7 -> java 6 `8c61e2d` [Geoff Anderson] Reverted jdk back to 6 `f14c507` [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup. `98b7253` [Geoff Anderson] Updated consumer tests to pre-populate kafka logs `e6a41f1` [Geoff Anderson] removed stray println `b15b24f` [Geoff Anderson] leftover KafkaBenchmark in super call `0f75187` [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test `f469f84` [Geoff Anderson] Tweaked readme, added example Vagrantfile.local `3d73857` [Geoff Anderson] Merged downstream changes `42dcdb1` [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast `7f7c3e0` [Geoff Anderson] Updated setup.py for kafkatest `c60125c` [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency `4f476fe` [Geoff Anderson] Moved aws scripts to vagrant directory `5af88fc` [Geoff Anderson] Updated README to include aws quickstart `e5edf03` [Geoff Anderson] Updated example aws Vagrantfile.local `96533c3` [Geoff] Update aws-access-keys-commands `25a413d` [Geoff] Update aws-example-Vagrantfile.local `884b20e` [Geoff Anderson] Moved a bunch of files to kafkatest directory `fc7c81c` [Geoff Anderson] added setup.py `632be12` [Geoff] Merge pull request #3 from confluentinc/verbose-client `51a94fd` [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0. `a80a428` [Geoff Anderson] Added shell program for VerifiableProducer. `d586fb0` [Geoff Anderson] Updated comments to reflect that throttler is not message-specific `6842ed1` [Geoff Anderson] left out a file from last commit `1228eef` [Geoff Anderson] Renamed throttler `9100417` [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. `0a5de8e` [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. `475423b` [Geoff Anderson] Convert class to string before adding to json object. `bc009f2` [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients) `c0526fe` [Geoff Anderson] Updates per review comments. `8b4b1f2` [Geoff Anderson] Minor updates to VerboseProducer `2777712` [Geoff Anderson] Added some metadata to producer output. `da94b8c` [Geoff Anderson] Added number of messages option. `07cd1c6` [Geoff Anderson] Added simple producer which prints status of produced messages to stdout. `a278988` [Geoff Anderson] fixed typos `f1914c3` [Liquan Pei] Merge pull request #2 from confluentinc/system_tests `81e4156` [Liquan Pei] Bootstrap Kafka system tests	9 years ago
Ismael Juma	66c6f9b1c4	kafka-2039; Update Scala to 2.10.5 and 2.11.6; patched by Ismael Juma; reviewed by Jun Rao	10 years ago
Onur Karaman	7c82afddc8	Added script for bin/kafka-consumer-groups.sh that was mistakenly dropped from the original patch for KAFKA-1476	10 years ago
Manikumar Reddy	995d0d3693	KAFKA-1109 Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS; reviewed by Neha Narkhede	10 years ago
Jun Rao	09234f0e52	trivial change to remove -UseCompressedOops option from script	10 years ago
Guozhang Wang	72601f783e	KAFKA-1624; bump up default scala version to 2.11.4 to compile with java 8; reviewed by Joe Stein, Gwen Shapira and Joel Koshy	10 years ago
Jarek Jarcec Cecho	409c367ceb	KAFKA-1654 Provide a way to override server configuration from command line; reviewed by Neha Narkhede	10 years ago
Stevo Slavic	f9d9b38143	kafka-1419 (followup); cross build for scala 2.11; patched by Stevo Slavic; reviewed by Jun Rao	10 years ago
Ivan Lyutov	4d0759713f	kafka-1419; cross build for scala 2.11; patched by Ivan Lyutov; reviewed by Joe Stein and Jun Rao	10 years ago
Jay Kreps	6b0ae4bba0	KAFKA-1291 Add wrapper scripts and usage information to each command.	11 years ago
Sriharsha Chintalapani	1363ed7c52	KAFKA-1438 Migrate client tools out of perf; reviewed by Neha Narkhede	11 years ago
Stevo Slavic	4bd33e5ba7	kafka-1210; Windows Bat files are not working properly; patched by Stevo Slavic; reviewed by Jun Rao	11 years ago
Martin Kleppmann	c3520fe7e0	kafka-server-stop.sh doesn't stop broker; reviewed by Neha Narkhede	11 years ago
Nathan Brown	edbed2823f	kafka-1278; More flexible helper scripts; patched by Nathan Brown; reviewed by Jun Rao	11 years ago
Jun Rao	d392f56b80	trivial change to use the correct path of kafka-clients.jar in kafka-run-class.sh	11 years ago
Joe Stein	a8297cc6ca	KAFKA-1254 remove vestigial sbt patch by Joe Stein; reviewed by Jun Rao	11 years ago
Jay Kreps	6b80dbb97b	KAFKA-1236 Fix various breakages in the perf tests. Make the producer test use either the old or the new producer.	11 years ago
Joe Stein	8014b6ff69	KAFKA-1158 run rat is not needed this is documented now in the release not part of the server running	11 years ago
Jun Rao	52614ad65e	kafka-1171; Gradle build for Kafka; patched by David Arthur, Joe Stein, Chris Freeman and Jun Rao; reviewed by Guozhang Wang, Joel Koshy and Neha Narkhede	11 years ago
Jason Rosenberg	74b42a8f81	KAFKA-1164 kafka should depend on snappy 1.0.5 (instead of 1.0.4.1); reviewed by Neha Narkhede	11 years ago
Philp K. Warren	c9028ad8c5	kafka-1214; Support arguments to zookeeper-shell.sh script; patched by Philp K. Warren; reviewed by Jun Rao	11 years ago
Jun Rao	58789d7cbe	kafka-1127; kafka and zookeeper server should start in daemon mode and log to correct position; patched by Raymond Liu; reviewed by Jun Rao	11 years ago
Jay Kreps	3e7c9c6abf	KAFKA-1081 Clean up shell scripts. Removed re-run of failed command. Use exec instead of subprocess execution.	11 years ago
Jun Rao	ace95c48a9	trivial change to remove unused command bin/kafka-add-partitions.sh	11 years ago
Jun Rao	71ed6ca336	kafka-1073; CheckReassignmentStatus is broken; patched by Jun Rao; reviewed by Guozhang Wang, Swapnil Ghike and Neha Narkhede	11 years ago
Vladislav Pernin	51de7c55d2	kafka-1031; Little modification to the stop script to be able to kill the proper process; patched by Vladislav Pernin; reviewed by Jun Rao	11 years ago
Jay Kreps	14af713252	Misc. minor house-keeping fixes: add reasonable GC settings, fix up README, fix up example configs, fix the logging for tools, use a log directory for logs instead of the root directory.	11 years ago
Sriram Subramanian	3817857b15	kafka-347; change number of partitions of a topic online; patched by Sriram Subramanian; reviewed by Neha Narkehede, Guozhang Wang, Joel Koshy and Jun Rao	11 years ago
Jun Rao	436dd25a48	kafka-259; Give better error message when trying to run shell scripts without having built/downloaded the jars yet; patched by Ashwanth Fernando; reviewed by Jun Rao	12 years ago
Cosmin Lehene	158baf661c	kafka-843; Re-add the release-zip sbt target; patched by Cosmin Lehene; reviewed by Jun Rao	12 years ago
Neha Narkhede	6dbf9212ae	KAFKA-826 Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x; reviewed by Swapnil Ghike, Neha Narkhede, Matt Christiansen, Scott Carey	12 years ago
Jay Kreps	c1ed12e44d	KAFKA-554 Dynamic per-topic configuration. This patch adds a mechanism for storing per-topic configurations in zookeeper and dynamically making config changes across the cluster. Reviewed by Neha and Jun.	12 years ago
Swapnil Ghike	2457bc49ef	KAFKA-513 Add state change log to Kafka brokers; reviewed by Neha Narkhede	12 years ago
Jun Rao	b84168a3e3	Fix windows build script - kafka-run-class.bat; patched by Mark deVilliers; reviewed by Jun Rao; kafka-751	12 years ago
Joe Stein	cb3359f9f1	KAFKA-731 use a variable for the ivy2 path in kafka-run-class.sh patch by Joe Stein reviewed by Neha Narkhede	12 years ago
Joe Stein	aeac593a90	KAFKA-728 snappy jar missing from path in kafka-run-class.sh patch by John Fung reviewed by Joe Stein	12 years ago
John Fung	b5d685bb7c	KAFKA-722 Fix the classpath in kafka-run-class.sh; reviewed by Neha Narkhede	12 years ago
Joe Stein	5e3e181ace	KAFKA-139 cross-compile multiple Scala versions the dependency jars moved and the bin scripts needed to point to the new locations	12 years ago
Derek Chen-Becker	45cb03d456	Re-applied old patch from KAFKA-139 Had to make some additional changes based on further mainline work. Leaving KafkaProject.scala in place as a reference for now.	12 years ago
Jun Rao	e15432efd3	provides windows batch script for starting Kafka/Zookeeper; patched by Antoine Vianey; reviewed by Jun Rao; kafka-581 git-svn-id: https://svn.apache.org/repos/asf/kafka/branches/0.8@1420001 13f79535-47bb-0310-9956-ffa450edef68	12 years ago
Jun Rao	7b2d9c3312	remove ProducerShell; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-638 git-svn-id: https://svn.apache.org/repos/asf/kafka/branches/0.8@1415002 13f79535-47bb-0310-9956-ffa450edef68	12 years ago

1 2 3

132 Commits (ac7695c32a4d8424bafcb8eb5409a4a4ad752c4a)