src-kafka

Commit Graph

Author	SHA1	Message	Date
Kamal C	43fb2df7a4	MINOR: Map `mkString` format updated to default java format This is a minor change but it helps to improve the log readability. Author: Kamal C <kamal.chandraprakash@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2709 from Kamal15/util	8 years ago
Edoardo Comar	c808e8955f	MINOR: FetchRequest.Builder maxBytes for version <3 The maxBytes field should be set to DEFAULT_RESPONSE_MAX_BYTES, the same way as the constructor using the Struct does. codeveloped with mimaison Author: Edoardo Comar <ecomar@uk.ibm.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2694 from edoardocomar/MINOR-FetchRequest	8 years ago
Eno Thereska	84a14fec29	KAFKA-4843: More efficient round-robin scheduler - Improves streams efficiency by more than 200K requests/second (small 100 byte requests) - Gets streams efficiency very close to pure consumer (see results in https://jenkins.confluent.io/job/system-test-kafka-branch-builder/746/console) - Maintains same fairness across tasks - Schedules all records in the queue in-between poll() calls, not just one per task. Author: Eno Thereska <eno@confluent.io> Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Damian Guy, Matthias J. Sax, Guozhang Wang Closes #2643 from enothereska/minor-schedule-round-robin	8 years ago
Ismael Juma	6feaa8a581	KAFKA-1449; Use CRC32C for checksum of V2 message format I manually tested that Crc32CTest and AbstractChecksums pass with JDK 9. I also verified that `Java9ChecksumFactory` is used in that case. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #2739 from ijuma/kafka-1449-crc32c	8 years ago
Colin P. Mccabe	aea5989d98	KAFKA-4944; Fix an "unread field" findbugs warning in streams examples Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Michael G. Noll <michael@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #2727 from cmccabe/KAFKA-4944	8 years ago
Colin P. Mccabe	7adf1e4148	KAFKA-4945; Suppress findbugs warnings about machine-generated code in jmh-benchmarks Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2729 from cmccabe/KAFKA-4945	8 years ago
Colin P. Mccabe	42284960da	KAFKA-4903; Remove dead code in Shell Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2692 from cmccabe/KAFKA-4903	8 years ago
Ismael Juma	4ce65d65df	KAFKA-4574; Ignore test_zk_security_upgrade until KIP-101 lands The transient failures make it harder to spot real failures and we can live without what is being tested (adding security to ZK via a rolling upgrade) until KIP-101 lands. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Apurva Mehta <apurva@confluent.io>, Jun Rao <junrao@gmail.com> Closes #2742 from ijuma/disable-zk-upgrade-test	8 years ago
Jason Gustafson	a0b8e435c9	MINOR: Support streaming decompression of fetched records for new format Author: Jason Gustafson <jason@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #2738 from hachikuji/streaming-compressed-iterator	8 years ago
Onur Karaman	edb372dcaf	KAFKA-4959; Remove controller concurrent access to non-threadsafe NetworkClient, Selector, and SSLEngine This brought down a cluster by causing continuous controller moves. ZkClient's ZkEventThread and a RequestSendThread can concurrently use objects that aren't thread-safe: * Selector * NetworkClient * SSLEngine (this was the big one for us. We turn on SSL for interbroker communication). As per the "Concurrency Notes" section from https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLEngine.html: > two threads must not attempt to call the same method (either wrap() or unwrap()) concurrently SSLEngine.wrap gets called in: * SslTransportLayer.write * SslTransportLayer.handshake * SslTransportLayer.close It turns out that the ZkEventThread and RequestSendThread can concurrently call SSLEngine.wrap: * ZkEventThread calls SslTransportLayer.close from ControllerChannelManager.removeExistingBroker * RequestSendThread can call SslTransportLayer.write or SslTransportLayer.handshake from NetworkClient.poll Suppose the controller moves for whatever reason. The former controller could have had a RequestSendThread who was in the middle of sending out messages to the cluster while the ZkEventThread began executing KafkaController.onControllerResignation, which calls ControllerChannelManager.shutdown, which sequentially cleans up the controller-to-broker queue and connection for every broker in the cluster. This cleanup includes the call to ControllerChannelManager.removeExistingBroker as mentioned earlier, causing the concurrent call to SSLEngine.wrap. This concurrent call throws a BufferOverflowException which ControllerChannelManager.removeExistingBroker catches so the ControllerChannelManager.shutdown moves onto cleaning up the next controller-to-broker queue and connection, skipping the cleanup steps such as clearing the queue, stopping the RequestSendThread, and removing the entry from its brokerStateInfo map. By failing out of the Selector.close, the sensors corresponding to the broker connection has not been cleaned up. Any later attempt at initializing an identical Selector will result in a sensor collision and therefore cause Selector initialization to throw an exception. In other words, any later attempts by this broker to become controller again will fail on initialization. When controller initialization fails, the controller deletes the /controller znode and lets another broker take over. Now suppose the controller moves enough times such that every broker hits the BufferOverflowException concurrency issue. We're now guaranteed to fail controller initialization due to the sensor collision on every controller transition, so the controller will move across brokers continuously. This patch avoids the concurrent use of non-threadsafe classes in ControllerChannelManager.removeExistingBroker by shutting down the RequestSendThread before closing the NetworkClient. Author: Onur Karaman <okaraman@linkedin.com> Reviewers: Joel Koshy <jjkoshy.w@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #2746 from onurkaraman/KAFKA-4959	8 years ago
Jiangjie Qin	23a0f09863	Minor: Remove the accidentally checked in file which broke checkStyle.	8 years ago
Dong Lin	8b05ad406d	KAFKA-4586; Add purgeDataBefore() API (KIP-107) Author: Dong Lin <lindong28@gmail.com> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com> Closes #2476 from lindong28/KAFKA-4586	8 years ago
Armin Braun	f3f9a9eafb	KAFKA-4569; Check for wakeup on every call to KafkaConsumer.poll Author: Armin Braun <me@obrown.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io> Closes #2699 from original-brownbear/KAFKA-4569	8 years ago
Damian Guy	1abed91bd2	KAFKA-4881: add internal leave.group.on.close config to consumer Author: Damian Guy <damian.guy@gmail.com> Reviewers: Ismael Juma, Guozhang Wang Closes #2650 from dguy/consumer-leave-group-config	8 years ago
Ismael Juma	d27e09e60c	MINOR: Use method handles instead of reflection for creating Snappy and LZ4 streams 1. Use Initialization-on-demand holder idiom that relies on JVM lazy-loading instead of explicit initialization check. 2. Method handles were designed to be faster than Core Reflection, particularly if the method handle can be stored in a static final field (the JVM can then optimise the call as if it was a regular method call). Since the code is of similar complexity (and simpler if we consider the whole PR), I am treating this as a clean-up instead of a performance improvement (which would require doing benchmarks). 3. Remove unused `ByteBufferReceive`. 4. I removed the snappy library from the classpath and verified that `CompressionTypeTest` (which uses LZ4) still passes. This shows that the right level of laziness is achieved even if we use one of the lazily loaded compression algorithms. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #2740 from ijuma/use-method-handles-for-compressed-stream-supplier	8 years ago
Ismael Juma	d348ac92c8	MINOR: Fix deserialization of abortedTransactions and lastStableOffset in FetchResponse Thanks to Dong Lin for finding the lastStableOffset issue. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Dong Lin <lindong28@gmail.com>, Jason Gustafson <jason@confluent.io> Closes #2737 from ijuma/fix-fetch-response-lso	8 years ago
Jason Gustafson	462767660b	HOTFIX: Fix unsafe dependence on class name in VerifiableClientJava Author: Jason Gustafson <jason@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2736 from hachikuji/hotfix-verifiable-clients	8 years ago
Jason Gustafson	5bd06f1d54	KAFKA-4816; Message format changes for idempotent/transactional producer (KIP-98) Author: Jason Gustafson <jason@confluent.io> Reviewers: Jun Rao <junrao@gmail.com>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #2614 from hachikuji/exactly-once-message-format	8 years ago
Damian Guy	f615c9e9d2	MINOR: only log first exception in RecordCollectorImpl producer callback Author: Damian Guy <damian.guy@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #2726 from dguy/producer-send-exception	8 years ago
Damian Guy	2269eed242	KAFKA-4919: Document that stores must not be closed when Processors are closed Author: Damian Guy <damian.guy@gmail.com> Reviewers: Michael G. Noll, Eno Thereska, Matthias J. Sax, Elias Levy, Guozhang Wang Closes #2725 from dguy/minor-processor-java-doc	8 years ago
Gwen Shapira	95ef40dd31	MINOR: Adding example to SMT documentation Author: Gwen Shapira <cshapi@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2721 from gwenshap/improve_smt_docs	8 years ago
Guozhang Wang	57278aa82d	MINOR: Improvements on Streams log4j 1. add thread id as prefix in state directory classes; also added logs for lock activities. 2. add logging for task creation / suspension. 3. add more information in rebalance listener logging. 4. add restored number of records into changlog reader. Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Eno Thereska, Damian Guy, Ewen Cheslack-Postava Closes #2702 from guozhangwang/KMinor-streams-task-creation-log4j-improvements	8 years ago
Bruce Szalwinski	763ea5aad1	KAFKA-4929: Transformation Key/Value type references should be to class name(), not canonicalName() Changing getCanonicalName() references to getName() so that docs update with "$" instead of ".". Also added a connect-plugin-discovery.sh CLI to list all of the transformations available. Author: Bruce Szalwinski <bruce.szalwinski@cdk.com> Reviewers: Gwen Shapira Closes #2720 from bruce-szalwinski/transforms and squashes the following commits: `ec3b5b9` [Bruce Szalwinski] remove connect-plugin-discovery. will submit in a different PR `eba0af7` [Bruce Szalwinski] Key / Value transformations are static nested classes and so are referenced using OuterClass$Key and OuterClass$Value.	8 years ago
Michael G. Noll	666beecec8	MINOR: MyProcessor doc example should implement, not extend `Processor` Author: Michael G. Noll <michael@confluent.io> Reviewers: Eno Thereska, Guozhang Wang Closes #2723 from miguno/trunk-streams-docs-typo-fix	8 years ago
Magnus Edenhill	d5cec01f2a	MINOR: Pluggable verifiable clients This adds support for pluggable VerifiableConsumer and VerifiableProducers test client implementations allowing third-party clients to be used in-place of the Java client in kafkatests. A new VerifiableClientMixin class is added and the standard Java Verifiable{Producer,Consumer} classes have been changed to use it. While third-party client drivers may be implemented with a complete class based on the Mixin, a simpler alternative which requries no kafkatest class implementation is available through the VerifiableClientApp class that uses ducktape's global param to specify the client app to use (passed to ducktape through the `--globals <json>` command line argument). Some existing kafkatest clients for reference: Go: https://github.com/confluentinc/confluent-kafka-go/tree/master/kafkatest Python: https://github.com/confluentinc/confluent-kafka-python/tree/master/confluent_kafka/kafkatest C++: https://github.com/edenhill/librdkafka/blob/0.9.2.x/examples/kafkatest_verifiable_client.cpp C#/.NET: https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/test/Confluent.Kafka.VerifiableClient This PR also contains documentation for the simplex JSON-based verifiable\* client protocol. There are also some minor improvements that makes troubleshooting failing client tests easier. Author: Magnus Edenhill <magnus@edenhill.se> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2048 from edenhill/pluggable_verifiable_clients	8 years ago
Matthias J. Sax	83824089f2	MINOR: improve hanging ResetIntegrationTest Sometimes `ResetIntegrationTest` hangs and thus the build times out. We suspect, that this happens if no data is written into the input topics. Right now, input data is written once and reused for both test cases. If for some reason, the broker gets recreated (between both test cases), no data will be available for the second test method and thus the test hangs. This change ensures, that input data is written for each test case individually. Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Ismael Juma, Eno Thereska, Guozhang Wang Closes #2630 from mjsax/minor-reset-integration-test	8 years ago
Damian Guy	b3beaebd44	MINOR: log start and end offsets for state store restoration Debug loggin of the start and end offsets used during state store restoration Author: Damian Guy <damian.guy@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #2718 from dguy/log-restore-offsets	8 years ago
Colin P. Mccabe	a7e3679d22	KAFKA-4924: Fix Kafka Connect API findbugs warnings Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2715 from cmccabe/KAFKA-4924	8 years ago
Matthias J. Sax	6f7780cf12	MINOR: updated incorrect JavaDocs for joins Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy, Eno Thereska, Guozhang Wang Closes #2670 from mjsax/fixJavaDocsRepartitioning	8 years ago
Damian Guy	c2653754d9	MINOR: remove unused log field from KStreamTransformValuesProcessor remove unused log field from KStreamTransformValuesProcessor Author: Damian Guy <damian.guy@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #2717 from dguy/remove-unused-log-para	8 years ago
Damian Guy	c74eec96c5	KAFKA-4863 Follow-up: Querying window store may return unwanted keys iterate over all keys returned from the rocksdb iterator so we don't miss any results Author: Damian Guy <damian.guy@gmail.com> Reviewers: Xavier Léauté, Guozhang Wang Closes #2713 from dguy/window-iter	8 years ago
Damian Guy	fef7fca2af	KAFKA-4594; Annotate integration tests and provide gradle build targets to run subsets of tests This uses JUnit Categories to identify integration tests. Adds 2 new build targets: `integrationTest` and `unitTest`. Author: Damian Guy <damian.guy@gmail.com> Reviewers: Eno Thereska <eno@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #2695 from dguy/junit-categories	8 years ago
Ryan P	05690f0c85	MINOR: Fix zookeeper-security-migration documentation example Incorrect option in example https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ZkSecurityMigrator.scala#L71 Author: Ryan P <ryan.n.pridgeon@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2714 from rnpridgeon/patch-1	8 years ago
Sachin Mittal	197a5d5a6d	KAFKA-4848: Fix retryWithBackoff deadlock issue Fixes related to handling of MAX_POLL_INTERVAL_MS_CONFIG during deadlock and CommitFailedException on partition revoked. Author: Sachin Mittal <sjmittal@gmail.com> Reviewers: Matthias J. Sax, Damian Guy, Guozhang Wang Closes #2642 from sjmittal/trunk	8 years ago
Colin P. Mccabe	5a2fcdd6d4	KAFKA-4894; Fix findbugs "default character set in use" warnings Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Jun Rao <junrao@gmail.com> Closes #2683 from cmccabe/KAFKA-4894	8 years ago
Michael G. Noll	783900c259	MINOR: Guard against NPE when throwing StreamsException on serializer mismatch Author: Michael G. Noll <michael@confluent.io> Reviewers: Damian Guy, Guozhang Wang Closes #2696 from miguno/trunk-sinknode-NPE	8 years ago
Nikki Thean	9e787716b0	KAFKA-4607: Validate the names of auto-generated internal topics I considered catching errors to add further information about naming internal state stores. However, Topic.validate() will throw an error that prints the offending name, so I decided not to add too much complexity. Author: Nikki Thean <nthean@etsy.com> Reviewers: Matthias J. Sax, Guozhang Wang, Eno Thereska, Damian Guy, Ismael Juma Closes #2331 from nixsticks/internal-topics	8 years ago
Colin P. Mccabe	ec8feb766f	KAFKA-4895; Fix findbugs "format string should use %n rather than \n" in tools Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #2684 from cmccabe/KAFKA-4895	8 years ago
Kamal C	01768e3321	MINOR: Fix non-null check in the constructor of ListenerName Author: Kamal C <kamal.chandraprakash@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2700 from Kamal15/precondition	8 years ago
Ben Stopford	6b11fb7af8	MINOR: Increase Throttle lower bound assertion in throttling_test Improves the reliability of this test by decreasing the lower bound (this is to be expected as throttling takes the first fetch to stabilise and will "over-fetch" for this first request) Author: Ben Stopford <benstopford@gmail.com> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #2698 from benstopford/throttling-test-fix	8 years ago
Eno Thereska	c5adf17aea	MINOR: Log error if offsets topic creation fails This problem is hard to debug otherwise as there error returned to the client (“Coordinator not available”) is not very informative. Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #2652 from enothereska/minor-warning-enforcement-offset	8 years ago
Guozhang Wang	cfda0368db	KAFKA-4859: Set shorter commit interval for integration tests with caching Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Armin Braun, Damian Guy, Jason Gustafson Closes #2682 from guozhangwang/K4859-cache-commit-interval	8 years ago
Guozhang Wang	b8fe2bb56c	KAFKA-4885: Add client.close as exception handler in streams system tests Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Eno Thereska, Damian Guy, Jason Gustafson Closes #2693 from guozhangwang/K4885-system-test-unexpected-exception-handler	8 years ago
Ismael Juma	1659ca1773	KAFKA-4901; Make ProduceRequest thread-safe If request logging is enabled, `ProduceRequest` can be accessed and mutated concurrently from a network thread (which calls `toString`) and a request handler thread (which calls `clearPartitionRecords()`). That can lead to a `ConcurrentModificationException` when iterating the `partitionRecords` map. The underlying thread-safety issue has existed since the server started using the Java implementation of ProduceRequest in 0.10.0. However, we were incorrectly not clearing the underlying struct until 0.10.2, so `toString` itself was thread-safe until that change. In 0.10.2, `toString` is no longer thread-safe and we could potentially see a `NullPointerException` given the right set of interleavings between `toString` and `clearPartitionRecords` although we haven't seen that happen yet. In trunk, we changed the requests to have a `toStruct` method instead of creating a struct in the constructor and `toString` was no longer printing the contents of the `Struct`. This accidentally fixed the race condition, but it meant that request logging was less useful. A couple of days ago, `AbstractRequest.toString` was changed to print the contents of the request by calling `toStruct().toString()` and reintroduced the race condition. The impact is more visible because we iterate over a `HashMap`, which proactively checks for concurrent modification (unlike arrays). We will need a separate PR for 0.10.2. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Onur Karaman <okaraman@linkedin.com>, Jun Rao <junrao@gmail.com> Closes #2689 from ijuma/produce-request-thread-safety	8 years ago
Vahid Hashemian	be11272818	MINOR: Fix a documentation typo Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com> Closes #2674 from vahidhashemian/minor/fix_typos_1703	8 years ago
Guozhang Wang	b9f812491f	MINOR: Improve log4j on stream thread and stream process Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Matthias J. Sax <matthias@confluent.io>, Eno Thereska <eno@confluent.io>, Damian Guy <damian.guy@gmail.com>, Jason Gustafson <jason@confluent.io> Closes #2685 from guozhangwang/KMinor-improve-log4j	8 years ago
Ismael Juma	962c378cc6	HOTFIX: Fix header in ByteArrayConverter Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com> Closes #2690 from ijuma/fix-header-in-byte-array-converter	8 years ago
Ewen Cheslack-Postava	52a15d7c0b	KAFKA-4783: Add ByteArrayConverter (KIP-128) Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #2599 from ewencp/kafka-4783-byte-array-converter	8 years ago
Damian Guy	9e4548df30	KAFKA-4863; Querying window store may return unwanted keys Make sure that the iterator returned from `WindowStore.fetch(..)` only returns matching keys, rather than all keys that are a prefix match. Author: Damian Guy <damian.guy@gmail.com> Reviewers: Eno Thereska, Guozhang Wang Closes #2662 from dguy/kafka-4863	8 years ago
Guozhang Wang	d3b8ff0243	HOTFIX: Fix apache headers in float serde class files hachikuji Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Jason Gustafson <jason@confluent.io> Closes #2679 from guozhangwang/KHotfix-serde-headers	8 years ago

1 2 3 4 5 ...

3363 Commits (43fb2df7a4b2bc7637dcba9436a5435cdcb4fb27) All Branches Search

3363 Commits (43fb2df7a4b2bc7637dcba9436a5435cdcb4fb27)

All Branches