You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
66 lines
3.0 KiB
66 lines
3.0 KiB
13 years ago
|
# .NET Kafka Client
|
||
|
|
|
||
|
|
This is a .NET implementation of a client for Kafka using C#. It provides for a basic implementation that covers most basic functionalities to include a simple Producer and Consumer.
|
||
|
|
|
||
|
|
The .NET client will wrap Kafka server error codes to the `KafkaException` class. Exceptions are not trapped within the library and basically bubble up directly from the TcpClient and it's underlying Socket connection. Clients using this library should look to do their own exception handling regarding these kinds of errors.
|
||
|
|
|
||
|
|
## Producer
|
||
|
|
|
||
|
|
The Producer can send one or more messages to Kafka in both a synchronous and asynchronous fashion.
|
||
|
|
|
||
|
|
### Producer Usage
|
||
|
|
|
||
|
|
string payload1 = "kafka 1.";
|
||
|
|
byte[] payloadData1 = Encoding.UTF8.GetBytes(payload1);
|
||
|
|
Message msg1 = new Message(payloadData1);
|
||
|
|
|
||
|
|
string payload2 = "kafka 2.";
|
||
|
|
byte[] payloadData2 = Encoding.UTF8.GetBytes(payload2);
|
||
|
|
Message msg2 = new Message(payloadData2);
|
||
|
|
|
||
|
|
Producer producer = new Producer("localhost", 9092);
|
||
|
|
producer.Send("test", 0, new List<Message> { msg1, msg2 });
|
||
|
|
|
||
|
|
### Asynchronous Producer Usage
|
||
|
|
|
||
|
|
List<Message> messages = GetBunchOfMessages();
|
||
|
|
|
||
|
|
Producer producer = new Producer("localhost", 9092);
|
||
|
|
producer.SendAsync("test", 0, messages, (requestContext) => { // doing work });
|
||
|
|
|
||
|
|
### Multi-Producer Usage
|
||
|
|
|
||
|
|
List<ProducerRequest> requests = new List<ProducerRequest>
|
||
|
|
{
|
||
|
|
new ProducerRequest("test a", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("1: " + DateTime.UtcNow)) }),
|
||
|
|
new ProducerRequest("test b", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("2: " + DateTime.UtcNow)) }),
|
||
|
|
new ProducerRequest("test c", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("3: " + DateTime.UtcNow)) }),
|
||
|
|
new ProducerRequest("test d", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("4: " + DateTime.UtcNow)) })
|
||
|
|
};
|
||
|
|
|
||
|
|
MultiProducerRequest request = new MultiProducerRequest(requests);
|
||
|
|
Producer producer = new Producer("localhost", 9092);
|
||
|
|
producer.Send(request);
|
||
|
|
|
||
|
|
## Consumer
|
||
|
|
|
||
|
|
The consumer has several functions of interest: `GetOffsetsBefore` and `Consume`. `GetOffsetsBefore` will retrieve a list of offsets before a given time and `Consume` will attempt to get a list of messages from Kafka given a topic, partition and offset. `Consume` allows for both a single and batched request function using the `MultiFetchRequest`.
|
||
|
|
|
||
|
|
### Consumer Usage
|
||
|
|
|
||
|
|
Consumer consumer = new Consumer("localhost", 9092);
|
||
|
|
int max = 10;
|
||
|
|
long[] offsets = consumer.GetOffsetsBefore("test", 0, OffsetRequest.LatestTime, max);
|
||
|
|
List<Message> messages = consumer.Consume("test", 0, offsets[0]);
|
||
|
|
|
||
|
|
### Consumer Multi-fetch
|
||
|
|
|
||
|
|
Consumer consumer = new Consumer("localhost", 9092);
|
||
|
|
MultiFetchRequest request = new MultiFetchRequest(new List<FetchRequest>
|
||
|
|
{
|
||
|
|
new FetchRequest("testa", 0, 0),
|
||
|
|
new FetchRequest("testb", 0, 0),
|
||
|
|
new FetchRequest("testc", 0, 0)
|
||
|
|
});
|
||
|
|
|
||
|
|
List<List<Message>> messages = consumer.Consume(request);
|