We Keep Coding

Display Kafka messages on web page

I have a Java Spring Application with a Tomcat server that listen on kafka topic. I want to display all messages in a real-time mode on the web page. Therefore, when a kafka messages is arrived in the backend I want to see it on my web page. I don't know a good approach to push kafka message directly to the front-end and display it on web page. Is someone could help my with a solution and some examples that could help? Thanks!

I have implemented a system like this in Java for my last employer, albeit not with Spring/Tomcat. It was consuming messages from Kafka and serving them on a web socket to be displayed in the browser. The approach I followed was to use akka-stream-kafka and akka-http for web-socket support. The benefit of that is both are based on akka-streams which makes it an easy fit for streaming data.
While you can embed akka-http in your spring app running inside tomcat, it may not feel the most natural choice any more as spring framework already has its own support for both kafka and websockets. However, if you're not familiar with either, then jumping on the akka approach may be easiest and the core logic goes along these lines (I can't share the code from work so have just put this together from the examples in the docs, not tested):
public Route createRoute(ActorSystem system) {
return path("ws", () -> {
ConsumerSettings<byte[], String> consumerSettings = ConsumerSettings.create(system, new ByteArrayDeserializer(), new StringDeserializer())
.withBootstrapServers("localhost:9092")
.withGroupId(UUID.randomUUID().toString()) //this is so that each client gets all messages. To be able to resume from where a client left off in case of disconnects, you can generate in on the client side and pass in the request
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
return handleWebSocketMessages(
Flow.fromSinkAndSourceCoupled(
Sink.ignore(),
Consumer.committableSource(consumerSettings, Subscriptions.topics("topic1"))
.map(msg -> TextMessage.create(msg.record().value()))
)
);
}
}
To expose this route you can follow the minimalistic example, the only difference being the route you define needs the ActorSystem:
final Http http = Http.get(system);
final ActorMaterializer materializer = ActorMaterializer.create(system);
final Flow<HttpRequest, HttpResponse, NotUsed> routeFlow = createRoute(system).flow(system, materializer);
final CompletionStage<ServerBinding> binding = http.bindAndHandle(routeFlow,
ConnectHttp.toHost("localhost", 8080), materializer);
Once you have your messages published to the websocket, the front end will code will of course depend on your UI framework of choice, the simplest code to consume ws messages from javascript is:
this.connection = new WebSocket('ws://url-to-your-ws-endpoint');
this.connection.onmessage = evt => {
// display the message
To easily display the message in the UI, you want the format to be something convenient, like JSON. If your Kafka messages are not JSON already, that's where the Deserializers in the first snippet come in, you can convert it to a convenient JSON string in the Deserializer or do it later on in the .map() called on the Source object.
Alternatively, if polling is an option you can also consider using the off-the-shelf Kafka Rest Proxy, then you only need to build the front-end.

how to await subscriptions established?

I have the following js code:
stompClient.subscribe('/topic/clients', function (calResult) {
updateClientsTable(JSON.parse(calResult.body));
});
$.get("/clients", null);
and following server code(last line invokes it):
#GetMapping(value = {"/clients"})
#ResponseBody
public void loadClients() {
brokerMessagingTemplate.convertAndSend("/topic/clients", clientService.getClientList());
}
Sometime front-end misses result of $.get("/clients", null);
As I understand problem: at the moment of result getting on front end, subscriptions is not happens.
if to put $.get("/clients", null); below in the code - all works fine.
Can you explain how to await subscriptions established?

I think it would make more sense to not mix REST requests with this messaging pattern.
Have you considered sending the "updateClients" command through SockJS into an "/apps/updateClients" channel which replies to the "/topic/clients" channel?

As #light_303 already mentioned, mixing HTTP requests with notification mechanism isn't good. You can register moment, when client connects (GET request on /clients), but you can't register when he disconnects.
You should think in one of the next ways. When user subscribes to /topic/clients:
You individually send him response with all client list and then push updates only.
You individually send him current server time or some kind of ID and then push updates only. User uses given time/ID in GET request to /clients and receives full client list on that moment. This option can be good in situation, when you have incremental updates (i. e. adding new elements to list) and otherwise not so good.
Check this question: Sending message to specific user on Spring Websocket.
This is actually ridiculous, how Spring can complicate things. I recommend you to look on another frameworks for real-time web communication, such as Vert.x or Netty and on Go programming language. Use WebSockets or SockJS instead of STOMP. All that technologies can give you more flexible and performant solution in obvious way. Also, check Centrifugo project, maybe it's relevant to your task.

You can use #SubscribeMapping annotation from spring-messaging.
If you have spring-messaging configured as described here and here, the server-side code could look like following:
#Controller
public class MessagingController {
#SubscribeMapping("/clients")
public List<Client> loadClients() {
return clientService.getClientList();
}
}
This way you don't have to call $.get("/clients", null); because JS message handler receives result of loadClients() call right after subscription happens. JS code would look like:
stompClient.subscribe('/topic/clients', function (calResult) {
updateClientsTable(JSON.parse(calResult.body));
});

Poco C++ websockets - how to use in non-blocking manner?

I am using the Poco C++ libraries to setup a websocket server, which clients can connect to and stream some data to their webinterface. So I have a loop which continuously sends data and I also want to listen if the clients closes the connection by using the receiveFrame() function, for the rest, the client is totally passive and doesn't send any data or whatsoever. The problem is that receiveFrame() blocks the connection, which is not what I want. I basically want to check if the client has not yet called the close() javascript function and stop streaming data if it has. I tried using
ws.setBlocking(false);
But now receiveFrame throws an exception every time it is called. I also tried removing receiveFrame entirely, which works if the connection is terminated by closing the browser but if the client calls the function close(), the server still tries to send data to the client. So how can I pull this off? Is there somehow a way to check if there are client frames to be received and if not to just continue?

You can repeatedly call Socket::select() (with timeout) in a separate thread; when you detect a readable socket, call receiveFrame(). In spite of the misleading name, Socket::select() call wraps epoll() or poll() on platforms where those are available.
You can also implement this in somewhat more complicated but perhaps a more elegant fashion with Poco::NotificationQueues, posting a notification every time when a socket is readable and reading data in the handler.

setBlocking() does not do what you would expect it to. Here's a little info on it:
http://www.scottklement.com/rpg/socktut/nonblocking.html
What you probably want to do is use setReceiveTimeout() on your socket to control how long it will wait for before giving you back control. Then test your response and loop everything if needed. The Poco docs have more info on how to use that part of the API. Just look up WebSockets.

AngularJS and WebSockets beyond

I just read this post, and I do understand what the difference is. But still in my head I have the question. Can/Should I use it in the same App/Website? Say I want the AngularJs to fetch content and update my page, connecting to a REST api and all of that top stuff. But on top of that I also want a realtime chat, or to trigger events on other clients when there is an update or a message received.
Does Angular support that? Or I need to use something like Socket.io to trigger those events? Does it make sense to use both?
If someone could help me or point me to some good reading about that if there is a purpose for using both of them together.
Hope I'm clear enough. thank you for any help.

Javascript supports WebSocket, so you don't need an additional client side framework to use it. Please take a look at this $connection service declared in this WebSocket based AngularJS application.
Basically you can listen for messages:
$connection.listen(function (msg) { return msg.type == "CreatedTerminalEvent"; },
function (msg) {
addTerminal(msg);
$scope.$$phase || $scope.$apply();
});
Listen once (great for request/response):
$connection.listenOnce(function (data) {
return data.correlationId && data.correlationId == crrId;
}).then(function (data) {
$rootScope.addAlert({ msg: "Console " + data.terminalType + " created", type: "success" });
});
And send messages:
$connection.send({
type: "TerminalInputRequest",
input: cmd,
terminalId: $scope.terminalId,
correlationId: $connection.nextCorrelationId()
});
Usually, since a WebSocket connection is bidirectional and has a good support, you can also use it for getting data from the server in request/response model. You can have the two models:
Publisher/Subscriber: Where the client declares its interest in some topics and set handlers for messages with that topic, and then the server publish (or push) messages whenever it sees fit.
Request/response: Where the client sends a message with a requestID (or correlationId), and listen for a single response for that requestId.
Still, you can have both if you want, and use REST for getting data, and WebSocket for getting updates.
In server side, you may need to use Socket.io or whatever server side framework in order to have a backend with WebSocket support.

As noted in the answer in your linked post, Angular does not currently have built-in support for Websockets. So, you would need to directly use the Websockets API, or use an additional library like Socket.io.
However, to answer your question of if you should use both a REST api and Websockets in a single Angular application, there is no reason you can't have both standard XmlHttpRequest requests for interacting with a REST api, using $http or another data layer library such as BreezeJS, for certain functionality included in various parts of the application and also use Wesockets for another part (e.g. real time chat).
Angular is designed to assist with handling this type of scenario. A typical solution to would be to create one or more controllers to handle the application functionality and update your page and then creating separate Services or Factories that encapsulate the data management of each of your data end points (i.e. the REST api and the realtime chat server), which are then injected into the Controllers.
There is a great deal of information available on using angular services/factories for managing data connections. If you're looking for a resource to help guide you on how to build an Angular application and where data services would fit in, I would recommend checking out John Papa's AngularJS Styleguide, which includes a section on Data Services.
For more information about factories and services, you can check out AngularJS : When to use service instead of factory

How to call particular node.js method from client side javascript

In my application i have created many methods in node.js file.How can i call the particular method from client side javascript.
Below is my node.js file
exports.method1=function(){
}
exports.method2=function(){
}
exports.method3=function(){
}

Your client should send a message, for example:
socket.emit("callMethod", {"methodName":"method3"});
And in your server:
socket.on("callMethod", function(data) {
if(data["methodName"] == "method3") {
exports.method3();
}
});
You don't call methods directly, you send events/messages.

I would avoid using sockets unless you really need to, from my experience they can be expensive. Sockets are great for intensive applications where a user stays engaged for awhile, otherwise I would suggest using a RESTful setup with javascript and node.js, for example:
http://blog.modulus.io/nodejs-and-express-create-rest-api
this way the socket doesn't always have to be open which causes more overhead anyway. REST will use http requests whereas sockets you will have direct connection via TCP. REST is better if your app won't be constantly engaging a user, but rather have updates here and there.