I have a Java Spring Application with a Tomcat server that listen on kafka topic. I want to display all messages in a real-time mode on the web page. Therefore, when a kafka messages is arrived in the backend I want to see it on my web page. I don't know a good approach to push kafka message directly to the front-end and display it on web page. Is someone could help my with a solution and some examples that could help? Thanks!
I have implemented a system like this in Java for my last employer, albeit not with Spring/Tomcat. It was consuming messages from Kafka and serving them on a web socket to be displayed in the browser. The approach I followed was to use akka-stream-kafka and akka-http for web-socket support. The benefit of that is both are based on akka-streams which makes it an easy fit for streaming data.
While you can embed akka-http in your spring app running inside tomcat, it may not feel the most natural choice any more as spring framework already has its own support for both kafka and websockets. However, if you're not familiar with either, then jumping on the akka approach may be easiest and the core logic goes along these lines (I can't share the code from work so have just put this together from the examples in the docs, not tested):
public Route createRoute(ActorSystem system) {
return path("ws", () -> {
ConsumerSettings<byte[], String> consumerSettings = ConsumerSettings.create(system, new ByteArrayDeserializer(), new StringDeserializer())
.withBootstrapServers("localhost:9092")
.withGroupId(UUID.randomUUID().toString()) //this is so that each client gets all messages. To be able to resume from where a client left off in case of disconnects, you can generate in on the client side and pass in the request
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
return handleWebSocketMessages(
Flow.fromSinkAndSourceCoupled(
Sink.ignore(),
Consumer.committableSource(consumerSettings, Subscriptions.topics("topic1"))
.map(msg -> TextMessage.create(msg.record().value()))
)
);
}
}
To expose this route you can follow the minimalistic example, the only difference being the route you define needs the ActorSystem:
final Http http = Http.get(system);
final ActorMaterializer materializer = ActorMaterializer.create(system);
final Flow<HttpRequest, HttpResponse, NotUsed> routeFlow = createRoute(system).flow(system, materializer);
final CompletionStage<ServerBinding> binding = http.bindAndHandle(routeFlow,
ConnectHttp.toHost("localhost", 8080), materializer);
Once you have your messages published to the websocket, the front end will code will of course depend on your UI framework of choice, the simplest code to consume ws messages from javascript is:
this.connection = new WebSocket('ws://url-to-your-ws-endpoint');
this.connection.onmessage = evt => {
// display the message
To easily display the message in the UI, you want the format to be something convenient, like JSON. If your Kafka messages are not JSON already, that's where the Deserializers in the first snippet come in, you can convert it to a convenient JSON string in the Deserializer or do it later on in the .map() called on the Source object.
Alternatively, if polling is an option you can also consider using the off-the-shelf Kafka Rest Proxy, then you only need to build the front-end.
Related
There is a very good utility called ttyd, which allows you to run a console application on your computer and display this console in the browser.
After startup, the utility starts an http web server on the specified port and when accessing localhost, a website with a web application that connects using web sockets to localhost:<port>/ws, and already with the help of them there is communication between the web application and the ttyd agent running on the computer.
I want to implement a client for ttyd in c#. I studied with the help of chrome tools what data the web application sends before receiving the data output to the console. This is just a string: {"authToken":"","columns":211,"rows":46} and tried to repeat the same actions in the c# client. But for some reason, no data from ttyd is returned to me.
Comparing the data output by ttyd to its console in the OS itself, it can be seen that it does not even create a process when accessing from my client.
Here is the code I use with the Websocket.Client package
var exitEvent = new ManualResetEvent(false);
var url = new Uri("ws://localhost:7681/ws");
using (var client = new WebsocketClient(url))
{
client.ReconnectTimeout = TimeSpan.FromSeconds(30);
client.ReconnectionHappened.Subscribe(info =>
Console.WriteLine($"Reconnection happened, type: {info.Type}"));
client.MessageReceived.Subscribe(msg => Console.WriteLine($"Message received: {msg}"));
client.Start();
Task.Run(() => client.Send("{\"AuthToken\":\"\",\"columns\":211,\"rows\":46}"));
exitEvent.WaitOne();
}
I have absolutely no idea how to get ttyd to send data to my client. Do you have any idea what action the browser is doing I'm missing in my c# client?
I tried different libraries for web sockets in c#, and also used postman with copying all the headers that the original web application sends to the ttyd agent, but this does not change anything. That is, ttyd, something is fundamentally interfering, as if my web client is not doing something that the browser is doing.
I have the following js code:
stompClient.subscribe('/topic/clients', function (calResult) {
updateClientsTable(JSON.parse(calResult.body));
});
$.get("/clients", null);
and following server code(last line invokes it):
#GetMapping(value = {"/clients"})
#ResponseBody
public void loadClients() {
brokerMessagingTemplate.convertAndSend("/topic/clients", clientService.getClientList());
}
Sometime front-end misses result of $.get("/clients", null);
As I understand problem: at the moment of result getting on front end, subscriptions is not happens.
if to put $.get("/clients", null); below in the code - all works fine.
Can you explain how to await subscriptions established?
I think it would make more sense to not mix REST requests with this messaging pattern.
Have you considered sending the "updateClients" command through SockJS into an "/apps/updateClients" channel which replies to the "/topic/clients" channel?
As #light_303 already mentioned, mixing HTTP requests with notification mechanism isn't good. You can register moment, when client connects (GET request on /clients), but you can't register when he disconnects.
You should think in one of the next ways. When user subscribes to /topic/clients:
You individually send him response with all client list and then push updates only.
You individually send him current server time or some kind of ID and then push updates only. User uses given time/ID in GET request to /clients and receives full client list on that moment. This option can be good in situation, when you have incremental updates (i. e. adding new elements to list) and otherwise not so good.
Check this question: Sending message to specific user on Spring Websocket.
This is actually ridiculous, how Spring can complicate things. I recommend you to look on another frameworks for real-time web communication, such as Vert.x or Netty and on Go programming language. Use WebSockets or SockJS instead of STOMP. All that technologies can give you more flexible and performant solution in obvious way. Also, check Centrifugo project, maybe it's relevant to your task.
You can use #SubscribeMapping annotation from spring-messaging.
If you have spring-messaging configured as described here and here, the server-side code could look like following:
#Controller
public class MessagingController {
#SubscribeMapping("/clients")
public List<Client> loadClients() {
return clientService.getClientList();
}
}
This way you don't have to call $.get("/clients", null); because JS message handler receives result of loadClients() call right after subscription happens. JS code would look like:
stompClient.subscribe('/topic/clients', function (calResult) {
updateClientsTable(JSON.parse(calResult.body));
});
I just read this post, and I do understand what the difference is. But still in my head I have the question. Can/Should I use it in the same App/Website? Say I want the AngularJs to fetch content and update my page, connecting to a REST api and all of that top stuff. But on top of that I also want a realtime chat, or to trigger events on other clients when there is an update or a message received.
Does Angular support that? Or I need to use something like Socket.io to trigger those events? Does it make sense to use both?
If someone could help me or point me to some good reading about that if there is a purpose for using both of them together.
Hope I'm clear enough. thank you for any help.
Javascript supports WebSocket, so you don't need an additional client side framework to use it. Please take a look at this $connection service declared in this WebSocket based AngularJS application.
Basically you can listen for messages:
$connection.listen(function (msg) { return msg.type == "CreatedTerminalEvent"; },
function (msg) {
addTerminal(msg);
$scope.$$phase || $scope.$apply();
});
Listen once (great for request/response):
$connection.listenOnce(function (data) {
return data.correlationId && data.correlationId == crrId;
}).then(function (data) {
$rootScope.addAlert({ msg: "Console " + data.terminalType + " created", type: "success" });
});
And send messages:
$connection.send({
type: "TerminalInputRequest",
input: cmd,
terminalId: $scope.terminalId,
correlationId: $connection.nextCorrelationId()
});
Usually, since a WebSocket connection is bidirectional and has a good support, you can also use it for getting data from the server in request/response model. You can have the two models:
Publisher/Subscriber: Where the client declares its interest in some topics and set handlers for messages with that topic, and then the server publish (or push) messages whenever it sees fit.
Request/response: Where the client sends a message with a requestID (or correlationId), and listen for a single response for that requestId.
Still, you can have both if you want, and use REST for getting data, and WebSocket for getting updates.
In server side, you may need to use Socket.io or whatever server side framework in order to have a backend with WebSocket support.
As noted in the answer in your linked post, Angular does not currently have built-in support for Websockets. So, you would need to directly use the Websockets API, or use an additional library like Socket.io.
However, to answer your question of if you should use both a REST api and Websockets in a single Angular application, there is no reason you can't have both standard XmlHttpRequest requests for interacting with a REST api, using $http or another data layer library such as BreezeJS, for certain functionality included in various parts of the application and also use Wesockets for another part (e.g. real time chat).
Angular is designed to assist with handling this type of scenario. A typical solution to would be to create one or more controllers to handle the application functionality and update your page and then creating separate Services or Factories that encapsulate the data management of each of your data end points (i.e. the REST api and the realtime chat server), which are then injected into the Controllers.
There is a great deal of information available on using angular services/factories for managing data connections. If you're looking for a resource to help guide you on how to build an Angular application and where data services would fit in, I would recommend checking out John Papa's AngularJS Styleguide, which includes a section on Data Services.
For more information about factories and services, you can check out AngularJS : When to use service instead of factory
I'm trying to determine how to setup a web socket for the first time ever so a working minimal example with static variables (IP address for example instead of getservbyname) will help me understand what is flowing where.
I want to do this the right way so no frameworks or addons for both the client and the server. I want to use PHP's native web sockets as described here though without over-complicating things with in-depth classes...
http://www.php.net/manual/en/intro.sockets.php
I've already put together some basic JavaScript...
window.onload = function(e)
{
if ('WebSocket' in window)
{
var socket = new WebSocket('ws://'+path.split('http://')[1]+'mail/');
socket.onopen = function () {alert('Web Socket: connected.');}
socket.onmessage = function (event) {alert('Web Socket: '+event.data);}
}
}
It's the PHP part that I'm not really sure about. Presuming we have a blank PHP file...
If necessary how do I determine if my server's PHP install has this socket functionality already available?
Is the request essentially handled as a GET or POST request in
example?
Do I need to worry about the port numbers? e.g. if
($_SERVER['SERVER_PORT']=='8080')
How do I return a basic message on the initial connection?
How do I return a basic message say, five seconds later?
It's not that simple to create a simple example, I'm afraid.
First of all you need to check in php configuration if the server is configured for sockets with the setting enable-sockets
Then you need to implement (or find) a websocket server that at least follows the Hybi10 specification (https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-10) of websockets. If you find the "magic number" 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 in the code for the header, you can be sure it does follow at least Hybi06 ...
Finally, you need to have access to an admin console on the server in order to execute the PHP websocket server using php -q server.php
EDIT: This is the one I've been using a year ago ... it might still work as expected with current browsers supporting Websockets: http://code.google.com/p/phpwebsocket/source/browse/trunk/+phpwebsocket/?r=5
In my application i have created many methods in node.js file.How can i call the particular method from client side javascript.
Below is my node.js file
exports.method1=function(){
}
exports.method2=function(){
}
exports.method3=function(){
}
Your client should send a message, for example:
socket.emit("callMethod", {"methodName":"method3"});
And in your server:
socket.on("callMethod", function(data) {
if(data["methodName"] == "method3") {
exports.method3();
}
});
You don't call methods directly, you send events/messages.
I would avoid using sockets unless you really need to, from my experience they can be expensive. Sockets are great for intensive applications where a user stays engaged for awhile, otherwise I would suggest using a RESTful setup with javascript and node.js, for example:
http://blog.modulus.io/nodejs-and-express-create-rest-api
this way the socket doesn't always have to be open which causes more overhead anyway. REST will use http requests whereas sockets you will have direct connection via TCP. REST is better if your app won't be constantly engaging a user, but rather have updates here and there.