I have a Java Spring Application with a Tomcat server that listen on kafka topic. I want to display all messages in a real-time mode on the web page. Therefore, when a kafka messages is arrived in the backend I want to see it on my web page. I don't know a good approach to push kafka message directly to the front-end and display it on web page. Is someone could help my with a solution and some examples that could help? Thanks!
I have implemented a system like this in Java for my last employer, albeit not with Spring/Tomcat. It was consuming messages from Kafka and serving them on a web socket to be displayed in the browser. The approach I followed was to use akka-stream-kafka and akka-http for web-socket support. The benefit of that is both are based on akka-streams which makes it an easy fit for streaming data.
While you can embed akka-http in your spring app running inside tomcat, it may not feel the most natural choice any more as spring framework already has its own support for both kafka and websockets. However, if you're not familiar with either, then jumping on the akka approach may be easiest and the core logic goes along these lines (I can't share the code from work so have just put this together from the examples in the docs, not tested):
public Route createRoute(ActorSystem system) {
return path("ws", () -> {
ConsumerSettings<byte[], String> consumerSettings = ConsumerSettings.create(system, new ByteArrayDeserializer(), new StringDeserializer())
.withBootstrapServers("localhost:9092")
.withGroupId(UUID.randomUUID().toString()) //this is so that each client gets all messages. To be able to resume from where a client left off in case of disconnects, you can generate in on the client side and pass in the request
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
return handleWebSocketMessages(
Flow.fromSinkAndSourceCoupled(
Sink.ignore(),
Consumer.committableSource(consumerSettings, Subscriptions.topics("topic1"))
.map(msg -> TextMessage.create(msg.record().value()))
)
);
}
}
To expose this route you can follow the minimalistic example, the only difference being the route you define needs the ActorSystem:
final Http http = Http.get(system);
final ActorMaterializer materializer = ActorMaterializer.create(system);
final Flow<HttpRequest, HttpResponse, NotUsed> routeFlow = createRoute(system).flow(system, materializer);
final CompletionStage<ServerBinding> binding = http.bindAndHandle(routeFlow,
ConnectHttp.toHost("localhost", 8080), materializer);
Once you have your messages published to the websocket, the front end will code will of course depend on your UI framework of choice, the simplest code to consume ws messages from javascript is:
this.connection = new WebSocket('ws://url-to-your-ws-endpoint');
this.connection.onmessage = evt => {
// display the message
To easily display the message in the UI, you want the format to be something convenient, like JSON. If your Kafka messages are not JSON already, that's where the Deserializers in the first snippet come in, you can convert it to a convenient JSON string in the Deserializer or do it later on in the .map() called on the Source object.
Alternatively, if polling is an option you can also consider using the off-the-shelf Kafka Rest Proxy, then you only need to build the front-end.
So to start off, I have a raspberry pi, running a lighttp server on arch. I have a webpage that will have a max of 10 people connected at a time. Each user will be given the tag "Master," or "observer." The webpage will have controls that only the "Master," can use.
Just a heads up: I am just learning most of this so I may be making some mistakes about how to accomplish this.
My original idea is has follows. When a user connects to the database their IP address would be grabbed and inserted into a SQLite database, along with a user tag, and time of connection. From there I would be able to query the database for the users tag when they tried to execute various commands.
Whatever I use needs to be very lightweight and not store cookies on the users device.
Here is the JavaScript I currently have, it probably isn't the most efficient, but I plan on getting it working then making it look nice.
This code is supposed to connect the databases and insert the user.
<script type="text/javascript" src="http://l2.io/ip.js?var=myip"></script>
<script type="application/javascript">
var db = openDatabase('userCon.contbl.sqlite', '1.0', 'contbl', 1024);
db.transaction(function(transaction) {
var ip = myip;
var conStatus = "master"
var date = new Date();
console.log('Inserting into the database ' + ip + ',' + conStatus +',' + date);
transaction.executeSql('INSERT INTO contbl(ipAd, conType, lastActive) VALUES (?,?,?,?)',[ip,conStatus,date], function(transaction, results) {
}, function (transaction, err){
console.log(err.message+":Error"); // here an error
});
});
</script>
<script type="application/javascript" src="http://jsonip.appspot.com/?callback=getip"> </script>
I am unable to connect to the SQLite database I created on the pi, which after my research may be because SQLite is supposed to be run locally and not on a server.
Is there some sort of work around to point to the SQLite database on the pi, or is there a better resource to use for this type of task?
EDIT:
I guess my original post was not specific enough. The basic idea is I need to be able to pass a tiny bit of information from a webpage, back to the server hosting it. i.e. User connect to the server and sends its IP then the server tags that IP as an Observer of Controller. From there the server will treat each person viewing the webpage differently based on how the user was tagged.
My original plan was to use a really light weight database like SQLite, but as I found out, SQLite is local use only. I need to do this on a sever with lower then 10 connections.
My hope is someone has a suggestion, or an example to solve this problem.
The most common way for javascript running on a web page to talk to a server these days is some kind of RESTful web service.
You'll want to find a server-side programming language that runs on the Pi. Maybe node.js? It's like javascript but on a server, see here: http://weworkweplay.com/play/raspberry-pi-nodejs/
You then write whatever methods you want in the server-side language, methods which talk to the database, then have your client-side javascript call those methods. Google "REST services node.js" and you'll find plenty of how-tos.
If the SQLite database is running on the server, and this code runs in the user's browser, it can't connect to the database. The only interaction between the browser and your server is HTTP, unless there's a gigantic security hole in your server.
So I am on a webproject that succesfully connects to and reads from an SQL database. The code that connects to it looks like this.
//From Here
var connection = new ActiveXObject("ADODB.Connection") ;
var connectionstring="Data Source=<server>;Initial Catalog=<catalog>;User ID=<user>;
Password=<password>;Provider=SQLOLEDB";
connection.Open(connectionstring);
var rs = new ActiveXObject("ADODB.Recordset");
//To Here
rs.Open("SELECT * FROM table", connection);
rs.MoveFirst
while(!rs.eof)
{
document.write(rs.fields(1));
rs.movenext;
}
rs.close;
connection.close;
Simple and effective and I have it working fine. But those first 4 (marked from here to here) run horribly slow and I have to reconnect every time I need to read from or write to the sql database... which is a lot for my project. So every time I run this code (which is on every other webpage I am creating in this project) I have to sit and wait for this code to run.
I have been told/ required for the project, to configure the design using javascript and spring framework. Apparently there is either a) a way to hold the connection so I don't have to run this code a every time I hit a go to new page or b) a different method of connection to the SQL database (something to replace those 4 lines of code. Both of which have to do with my using the Spring Framework.
I have never used Spring Framework before and need to learn fast. Been watching their website tutorials but still have no idea what to do or how to do it. Let me know if have any ideas.
Please and Thanks.
I think you got something wrong here: You aren't supposed to access your database from JavaScript because any user can access any data that way, or delete everything.
You should access the database server-side only.
You need connection pooling, which means, instead of opening and closing a new connection every time, keep the one that you were about to close, and see if you can reuse it later.
Unfortunately, I've never seen anything in JavaScript that allows you to reuse objects across pages, so there's no way afaik to do this in JavaScript. It's considered a very bad practice to connect from a browser directly to a database, anyway.
Usually, what you do is put a java or C# application server in between, and let these to the database access, using a connection pool. From your javascript, you then do an AJAX call to the application server, which will use the connection pool, and return e.g. JSON objects.
See for example http://tomcat.apache.org/tomcat-6.0-doc/jndi-datasource-examples-howto.html#MySQL_DBCP_Example
Can JavaScript connect with MySQL? If so, how?
Client-side JavaScript cannot access MySQL without some kind of bridge. But the above bold statements that JavaScript is just a client-side language are incorrect -- JavaScript can run client-side and server-side, as with Node.js.
Node.js can access MySQL through something like https://github.com/sidorares/node-mysql2
You might also develop something using Socket.IO
Did you mean to ask whether a client-side JS app can access MySQL? I am not sure if such libraries exist, but they are possible.
EDIT: Since writing, we now have MySQL Cluster:
The MySQL Cluster JavaScript Driver for Node.js is just what it sounds like it is – it’s a connector that can be called directly from your JavaScript code to read and write your data. As it accesses the data nodes directly, there is no extra latency from passing through a MySQL Server and need to convert from JavaScript code//objects into SQL operations. If for some reason, you’d prefer it to pass through a MySQL Server (for example if you’re storing tables in InnoDB) then that can be configured.
JSDB offers a JS interface to DBs.
A curated set of DB packages for Node.js from sindresorhus.
If you want to connect to a MySQL database using JavaScript, you can use Node.js and a library called mysql. You can create queries, and get results as an array of registers. If you want to try it, you can use my project generator to create a backend and choose MySQL as the database to connect. Then, just expose your new REST API or GraphQL endpoint to your front and start working with your MySQL database.
OLD ANSWER LEFT BY NOSTALGIA
THEN
As I understand the question and correct me if I am wrong, it refers to the classic server model with JavaScript only on the client-side. In this classic model, with LAMP servers (Linux, Apache, MySQL, PHP) the language in contact with the database was PHP, so to request data to the database you need to write PHP scripts and echo the returning data to the client. Basically, the distribution of the languages according to physical machines was:
Server Side: PHP and MySQL.
Client Side: HTML/CSS and JavaScript.
This answered to an MVC model (Model, View, Controller) where we had the following functionality:
MODEL: The model is what deals with the data, in this case, the PHP scripts that manage variables or that access data stored, in this case in our MySQL database and send it as JSON data to the client.
VIEW: The view is what we see and it should be completely independent of the model. It just needs to show the data contained in the model, but it shouldn't have relevant data on it. In this case, the view uses HTML and CSS. HTML to create the basic structure of the view, and CSS to give the shape to this basic structure.
CONTROLLER: The controller is the interface between our model and our view. In this case, the language used is JavaScript and it takes the data the model send us as a JSON package and put it inside the containers that offer the HTML structure. The way the controller interacts with the model is by using AJAX. We use GET and POST methods to call PHP scripts on the server-side and to catch the returned data from the server.
For the controller, we have really interesting tools like jQuery, as "low-level" library to control the HTML structure (DOM), and then new, more high-level ones as Knockout.js that allow us to create observers that connect different DOM elements updating them when events occur. There is also Angular.js by Google that works in a similar way, but seems to be a complete environment. To help you to choose among them, here you have two excellent analyses of the two tools: Knockout vs. Angular.js and Knockout.js vs. Angular.js. I am still reading. Hope they help you.
NOW
In modern servers based in Node.js, we use JavaScript for everything. Node.js is a JavaScript environment with many libraries that work with Google V8, Chrome JavaScript engine. The way we work with these new servers is:
Node.js and Express: The mainframe where the server is built. We can create a server with a few lines of code or even use libraries like Express to make even easier to create the server. With Node.js and Express, we will manage the petitions to the server from the clients and will answer them with the appropriate pages.
Jade: To create the pages we use a templating language, in this case, Jade, that allow us to write web pages as we were writing HTML but with differences (it take a little time but is easy to learn). Then, in the code of the server to answer the client's petitions, we just need to render the Jade code into a "real" HTML code.
Stylus: Similar to Jade but for CSS. In this case, we use a middleware function to convert the stylus file into a real CSS file for our page.
Then we have a lot of packages we can install using the NPM (Node.js package manager) and use them directly in our Node.js server just requiring it (for those of you that want to learn Node.js, try this beginner tutorial for an overview). And among these packages, you have some of them to access databases. Using this you can use JavaScript on the server-side to access My SQL databases.
But the best you can do if you are going to work with Node.js is to use the new NoSQL databases like MongoDB, based on JSON files. Instead of storing tables like MySQL, it stores the data in JSON structures, so you can put different data inside each structure like long numeric vectors instead of creating huge tables for the size of the biggest one.
I hope this brief explanation becomes useful to you, and if you want to learn more about this, here you have some resources you can use:
Egghead: This site is full of great short tutorials about JavaScript and its environment. It worths a try. And the make discounts from time to time.
Code School: With a free and very interesting course about Chrome Developer tools to help you to test the client-side.
Codecademy: With free courses about HTML, CSS, JavaScript, jQuery, and PHP that you can follow with online examples.
10gen Education: With everything you need to know about MongoDB in tutorials for different languages.
W3Schools: This one has tutorials about all this and you can use it as a reference place because it has a lot of shortcode examples really useful.
Udacity: A place with free video courses about different subjects with a few interesting ones about web development and my preferred, an amazing WebGL course for 3D graphics with JavaScript.
I hope it helps you to start.
Have fun!
No, JavaScript can not directly connect to MySQL. But you can mix JS with PHP to do so.
JavaScript is a client-side language and your MySQL database is going to be running on a server
Bit late but recently I have found out that MySql 5.7 got http plugin throuh which user can directly connect to mysql now.
Look for Http Client for mysql 5.7
I think you would need to add something like PHP into the equation. PHP to interact with the database and then you could make AJAX calls with Javascript.
Simple answer is: no.
JavaScript is a client-side language that runs in the browser (node.js notwithstanding) and MySQL is a server-side technology that runs on the server.
That means you typically use a server-side language like ASP.NET or PHP to connect to the database.
YES? Have a look a meteor. Links:
http://meteor.com/screencast and http://net.tutsplus.com/tutorials/javascript-ajax/whats-this-meteor-thing/
I don't understand how it is done. But Nettuts+ put this thing in the javascript-ajax section, maybe magic happens.
It also shows some way to connect and insert to MongoDB with JS, like this:
Products.insert({Name : "Hammer", Price : 4.50, InStock : true});
Products.insert({Name : "Wrench", Price : 2.70, InStock : true});
Products.insert({Name : "Screw Driver", Price : 3.00, InStock : false});
Products.insert({Name : "Drill", Price : 5.25, InStock : true});
Yes. There is an HTTP plugin for MySQL.
http://blog.ulf-wendel.de/2014/mysql-5-7-http-plugin-mysql/
I'm just googling about it now, which led me to this stackoverflow question. You should be able to AJAX a MySQL database now or in the near future (they claim it's not ready for production).
Depending on your environment, you could use Rhino to do this, see Rhino website. This gives you access to all of the Java libraries from within JavaScript.
You can send AJAX requests to some server-side RESTful wrappers for MySQL, such as DBSlayer, PhpRestSQL or AlsoSQL (for Drizzle, a fork of MySQL).
Yes you can. MySQL connectors use TCP for connection, and in JS there is an little modified version of TCP client called Websocket. But you can't directly connect to MySQL server with websocket. You will need some 3rd party bridge between websocket and mysql. It receive query from websocket, send it to mysql, response result and resend to JS.
And this is my example bridge written in C# with websocket-sharp library:
class JSQLBridge : WebSocketBehavior
{
MySqlConnection conn;
protected override void OnMessage(MessageEventArgs e)
{
if (conn == null)
{
try
{
conn = new MySqlConnection(e.Data);
conn.Open();
}
catch (Exception exc)
{
Send(exc.Message);
}
}
else
{
try
{
MySqlCommand cmd = new MySqlCommand(e.Data, conn);
cmd.ExecuteNonQuery();
Send("success");
}
catch (Exception exc)
{
Send(exc.Message);
}
}
}
protected override void OnClose(CloseEventArgs e)
{
if (conn != null)
conn.Close();
}
}
JS side:
var ws = new WebSocket("ws://localhost/");
ws.send("server=localhost;user=root;database=mydb;");
ws.send("select * from users");
JavaScript can't connect directly to DB to get needed data but you can use AJAX. To make easy AJAX request to server you can use jQuery JS framework http://jquery.com. Here is a small example
JS:
jQuery.ajax({
type: "GET",
dataType: "json",
url: '/ajax/usergroups/filters.php',
data: "controller=" + controller + "&view=" + view,
success: function(json)
{
alert(json.first);
alert(json.second);
});
PHP:
$out = array();
// mysql connection and select query
$conn = new mysqli($servername, $username, $password, $dbname);
try {
die("Connection failed: " . $conn->connect_error);
$sql = "SELECT * FROM [table_name] WHERE condition = [conditions]";
$result = $conn->query($sql);
if ($result->num_rows > 0) {
// output data of each row
while($row = $result->fetch_assoc()) {
$out[] = [
'field1' => $row["field1"],
'field2' => $row["field2"]
];
}
} else {
echo "0 results";
}
} catch(Exception $e) {
echo "Error: " . $e->getMessage();
}
echo json_encode($out);
No.
You need to write a wrapper in PHP, and then export the returned data (probably as Json). NEVER, get from your "_GET" the SQL code, as this is called an SQL injection (people who learn this will have full control over your database).
This is an example I wrote:
function getJsonData()
{
global $db;
if (!$db->isConnected()) {
return "Not connected";
}
$db->query("SELECT * FROM entries");
$values = array();
while( $v = $db->fetchAssoc()){
$values[] = $v;
}
return json_encode($values);
}
switch (#$_GET["cmd"]){
case 'data':
print getJsonData();
exit;
default:
print getMainScreen();
exit;
}
Do learn about SQL injections please.
You can connect to MySQL from Javascript through a JAVA applet. The JAVA applet would embed the JDBC driver for MySQL that will allow you to connect to MySQL.
Remember that if you want to connect to a remote MySQL server (other than the one you downloaded the applet from) you will need to ask users to grant extended permissions to applet. By default, applet can only connect to the server they are downloaded from.
If you're not locked on MySQL you can switch to PostgreSQL. It supports JavaScript procedures (PL/V8) inside the database. It is very fast and powerful. Checkout this post.
Of course you can. In Nodejs you can connect server side JavaScript with MySQL using MySQL driver.
Nodejs-MySQL
is there any possibility to access and retrieve data from MongoDB directly over JavaScript form browser without backend processing?
MongoDB natively does not have a fully usable REST interface. You will need either Python based Sleepy Mongoose or Node.js based MongoDB Rest
Please see http://www.mongodb.org/display/DOCS/Http+Interface for more information.
Mongo's inbuilt REST interface is very basic & read only.
If you happen to host your database on MongoLabs, they also expose a REST API. Watch out, it's not secure, as an API key is passed with each request, but your could use it to retrieve public data through Javascript:
https://support.mongolab.com/entries/20433053-Is-there-a-REST-API-for-MongoDB-
Once you have your mongolab db setup, you can access it thru REST request such as
$.getJSON("https://api.mongolab.com/api/1/databases/your-db/collections/your-collection/?apiKey=your-key", function(json) {
//console.log( "JSON Data: " + json );
});
Not in the standard security context.
If you have a situation where you have elevated security rights (such as inside a custom browser extension) then it may become possible.