I'm creating a website that scraps with BeautifulSoup4 articles from other website and gives the user the articles as an output.
The articles are being saved in a data base. Each articles gets an id.
Current relevant code part for example:
#scrap.py
id = 0
if 'article' == 'new': #clearly this is just for the example...
id = id+1
I want the user will be able to see in real-time how many articles have been collected. Current relevant HTML code part for example:
<!--index.html-->
<div id="cur_id">{{ cur_id }}</div>
I'm rendering the id with flask to the HTML page. Current relevant code part for example:
#main.py
#app.route('/')
#app.route('/templates/index.html')
def index (name=None):
cur_id = scrap.id
return render_template('index.html', name=name, cur_id=cur_id)
How can i make the <div> in the HTML to get updated in real-time every time id gets updated?
Thanks!
Flask can do nothing to change the content of an already served webpage. As soon as the HTML is created and send to the client, flask is "done". Then the browser does stuff like interpreting the HTML, interpreting javscript (and possible adding interactions to the webpage) and lots of other things. Note that the client doesn't need to do this, there are clients that only download the HTML and are done (e.g. curl/wget).
So changing content on a webpage can only happen clientside, which essentially means javascript. If we have a new value, how do we update the content? Vanillajs is relativly simple:
var new_data = "3";
cur_id = document.getElementById('cur_id');
cur_id.innerHTML = new_data;
There are lots of libraries which help you with updating content. Jquery is an often used general purpose library. There are entire libraries which only focus on how to update the UI with new content, e.g. vue, react, ember and probably lots of others. react and ember also provide a server side if I remember correctly, so if you don't want to do vanillajs vue might be worth a look. But if you only want to update a single value you can't really beat the two lines vanillajs.
In the above snippet the new value is hardcoded which isn't helpful. How do we get new values to the client? If you really want realtime information websockets are the way to go. These are somewhat new and you should check if browser compatibility is acceptable for you.
cur_id = document.getElementById('cur_id');
ws = new WebSocket("wss://echo.websocket.org")
ws.onmessage = function(event) {cur_id.innerHTML = event.data};
simulate_update = function() {ws.send(Number(cur_id.innerHTML) + 1)};
ws.onopen = function(event) {setInterval(simulate_update, 2000)};
This uses an echo websocket server to simulate the server sending new information, but ofcourse it should connect to your websocket server. Flask per default cannot use websockets, but there some plugins like flask-sockets which make this possible. The "protocol" used above is very dumb and depending on your needs you want something smarter.
There are libraries which help with writing websocket code, often with integrated protocols. I think the most often used one is socketio which can fallback to different methods to exchange data (long polling, ajax) and puts a event/RPC/pubsub framework in front as far as I remember. It should also deal with reconnecting, heartbeat and other problems which one may not initially think about. There are lots of different websocket libraries.
Related
1) One way to put javascript on the client side is EJS e.g.
<h1> <%= title %> </h1>
where title is a variable.
2) Another way is to use back ticks and insert html or append etc using a library like jQuery
$('h1').html(`{obj.title}`);
3) A third way is to use react js esx, so import all the files on the client side if you are going down that route and add an variable as follows:
<div id="holder1"></diV>
var title = React.createClass({
render: function(){
return(
<h1 className="title1">{this.props.title1}<h1>
)
}
});
ReactDOM.render(<title
title1: "Hello World" />,
document.getElementById("holder1")
);
My question is how does react.js handle security so that the javascript cannot be manipulated such as a password on the client side (Not server side) and for 1-2 how can you enforce data hiding and prevent someone from changing the values. Can this only be done using server side react.js using node.
Let me clarify: How does React.js and other javascript libraries make the front end more secure... not just passwords... that can be handled with bcrypt hashes and https.
It's not entirely clear what you are asking, but as a very general rule, anything in the client can't be trusted.
Input posted to the server from the client can't be trusted, and should not be later displayed without proper precautions.
Web security is a huge topic. So I suggest you break your question down into smaller chunks and really identify what you are trying to ask.
Regarding React - it's not doing anything special around passwords. Its the responsibility of the backend service layer to protect from malicious input - there is a very real chance that a React component would send back crappy/malicious data. (not Reacts fault, nor is it the intent of React)
The only thing React has in this regard is {} vs dangerouslySetInnerHtml, but again, thats not going to protect passwords or the like.
I wrote a React image gallery or slideshow. I need to make the alt text indexable by search engines, but because my server is in PHP, React.renderToString is of limited use.
The server is in PHP + MySQL. The PHP uses Smarty, a decent PHP template engine, to render the HTML. The rest of the PHP framework is my own. The Smarty template has this single ungainly line:
<script>
var GalleryData = {$gallery};
</script>
which is rendered by the PHP's controller function as follows:
return array(
'gallery' => json_encode($gallery),
);
($gallery being the result table of a MySQL query).
And my .js:
React.render(<Gallery gallery={GalleryData} />, $('.gallery').get(0));
Not the most elegant setup, but given that my server is in PHP there doesn't seem to be much of a better way to do it (?)
I did a super quick hack to fix this at first shot - I copied the rendered HTML from Firebug, and manually inserted it into a new table in the DB. I then simply render this blob of code from PHP and we're good to go on the browser.
There was one complication which is that because React components are only inserted into the DOM as they're first rendered (as I understand it), and because the gallery only shows one image slide at a time, I had to manually click through all slides once before saving the HTML code out.
Now however the alt text is editable by CMS and so I need to automate this process, or come up with a better solution.
Rewriting the server in Node.js is out of the question.
My first guess is that I need to install Node, and write a script that creates the same React component. Because the input data (including the alt text) has to come from MySQL, I have a few choices:
connect to the MySQL DB from Note, and replicate the query
create a response URL on the PHP side that returns only the JSON (putting the SQL query into a common function)
fetch the entire page in Node but extracting GalleryData will be a mess
I then have to ensure that all components are rendered into the DOM, so I can script that by manually calling the nextSlide() method as many times as there are slides (less one).
Finally I'll save the rendered DOM into the DB again (so the Node script will require a MySQL connection after all - maybe the 1st option is the best).
This whole process seems very complicated for such a basic requirement. Am I missing something?
I'm completely new to Node and the whole idea of building a DOM outside of the browser is basically new to me. I don't mind introducing Node into my architecture but it will only be to support React being used on the front-end.
Note that the website has about 15,000 pageviews a month, so massive scalability isn't a consideration - I don't use any page caching as it simply isn't needed for this volume of traffic.
I'm likely to have a few React components that need to be rendered statically like this, but maintaining a small technical overhead (e.g. maintaing a set of parallel SQL queries in Node) won't be a big problem.
Can anyone guide me here?
I think you should try rendering React components on server-side using PHP. Here is a PHP lib to do that.
But, yes, you'll basically need to use V8js from your PHP code. However, it's kind of experimental and you may need to use other around. (And this "other way around" may be using Node/Express to render your component. Here is some thoughts on how to do it.)
Our website development team manages many web sites (each with a different vanity URL) on our INTRAnet. We'd like to implement something (code snippet) that is easy to add the Application.cfm/OnRequestEnd.cfm page which would insert a record into the database tracking things like page, url, querystring, userid, etc - basic stuff. Inserting the record is not a big deal. What I'd like to know is from a performance stand point, what would you all recommend so that we dont' get a bottleneck of inserts queued up as employees hit the various sites. We can't use jQuery since not every site will have the same version of jQuery so we really are limited to just using Coldfusion - I think.
Ideally, what we'd like to be able to do is create one main tracking file on our main server and reference that file from all of our other sites. Then if/when we need to make an update, we can make a global change - kind of like how Google Analytics does, just not nearly as much details.
On all of the sites we support, we do have our department logo on those pages. I thought about building a tracking process into the loading of the image, much like they do with emails.
Any thoughts on this would be appreciated of if you have a better idea - I'm all ears.
UPDATED
Regarding the image processing, I could not find the original link for the tutorial from easycfm.com but I found what appears to be the identical code here: http://www.experts-exchange.com/Software/Server_Software/Web_Servers/ColdFusion/A_2386-Track-your-Emails-using-coldfusion.html
I would say that you should not try to prematurely optimise this until you have a reason to. If you find you have a problem down the track, you can deal with it then.
Create a Logging CFC and implement a method which receives the metrics you wish to log, and then wraps that up into a record to insert into the DB, and use <cfquery> to write them to a DB table. Call the function from OnRequestEnd.cfm.
I specifically suggest rolling this into a Logging CFC because if you need to change the implementation of how you log things later on, you can just change the inner workings of the logging method, instead of having to mess with all your sites.
Adam Cameron's answer is probably the way you should go. On top of that what I would suggest is inserting those records into a table with no additional indexes. This will cause the inserts to be very quick. Then you can use a scheduled database job to move this data into a nice normalised schema that is quick to analyse.
I wouldn't write off using JavaScript just because each site will have a different version of JQuery. People still used to be able to get stuff done prior to the existence of JQuery it is just a JavaScript library. It is very basic JavaScript to include dynamically include a script file that points at your tracking file with the relevant parameters.
<script type="text/javascript">
var script = document.createElement("script");
var body = document.getElementsByTagName("body")[0];
script.setAttribute("src", "http://www.yourdomain.com/trackingfile.cfm?" + yourparams);
body.appendChild(script);
</script>
As you can see, no need for JQuery.
I would like to use the Glimpse client viewer in a web application in order to render some JSON. I am unable to use the Glimpse server implementation on the site in question. However, I can implement my own IHttpHandler to render the information using the Glimpse JSON format.
Has anyone done this and posted details on how to do it? If not, can anyone tell me the steps required to get this up and running? Alternatively, are there any other similar viewer frameworks out there?
Note: I am poking around the source and have seen the client js etc. I will continue down the source hacking route, but was hoping someone may have some shortcuts for me!
As Nik said I would be interested in what you are trying to do. But in the mean time the best place to look is http://getglimpse.com/Protocol.
If you look on this page you will see that we have built a protocol tester. This allows you to put in any JSON and see the output.
If you want to do this yourself, have a look at http://getglimpse.com/Scripts/Protocol/LayoutExample.js and you will see how we do this without using the whole of Glimpse.
You will see that we are doing something like the following:
var data = { test : 'test', hello : 'hello' };
var html = $Glimpse.glimpseProcessor.build(data, 0, false)
$('.panel').html(html);
I know this isn't as nice as it could be but it wasn't designed with this in mind.
We are currently working on refactoring the client code to make this all better.
We haven't really documented all of this yet.
Your best bet is to look at first glimpse javascript file that gets rendered to a page - it is the data file. If you can output data in that format, which is basically just one object of key value pairs, then the client will pick up the data and render it.
You might also want to look at the Glimpse.PHP implementation, since they'vve had to do the same thing you are.
I consistently come across this code smell where I am duplicating markup, and I'm not really sure how to fix it. Here's a typical use case scenario:
Let's say we'd like to post comments to some kind of article. Underneath the article, we see a bunch of comments. These are added with the original page request and are generated by the templating engine (Freemarker in my case, but it can be PHP or whatever).
Now, whenever a user adds a comment, we want to create a new li element and inject it in the current page's list of comments. Let's say this li contains a bunch of stuff like:
The user's avatar
Their name
A link to click to their profile or send them a private message
The text they wrote
The date they wrote the comment
Some "edit" and "delete" links/buttons if the currently logged in user has permission to do these actions.
Now, all of these things were already written in our template that originally generated the page... so now we have to duplicate it inside of Javascript!
Sure, we can use another templating language - like Jquery's Template plugin - to ease the pain generating and appending this new li block... but we still end up with duplicate html markup that is slightly different because we can't use macros or other conveniences provided to us by the templating language.
So how do we refactor out the duplication? Is it even possible, or do we just put up with it? What are the best practices being used to solve this problem?
This is a common problem and becomes more obvious as the UI complexity increases, and changes have to be done on both the server and client templates. This problem is fixable by using a the same template markup on both the client and server sides. The template processors must be written in both JavaScript and the server side language.
Two other solutions that are cleaner than the above approach, but both have their own problems:
Do everything client side
Do everything server side
If all markup generation is done on the client side, then the server acts more or less like a web service which only sends back data in whatever formats suits the application. JSON, and XML are really popular formats for most web services nowadays. The client always generates the necessary HTML and JS. If going with this approach, the boundary between the client and server must be well defined. Since the client has limited knowledge of what happens on the server, this means that proper error codes must be defined. State management will become harder since most/all server interaction will be happening asynchronously. An example of adding a comment with this approach may look like:
$('#add-comment').click(function() {
var comment = $('#comment-box').text();
$.ajax('http://example.com/add', {
success: function() {
addCommentRow(comment);
},
...
});
});
function addCommentRow(comment) {
var user = currentUser().name;
var html = "<li><b>{user}</b> says {comment}</li>";
html = html.replace("{user}", user).replace("{comment}", comment);
var item = $('<li>').html(html);
$('#comments').append(item);
}
The other approach is to do everything server side. Whenever a change happens, shoot a request to the server, and ask it for the updated view. With a fast backend, response times under a second, and proper indicators of network activity, the application should seem very responsive despite everything happening on the server. The above example would be simplified to:
$('#add-comment').click(function() {
$.ajax('http://example.com/add', {
success: function(response) {
$('#comments').html(response);
},
...
});
});
Although this seems a lot more cleaner on the client side than the previous approach, we have just moved the markup generation up to the server. However, if the application is not very AJAXy like Google Maps, then this approach may be easier to work with. Again, it's a matter of how complicated the application is, and perhaps maintaining state client side is a necessity for you, in which case you may want to go with the previous approach.