I am creating a web application with Django (back end) and JavaScript (front end).
The main scenario is the following: a client asks the server to train a neural network (which can take quite a lot: seconds or even minutes, sometimes). Immediately after sending the request, the client shows to the user how the training session is going; it does so by retrieving and displaying the training error, which is actually just a number.
For training the network I made this function, which I implemented as a generator:
def train(network, trainset):
# [setup variables...]
while (current_error > target_error):
# [continue to train the network...]
# [update current_error...]
yield current_error
On the front end (a simple HTML page) I have a JS script for retrieving data dynamically from a Django view. My Django view goes something like this:
def getError(request):
# ...
generator = train(net, dataset)
return StreamingHttpResponse(generator)
At the moment I can retrieve the data with an AJAX call but I always have to wait for the server to train the whole neural network before I can see the training errors.
My problem is that I would like the user to view the error updates while the server is training the network, not after: that's why I implemented the training function as a generator (and that's why I chose to return a StreamingHttpResponse).
What can I do for retrieving the data while the server is computing? Specifically, I would like to update the front end each time the training generator yields a number.
Initial analysis:
Let's have a look at the documentation:
yield:
The yield expression is used when defining a generator function
generator:
A function which returns a generator iterator. It looks like a normal function except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function.
From the aforementioned it is clear that your code sample works well but not as intended:
The train() function yields a generator which gets "transmitted" to your front-end from the getError after the whole calculation is done.
Reform the problem based on analysis:
With the above mentioned, your problem will be to get the calculations of each step of the train() process and display them to the front-end.
A skeleton of a solution:
Create a global variable (yes I said it) list.
Refactor train() function to use the previous list:
train_errors = []
def train(network, trainset):
# [setup variables...]
while (current_error > target_error):
# [continue to train the network...]
# [update current_error...]
train_errors.append(current_error)
Assuming that train() is located in views.py file, refactor the getError() to return the train_errors list:
from django.http import JsonResponse
def getError(request):
# ...
return JsonResponse(train_errors, safe=False)
I utilize JsonResponse, but feel free to use anything that suites you.
Make your front-end to repeat the AJAX call every N seconds, by utilizing javascript's setInterval, and display the differences from the previous state of the data.
Example of use:
var myIntervalCall = setInterval(myAJAXcall, 500);
function myAJAXcall{
// Create your AJAX call in here
// And sufficient code to display the changes on the received data.
}
The above code will make myAJAXcall to the back-end, every 0.5 sec
The above will allow you to make an initial approach to your problem and you can/must expand on this (barbaric) solution.
EDIT/Disclaimer (due to #George and #brunodesthuilliers comments):
The above, is not a good solution!! It is only a quick solution to get the author "unstuck" from his current problem and help him move on with his development. The "solution" provides the basis for a thought process, considering the problem at hand (the actual problem that the author has at the moment) and therefore provides a minimal and initial approach to a solution.
The aforementioned distiled: DO NOT TAKE THIS SOLUTION AS CANON! it is only a "jump start".
Solution 1: use celery for the computation, storing progress in redis, and have your front side js code poll a view that will get progress from a dedicated view.
Solution 2: use a websocket.
Related
I'm making a bot that could go though my clients website and get details for the products they have and since the products' pricing, availability, promos & etc. changes multiple times in a week then I need to update my spreadsheet too manually for the rest of my team to use. Finding those changes for each product is my job which is very prone to error. If only they could provide me with an API, life would have much been easier but no.
My client told me that I've been spamming their error report and tripping their alarms on their server error logs and the functions/files they found are the helper functions that I injected into the DOMWorld on their site using page.addScriptTag('./client/helper.js') and the helper functions inside that file (wasn't sure if that is all they found but so far they have only mentioned that file and the functions inside it).
const product_name = await page.$eval('#product-name', product => {
getProductName(product )
})
The getProductName() function comes from this file ./client/helper.js.
//This simply removes unwanted characters
function getProductName(product ) {
return product.innerHTML.replace(/[\s]{2}/g,'')
}
I wasn't aware that they could see this as I thought puppeteer runs locally on the client-side, on my computer only.
I have few ideas that could make this go undetected and silence their alarms and errors.
First would be to NOT use page.addScriptTag('./client/helper.js') at all and put the codes of this file straight into the main file then do further processes after getting the innerHTML.
const product_name = await page.$eval('#product-name', product => {
return product.innerHTML
})
const final_product_name = product_name.replace(/[\s]{2}/g,'')
This does work but my only concerns is if will this still be detectable as I am still running an anonymous function inside page.$eval(). That I am still trying to find out if my client complains again.
So my question really is..
Will removing page.addScriptTag() and using anonymous function be undetectable?
is there any other way to scrape string in the DOMWorld ? (if it turns out that anonymous function still triggers errors)
Is this because i'm using headless chrome?
Thank you for reading.
This question may be a bit confusing, so let me give you some background. Eel is a Python module where you can take functions made in Python and use them in Javascript, and vice versa. What I want to do is take a json made from a Python function, put it in Javascript, and make a table based on the json that was taken from the Python side. Here's an example.
python.py
def json_example():
json = [
{
"key": "value1"
},
{
"key": "value2"
}
]
return json
js.html
<body>
<div></div>
</body>
<script>
function js_example() {
# This is where the function from Python is called
var json_obj = eel.json_example();
var tbl = $("<table/>").attr("id", "example_table");
$("div").append(tbl);
for(var i=0; i<json_obj.length; i++){
var tr="<tr>";
var td="<td>"+obj[i]["key"]+"</td></tr>";
$('#example_table').append(tr+td);
}
}
I tested both of these functions out separately with a few changes and they both work. However, here's where the problem starts. For some reason, the Javascript part is not getting anything from the function it calls from the Python code. the variable "json_obj" should be equal to the json I made in the Python function, but for some reason the return value of the function isn't creating tangible data that can be manipulated in the Javascript, basically returning nothing. And the eel transfer itself works as well. If you replace "return" with "print", it will print the json in the console.
Also, please don't tell me to just put the json itself in the Javascript. I have a reason for needing the json to come from the Python side.
So basically, here's my question: how do you get a Python function to create a value that can be manipulated in Javascript?
The problem is that when eel exposes a function what it actually does is it creates a new function that will return a promise containing the return value of your python function.
So you should have something like this instead:
let json_obj = '';
eel.json_example()(x => json_obj = x);
If you need more help on callbacks, refer to https://github.com/ChrisKnott/Eel.
Convert to json within Python if you're calling Python to begin with and send json to JS in the return value.
See: https://github.com/ChrisKnott/Eel/tree/master/examples/03%20-%20sync_callbacks
To do a synchronous operation that will take time to complete in Python and then return a value into JS, use;
let n = await eel.py_random()();
which is really
let mySlowReturnValueFromPython = await eel.myExposedPythonFunction()();
In fact I tried to code in my own promises and I was getting back garbage that looked like eel promises. The browser maintains a thread while you call this way, so the user can kick off a long Python operation and still interact with the GUI.
I'd note that you can still call things that update the GUI asynchronously. If you have a bunch of Python functions which return ready-made HTML as I do, then you can kick them all off in a row without waiting and they will update the divs whenever they return. I use this;
eel.expose(updateDiv);
function updateDiv(newData, divToUpdate)
{
var fieldToUpdate = document.getElementById(divToUpdate)
fieldToUpdate.innerHTML = newData
}
Then I call my Python function which gets the data synchronously, packs it up into a ready-made HTML chunk for the GUI, and then calls updateDiv from Python. I'm actually really enjoying the power that this "ping pong" interaction between a synchronous codebase and an asynchronous one give me when working with a GUI. Worlds better than arfing about with TK.
I hope this helps you and you can struggle less with it than I did. Once you understand how this works, Eel is really great. It apparently handles sync for you, just hand it a blank callback (or whatever black magic that is). What a great lib! Eel is just perfect for locally hosted GUI's. I'd like to see a better GUI framework than HTML/CSS/JS - the truth is that there really isn't one, those things are so well tested and stable, with so many available examples for whatever you could want to create.
I'd really like to see this become the native Python GUI solution. Browsers are extremely cross-platform and the only problem to solve when porting becomes interfacing the browser to Python.
I read a lot about Express / SocketIO and that's crazy how rarely you get some other example than a "Hello" transmitted directly from the app.js. The problem is it doesn't work like that in the real world ... I'm actually desperate on a logic problem which seems far away from what the web give me, that's why I wanted to point this out, I'm sure asking will be the solution ! :)
I'm refactoring my app (because there were many mistakes like using the global scope to put libs, etc.) ; Let's say I've got a huge system based on SocketIO and NodeJS. There's a loader in the app.js which starts the socket system.
When someone join the app it require() another module : it initializes many socket.on() which are loaded dynamically and go to some /*_socket.js files in a folder. Each function in those modules represent a socket listener, then it's way easier to call it from the front-end, might look like this :
// Will call `user_socket.js` and method `try_to_signin(some params)`
Queries.emit_socket('user.try_to_signin', {some params});
The system itself works really well. But there's a catch : the module that will load all those files which understand what the front-end has sent also transmit libraries linked with req/res (sessions, cookies, others...) and must do it, because the called methods are the core of the app and very often need those libraries.
In the previous example we obviously need to check if the user isn't already logged-in.
// The *_socket.js file looks like this :
var $h = require(__ROOT__ + '/api/helpers');
module.exports = function($s, $w) {
var user_process = require(__ROOT__ + '/api/processes/user_process')($s, $w);
return {
my_method_called: function(reference, params, callback) {
// Stuff using $s, $w, etc.
}
}
// And it's called this way :
// $s = services (a big object)
// $w = workers (a big object depending on $s)
// They are linked with the req/res from the page when they are instantiated
controller_instance = require('../sockets/'+ controller_name +'_socket')($s, $w);
// After some processes ...
socket_io.on(socket_listener, function (datas, callback) {
// Will call the correct function, etc.
$w.queries.handle_socket($w, controller_name, method_name, datas);
});
The good news : basically, it works.
The bad news : every time I refresh the page, the listeners double themselves because they are in a loop called on page load.
Below, this should have been one line :
So I should put all the socket.on('connection'...) stuff outside the page loading, which means when the server starts ... Yes, but I also need the req/res datas to be able to load the libraries, which I get only when the page is loaded !
It's a programing logic problem, I know I did something wrong but I don't know where to go now, I got this big system which "basically" works but there's like a paradox on the way I did it and I can't figure out how to resolve this ... It's been a couple of hours I'm stuck.
How can I refacto to let the possibility to get the current libraries depending on req/res within a socket.on() call ? Is there a trick ? Should I think about changing completely the way I did it ?
Also, is there another way to do what I want to do ?
Thank you everyone !
NOTE : If I didn't explain well or if you want more code, just tell me :)
EDIT - SOLUTION : As seen above we can use sockets.once(); instead of sockets.on(), or there's also the sockets.removeAllListeners() solution which is less clean.
Try As Below.
io.sockets.once('connection', function(socket) {
io.sockets.emit('new-data', {
channel: 'stdout',
value: data
});
});
Use once instead of on.
This problem is similar as given in the following link.
https://stackoverflow.com/questions/25601064/multiple-socket-io-connections-on-page-refresh/25601075#25601075
A kind of google docs that says "Saving..." when user edit text. Or all kind of web site that uses ajax and has to save something on server.
I have understand that Meteor save on MongoDB (server side) through Method.Call (not stubs). But how can know when all method.call (write to db) are ended? (not on client side (minimongo))
EDIT:
In my case, there are 3 Method for CRUD operations, defined on client and server:
Insert
Update
Remove
I would that user is notified with a "saving..." message, while this method running on server.
Could be use the async callback when call method from client?
The async callback is indeed what you want. In the simplest case:
Session.set("theThingIsLoading", true);
Meteor.call("doTheThing", function (error, result) {
if (error) {/* report it somehow */}
Session.set("theThingIsLoading", false);
});
{{#if theThingIsLoading}}
<p>Loading...</p>
{{/if}}
Where theThingIsLoading is a helper which does Session.get("theThingIsLoading").
I would suggest the atmosphere package "auto nprogress". It will show a fancy loading bar at the top of the screen when an action is processed.
Hope it's kind of what you're looking for!
https://atmospherejs.com/package/auto-nprogress
I think user3374348's answer could lead to strange behavior if multiple methods are called.
I suggest:
increaseOngoingMethods=function(){
Session.set("ongoingMethods")=Session.get("ongoingMethods")+1;
}
decreaseOngoingMethods=function(){
Session.set("ongoingMethods")=Session.get("ongoingMethods")-1;
}
increaseOngoingMethods();
Meteor.call("doTheThing", decreaseOngoingMethods);
UI.registerHelper('Methodsareongoing', function() {
return !!Session.get("ongoingMethods")
});
{{#if Methodsareongoing}}
<p>Loading...</p>
{{/if}}
Note:
No paranthesis on the callback function in the methodcall.
No var in front of the declaration of the increase and decrease functions - that way they should become globally available.
Methodsareongoing is available in all templates due to the UI.registerHelper
EDIT: Andrew Mao's comment made me realize that I probably misunderstood the question!
If you want a 'saving' status showing one place this answer is very wrong...
This answer assumes you want to show something like:
John Smith saving
Where John Smith is part of the contextual object.
If that is not the case - please see my other answer
Original answer:
Have you tried using the 'isSimulation' property?
{{#if isSimulation}}
<b>Saving</b>
{{/if}}
If you methods are available on the client as well it should work.
If not - could you please add your methods to this thread
UPDATE:
You updated your post - simple CRUD operations should be able to run on the client.
If you try this but never see the 'saving' bit on your page it is because it is so fast :-/
In the good book Discover Meteor, to illustrate this technique they actully use a fancy delay method so the user is able to see the model simulation...
We're planning on rebuilding our service at my workplace, creating a RESTful API and such and I happened to stumble on an interesting question: can I make my JS code in a way that it mimics my API design?
Here's an example to illustrate what I mean:
We have dogs, and you can access those dogs doing a GET /dogs, and get info on a specific one by GET /dogs/{id}.
My Javascript code would then be something like
var api = {
dogs : function(dogId) {
if ( dogId === undefined ) {
//request /dogs from server
} else {
//request /dogs/dogId from server
}
}
}
All if fine and dandy with that code, I just have to call api.dogs() or api.dogs(123) and I'll get the info I want.
Now, let's say those dogs have a list of diseases (or whatever, really) which you can fetch via GET /dogs/{id}/disases. Is there a way to modify my Javascript so that the previous calls will remain the same - api.dogs() returns all dogs and api.dogs(123) returns dog 123's info - while allowing me to do something like api.dogs(123).diseases() to list dog 123's diseases?
The simplest way I thought of doing it is by having my methods actually build queries instead of retrieving the data and a get or run method to actually run those queries and fetch the data.
The only way I can think of building something like this is if I could somehow, when executing a function, if some other function is chained to the object, but I don't know if that's possible.
What are your thoughts on this?
I cannot give you a concrete implementation, but a few hints how you could accomplish what you want. It would be interesting to know, what kind of Server and framework you are using.
Generate (Write yourself or autogenerate from code) a WADL describing your Service and then try do generate the Code for example with XSLT
In my REST projects I use swagger, that analyzes some common Java REST Implementation and generates JSON descriptions, that you could use as a base for Your JavaScript API
It can be easy for simple REST Apis but gets complicated as the API divides into complex hierarchies or has a tree structure. Then everything will depend on an exact documentation of your service.
Assuming that your JS application knows of the services provided by your REST API i.e send a JSON or XML file describing the services, you could do the following:
var API = (function(){
// private members, here you hide the API's functionality from the outside.
var sendRequest = function (url){ return {} }; // send GET request
return {
// public members, here you place methods that will be exposed to the public.
var getDog = function (id, criteria) {
// check that criteria isn't an invalid request. Remember the JSON file?
// Generate url
response = sendRequest(url);
return response;
};
};
}());
var diseases = API.getDog("123", "diseases");
var breed = API.getDog("123", "breed");
The code above isn't 100% correct since you still have to deal with AJAX call but it is more or less what you what.
I hope this helps!