We Keep Coding

Express server that creates maximum of 2 child/worker processes

I'm experimenting with node and it's child_process module.
My goal is to create server which will run on maximum of 3 processes (1 main and optionally 2 children).
I'm aware that code below may be incorrect, but it displays interesting results.
const app = require ("express")();
const {fork} = require("child_process")
const maxChildrenRuning = 2
let childrenRunning = 0
app.get("/isprime", (req, res) => {
if(childrenRunning+1 <= maxChildrenRuning) {
childrenRunning+=1;
console.log(childrenRunning)
const childProcess = fork('./isprime.js');
childProcess.send({"number": parseInt(req.query.number)})
childProcess.on("message", message => {
console.log(message)
res.send(message)
childrenRunning-=1;
})
}
})
function isPrime(number) {
...
}
app.listen(8000, ()=>console.log("Listening on 8000") )
I'm launching 3 requests with 5*10^9'ish numbers.
After 30 seconds I receive 2 responses with correct results.
CPU stops doing hard work and goes idle
Surprisingly after next 1 minute 30 seconds 1 thread starts to proceed, still pending, 3rd request and finishes after next 30 seconds with correct answer. Console log displayed below:
> node index.js
Listening on 8000
1
2
{ number: 5000000029, isPrime: true, time: 32471 }
{ number: 5000000039, isPrime: true, time: 32557 }
1
{ number: 5000000063, isPrime: true, time: 32251 }
Either express listens and checks pending requests once for a while or my browser sends actual requests every x time while pending. Can anybody explain what is happening here and why? How can I correctly achieve my goal?

The way your server code is written, if you receive a /isprime request and two child processes are already running, your request handler for /isprime does nothing. It never sends any response. You don't pass that first if test and then nothing happens afterwards. So, that request will just sit there with the client waiting for a response. Depending upon the client, it will probably eventually time out as a dead/inactive request and the client will shut it down.
Some clients (like browsers) may assume that something just got lost in the network and they may retry the request by sending it again. It would be my guess that this is what is happening in your case. The browser eventually times out and then resends the request. By the time it retries, there are less than two child processes running so it gets processed on the retry.
You could verify that the browser is retrying automatically by going to the network tab in the Chrome debugger and watching exactly what the browser sends to your server and watch that third request, see it timeout and see if it is the browser retrying the request.
Note, this code seems to be only partially implemented because you initially start two child processes, but you don't reuse those child processes. Once they finish and you decrement maxChildrenRuning, your code will then start another child process. Probably what you really want to do is to keep track of the two child processes you started and when one finishes, add it to an array of "available child processes" so when a new request comes in, you can just use an existing child process that is already started, but idle.
You also need to either queue incoming requests when all the child processes are full or you need to send some sort of error response to the http request. Never sending an http response to an incoming request is a poor design that just leads to great inefficiencies (connections hanging around much longer than needed that never actually accomplish anything).

Scraping a rendered javascript webpage

I'm trying to build a short Python program that extracts Pewdiepie's number of subscribers which is updated every second on socialblade to show it in the terminal. I want this data like every 30 seconds.
I've tried using PyQt but it's slow, i've turned to dryscrape, slightly faster but doesn't work either as I want it to. I've just found Invader and written some short code that still has the same problem : the number returned is the one before the Javascript on the page is executed :
from invader import Invader
url = 'https://socialblade.com/youtube/user/pewdiepie/realtime'
invader = Invader(url, js=True)
subscribers = invader.take(['#rawCount', 'text'])
print(subscribers.text)
I know that this data is accessible via the site's API but it's not always working, sometimes it just redirect to this.
Is there a way to get this number after the Javascript on the page modified the counter and not before ? And which method seems the best to you ? Extract it :
from the original page which always returns the same number for hours ?
from the API's page which bugs when not using cookies in the code and after a certain amount of time ?
Thanks for your advices !

If you want scrape a web page that has parts of it loaded in by javascript you pretty much need to use a real browser.
In python this can be achieved with pyppeteer:
import asyncio
from pyppeteer import launch
async def main():
browser = await launch(headless=False)
page = await browser.newPage()
await page.goto('https://socialblade.com/youtube/user/pewdiepie/realtime',{
'waitUntil': 'networkidle0'
})
count = int(await page.Jeval('#rawCount', 'e => e.innerText'))
print(count)
asyncio.get_event_loop().run_until_complete(main())
Note: It does not seems like the website you mentioned above is updating the subscriber count frequently any more (even with JavaScript). See: https://socialblade.com/blog/abbreviated-subscriber-counts-on-youtube/
For best success and reliability you will probably need to set the user agent(page.setUserAgent in pyppeteer) and keep it up to date and use proxies (so your ip does not get banned). This can be a lot of work.
It might be easier and cheaper (in time and than buying a large pool of proxies) to use a service that will handle this for you like Scraper's Proxy. It supports will use a real browser and return the resulting html after the JavaScript has run and route all of our requests through a large network of proxies, so you can sent a lot of requests without getting you ip banned.
Here is an example using the Scraper's Proxy API getting the count directly from YouTube:
import requests
from pyquery import PyQuery
# Send request to API
url = "https://scrapers-proxy2.p.rapidapi.com/javascript"
params = {
"click_selector": '#subscriber-count', # (Wait for selector work-around)
"wait_ajax": 'true',
"url":"https://www.youtube.com/user/PewDiePie"
}
headers = {
'x-rapidapi-host': "scrapers-proxy2.p.rapidapi.com",
'x-rapidapi-key': "<INSERT YOUR KEY HERE>" # TODO
}
response = requests.request("GET", url, headers=headers, params=params)
# Query html
pq = PyQuery(response.text)
count_text = pq('#subscriber-count').text()
# Extract count from text
clean_count_text = count_text.split(' ')[0]
clean_count_text = clean_count_text.replace('K','000')
clean_count_text = clean_count_text.replace('M','000000')
count = int(clean_count_text)
print(count)
I know this is a bit late, but I hope this helps

Preventing client side abuse/cheating of repeating Ajax call that rewards users

I am working on a coin program to award the members for being on my site. The program I have makes two random numbers and compares them, if they are the same, you get a coin. The problem I have is someone could go in the console and get "free" coins. They could also cheat by opening more tabs or making a program to generate more coins right now which I am trying to stop. I am thinking about converting it over to php from js to stop the cheating (for the most part) but I don't know how to do this. The code in question is:
$.ajax({
type: 'post',
url: '/version2.0/coin/coins.php',
data: {Cid : cs, mode : 'updateCoins'},
success: function (msg) {
window.msg=msg;
}});
And the code for the console is that with a loop around it. In the code above, "cs" is the id of the member so by replacing it with their id would cause them to get all the coins they would want.
Should I just have an include with variable above it? But then how would I display the success message which has the current number of coins. Also, this code is in a setInterval function that repeats every 15 milliseconds.

There are multiple ways you could do this, but perhaps the simplest would be to go in your server side code - when a request comes in, you check the time of last coin update, if there ins't one, you run your coin code and save the time of this operation in their session. If there is a stored time, ensure that it is beyond the desired time. If it is, continue to the coin update. If it isn't, simply respond with a 403 or other failure code.
In pseudo code:
if (!$userSession['lastCoinTime'] || $currentTime + $delay > $userSession['lastCoinTime']) {
// coin stuff
$userSession['lastCoinTime'] = // new time
} else {
// don't give them a chance at coin, respond however you want
}
However, since you're talking about doing this check every 15ms, I would use websockets so that the connection to the server is ongoing. Either way, the logic can be comparable.
Just in case there's any uncertainty about this, definitely do ALL of the coin logic on the server. You can never trust the user for valid data coming in. The most you can trust, depending on how your authentication is setup, is some kind of secret code only they would have that would just let you know who they are, which is a technique used in place of persistent sessions. Unless you're doing that, you would rely on the session to know who the user is - definitely don't let them tell you that either!

Check/Log how much bandwidth PhantomJS/CasperJS used

Is it possible to check/log how much data has been transferred during each run of PhantomJs/CasperJS?
Each instance of Phantom/Casper has a instance_id assigned to it (by the PHP function that spun up the instance). After the run has finished, the amount of data transferred and the instance_id will have to make its way to be inserted into a MySQL database, possibly via the PHP function that spawned the instance. This way the bandwidth utilization of individual phantomjs runs can be logged.
There can be many phantom/casper instances running, each lasting a minute or two.

The easiest and most accurate approach when trying to capture data is to get the collector and emitter as close as possible. In this case it would be ideal if phantomjs could capture that data that you need and send it back to your PHP function to associate it to the instance_id and do the database interaction. Turns out it can (at least partially).
Here is one approach:
var page = require('webpage').create();
var bytesReceived = 0;
page.onResourceReceived = function (res) {
if (res.bodySize) {
bytesReceived += res.bodySize;
}
};
page.open("http://www.google.com", function (status) {
console.log(bytesReceived);
phantom.exit();
});
This captures the size of all resources retrieved, adds them up, and spits out the result to standard output where your PHP code is able to work with it. This does not include the size of headers or any POST activity. Depending upon your application, this might be enough. If not, then hopefully this gives you a good jumping off point.

setInterval alternative

In my app I am polling the webserver for messages every second and displaying them in the frontend.
I use setInterval to achieve this. However as long as the user stays on that page the client keeps polling the server with requests even if there is no data. The server does give an indication when no more messages are being generated by setting a variable.
I thought of using this variable to clearInterval and stop the timer but that didn't work. What else can I use in this situation?
I am using jquery and django. Here is my code:
jquery:
var refresh = setInterval(
function ()
{
var toLoad = '/myMonitor'+' #content';
$('#content').load(toLoad).show();
}, 1000); // refresh every 1000 milliseconds
});
html:
div id=content is here
I can access the django variable for completion in html with each refresh. How can I set clearInterval if at all ?
Note: stack overflow does not let me put is &gt &lt so html is incomplete
Thanks
Updated 03/16/2010
I must be doing something wrong. But cannot figure it out. Here is my script with clearTimer and it does not work.
var timer = null;
$(function(){
if ("{{status}}" == "False")
{
clearInterval(timer);
}
else
{
timer = setInterval(
function(){
var toLoad = '/myMonitor'+' #content';
$('#content').load(toLoad).show();}
,1000); // refresh every 1000 milliseconds
}
});
status is a boolean set in "views.py" (Django).
Thanks a bunch.

A couple people have already answered with specific resources to your problem, so I thought I would provide a bit of background.
In short, you want the server to push data to the browser to avoid extensive client-side polling. There isn't a good cross-browser way to support server push, so a common solution that requires much less polling is to use the Comet (another cleaning product, like AJAX) long-poll technique.
With Comet, the browser makes a request, and the server keeps the connection open without responding until new data is available. When the server does has new data, it sends it over the open connection and the browser receives it right away. If the connection times out, the browser opens a new one. This lets the server send data to the client as soon as it becomes available. As others have indicated, this approach requires special configuration of your web server. You need a script on the server that checks for data at an interval and responds to the client if it exists.
Something to keep in mind with this approach is that most web servers are built to get a request from a client and respond as quickly as possible; they're not intended to be kept alive for a long period of time. With Comet you'll have far more open connections than normal, probably consuming more resources than you expect.

Your clearInterval check is only checking when the document ready event is fired.
If the code you gave is exactly what's in the browser, then you're comparing the string "{{status}}" to the string "False". I'd rather watch paint dry than wait for that to evaluate as true.
What if your requests taking longer than 1 second to complete? : You'll flood your server with requests.
function update () {
$('#content').show().load('/myMonitor'+' #content', function (response, status) {
if (!/* whatever you're trying to check*/) {
setTimeout(update, 1000);
};
});
};
$(document).ready(function () {
update();
});
Is closer than where you were, but you still need to work out how you're going to decide when you want to stop polling.