I'm trying to read some data from 200+ web pages with PhantomJS and typescript/rxjs
What I came up so far is this
Observable.fromPromise(phantom.createPage()).flatMap(jobPage => {
return Observable.fromPromise(jobPage.open(url)).flatMap(status => {
if (status !== "success") {
console.error("Couldn't load job page for url " + url + " Status: " + status);
jobPage.close();
return Observable.of(undefined)
} else {
return Observable.fromPromise(jobPage.evaluate(function () {
//do some content reading, return data
return data;
}));
}
});
})
And it works, but with every page it gets slower and slower, and finally ends with Memory Exhausted message from Phantom. I guess it's because I do not close the web pages I'm creating, but I dont have any idea how to do it such case (flatMap creates a new one, I need it for extraction later, and Observable.fromPromise() does not allow me to close the page after I'm done.
Any help is appreciated
Ok, figured it out, just need to use
Observable.fromPromise(phantom.createPage()).flatMap(jobPage => {
//stuff as before
}).finally(function(){
jobPage.close();
})
Related
sorry if this is a rookie question, but here it goes. Since electron doesn't support prompts I am trying to use [Smalltalk] (https://www.npmjs.com/package/smalltalk} to trigger one to insert an URL but it is not working, I have tried several options and still not working, here is the code:
// Insert Link //
const smalltalk = require('smalltalk');
function run(cmd, ele, value = null) {
let status = document.execCommand(cmd, true, value);
if (!status) {
switch (cmd) {
case 'insertLink':
smalltalk.prompt('Enter url')
.then((value) => {
console.log(value);
})
.catch(() => {
console.log('cancel');
});
if (value.slice(0, 4) != 'http') {
value = 'http://' + value;
}
document.execCommand('createLink', false, value);
// Overrides inherited attribute "contenteditable" from parent
// which would otherwise prevent anchor tag from being interacted with.
atag = document.getSelection().focusNode.parentNode;
atag.setAttribute("contenteditable", "false");
break;
}
}
}
Thanks in advance for your help!
in the end, I got it sorted by working with the 'insert link' functionality in javascript, as per this thread.
Clearly it was my limited knowledge of what was preventing me from getting this or other solutions to work on electron. It is all sorted now. Thanks again for your help and your time!
I'm using Chrome v67 and FileSaver.js.
This code works in FF & Edge but not in Chrome.
var ajaxSettings = {
url: "my/api/method",
data: JSON.stringify(myItems),
success: function (data) {
if (data) {
var dateToAppend = moment().format("YYYYMMDD-HHmmss");
createPdf("pdfDoc_" + dateToAppend + ".pdf", data[0]);
saveFile("txtDoc_" + dateToAppend + ".txt", data[1]);
//sleep(2000).then(() => {
// saveFile("txtDoc_" + dateToAppend + ".txt", data[1]);
//});
}
}
}
//function sleep(time) {
//return new Promise((resolve) => setTimeout(resolve, time));
//}
function createPdf(filename, pdfBytes) {
if (navigator.appVersion.indexOf("MSIE 9") > -1) {
window.open("data:application/pdf;base64," + encodeURIComponent(pdfBytes));
}
else {
saveFile(fileName, data);
}
}
function saveFile(fileName, data) {
var decodedByte64 = atob(data);
var byteVals = new Array(decodedByte64.length);
for (var i = 0; i < decodedByte64.length; i++) {
byteVals[i] = decodedByte64.charCodeAt(i);
}
var byte8bitArray = new Uint8Array(byteVals);
var blob = new Blob([byte8bitArray]);
saveAs(blob, fileName); //FileSaver.js
}
The result of calling the API is an array with 2 byte arrays in it.
The byte arrays are the documents.
If I run this code, as a user would, then what happens is that the first document gets "downloaded" but the second does not. Attempting to do this a second time without refreshing the page results in no documents being "downloaded". The "download" word is in quotes because it already has been downloaded, what I'm really trying to do is generate the documents from the byte arrays.
This is the strange bit ... if I open the console and place a breakpoint on the "saveFile" call and immediately hit continue when the debugger lands on the breakpoint then all is well with the world and the 2 documents get downloaded.
I initially thought it was a timing issue so I put a 2 second delay on this to see if that was it but it wasn't. The only thing I've managed to get working is the breakpoint which I'm obviously not going to be able to convince the users to start doing no matter how much I want them to.
Any help or pointers are much appreciated
You probably faced a message at the top left corner of your browser stating
https://yoursite.com wants to
Download multiple files
[Block] [Allow]
If you don't have it anymore, it's probably because you clicked on "Block".
You can manage this restriction in chrome://settings/content/automaticDownloads.
I'm trying to fetch some data, and THEN fetch some other data. Problem is, there's a loop with a timeout in the mix, which makes everything so complicated. And it doesn't work.
In the order, what I'm triying to do:
Loop through listOfGroups and fetch one or many get request to my backend
When that is done, fetch one time another get request
So, here's my code:
var i = 0;
var arrayOfPostURL = [];
isFinished = false;
while (isFinished == false)
if(i == listOfGroups.length) {
return fetch("/update_db_fcb_groups/" + theID + "/" + arrayOfPostURL).then((response) => {
response.json().then((data) => {
this.setState({onFcbGroups: arrayOfPostURL})
isFinished = true;
});
});
}
if(i<listOfGroups.length){
setTimeout(function(){
fetch("/post_to_fcb_group/" + listOfGroups[i] + "/" + theID).then((response) => {
// console.log(response);
response.json().then((data) => {
arrayOfPostURL.push("+" + data.url)
i++;
});
});
// console.log(arr[i])
},5000)
}
}
This code even freezes the browser (Google Chrome) !
Any ideas?
It looks like you're using a while loop when you could be using a for.
var arrayOfPostURL = [];
for (let group of listOfGroups) {
setTimeout(function() {
fetch("/post_to_fcb_group/" + group + "/" + theID).then((response) => {
response.json().then((data) => {
arrayOfPostURL.push("+" + data.url)
});
});
}, 5000)
}
fetch("/update_db_fcb_groups/" + theID + "/" + arrayOfPostURL).then((response) => {
response.json().then((data) => {
this.setState({onFcbGroups: arrayOfPostURL})
});
});
Breaking your code down like this reveals a couple other issues.
Your setTimeouts will all finish around the same time. You're just queueing a bunch of fetches that will each take place 5 seconds after they were queued. If you meant to wait 5 seconds between each fetch, this is not the way to do so.
Your final fetch concatenates an array into the URL. That will end up looking something like "/your/url/+url1,+url2". Is that what you intended? It's a fairly unusual URL schema. You might want to change that call to a POST or PUT and pass in a JSON array in the body.
Because you're calling setTimeout on all of your fetches, your final fetch will actually finish when the loop completes which could be before any of the other fetches execute. You likely want to use Promise.all or something similar.
I have implemented the browser push notification functionality and its working fine. I used this guide as the reference https://developers.google.com/web/fundamentals/getting-started/push-notifications/step-01?hl=en
However as payload is still not supported, I decided to query my server to get the notification data for each user which is also working fine.
There is one issue though. For some cases, after getting data from the server, I want to control whether to show the notification or not. I am not able to figure out how to do this. I tried returning false, throwing errors etc. But is always shows the default notification even if I don't call showNotification method. Let me know how to solve this. Following is the relevant code
self.addEventListener('push', function(event) {
event.waitUntil(
fetch('/getPushNotificationData/').then(function(response){
if (response.status !== 200) {
// I don't want to show any notification in this case
console.log('Looks like there was a problem. Status Code: ' + response.status);
throw new Error();
}
return response.json().then(function(data){
var shouldDisplay = data.shouldDisplay;
if (shouldDisplay=='1'){
var title = data.title;
var message = data.message;
var url = data.url;
return self.registration.showNotification(title, {
body: message,
data: url
});
}
else{
// I don't want to show any notification in this case also
return true;
}
});
})
);
});
I have a little JavaScript XMPP client, written with Strophe, that connects to a server hosted on hosted.im. I think hosted.im uses ejabberd on their backend.
I establish the connection using
Strophe.Connection(myBoshService), and am able to send chat messages back and forth. However, after a certain time, it seems, there is an automatic disconnect if there is no activity.
Now, my question is, what would be a good way to keep the session active, so that it does not disconnect. Disconnect time seems to be very short, about 60 seconds or so.
Should I send some kind of activity back and forth to keep it open? Or, which seems simpler to me, should I somehow change the timout of the session. If so, where can I change this? Is this a server-setting, irregardless of the Strophe.Connection object, or can I set the timeout when initializing Strophe.Connection?
Thanks for any and all help.
Best regards,
Chris
Edit: Here is the code I use for connecting:
I manage the connection through a global variable Hello (yes, name is awkward, I took it from an example):
var Hello = {
connection: null,
start_time: null,
partner: {
jid: null,
name: null
},
log: function (msg) {
$('#log').append("<p>" + msg + "</p>");
},
send_ping: function (to) {
var ping = $iq({
to: to,
type: "get",
id: "ping1"}).c("ping", {xmlns: "urn:xmpp:ping"});
Hello.log("Sending ping to " + to + ".");
console.log("Sending ping to " + to + ".");
Hello.start_time = (new Date()).getTime();
Hello.connection.send(ping);
},
handle_pong: function (iq) {
var elapsed = (new Date()).getTime() - Hello.start_time;
Hello.log("Received pong from server in " + elapsed + "ms.");
console.log('Received pong from server in " + elapsed + "ms.');
$('#login').hide();
$('#chat').show();
//window.location = "chat.html";
//Hello.connection.disconnect();
return true;
},
//"<active xmlns="http://jabber.org/protocol/chatstates"/><body xmlns="http://jabber.org/protocol/httpbind">tuiuyi</body>"
displayIncomingText: function (text) {
var body = $(text).find("xml > body");
if (body.length === 0)
{
body = $(text).find('body');
if (body.length > 0)
{
body = body.text();
$('#chattext').append("<p>"+ body + "</p>");
}
else
{
body = null;
}
}
return true;
},
readRoster: function (iq) {
$(iq).find('item').each(function () {
var jid = $(this).attr('jid');
var name = $(this).attr('name') || jid;
Hello.partner.name = name;
Hello.partner.jid = jid;
});
return true;
}
};
The main relevant objects here are Hello.connect and Hello.partner, which stores the jid of the only person on the accounts roster, as this is a one to one chat.
Then, in $(document).ready, I bind two buttons to connect and send messages respectively:
$(document).ready(function () {
$('#chat').hide();
$('#chatSend').bind('click', function () {
Hello.connection.send(
$msg(
{to : Hello.partner.jid, type : 'chat'}
).c('body').t($('#chattextinput').val())
);
$('#chattext').append("<p align='right'>" + $('#chattextinput').val() + "</p>");
});
$('#SignIn').bind('click', function () {
$(document).trigger('connect', {
jid: $('#eMail').val(), password: $('#password_f').val()
}
);
});
});
Clicking the sign-in button triggers the event "connect":
$(document).bind('connect', function (ev, data) {
console.log('connect fired');
var conn = new Strophe.Connection("http://bosh.metajack.im:5280/xmpp-httpbind");
conn.connect(data.jid, data.password, function (status) {
console.log('callback being done');
if (status === Strophe.Status.CONNECTED) {
alert('connected!');
$(document).trigger('connected');
alert('Connected successfully');
} else if (status === Strophe.Status.DISCONNECTED) {
$(document).trigger('disconnected');
}
else
{
Hello.log("error");
console.log('error');
}
});
Hello.connection = conn;
});
This creates the Strophe.Connection and stores it in Hello.connection. Also, it sets the callback function of the connection object. This code is taken straight from an example in a Strophe.js book. Anyway, the callback checks the status of the connection, and if status === Strophe.Status.DISCONNECTED, triggers "disconnected", which only does this:
$(document).bind('disconnected', function () {
Hello.log("Connection terminated.");
console.log('Connection terminated.');
// remove dead connection object
Hello.connection = null;
});
Anyway, what is happening is that, for some reason, in the callback set with conn.connect, after a short time, the status evaluates to Strophe.Status.DISCONNECTED, and I am not sure why, unless somewhere, either in the server or in the connection object, there is a timeout specified which seems to be ca. 60 seconds.
As to a log of the stanzas going back and forth, I guess I would need to quickly write a handler to see all incoming stanzas, or is it possible to see a log of all stanzas between the client and server in ejabberd?
For the sake of other people who come upon this and have a similar problem, the solution in this case was that the servers at hosted.im send a ping request every 60 seconds to check if the client is still online.
This ping request looks like this:
<iq from="testserver.p1.im" to="chris#testserver.p1.im/23064809721410433741569348" id="164323654" type="get"> <ping xmlns="urn:xmpp:ping"></ping> </iq>
What is needed, of course, is to form a response, which will look something like this:
<iq from="chris#testerver.p1.im" to="testserver.p1.im" id="164323654" type="result" xmlns="jabber:client"><ping xmlns="urn:xmpp:ping"/></iq>
Note the "to"-attribute. I omitted it at the beginning as I was under the assumption a message sent with no to-attribute is automatically assumed to be a client->server message. Not in this case however. Not sure if this is the case in general, or whether it is an oddity of servers at hosted.im.
Thanks to everyone for their comments and suggestions!
Best regards,
Chris