We Keep Coding

Retrieve html content of a page several seconds after it's loaded

I'm coding a script in nodejs to automatically retrieve data from an online directory.
Knowing that I had never done this, I chose javascript because it is a language I use every day.
I therefore from the few tips I could find on google use request with cheerios to easily access components of dom of the page.
I found and retrieved all the necessary information, the only missing step is to recover the link to the next page except that the one is generated 4 seconds after loading of page and link contains a hash so that this step Is unavoidable.
What I would like to do is to recover dom of page 4-5 seconds after its loading to be able to recover the link
I looked on the internet, and much advice to use PhantomJS for this manipulation, but I can not get it to work after many attempts with node.
This is my code :
#!/usr/bin/env node
require('babel-register');
import request from 'request'
import cheerio from 'cheerio'
import phantom from 'node-phantom'
phantom.create(function(err,ph) {
return ph.createPage(function(err,page) {
return page.open(url, function(err,status) {
console.log("opened site? ", status);
page.includeJs('http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js', function(err) {
//jQuery Loaded.
//Wait for a bit for AJAX content to load on the page. Here, we are waiting 5 seconds.
setTimeout(function() {
return page.evaluate(function() {
var tt = cheerio.load($this.html())
console.log(tt)
}, function(err,result) {
console.log(result);
ph.exit();
});
}, 5000);
});
});
});
});
but i get this error :
return ph.createPage(function (page) {
^
TypeError: ph.createPage is not a function
Is what I am about to do is the best way to do what I want to do? If not what is the simplest way? If so, where does my error come from?

If You dont have to use phantomjs You can use nightmare to do it.
It is pretty neat library to solve problems like yours, it uses electron as web browser and You can run it with or without showing window (You can also open developer tools like in Google Chrome)
It has only one flaw if You want to run it on server without graphical interface that You must install at least framebuffer.
Nightmare has method like wait(cssSelector) that will wait until some element appears on website.
Your code would be something like:
const Nightmare = require('nightmare');
const nightmare = Nightmare({
show: true, // will show browser window
openDevTools: true // will open dev tools in browser window
});
const url = 'http://hakier.pl';
const selector = '#someElementSelectorWitchWillAppearAfterSomeDelay';
nightmare
.goto(url)
.wait(selector)
.evaluate(selector => {
return {
nextPage: document.querySelector(selector).getAttribute('href')
};
}, selector)
.then(extracted => {
console.log(extracted.nextPage); //Your extracted data from evaluate
});
//this variable will be injected into evaluate callback
//it is required to inject required variables like this,
// because You have different - browser scope inside this
// callback and You will not has access to node.js variables not injected
Happy hacking!

Electron - Processing Input

I recently started to get my feet wet with Electron. I really like the principles behind it but I find it a little confusing to do some things.
For example, how do you process user input? I've an main.js and a BrowserWindow pointing to a local html file (containing some user settings with input field).
How do I access this data when the HTML form is submitted (to either the same file or another one)?
main.js
const {app, BrowserWindow} = require('electron')
let win
function createWindow () {
win = new BrowserWindow({width: 800, height: 600})
win.loadURL('file://' + __dirname + '/index.html')
// Emitted when the window is closed.
win.on('closed', () => {
win = null
})
// Open the DevTools.
// win.webContents.openDevTools()
}
app.on('window-all-closed', () => {
if (process.platform !== 'darwin') {
app.quit()
}
})
app.on('activate', () => {
if (win === null) {
createWindow()
}
})
// In this file you can include the rest of your app's specific main process
// code. You can also put them in separate files and require them here.
//Start the main window
app.on('ready', createWindow)
index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Document</title>
</head>
<body>
<form action="" method="post">
<input type="text" name="test-1">
</form>
</body>
</html>

With Electron, node.js is not acting as a webserver with routes like it would be in a typical web application scenario. Instead of sending requests to routes, you would create a single page application using a javascript framework like Angular, React, Knockout, etc. At that point, you no longer need to handle routing. You would tie your 'Submit' click event to a javascript function directly within the page, and process the input from there.
You can do everything from the page's javascript context that you can do from the node.js main process context. For instance, if you needed to access the file system from your page, you would use the Remote module to gain access to the node.js native APIs.
For example:
// Gain access to the node.js file system api
function useNodeApi() {
const remote = require('electron').remote;
const fs = remote.require('fs');
fs.writeFile('test.txt', 'Hello, I was written by the renderer process!');
}
I've rarely come across a situation where I needed to pass control back to the main process to accomplish something. Once the BrowserWindow launches, anything you could ever need to do could be done from the renderer process. This pretty much eliminates the need to do things like submit form posts via http.

ReferenceError while using sdk/tabs in firefox webextension

This is my first time learning to build a firefox addon. I want store all the open tabs in a window and for that I require sdk/tabs.
Here is my js file:
/*
Given the name of a beast, get the URL to the corresponding image.
*/
debugger;
var tabs = require("sdk/tabs");
function beastNameToURL(beastName) {
switch (beastName) {
case "Save Session":
debugger;
for (let tab of tabs)
console.log(tab.url);
return;
case "Load Session":
debugger;
return chrome.extension.getURL("beasts/snake.jpg");
case "Turtle":
return chrome.extension.getURL("beasts/turtle.jpg");
}
}
/*
Listen for clicks in the popup.
If the click is not on one of the beasts, return early.
Otherwise, the text content of the node is the name of the beast we want.
Inject the "beastify.js" content script in the active tab.
Then get the active tab and send "beastify.js" a message
containing the URL to the chosen beast's image.
*/
document.addEventListener("click", function(e) {
if (!e.target.classList.contains("btn")) {
return;
}
var chosenBeast = e.target.textContent;
var chosenBeastURL = beastNameToURL(chosenBeast);
chrome.tabs.executeScript(null, {
file: "/content_scripts/beastify.js"
});
chrome.tabs.query({active: true, currentWindow: true}, function(tabs) {
chrome.tabs.sendMessage(tabs[0].id, {beastURL: chosenBeastURL});
});
});
When I reach the var tabs = require("sdk/tabs") line I get a Reference error.
Github : https://github.com/sagar-shah/Session-manifest
Kindly let me know how do I resolve this error. This being my first time with add-ons I am completely lost.
Thanks in advance.
Update:
Tried to declare it globally in the js file. Now I am getting undefined error for tabs.
Update2:
I was mixing up development using sdk and webextensions as pointed out by #matagus. I have decided to go with development using the webextensions. Link to the new repository has been updated.

The error is on package.json line 6: you're telling to the addon sdk that the main file of your addon is manage.json. According to [the docs] the value of main should be:
A string representing the name of a program module that is located in one of the top-level module directories specified by lib. Defaults to "index.js".
So you need to change its value to index.js.
Besides that, I think you're missing a difference between Firefox addon built using the addon-sdk (which do not have a ´manifest.json´ and that you build using jpm tool) and the new WebExtensions which do require you to write a ´manifest.json´ like the one already have.
UPDATE:
Again: you're missing the difference between WebExtensions and SDK-based addons. Now you made a WebExtension but you're trying to use the SDK. It isn't possible. Just use chrome.tabs directly instead of trying to import it from the sdk (var tabs = require("sdk/tabs");).

Retry loading a webpage when failed in electron

I'm using electron to display some webpages. Below is my coding:
var app = require('app');
var ipc = require('ipc');
var BrowserWindow = require('browser-window');
var settings = require('./settings');
var mainWindow = null;
app.on('window-all-closed', function() {
if (process.platform != 'darwin') {
app.quit();
}
});
app.on('ready', function(){
var mainWindow = new BrowserWindow({
fullscreen: true,
autoHideMenuBar: true
})
mainWindow.loadUrl('file://' + __dirname + '/index.html') // FIRST WEBPAGE
mainWindow.on('closed', function() {
mainWindow = null;
});
ipc.on('redirect', function(){
mainWindow.loadUrl('http://192.168.1.10/page2') // SECOND WEBPAGE
mainWindow.webContents.on("did-fail-load", function() {
console.log("did-fail-load");
mainWindow.loadUrl('file://' + __dirname + '/index.html');
// REDIRECT TO FIRST WEBPAGE AGAIN
});
});
It will first go into the first webpage, then after it received a command "redirect" from the javascript using ipc module, it will redirect to a second webpage. But I need to check whether or not the second webpage can be connected. If it cannot be connected (did-fail-load), it will go to the first webpage again. And the cycles goes on.
I use the console.log("did-fail-load") to check whether or not it had failed to connect to the second page. But I found out that it had duplicated the call. The first time it fail to connect to second webpage, there is one console.log("did-fail-load"), when it retry the second time, there is two console.log("did-fail-load") appear, and the 3rd time it retry, there is three console.log("did-fail-load") appear. Is it that it some how got duplicated calls on mainWindow?
What is the best way to retry loading a webpage when it failed in electron?

This is an old post, but I feel the question was never actually answered for OP.
You are seeing multiple console.log messages because a new did-fail-load callback is added everytime a redirect happens. You need to add the callback only once, outside of the ipc.on('redirect') callback. Example:
app.on('ready', function(){
var mainWindow = new BrowserWindow({
fullscreen: true,
autoHideMenuBar: true
})
mainWindow.loadUrl('file://' + __dirname + '/index.html') // FIRST WEBPAGE
mainWindow.on('closed', function() {
mainWindow = null;
});
/* Set did-fail-load listener once */
mainWindow.webContents.on("did-fail-load", function() {
console.log("did-fail-load");
mainWindow.loadUrl('file://' + __dirname + '/index.html');
});
});
/* This is called every time a redirect occurs,
* so don't add any listeners here. Only add code
* to handle the redirect
*/
ipc.on('redirect', function(){
mainWindow.loadUrl('http://192.168.1.10/page2') // SECOND WEBPAGE
});

You have to ask yourself the question: "Why did the load fail?" and "Would it load now?"
The why defines the best way how to check if your webpage would load. When you check the url/server before you load you make sure that it can be loaded. Then it won't be a need to reload.
From your code I would guess, that you want to check if the server inside your network is running.
To do that you could use the node module node-net-ping https://github.com/stephenwvickers/node-net-ping
Install the module via npm inside your app:
npm install node-net-ping --save
Load the module on top via require:
var ping = require ("net-ping");
Check if their server is available:
var session = ping.createSession ();
session.pingHost ('192.168.1.10', function (error, target) {
if (!error)
// Load second page
});
Another maybe better solution is to check the request before you load the url. This is also done with the node.js part of electron. Answer is copied from here:
Node.js - How to check status of a URL within a http request
var request = require('request');
request('http://www.google.com', function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log("URL is OK")
// Load second page
}
Depending on the answer to "Why did the load fail?" you should create the check.

How do run reload() and open() in Atom-Shell

1) Is this javascript call not supported on Atom-shell?
window.open('#/openpage','_self',false)
2) In NodeJS-Webkit, I could either reload the HTML with win.reload() without the toolbar and buttons. Is Atom-shell support this API as well?
3) In the app.js that is program to index.html into the Atom-shell
var BrowserWindow = require('browser-window');
and index.html tried to execute a reload command and failed that BrowserWindow is not defined.
BrowserWindow.reload()

browser-window only works in the Browser process (i.e. the one your app starts in). Try this:
var remote = require('remote');
remote.getCurrentWindow().reload();

For Reload :
const { BrowserWindow } = require('electron').remote
BrowserWindow.getCurrentWindow().reload();
For Close:
BrowserWindow.getCurrentWindow().on('close', () => {
// window was closed...
})
Documentation :
https://www.electronjs.org/docs/api/remote