So I created a desktop electron application with javascript html css etc I have a bot that I want to run when a button is clicked by the user the bot is written in python. what the bot does is web scraping using selenium and chrome driver im just wondering is there a way where I could store the bot and its source code outside the clients computer so the source code is not visible and still give the client the ability to use the bot to webscrape.
sorry if this is a rookie question im coming from c++ & swift mobile development and im a junior CS student so im just teaching myself new stuff.
I agree with Chris G in that it would be considered best practice to create a web app with one of Python's many web frameworks (Django, FastAPI, Flask, etc).
Alternatively, with the python-shell package this can be done quite simply with electron:
const { app, BrowserWindow } = require('electron');
const pyshell = require('python-shell')
function createWindow() {
window = new BrowserWindow({ width: 600, height: 450 });
window.loadFile('index.html');
pyshell.run('your_script.py', function (err, results) {
if (err) {
throw err;
}
});
}
app.on('ready', createWindow);
app.on('window-all-closed', () => {
if (process.platform !== 'darwin') {
app.quit()
}
}
Source
Then, with a simple python script your_script.py:
a = 'Foo'
b = 'Bar'
print(a + b)
This example is quite simple. Creating your own web facing API would be your best bet if you don't want to run into any compatibility issues when shipping your app.
Related
I've been worked on a vue project.
This vue project use the nodejs API I've created, in simple way, they are two entire differents project which are not located in the same directory and they are launched separately.
The problem is whenever I debug a route with node --inspect --debug-break event_type.controller.js for example named:
"/eventtype/create"
exports.create = (req, res) => {
const userId = jwt.getUserId(req.headers.authorization);
if (userId == null) {
res.status(401).send(Response.response401());
return;
}
// Validate request
if (!req.body.label || !req.body.calendarId) {
res.status(400).send(Response.response400());
return;
}
const calendarId = req.body.calendarId; // Calendar id
// Save to database
EventType.create({
label: req.body.label,
}).then((eventType) => {
Calendar.findByPk(calendarId).then((calendar) => {
eventType.addCalendar(calendar); // Add a Calendar
res.status(201).send(eventType);
}).catch((err) => {
res.status(500).send(Response.response500(err.message));
});
}).catch((err) => {
res.status(500).send(Response.response500(err.message));
});
};
Even if I create a breakpoint on const userId = jwt.getUserId(req.headers.authorization);
and from my vue app I trigger the api createEventType event, my break point is not passed.
Also when I press f8 after the breakpoint on my first line with the debugger, my file close automatically.
I do not use VS Code but Vim for coding but I've heard that maybe Vs Code could allow a simplified way to debug nodesjs application.
NOTE: I use the V8 node debugger.
For newer NodeJS versions (> 7.0.0) you need to use
node --inspect-brk event_type.controller.js
instead of
node --inspect --debug-break event_type.controller.js
to break on the first line of the application code. See https://nodejs.org/api/debugger.html#debugger_advanced_usage for more information.
The solution (even if it's not really a solution) has been to add console.log to the line I wanted to debug.
I want to do some event eg. clicks in a website. I can do it in chrome with javascript (or chrome extension), but is it possible to do without opening chrome but with server side code? No API is provided. It's not scraping but perform some sort of action.
NodeJS uses Google V8 engine to interpret the JavaScript code. It does not run in a browser environment and therefore it lacks DOM and event handling. However, you can actually mock browser in NodeJS environment using mock-browser package.
const MockBrowser = require('mock-browser/lib/MockBrowser')
const mockBrowser = new MockBrowser()
global.window = mockBrowser.getWindow()
global.document = mockBrowser.getDocument()
global.navigator = mockBrowser.getNavigator()
However, you should be careful with this approach, as some methods (e.g. getComputedStyle) still will not work.
Maybe you should reconsider why you want to use DOM and events on the server side.
PhantomJS: Headless browser for NodeJS
PhantomJS is a headless browser for NodeJS that is used for testing, scraping, etc. It provides you with a full-featured browser that can simulate a browser.
Using CasperJS for scraping
If you want to scrape websites, you may use a library called CasperJS that itself uses PhantomJS. An example:
var casper = require('casper').create();
var links;
function getLinks() {
// Scrape the links from top-right nav of the website
var links = document.querySelectorAll('ul.navigation li a');
return Array.prototype.map.call(links, function (e) {
return e.getAttribute('href')
});
}
// Opens casperjs homepage
casper.start('http://casperjs.org/');
casper.then(function () {
links = this.evaluate(getLinks);
});
casper.run(function () {
for(var i in links) {
console.log(links[i]);
}
casper.done();
});
I recently started to get my feet wet with Electron. I really like the principles behind it but I find it a little confusing to do some things.
For example, how do you process user input? I've an main.js and a BrowserWindow pointing to a local html file (containing some user settings with input field).
How do I access this data when the HTML form is submitted (to either the same file or another one)?
main.js
const {app, BrowserWindow} = require('electron')
let win
function createWindow () {
win = new BrowserWindow({width: 800, height: 600})
win.loadURL('file://' + __dirname + '/index.html')
// Emitted when the window is closed.
win.on('closed', () => {
win = null
})
// Open the DevTools.
// win.webContents.openDevTools()
}
app.on('window-all-closed', () => {
if (process.platform !== 'darwin') {
app.quit()
}
})
app.on('activate', () => {
if (win === null) {
createWindow()
}
})
// In this file you can include the rest of your app's specific main process
// code. You can also put them in separate files and require them here.
//Start the main window
app.on('ready', createWindow)
index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Document</title>
</head>
<body>
<form action="" method="post">
<input type="text" name="test-1">
</form>
</body>
</html>
With Electron, node.js is not acting as a webserver with routes like it would be in a typical web application scenario. Instead of sending requests to routes, you would create a single page application using a javascript framework like Angular, React, Knockout, etc. At that point, you no longer need to handle routing. You would tie your 'Submit' click event to a javascript function directly within the page, and process the input from there.
You can do everything from the page's javascript context that you can do from the node.js main process context. For instance, if you needed to access the file system from your page, you would use the Remote module to gain access to the node.js native APIs.
For example:
// Gain access to the node.js file system api
function useNodeApi() {
const remote = require('electron').remote;
const fs = remote.require('fs');
fs.writeFile('test.txt', 'Hello, I was written by the renderer process!');
}
I've rarely come across a situation where I needed to pass control back to the main process to accomplish something. Once the BrowserWindow launches, anything you could ever need to do could be done from the renderer process. This pretty much eliminates the need to do things like submit form posts via http.
I am working on web scraping for few task to complete.
I have used node-js request module for page scraping.
It is working fine and great for cookie-session and all.
But it fails when time comes to render Dynamic pages build with some javascript framework like ANGULAR or BACKBONE etc.
I am trying for phantomjs to overcome this thing as i found on google that it is helpful to come over such case.
I also found one nodejs bridge for phantomjs phantom
With phantomjs and this bridge module i am able to achieve same thing nothing more.
var phantom = require('phantom');
var fs = require('fs');
var sitepage = null;
var phInstance = null;
phantom.create()
.then(instance => {
phInstance = instance;
console.log("Instance created");
return instance.createPage();
})
.then(page => {
sitepage = page;
console.log("createing page");
return page.open('https://paytm.com/shop/p/carrier-estrella-plus-1-5-ton-3-star-window-ac-LARCARRIER-ESTRPLAN5550519593A34?src=grid&tracker=%7C%7C%7C%7C%2Fg%2Felectronics%2Flarge-appliances%2F1-5-ton-3-star-ac-starting-at-rs-22699%7C88040%7C1');
})
.then(status => {
//console.log(status);
console.log("getting content of page");
return sitepage.property('content');
})
.then(content => {
console.log("success");
//console.log(content);
fs.writeFile("ok.text", content);
sitepage.close();
phInstance.exit();
})
.catch(error => {
console.log("errr");
//console.log(error);
phInstance.exit();
});
Above is code which i am trying for load one of dynamic website page which is build with angular framework.
Can anybody guide me for same or correct in above code where i am missing right things.
You're getting the content of the page before the dynamic code has run, you need to wait for the load to be completed.
The block behind the page.open would need to wait for the page to complete, if there is an element you know is being fetched from the back end you can lie in wait for that element (see the waitfor example in phantomjs doc).
I'm a Java developer, but have played around with javascript very little. I am looking to develop a small Node JS app to parse a dynamic web page... so we need some way to wait until the page is fully loaded. I managed to get a node js project running with a hello world app.
I then updated the project to support PhantomJS via the PhantomJS Node bridge (https://github.com/amir20/phantomjs-node). I was able to successfully run one of their (PhantomJS Node bridge) samples in my node project (see below). While this will successfully write the contents of the web page to a file, the content is not complete, as it does not contain the dynamic data (retrieved via javascript/AJAX).
Can someone tell me a code modification to the below that will allow it to wait until the page is fully loaded prior to writing the file?
** Edit - Just saw where another user has basically the exact same issue, but is unanswered: Dynamic scraping using nodejs and phantomjs
Node js version 6.20, phantom js version 2.1.1, phantom js node (bridge) version 2.1.2
var phantom = require('phantom');
var sitepage = null;
var phInstance = null;
phantom.create()
.then(instance => {
phInstance = instance;
return instance.createPage();
})
.then(page => {
sitepage = page;
return page.open('http://www.somesite.com');
})
.then(status => {
console.log(status);
return sitepage.property('content');
})
.then(content => {
var fs = require('fs');
fs.writeFile("output.html", content, function(err) {
if(err) {
return console.log(err);
}
});
sitepage.close();
phInstance.exit();
})
.catch(error => {
console.log(error);
phInstance.exit();
});