I am trying to make a crawler, and as the data is not showing in the page source, I can only execute the javascript with the web driver and get the response, and then do the data analysis.
The script is simplified, like this, use Promise.
var res = ""
function f1() {
p = window.Promise
a = p.resolve(5).then(function(value) {
console.log(value)
res = value
return res
})
return a
}
console.log(f1()) // Promise object
console.log("result = ", res) // res is empty
My program is like this, writing with c#:
public void GetParameters(string url)
{
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl(url);
IJavaScriptExecutor js = driver as IJavaScriptExecutor;
string script = readFile(#"C:\myscript.js", Encoding.UTF8).TrimEnd(); // The script is a bit long, so I save it to a local file.
var json = js.ExecuteScript(script);
Console.WriteLine("Get the return value");
Console.WriteLine(json);
driver.Close();
}
I want to get the value, and I know then always return Promise object, so I define a variable, and want to store the value to it. But seems that, the Promise is executed asynchronous, so the res always be empty.
OK, I'm writing a crawler, and use Selenium to get the response from the server (Selenium.webdriver can open a web browser, and then execute script), I need get the result, as it will use by another program. So, I can not just add another .then, just output the value. Maybe I can save to a local file, and then read it, but I think it is inefficient.
Anyone can help?
When you try to log the global res, the result hasn't been computed yet. When using Promises, you have to get the result asynchronously, using .then, like this:
f1().then(res => console.log("result =", res));
Try this
You're using promises incorrectly.
f1().then(res => console.log('result=', res)).
Related
I have to compare few rows derived from potgres with data derived from the dynamo DB. problem is in both cases i have to iterate over data & unless i can assign the value to a variable i will have to query the same data again from either one of two databases.
function postgressdata(sql){
return new Promise(function(resolve, reject) {
client_postgres.query(sql, (err, res) => {
resolve(res);
});
});
}
let get_creatives = postgressdata(sql);
get_creatives.then(function(result) {
console.log(result); // prints the result
let new_result = result;
}
console.log(new_result) // does not print result
How Can I assign the data to a variable here & use it anywhere?
I am a newbie to NODEJS so pardon me if I am asking silly question.
Nodejs is non-blocking.
so when this block of code runs
let get_creatives = postgressdata(sql);
get_creatives.then(function(result) {
console.log(result); // prints the result
let new_result = result;
}
console.log(new_result) // does not print result
what will happen is you will get nothing printed first because .then() will be called when the promise is resolved so this line
console.log(new_result) // does not print result
will be executed first which doesn't have the value yet obviously.
after that when the promise is resolved .then will be called and your value will be set.
now how to solve this?
I am assuming you're sending the value back to the callee so you can either use the async/await structure or you can use a callback, depending on how you need the value.
You can just keep a reference to the promise as your cache. A promise in JavaScript represents the result of an async operation, not the operation itself.
If you have something like:
async function runQuery() {
const result = await db.query(sql`SELECT * FROM creatives;`);
console.log('ran query');
return result;
}
const creativesPromise = runQuery();
async function useResult() {
console.log(await creativesPromise);
}
useResult();
useResult();
useResult();
even though we use the resulting list of creatives three times, "ran query" will only get logged once. Every use of the query results will need to be async, because you never know whether the query has finished running when you request the data, but the query will only be run once.
I am building an application customiser in SPFX and I am using pnp/sp to get data from a Sharepoint list - all easy so far. I have figured out the code like this, but it is just returning [object promise] here is my code , any help would be brilliant.
I am calling the function like this :
public emailAddressGetter = this.GetSharePointData();
I am trying to show the output like this :
${escape(this.emailAddressGetter.toString())}
and this is the promise I am executing :
private async GetSharePointData(): Promise<any>
{
let myVar : string;
var resultData: any = await sp.web.lists
.getByTitle('Emails')
.items
.select('EmailAddress')
.getById(99)
.get().then((r => {
myVar = r.EmailAddress;
}));
console.log(myVar);
return myVar;
}
any help would be appreciated, I know I am almost there :) thanks guys
I think your GetSharePointData returns a Promise, because it has async declaration, so you need to execute code asynchronously and wait for the result.
Instead of:
public emailAddressGetter = this.GetSharePointData();
${escape(this.emailAddressGetter.toString())}
Try:
this.GetSharePointData()
.then(res => {
// res here is myVar
${escape(res.toString())};
});
Firstly fix your code's type annotations. You are completely defeating the point of TypeScript by suppressing the errors the language exists to catch by specifying vacuous types instead of leveraging inference. This isn't Java.
async GetSharePointData() { // return type is inferred as `Promise<string>`
const result = await sp.web.lists // the `any` you had here was worse than useless.
.getByTitle('Emails')
.items
.select('EmailAddress')
.getById(99)
.get();
const emailAddress= result.emailAddress;
console.log(emailAddress);
return emailAddress;
}
Now onto async functions and promises. An async function or method always returns a promise. Assigning the result of calling such a function directly to a property or variable will always result in the behavior you described
GetSharePointData().toString() === "[object Promise]"
The correct approach to setting the property emailAddressGetter (BTW that's a terrible name for that property either way) to the email address that the promise eventually resolves with depends on the context, but here is something you might do.
constructor() {
this.emailAddressPromise = this.GetSharePointData();
this.emailAddressPromise.then(emailAddress => this.emailAddress = emailAddress);
}
But that could be awful and unnecessary unpredictable depending on what you are trying to do.
I'm new to using csv-parse and this example from the project's github does what I need with one exception. Instead of outputting via console.log I want to store data in a variable. I've tried assigning the fs line to a variable and then returning data rather than logging it but that just returned a whole bunch of stuff I didn't understand. The end goal is to import a CSV file into SQLite.
var fs = require('fs');
var parse = require('..');
var parser = parse({delimiter: ';'}, function(err, data){
console.log(data);
});
fs.createReadStream(__dirname+'/fs_read.csv').pipe(parser);
Here is what I have tried:
const fs = require("fs");
const parse = require("./node_modules/csv-parse");
const sqlite3 = require("sqlite3");
// const db = new sqlite3.Database("testing.sqlite");
let parser = parse({delimiter: ","}, (err, data) => {
// console.log(data);
return data;
});
const output = fs.createReadStream(__dirname + "/users.csv").pipe(parser);
console.log(output);
I was also struggling to figure out how to get the data from csv-parse back to the top-level that invokes parsing. Specifically I was trying to get parser.info data at the end of processing to see if it was successful, but the solution for that can work to get the row data as well, if you need.
The key was to wrap all the stream event listeners into a Promise, and within the parser's callback resolve the Promise.
function startFileImport(myFile) {
// THIS IS THE WRAPPER YOU NEED
return new Promise((resolve, reject) => {
let readStream = fs.createReadStream(myFile);
let fileRows = [];
const parser = parse({
delimiter: ','
});
// Use the readable stream api
parser.on('readable', function () {
let record
while (record = parser.read()) {
if (record) { fileRows.push(record); }
}
});
// Catch any error
parser.on('error', function (err) {
console.error(err.message)
});
parser.on('end', function () {
const { lines } = parser.info;
// RESOLVE OUTPUT THAT YOU WANT AT PARENT-LEVEL
resolve({ status: 'Successfully processed lines: ', lines });
});
// This will wait until we know the readable stream is actually valid before piping
readStream.on('open', function () {
// This just pipes the read stream to the response object (which goes to the client)
readStream.pipe(parser);
});
// This catches any errors that happen while creating the readable stream (usually invalid names)
readStream.on('error', function (err) {
resolve({ status: null, error: 'readStream error' + err });
});
});
}
This is a question that suggests confusion about an asynchronous streaming API and seems to ask at least three things.
How do I get output to contain an array-of-arrays representing the parsed CSV data?
That output will never exist at the top-level, like you (and many other programmers) hope it would, because of how asynchronous APIs operate. All the data assembled neatly in one place can only exist in a callback function. The next best thing syntactically is const output = await somePromiseOfOutput() but that can only occur in an async function and only if we switch from streams to promises. That's all possible, and I mention it so you can check it out later on your own. I'll assume you want to stick with streams.
An array consisting of all the rows can only exist after reading the entire stream. That's why all the rows are only available in the author's "Stream API" example only in the .on('end', ...) callback. If you want to do anything with all the rows present at the same time, you'll need to do it in the end callback.
From https://csv.js.org/parse/api/ note that the author:
uses the on readable callback to push single records into a previously empty array defined externally named output.
uses the on error callback to report errors
uses the on end callback to compare all the accumulated records in output to the expected result
...
const output = []
...
parser.on('readable', function(){
let record
while (record = parser.read()) {
output.push(record)
}
})
// Catch any error
parser.on('error', function(err){
console.error(err.message)
})
// When we are done, test that the parsed output matched what expected
parser.on('end', function(){
assert.deepEqual(
output,
[
[ 'root','x','0','0','root','/root','/bin/bash' ],
[ 'someone','x','1022','1022','','/home/someone','/bin/bash' ]
]
)
})
As to the goal on interfacing with sqlite, this is essentially building a customized streaming endpoint.
In this use case, implement a customized writable stream that accepts the output of parser and sends rows to the database.
Then you simply chain pipe calls as
fs.createReadStream(__dirname+'/fs_read.csv')
.pipe(parser)
.pipe(your_writable_stream)
Beware: This code returns immediately. It does not wait for the operations to finish. It interacts with a hidden event loop internal to node.js. The event loop often confuses new developers who are arriving from another language, used to a more imperative style, and skipped this part of their node.js training.
Implementing such a customized writable stream can get complicated and is left as an exercise for the reader. It will be easiest if the parser emits a row, and then the writer can be written to handle single rows. Make sure you are able to notice errors somehow and throw appropriate exceptions, or you'll be cursed with incomplete results and no warning or reason why.
A hackish way to do it would have been to replace console.log(data) in let parser = ... with a customized function writeRowToSqlite(data) that you'll have to write anyway to implement a custom stream. Because of asynchronous API issues, using return data there does not do anything useful. It certainly, as you saw, fails to put the data into the output variable.
As to why output in your modified posting does not contain the data...
Unfortunately, as you discovered, this is usually wrong-headed:
const output = fs.createReadStream(__dirname + "/users.csv").pipe(parser);
console.log(output);
Here, the variable output will be a ReadableStream, which is not the same as the data contained in the readable stream. Put simply, it's like when you have a file in your filesystem, and you can obtain all kinds of system information about the file, but the content contained in the file is accessed through a different call.
I have this code jQuery code fragment:
$.get('/api/' + currentPage).done(function(data) { ... })
.fail(...)
I want to replace $.get('/api/'+currentPage) with a promise that always succeeds and returns a specific value for data. Something like:
let myData = { ... } // value of data I want to pass to the done function
(new AlwaysSucceeds(myData)).done(function(data) { ... })
.fail(...)
I could cobble up a dummy object, or I could extract out the done function but I want to keep changes to the code to a minimum.
Is there a way to do this?
UPDATE: To help clarify what's going, the code I am working with is (here). Normally this app is served from a nodejs server which implements the /api/... call, but I am converting it to be served
from a static page server. I know what is going to be returned from
the $.get call. To keep changes to the code clean I simply want to
change that line to:
let myData = {...}
// $.get('/api/' + currentPage) -- comment out the $.get call
(SOMETHINGHERE(myData)).done(function(data) {
The SOMETHINGHERE expression needs to implement .done(f)
which will call the function f with myData and then return
some object which implements .fail(...) which does nothing.
You can just replace $.get(...) with a function that returns a promise that is already resolved with the data you already have. And, the shortest way to get an already resolved jQuery promise, resolved with a particular value, is this:
$.when(myData).done(...)
The more text book way to do it in jQuery is:
$.Deferred().resolve(myData).done(...)
And, if you care to switch your logic to the the ES6 standard (instead of the non-standard jQuery promise behaviors), then you could use this:
Promise.resolve(myData).then(...).catch(...)
You can achieve this by implementing AlwaysSuceeds constructor function. Please see below example.
function AlwaysSucceeds(data) {
this.data = data;
}
AlwaysSucceeds.prototype.done = function(fn) {
fn(this.data);
return this;
}
AlwaysSucceeds.prototype.fail = function(fn) {
return this;
}
var myData = {
a: 1
};
(new AlwaysSucceeds(myData)).done(function(data) {
console.log(data)
}).fail(function(data){
})
Since jQuery Ajax functions just return $.Deferred objects, you can just substitute an immediately-resolved Deferred:
$.Deferred().resolve(myData).then(...)
In this particular case, if you want to make it easy to switch between synchronous and asynchronous code, and you have access to async/await, you can just use those directly:
try {
const data = await Promise.resolve($.get('/api/' + currentPage));
// code in done
} catch (err) {
// code in fail
}
would become
try {
const data = myData;
// code in done
} catch (err) {
// code in fail (never runs unless other code throws exceptions)
}
It's not clear what you actually want but be carufull using jQuery Deferred with native promises, the deferred has some non standard methods that native promises don't have.
So to be save I always assume there is a thenable, something that has a then with that you can pretty much do whatever you want.
jQuery Deferred do not behave like native promises either (depending on version):
$.Deferred().reject("hello world")
.then(
undefined
,x=>x
)
.then(
x=>console.log("Never happens",x)
)
Promise.reject("hello world")
.then(
undefined
,x=>x
);
.then(
x=>console.log("Well behaved",x)
);
Promise.resolve().then(x=>{throw "nope"})
.then(undefined,err=>console.warn(err));
$.Deferred().resolve().then(x=>{throw "nope"})//crashes
.then(undefined,err=>err);
So it will be saver to use native promises and polyfill with something that behaves like native.
To answer the question about non failing promise, if you want to make a request but return a default when it rejects and keep returning the same once resolves or rejects you can do:
const get = (p=>{
(url) => {
p = p ||
//return native promise or properly polyfilled one
Promise.resolve($.get(url))
.then(
undefined,
_=> {defaultobject:true}
);
return p;
}
})();
Your get function will return a native promise so no fail, done and other things that are non standard. Combining "promises" from different libraries and native promises it would be best to only use then
I'm struggling to get the actual text of the text box as I need it as a text to store it in a variable rather than comparing it against a value, because I need to add it to the end of the url to call another page.
I tried using the code suggested by ebeal but it didn't do what I want:
var access_token = driver.findElement(webdriver.By.name("AccToken"))
.getAttribute("value")
.then(console.log);
// This outputs the right result but only to the console as I can't save it to a variable
var access_token = driver.findElement(webdriver.By.name("AccToken"))
.getText();
access_token = access_token.then(function(value){
console.log(value);
});
console.log("the new one : " + access_token);
// this one outputs : the new one: Promise::304 {[[PromiseStatus]]: "pending"}
Any idea?
WebdriverJS is purely asynchronous. Meaning, you need to provide a callback and instantiate your variable inside the callback rather than simply assigning the call the results of the function to your variable.
That's why you will always get a promise everytime you console.log your access_token variable. The webdriverjs docs explain a little about how promises work in selenium-webdriver https://code.google.com/p/selenium/wiki/WebDriverJs#Understanding_the_API
You can do the following to assign the text to a variable:
var access_token;
var promise = driver.findElement(webdriver.By.name("AccToken")).getText();
promise.then(function(text) {
access_token = text;
});
I highly recommend WebdriverIO as it takes away from the pain of having to write your own promises. http://webdriver.io/
So this is one thing that I have had to learn the hard way, so I hope this helps:
var access_token = await driver.findElement(webdriver.By.name("AccToken"))
.getAttribute("value")
.then((value) => { return value; });
I'm not sure which version of Webdriver you are using, but you may have some luck using WebdriverIO. Specifically its getText() function which will return a callback with the text so you can use it elsewhere.
http://webdriver.io/api/property/getText.html
client.getText('#elem').then(function(text) {
console.log(text);
});
This should work just fine if you are looking to just get the value. If you are using the new await ES6 syntax no need to "then" the promise.
const { Builder, By, Key, until } = require('selenium-webdriver');
const assert = require('assert');
let access_token = await driver.findElement(By.name("AccToken")).getAttribute("value");
Then you can even just assert:
assert.equal(access_token, "your token value here");
More information can be found at the documentation for selenium-webdriver. Take a look at the Webdriver instance methods for a closer look. Good luck!