We Keep Coding

Data not returning from async function with database connection

The goal is to call a function from my main script that connects to a database, reads a document from it, stores pieces of that document in a new object, and returns that object to my main script. The problem is I cannot get it all to work together. If I try one thing, I get the results but my program locks up. If I try something else I get undefined results.
Long story short, how do I open a database and retrieve something from it to another script.
The program is a quiz site and I want to return the quiz name and the questions.
const myDb = require('./app.js');
var myData = myDb.fun((myData) => {
console.log(myData.quizName);
});
Here is the script that tries to open the database and find the data
const { MongoClient } = require("mongodb");
const {mongoClient} = require("mongodb");
const uri = connection uri goes here but my name is hard coded into it at the moment so I removed for privacy
const client = new MongoClient(uri);
const fun = async (cback) => {
try {
await client.connect();
const database = client.db('Quiz-Capstone');
const quizzes = database.collection('Quiz');
const query = {quizName: "CIS01"};
const options = {
sort: {},
projection: {}
};
const quiz = await quizzes.findOne(query, options);
var quizObject = {
quizName: quiz.quizName,
quizQuestions: quiz.quizQuestions
}
//console.log(testOb);
} finally {
await client.close();
cback(quizObject);
}
}
fun().catch(console.dir);
module.exports = {
fun: fun
}
UPDATE: Still stuck. I have read several different threads here about asynchronous calls and callbacks but I cannot get my function located in one file to return a value to the caller located in another file.

Connecting to a running process in Winappdriver using Javascript

I am fairly new to JS/Winappdriver.
The application I am trying to test is a windows based "Click Once" application from .Net, so I have to go to a website from IE and click "Install". This will open the application.
Once the application is running, I have no way to connect the application to perform my UI interactions while using JavaScript.
Using C#, I was looping through the processes looking for a process name, get the window handle, convert it to hex, add that as a capability and create the driver - it worked. Sample code below,
public Setup_TearDown()
{
string TopLevelWindowHandleHex = null;
IntPtr TopLevelWindowHandle = new IntPtr();
foreach (Process clsProcess in Process.GetProcesses())
{
if (clsProcess.ProcessName.StartsWith($"SomeName-{exec_pob}-{exec_env}"))
{
TopLevelWindowHandle = clsProcess.Handle;
TopLevelWindowHandleHex = clsProcess.MainWindowHandle.ToString("x");
}
}
var appOptions = new AppiumOptions();
appOptions.AddAdditionalCapability("appTopLevelWindow", TopLevelWindowHandleHex);
appOptions.AddAdditionalCapability("ms:experimental-webdriver", true);
appOptions.AddAdditionalCapability("ms:waitForAppLaunch", "25");
AppDriver = new WindowsDriver<WindowsElement>(new Uri(WinAppDriverUrl), appOptions);
AppDriver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(60);
}
How do I do this in Javascript ? I can't seem to find any code examples.
Based on an example from this repo, I tried the following in JS to find the process to latch on to but without luck.
import {By2} from "selenium-appium";
// this.appWindow = this.driver.element(By2.nativeAccessibilityId('xxx'));
// this.appWindow = this.driver.element(By2.nativeXpath("//Window[starts-with(#Name,\"xxxx\")]"));
// this.appWindow = this.driver.elementByName('WindowsForms10.Window.8.app.0.13965fa_r11_ad1');
// thisappWindow = this.driver.elementByName('xxxxxxx');
async connectAppDriver(){
await this.waitForAppWindow();
var appWindow = await this.appWindow.getAttribute("NativeWindowHandle");
let hex = (Number(ewarpWindow)).toString(16);
var currentAppCapabilities =
{
"appTopLevelWindow": hex,
"platformName": "Windows",
"deviceName": "WindowsPC",
"newCommandTimeout": "120000"
}
let driverBuilder = new DriverBuilder();
await driverBuilder.stopDriver();
this.driver = await driverBuilder.createDriver(currentEwarpCapabilities);
return this.driver;
}
I keep getting this error in Winappdriver
{"status":13,"value":{"error":"unknown error","message":"An unknown error occurred in the remote end while processing the command."}}
I've also opened this ticket here.
It seems like such an easy thing to do, but I couldn't figure this one out.
Any of nodes packages I could use to get the top level window handle easily?
I am open to suggestions on how to tackle this issue while using JavaScript for Winappdriver.

Hope this helps some one out there,
Got around this by creating an exe using C# that generated hex of the app to connect based on the process name, it looks like something like this.
public string GetTopLevelWindowHandleHex()
{
string TopLevelWindowHandleHex = null;
IntPtr TopLevelWindowHandle = new IntPtr();
foreach (Process clsProcess in Process.GetProcesses())
{
if (clsProcess.ProcessName.StartsWith(_processName))
{
TopLevelWindowHandle = clsProcess.Handle;
TopLevelWindowHandleHex = clsProcess.MainWindowHandle.ToString("x");
}
}
if (!String.IsNullOrEmpty(TopLevelWindowHandleHex))
return TopLevelWindowHandleHex;
else
throw new Exception($"Process: {_processName} cannot be found");
}
Called it from JS to get the hex of the top level window handle, like this,
async getHex () {
var pathToExe =await path.join(process.cwd(), "features\\support\\ProcessUtility\\GetWindowHandleHexByProcessName.exe");
var pathToDir =await path.join(process.cwd(), "features\\support\\ProcessUtility");
const result = await execFileSync(pathToExe, [this.processName]
, {cwd: pathToDir, encoding: 'utf-8'}
, async function (err, data) {
console.log("Error: "+ err);
console.log("Data(hex): "+ data);
return JSON.stringify(data.toString());
});
return result.toString().trim();
}
Used the hex to connect to the app like this,
async connectAppDriver(hex) {
console.log(`Hex received to connect to app using hex: ${hex}`);
const currentAppCapabilities=
{
"browserName": '',
"appTopLevelWindow": hex.trim(),
"platformName": "Windows",
"deviceName": "WindowsPC",
"newCommandTimeout": "120000"
};
const appDriver = await new Builder()
.usingServer("http://localhost:4723/wd/hub")
.withCapabilities(currentAppCapabilities)
.build();
await driver.startWithWebDriver(appDriver);
return driver;
}

Solution:
In WebDriverJS (used by selenium / appium), use getDomAttribute instead of getAttribute. Took several hours to find :(
element.getAttribute("NativeWindowHandle")
POST: /session/270698D2-D93B-4E05-9FC5-3E5FBDA60ECA/execute/sync
Command not implemented: POST: /session/270698D2-D93B-4E05-9FC5-3E5FBDA60ECA/execute/sync
HTTP/1.1 501 Not Implemented
let topLevelWindowHandle = await element.getDomAttribute('NativeWindowHandle')
topLevelWindowHandle = parseInt(topLevelWindowHandle).toString(16)
GET /session/DE4C46E1-CC84-4F5D-88D2-35F56317E34D/element/42.3476754/attribute/NativeWindowHandle HTTP/1.1
HTTP/1.1 200 OK
{"sessionId":"DE4C46E1-CC84-4F5D-88D2-35F56317E34D","status":0,"value":"3476754"}
and topLevelWindowHandle have hex value :)

Is there an example site that uses navigator.serial?

I noticed that Chrome Canary has an implementation of a web serial api at navigator.serial, and I'm interested in looking at it. The previous API for serial ports chrome.serial implements listener callbacks, while this new API seems to deal in streams.
I've looked at the example at https://wicg.github.io/serial/#usage-example, but it seems pretty bare bones.
<html>
<script>
var port;
var buffy = new ArrayBuffer(1);
var writer;
buffy[0]=10;
const test = async function () {
const requestOptions = {
// Filter on devices with the Arduino USB vendor ID.
//filters: [{ vendorId: 0x2341 }],
};
// Request an Arduino from the user.
port = await navigator.serial.requestPort(requestOptions);
// Open and begin reading.
await port.open({ baudrate: 115200 });
//const reader = port.in.getReader();
const reader = port.readable.getReader();
writer = port.writable.getWriter();
//const writer = port.writable.getWriter();
//writer.write(buffy);
while (true) {
const {done, data} = await reader.read();
if (done) break;
console.log(data);
}
} // end of function
</script>
<button onclick="test()">Click It</button>
</html>
I'd like to find a working example, and eventually find a way to migrate an app from chrome.serial to navigator.serial

Hey battling this as well. To enable this 'experimental api', open Canary, and punch this into the url: chrome://flags/#enable-experimental-web-platform-features
Enable that feature. Now you can use it.

Extract public posts from Facebook page without API/APP key/token/secret

Just to clarify in advance, I don't have a Facebook account and I have no intent to create one. Also, what I'm trying to achieve is perfectly legal in my country and the USA.
Instead of using the Facebook API to get the latest timeline posts of a Facebook page, I want to send a get request directly to the page URL (e.g. this page) and extract the posts from the HTML source code.
(I'd like to get the text and the creation time of the post.)
When I run this in the web console:
document.getElementsByClassName('userContent')
I get a list of elements containing the text of the latest posts.
But I'd like to extract that information from a nodejs script. I could probably do it quite easily using a headless browser like puppeteer or the like, but that would create a ton of unnecessary overhead. I'd really like to a simple approach like downloading the HTML code, passing it to cheerio and use cheeriio's jQuery-like API to extract the posts.
Here is my attempt of trying exactly that:
// npm i request cheerio request-promise-native
const rp = require('request-promise-native'); // requires installation of `request`
const cheerio = require('cheerio');
rp.get('https://www.facebook.com/pg/officialstackoverflow/posts/').then( postsHtml => {
const $ = cheerio.load(postsHtml);
const timeLinePostEls = $('.userContent');
console.log(timeLinePostEls.html()); // should NOT be null
const newestPostEl = timeLinePostEls.get(0);
console.log(newestPostEl.html()); // should NOT be null
const newestPostText = newestPostEl.text();
console.log(newestPostText);
//const newestPostTime = newestPostEl.parent(??).child('.livetimestamp').title;
//console.log(newestPostTime);
}).catch(console.error);
unfortunately $('.userContent') does not work. However, I was able to verify that the data I'm looking for is embedded somewhere in that HTML code.
But I couldn't really come up with a with a good regex approach or the like to extract that data.
Depending on the post content the number of HTML tags within the post varies heavily.
Here is a simple example of a post containing one link:
<div class="_5pbx userContent _3576" data-ft="{"tn":"K"}"><p>We're proud to be named one of Built In NYC's Best Places to Work in 2019, ranking in the top 10 for Best Midsize Places to Work and top 3 (!) for Best Perks and Benefits. See what it took to make the list and check out our profile to see some of our job openings. http://*******/2H3Kbr2</p></div>
Formatted in a more readable form it looks somewhat like this:
<div class="_5pbx userContent _3576" data-ft="{"tn":"K"}">
<p>
We're proud to be named one of Built In NYC's Best Places to Work in
2019, ranking in the top 10 for Best Midsize Places to Work and top 3 (!) for
Best Perks and Benefits. See what it took to make the list and check out our
profile to see some of our job openings.
SHORT_LINK.....
</p>
</div>
This regex seems to work okay, but I don't think it is very reliable:
/<div class="[^"]+ userContent [^"]+" data-ft="[^"]+">(.+?)<\/div>/g
If for example the post contained another div-element then it wouldn't work properly. In addition to that I have no way of knowing the time/date the post was created using this approach?
Any ideas how I could relatively reliably extract the most recent 2-3 posts including the creation date/time?

Okay, I finally figured it out. I hope this will be useful to others. This function will extract the 20 latest posts, including the creation time:
// npm i request cheerio request-promise-native
const rp = require('request-promise-native'); // requires installation of `request`
const cheerio = require('cheerio');
function GetFbPosts(pageUrl) {
const requestOptions = {
url: pageUrl,
headers: {
'User-Agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0'
}
};
return rp.get(requestOptions).then( postsHtml => {
const $ = cheerio.load(postsHtml);
const timeLinePostEls = $('.userContent').map((i,el)=>$(el)).get();
const posts = timeLinePostEls.map(post=>{
return {
message: post.html(),
created_at: post.parents('.userContentWrapper').find('.timestampContent').html()
}
});
return posts;
});
}
GetFbPosts('https://www.facebook.com/pg/officialstackoverflow/posts/').then(posts=>{
// Log all posts
for (const post of posts) {
console.log(post.created_at, post.message);
}
});
Since Facebook messages can have complicated formatting the message is not plain text, but HTML. But you could remove the formatting and just get the text by replacing message: post.html() with message: post.text().
Edit:
If you want to get more than the latest 20 posts, it is more complicated. The first 20 posts are served statically on the initial html page. All following posts are retrieved via ajax in chunks of 8 posts.
It can be achieved like that:
// make sure your node.js version supports async/await (v10 and above should be fine)
// npm i request cheerio request-promise-native
const rp = require('request-promise-native'); // requires installation of `request`
const cheerio = require('cheerio');
class FbScrape {
constructor(options={}) {
this.headers = options.headers || {
'User-Agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0' // you may have to update this at some point
};
}
async getPosts(pageUrl, limit=20) {
const staticPostsHtml = await rp.get({ url: pageUrl, headers: this.headers });
if (limit <= 20) {
return this._parsePostsHtml(staticPostsHtml);
} else {
let staticPosts = this._parsePostsHtml(staticPostsHtml);
const nextResultsUrl = this._getNextPageAjaxUrl(staticPostsHtml);
const ajaxPosts = await this._getAjaxPosts(nextResultsUrl, limit-20);
return staticPosts.concat(ajaxPosts);
}
}
_parsePostsHtml(postsHtml) {
const $ = cheerio.load(postsHtml);
const timeLinePostEls = $('.userContent').map((i,el)=>$(el)).get();
const posts = timeLinePostEls.map(post => {
return {
message: post.html(),
created_at: post.parents('.userContentWrapper').find('.timestampContent').html()
}
});
return posts;
}
async _getAjaxPosts(resultsUrl, limit=8, posts=[]) {
const responseBody = await rp.get({ url: resultsUrl, headers: this.headers });
const extractedJson = JSON.parse(responseBody.substr(9));
const postsHtml = extractedJson.domops[0][3].__html;
const newPosts = this._parsePostsHtml(postsHtml);
const allPosts = posts.concat(newPosts);
const nextResultsUrl = this._getNextPageAjaxUrl(postsHtml);
if (allPosts.length+1 >= limit)
return allPosts;
else
return await this._getAjaxPosts(nextResultsUrl, limit, allPosts);
}
_getNextPageAjaxUrl(html) {
return 'https://www.facebook.com' + /"(\/pages_reaction_units\/more[^"]+)"/g.exec(html)[1].replace(/&/g, '&') + '&__a=1';
}
}
const fbScrape = new FbScrape();
const minimum = 28; // minimum number of posts to request (gets rounded up to 20, 28, 36, 44, 52, 60, 68 etc... because of page sizes (page1=20; all_following_pages=8)
fbScrape.getPosts('https://www.facebook.com/pg/officialstackoverflow/posts/', minimum).then(posts => { // get at least the 28 latest posts
// Log all posts
for (const post of posts) {
console.log(post.created_at, post.message);
}
});

Mobile Safari 10 IndexedDB Blobs

IndexedDB in Safari 10 supports blobs now. This works fine on desktop, however mobile Safari on iOS 10 throws an error:
UnknownError
and sometimes in combination:
TransactionInactiveError (DOM IDBDatabase Exception): Failed to store record in an IDBObjectStore:
The transaction is inactive or finished.
The code (shortened):
var indexedDB = window.indexedDB || window.webkitIndexedDB || window.mozIndexedDB || window.msIndexedDB,
READ_WRITE = IDBTransaction && IDBTransaction.READ_WRITE ? IDBTransaction.READ_WRITE : 'readwrite',
storeName = 'files',
db;
init: function() {
var request = indexedDB.open('mydb');
request.onerror = ...;
request.onupgradeneeded = function() {
db = request.result;
db.createObjectStore(storeName);
};
request.onsuccess = function() {
db = request.result;
};
},
save: function(id, data) {
var put = function(data) {
var objectStore = db.transaction([storeName], READ_WRITE).objectStore(storeName),
request = objectStore.put(data, id);
request.onerror = ...;
request.onsuccess = ...;
};
// not all IndexedDB implementations support storing blobs, only detection is try-catch
try {
put(data);
} catch(err) {
if (data instanceof Blob) {
Helpers.blobToDataURL(data, put);
}
}
}
On Mobile Safari 10 .put() doesn't throw like before, only later in the async error-callback.
Base64 strings work fine.
Bug in Mobile Safari or do I have to change code?
Test Case: http://fiddle.jshell.net/j7wh60vo/7/

Ran across the same problem. Chrome 54 and Safari 10 work fine on desktop, but on Mobile Safari I kept getting the Unknown error when trying to store a Blob into IndexedDB. I can confirm that this really is just an issue with Blobs on Mobile Safari, and not some misuse of the API.
Fortunately, ArrayBuffers work fine. So I instead downloaded the images like:
xhr.open('GET', url, true);
xhr.responseType = 'arraybuffer';
Then saved them into IndexedDB as ArrayBuffers, and converted them to Blobs after pulling them out to get a url:
putRequest = objectStore.put(arrayBuffer, id);
putRequest.onsuccess = function(event) {
objectStore.get(id).onsuccess = function(event) {
var blob = new Blob([event.target.result], { type: 'image/jpeg'});
var URL = window.URL || window.webkitURL;
blobUrl = URL.createObjectURL(blob);
};
};
I'd rather not have to convert ArrayBuffers to Blobs like this as I assume there is a performance penalty. But it works.

That error looks to me like you have to change the code. That error does not indicate an issue with blobs. That error indicates you have a problem somewhere in how you call functions. To better answer your question, you need to post more of the surrounding code. Specifically, display the parts of the code where you create the transaction and where you create requests on the transaction.
Edit: first, remove the window.indexedDB stuff. Second, do not use 'db' in the way you are using it, because that will not work, the db may be closed by the time save is called.
function save(id, data) {
var openRequest = indexedDB.open(...);
openRequest.onerror = console;
openRequest.onsuccess = function(event) {
var db = openRequest.result;
// Open the transaction
var tx = db.transaction(storeName, 'readwrite');
var store = tx.objectStore(storeName);
// Immediately use the transaction
try {
var putRequest = tx.put(data, id);
putRequest.onerror = console;
} catch(error) {
console.log(error);
}
};
}
Edit2: Additional notes:
Prefixes have been removed, just use indexedDB, not mozIndexedDB or webkitIndexedDB etc
Transaction mode constants have been removed, use either 'readonly' or 'readwrite', or nothing (defaults to readonly)
I am somewhat confused how you are calling request = transaction.put. As far as I am aware, there is no method IDBTransaction.prototype.put as shown in the spec https://w3c.github.io/IndexedDB/#idbtransaction. I am confused as to why the Mozilla docs show an example with transaction.put. Inspecting the prototype of IDBTransaction in Chrome 55 does not show a put method.
There is IDBObjectStore.prototype.put. Your code should not be working at all, on any platform, as it is currently written. So if it did ever work, I am surprised. You should only be using something like var store = transaction.objectStore('store'); store.put(obj); where you call put on the object store.