We Keep Coding

Apify - How to Include Failed Results in Dataset

We are using the Apify Web Scraper actor to create a URL validation task that returns the input URL, the page's title, and the HTTP response status code. We have a set of 5 test URLs we are using: 4 valid, and 1 non-existent. The successful results are always included in the dataset, but never the failed URL.
Logging indicates that the pageFunction is not even reached for the failed URL:
2021-05-05T14:50:08.489Z ERROR PuppeteerCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"http://www.invalidurl.com","retryCount":1,"id":"XS9JTk8dYRM8bpM"}
2021-05-05T14:50:08.490Z Error: gotoFunction timed out after 30 seconds.
2021-05-05T14:50:08.490Z at PuppeteerCrawler._handleRequestTimeout (/home/myuser/node_modules/apify/build/crawlers/puppeteer_crawler.js:387:15)
2021-05-05T14:50:08.496Z at PuppeteerCrawler._handleRequestFunction (/home/myuser/node_modules/apify/build/crawlers/puppeteer_crawler.js:329:26)
Eventually it times out, based on our settings:
2021-05-05T14:50:42.052Z ERROR Request http://www.invalidurl.com failed and will not be retried anymore. Marking as failed.
2021-05-05T14:50:42.052Z Last Error Message: Error: gotoFunction timed out after 30 seconds.
I tried wrapping the code in the pageFunction in a try/catch block, but again, because the pageFunction is not getting reached for the invalid URL, that does not resolve the issue. Is there a way to still include the failed result in the dataset with a hard-coded response status code of '000'? (See pageFunction code below.) Please let me know if I can provide any additional information, and thanks in advance!
async function pageFunction(context) {
context.log.info("Starting pageFunction");
// use jQuery as $
const { request, jQuery: $ } = context;
const { url } = request;
context.log.info("Trying " + url);
let title = null;
let responseCode = null;
try {
context.log.info("In try block for " + url);
title = $('title').first().text().trim();
responseCode = context.response.status;
} catch (error) {
context.log.info("EXCEPTION for " + url);
title = "";
responseCode = "000";
}
return {
url,
title,
responseCode
};
}

you can use https://sdk.apify.com/docs/typedefs/puppeteer-crawler-options#handlefailedrequestfunction:
you can then push it to the when all retries fail:
handleFailedRequestFunction: async ({ request }) => {
// failed all retries
await Apify.pushData({ url: request.url, responseCode: '000' });
}

jQuery.GET intermittent failure in Safari in ipad

I have the following utility function which I use as a standard way of adding headers to an HTTP GET request.
const get1 = function(url, callback) {
console.log("in get1 with url : " + url);
return new Promise(function(resolve, reject) {
firebase.auth().currentUser.getIdToken()
.then(function(idToken) {
return {
'Authorization': 'Bearer ' + idToken,
'ACTING_ON_BEHALF_OF': ns.actingAs
};
})
.then(function(headers) {
resolve($.get({
url: url,
headers: headers,
dataType: "json",
error: function(jqXHR, textStatus, errorThrown) {
alert(textStatus);
alert(errorThrown);
alert(jqXHR.responseText);
}
}));
});
});
};
I call it with structures similar to this
const targetUrl = ns.operation + "/" + ns.stage + "/" + ns.category;
utils.get1("/api/gameconfig/v1/oddoneout/" + targetUrl)
.then(function(data) {
console.log("THEN Block")
// blah blah blah
});
This seems to work reliably in all browsers except Safari on iPad where, periodically, the GET request appears to be executed but the .then block never gets entered.
Debugging the javascript seems to indicate that the HTTP request is still executing, but the server side seems to think that it has successfully completed the request and returned a response.
I'm at a loss as to why I see this behaviour. There doesn't seem any pattern to it. Sometimes I can make 50 calls without a problem, sometimes it fails more or less immediately.
Any help will be gratefully received - even if it's hints about how to debug this.
I'm using jQuery 3.3.1 if that makes any difference.

Redirecting URL With Token Parameter That Contains '+'

I'm having issues creating a redirect to an URL from an API that I'm using within my application. I request a session token which is used in the redirect URL, but I experience issues when the token contains a + in it. My guess is that the browser is picking this up as a space instead of an actual token character which is causing the redirect to fail ever so often.
Here is an example token that can come back from a request to the API: 1w5OX65MRj+3J9R5AXjMWQLAAXIo5TXa
Looking at the network tab I see that it tries to request the redirect with this token instead:
1w5OX65MRj 3J9R5AXjMWQLAAXIo5TXa, which would be why it's causing issues.
I tried replacing the + with %2B but it seems like my application isn't replacing it at all which is a little odd to me.
Here is my code:
let token = "";
$.get('/_token_req', {contentType: "application/x-www-form-urlencoded"}, (response) => {
//console.log(response);
token = response;
token = token.replace(/\+/g, "%2B"); // this doesn't replace the + character for some reason
$.get('/_redirect', {token: response}, (response) => {
//console.log(response);
if(response == "OK"){
window.location.href = "https://someapi/payments/?auth_token=" + token;
}
})
})
I don't know much about URL encoding, but if someone could point me in the right direction, that would be super helpful. Thanks!

You have 2 issues with your code:
One that you need to use encodeURIComponent to encode any symbols in your token so that it can be sent in appropriate manner.
Two, while calling the second request $.get('/_redirect', you didn't use the replaced token but the simple response that you received from earlier request.
Change your code to this to eliminate both errors:
let token = "";
$.get('/_token_req', {contentType: "application/x-www-form-urlencoded"}, (response) => {
token = encodeURIComponent(response);
$.get('/_redirect', {token: token}, (response) => {
if(response == "OK"){
window.location.href = "https://someapi/payments/?auth_token=" + token;
}
})
})

Javascript getJSON to PHP file in IIS using jsonp with variable

I am hosting an HTML5, JavaScript, and CSS3 https application on Windows IIS (Internet Information Services). How the root directory looks is this:
index.html, src/index.js, src/send_payment.php
I am trying to return a simple string at the moment from the php file using getJSON with jsonp (for security). Here is non-working code:
index.js
var json_obj = {
"data_value": 5489798123489725,
"payment_amount": 10.50,
}
var json_str = JSON.stringify(json_obj);
$.getJSON("https://localhost:443/src/send_payment.php?callback=?", "json_str=" + json_str, function (data) {
try {
console.log("SUCCESS >> " + JSON.stringify(data));
}
catch (e) {
console.log("ERROR >> " + e.toString());
}
});
send_payment.php
<?php
try {
// 1. get data
$json_str = $_GET["json_str"];
// 2. parse the json string
$json_obj = json_decode($json_str);
// 3. get the parameters
$data_value = $json_obj->{"data_value"};
$payment_amount = $json_obj->{"payment_amount"};
}
catch (Exception $e) {
trigger_error("ERROR >> exception = " + $e->getMessage(), E_USER_ERROR);
}
return "test successful";
?>
I'm not sure if the code is right or missing anything, but the issue is that I get 404 (Page not found) from getJSON. Is the URL wrong? I am accessing IIS locally, thus the localhost in the URL. I get error 405 (Method not allowed) when using AJAX POST instead with the same URL. Thank you.

I would recommend you to go for $.ajax and POST method like below:
$.ajax({
type: 'POST',
url: 'src/send_payment.php',
async: false,
data: {'data_value': 5489798123489725,'payment_amount': 10.5 },
success: function (response) {
//do whatever you want here }
});

Questions about the scopes, node.js, and express

I really suck at understanding scopes and other things of that nature in just about every language. Right now I am building an express application that takes user input and then queries an arbitrary api and then feeds it to the console. To handle the rest api, I am using shred. I know I can use nodes built in get request, but for some reason, I could never get it to work. The user makes the following get request to my app, /query?query=. This is what I have now. I can't really describe what I'm doing so pleas read the code comments.
var http = require('http');
var Shred = require("shred");
var assert = require("assert");
exports.query = function(req, res){
//thequery is the string that is requested
var thequery = req.query.query;
var shred = new Shred();
console.log("user searched" + " " + thequery);
console.log();
//The if statement detects if the user searched a url or something else
if (thequery.indexOf("somearbitratyrestapi.com") !== -1){
console.log("a url was searched");
//find info on the url
var thedata = shred.get({
url: "http://somearbitratyrestapi.com/bla/v2" + thequery,
headers: {
Accept: "application/json"
},
on: {
// You can use response codes as events
200: function(response) {
// Shred will automatically JSON-decode response bodies that have a
// JSON Content-Type
//This is the returned json
//I want to get this json Data outside the scope of this object
console(response.content.body);
},
// Any other response means something's wrong
response: function(response) {
console.log("ohknowz");
}
}
});
//I want to be able to see that json over here. How do?
}else{
console.log("another thing was searched");
}
/*
res.render('search-results', {
result: 'you gave me a url',
title: 'you gave me a url'
});
*/
};
I tried doing this
var http = require('http');
var Shred = require("shred");
var assert = require("assert");
exports.query = function(req, res){
//thequery is the string that is requested
var thequery = req.query.query;
var shred = new Shred();
//I created a variable outside of the object
var myjson;
console.log("user searched" + " " + thequery);
console.log();
//The if statement detects if the user searched a url or something else
if (thequery.indexOf("somearbitratyrestapi.com") !== -1){
console.log("a url was searched");
//find info on the url
var thedata = shred.get({
url: "http://somearbitratyrestapi.com/bla/v2" + thequery,
headers: {
Accept: "application/json"
},
on: {
// You can use response codes as events
200: function(response) {
// Shred will automatically JSON-decode response bodies that have a
// JSON Content-Type
//This is the returned json
//I set myjson to the returned json
myjson = response.content.body
},
// Any other response means something's wrong
response: function(response) {
console.log("ohknowz");
}
}
});
//Then I try to output the json and get nothing
console.log(myjson);
}else{
console.log("another thing was searched");
}
/*
res.render('search-results', {
result: 'you gave me a url',
title: 'you gave me a url'
});
*/
};
Sorry for the bad explanation of my problem. Can someone please help or explain what is going on.

So you think you need to move data out of your nested scope, but the opposite is true. Within the nested scope where you have access to your upstream JSON response, you need to access the res object and send it though:
myjson = response.content.body
res.send(myjson);
However, long term you'll need to do some more node tutorials and focus on how to use callbacks to avoid deeply nested function scopes.