We Keep Coding

XMLHttpRequest : how to navigate the "Link" header

Context
I'm using an XMLHttpRequest to request a resource from the GitHub API. Since the response is paginated, I want to use the Link header that is provided.
When logging request.getResponseHeader("link") in the response callback, I get either null (if not paginated) or something like:
<https://api.github.com/repositories/14173222/forks?sort=stargazers&per_page=100&page=2>; rel="next", <https://api.github.com/repositories/14173222/forks?sort=stargazers&per_page=100&page=4>; rel="last"
Question
Is there an easy way to navigate this returned String without having to parse it?
Something along the lines of request.getResponseHeader("link").get("next") and which would return https://api.github.com/repositories/14173222/forks?sort=stargazers&per_page=100&page=2
.
My actual goal is to extract the next page's number (here, for example, I'd like a function which returns 2 since it is the next page to be requested).

I won't mark this as an accepted answer since it doesn't completely answer the question, but this is nonetheless what I ended up doing:
/** Paginated request. Pages index start at 1. */
function request_fork_page(page_number, user, repo, token) {
// ...
/* Pagination (beyond 100 forks). */
const link_header = request.getResponseHeader("link");
if (link_header) {
let contains_next_page = link_header.indexOf('>; rel="next"');
if (contains_next_page !== -1) {
request_fork_page(++page_number, user, repo);
}
}
}

Zapier code: trigger multiple webhooks

I'm trying to send multiple webhooks based on the number of items in a JSON array. I'm using a example from:
how to trigger webhook from zapier code
I tried it and it almost works as planned. I have a problem that when I have a JSON array of 4 items, it sends 16 webhooks instead of 4. If I have A JSON array of 3 items, it sends out 9 webhooks instead of 3.
I use inputData.items for insertion of the JSON array. Does anyone know why the items in the JSON array are multiplied?
I used:
const elements = JSON.parse(inputData.items)
var body = elements;
var options = {
"url": "URL.COM",
"method": "POST",
"headers": {'Content-Type': 'application/json'},
"body": JSON.stringify(body)
},
requests = elements.map(mapDataToSettings);
function mapDataToSettings(elem) {
var settings = Object.assign({}, options);
settings.data = JSON.stringify(elem);
return settings;
};
Promise.all(requests.map(grabContent))
.then(function(data){ callback(null, {requestsMade: data});});
function grabContent(options) {
return fetch(options.url, options)
.then(function(res) {return res.json();});
};
Does anyone see why my webhooks are triggering to often?
Thanks

David here, from the Zapier Platform team.
It's hard to say for sure without seeing your zap, but my guess is that you're feeding .items in from a previous Code step where it's returned as an array? If so, you're running into an undocumented code feature where subsequent steps are run for each item in the array. This is usually desirable, but since you're doing a loop in the code, you don't need it.
Assuming that's the case, you've got two options:
Return a sjon string from the previous step (instead of an array) so this code step only runs once.
Change this code to only receive one item and perform a request with it.
If something else is going on, update your question with the zap id (it's not sensitive info) and I can take a look!

Can someone explain an ENOBUFS error?

I'm making a bunch calls to a database that contains a large amount of data on a Windows 7 64 bit OS. As the calls are queuing up I get the error (for ever HTTP call after the first error):
Error: connect ENOBUFS *omitted* - Local (undefined:undefined)
From my google searching I've learned that this error means that my buffer has grown too large and my system's memory can no longer handle the buffer's size.
But I don't really understand what this means. I'm using node.js to with an HTTPS library to handle my requests. When the requests are getting queued and the sockets are opening is the buffer's size allocated in RAM? What will allow the buffer to expand to a greater size? Is this simply a hardware limitation?
I've also read that some OS are able to handle the size of the buffer better than other OS's. Is this the case? If so which OS would be better suited for running a node script that needs to fetch a lot of data via HTTPS requests?
Here's how I'm doing my requests.
for (let j = 0; j < dataQueries; j++) {
getData(function())
}
function getData(callback){
axios.get(url, config)
.then((res) => {
// parse res
callback(parsedRes(res))
}).catch(function(err) {
console.log("Spooky problem alert! : " + err);
})
}
I've omitted some code for brevity, but this is generally how I'm doing my requests. I have a for loop that for every iteration launches a GET request via axios.
I know there is an axios.all command that is used for storing the promise the axios.HTTPMethod gives you, but I saw no change in my code when I set it up to store promises and then iterate over the promises via axios.all

Thanks #Jonasw for your help, but there is a very simple solution to this problem.
I used the small library throttled-queue to get the job done. (If you look at the source code it would be pretty easy to implement your own queue based on this package.
My code changed to:
const throttledQueue = require('throttled-queue')
let throttle = throttledQueue(15, 1000) // 15 times per second
for (let j = 0; j < dataQueries; j++) {\
throttle(function(){
getData(function(res){
// do parsing
})
}
}
function getData(callback){
axios.get(url, config)
.then((res) => {
// parse res
callback(parsedRes(res))
}).catch(function(err) {
console.log("Spooky problem alert! : " + err);
})
}

In my case this got resolved by deleting the autogenerated zip files from my workspace, which got created every time I did cdk deploy. Turns out that my typescript compiler treated these files as source files and counted them into the tarball.

Youre starting a lot of data Queries at the same time. You could chain them up using a partly recursive function, so that theyre executed one after another:
(function proceedwith(j) {
getData(function(){
if(j<dataQueries-1) proceedwith(j+1);
});
})(0)

Experienced the same issue when starting too many requests.
Tried throttled-queue, but wasn't working correctly.
system-sleep worked for me, effectively slowing down the rate at which the requests were made. Sleep is best used in synchronized code, to block before using sync/async code.
Example: (using sleep to limit the rate updateAddress() is called)
// Asynchronus call (what is important is that forEach is synchronous)
con.query(sql, function (err, result) {
if (err) throw err;
result.forEach(function(element) {
sleep(120); // Blocking call sleep for 120ms
updateAddress(element.address); // Another async call (network request)
});
});

I'm facing difficulty in creating cookie using node js with AWS Lambda

I i'm trying to create a cookie on node js side with the help of aws lambda. I have tried this below link to create cookie but I'm able to set cookie with context.done or context.successed. With this aproch cookie get printed on browser and your code get exits. What I'm trying to do its set a cookie and then execute other functions.
With navtive node js, you have to use http.createClient or createServer to get http.serverResponse but with aws lambda you can't do it.
AWS link to create cookie with AWS api gatewal and lambda. I'm able to achieve this.
https://aws.amazon.com/blogs/compute/simply-serverless-using-aws-lambda-to-expose-custom-cookies-with-api-gateway/
Edit
Adding code what I'm trying to do,
exports.handler = (event, context, callback) => {
// TODO implement
console.log("{'Cookie':event['Cookie']}");
var date = new Date();
// Get Unix milliseconds at current time plus 365 days
date.setTime(+ date + (365 * 86400000)); //24 \* 60 \* 60 \* 100
var cookieVal = Math.random().toString(36).substring(7); // Generate a random cookie string
var cookieString = "mycookie="+cookieVal+"; expires="+date.toGMTString()+";";
var errorXml = '<?xml version="1.0" encoding="UTF-8"?><VAST version="3.0"><error>Ad Not Available</error></VAST>';
console.log(event.params.header["Cookie"]);
if(event.params.header["Cookie"]){
console.log(errorXml);
callback(null, "test");
}else{
console.log("setting cookie");
callback(null, {errorXml,"Cookie":cookieString});
}
};
With I'm able to set the cookie but as my o/p is xml, and I'm passing Cookie in callback function (I think this is the only way to set cookie in node js using lambda. I may be wrong). application/xml is creating issue for valid xml. Once cookie is set code isn't going in else loop. In my application is I can check if Cookie is set in header or not and pass it to some variable. Now I'm having issue in setting cookie, with callback.

Writing this answer based on assumptions. In case it doesn't help, Please provide some code to get into details. Also, the exact code without any further additions in the Link you provided works for me.
With this approach cookie get printed on browser and your code get exits.
I suppose that your function exits before successful execution.
Reasons may be:
Timeout: This reason can be verified through the log [Task timed out after x seconds]
Lambda being Asynchronous : If the cookie gets printed after callback and function exits, it may be possible that your other functions have greater execution time and callback is called before others' execution completes.
This can be verified by adding a console.log("test data"); inside every other function to track their execution.
What I'm trying to do its set a cookie and then execute other functions.
well, if your cookie doesn't need any data from 'other functions', this can be achieved, but if you call context or any callbacks outside 'other functions', chances are, asynchronous Node will execute context.done() first.
My suggestions:
case 1:
If you have nested functions and cookie needs data from one of those--
You should go for promises, this will ensure, data is returned and post that cookie can be set.
More about Promise can be found: Here
var Promise = require('promise');
var promise = new Promise(function (resolve, reject) {
get('http://www.google.com', function (err, res) {
if (err) reject(err);
else resolve(res);
});
});
case 2:
If no dependency on other functions--
Try using callback
something like :
exports.handler = function(event, context, callback) {
otherFunctions(){}
callback(null, {"Cookie": cookieString});
}

Parse iOS SDK: Understanding Cloud Code

Scenario = I am slowly but surely wrapping my head around what is going on with Parse's cloud code features. I just need some help from those who would like to answer some short, relatively simple questions about what is going on in some sample cloud code functions.
The code I will use in this example is below
1) cloud code
Parse.Cloud.define('editUser', function(request, response) {
var userId = request.params.userId,
newColText = request.params.newColText;
var User = Parse.Object.extend('_User'),
user = new User({ objectId: userId });
user.set('new_col', newColText);
Parse.Cloud.useMasterKey();
user.save().then(function(user) {
response.success(user);
}, function(error) {
response.error(error)
});
});
2) called from iOS
[PFCloud callFunction:#"editUser" withParameters:#{
#"userId": #"someuseridhere",
#"newColText": #"new text!"
}];
This code was taken from here
Question 1 =
(request, response)
I am confused by what this is. Is this like typecasting in iOS where I am saying (in the iOS call) I want to pass an NSString into this function ("userId") and inside the cloud code function I'm going to call it "request"? Is that what's going on here?
Question 2 =
Parse.Object.extend('_User')
Is this grabbing the "User" class from the Parse database so that a "PFObject" of sorts can update it by creating a new "user" in the line below it?
Is this like a...
PFObject *userObject = [PFObject objectWithClassName:#"User"]?
Question 3 =
user.set('new_col', newColText)
This obviously 'sets' the values to be saved to the PFUser (~I think). I know that the "newColText" variable is the text that is to be set - but what is 'new_col'? Only thing I can think of is that this sets the name of a new column in the database of whatever type is being passed through the "request"?
Is this like a...
[[PFUser currentUser] setObject: forKey:]
Question 4 =
Parse.Cloud.useMasterKey()
Without getting too technical, is this basically all I have to type before I can edit a "User" object from another User?
Question 5 =
user.save().then(function(user) {
response.success(user);
}
Is this like a...
[user saveInBackgroundWithBlock:]?
and if so, is
function(error) {
response.error(error)
just setting what happens if there is an error in the saveInBackgroundWithBlock?
Please keep in mind, I know iOS - not JavaScript. So try to be as descriptive as possible to someone who understands the Apple realm.

Here's my take on your questions:
The request parameter is for you to access everything that is part of the request/call to your cloud function, it includes the parameters passed (request.params), the User that is authenticated on the client (request.user) and some other things you can learn about in the documentation. The response is for you to send information back to the calling code, you generally call response.success() or response.error() with an optional string/object/etc that gets included in the response, again documentation here.
That's a way of creating an instance of a User, which because it is a special internal class is named _User instead, same with _Role and _Installation. It is creating an instance of the user with an ID, not creating a new one (which wouldn't have an ID until saved). When you create an object this way you can "patch" it by just changing the properties you want updated.
Again, look at the documentation or an example, the first parameter is the column name (it will be created if it doesn't exist), the second value is what you want that column set to.
You have to do Parse.Cloud.useMasterKey() when you need to do something that the user logged into the client doesn't have permission to do. It means "ignore all security, I know what I'm doing".
You're seeing a promise chain, each step in the chain allows you to pass in a "success" handler and an optional "error" handler. There is some great documentation. It is super handy when you want to do a couple of things in order, e.g.
Sample code:
var post = new Parse.Object('Post');
var comment = new Parse.Object('Comment');
// assume we set a bunch of properties on the post and comment here
post.save().then(function() {
// we know the post is saved, so now we can reference it from our comment
comment.set('post', post);
// return the comment save promise, so we can keep chaining
return comment.save();
}).then(function() {
// success!
response.success();
}, function(error) {
// uh oh!
// this catches errors anywhere in the chain
response.error(error);
});

I'm pretty much at the same place as you are, but here are my thoughts:
No, these are the parameters received by the function. When something calls the editUser cloud function, you'll have those two objects to use: request & response. The request is basically what the iOS device sent to the server, and response is what the server will send to the iOS device.
Not quite that. It's like creating a subclass of _User.
Think of Parse objects types as a database table and it's instances as rows. The set will set (derp) the value of 'newColText' to the attribute/column 'new_col'.
Not sure, never used that function as I don't handle User objects. But might be that.
Pretty much that. But it's more sort of like (pseudo-code, mixing JS with Obj-C):
[user saveInBackgroundWithBlock:^(BOOL succeeded, NSError *error){
if(error){
response.error(error); // mark the function as failed and return the error object to the iOS device
}
else{
response.success(user); // mark the function call as successful and return the user object to the iOS device
}
}];