I'm writing a wrapper around fetch that I would like to add something to the URL before making the request e.g. identifying query parameters. I can't figure out how to make a copy of a given a Request object with a different URL than the original. My code looks like:
// My function which tries to modify the URL of the request
function addLangParameter(request) {
const newUrl = request.url + "?lang=" + lang;
return new Request(newUrl, /* not sure what to put here */);
}
// My fetch wrapper
function myFetch(input, init) {
// Normalize the input into a Request object
return Promise.resolve(new Request(input, init))
// Call my modifier function
.then(addLangParameter)
// Make the actual request
.then(request => fetch(request));
}
I tried putting the original request as the second arguent to the Request constructor, like so:
function addLangParameter(request) {
const newUrl = request.url + "?lang=" + lang;
return new Request(newUrl, request);
}
which seems to copy most of the attributes of the old request but doesn't seem to preserve the body of the old request. For example,
const request1 = new Request("/", { method: "POST", body: "test" });
const request2 = new Request("/new", request1);
request2.text().then(body => console.log(body));
I would expect to log "test", but instead it logs the empty string, because the body is not copied over.
Do I need to do something more explicit to copy all of the attributes correctly, or is there a nice shortcut that will do something reasonable for me?
I'm using the github/fetch polyfill, but have tested with both the polyfill and the native fetch implementation in the lastest Chrome.
It looks like your best bet is to read the body using the Body interface that Requests implement:
https://fetch.spec.whatwg.org/#body
This can only be done asynchronously since the underlying "consume body" operation always reads asynchronously and returns a promise. Something like this should work:
const request = new Request('/old', { method: 'GET' });
const bodyP = request.headers.get('Content-Type') ? request.blob() : Promise.resolve(undefined);
const newRequestP =
bodyP.then((body) =>
new Request('/new', {
method: request.method,
headers: request.headers,
body: body,
referrer: request.referrer,
referrerPolicy: request.referrerPolicy,
mode: request.mode,
credentials: request.credentials,
cache: request.cache,
redirect: request.redirect,
integrity: request.integrity,
})
);
After doing that, newRequestP will be a promise that resolves to the request you want. Luckily, fetch is asynchronous anyway so your wrapper shouldn't be significantly hampered by this.
(Note: Reading the body using .blob() off of a request that does not have a body seems to return a zero-length Blob object, but it's incorrect to specify any body, even a zero-length one, on a GET or HEAD request. I believe that checking if the original request had Content-Type set is an accurate proxy for whether it has a body, which is what we really need to determine.)
Related
I started writing a program that will automate user actions, by now it's meant to be an easier menu to make faster actions by just sending requests to the official website by clicks on my own page. (something like web-bot but not exactly).
My problem is when i send login request in response i get back user_id, server, session_id etc. And I need to save that session_id to make the other actions.
How can i save this to variable.
All in JavaScript.
I was looking in the internet since yesterday for an answer and i still can't find (or understand) how to get this
I tried
function login(){ fetch('url', { method: 'POST', headers: { //headers }, body: //my Id's }) })
//There's the problem to solve
.then(res => res.text()) .then(data => obj = data) .then(() => console.log(obj)) console.log(body.session_id);
// I even tried the substring but 1. It won't work as I want because There are sometimes //more and less letters. 2. I get and error "Cannot read properties of undefined (reading //'substr')"
`session = obj;
session_id = session.substring(291,30)
console.log(session_id)`
It looks like you're using the text() method on the Response object returned from fetch(), which will give you a string representation of the response.
You probably want to be using the json() method instead, and from that you can get your session_id.
This guide has some more useful information that may help you: https://javascript.info/fetch
Ok it works now with
`async function login(){ let session = await fetch('url', {
//code
}
let result = await session.json();
console.log(result);
session_id = result.data.user.session_id;
user_id = result.data.user.id;`
When a new domain is attached, I made a logic to check whether the domain works normally through fetch.
fetch("https://" + domain).then((result)=>{
const isOk = result.status === 200
...
})
However, I thought it would be inefficient to fetch the entire HTML doc through fetch just to check if it works.
How can I get only the status value?
You can request only the headers, as opposed to the whole page, using the HEAD method. The server should return the same headers as it would if the GET method was used, but no body. You can request this behaviour using the method option:
fetch(url, { method: 'HEAD' })
Event shouldCrawl belongs to js-crawler's config object, and the callback function it has as value must return boolean in order to tell the crawler whether or not to crawl the URL received as an argument.
I'm using axios and HEAD method to retrieve the resource's headers. Will return true to shouldCrawl when content-type contains text/html in order to prevent the crawler from downloading files and garbage.
My code:
this.crawler = new Crawler().configure({
shouldCrawl: async(sUrl)=> {
const crawlWhenHtml = async()=> { //return false;
return axios({
url: sUrl,
method: 'head'
}).then(res=>{
return (res.headers['content-type'].indexOf('text/html') >= 0?
true:false);
}).catch(error=>{
return false;
});
}
return await crawlWhenHtml();
}
});
I can't get shouldCrawl and crawlWhenHtml in sync.
If I make the callback returns false (see commented sentence), shouldCrawl ignores it and crawls the URL anyway. This happens since I made the mentioned callback async.
But without making it async I cant't wait for axios completes the request before returning a boolean to shouldCrawl.
How can I unravel this?
So I am testing a cloudflare web worker script and i can't seem to get my code to work with POST requests and such.
url_without_query_strings = request.url.split('?')[0] //remove all query strings
const response = await fetch(url_without_query_strings, {
method: request.method,
headers: request.headers
})
return response
Can anyone see what I am doing wrong ?
The problem is that you are only copying method and headers from the request, but it has more properties than that. POST requests, for example, have a body property which your code is not copying.
In order to perform a fetch that inherits everything from the original request except the URL, do:
const response = await fetch(url_without_query_strings, request)
That is, pass the request itself as the second parameter, rather than a dict. This works because a request object has property names matching exactly all of the options that fetch()'s second parameter expects.
Note that, awkwardly, if you want to modify any request property other than the URL, but keep other properties the same, then you must pass the request as the first parameter and specify the modifications in the second parameter:
const response = await fetch(request, {headers: myHeaders})
This means that if you want to modify the URL and some other property, you need to perform two steps -- first created a new Request object that changes the URL, then modify the headers:
let request2 = new Request(newUrl, request)
const response = await fetch(request2, {headers: myHeaders})
Or, of course, you could do the opposite order:
let request2 = new Request(request, {headers: myHeaders})
const response = await fetch(newUrl, request2)
Or, alternatively, for the specific case of headers, you can take advantage of the fact that once you've constructed your own Request object, you're allowed to directly modify its headers:
let request2 = new Request(newUrl, request)
request2.headers.set("X-Foo", "Bar")
const response = await fetch(request2)
Specifically I am interested in changing all responses with code 403 to code 404, and changing all responses with code 301 to 302. I do not want any other part of the response to change, except the status text (which I want to be empty). Below is my own attempt at this:
addEventListener("fetch", event => {
event.respondWith(fetchAndModify(event.request));
});
async function fetchAndModify(request) {
// Send the request on to the origin server.
const response = await fetch(request);
const body = await response.body
newStatus = response.status
if (response.status == 403) {
newStatus = 404
} else if (response.status == 301) {
newStatus = 302
}
// Return modified response.
return new Response(body, {
status: newStatus,
statusText: "",
headers: response.headers
});
}
I have confirmed that this code works. I would like to know if there is any possibility at all that this overwrites part of the response other than the status code or text, and if so, how can I avoid that? If this goes against certain best practices of Cloudflare workers or javascript, please describe which ones and why.
You've stumbled on a real problem with the Fetch API spec as it is written today.
As of now, status, statusText, and headers are the only standard properties of Response's init structure. However, there's no guarantee that they will remain the only properties forever, and no guarantee that an implementation doesn't provide additional non-standard or not-yet-standard properties.
In fact, Cloudflare Workers today implements a non-standard property: webSocket, which is used to implement WebSocket proxying. This property is present if the request passed to fetch() was a WebSocket initiation request and the origin server completed a WebSocket handshake. In this case, if you drop the webSocket field from the Response, WebSocket proxying will break -- which may or may not matter to you.
Unfortunately, the standard does not specify any good way to rewrite a single property of a Response without potentially dropping unanticipated properties. This differs from Request objects, which do offer a (somewhat awkward) way to do such rewrites: Request's constructor can take another Request object as the first parameter, in which case the second parameter specifies only the properties to modify. Alternately, to modify only the URL, you can pass the URL as the first parameter and a Request object as the second parameter. This works because a Request object happens to be the same "shape" as the constructor's initializer structure (it's unclear if the spec authors intended this or if it was a happy accident). Exmaples:
// change URL
request = new Request(newUrl, request);
// change method (or any other property)
request = new Request(request, {method: "GET"});
But for Response, you cannot pass an existing Response object as the first parameter to Response's constructor. There are straightforward ways to modify the body and headers:
// change response body
response = new Response(newBody, response);
// change response headers
// Making a copy of a Response object makes headers mutable.
response = new Response(response.body, response);
response.headers.set("Foo", "bar");
But if you want to modify status... well, there's a trick you can do, but it's not pretty:
// Create an initializer by copying the Response's enumerable fields
// into a new object.
let init = {...response};
// Modify it.
init.status = 404;
init.statusText = "Not Found";
// Work around a bug where `webSocket` is `null` but needs to be `undefined`.
// (Sorry, I only just noticed this when testing this answer! We'll fix this
// in the future.)
init.webSocket = init.webSocket || undefined;
// Create a new Response.
response = new Response(response.body, init);
But, ugh, that sure was ugly.
I have proposed improvements to the Fetch API to solve this, but I haven't yet had time to follow through on them. :(