We Keep Coding

How more than 2 parallel api requests can change database without skipping one or more of them?

I have an endpoint uploads images and updates a database table.
I send 3 requests to this endpoint at same time. Actually this problem happens when I send more than 2 API requests.
First request that comes to endpoint uploads images and updates database table successfully.
Second request that comes to endpoint uploads images sees database changes of first request, and updates database table successfully.
Third request that comes to endpoint uploads images, doesn't see database changes of second request, and updates database table successfully.
As a result; only database changes of first request and third request apply. Database changes of second request is not able to applied successfully or is overridden, somehow.
I use pg npm package.
Is problem in my code or in pg package.
How can this problem be solved?
Controller:
#UseStaffPermissionsGuards('upsert', 'VehicleCondition')
#ApiBody({ type: VehiclePhotoConditionInfoImageDTO })
#ApiResponse({ status: 201 })
#Post(':id/photos/:photoConditionId/image')
#ApiConsumes('multipart/form-data')
#UseInterceptors(FilesInterceptor('images'), FilesToBodyInterceptor)
async upsertImages(
#Param('id') vehicleId: string,
#Param('photoConditionId') photoConditionId: string,
#Body() vehiclePhotoConditionInfoImages: VehiclePhotoConditionInfoImageDTO,
): Promise<void> {
return this.vehiclePhotoConditionService.upsertImages(
vehicleId,
photoConditionId,
vehiclePhotoConditionInfoImages,
);
}
Service:
async upsertImages(
vehicleId: string,
vehiclePhotoConditionId: string,
vehiclePhotoConditionImage: VehiclePhotoConditionInfoImageDTO,
): Promise<void> {
await this.isVehicleExist(vehicleId);
const vehiclePhotoCondition = await this.getOne(vehicleId, vehiclePhotoConditionId);
if (!vehiclePhotoCondition) {
throw new BadRequestException(
`The vehicle photo condition ${vehiclePhotoConditionId} is not found`,
);
}
const imageKeys = await this.handleImages(vehiclePhotoConditionId, vehiclePhotoConditionImage);
const updatedVehiclePhotoConditions = vehiclePhotoCondition.info.map((data) => {
if (data.vehiclePart === vehiclePhotoConditionImage.vehiclePart) {
data.uploadedImagesKeys.push(...imageKeys);
}
return data;
});
const query = sql
.update('vehicle_photo_condition', {
info: JSON.stringify(updatedVehiclePhotoConditions),
updated_at: sql('now()'),
})
.where({ id: vehiclePhotoConditionId });
await this.db.query(query.toParams());
}

I solved the problem. I am posting correct code.
Here is the explanation:
In previous code, I was updating inner jsonb array of objects in code and because of the fact that the below code took some time and asynchronicity of the NodeJS, previous request can take more time than the other requests that will come later and this situation can cause data inconsistency.
Here is the previous code:
const updatedVehiclePhotoConditions = vehiclePhotoCondition.info.map((data) => {
if (data.vehiclePart === vehiclePhotoConditionImage.vehiclePart) {
data.uploadedImagesKeys.push(...imageKeys);
}
return data;
});
In current code, I am updating inner jsonb array of object in database and let database do this operation. So, no data consistency happened.
Here is current code:
const query = {
text: `
UPDATE vehicle_photo_condition
SET info = s.json_array
FROM (
SELECT
jsonb_agg(
CASE WHEN obj ->> 'vehiclePart' = $1 THEN
jsonb_set(obj, '{uploadedImagesKeys}', $2)
ELSE obj END
) as json_array
FROM vehicle_photo_condition, jsonb_array_elements(info) obj WHERE id = $3
) s WHERE id = $3`,
values: [
vehiclePhotoConditionImage.vehiclePart,
JSON.stringify(imageKeys),
vehiclePhotoConditionId,
],
};
async upsertImages(
vehicleId: string,
vehiclePhotoConditionId: string,
vehiclePhotoConditionImage: VehiclePhotoConditionInfoImageDTO,
): Promise<void> {
const vehiclePhotoCondition = await this.getOne(vehicleId, vehiclePhotoConditionId);
if (!vehiclePhotoCondition) {
throw new BadRequestException(
`The vehicle photo condition ${vehiclePhotoConditionId} is not found`,
);
}
const imageKeys = await this.handleImages(vehiclePhotoConditionId, vehiclePhotoConditionImage);
const query = {
text: `
UPDATE vehicle_photo_condition
SET info = s.json_array
FROM (
SELECT
jsonb_agg(
CASE WHEN obj ->> 'vehiclePart' = $1 THEN
jsonb_set(obj, '{uploadedImagesKeys}', $2)
ELSE obj END
) as json_array
FROM vehicle_photo_condition, jsonb_array_elements(info) obj WHERE id = $3
) s WHERE id = $3`,
values: [
vehiclePhotoConditionImage.vehiclePart,
JSON.stringify(imageKeys),
vehiclePhotoConditionId,
],
};
await this.db.query(query);
}

Strapi closing connection to host before response

I have a local Java Service that exposes a single endpoint where the response can take over 5min sometimes depending on the request size. I want to "hide" this endpoint behind Strapi, so that a user that is not authenticated through Strapi, can't access this endpoint. What i've done is, created a custom Route in Strapi, and created a custom controller function for this route:
async optimizeRoute(ctx) {
try {
console.log(`${new Date()} - Optimize route started`)
const vehicles = ctx.request.body.vehicles.map(vehicle => {
return {
id: vehicle.id,
licenseplate: vehicle.licenseplate,
maxweight: vehicle.maxweight,
maxlength: vehicle.maxlength,
geojson: vehicle.geojson,
maxtime: moment(vehicle.endtime).diff(moment(vehicle.starttime), 'seconds') * 1.1
}
});
const orders = ctx.request.body.orders.map(order => {
return {
id: order.id,
weight: order.weight,
length: order.length,
geojson: order.geojson
}
});
console.log(`Using ${vehicles.length} vehicles, and ${orders.length} orders!`)
const response = await axios.post('http://127.0.0.1:9090/optimizeroutes',
{ vehicles: vehicles, orders: orders }, {timeout: 0}
);
return response.data;
} catch (error) {
ctx.throw(error.response.status, error.response);
}
}
But what happens when i do this, is Strapi closes the connection after 1-2 min to the requester before the response is returned.
Using Strapi Beta3.17.5.
Anywhere i can configure or debug to wait for the axios to return a response then return the response to the requester??

The default server timeout value is 2 min, Please check the image for reference.
So in server.js you can increase this timeout value by passing callback function
So the code will look like below
const strapi = require('strapi');
strapi(/* {...} */).start(()=>{
strapi.server.setTimeout(0);
});
setting timeout to 0 strapi will wait forever or you can add value in milliseconds.

Thank you for putting me on the right track #NS23, but i couldn't user server.js to start my server, and console logging strapi.server returned undefined.
What i did though was access strapi.server in Bootstrap.js, which then logging strapi.server, could see the timeout was changed from 120000 to 0

Array of JSON object is not empty but cannot iterate with foreach, show zero length [duplicate]

This question already has an answer here:
How to get data from firestore DB in outside of onSnapshot
(1 answer)
Closed 3 years ago.
I read a collection about book checkout from firestore in create().
The data is store in checkout array that declared in data().
export default {
name: "checkout-history",
data() {
return {
checkout: []
]
};
},
created() {
db.collection("checkout")
.get()
.then(querySnapshot => {
querySnapshot.forEach(doc => {
const data = {
id: doc.id, // firebase document id
book_did: doc.data().book_did,
borrowed_date: doc.data().borrowed_date,
copies_did: doc.data().copies_did,
due_date: doc.data().due_date
};
this.checkout.push(data); // push to array
});
});
console.log(this.checkout) // show data, as in the image below
console.log(this.checkout.length) // get 0
console.log(Object.keys(this.checkout).length) // get 0
}
......
When I run console.log(this.checkout);, console show this:
However, I cannot iterate it, the console show this.checkout.length is 0
I also tried to use Object.keys but no luck.
Object.keys(this.checkout).forEach(key => {
const keys = this.checkout[key];
console.log(keys);
});
I really don't know what to do anymore.
I read many answers online and tried most of them, but none of them work.

I guess you are executing your code before the completion of the request.
If you hover over the little blue i icon, it says:
Value below was evaluated just now.

Data is loaded from Firestore (and from most modern web APIs) asynchronously. This means that the rest of your code continues to execute after you start the query, and then when the data comes back from the database, your then() callback is called. This in turn means that all code that needs access to the data from the database, must be inside the then() callback.
db.collection("checkout")
.get()
.then(querySnapshot => {
querySnapshot.forEach(doc => {
const data = {
id: doc.id, // firebase document id
book_did: doc.data().book_did,
borrowed_date: doc.data().borrowed_date,
copies_did: doc.data().copies_did,
due_date: doc.data().due_date
};
this.checkout.push(data); // push to array
console.log(this.checkout) // show data, as in the image below
console.log(this.checkout.length) // get 0
console.log(Object.keys(this.checkout).length) // get 0
});
});
}
......
Also see:
How to get data from firestore DB in outside of onSnapshot
Why are Firebase APIs asynchronous?

nodejs recursively call same api and write to excel file sequentially

I need to call an API recursively using request promise after getting result from API need to write in an excel file , API sample response given below
{
"totalRecords": 9524,
"size": 20,
"currentPage": 1,
"totalPages": 477,
"result": [{
"name": "john doe",
"dob": "1999-11-11"
},
{
"name": "john1 doe1",
"dob": "1989-12-12"
}
]
}
Now I want to call this API n times, here n is equal to totalPages, after calling each API I want to write response result to the excel files.
First write page 1 response result to excel then append page 2 response result to excel file and so on..
I have written some sample code given below
function callAPI(pageNo) {
var options = {
url: "http://example.com/getData?pageNo="+pageNo,
method: 'GET',
headers: {
'Content-Type': 'application/json'
},
json: true
}
return request(options)
}
callAPI(1).then(function (res) {
// Write res.result to excel file
}).catch(function (err) {
// Handle error here
})
But facing problem calling recursively API and maintaining sequentially like write page 1 result first to excel file then page 2 result append to excel and so on..
Any code sample how to achieve in nodejs

You want to do something like this:
function getAllPages() {
function getNextPage(pageNo) {
return callAPI(pageNo).then(response => {
let needNextPage = true;
if (pageNo === 1) {
// write to file
} else {
// append to file
}
if (needNextPage) {
return getNextPage(pageNo+1);
} else {
return undefined;
}
});
}
return getNextPage(1);
}
Obviously change that 'needNextPage' to false to stop the recursion when you're done

So you want to do 477 requests in sequence? How long do you wanna wait for this to finish? Even in paralell, this would be still too long for me.
Best: write an API that can return you a batch of pages at once. Reducing the number of requests to the backend. Maybe something like http://example.com/getData?pages=1-100 and let it return an Array; maybe like
[
{
"totalRecords": 9524,
"currentPage": 1,
"totalPages": 477,
"result": [...]
},
{
"totalRecords": 9524,
"currentPage": 2,
"totalPages": 477,
"result": [...]
},
...
]
or more compact
{
"totalRecords": 9524,
"totalPages": 477,
"pages": [
{
"currentPage": 1,
"result": [...]
},
{
"currentPage": 2,
"result": [...]
},
...
]
}
Sidenote: writing the size of the results array into the json is unnecessary. This value can easily be determined from data.result.length
But back to your question
Imo. all you want to run in sequence is adding the pages to the sheet. The requests can be done in paralell. That already saves you a lot of overall runtime for the whole task.
callApi(1).then(firstPage => {
let {currentPage, totalPages} = firstPage;
//`previous` ensures that the Promises resolve in sequence,
//even if some later request finish sooner that earlier ones.
let previous = Promise.resolve(firstPage).then(writePageToExcel);
while(++currentPage <= totalPages){
//make the next request in paralell
let p = callApi(currentPage);
//execute `writePageToExcel` in sequence
//as soon as all previous ones have finished
previous = previous.then(() => p.then(writePageToExcel));
}
return previous;
})
.then(() => console.log("work done"));
or you wait for all pages to be loaded, before you write them to excel
callApi(1).then(firstPage => {
let {currentPage, totalPages} = firstPage;
let promises = [firstPage];
while(++currentPage < totalPages)
promises.push(callApi(currentPage));
//wait for all requests to finish
return Promise.all(promises);
})
//write all pages to excel
.then(writePagesToExcel)
.then(() => console.log("work done"));
or you could batch the requests
callApi(1).then(firstPage => {
const batchSize = 16;
let {currentPage, totalPages} = firstPage;
return Promise.resolve([ firstPage ])
.then(writePagesToExcel)
.then(function nextBatch(){
if(currentPage > totalPages) return;
//load a batch of pages in paralell
let batch = [];
for(let i=0; i<batchSize && ++currentPage <= totalPages; ++i){
batch[i] = callApi(currentPage);
}
//when the batch is done ...
return Promise.all(batch)
//... write it to the excel sheet ...
.then(writePagesToExcel)
//... and process the next batch
.then(nextBatch);
});
})
.then(() => console.log("work done"));
But don't forget to add the error handling. Since I'm not sure how you'd want to handle errors with the approaches I've posted, I didn't include the error-handling here.
Edit:
can u pls modify batch requests, getting some error, where you are assigning toalPages it's not right why the totalPages should equal to firstPage
let {currentPage, totalPages} = firstPage;
//is just a shorthand for
let currentPage = firstPage.currentPage, totalPages = firstPage.totalPages;
//what JS version are you targeting?
This first request, callApi(1).then(firstPage => ...) is primarily to determine currentIndex and totalLength, as you provide these properties in the returned JSON. Now that I know these two, I can initiate as many requests in paralell, as I'd want to. And I don't have to wait for any one of them to finish to determine at what index I am, and wether there are more pages to load.
and why you are writing return Promise.resolve([ firstPage ])
To save me some trouble and checking, as I don't know anything about how you'd implement writePagesToExcel.
I return Promise.resolve(...) so I can do .then(writePagesToExcel). This solves me two problems:
I don't have to care wether writePagesToExcel returns sync or a promise and I can always follow up with another .then(...)
I don't need to care wether writePagesToExcel may throw. In case of any Error, it all ends up in the Promise chain, and can be taken care of there.
So ultimately I safe myself a few checks, by simply wrapping firstPage back up in a Promise and continue with .then(...). Considering the amounts of data you're processing here, imo. this ain't too much of an overhead to get rid of some potential pitfalls.
why you are passing array like in resolve
To stay consistent in each example. In this example, I named the function that processes the data writePagesToExcel (plural) wich should indicate that it deals with multiple pages (an array of them); I thought that this would be clear in that context.
Since I still need this seperate call at the beginning to get firstPage, and I didn't want to complicate the logic in nextBatch just to concat this first page with the first batch, I treat [firstPage] as a seperate "batch", write it to excel and continue with nextBatch

function callAPI(pageNo) {
var options = {
url: "http://example.com/getData?pageNo="+pageNo,
method: 'GET',
headers: {
'Content-Type': 'application/json'
},
json: true
}
return request(options)
}
function writeToExcel(res){console.log(res)} //returns promise.
callAPI(1).then(function (res) {
if(res){
writeToExcel(res).then(() => {
var emptyPromise = new Promise(res => setTimeout(res, 0));
while(res && res.currentPage < res.totalPages){
emptyPromise = emptyPromise.then(() => {
return callAPI(res.currentPage).then(function (res){
if(res){
writeToExcel(res)
}
});
}
}
return emptyPromise;
});
}
}).catch(function (err) {
// Handle error here
})

Fetch, store and use JSON variable with express

I've been struggling trying to understand how to access and and process data using HTTP method routes (get, put, post...). So far, I've been able to fetch the JSON data and store it in a global variable.
var pokedata;
fetch('https://raw.githubusercontent.com/Biuni/PokemonGO-Pokedex/master/pokedex.json')
.then(function (res) {
return res.json();
}).then(function (json) {
pokedata = json;
}).catch(function () {
console.log("It was not possible to fetch the data.")
});
I wanted to send responses HTTP GETs to http://localhost:3000/pokemon/XXX/ with some data about that Pokémon number XXX (which is in the JSON called pokedata). However, any attempt at looping through the data inside GET, triggers the error:
app.get('/pokemon/:pokemonid', function (req, res) {
//not the desired behaviour but a sample of what doesn't work.
for (let {name: n, weight: w, height: h} of pokedata) {
res.send(n, w, h);
}
});
TypeError: pokedata[Symbol.iterator] is not a function
Can't seem to find anything related in the express docs. Any help is well received.

pokedata is an object, and you don't want to iterate on it. Instead you want to iterate on pokedata.pokemon, which is an array of the pokemon. So a small modification to your code is all that's needed:
for (let {name: n, weight: w, height: h} of pokedata.pokemon) {
res.send(n, w, h);
}