I'm trying to update a column in my PostgresQL database using Nodejs:
res.rows.forEach((tmdbID) => {
(async () => {
const json = await fetchMovieData(tmdbID.tmdb_id);
const overview = json.overview.replace('\'', '\\\'');
console.log(overview);
pool.query(`UPDATE "Movie" SET overview = '${overview}' WHERE tmdb_id = ${json.id}`);
})();
});
async function fetchMovieData(tmdbID) {
const response = await fetch(`https://api.themoviedb.org/3/movie/${tmdbID}?api_key=a8f7039633f2065942cd8a28d7cadad4&language=en-US`);
const data = response.json();
return data;
}
The error I'm getting:
(node:1412) UnhandledPromiseRejectionWarning: error: syntax error at or near "s"
It happens on this string:
The Bride unwaveringly continues on her roaring rampage of revenge against the band of assassins who had tried to kill her and her unborn child. She visits each of her former associates one-by-one, checking off the victims on her Death List Five until there \'s nothing left to do … but kill Bill.
I'm trying to escape the 'but it doesn't seem to be working. Any tips?
Don't try to pass everything as a String, that is how SQL Injection happens. A better approach is to use parameters.
I'm not exactly sure which library are you using, but the syntax should be something similar to this:
pool.query('UPDATE "Movie" SET overview = ? WHERE tmdb_id = ?', [overview, json.id]);
Related
I am using Couchbase in a node app. Every time I insert a document, I am using a random UUID.
It inserts fine and I could retrieve data based on this id.
But in reality, I actually want to search by a key called url in the document. To be able to get or update or delete a document.
I could possibly add the url as the id I suppose but that is not what I see in any database concepts. Ids are not urls
or any unique names. They are typically random numbers or incremental numbers.
How could I approach this so that I can use a random UUID as id but be able to search by url?
Cos lets say the id was 56475-asdf-7856, I am not going to know this value to search for right.
Whereas if the id was https://www.example.com I know about this url and searching for it would give me what I want.
Is it a good idea making the url the id.
This is in a node app using Couchbase.
databaseRouter.put('/update/:id', (req, res) => {
updateDocument(req)
.then(({ document, error }) => {
if (error) {
res.status(404).send(error);
}
res.json(document);
})
.catch(error => res.status(500).send(error));
});
export const updateDocument = async (req) => {
try {
const result = await collection.get(req.params.id); // Feels like id should be the way to do this, but doesn't make sense cos I won't know the id beforehand.
document.url = req.body.url || document.url;
await collection.replace(req.params.id, document);
return { document };
} catch (error) {
return { error };
}
};
I think it's okay to use URLs as IDs, especially if that's the primary way you're going to lookup documents, and you don't need to change the URL later. Yes, often times IDs are numbers or UUIDs, but there is no reason you have to be restricted to this.
However, another approach you can take is to use a SQL query (SQL++, technically, since this is a JSON database).
Something like:
SELECT d.*
FROM mybucket.myscope.mydocuments d
WHERE d.url = 'http://example.com/foo/baz/bar'
You'll also need an index with that, something like:
CREATE INDEX ix_url ON mybucket.myscope.mydocuments (url)
I'd recommend checking out the docs for writing a SQL++ query (sometimes still known as "N1QL") with Node.js: https://docs.couchbase.com/nodejs-sdk/current/howtos/n1ql-queries-with-sdk.html
Here's the first example in the docs:
async function queryPlaceholders() {
const query = `
SELECT airportname, city FROM \`travel-sample\`.inventory.airport
WHERE city=$1
`;
const options = { parameters: ['San Jose'] }
try {
let result = await cluster.query(query, options)
console.log("Result:", result)
return result
} catch (error) {
console.error('Query failed: ', error)
}
}
In my programming in Node.JS I made some code that worked, but I want to understand the concept of how it works.
In the code in question I use knex to retrieve information from a MySQL database. I import the knex module for that:
const config = {...}
const knex = require('knex')(config)
So far nothing new, only this time I needed to do a nested query and it was the first time. In this case, I consulted the sales data and then the sale items. I did it as follows:
const getSales = async () => {
await knex("tbl_vendas")
.select(knex.raw("tbl_vendas.id_vendas, tbl_vendedores.nome AS vendedor, " +
"tbl_clientes.nome_razaosocial AS cliente, tbl_vendas.data, tbl_vendas.hora, " +
"tbl_vendas.cupom, tbl_vendas.total"))
.leftOuterJoin("tbl_vendedores", "tbl_vendedores.id_vendedores", "tbl_vendas.id_vendedores")
.leftOuterJoin("tbl_clientes", "tbl_clientes.id_clientes", "tbl_vendas.id_clientes")
.then(sales => {
const rows = sales.map(sale => {
return knex("tbl_vendas_itens")
.select("tbl_vendas_itens.id_vendas_itens AS id_item", "tbl_produtos.descricao",
"tbl_vendas_itens.qtde", "tbl_vendas_itens.vl_unitario", "tbl_vendas_itens.desconto",
"tbl_vendas_itens.vl_total")
.leftOuterJoin("tbl_produtos", "tbl_vendas_itens.id_produtos", "tbl_produtos.id_produtos")
.where("tbl_vendas_itens.id_vendas", "=", sale.id_vendas)
.then(sales_items => {
const newRow = { ...sale, itens: [...sales_items] }
return newRow
})
})
return Promise.all(rows)
})
.then(console.log);
}
Writing this code was pretty intuitive and it worked, but then I was amazed that I used the knex constant twice, one inside the other, and it didn't hurt.
I ran a console.log(typeof(knex)) to find out what it was and it returned that it is a function.
Could someone explain the theory behind the use of knex inside the other and help me understand why it is okay to do this?
I can't believe that I'm asking an obvious question, but I still get the wrong in console log.
Console shows crawl like "[]" in the site, but I've checked at least 10 times for typos. Anyways, here's the javascript code.
I want to crawl in the site.
This is the kangnam.js file :
const axios = require('axios');
const cheerio = require('cheerio');
const log = console.log;
const getHTML = async () => {
try {
return await axios.get('https://web.kangnam.ac.kr', {
headers: {
Accept: 'text/html'
}
});
} catch (error) {
console.log(error);
}
};
getHTML()
.then(html => {
let ulList = [];
const $ = cheerio.load(html.data);
const $allNotices = $("ul.tab_listl div.list_txt");
$allNotices.each(function(idx, element) {
ulList[idx] = {
title : $(this).find("list_txt title").text(),
url : $(this).find("list_txt a").attr('href')
};
});
const data = ulList.filter(n => n.title);
return data;
}). then(res => log(res));
I've checked and revised at least 10 times
Yet, Js still throws this result :
root#goorm:/workspace/web_platform_test/myapp/kangnamCrawling(master)# node kangnam.js
[]
Mate, I think the issue is you're parsing it incorrectly.
$allNotices.each(function(idx, element) {
ulList[idx] = {
title : $(this).find("list_txt title").text(),
url : $(this).find("list_txt a").attr('href')
};
});
The data that you're trying to parse for is located within the first index of the $(this) array, which is really just storing a DOM Node. As to why the DOM stores Nodes this way, it's most likely due to efficiency and effectiveness. But all the data that you're looking for is contained within this Node object. However, the find() is superficial and only checks the indexes of an array for the conditions you supplied, which is a string search. The $(this) array only contains a Node, not a string, so when you you call .find() for a string, it will always return undefined.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/find
You need to first access the initial index and do property accessors on the Node. You also don't need to use $(this) since you're already given the same exact data with the element parameter. It's also more efficient to just use element since you've already been given the data you need to work with.
$allNotices.each(function(idx, element) {
ulList[idx] = {
title : element.children[0].attribs.title,
url : element.children[0].attribs.href
};
});
This should now populate your data array correctly. You should always analyze the data structures you're parsing for since that's the only way you can correctly parse them.
Anyways, I hope I solved your problem!
I'm using React to build a web app. At one point I have a list of ids, and I want to use those to retrieve a list of items from a database, get a list of metrics from each one, and then push those metrics to an array. My code so far is:
useEffect(() => {
const newMetrics = [];
currentItems.forEach((item) => {
const url = `items/listmetrics/${item.id}`;
Client.getData(url).then(async (metrics) => {
let promises = metrics.map((metricId: string) => {
// Get metric info
const urlMetric = `metrics/${metricId}`;
return Client.getData(urlMetric);
});
await Promise.all(promises).then((metrics: Array<any>) => {
metrics.forEach((metric: MetricModel) => {
const metricItem = {
id: metric.id,
metricName: metric.name
};
newMetrics.push(metricItem);
}
});
});
});
setMetrics(newMetrics);
});
}, [currentItems]);
where "metrics" is a state variable, set by setMetrics.
This appears to work ok, but when I try to access the resulting metrics array, it seems to be in the wrong format. If I try to read the value of metrics[0], it says it's undefined (although I know there are several items in metrics). Looking at it in the console, metrics looks like this:
However, normally the console shows arrays like this (this is a different variable, I'm just showing how it's listed with (2) [{...},{...}], whereas the one I've created shows as []):
I'm not confident with using Promise.all, so I suspect that that's where I've gone wrong, but I don't know how to fix it.
Background: Been trying for the last 2 day to resolve this myself by looking at various examples from both this website and others and I'm still not getting it. Whenever I try adding callbacks or async/await I'm getting no where. I know this is where my problem is but I can't resolve it myself.
I'm not from a programming background :( Im sure its a quick fix for the average programmer, I am well below that level.
When I console.log(final) within the 'ready' block it works as it should, when I escape that block the output is 'undefined' if console.log(final) -or- Get req/server info, if I use console.log(ready)
const request = require('request');
const ready =
// I know 'request' is deprecated, but given my struggle with async/await (+ callbacks) in general, when I tried switching to axios I found it more confusing.
request({url: 'https://www.website.com', json: true}, function(err, res, returnedData) {
if (err) {
throw err;
}
var filter = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str.split(",").map(function(a) { return `"trades.` + a + `.raw", `; }).join("");
var neater = addToStr.substr(0, addToStr.length-2);
var final = "[" + neater + "]";
// * * * Below works here but not outside this block* * *
// console.log(final);
});
// console.log(final);
// returns 'final is not defined'
console.log(ready);
// returns server info of GET req endpoint. This is as it is returning before actually returning the data. Not done as async.
module.exports = ready;
Below is an short example of the JSON that is returned by website.com. The actual call has 200+ 'result' objects.
What Im ultimately trying to achieve is
1) return all values of "instrument_name"
2) perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array.
["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
4) export/send this array to another file.
5) The array will be used as part of another request used in a websocket connection. The array cannot be hardcoded into this new request as the values of the array change daily.
{
"jsonrpc": "2.0",
"result": [
{
"kind": "option",
"is_active": true,
"instrument_name": "26JUN20-8000-C",
"expiration_timestamp": 1593158400000,
"creation_timestamp": 1575305837000,
"contract_size": 1,
},
{
"kind": "option",
"is_active": true,
"instrument_name": "25SEP20-8000-C",
"expiration_timestamp": 1601020800000,
"creation_timestamp": 1569484801000,
"contract_size": 1,
}
],
"usIn": 1591185090022084,
"usOut": 1591185090025382,
"usDiff": 3298,
"testnet": true
}
Looking your code we find two problems related to final and ready variables. The first one is that you're trying to console.log(final) out of its scope.
The second problem is that request doesn't immediately return the result of your API request. The reason is pretty simple, you're doing an asynchronous operation, and the result will only be returned by your callback. Your ready variable is just the reference to your request object.
I'm not sure about what is the context of your code and why you want to module.exports ready variable, but I suppose you want to export the result. If that's the case, I suggest you to return an async function which returns the response data instead of your request variable. This way you can control how to handle your response outside the module.
You can use the integrated fetch api instead of the deprecated request. I changed your code so that your component exports an asynchronous function called fetchData, which you can import somewhere and execute. It will return the result, updated with your logic:
module.exports = {
fetchData: async function fetchData() {
try {
const returnedData = await fetch({
url: "https://www.website.com/",
json: true
});
var ready = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str
.split(",")
.map(function(a) {
return `"trades.` + a + `.raw", `;
})
.join("");
var neater = addToStr.substr(0, addToStr.length - 2);
return "[" + neater + "]";
} catch (error) {
console.error(error);
}
}
}
I hope this helps, otherwise please share more of your code. Much depends on where you want to display the fetched data. Also, how you take care of the loading and error states.
EDIT:
I can't get responses from this website, because you need an account as well as credentials for the api. Judging your code and your questions:
1) return all values of "instrument_name"
Your map function works:
var filter = returnedData.result.map(entry => entry.instrument_name);
2)perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array. ["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
This can be done using this function
const manipulatedData = filter.map(val => `trades.${val}.raw`);
You can now use manipulatedData in your next request. Being able to export this variable, depends on the component you use it in. To be honest, it sounds easier to me not to split this logic into two separate components - regarding the websocket -.