NodeJS - How to get response of URL? - javascript

I have an url adress. When I go that adress in browser it shows a json file. I need to get that json file in NodeJs.
How can I do that?
Edit: This is my current code:
const got = require('got');
var url = "https://www.instagram.com/p/CFcyO54Hc7k/?__a=1";
got(url).json().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
And This is the result I get:
Promise { <pending> }
Hint: hit control+c anytime to enter REPL.
RequestError: Unexpected token < in JSON at position 0 in "https://www.instagram.com/accounts/login/?next=/p/CFcyO54Hc7k/%3F__a%3D1"
at Object.parseBody [as default] (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/parse-body.js:22:15)
at /home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/index.js:157:40
at JSON.parse (<anonymous>)
at parseJson (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/index.js:118:35)
at Object.parseBody [as default] (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/parse-body.js:11:48)
at /home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/index.js:157:40
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
name: 'ParseError',
code: undefined,
timings: {
start: 1601274389096,
socket: 1601274389096,
lookup: 1601274389106,
connect: 1601274389117,
secureConnect: 1601274389128,
upload: 1601274389128,
response: 1601274389467,
end: 1601274389511,
error: undefined,
abort: undefined,
phases: {
wait: 0,
dns: 10,
tcp: 11,
tls: 11,
request: 0,
firstByte: 339,
download: 44,
total: 415
}
}
}
My node version is 12.16.1

If you want to get it as json you can use the fetch api (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) then in the promises you can read the json as an object in your js code.
fetch('http://server/file.json').then(resp=>resp.json()).then(data=>{do something with data});

You can use axios in NodeJS.
axios.get('https://someapiendpoint.domain.com/v1/api')
.then((response) => {
console.log(response.data);
});

You can either parse the JSON yourself or you can get a library for making an Http request that parses the JSON automatically for you. There are lots of libraries to choose from here. My favorite is got() and it will let you parse the JSON with a simple .json() added to it so the resolved value you get back is a Javascript object.
const got = require('got');
got("https://www.instagram.com/p/CFcyO54Hc7k/?__a=1").json().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
When I run the above code with your instagram URL, I get this Javascript object back:
{
graphql: {
shortcode_media: {
__typename: 'GraphSidecar',
id: '2404017227651993316',
shortcode: 'CFcyO54Hc7k',
dimensions: [Object],
gating_info: null,
fact_check_overall_rating: null,
fact_check_information: null,
sensitivity_friction_info: null,
media_overlay_info: null,
media_preview: null,
display_url: 'https://scontent-sjc3-1.cdninstagram.com/v/t51.2885-15/e35/119980944_890957481429986_8757958615208227911_n.jpg?_nc_ht=scontent-sjc3-1.cdninstagram.com&_nc_cat=102&_nc_ohc=w7zBW_FdRNgAX9BYknl&_nc_tp=18&oh=11424efc118b55180e4d3302d75181ca&oe=5F99EB23',
display_resources: [Array],
is_video: false,
tracking_token: 'eyJ2ZXJzaW9uIjo1LCJwYXlsb2FkIjp7ImlzX2FuYWx5dGljc190cmFja2VkIjp0cnVlLCJ1dWlkIjoiZGIyMzdkMzgzYWVhNDY5YmI0ZWVkZWI4YzlhZDY0YWUyNDA0MDE3MjI3NjUxOTkzMzE2In0sInNpZ25hdHVyZSI6IiJ9',
edge_media_to_tagged_user: [Object],
edge_media_to_caption: [Object],
caption_is_edited: false,
has_ranked_comments: false,
edge_media_to_parent_comment: [Object],
edge_media_to_hoisted_comment: [Object],
edge_media_preview_comment: [Object],
comments_disabled: false,
commenting_disabled_for_viewer: false,
taken_at_timestamp: 1600801207,
edge_media_preview_like: [Object],
edge_media_to_sponsor_user: [Object],
location: null,
viewer_has_liked: false,
viewer_has_saved: false,
viewer_has_saved_to_collection: false,
viewer_in_photo_of_you: false,
viewer_can_reshare: true,
owner: [Object],
is_ad: false,
edge_web_media_to_related_media: [Object],
edge_sidecar_to_children: [Object],
edge_related_profiles: [Object]
}
}
}
If you'd rather modify your existing code, then you will probably need to use JSON.parse(). You will have to add your existing code to your question for us to help with that.

Related

Firebase Firestore collection get method causing error

I have a collection called 'users' in Firestore, and it is structured like this
users
->
document
-> email, name
db.collection('users').get().then(snapshot => {
snapshot.docs.forEach(doc => {
console.log(doc)
});
});
This code is supposed to fetch all the documents in the collection (there are about 15), but when I run it, I get a long error (below)
QueryDocumentSnapshot {
_firestore: Firestore {
_delegate: Firestore$1 {
type: 'firestore',
_persistenceKey: '[DEFAULT]',
_settings: [FirestoreSettingsImpl],
_settingsFrozen: true,
_app: [FirebaseAppImpl],
_databaseId: [DatabaseId],
_credentials: [FirebaseCredentialsProvider],
_queue: [AsyncQueueImpl],
_firestoreClient: [FirestoreClient]
},
_persistenceProvider: IndexedDbPersistenceProvider {},
INTERNAL: { delete: [Function: delete] },
_appCompat: FirebaseAppImpl {
firebase_: [Object],
isDeleted_: false,
name_: '[DEFAULT]',
automaticDataCollectionEnabled_: false,
options_: [Object],
container: [ComponentContainer]
}
},
_delegate: QueryDocumentSnapshot$1 {
_firestore: Firestore$1 {
type: 'firestore',
_persistenceKey: '[DEFAULT]',
_settings: [FirestoreSettingsImpl],
_settingsFrozen: true,
_app: [FirebaseAppImpl],
_databaseId: [DatabaseId],
_credentials: [FirebaseCredentialsProvider],
_queue: [AsyncQueueImpl],
_firestoreClient: [FirestoreClient]
},
How should I fetch documents properly without this error?
Firebase version - 8.2.3
Thanks
This is not an error: With your code you are actually printing the QueryDocumentSnapshot Objects in your console.
You should call the data() method to get the documents data, as follows:
db.collection('users').get().then(snapshot => {
snapshot.docs.forEach(doc => {
console.log(doc.data()); // Or console.log(JSON.stringify(doc.data()));
});
});
Also note that a QuerySnapshot has a forEach() method and therefore you could do:
db.collection('users').get().then(snapshot => {
snapshot.forEach(doc => {
console.log(doc.data());
});
});

Invalid form body when attempting to push slash commands

I have looked far and wide for answers to my question and have yet to find anything that helped me, I try to push my slash commands globally and they do not work as in discord shoots back with this when I execute the code:
Started refreshing 13 application (/) commands.
DiscordAPIError[50035]: Invalid Form Body
0.name[BASE_TYPE_REQUIRED]: This field is required
1.name[BASE_TYPE_REQUIRED]: This field is required
2.name[BASE_TYPE_REQUIRED]: This field is required
3.name[BASE_TYPE_REQUIRED]: This field is required
4.name[BASE_TYPE_REQUIRED]: This field is required
5.name[BASE_TYPE_REQUIRED]: This field is required
6.name[BASE_TYPE_REQUIRED]: This field is required
7.name[BASE_TYPE_REQUIRED]: This field is required
8.name[BASE_TYPE_REQUIRED]: This field is required
9.name[BASE_TYPE_REQUIRED]: This field is required
10.name[BASE_TYPE_REQUIRED]: This field is required
11.name[BASE_TYPE_REQUIRED]: This field is required
12.name[BASE_TYPE_REQUIRED]: This field is required
at SequentialHandler.runRequest (/home/ayden/microwave.js/node_modules/#discordjs/rest/dist/index.js:753:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async SequentialHandler.queueRequest (/home/ayden/microwave.js/node_modules/#discordjs/rest/dist/index.js:565:14)
at async REST.request (/home/ayden/microwave.js/node_modules/#discordjs/rest/dist/index.js:999:22)
at async /home/ayden/microwave.js/deploy.js:25:16 {
rawError: {
code: 50035,
errors: {
'0': [Object],
'1': [Object],
'2': [Object],
'3': [Object],
'4': [Object],
'5': [Object],
'6': [Object],
'7': [Object],
'8': [Object],
'9': [Object],
'10': [Object],
'11': [Object],
'12': [Object]
},
message: 'Invalid Form Body'
},
code: 50035,
status: 400,
method: 'PUT',
url: 'https://discord.com/api/v10/applications/867964961417203743/commands',
requestBody: {
files: undefined,
json: [
undefined, undefined,
undefined, undefined,
undefined, undefined,
undefined, undefined,
undefined, undefined,
undefined, undefined,
undefined
]
}
}
Here is the code from the deploy.js file:
const { REST } = require('#discordjs/rest');
const { Routes } = require('discord-api-types/v10');
const { clientId, guildId, token } = require('./config.json');
const fs = require('node:fs');
const commands = [];
const commandFiles = fs.readdirSync('./commands').filter(file => file.endsWith('.ts'));
for (const file of commandFiles) {
const command = require(`./commands/${file}`);
commands.push(JSON.stringify(command.data));
}
const rest = new REST({ version: '10' }).setToken(token);
(async () => {
try {
console.log(`Started refreshing ${commands.length} application (/) commands.`);
const data = await rest.put(
Routes.applicationCommands(clientId),
{ body: commands },
);
console.log(`Successfully reloaded ${data.length} application (/) commands.`);
} catch (error) {
console.error(error);
}
})();
If you could help me solve this that would be much appreciated!
The problem is that you are trying to push command.data to the commands array.
Looking at one of your command files, I noticed that that property doesn’t exist. Instead, the Slash Command Builder is located at <command>.help.data (e. g. in your about.js, you are using:
module.exports.help = {
name: "about",
cat: "Info",
description: "Get information about the bot",
aliases: "",
data: new SlashCommandBuilder().setName("about").setDescription("Get information about the bot"),
}
This module.exports points to the export property .help and has the property .data which contains the SlashCommandBuilder.
To solve this issue, just change the deploy.js line 11 to the following:
commands.push(JSON.stringify(command.help.data));
This points to the correct location of the SlashCommandBuilder - preventing the array to just be filled with undefined because the property given doesn’t exist.
I hope this helps in resolving the problem.

How to prevent ava displaying long objects when a test fails

Given: Express app, AVA, supertest
When: I test generated html in response and the test case fails
Then: AVA displays the whole response object in the console which slows down analysis of the issue
Example of the test:
test('Positive case: book is found in /api/v1/books/1', async (t) => {
t.plan(2)
const response = await request(app).get('/api/v1/books/1')
t.is(response.status, 200)
const expectedBook = '<h3>Book 1</h3>'
t.truthy(response.res.text.match(expectedBook), 'Book title is not found')
})
Example of the output in the console
/express-project/src/books/index.test.js:22
21: const text = response.res.text
22: t.truthy(response.res.text.match(expectedBook), 'Book t…
23: })
Book title is not found
Value is not truthy:
null
response.res.text.match(expectedBook)
=> null
expectedBook
=> '<h3>Book 2</h3>'
response.res.text
=> '<!DOCTYPE html><html><head><title>BOOK</title><link rel="stylesheet" href="/stylesheets/style.css"></head><body><h1>BOOK</h1>
<h3>Book 1</h3><h4></h4></body></html>'
response.res
=> IncomingMessage {
_consuming: false,
_dumped: false,
_events: {
close: Function bound emit {},
data: [
Function {},
Function {},
Function bound emit {},
],
end: [
Function responseOnEnd {},
Function {},
Function bound emit {},
],
error: [
Function bound onceWrapper { … },
Function bound emit {},
],
},
_eventsCount: 4,
_maxListeners: undefined,
_readableState: ReadableState [
awaitDrain: 0,
.......... VERY LONG LIST WITH HUNDREDS OF LINES
SO HAVE TO SCROLL UP AND UP AND UP BEFORE YOU GET TO THE POINT
Ava is trying to be helpful in debugging the failed test so Ava puts in the console respectively
response.res.text
response.res
response
and it generates hundreds of even thousands of lines in the console
So the solution is pretty simple - use an intermediate variable for the assertion
Instead of
t.truthy(response.res.text.match(expectedBook), 'Book title is not found')
use
const text = response.res.text
t.truthy(text.match(expectedBook), 'Book title is not found')

Selenium NodeJS print cookies to console

I'm using the selenium-webdriver npm module for Node.JS, however I am having trouble writing the cookies to the console. I used the example code from the NPM page (Under usage)
var webdriver = require('selenium-webdriver'),
By = webdriver.By,
until = webdriver.until;
var driver = new webdriver.Builder()
.forBrowser('firefox')
.build();
driver.get('http://www.google.com/ncr');
driver.findElement(By.name('q')).sendKeys('webdriver');
driver.findElement(By.name('btnG')).click();
driver.wait(until.titleIs('webdriver - Google Search'), 1000);
console.log(driver.manage().getCookies());
driver.quit();
Now, I would expect console.log to write the dictionary I've seen referenced in other questions, however I get the following output:
ManagedPromise {
flow_:
ControlFlow {
propagateUnhandledRejections_: true,
activeQueue_:
TaskQueue {
name_: 'TaskQueue::5',
flow_: [Circular],
tasks_: [Object],
interrupts_: null,
pending_: null,
state_: 'new',
unhandledRejections_: Set {} },
taskQueues_: Set { [Object] },
shutdownTask_: null,
hold_:
Timeout {
_called: false,
_idleTimeout: 2147483647,
_idlePrev: [Object],
_idleNext: [Object],
_idleStart: 231,
_onTimeout: [Function: wrapper],
_repeat: [Function] } },
stack_: { [Task: WebDriver.manage().getCookies()] name: 'Task' },
parent_: null,
callbacks_: null,
state_: 'pending',
handled_: false,
value_: undefined,
queue_: null }
I get no errors in my console, but I'm not getting the cookies I expected either. I'm using the latest version of node, v5.9.1, and the latest version of selenium-webdriver. For some reason the console.log code is called before selenium's instance of Firefox even starts. How would I go about fixing this?
You can use .then(). Refer to When should we use .then with Protractor Promise? for more information.
driver.manage().getCookies().then(function (cookies) {
console.log(cookies);
});

node.js: Mongodb db.collection.find() not working while collection.insert works

I'm using node.js/express and I have a Mongodb to store some sets of data. On a webpage the user can enter, edit and delete data (which all works fine). For example, to add data I have the following code:
router.post('/addset', function(req,res) {
var db = req.db;
var collection = db.get('paramlist');
collection.insert(req.body, function(err, result){
res.send(
(err === null) ? { msg: '' } : { msg: err }
);
});
});
In my app.js file I include the lines
// Database
var mongo = require('mongodb');
var monk = require('monk');
var db = monk('localhost:27017/paramSet1');
as well as
app.use(function(req,res,next){
req.db = db;
next();
});
to make the database accessible in the rest of the code (following this tutorial: http://cwbuecheler.com/web/tutorials/2013/node-express-mongo/ , I'm a beginner with these things).
So all this works fine. My problem is the following: I would like to add a test if a dataset with the same name is already in the database and give a message to the user. Following this answer How to query MongoDB to test if an item exists? I tried using collection.find.limit(1).size() but I get the error
undefined is not a function
I tried the following things. In the cost above (router.post) I tried adding after the line var collection...
var findValue = collection.find({name:req.body.name});
If i then do console.log(findValue), I get a huge output JSON. I then tried console.log(findValue.next()) but I get the same error (undefined is not a function). I also tried
collection.find({name:req.body.name}).limit(1)
as well as
collection.find({name:req.body.name}).limit(1).size()
but also get this error. So in summary, collection.insert, collection.update and collection.remove all work, but find() does not. On the other hand, when I enter the mongo shell, the command works fine.
I would be grateful for any hints and ideas.
Edit:
The output to console.log(findValue) is:
{ col:
{ manager:
{ driver: [Object],
helper: [Object],
collections: [Object],
options: [Object],
_events: {} },
driver:
{ _construct_args: [],
_native: [Object],
_emitter: [Object],
_state: 2,
_connect_args: [Object] },
helper: { toObjectID: [Function], isObjectID: [Function], id: [Object] },
name: 'paramlist',
col:
{ _construct_args: [],
_native: [Object],
_emitter: [Object],
_state: 2,
_skin_db: [Object],
_collection_args: [Object],
id: [Object],
emitter: [Object] },
options: {} },
type: 'find',
opts: { fields: {}, safe: true },
domain: null,
_events: { error: [Function], success: [Function] },
_maxListeners: undefined,
emitted: {},
ended: false,
success: [Function],
error: [Function],
complete: [Function],
resolve: [Function],
fulfill: [Function],
reject: [Function],
query: { name: 'TestSet1' } }
find returns a cursor, not the matching documents themselves. But a better fit for your case would be to use findOne:
collection.findOne({name:req.body.name}, function(err, doc) {
if (doc) {
// A doc with the same name already exists
}
});
If you're using the method on that website http://cwbuecheler.com/web/tutorials/2013/node-express-mongo/
you have to change your code
collection.find({name:req.body.name}).limit(1)
and use it like this
collection.find({name:req.body.name},{limit:1})
and if you want to add more options
do like this
collection.find({name:req.body.name},{limit:1,project:{a:1},skip:1,max:10,maxScan:10,maxTimeMS:1000,min:100})
You can find everything here http://mongodb.github.io/node-mongodb-native/2.0/api/Cursor.html
you can use like that
app.get('/', (req, res) => {
db.collection('quotes').find().toArray((err, result) => {
console.log(result);
})
})
The first thing that looks wrong is your syntax is incorrect for find. You need to call it as a function. Try:
collection.find({name:req.body.name}).limit(1).size();
Probably you might have missed to connect a Database.
Try adding below code before executing
mongodb.connect(dbURI).then((result)=>console.log('connected to DB')).catch((err)=>console.log('connected to DB'));

Categories