We Keep Coding

Is there was way to limit the number of queries from youtube api?

I am creating a tampermonkey script that will read a line of text that contains an artist and song title from a webpage and will create a link that will open a new tab to the first result that is returned from the youtube api. My issue is that when the page is loaded and say there are 5 songs on this particular page, according to my youtube api dashboard, I am making about 1200 api requests for that single page. Im guessing that its because when I query a single artist/song, its getting every result on its server. I tried limiting my maxResults to 1, but this doesn't help. Since youtube decreased the number of queries a single API gets, I would like to know if there is a way to reduce the number of queries it makes. Realistically, I just need the first result from the GET request, since chances are that will be the correct video.
here is a snippet of my code that parses through the json data:
function getLink(artist, song){
// API Key
var key = "MY_KEY";
// Setup url for api
var url = 'https://www.googleapis.com/youtube/v3/search?part=id&q=' + artist + "-" + song + '&maxResults=1&key=' + key;
// call api and get videoId
var xhReq = new XMLHttpRequest();
xhReq.open("GET", url, false);
xhReq.send(null);
var id = JSON.parse(xhReq.responseText);
return id.items[0].id.videoId;
}

I'm not sure if I'm looking at the same Google doc, but it says the part parameter should be snippet for a Search:list call. I know different calls take up different quota amounts. Using the query below I get only one song returned. It seems to use only the normal 100 units of quota required for Search:list:
https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=1&q=Adele+Hello&fields=items(id(videoId))&key=API_key
It returns:
{
"items": [
{
"id": {
"videoId": "YQHsXMglC9A"
}
}
]
}

Be sure to escape artist and title with something like encodeURIComponent. It could be that the strings in those variables are breaking the intended URL you want to process.
Even more, if you do a test with the Youtube API sandbox, you could check it by yourself. In my case I've tried "Radiohead" and "Paranoid Android" with 3 max results and it's working fine. See here

JavaScript: Import video from youtube channels as API json to my website [duplicate]

We need a video list by channel name of YouTube (using the API).
We can get a channel list (only channel name) by using the below API:
https://gdata.youtube.com/feeds/api/channels?v=2&q=tendulkar
Below is a direct link of channels
https://www.youtube.com/channel/UCqAEtEr0A0Eo2IVcuWBfB9g
Or
WWW.YouTube.com/channel/HC-8jgBP-4rlI
Now, we need videos of channel >> UCqAEtEr0A0Eo2IVcuWBfB9g or HC-8jgBP-4rlI.
We tried
https://gdata.youtube.com/feeds/api/videos?v=2&uploader=partner&User=UC7Xayrf2k0NZiz3S04WuDNQ
https://gdata.youtube.com/feeds/api/videos?v=2&uploader=partner&q=UC7Xayrf2k0NZiz3S04WuDNQ
But, it does not help.
We need all the videos posted on the channel. Videos uploaded to a channel can be from multiple users thus I don't think providing a user parameter would help...

You need to look at the YouTube Data API. You will find there documentation about how the API can be accessed. You can also find client libraries.
You could also make the requests yourself. Here is an example URL that retrieves the latest videos from a channel:
https://www.googleapis.com/youtube/v3/search?key={your_key_here}&channelId={channel_id_here}&part=snippet,id&order=date&maxResults=20
After that you will receive a JSON with video ids and details, and you can construct your video URL like this:
http://www.youtube.com/watch?v={video_id_here}

First, you need to get the ID of the playlist that represents the uploads from the user/channel:
https://developers.google.com/youtube/v3/docs/channels/list#try-it
You can specify the username with the forUsername={username} param, or specify mine=true to get your own (you need to authenticate first). Include part=contentDetails to see the playlists.
GET https://www.googleapis.com/youtube/v3/channels?part=contentDetails&forUsername=jambrose42&key={YOUR_API_KEY}
In the result "relatedPlaylists" will include "likes" and "uploads" playlists. Grab that "upload" playlist ID.
Also note the upload playlist id is your channelId prefixed with UU instead of UC.
Next, get a list of videos in that playlist:
https://developers.google.com/youtube/v3/docs/playlistItems/list#try-it
Just drop in the playlistId!
GET https://www.googleapis.com/youtube/v3/playlistItems?part=snippet%2CcontentDetails&maxResults=50&playlistId=UUpRmvjdu3ixew5ahydZ67uA&key={YOUR_API_KEY}

Here is a video from Google Developers showing how to list all videos in a channel in v3 of the YouTube API.
There are two steps:
Query Channels to get the "uploads" Id. eg https://www.googleapis.com/youtube/v3/channels?id={channel Id}&key={API key}&part=contentDetails
Use this "uploads" Id to query PlaylistItems to get the list of videos. eg https://www.googleapis.com/youtube/v3/playlistItems?playlistId={"uploads" Id}&key={API key}&part=snippet&maxResults=50

To get channels list :
Get Channels list by forUserName:
https://www.googleapis.com/youtube/v3/channels?part=snippet,contentDetails,statistics&forUsername=Apple&key=
Get channels list by channel id:
https://www.googleapis.com/youtube/v3/channels/?part=snippet,contentDetails,statistics&id=UCE_M8A5yxnLfW0KghEeajjw&key=
Get Channel sections:
https://www.googleapis.com/youtube/v3/channelSections?part=snippet,contentDetails&channelId=UCE_M8A5yxnLfW0KghEeajjw&key=
To get Playlists :
Get Playlists by Channel ID:
https://www.googleapis.com/youtube/v3/playlists?part=snippet,contentDetails&channelId=UCq-Fj5jknLsUf-MWSy4_brA&maxResults=50&key=
Get Playlists by Channel ID with pageToken:
https://www.googleapis.com/youtube/v3/playlists?part=snippet,contentDetails&channelId=UCq-Fj5jknLsUf-MWSy4_brA&maxResults=50&key=&pageToken=CDIQAA
To get PlaylistItems :
Get PlaylistItems list by PlayListId:
https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,contentDetails&maxResults=25&playlistId=PLHFlHpPjgk70Yv3kxQvkDEO5n5tMQia5I&key=
To get videos :
Get videos list by video id:
https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=YxLCwfA1cLw&key=
Get videos list by multiple videos id:
https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=YxLCwfA1cLw,Qgy6LaO3SB0,7yPJXGO2Dcw&key=
Get comments list
Get Comment list by video ID:
https://www.googleapis.com/youtube/v3/commentThreads?part=snippet,replies&videoId=el****kQak&key=A**********k
Get Comment list by channel ID:
https://www.googleapis.com/youtube/v3/commentThreads?part=snippet,replies&channelId=U*****Q&key=AI********k
Get Comment list by allThreadsRelatedToChannelId:
https://www.googleapis.com/youtube/v3/commentThreads?part=snippet,replies&allThreadsRelatedToChannelId=UC*****ntcQ&key=AI*****k
Here all api's are Get approach.
Based on channel id we con't get all videos directly, that's the important point here.
For integration https://developers.google.com/youtube/v3/quickstart/ios?ver=swift

Here is the code that will return all video ids under your channel
<?php
$baseUrl = 'https://www.googleapis.com/youtube/v3/';
// https://developers.google.com/youtube/v3/getting-started
$apiKey = 'API_KEY';
// If you don't know the channel ID see below
$channelId = 'CHANNEL_ID';
$params = [
'id'=> $channelId,
'part'=> 'contentDetails',
'key'=> $apiKey
];
$url = $baseUrl . 'channels?' . http_build_query($params);
$json = json_decode(file_get_contents($url), true);
$playlist = $json['items'][0]['contentDetails']['relatedPlaylists']['uploads'];
$params = [
'part'=> 'snippet',
'playlistId' => $playlist,
'maxResults'=> '50',
'key'=> $apiKey
];
$url = $baseUrl . 'playlistItems?' . http_build_query($params);
$json = json_decode(file_get_contents($url), true);
$videos = [];
foreach($json['items'] as $video)
$videos[] = $video['snippet']['resourceId']['videoId'];
while(isset($json['nextPageToken'])){
$nextUrl = $url . '&pageToken=' . $json['nextPageToken'];
$json = json_decode(file_get_contents($nextUrl), true);
foreach($json['items'] as $video)
$videos[] = $video['snippet']['resourceId']['videoId'];
}
print_r($videos);
Note: You can get channel id at
https://www.youtube.com/account_advanced after logged in.

Below is a Python alternative that does not require any special packages. By providing the channel id it returns a list of video links for that channel. Please note that you need an API Key for it to work.
import urllib
import json
def get_all_video_in_channel(channel_id):
api_key = YOUR API KEY
base_video_url = 'https://www.youtube.com/watch?v='
base_search_url = 'https://www.googleapis.com/youtube/v3/search?'
first_url = base_search_url+'key={}&channelId={}&part=snippet,id&order=date&maxResults=25'.format(api_key, channel_id)
video_links = []
url = first_url
while True:
inp = urllib.urlopen(url)
resp = json.load(inp)
for i in resp['items']:
if i['id']['kind'] == "youtube#video":
video_links.append(base_video_url + i['id']['videoId'])
try:
next_page_token = resp['nextPageToken']
url = first_url + '&pageToken={}'.format(next_page_token)
except:
break
return video_links

Thanks to the references shared here and elsewhere, I've made an online script / tool that one can use to obtain all videos of a channel.
It combines API calls to youtube.channels.list, playlistItems, videos. It uses recursive functions to make the asynchronous callbacks run the next iteration upon getting a valid response.
This also serves to limit the actual number of requests made at a time, hence keeping you safe from violating YouTube API rules. Sharing shortened snippets and then a link to the full code. I got around the 50 max results per call limitation by using the nextPageToken value that comes in the response to fetch the next 50 results and so on.
function getVideos(nextPageToken, vidsDone, params) {
$.getJSON("https://www.googleapis.com/youtube/v3/playlistItems", {
key: params.accessKey,
part: "snippet",
maxResults: 50,
playlistId: params.playlistId,
fields: "items(snippet(publishedAt, resourceId/videoId, title)), nextPageToken",
pageToken: ( nextPageToken || '')
},
function(data) {
// commands to process JSON variable, extract the 50 videos info
if ( vidsDone < params.vidslimit) {
// Recursive: the function is calling itself if
// all videos haven't been loaded yet
getVideos( data.nextPageToken, vidsDone, params);
}
else {
// Closing actions to do once we have listed the videos needed.
}
});
}
This got a basic listing of the videos, including id, title, date of publishing and similar. But to get more detail of each video like view counts and likes, one has to make API calls to videos.
// Looping through an array of video id's
function fetchViddetails(i) {
$.getJSON("https://www.googleapis.com/youtube/v3/videos", {
key: document.getElementById("accesskey").value,
part: "snippet,statistics",
id: vidsList[i]
}, function(data) {
// Commands to process JSON variable, extract the video
// information and push it to a global array
if (i < vidsList.length - 1) {
fetchViddetails(i+1) // Recursive: calls itself if the
// list isn't over.
}
});
See the full code here, and live version here. (Edit: fixed github link)
Edit: Dependencies: JQuery, Papa.parse

Short answer:
Here's a library called scrapetube That can help with that.
pip install scrapetube
import scrapetube
import simplejson as json
videos = scrapetube.get_channel("UC9-y-6csu5WGm29I7JiwpnA")
for video in videos:
print(video['videoId'])
print(video['title']['runs'][0]['text'])
print(video['publishedTimeText']['simpleText'])
print('\r\n')
# DEBUG: print(json.dumps(video))
Long answer:
The module mentioned above was created by me due to a lack of any other solutions. Here's what i tried:
Selenium. It worked but had three big drawbacks: 1. It requires a web browser and driver to be installed. 2. has big CPU and memory requirements. 3. can't handle big channels.
Using youtube-dl. Like this:
import youtube_dl
youtube_dl_options = {
'skip_download': True,
'ignoreerrors': True
}
with youtube_dl.YoutubeDL(youtube_dl_options) as ydl:
videos = ydl.extract_info(f'https://www.youtube.com/channel/{channel_id}/videos')
This also works for small channels, but for bigger ones i would get blocked by youtube for making so many requests in such a short time (because youtube-dl downloads more info for every video in the channel).
So i made the library scrapetube which uses the web API to get all the videos.

Try with like the following. It may help you.
https://gdata.youtube.com/feeds/api/videos?author=cnn&v=2&orderby=updated&alt=jsonc&q=news
Here author as you can specify your channel name and "q" as you can give your search key word.

Since everyone answering this question has problems due to the 500 video limit here's an alternate solution using youtube_dl in Python 3. Also, no API key is needed.
Install youtube_dl: sudo pip3 install youtube-dl
Find out your target channel's channel id. The ID is going to start with UC. Replace the C for Channel with U for Upload (i.e. UU...), this is the upload playlist.
Use the playlist downloader feature from youtube-dl. Ideally you do NOT want to download every video in the playlist which is the default, but only the metadata.
Example (warning -- takes tens of minutes):
import youtube_dl, pickle
# UCVTyTA7-g9nopHeHbeuvpRA is the channel id (1517+ videos)
PLAYLIST_ID = 'UUVTyTA7-g9nopHeHbeuvpRA' # Late Night with Seth Meyers
with youtube_dl.YoutubeDL({'ignoreerrors': True}) as ydl:
playd = ydl.extract_info(PLAYLIST_ID, download=False)
with open('playlist.pickle', 'wb') as f:
pickle.dump(playd, f, pickle.HIGHEST_PROTOCOL)
vids = [vid for vid in playd['entries'] if 'A Closer Look' in vid['title']]
print(sum('Trump' in vid['title'] for vid in vids), '/', len(vids))

Just in three steps:
Subscriptions: list ->
https://www.googleapis.com/youtube/v3/subscriptions?part=snippet&maxResults=50&mine=true&access_token={oauth_token}
Channels: list ->
https://www.googleapis.com/youtube/v3/channels?part=contentDetails&id={channel_id}&key={YOUR_API_KEY}
PlaylistItems: list ->
https://www.googleapis.com/youtube/v3/playlistItems?part=snippet&playlistId={playlist_id}&key={YOUR_API_KEY}

Recently I had to retrieve all videos from a channel, and according to YouTube developer documentation:
https://developers.google.com/youtube/v3/docs/playlistItems/list
function playlistItemsListByPlaylistId($service, $part, $params) {
$params = array_filter($params);
$response = $service->playlistItems->listPlaylistItems(
$part,
$params
);
print_r($response);
}
playlistItemsListByPlaylistId($service,
'snippet,contentDetails',
array('maxResults' => 25, 'playlistId' => 'id of "uploads" playlist'));
Where $service is your Google_Service_YouTube object.
So you have to fetch information from the channel to retrieve the "uploads" playlist that actually has all the videos uploaded by the channel: https://developers.google.com/youtube/v3/docs/channels/list
If new with this API, I highly recommend to turn the code sample from the default snippet to the full sample.
So the basic code to retrieve all videos from a channel can be:
class YouTube
{
const DEV_KEY = 'YOUR_DEVELOPPER_KEY';
private $client;
private $youtube;
private $lastChannel;
public function __construct()
{
$this->client = new Google_Client();
$this->client->setDeveloperKey(self::DEV_KEY);
$this->youtube = new Google_Service_YouTube($this->client);
$this->lastChannel = false;
}
public function getChannelInfoFromName($channel_name)
{
if ($this->lastChannel && $this->lastChannel['modelData']['items'][0]['snippet']['title'] == $channel_name)
{
return $this->lastChannel;
}
$this->lastChannel = $this->youtube->channels->listChannels('snippet, contentDetails, statistics', array(
'forUsername' => $channel_name,
));
return ($this->lastChannel);
}
public function getVideosFromChannelName($channel_name, $max_result = 5)
{
$this->getChannelInfoFromName($channel_name);
$params = [
'playlistId' => $this->lastChannel['modelData']['items'][0]['contentDetails']['relatedPlaylists']['uploads'],
'maxResults'=> $max_result,
];
return ($this->youtube->playlistItems->listPlaylistItems('snippet,contentDetails', $params));
}
}
$yt = new YouTube();
echo '<pre>' . print_r($yt->getVideosFromChannelName('CHANNEL_NAME'), true) . '</pre>';

Using API version 2, which is deprecated, the URL for uploads (of channel UCqAEtEr0A0Eo2IVcuWBfB9g) is:
https://gdata.youtube.com/feeds/users/UCqAEtEr0A0Eo2IVcuWBfB9g/uploads
There is an API version 3.

From https://stackoverflow.com/a/65440501/2585501:
This method is especially useful if a) the channel has more than 50 videos or if b) desire youtube video ids formatted in a flat txt list:
Obtain a Youtube API v3 key (see https://stackoverflow.com/a/65440324/2585501)
Obtain the Youtube Channel ID of the channel (see https://stackoverflow.com/a/16326307/2585501)
Obtain the Uploads Playlist ID of the channel: https://www.googleapis.com/youtube/v3/channels?id={channel Id}&key={API key}&part=contentDetails (based on https://www.youtube.com/watch?v=RjUlmco7v2M)
Install youtube-dl (e.g. pip3 install --upgrade youtube-dl or sudo apt-get install youtube-dl)
Download the Uploads Playlist using youtube-dl: youtube-dl -j --flat-playlist "https://<yourYoutubePlaylist>" | jq -r '.id' | sed 's_^_https://youtu.be/_' > videoList.txt (see https://superuser.com/questions/1341684/youtube-dl-how-download-only-the-playlist-not-the-files-therein)

Posting long after the original question was asked, but I made a python package that does this using a very simple API. It gets all the videos uploaded to a channel, but I'm not sure about this part (included in the original question):
Videos uploaded to a channel can be from multiple users thus I don't think providing a user parameter would help...
Maybe YouTube changed in the 8 years since this question was posted, but if it didn't, the package I made might not cover this case.
To use the API:
pip3 install -U yt-videos-list # macOS
pip install -U yt-videos-list # Windows
# if that doesn't work, try
python3 -m pip install -U yt-videos-list # macOS
python -m pip install -U yt-videos-list # Windows
Then open up a python interpreter
python3 # macOS
python # Windows
and run the program:
from yt_videos_list import ListCreator
lc = ListCreator()
help(lc) # display API information - shows available parameters and functions
my_url = 'https://www.youtube.com/user/1veritasium'
lc.create_list_for(url=my_url)
Python documentation (will be updated most frequently, so check this page for updates!)
Repository homepage
PyPI page

Sample solution in Python. Help taken from this video: video
Like many other answers, upload id is to be retrieved from the channel id first.
import urllib.request
import json
key = "YOUR_YOUTUBE_API_v3_BROWSER_KEY"
#List of channels : mention if you are pasting channel id or username - "id" or "forUsername"
ytids = [["bbcnews","forUsername"],["UCjq4pjKj9X4W9i7UnYShpVg","id"]]
newstitles = []
for ytid,ytparam in ytids:
urld = "https://www.googleapis.com/youtube/v3/channels?part=contentDetails&"+ytparam+"="+ytid+"&key="+key
with urllib.request.urlopen(urld) as url:
datad = json.loads(url.read())
uploadsdet = datad['items']
#get upload id from channel id
uploadid = uploadsdet[0]['contentDetails']['relatedPlaylists']['uploads']
#retrieve list
urld = "https://www.googleapis.com/youtube/v3/playlistItems?part=snippet%2CcontentDetails&maxResults=50&playlistId="+uploadid+"&key="+key
with urllib.request.urlopen(urld) as url:
datad = json.loads(url.read())
for data in datad['items']:
ntitle = data['snippet']['title']
nlink = data['contentDetails']['videoId']
newstitles.append([nlink,ntitle])
for link,title in newstitles:
print(link, title)

That's my Python solution, using Google API.
Observations:
Create a .env file to store your API Developer Key, and put it in your .gitignore file
The parameter "forUserName" should be set with the name of the Youtube Channel (username). Alternatively, you can use the channel id, setting the parameter "id", instead of "forUserName".
The object "playlistItem" gives you access to each video. I'm showing only its title but there are many other properties.
import os
import googleapiclient.discovery
from decouple import config
def main():
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtube"
api_version = "v3"
DEVELOPER_KEY = config('API_KEY')
youtube = googleapiclient.discovery.build(
api_service_name, api_version, developerKey = DEVELOPER_KEY)
request = youtube.channels().list(
part="contentDetails",
forUsername="username",
# id="oiwuereru8987",
)
response = request.execute()
for item in response['items']:
playlistId = item['contentDetails']['relatedPlaylists']['uploads']
nextPageToken = ''
while (nextPageToken != None):
playlistResponse = youtube.playlistItems().list(
part='snippet',
playlistId=playlistId,
maxResults=25,
pageToken=nextPageToken
)
playlistResponse = playlistResponse.execute()
print(playlistResponse.keys())
for idx, playlistItem in enumerate(playlistResponse['items']):
print(idx, playlistItem['snippet']['title'])
if 'nextPageToken' in playlistResponse.keys():
nextPageToken = playlistResponse['nextPageToken']
else:
nextPageToken = None
if __name__ == "__main__":
main()
Example for the .env file
API_KEY=<Key_Here>

Using the gapi JavaScript API, you can do this
<script src="https://apis.google.com/js/api.js"></script>
const start = () => {
gapi.client
.init({
apiKey: "your_youtubeApiKey",
discoveryDocs: ["https://www.googleapis.com/discovery/v1/apis/youtube/v3/rest"],
scope: "https://www.googleapis.com/auth/youtube.readonly",
})
.then(() => {
console.log("gapi.client initiated");
})
.then(() =>
gapi.client.youtube.channels.list({
part: "snippet,contentDetails,statistics",
id: "youtube_channelId",
// forUsername: 'Bankless',
})
)
.then(
(res) =>
// get the youtube related playlist id
res.result.items[0].contentDetails.relatedPlaylists.uploads
)
.then((playlistId) =>
gapi.client.youtube.playlistItems.list({
part: "snippet",
playlistId,
maxResults: 50,
})
)
.then((res) =>
// get youtube videos snippets
res.result.items.map((item) => item.snippet)
)
.then((snippets) =>
snippets.map((snippet) => {
const { title, description, resourceId } = snippet;
const { videoId } = resourceId;
return { title, description, videoId };
})
)
.then((videos) => {
console.log(videos);
})
.catch((err) => console.error(err));
};
gapi.load("client", start);
Docs:
https://github.com/google/google-api-javascript-client
https://developers.google.com/youtube/v3/guides/auth/client-side-web-apps#callinganapi

You have to get the channel_id of the video you want to get the data from.
For getting the channel_id using the video_id, you can use the videos:list endpoint of the YouTube Data API - add video_id in Id parameter. example.
Then, with the channel_id, change the second character to "U" :
This modified id is the Uploads playlist of that said YouTube channel.
With this Uploads playlist_id, you can use the Playlistitem:list endpoint of the YouTube Data API to retrieve all the uploaded videos from the channel.
In the part parameter add "id,snippet,contentDetails,status".
and in playlistID add the modified channel ID.
and then execute.

Node.Js async request inside a Array.forEach not completing before wiriting a json file

I'm making a web scraper Node.js app that harvests job description text from various urls.. I currently have an array of job objects named jobObj and the code cycles through each url, makes a request for html, loads using cheerio module then finally makes a new object with jobName and jobDesc keys and pushes it onto a new array of objects that is then written as a json file....
All this currently works however the completeness of the written json file is very random and usually only contains one complete job account. I thought this may be due to the forEach loop completing much quicker than the asynchronous Request function thus resulting in execution of the fs.writefile before request callback is completed. I've added a counter to monitor at what stage the requests are at and only write the json file once counter===jobObj.length but still the json file is not fully complete.
I'm new to node.js if someone could please point out my error it would be greatly appreciated!
var jobObj = [
{
id:1,
url:"https://www.indeed.co.uk/cmp/Daffodil-IT/jobs/Lead-Junior-Website-Developer-59ea7d446bdf1253?q=Junior+Web+Developer&vjs=3",
},
{
id:2,
url:"https://www.indeed.co.uk/cmp/Crush-Design/jobs/Middleweight-Web-Developer-541331b7885c03cf?q=Web+Developer&vjs=3",
},
{
id:3,
url:"https://www.indeed.co.uk/cmp/Monigold-Solutions/jobs/Graduate-Web-Software-Engineer-a5787dc322c0ca36?q=Web+Developer&vjs=3",
},
{
id:4,
url:"https://www.indeed.co.uk/cmp/ZOO-DIGITAL-GROUP-PLC/jobs/Web-Developer-5cdde1c3b0b7b8d0?q=Web+Developer&vjs=3",
},
{
id:5,
url:"https://www.indeed.co.uk/viewjob?jk=9cc3d8c637c41067&q=Web+Developer&l=Sheffield&tk=1cf5di52e9u0ocam&from=web&vjs=3",
}
];
app.get('/myform', function(req, res){
res.send("<h1>" + `scanning ${jobObj.length} urls for job description text` + "</h1>");
//make assign input form data to node "url" variable
//Compnonents for a request counter
var jobs = new Array;
function scrapeFinished(){console.log("all websites scraped!");};
var itemsProcessed = 0;
jobObj.forEach(function(item){
request(item.url, function(err, res, html){
if(!err){
var $ = cheerio.load(html);
var newJob = new Object;
$('#job_summary').each(function(){
var data = $(this);
var textout = data.text();
newJob.jobDesc = textout;
});
$('.jobtitle').each(function(){
var data = $(this);
var jobtitle = data.text();
newJob.jobName = jobtitle;
});
jobs.push(newJob);
itemsProcessed++;
console.log(item.url + " scraped");
if(itemsProcessed === jobObj.length){
scrapeFinished();
fs.writeFile('output.json', JSON.stringify(jobs, null, "\t"), function(err){
if(!err){console.log("output.json file written")}
})
}
}
})
})
})
And finally this is what I get on fs.writefile
[
{},
{
"jobDesc": "We are a successful design and digital agency that works with some great clients on a wide range of digital projects.We simply need more developers to join our great team to deliver even more great work.The projects we work on are all php based, typically built using WordPress, Laravel or flat html.We are seen as a premium agency because of the quality and complexity of the work we do.That means you will have to do more that just manipulate a theme - you will have to code. But you will be given the space, time and support to do so.We want you to be proud of the work you do, because the reputation of the agency need you to be.Key skills we will want you to bringCSS (CSS3) & HTMLAt least some knowledge of MySQL and JavaScript.At least some knowledge PHP (seniors will be tested)PhotoshopWhat you will want that we can giveA good place to work with a friendly teamA chance to develop your coding craftA decent range of projects to challenge yourselfA senior developer on hand to coach and adviseA successful company with an optimistic outlook, growth plans and a secure futureExactly how much experience you have can vary, but you must have some. And the more experience you have, the more we will pay you.We are based in offices we own in the centre of Chesterfield will two staff that do the short commute from Sheffield.If you think this job sounds interesting, we would love to hear from you, please apply!(though not agencies please)Job Type: Full-timeSalary: £22,000.00 to £30,000.00 /yearExperience:development: 2 years",
"jobName": "Middleweight web developer"
},
{},
{},
{}
]

forEach run in synchronous fashion so no matter how much time it will get to scrap the webpage, it won't run for the next array item. What can cause trouble here is your code maybe pushing an object in the array before scrapping the webpage. This is why you are getting an empty object. What you do is to push object after your program finishes visiting the URL.
Do something like this
jobObj.forEach(function (item) {
var newJob = new Object;
request(item.url, function (err, res, html) {
// Scrap URL and save values in newJob
});
jobs.push(newJob);
});
if your code still pushes an empty object before completing the request then consider using Async module.

How to get spotify artist id for the spotify endpoint url?

I am trying to get the top ten tracks for an artist, but I don't know how to get Spotify id for that particular artist? this is the end point.
GET /v1/artists/{id}/top-tracks
The premise is this, i should be able to type an artist name, and I should get the JSON data for the top ten tracks.
function test() {
var query = $('#searchArtist').val();
console.log(query);
$.ajax({
url: 'https://api.spotify.com/v1/artists/' + query + '/top-tracks?country=IN',
success: function(response) {
console.log(response)
}
})
}
$('#search').on('click', test);
Am I supposed to first authorize? Because I searched the document and fetching the top ten tracks data doesn't require authorization.
Also, to let you know. There is no backend environment here, as mentioned in the docs, it is not required for simple data fetch.

What TZHX said. You need two seperate requests. Javascript in the browser is not my main environment. I will explain with curl and jq in bash. Hopefully it is clear enough.
$ API_ARTIST_URL=$(curl -s 'https://api.spotify.com/v1/search?q=Daft+Punk&type=artist' | jq -r '.artists.items[0].href')
$ echo $API_ARTIST_URL
https://api.spotify.com/v1/artists/4tZwfgrHOc3mvqYlEYSvVi
$ curl -s "$API_ARTIST_URL/top-tracks?country=US" | jq -r '.tracks[].name'
Get Lucky - Radio Edit
One More Time
Instant Crush
Get Lucky
Lose Yourself to Dance
Around The World
Harder Better Faster Stronger
Doin' it Right
Something About Us
Give Life Back to Music
Another detail. ?country=IN will not yield any results, because Spotify has not launched in India yet.

Creating LinkedNotebooks from API

I am working with the Evernote api trying to make linked notebooks from within our web application.
We are using evernote-sdk-js version of the API.
We are able to create the Shared Notebook from the API. Also we are able to create a Linked Notebook and receive a valid response. The problem is the linked notebook is not working. The notebook is returned in the ListLinkedNotebook api call but when I try to authorize it I receive an error.
Error { identifier: 'SharedNotebook.id', key: 'xxxx' }
From what I've read means the notebook is no longer shared by the owner. This is not true as the owner account still shows the pending request in the 'share' window.
Here is the logical flow:
1) User enters email address of the person they wish to share notebook with. A request is made to our server to create a ShareNotebook.
We create a Shared Notebook like this: (From Owner Account)
var notebook = new Evernote.SharedNotebook();
notebook.notebookGuid = guid;
notebook.email = email;
notebook.privilege = Evernote.SharedNotebookPrivilegeLevel.FULL_ACCESS;
var userclient = new Evernote.Client({token: token, sandbox: sandbox});
var notestore = userclient.getNoteStore();
notestore.createSharedNotebook(userclient.token, notebook, function(err, results){
callback(err, results);
});
With an example response like:
{
allowPreview:null
email:"...#gmail.com"
id:12345
notebookGuid:"d2fa3cbe-...."
notebookModifiable:false
privilege:4
recipientSettings:null
requireLogin:null
serviceCreated:1469079222000
serviceUpdated:1469079222000
shareKey:"xxxx-sxxx"
userId:999xxxxx
username:null
}
2) Since Evernote doesn't send an email from the API request we manually send the user a link to activate the Shared Notebook.
Code to create LinkedNotebook: (From Shared User Account)
var notebook = new Evernote.LinkedNotebook;
notebook.shareName = options.shareName;
notebook.username = options.username;
notebook.shardId = options.shardId;
notebook.shareKey = options.shareKey;
var userclient = new Evernote.Client({token: token, sandbox: sandbox});
var notestore = userclient.getNoteStore();
notestore.createLinkedNotebook(userclient.token, notebook, function(err, results){
callback(err, results);
});
And the example response:
{
shareName: 'Notebook Name',
username: '...',
shardId: 'sxxx',
shareKey: 'xxxx-sxxx',
uri: null,
guid: '4f8df3c2-...',
updateSequenceNum: 630,
noteStoreUrl: 'https://www.evernote.com/shard/sxxx/notestore',
webApiUrlPrefix: 'https://www.evernote.com/shard/sxxx/',
stack: null,
businessId: null
}
Has anyone had any luck creating a Linked Notebook from the API? It seems like the createLinkedNotebook API call is not working correctly. Thanks for your help.

tl;dr you can't do what you're trying to do right now, sorry.
So there are a couple pieces missing here, not something you can do much about.
The first piece is that the shared notebook must be "claimed" by a user (the recipient), during which it gets assigned to the user. Until it's claimed, a LinkedNotebook pointing to the SharedNotebook won't actually give you access to the underlying Notebook - authenticateToSharedNotebook will fail.
The second piece is the shareKey itself. We renamed the struct member you see as shareKey to globalId on the SharedNotebook and sharedNotebookGlobalId on the LinkedNotebook - we're currently in the process of updating our developer documentation, and then we'll update the sdks. To "claim" a sharedNotebook, the recipient needs to call authenticateToSharedNotebook with the shareKey, not the globalId. The catch is that we don't expose a method to generate the real shareKey, so there's no way for third-party clients to generate this and join the shared notebook. Once it is claimed, you can call authenticateToSharedNotebook with the globalId to get access to the notebook.
There's a shareNotebook method we added that send an email as part of the API call. That email contains a link for the user to click on which contains the shareKey and allows claiming a SharedNotebook, but that isn't currently in the SDKs or documented. We'll hopefully get it up with this round of doc updates.