Getting only one part of an HTTP GET Request - javascript

I want to use the Rest API of Github, but I only need a small part of the info returned in JSON from the GET request. Is it possible to get exactly what I need in order to avoid the extra network usage? This will be done in javascript, but I guess if it's possible it would work on different frameworks.

Related

architecture for get and store api request data

This is more of a architectural questions. An external platform had product and price information for let's say, books. There is an API available to get this information.
What I read is that it should be possible to create a function in Javascript and connect the Javascript to a page where you want to show the data on my own website. This would mean that for each page request an API-call is made. Since the requested information only changes once a day maximum this does not sound the most efficient solution.
Can someone advise a better solution? Something into the direction of a similar php or javascript function that does the request on the background, schedule an update and import the data into mysql? If so, what language would be most common.
I need the solution for a Joomla/php/mysql environment
Here's a simple idea - fetch and store results from the API (ones you think aren't gonna change in a day), either on disk, or in the database, and later use these stored results to retrieve what you otherwise would've fetched from the API.
Since storing anything in frontend JS across page reloads isn't easy, you need to make use of PHP for that. Based on what's given, you seem to have two ways of calling the API:
via the frontend JS (no-go)
via your PHP backend (good-to-go)
Now, you need to make sure your results are synced every (say) 24 hours.
Add a snippet to your PHP code that contains a variable $lastUpdated (or something similar), and assign it the "static" value of the current time (NOT using time()). Now, add a couple of statements to update the stored results if the current time is at least 24 hours greater than $lastUpdated, followed by updating $lastUpdated to current time.
This should give you what you need with one API call per day.
PS: I'm not an expert in PHP, but you can surely figure out the datetime stuff.
It sounds like you need a cache, and you're not the first person to run into that problem - so you probably don't need to reinvent the wheel and build your own.
Look into something like Redis. There's an article on it available here as well: https://www.compose.com/articles/api-caching-with-redis-and-nodejs/

what's the fastest way to web-scraping and get different items from a website

I've been working on a web-scraping project written in Python using Selenium and requests. Each time when I need data I send a request to get it (or using Selenium) and each request takes time.
My question is, is there any option to get a lot of different data ( such as product's name+price+shipping) in 1 request in Python or even in javascript?
Given a url for example: https://www.amazon.com/Dell-Inspiron-7573-i7-8550U-Windows/dp/B07NRC8ZXC/ref=sr_1_1_sspa?keywords=laptop+i7&qid=1572590892&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEyNjFPSFdTOEpVODdQJmVuY3J5cHRlZElkPUEwNTU5Nzk1MlBGWFkxU0JKOVlLNiZlbmNyeXB0ZWRBZElkPUEwNjU0MzYwM0NRT01ER1oxSDdMOCZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=
NOTE: you marked this question as duplicated with this question - How to click on Load More button within Google Trends and print all the titles through Selenium and Python
PLEASE, those are different questions, I'm asking about getting this product's price+name+shipping who are under different "scopes", she asked about getting the same "objects" (such as names)!!!
so if you know any other way to get those things instead of sending 3 different requests to get them, I would love to hear about them... I've heard that there is a way to combine a few requests as one big request in javascript, is it true?
still i'm waiting for an answer, anyone?

Object inside Array as External File for Express

I searched around and found pieces of what I want to do and how to do it but I have a feeling combining them all is a process I shouldnt do and that I should write it a different way.
Currently I had a small application that would use a MSSQL library in Node to Query a SQL Server with a sql command and get the results and store it as an object. Which then I use express and some javascript to decipher it or modify it before calling it with an AJAX call and responding it as a proper JSON object.
SQLDB -> NodeJs -> API -> localhost
My problem now is I want to repurpose this and expand it. Currently storing the SQLDB responses as objects inside an array is becoming a huge memory problem. Considering some of these requests can be hundred thousands of rows with hundreds of columns, the node process begins eatingup outrageous amounts of RAM.
I then thought, maybe I could easily just take that object when it comes through in the result and write it to file. Then when ajax calls come to expressjs, it can read from the file and res.json from there.
Will this work if say 50-200 some people request data at the same time? Or should I look for another method?

avoid mismatch between server side and client side result

I use PHP and Javascript. In my website some results are processed server side some client side.
Using javascript only, prevents your website from being crawled correctly by search engines and using PHP only prevents correct real time response.
The problem is how to grantee both js functions and PHP functions give the same result? for example suppose there is a function which gives relative time:
JS:
function relative_time(timestamp)
{
...
}
PHP:
function relative_time($timestamp)
{
...
}
Keeping both functions matched with each other is not easy since I want to edit both. For example if both give us:
one year ago
And I change PHP only, to give me:
a year ago
Then JS is not updated too. Is there any standard way to ensure both will act in the same way?
unfortunately js function cannot be called on server side.
If there is some complicated logic, you should implement it server-side and just pull the results via AJAX. That way you only need to maintain the PHP code and provide a kind of AJAX API for access via JS.
I think you just need to make a decision where it is to be done, because if they do vary which is to be dominant ? (that's the one that should be doing it)
Also, how are you saving server load by doing it in both locations ?
Avoid this by making a decision for which code is to do it, failing that, Put a note in you code at both locations reminding yourself to update both locations ?

How to show a friendly error message using Open-flash-charts2?

If my JSON data-file comes from a database result set and that result set is empty, how do I tell OFC2 to display an error message, instead of crashing because of a malformed JSON string?
Add tags for javascript and actionscript-3 to this question and you should get a load more views and useful responses than you currently are, with more precise details than I am giving. Post the actual JSON string that is causing you the problem and that you would like to be guarded against. That way people can suggest a regexp to catch it, treating it as a string rather than as JSON data at some point before JSON.decode() happens.
In more detail:
You can catch it in two places. One route is to switch over to using the javascript interface to OFC2 and use client side javascript to detect the bad string. This allows you to modify the JSON string client side. See http://teethgrinder.co.uk/open-flash-chart-2/tutorial-5.php for that approach. One downside is that the clients must have javascript enabled for this to work.
Alternatively, since OFC2 is LGPL, you or an actionscript developer can dive into the OFC2 source code and do the same thing there. I am not an actionscript developer so you are better off ensuring you get a reply from one.
The main thing is to add those two tags to this question. 22 Views is way too low for a question with a bounty of 500. Hope this helps.
Several solution avenues are possible, depending on your level of access to the server and your knowledge of JavaScript and/or any server-side platforms.
With access to database
Depending on the kind of data you are displaying, it might be possible to add dummy records for those queries that would otherwise have returned an empty set. If you have access to the query definition, you may check for the empty set in the DB-query. For example, if you're on MS SQL Server you could check the condition with some T-SQL statements.
With access to server
If you have access to the server side script generating the dataset, add a condition that returns some default value that OFC2 will handle correctly.
With access to another server or serverlocation
If you don't have access to the specific script, you may consider creating a new script at another location that queries the original script and replaces empty results with the default value.
Client-side only
You can add the JavaScript open_flash_chart_data function (see tutorial) to replace empty datasets. OFC2 can use that function as data source.
Hope this helps.

Categories