On yahoo finance there is an option to download the historical data for any given Ticker symbol. However I am most interested in Yahoo Finance Premium's 'Visitor Trends' data. This 'Visitor Trends' data is shown with an interactive chart which overlays the visitor trends over the stock's rising and falling performance. The interactive chart reacts to the users cursor and makes it easy to see where spikes in a given stock's page visits correlates to a change in its price.
I want to download the visitor trends data to build a dataset, but there is no way to direct download through yahoo finance.
I have been looking for ways to scrape the data I need by intercepting it before it is rendered by the chart. Has anyone dealt with a similar puzzle/hurdle? If so, what should my approach be?
Sorry if am not specific enough. I can get more in depth if necessary.
If you are looking to scrape or take information from a website look into puppeteer. It is a library designed for exactly that
Related
I am building a website that allows users to find locations and areas around the world based on data other users will be allowed to submit. I have a database containing all of the location data, latitudes and longitudes.
I have implemented the Google Maps Javascript API, as well as the Static API for simple things in the website. What I would like to do is allow users to view all of the database's map points on a world map for the user to drag and zoom to the different areas to view and select a pin for more information about that location.
There is an example on Google's site that shows how to do something like this here. But what if I'd like to hide this data from the client, rather than passing to them an entire XML file with all locations?
I supposed what I would like to do is make it more difficult for someone to intercept all of the coordinates and save them to their computer. This might be a subjective question because of my novice understanding of this and I understand it is probably a trade-off type situation.
So basically, you don't want to make it easy for someone to take all your data at once?
A fairly straight-forward strategy for this would be to provide search and filter controls that interact with discrete apis (ajax services). You could have an API that provides summary data about how many points are in one area and provide guided search constraints to let the user drill down further.
You could then only return concrete data points once the user gets to a specific level of depth.
Does that help?
I work for a non-profit that holds free sports and physical activity events. I'm am trying to set up a system to store and collect information about our membership and the events we hold, using Google Docs as the user interface and Google Cloud SQL for data storage.
Like most non-profits, we do not have a lot of resources available for advanced computer programming. I would like to use Google Forms as a simple UI that our research and evaluation staff can use to build data collection tools without coding. The ease of access to the responses in Google Sheets is great; however, we hold a lot of events and will quickly exceed the 2 million cell limit. So, I think we will need to store the responses in a SQL database.
What I would like to do is modify the action performed on submit, such that the form:
Does NOT submit to a sheet
Connects to a SQL table (I've set this up on Google Cloud SQL)
Dumps the responses into the correct columns
If the form was modified, add any new columns to the table (like Forms does with attached sheets).
I know that Google Apps Script can connect to external databases through the JDBC service (https://developers.google.com/apps-script/guides/jdbc) and I know that I will need to use the getItemResponse method (https://groups.google.com/d/msg/google-appengine-stackoverflow/laLkcneaqZo/PfPKKYlmva8J) to execute this.
I have very little experience with Google Apps Script or JavaScript in general....basically, the most I've done is finish the Codecademy JavaScript course. However, I am a quick learner and I'm looking for a little example code that might get me started (remembering that I have almost no experience).
I've searched through this forum, GitHub and everywhere else I can think of, but cannot find something that is quite right. I'm mostly pointed to the JBDC documentation, but do not know how to use it. Any help you can provide would be much appreciated.
Brett
I wrote some articles about this at my blog, Bit Vectors. This article might cover most of what you need. That specific article explains how to build a Google Apps Script solution tied to a Google Sheet file. The GAS solution front-ends Cloud SQL. The end-user sees a web page / form, and if you want, the Google Sheet itself. To answer your list items:
1) The web form does write to a sheet, but in the software, you can easily prevent this
2) The web form definitely connects to your Cloud SQL table / database
3) The solution writes to the table(s) / column(s) you want through stored procedures
4) The solution will not modify the data table structure if you change the form - you would have to handle this change yourself
See a demo here.
HTH!
Frank
I have few excel pivot bar charts and trend analysis graphs.
Is there a way to display the excel charts on a webpage, without losing the interaction capabilities?
So, from the excel users can filer,dig deeper etc.
Can i do the same using some other technologies?
Any technology stack is welcome, open source more so.
I am trying out Kibana right now, but doesn't seem to have all the capabilities as excel does.
[Update] I tried out Google charts, but that sends data to Google server, which cannot be allowed.
I don't like to post an answer to my own question, But http://www.highcharts.com/ is my favorite now. Very easy to use charts and quite zingy.
Minor point : it's not free for commercial usage, so we will have to get a license for it.
I am looking for a way to extract built in property boundary data from the Google API to know where to highlight areas for information needs.
It's easy enough drawing a polygon, but I want to know where to draw it in a more automated fashion. Perhaps similar to Geocoding for extracting an address location, except the geocoder only provides a central coordinate and a bounding box (for screen navigation) unfortunately.
Does Google provide this property data in some form?
Thanks
Having looked through the Maps APIs fairly extensively and no one suggesting otherwise. I think it's pretty conclusive Google Maps does not provide property boundaries.
The best way to go is to look for a service that does have the data to integrate into Maps, but likely costs money.
As an example these may be:
Your local government's land services
RPData.com - http://www.rpdata.com/residential_property_information/residential_property_information.html
ReportAllUSA.com (if you are American) - http://reportallusa.com/
None of these are guaranteed, looking into them myself, but may also be different from person to person depending on costs. If I find a free one, I'll give a shout.
For anyone looking to implement this on Australian Region.
State & Local councils have open data free for usage.
I am working with Brisbane, QLD Australia:
For Brisbane: https://www.data.brisbane.qld.gov.au/data/dataset?q=parcel
For QLD: https://www.data.qld.gov.au/dataset?q=parcel
P.S parcel dataset contains information about individual Property Listings. It does have a learning curve & takes sometime to use open data.
I have some usage queries for my web app's database, the results of which I want to display graphically. Is there an easy-to-use API that exists for this purpose?
I want to show things like average query-time per user (a small user-base), average query time per day, and things like that. I think it would be cool to show these on a two-axis graph.
I am displaying this data on my site, so a jQuery/Javascript/HTML solution for rendering information into graphs would be ideal.
FusionCharts has easy to use graphs - nice xml interface - but not free.
edit:
this is a good link for several options:
The Google Chart API can be handy for this kind of thing. If you want your own hosted solution, High Charts is pretty nice.
There are tons of good javascript/jquery charting controls here:
https://stackoverflow.com/search?q=javascript+charting+library