We Keep Coding

Looking for ways to aggregate info/data from different websites

I'm a fresh-out-of-college programmer with some experience in Python and Javascript, and I'm trying to develop either a website or just a back-end system that will aggregate information from online market websites which don't have any API (or none that I've found, anyway). Ideally I would also want the system that can write to local storage to track changes to the data over time in some kind of database, but that's down the road a bit.
I've already pounded out some javascript that can grab the data I want, but apparently there doesn't seem to be a way to access or act upon data from other websites due to data security protections or to save the data to local storage in order to be read from other pages. I know that there are ways to aggregate data, as I've seen other websites that do this.
I can load websites in Python using the urllib2 and use regular expressions to parse what I want from some pages, but on a couple of the desired sites I would need to log into the website before I can access the data I want to gather.
Since I am relatively new to programming, is there an ideal tool / programming language that would streamline or simplify what I'm trying to do?
If not, could you please point me in the right direction for how I might go about this? After doing some searching, there seems to be a general lack of cross-domain data gathering and aggregation. Maybe I'm not even using the right terminology to describe what I'm trying to do.
Whichever way you would look at this, please help! :-)

i suggest you use selenium webdriver to login to get cookie,and use requests library to scrap the message.That is what my company do in the scraping system.if you only use selenium webdriver, you will need to many memory and cpu capacity.
if you are good at html and js,it is a good way for you to use requests library to Simulate logging.
for the website you must log in,the most import thing is to get cookie.

Suggestion for the best way to store persistent data for a light-weight, portable JS-based web app

I'm still new to web development. To learn more about JavaScript(JS) and web development, I am thinking of writing a simple web app which pulls and records time-series data (say, the price of a stock) periodically and draws a live chart showing the historical data. In addition to price data, I would like the app to record/maintain some user-related info such as the ticker of the stock(s) associated to each user.
Ideally, I would like to keep the app light-weight and portable/standalone (meaning, reduce the dependency as much as possible, and the end user hopefully doesn't have to do a lot of configuration/install of dependencies). The issue that I cannot figure out is where to store the historical data. I looked around for database solutions which will allow the app to write data directly from the browser (that is, using JS) to the client's machine. LocalStorage and IndexDB are non-persistent as far as I understand. Some suggested using PouchDB, but upon looking at it closer, it seems like the user need to install CouchDB or some compatible DB (say, SQLite). But that means I cannot share my app with users who aren't technical enough to install and configure CouchDB or SQLite on their machine before using my app.
If anyone could share some insights as to which DB might allow a JS-based app to write persistent data to the client's machine (if such thing even exist), that would be greatly helpful. If there is no such DB solution, please feel free to let me know alternative solutions that would allow the goal of building a simple, portable, JS-based web app. Thank you!

I think the best solution is to use Electron.js. The whole idea of this framework is to create web apps that can reside on client machines. You could package up any DB option you want, or even better, just include an API to your backend through the web app and it will work on your client machine like I think you want it to.
As for DB options, there is a great thread on S.O. that talks about what is possible. It looks like knex.js is your best bet (full disclosure - I haven't used knex).

Back-end solution for pure-Javascript site with link sharing

I'm developer-beginner and I would like to do simple card-sorting site.
hundreds of cards with plain text
no log-ins, it must be as simple for user as possible
few dozens users per day
final state of sorting should be saved under unique link, like site.com/q56we1cfuz4 or site.com/link.php?unique=q56we1cfuz4
User sorts cards as he/she wishes and the result is saved into browser sessions storage (few kb of data in JSON or so). So far, so good.
Question:
As I know only Javascript:
If I need to share the state of the page (data in session storage) with some unique link, is something like Firebase.com good solution for such kind of back-end? Or some simple DB with help od Node.js?
In order to "catch" the link when someone click at unique URL (site.com/link.php?unique=q56we1cfuz4), I still need some server-side script which will ask the DB, like PHP header redirect, right?

Your questions are a little fuzzy, no problem tho. You are just getting into web dev, so there's a lot to wrap your head around and all of the options can be pretty confusing. Some people will complain about opinionated answers, and I'm not going to claim to be objective here, but here are a few tips that I think will get you pointed in a better direction...
Firstly, yes - firebase is a good solution for you to try working with. Aside from the fact that it will give you the db/storage features you need, it's a realtime db, which will give you a lot more than just storage in the long run, and will better equip you for the future web development trends. The firebase API is (mostly) designed to work asynchronously, so from a javascript perspective, it falls right in line with the kind of code you'll end up learning to write.
Re: the other aspect of your question - server-side - check out nodeJS. It's basically a server-side javascript platform that will allow you to use the same skills you're learning to write client-side code for the server. Also check out expressJS, a nodeJS package that provides you the http-server, and allows you to handle dynamic urls, etc. - the bits you were thinking about when you made a reference to PHP.
Hopefully this will help you focus on a few specific tools to familiarize yourself with as you learn web development, rather than having to struggle with everything from new languages, platforms, and way too many libraries, frameworks and techniques to wrap your head around. Good luck!

Connecting to couch db from a webpage

I used to use mysql databases and a php backend for my web applications. Now that I am diving deeper into the the javascript world I am looking for a more compatible way to store my data. I like the idea of writing everything in one language, and javascript seems to be capable to make that job.
I started reading about couchDB and the JSON/javaScipt approach seems to fit perfectly to my needs. My question is now, which would be a convenient way to connect to the database from my pages.
I found this post, which explains how to connect to a couchDB directly from a client-side javascript using a library called couch.js. The post is quite a few years old and the library I cannot find anywhere in the web. So I am asking myself If this approach is still up to date.
I was thought that one important reason for using a server backend is to provide a security mechanism to protect the database authentification process. So: Is it generally secure to save an (encrypted) db pwd in a client side script?
So what I am looking for is a quite simple way to access and store json-like data from webpages. Are there any suggestions?
The hosting of the production solution should also be not that difficult and in the end not expansive, since it's rather meant for smaller projects at the moment.
If the post does not fit to the forum, let me know a better place for it..

I have been using the CouchDB jQuery Plugin for development. At first I was hesitant as well to use it due to most documentation being at least a few years old. So far it has worked really well and been very easy to use, especially if you are familiar with JavaScript/jQuery.
...
You can get the plugin from Github
CouchDB jQuery Plugin
Or you can save it directly from Futon:
http://localhost:5984/_utils/script/jquery.couch.js (where local host is your IP and 5984 is the port CouchDB is on)
Whatever you do, make sure you enable CORS on your database. There are instructions on the Github Readme page, or this script is super useful and easy to use through npm.
...
Documentation can be found here.
This reference has also been very useful for me.
...
Good luck!

I just released Slouch, a JavaScript client for CouchDB that can be used in both the browser and in node. I hope this helps!

Is a database required for a "quiz" type of game?

I don't know much about databases, I've been asking a few questions about them lately to get a better understanding but I'm still a bit confused about what does and doesn't need one.
I'm making a simple application using HTML/CSS/JavaScript, it has a few quizzes and "tutorials" targeted towards children. I don't want the next tutorial/quiz to be unlocked until the previous one is completed.
So for that would I need a database so that it "saves" when one is completed? I don't need to save scores or anything like that, they just get to move on once they get a passing score.

Any other requirements such as saving to a profile or needing to persist between sessions (e.g. changing of device)?
Browsers have localStorage APIs now which allow you to save a lot of the data (and keep it for a set duration of time). There are also good'ol'fashioned cookies which allow you save pieces of information as well.
Keep in mind that both of the above mandate the user use the same browser and allow these mechanisms. Obviously using "private"/"incognito" browsing would also affect saving status.
It's up to what you feel the requirements are.
EDIT Just saw your mention of a mobile app. If you're planning on allowing the experience to transcend devices, you'll need a database. otherwise, you'll be relying heavily on if they use cross-device sync (like Chrome and Firefox do with bookmarks, passwords, etc.)

If you don't mind that people can do a "view source" on the webpage or use every browsers' developer tools to find out the answers or move on to the next tutorial or quiz, then you can use cookies to store the user's status. Or you can use the preferable Web Storage API.
You might want to look at Firebase. Using just simple JavaScript on the web browser, you can have users with logins (or just allow them to login via Facebook or other services) very easily. And then you can store and retrieve data very easily as well, like quizzes, tutorials and results. This way nobody can see the answers even if they're adept at analyzing the webpage.

When you don't use database, before any check, you have to load all data in your static page.
So My sloution: store students situation in a cookie. On each page check cookie status and then use Jquery remove() to remove (Client-side) those parts of page that he/she can not access.
EDIT
This wont work when JavaScript is disabled.

There seems to be a lot of ideas but no clarifying on the database subject.
TL;DR is: No.
Now for the specifics. A database is nothing more than a way to store information. While traditional "SQL" databases (it is pronounced "Sequel" as in "My Sequel" for MySQL) have concepts of tables, where you define columns with items to store and saves each row with its value, much like an Excel file, some databases like Redis store key-value pairs and others lide MongoDB store JavaScript Objects.
You can store information in the source code (As Variables possibly) or in a file. A database is a way to organize that information.
With that said, in your case, you probably need a backend or an API. An API is basically a means of communication with a server through AJAX (JavaScript in the browser asks for stuff). That would be your way to retrieve information from the server as needed, so that users wouldn't see the answers before they answer.
With that out of the way, there are some options. FireBase (As noted on other answer) and AppBase are easy ways to integrate this concept with little effort. But they tie you and your information to their system, and they are mostly targeting more resource intensive apps.
Since you are using JS and seem to be enjoying your learning experience, I would suggest you consider suing NodeJS and defining the data as either a JSON file or a variable in JS. You keep working on your problem but add options and get to learn some stuff.
If you decide to integrate a database and possibly do some neat stuff, you have most of the groundwork done already.
If NodeJS picks your interest, Mean.IO and KrakenJS are, in my opinion, the best places to start, though they may both seem overkill in your specific case.
Do consider though: A database is just a small possible piece in a puzzle, and it's mostly a horrible way to name some of the software that tries to organize your information. Consider first if you need to organize information, and what and how do you need to organize, then start thinking if databases are the best way to organize it.