I am working on a project where I want to scrape a website that produces content periodically, store the content in a MongoDB cluster, preview the documents in a chrome extensions.
First two step are completed, the website is being scraped through a python script (automated through GitHub actions) and dumped in a MongoDB cluster.
I am having trouble going through the third step, what is an optimal way to fetch the data from MongoDB and preview you them in an extension. Could this be achieved using Python? Could this be done client-side Javascript? or is Node.JS needed here?
Any ideas are welcome, excuse me for a vague question but I have ran out of resources.
You will need your Chrome extension to call a service/server somewhere else. Luckily, if all you need to do is get a document back that is very easy to do!
You can use various programming languages or even a hosted service like MongoDB Atlas. Then make a call to whatever page/api you've made inside your chrome extension.
While you can use a lot of different programming languages and frameworks MongoDB has some easy to follow tutorials:
Python with PyMongo
Node.JS
MongoDB Atlas (a hosted service with a free tier, also has Python, Node.JS directions)
Related
I need to create a web application that perform queries on a remote Oracle DB Server instance.
Unfortunately it is not possible to install any additional software on the machines - and yes, I know that connecting directly from JavaScript to the Oracle DB Server is dangerous - but I need to do this.
In previous versions of the application the ActiveXComponent in the Internet Explorer was used. But this does not work in any other browser and I can't use Internet Explorer any longer.
So, does anyone has an idea that solves this problem only with the help of JavaScript?
Take it easy,
Julian
It's not possible to do this from a browser-based JavaScript environment - you don't even get to the point of it being dangerous, it simply won't work to begin with.
You won't be able to load the database drivers, and browser-based JS won't be able to make that kind of socket connection, it can only make HTTP connections (for security reasons, mainly).
The bottom line is, you need some server-side code as an intermediary - either your own creation, or provided by someone else. This is universal for all database platforms. (Even ones you may find which might expose functionality via a HTTP API are like this, it's just that the vendor wrote the API using some server-side code and then deployed it in front of the database - it's still not a direct connection to the DBMS).
Thank you for your answers.
I've found an other solution (java programm connects to the database and shows the webpage in a JWebBrowser). Working for my use-case.
I'm currently working on a project that involves parsing through a user-supplied file, doing computations with that data, and visualizing the results using graphing utilities. Right now, I'm stuck with using Python as the back-end because it has scientific libraries unavailable in JavaScript, but I want to move the entire tool to a web server, where I can do much slicker visualizations using D3.js.
The workflow would be something like: obtain the file contents from the browser, execute the Python script with the contents, return jsonified objects of computed values, and plot those objects using D3. I already have the back-end and front-end working by themselves, but want to know: How can I go about bridging the two? From what I've gathered, I need to do something along the lines of launching a server, sending AJAX requests to the server, and retrieving data from the server. But with the number of frameworks out there (Flask, cgi, apache, websockets, etc.), I'm not really sure where to start. This will likely only be a very simple web app with just a file submit page and a data visualization page. Any help is appreciated!
Apache is a web server, flask is a web framework in python, websockets are a protocol, and cgi is something totally different, and none of them help you on the front end.
You could deploy a simple backend in flask or django or pylons or any other python web framework. I like django, but it may be a little heavy for your purpose, flask is a bit more lightweight. You deploy them to a server with a web server installed and use something like apache to distribute.
Then you need a front end and a way of delivering your front end. Flask / Django are both fully capable of doing so in conjunction with a web server, or you could use a static file server like Amazon S3.
On your front end, you need to load D3 and probably some kind of utility like jQuery to load and parse your data from the back end, then use D3 however you like to present it on screen.
Flask is easy to get up and running and is Python based. It works well with REST APIs and data sent by JSON (or JSON API).
This is one solution with which I have some experience and which seems to work well and is not hard to get up and running (and natural to work with Python). I can't tell you whether it is the best solution for your needs, but it should work.
If you are overwhelmed and don't know where to start, you can pick one of the options and google search for a tutorial. With a decent tutorial, you should have an example up and running by the end of the tutorial, and then you will know if you are comfortable working with it and have an idea whether it will meet your needs.
Then you could do a proof-of-concept; make a small app that just handles one small part (the one you are most concerned about handling, perhaps) and write something which will do it.
By then, you can be pretty sure you have a good tool for the purpose (unless you were convinced otherwise by the proof-of-concept -- in which case, try again with another option :-))
I have been tasked with researching technology for generating a dynamic report with charts & visuals.
There are countless BI tools available and online solutions that fit the bill, however, the customers have considerable IT restrictions preventing the installation of applications or sharing data across the internet.
One solution I've dreamed up is to serve up an HTML document for offline consumption in a browser. With javaScript bundled in & data added dynamically prior to sending to the client, I believe this would be possible in much the same way a website functioning offline is possible.
How feasible is this? If it's possible, what technologies are recommended? So far I've considered building this using AngularJS since an SPA could mimic a one-page report well & javaScript bundling is readily available.
Are there better solutions? Which?
May be your clients will be okay with providing them files by SFTP. Say, if you are not using a database and are ok with building charts based on files, you can try to provide end users with the files (txt, csv, xml...). Then you will be able to execute client-side code with no server needed. You may even go for some open-source Javascript libraries for data visualisation like D3. All you have to do is to allow local file access for user's browser.
Otherwise you need a local server. I would recommend you to use Python python's SimpleHTTPServer or node.js may fit you.
I'm still new to web development. To learn more about JavaScript(JS) and web development, I am thinking of writing a simple web app which pulls and records time-series data (say, the price of a stock) periodically and draws a live chart showing the historical data. In addition to price data, I would like the app to record/maintain some user-related info such as the ticker of the stock(s) associated to each user.
Ideally, I would like to keep the app light-weight and portable/standalone (meaning, reduce the dependency as much as possible, and the end user hopefully doesn't have to do a lot of configuration/install of dependencies). The issue that I cannot figure out is where to store the historical data. I looked around for database solutions which will allow the app to write data directly from the browser (that is, using JS) to the client's machine. LocalStorage and IndexDB are non-persistent as far as I understand. Some suggested using PouchDB, but upon looking at it closer, it seems like the user need to install CouchDB or some compatible DB (say, SQLite). But that means I cannot share my app with users who aren't technical enough to install and configure CouchDB or SQLite on their machine before using my app.
If anyone could share some insights as to which DB might allow a JS-based app to write persistent data to the client's machine (if such thing even exist), that would be greatly helpful. If there is no such DB solution, please feel free to let me know alternative solutions that would allow the goal of building a simple, portable, JS-based web app. Thank you!
I think the best solution is to use Electron.js. The whole idea of this framework is to create web apps that can reside on client machines. You could package up any DB option you want, or even better, just include an API to your backend through the web app and it will work on your client machine like I think you want it to.
As for DB options, there is a great thread on S.O. that talks about what is possible. It looks like knex.js is your best bet (full disclosure - I haven't used knex).
This is more a general question, I'm not asking for implementation details.
I have a Python script running on a machine which monitors an experiment and gathers live data from it. I want to visualise the data on a website and decided to go with d3.js
My current setup however is absolutely basic:
The python script is running in an endless loop and appends new data
to a text-file (CSV file).
There is an HTML-page on the web
server with a javascript, which grabs the file and creates a graph
using d3.js.
I already thought about replacing the text-file with a sqlite database, but I'm sure there is a more sophisticated and easy way to do it via network.
So I am looking for a more solid solution and searched the web for different kind of strategies but most of those involve bigger libraries (Tornado, node.js etc.).
I would be happy with quite basic socket-like solution and tried to figure out how it would work with WebSockets but not sure how to handle this.
So my question is, what kind of design would you recommend or use if your only goal is to have a Javascript running in a browser and waiting/listening for data (let's say every few seconds a float), which is sent from a Python script?