For my college project, I'm developing an application that analyzes/detects anomalies in videos.
The backend is Python and the frontend is JS/NodeJS. I setup the backend so it runs 4 ML models and writes the outputs to 4 separate .txt files. I created a JS file that reads the .txt files and reformats the data to be saved as 4 .json files. Currently, the frontend just reads the .json files and displays the data... I run npm start to display the application in the browser at localhost:3000. Here is my problem:
In the application, the user can upload a .mp4 file and watch the video (react-player). I think I can setup a button interaction to execute a script that runs the backend .py file. However, I don't know how to continue from there. I need to exchange data twice: First, I need to download the video to the .py file local directory as input to the ML models. Once the models have finished running, I need to
transfer the outputs (Python list or .json format... I know how to do either) back to the application so it can be displayed to the user.
I'm reading online that I can execute data transfer using jquery and/or ajax calls? Or maybe I setup a database? I have 0 experience w/ any of that so I'd like advice on the easiest approach. If you can recommend resources that can help me learn more on the topic, that'd also be helpful.
You can use WebSocket to transfer the files, initially after sending the video to the backend there would be quite a bit of delay for the ML model to process it and come up with the output JSON files, WebSocket would make it easier to make this communication.
(I'm not saying it's not possible to use a rest API to do the same, but it would be easier to use WebSockets)
Related
i have written a game in javascript with the p5.js library. Now i want to host the game on a server to conduct a survey on a service like amazon turk. Ideally the clients recieve a URL to the game and play it while in-game actions are tracked and stored in node.js or on the server and exported as a .csv file once they are done playing. After they finish the game the csv. file should be sent automatically to a location that i can then access. I have zero experience in server hosting or similar topics.
So a couple questions arise:
Is a hosting service like Heroku suitable for hosting the game?
Do i need to use node.js to make this happen?
Which of those two would extract the data and store it to a csv? And where is the file stored?
How do i get or access the csv. after?
Any alternative takes to solve the problem?
Thanks alot in advance!
github repository: https://github.com/luuuucaaa/schaeffers-charade
game on github pages: https://luuuucaaa.github.io/schaeffers-charade/
If I were you, I would do it like below:
Host
Since your project is basically a html & JavaScript static contents,
AWS S3's static hosting would be sufficient (Also, the current git hub pages is another option if you just want to host it).
Hosting on node.js environment is also available using webpack serving, but it requires additional works. (but if you require other npm packages to generate .csv file, you need webpack anyway to bundle js file and attach it to html)
Data Storing
Two ways are considerable,
the first is to store it on the filesystem. Generate .csv via JS script within your app, and save it where the app is hosted (if you go with s3, you can access it afterwards, but I'm not sure if it can write objects by script)
The second is to post the data to another API endpoint. (for example building an API Gateway on AWS that triggers Lambda, which stores it on S3)
It's merely an example and I don't know exactly what you want to achieve, but take it into considerations. Good luck. Cool game BTW.
I need reflection, vision and documentation on my problem.
I wrote a python script to calculate something from an API and export the result in a CSV file. Then, I use a JavaScript script to display the data from this CSV file on a building website.
I need to have the latest data available for my website, so I opened a VM instance in Google Cloud Platform (Google Compute Engine) and set a Crontab job to run automatically my python script. The calculation is now executed every day and the result is exported to the CSV file, but stored in this VM instance.
Here is my goal: How can I get my CSV file on my website? The CSV is always on the virtual machine and I do not know how to communicate with my JavaScript script to the VM. Do I have to communicate directly with the VM? Do I have to go through another step before (server, API, etc.)?
I cannot find a specific solution for my problem on the internet.
Thanks in advance.
How can I get my CSV file on my website?
By making your python script output the CSV into your website's root folder.
Example, if you're running apache, chances are your root folder is somewhere in /var/www/html/...
If the script is generated from another machine (not the one with your website), then I would host it and make the server hosting your website fetch it via cronjob.
Basically:
If your CSV is generated from the same machine as the website that will use it - simply output it to the website's folder
If your CSV is generated from another machine, make it publicly accessible and have your website's machine cronjob fetch that CSV a few minute after it's generated.
I would suggest you a different approach than using a VM for calculating and storing your CSV.
The idea would be using a Python Cloud Function instead, that will be run by Cloud Scheduler and Pub/Sub.
This function will generate your CSV file, that will be stored on Cloud Storage. Here you can find an example of how to upload an object to Cloud Storage using Python.
Then, you need to give your website the ability to access that CSV file in Cloud Storage when required. As indicated by #guillaumeblaquiere, the exact way will be dependent on where your website is hosted: especially, it will constraint the auth mechanisms that you need to use to download your data. The documentation in GCP provides several examples about this matter.
The documentation of Google Cloud provides an example of the proposed architecture.
I came across an article that, in a certain way, describes the setup I suggested as well.
Please, consider review the costs of the different mentioned products.
I want to upload file into folder from which my Angular app is served while running on localhost. I'm not able to find any solution without using backend.
For example I just want to upload an image file and that file should copy in specified folder of the project. This should be done only with Angular without using any Backend script or hitting any API endpoint.
Depending on your webhost, you can make your assets-folder accessible via FTP.
Making a FTP-call from javascript (angular is javascript) isn't that difficult. And there are plenty of example and questions about it on the internet (like this)
Why you wouldn't do that:
The credentials for your ftp-connection will be accessible in the compiled javascript-code. With a little bit of effort, everyone can find it.
Each gate you open through the webhosts firewall, is a extra vulnerability. Thats why everybody will recommend you to add an API endpoint for uploading files so that you keep holding the strings of what may be uploaded.
Edit:
As I read your question again and all the sub-answers, I (think) figured out that you are building an native-like app with no back-end, just an angular-single page front-end application. An I can understand why (you can run this on every platform in an application that supports javascript), but the problem you are encountering is only the first of a whole series.
If this is the case, I wouldn't call it uploadingas you would store it locally.
But the good news is that you have localstoragefor your use to store temporary data on the HDD of the client. It isn't a very large space but it is something...
The assets folder is one of the statically served folders of the Angular app. It is located on the server so you can't add files to it without hitting the server (HTTP server, API, or whatever else...).
Even when running your app on localhost, there's a web server under the hood, so it behaves exactly the same than a deployed application, and you can't add files to the assets folder via the Angular app.
I don't know what exactly you want to do with your uploaded files, but:
If you want to use them on client side only, and in one user session, then you can just store the file in a javascript variable and do what you want with it
If you want to share them across users, or across user sessions, then you need to store them on the server, and you can't bypass an API or some HTTP server configuration
Based on your clarification in one of your comments:
I'm trying to develop a small speed test application in which user can upload any file from his system to check upload and download speed.
The only way to avoid having you own backend is to use 3rd party API.
There are some dedicated speed test websites, which also provide API access. E.g.:
https://myspeed.today
http://www.speedtest.net
https://speedof.me/api.html
Some more: https://duckduckgo.com/?q=free+speedtest+api
Note, that many of these APIs are paid services.
Also, I've been able to find this library https://github.com/ddsol/speedtest.net, which might indicate that speedtest.net has some kind of free API tier. But this is up to you to investigate.
This question might also be of help, as it shows using speedtest.net in React Native: Using speedtest.net api with React Native
You can use a third party library such ng-speed-test. For instance here is an Angular library which has an image hosted on a third party server (ie GitHub) to test internet speed.
I am currently investigating a way to upload Images to a Heroku repo where I have a python-application that takes in the images, has them classified and saves the results in a .csv file.
The Images can be selected for upload via a website that uses Javascript and HTML.
My Question now is, how would I best enable the upload from the website to the Heroku App?
Bearing in mind that the Frontend is currently running on my local machine and that I want to use Heroku as a Backend to take in either Images or Strings.
Will I need an SSH-connection to a separate Web server? Will I need to use Amazon S3?
Not looking for a complete Solution to my problem per se, but if someone could point me in the right direction as to what I will need to solve my problem that would be great.
You could upload an image to Heroku however there two problems with that
Heroku router times out requests after 30 seconds which means that if your users have a spotty connection and/or huge files the upload will fail
Heroku's ephemeral filesystem means that you must process this file in your web process because workers run on different dynos and don't have access to your web dyno filesystem. So that's another strike at 30 seconds timeout.
Your best bet is to have your users upload their files directly to s3 from their browsers. We had a good experience with filestack.com js widget, but there are other ways.
Your page will then ping your backend with this newly uploaded file's s3 url. The backend will launch an asynchronous job using Heroku worker to process it.
This neatly solves all issues with timeouts and blocking your web dynos.
I recently started developing an application in Angular JS and Node Js, it's completely a new stuff for me, I am right now into tricky situation where I want to upload the images or files to a folder where my Angular application is hosted, I am using node js as REST API which is running on the different server whereas Angular JS part is hosted on the different server, all I want to do is to save files uploaded by the users to Angular JS server, I don't know exactly how achieve it. I can make calls to my Node js server and pass file to it and save it their, but my problem is that node js app hosted somewhere else and my angular js is app host somewhere, I want to save my files to angular js server folder
Any suggestions and help from you guys would be much appreciated
Thanks in Advance
You can use ng-file-upload and angular-file-upload, there are other questions already answered.
File Upload using AngularJS
Try to understand my scenario I have two different servers.... My requests are handled by node js app whereas front end part is written in angular js which on the different server, all I want to do is to save file on angular hosted server not on the node server.... If I am not wrong, to upload a file we need make a request to the server and backend server is node js which is somewhere else where I don't want to save files... I am thinking of sending files to node server and from there I will make a use of ftp call through node to save files to my angular hosted server... It's quite inconvenient method but I feel I can achieve what I want from it...