I'm relatively new to javascript but I want to get some data from a csv file that is saved online and gets updated each hour.
The data should be displayed on a table later on but I have some problems with saving it to an array. The csv file is comma seperated, has 9 columns, over 6000 rows and is a long string of text, so no linebreaks. The first row contains usernames and each username with special characters is conclosed with quotation marks.
I've tried several codes over the past few days, but none worked. Can I parse a online CSV file into an array at all? Is there an alternative like with SQL or saving the file to my server?
Remember: The file gets updated each hour..
NOTE: There are not really problems with the codes I've found, all of these were tested by others and seemed to work. But only for local files, not actual URLs!
You can use this https://code.google.com/p/jquery-csv/ plugin and it is possible to convert multi-line csv into 2D-array using $.csv.toArrays(csv) or to an object using $.csv.toObjects(csv). Check this post or this one for more info
$.ajax({
url: "urlto/filename.csv",
success: function (data) {
var arr = $.csvtoArray(data);
_oncomplete(arr);
},
dataType: "text",
});
_oncomplete: function (arr) {
//Your array here
}
You can have a look at papaparse for a solid and full-featured CSV parsing library.
setInterval javascript function lets you update the data every hour, in case you decide to develop this part on the client.
Is there an alternative like with SQL or saving the file to my server?
Yes there are alternatives, the right architecture depends on your use case. How many visitors will go to your web page and view the results, how critical your application is, how reliable the data source is, etc. If you're not sure about these you should talk to a web developer with more experience around these questions.
You may want to parse the CSV file every hour on the server and store a copy of the data there, to serve to your visitors. This way, if the upstream data source is unavailable, you still have a copy of the data from the past.
P.S.:
I've tried several codes over the past few days, but none worked
stackOverflow is about this: getting help about specific problems in your code, rather than asking general questions (answer to those can be found easily using a search engine).
Related
Intro
Hi, I was looking for answer in the whole Internet (in some way I kind of feel that I know every question in Stack Overflow), but the answers were never appropriate to what I'm looking for. I was trying to avoid posting question here, but situation forced me to do this.
Sorry if the answer is simpler than I think.
I'm in the middle of building my first app in Electron using JavaScript. I think that I should describe it in few words, so flam:ngo™ (which is projects name) should work like this:
User will upload two files:
file with tables (like XLSX or DOC)
file with data and blank spaces (which will be used as a template)
App will import from tables.
Now app should let user choose which rows he's interested in and where in uploaded file he wants them to be placed.
flam:ngo save document in PDF (or DOC).
Clue
Right now I need solutions just for myself and in little simpler form. For now I need flam:ngo just to work with one specify XLSX and with one DOC template, but I stuck. I know which rows in document I will always need, but I don't know what should I write to specify in JS's code that I need exactly this ones (like hey, app, pick only this one, this one and maybe this one) while JS is reading file and I don't know how to create new DOC (or PDF) file, how to write data in specified blank spaces and then at the end: how to save it in direction which I should choose for every time I'm using an app - everything in one, maybe two, processes.
Ending
Could you, please, help me: for now I have implemented file uploader which is importing file in XLSX and which is saving it as CSV, XML oraz HTML. What should I do to keep moving forward?
I really appreciate your help!
PS. For better explanation:
For now this should look like this:
1. "template.docx" is uploaded in the core
2. user uploads .xlsx
3. app reads tables and select rows chosen in code
4. app puts data from chosen rows in specify places
5. app saves file > new life of the app :)
Ok, so it's been a while and app has been already written by me. Maybe this will help someone in the future:
Solution
Packages that helps me with this issue was:
js-xlsx which converts my file to simple HTML file, where cells from my XLSX template file have always the same ID (which was important for me to work with document templates).
officegen which helps me write brand new document based on earlier prepared template.
Rest of it was just Vanilla JS.
Is it possible to be able to upload an excel document with varying ranges of data, and have that data dynamically displayed in a basic form of chart(bar, pie, etc.) on our company website.
After doing some research I figured the only two possible ways to maybe do something like this is to use a very complicated macro in VBA or a Javascript parser to read the data and display it then. The data that will eventually go in here will have sensitive information so I cannot use google charts or anything like that.
This problem has to be divided into two parts.
One -part is to gather and process the information needed to display the chart.
Second - This is the easiest, a way to display a chart in HTML. For this, you can use www.c3js.org javascript library to display the chart in HTML.
Regarding part one, it depends in which technology is built your website.
For example, If it is in php, you will need to find a library in php, which can read and parse excel files.
Then you have to create a service in your website, where the data is going to be provided. For example,
www.yourcompany.com/provideChartData.php
You can format the response as json format.
Once you have solved that, you only have to call the service from your page, and the data will be dynamically displayed. You can call it using jquery library for javascript ($.post("www.yourcompany.com/provideChartData.php",function (data) { code to display chart ....}))
There is no real easy way to do this that I have found. I have had to manually parse these things in the past but there are some libraries out there for node that might help you.
https://www.npmjs.com/package/node-xlsx
You can also export form excel as CSV. When you do this, me sure to set the custom separator to something other than ',' and you should be fine to import it into a large array and get the data/charts you need.
https://github.com/wdavidw/node-csv
Hope that helps.
I'm working on a small project where I'm going to read some parameters from a SQLite database. The data is generated from a Linux server running a C code. I then want to create a script using JavaScript to fetch the data from the database. I've tried with alasql.js but it takes very long time (~1 minute) before I get the list with the parameters from two tables.
SELECT sensors.id, sensors.sensorname, sensors.sensornumber, information.sensorvalue, information.timestamp FROM sensors INNER JOIN information ON sensors.id=information.sensorid
I've been reading about IndexedDB but seems like it only works with JavaScript but not with C-code. Please, correct me if I'm wrong. The clue here is that I want a database that supports writing to database from C-code and reading from database from JavaScript. The database can be read either from file:// schema or an IP address.
Would appreciate any help regarding this problem. Thanks!
Unfortunately, SQLite internal file format is complicated to fast parsing with JavaScript. One of the reasons, that the only browser side library which can read it is SQL.js and it is relatively slow, because it can not read data by selected pages from the database file, but only the whole file.
One of the options: you can switch from SQLite format to CSV or TSV plain tet formats, which can be eaily and quickly send to the browser and be parsed with jQuery.CSV, PapaParse, AlaSQL or any other CSV parsing libraries, like:
alasql('SELECT * FROM TSV("mydata.txt",{headers:true})',[],function(data){
// data
});
Another alternative: you can write simple server on node.js with native support of SQLite and then provide requested records in JSON format with ajax to your application (like in this article)
I can't really match an answer to this issue so I've decided to post this question hoping to find a solution.
I would like to use preferably JQuery / javascript or even PHP to find the most recent JSON files in a specified directory.
Why I would like to do this? Because when a user of my html5/javascript webapplication saves some array of objects to a JSON file, then besides the original work JSON file, there is a backup file that is created with a random name and which is the exact copy of the original JSON file.
If something happens to the original JSON file I would like the user to be able to open the most recent backup files from the backups directoy and select the right one to be recovered.
To open a JSON file I usually use this code:
$.getJSON('main/backups/random1345004.json', function(info){ ... });
Now the trouble is that in case of a backup I don't know the name of the JSON file that should be opened, because each file is unic and has a Math.random() generated name when it's created.
So I repeat the question: Is there a way to open from the backups directory the most recently created, randomly named, JSON files?
If not I might try to use .getTime() javascript method instead of Math.random() to control the names of the backup files created and then use a loop to search for a valid backup file name. That's a hunch, but I would not like to make anything stupid if there is a better solution, without loops.
Security is not a concern for me at this rate.
Thank you for any help provided!
If your server supports WebDav or FTP on your main\backups folder then you could search for all files greater today and then select the more recent.
--- Addition ---
With PHP, take a look at sort files by date in PHP
You could replace your $.getJSON() call with a standard ajax request:
<script>
$.ajax({
url : "getMostRecentBackup.php",
datatype : "json"
})
.done(function(data){
console.log( data.toSource() );
})
.fail(function() {
alert( "error" );
});
</script>
getMostRecentBackup.php will read backup directory and return a JSON object containing the most recent backup file, please read this Topic: sort files by date in PHP
I am trying to build this application that when provided a .txt file filled with isbn numbers will visit the isbn.nu page for that isbn number by simply appending the isbn to the url www.isbn.nu/your isbn number.
After pulling up the page, I want to scan it for information about the book, and store that in an excel file.
I was thinking about creating a file stream of the url in Java, but I am not really sure how to extract the information from the html page. Storing the information will be done using the JExcel Java package.
My best guess would be using javascript to extract the information, but I don't know how to call the javascript from my java program.
Is my idea plausible? if not, what do you guys suggest I do.
my goal: retrieve information from an html page and store it in an excel file for each ISBN in a text file. There can be any number of isbn's in a text file.
This isn't homework btw, I am simply doing this for an organization that donates books to Sudan. Currently they have 5 people cataloging these books manually and I am one of them.
Jsoup is a useful tool for parsing a web page and getting data from it. You can do it in Java and it's pretty easy.
You can parse the text file, build the URL with a string, send it in with JSoup then use JSoup to parse out the information using the html tags on the page. Then you can store it out however you want. You really don't need to use Javascript at all if you're more comfortable with Java.
Example for reading a page and parsing it with Jsoup:
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
Elements newsHeadlines = doc.select("#mp-itn b a");
Use a div in which you load your link (example here how to do that http://api.jquery.com/load/).
After that when load is complete you can check what is the name of the div's or spans used in the webpage and get that content with val (http://api.jquery.com/val/) or text (http://api.jquery.com/text/)
Here is text from the main page of www.isbn.nu:
Please note that isbn.nu is designed for manual searching by individuals. It is not intended as an information resource for automated retrieval, nor as a research tool for companies. isbn.nu reserves the right to deny access based on excessive requests.
Why not just use the free Google books API that would return book details in XML format. There are many classes available in Java to parse XML feeds and would make your life much easier.
See http://code.google.com/apis/books/ for more info.
Here are the steps needed:
Create CURL request (you can use multiple curl requests)
Get body data
Parse data
Make excel file
You can read HTML information using this guide.
A simple solution might be to use a Google Docs spreadsheet function like ImportXML(URL,path-expression).
More information and examples here:
http://www.seerinteractive.com/blog/importxml-cookbook/
http://www.distilled.net/blog/distilled/guide-to-google-docs-importxml/
http://blog.ouseful.info/2008/10/14/data-scraping-wikipedia-with-google-spreadsheets/