Meteor-package: Import JSON-array to mongoDB - javascript

I'm trying to create a meteor-package to import JSON-files to collections in a mongoDB. But I'm not quite sure, if this is possible.
So I want the user to upload a JSON-file. In a input-field the user also types the collection-name which has to be used for the import. After this the JSON-array should be saved to the given collection.
HTML:
<template name="importJSON">
<form id="importJson">
<input type="text" id="collection">
<input type="file" id="file">
</form>
</template>
meteor:
Template.importJSON.events({
'submit #importJson': function(e){
e.preventDefault();
var collection = e.target.collection.value;
var obj = JSON.parse(response);
db.collection.insert(obj);
}
});
So I have three problems with that:
1) How do I have to do the upload itself, as the file should be uploaded temporarily
2) How can I use the collection name given in the input-field?
3) How do I import the data in a correct way? Insert would just append the new data to the existing data, wouldn't it?


So to answer your three problems:
1) How do I have to do the upload itself, as the file should be uploaded temporarily
If all you need to do is read the file, then insert it into a collection. In that case you do not need to even upload the file. Just read the file in client side. Here is an article on Reading files in JavaScript using the File APIs.
2) How can I use the collection name given in the input-field?
Say the collection name given in the input-field is products and assume you have a sample data file like this:
{
"name": "Product",
"id": "Product identifier",
"name": "Name of the product",
"price": "9990",
"tags": ["tag1", "tag2"]
}
At this point you need to decide how you do this. If you already have a Products collection on server side.
<your app name>/lib/collections/products.js
Products = new Meteor.Collection('Products');
Then in your client side code:
var rawproducts = (content of the file you read using the File API -- as mentioned above)
var newproducts = JSON.parse(rawproducts);
for (var i = 0; i < newproducts.length; i++) {
Products.insert(newproducts[i]);
}
You can also test this out locally by creating a local only collection.
//pass null as collection name, it will create
//local only collection
Products = new Mongo.Collection(null);
for (var i = 0; i < newproducts.length; i++) {
Products.insert(newproducts[i]);
}
Note: If you use a local only collection your data is still on the client. You'll need to sync that with the server (as described above) for persistence.
3) How do I import the data in a correct way? Insert would just append the new data to the existing data, wouldn't it?
Importing the data as shown above will keep on appending the data. You may want to think about how to de-dupe the data or completely overwrite the existing data.
Hope this helps.
Update:
How to delete all the elements from a collection?
Products.remove({}) // remove every product

Related

get only specific element of a JSON database stored in server by url request

my website relies on a database which is a big JSON file like this:
var myjsonData =
[ {
"ID": 0,
"name": "Henry",
"surname": "McLarry",
"...": "...",
}]
I do generate this data every month at high cost to me, therefore I would like to avoid calling it straight in my html <head>, because this will allow any user to download the full database in no time.
I would like to build a "something" that can only call specific items from the json file (just the only one I want to show) without "exposing" the full .json onto client side.
today I use the call
var myvar= myjsonData.ID.Name
to get "Henry" into myvar, I would like to build something like
var myvar = mycallfunction(ID,Name)
I did try with PHP as intermediary but the ajax calls from javacript doesn't allow me to fetch the data.
Can I use JQuery with the JSON Url to get only the item I need?

What you can do is parse your json for an object. So you can get any value you want from json.
Example:
var myjsonData = '{"ID": 0,"name": "Henry","surname": "McLarry"}';
obj = JSON.parse(myjsonData);
console.log(myjsonData.ID); //print the id
console.log(myjsonData.name); //print the name
console.log(myjsonData.surname); //print the surname

So you have a NoSQL Database which has only one kind of Document that is the full JSON element you use in your website. In that scenario you have three options:
Depending on the NoSQL Database you're using you can limit the fields which will be returned(I.e: For MongoDB you can look here: https://docs.mongodb.com/manual/tutorial/project-fields-from-query-results/)
Change the way you store you data into more modular documents and make the logic to connect them in you application. So instead of one big document you'll have modular ones as Users, Products, Transactions and etc and you can use your application to query them individually.
Build a Server Side logic as an API to deal with your data and provide only what you need, so the API(Which can be node.js, php, or any you may like) will get the full JSON it`s endpoints will only the data you want. For example: myapi.com/getUser, myapi.com/getProducts and so on.
If you're able to provide more info on the technologies you're using that would help us. Hope that helped :).

Copy data from a dynamic website using scrapy

I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values ​​are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?

You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions

Print data from json file

I have json file with given data (total.json)
var data = {"trololo":{"info":"61511","path".... }}
I need to get object "info" and then print data "61511" in alert window
I include my json like
var FILE = 'total'
var data_file_names = {};
data_file_names[FILE] = 'total.json';
And then i use it like
var data_trololo = data_file_names[FILE];
Plese, help me print object "info". Maybe there is another way to solve this problem

You need to make an ajax call to the json file. Then you can access the array like the below example.
Note : Your json wasn't properly formatted.
var data = {
"trololo":{
"info": ["61511","path"]
}
};
console.log(data.trololo.info[0]); //this one will print 61511

Usually one can make an ajax call to read the file on the server.
But if you are ok with using HTML5 features then go through the link find out how to read the file on the browser itself. Though File API being part of HTML5 spec is stable across browsers.
http://www.html5rocks.com/en/tutorials/file/dndfiles/

Convert a Json Object to a file Object

I have some java-script functions. I am using input type file for selecting multiple files. In the on-change event I am getting a list files objects which were uploaded and then I am storing those objects in a hidden field. I want to retrieve those objects as file objects only. How can i accomplish that?
The code I am using is as follows:-
<input type="file" multiple onchange="ehDisplayFileNames(event);" />
function ehDisplayFileNames(event) {
myFilesList = $('#' + event.target.id)[0]; // $('#chooseFiles')[0] gives the DOM element of the HTML Element as opposed to $('#store-input') which gives the jQuery object
$("#FilesToUpload").val(JSON.stringify(filesToUpload));
}
function getAllFiles(){
var filesToUpload = [];
filesToUpload = $.parseJSON($("#FilesToUpload").val()); // returns json objects instead i need file objects
}
Thank you for the help.

I was looking for a similar thing.. I ended up using the solution provided here as there is no way to serialize API object... https://stackoverflow.com/a/20535454/606810
Here is another source i found useful for image/file storage: https://hacks.mozilla.org/2012/02/saving-images-and-files-in-localstorage/
Hope it helps!

whats a file Object? You mean JSON string to JS Object?
If yes just use:
var myObject = jQuery.parseJSON( jsonString ) ;
and it will be a object
REWRITE::
<input type="file" multiple onchange="ehDisplayFileNames(event);" />
add ID to it
<input type="file" id="myFile" multiple onchange="ehDisplayFileNames(event);" />
then you have your object:
$("#myFile")

Here is the solution you are looking for. :-)
I assume what you are ultimately trying to do is to store the values (of which there may be multiples). In addition, you may be trying to display them to the user instead of showing the default text that says "4 files selected" for example. And also, as you had indicated, you want to be able to programmatically manipulate the list of selected files.
The problem is that can't be done in the way you are approaching it. That's because the <input type="file" /> element is protected, and the full file path will never be accessible. Therefore, when you attempt to grab the values, they would only be the filenames themselves but not the full file path. For example, if you selected a file called dog.jpg from your desktop, the browser knows the path is "file:\c:\windows\users\someuser\desktop\dog.jpg" but when you programmatically try to retrieve it - you will only get the value "dog.jpg". With just the filename, you can not reconstruct the full file paths and so you can't do anything meaningful with them in regards to eventually sending them to be processed by the server.
Therefore, I have crafted a solution for you which uses a single file upload button. When a user selects a file, it will add another as needed. In addition, when there are more than one files selected, the user can choose to remove any of them as needed.
You would process the selected files on your server in the following way... When the form data is sent to the form action (in this case "/sendfiles"), your server side code would process the incoming data for each form element. In the example below, the input elements of type files (having the name attribute "myFiles") would be sent as an array, where each array element would contain the file data of each one that was specified by the user.
jQuery code:
var getUploadFragment = function(){return $('<div class="files-cont"><input type="file" name="myfiles" class="files" /></div>')};
$(document).ready(function(){
$("#my_form")
.append(getUploadFragment())
.on("change", "input[name=myfiles]:last", function(){
var lastFileContainer = $(".files-cont:last");
lastFileContainer.after(getUploadFragment());
if($(".files-cont").length > 1){
lastFileContainer.append('[Remove]');
}
})
.on("click", ".remove", function(){
if($(".files-cont").length > 1){
$(this).parent().remove();
}else{
$(this).remove();
}
});
});
with the following form defined in the body:
<form id="my_form" action="/sendfiles" method="post"></form>

How do you get all list information from Django view in Javascript?

I am trying to make a small web app using django and javascript however I have ran into a problem.
In my model I have an entity name "entry" which has attributes id, name, location_lat and location_lon all as char arrays. I pass entrys associated to a certain sport to a "map.html" file using the following code:
def sportpage(request,sportname):
t = get_template('map.html')
try:
entry_list = Feature.objects.filter(sport=sportname)
except BaseException:
raise Http404
sport_list = Sport.objects.all().order_by('name')
context = RequestContext(request,{'sport_list':sport_list,'entry_list':entry_list, 'page_title':sportname})
return HttpResponse(t.render(context))
and in the JavaScript file I attempt to read "entry_list" in using:
var data = "{{entry_list|safe}}";
However the data only contains a list of the entry names but not the location_lat or location_lon attributes. How would I gain access to these?
Thank you very much for any help

Categories