I'm am developing a mobile application with cordova. The applications needs in some point to be off-line which means that it has to have data in an internal database.
The data consists on a bunch of fields some of them are links to files. For example:
var restaurant = {
name: 'Restaurant X'
address: 'So far'
logo: http://www.restaurantX.com/logo.png
image: pics/restaurant-x.jpg
}
When the user decides to be off-line we should extract all the relevant information from the database (This point is cleared) and download all the related images (This is where I am stuck). At this point I thought in two options the first one is download all the files to the mobile as files or as data in the database, however, the project chief said that It was mandatory that images were saved as blob on the database.
I've been looking for a function to convert the content of the file into a byte Array for saving it into the database but I need a syncronous function since I have to wait for the pictures before of saving the row in the database.
function(tx, row){
var query = "INSERT INTO Restaurants (name, address, logo, image) values (?,?,?,?)";
var logo = null; // I need to fill it with the logo
var picture = null; // I need to fill it with the picture
tx.executeSql(query, [row["nombre"], row["address"],logo,picture], onSuccess, onError);
});
I'm sorry if the question is too basis, I'm practically new at JavaScript.
You should definitely take a look at promises, if your image serialization routine is async and you want to store it on serialize success. More detailed info about promises you can check here
Related
I have a profile to keep data from the user, and he can edit when he want. To keep the session, I pass some info from the user in localStorage, because in this project we don't use a library like redux, and one of these fields is "picture" to represent the photo in user's profile.
The problem consists when I try to "save" the image in the localStorage, when I upload the input and save it, it's ok, the picture saves in the database, and in localStorage that link (where the picture is hosted in the db) is correct, but if I try to edit some other field, like "name" for example and I click to apply the changes, the picture should be the same that I save previously, but the result is a file object and the url is not found in the database. It's like that object file replace that field in the localStorage.
Here I detach some lines of the code to guide me a little.
When I have a succesfull login, I saved the picture from data supplied from the api. PictureURL concat the url from the api with the picture URL that I received from the call.
------------
localStorage.setItem("picture", pictureURL);
------------
In my profile component, I have a state that initially load all the data from the localStorage, picture included. This state I'll use later to send the user to the api
const [userState, setUserState] = useState({
--------------------------------
picture: localStorage.getItem("picture"),
------------------
});
Then in my component, I have an input to upload the new image that I want to change
<Input
type={"file"}
size={"sm"}
mt={"10%"}
ml={25}
accept="image/*"
onChange={imageChange}
></Input>
imageChange is a function that consists in read the file, with FileReader, store the result in another state call imgData (that functionality is to show a preview of that image), and then set the picture in the main state userState that I mentioned before
const imageChange = (e) => {
const reader = new FileReader();
reader.addEventListener("load", () => {
setImgData(reader.result);
});
reader.readAsDataURL(e.target.files[0]);
setUserState({ ...userState, picture: e.target.files[0] });
};
And finally, I have a handler to pass all the data from the userState to a formData and send to the api, and when it's all ok, set the localStorage fields with the new data
---------------------------
localStorage.setItem("picture", userState.picture);
---------------------------
Sorry if it's too long, but I don't know where I have the error, maybe when I manage the picture at the beginning, or it's a problem in the backend, but idk! I have 3 days with this problem and I can't see what's going on.
First of all, thanks a lot for the help. All the critics to become more "clean" and to grow up are welcome :)
I have a bulk amount of data in CSV format. I am able to upload that data with python by converting them to the dictionary (dict) with loop. the whole data is getting uploaded.
but now I want to upload bulk data to firebase and images to storage and I want to link between each document and image because i am working on e-commerce react app. so that I can retrieve documents along with images.
which is a good way to do this? should I do this with javascript or python?
I uploaded data manually to firebase by importing from there but again I am unable to upload bulk images to storage and also unable to create references between them. please give me a source where I can find this solution
This is tough, because it's hard to fully understand how exactly your images and CSV's are linked, but generally if you need to link something to items stored in Firebase, you can get a link either manually (go into storage, click and item, and the 'Name' Field on the right hand side is a link), or you can get it when you upload it. So for example, I have my images stored in firebase, and a postgres database with a table storing the locations. In my API (Express), when I post the image to blob storage, I create the URL of the item, and post that as an entry in my table, as well as setting it to be the blobs name. I'll put the code here, but obviously it's a completely different architecture to your problem, so i'll try and highlight the important bits (it's also JS, not Python, sorry!):
const uploadFile = async () => {
var filename = "" + v4.v4() + ".png"; //uses the uuid library to create a unique value
const options = {
destination: filename,
resumable: true,
validation: "crc32c",
metadata: {
metadata: {
firebaseStorageDownloadTokens: v4.v4(),
},
},
};
storage
.bucket(bucketName)
.upload(localFilename, options, function (err, file) {});
pg.connect(connectionString, function (err, client, done) {
client.query(
`INSERT INTO table (image_location) VALUES ('${filename}')`, //inserts the filename we set earlier into the postgres table
function (err, result) {
done();
if (err) return console.error(err);
console.log(result.rows.length);
}
);
});
console.log(`${filename} uploaded to ${bucketName}.`);
};
Once you have a reference between the two like this, you can just get the one in the table first, then use that to pull in the other one using the location you have stored.
I'm making a web scraper Node.js app that harvests job description text from various urls.. I currently have an array of job objects named jobObj and the code cycles through each url, makes a request for html, loads using cheerio module then finally makes a new object with jobName and jobDesc keys and pushes it onto a new array of objects that is then written as a json file....
All this currently works however the completeness of the written json file is very random and usually only contains one complete job account. I thought this may be due to the forEach loop completing much quicker than the asynchronous Request function thus resulting in execution of the fs.writefile before request callback is completed. I've added a counter to monitor at what stage the requests are at and only write the json file once counter===jobObj.length but still the json file is not fully complete.
I'm new to node.js if someone could please point out my error it would be greatly appreciated!
var jobObj = [
{
id:1,
url:"https://www.indeed.co.uk/cmp/Daffodil-IT/jobs/Lead-Junior-Website-Developer-59ea7d446bdf1253?q=Junior+Web+Developer&vjs=3",
},
{
id:2,
url:"https://www.indeed.co.uk/cmp/Crush-Design/jobs/Middleweight-Web-Developer-541331b7885c03cf?q=Web+Developer&vjs=3",
},
{
id:3,
url:"https://www.indeed.co.uk/cmp/Monigold-Solutions/jobs/Graduate-Web-Software-Engineer-a5787dc322c0ca36?q=Web+Developer&vjs=3",
},
{
id:4,
url:"https://www.indeed.co.uk/cmp/ZOO-DIGITAL-GROUP-PLC/jobs/Web-Developer-5cdde1c3b0b7b8d0?q=Web+Developer&vjs=3",
},
{
id:5,
url:"https://www.indeed.co.uk/viewjob?jk=9cc3d8c637c41067&q=Web+Developer&l=Sheffield&tk=1cf5di52e9u0ocam&from=web&vjs=3",
}
];
app.get('/myform', function(req, res){
res.send("<h1>" + `scanning ${jobObj.length} urls for job description text` + "</h1>");
//make assign input form data to node "url" variable
//Compnonents for a request counter
var jobs = new Array;
function scrapeFinished(){console.log("all websites scraped!");};
var itemsProcessed = 0;
jobObj.forEach(function(item){
request(item.url, function(err, res, html){
if(!err){
var $ = cheerio.load(html);
var newJob = new Object;
$('#job_summary').each(function(){
var data = $(this);
var textout = data.text();
newJob.jobDesc = textout;
});
$('.jobtitle').each(function(){
var data = $(this);
var jobtitle = data.text();
newJob.jobName = jobtitle;
});
jobs.push(newJob);
itemsProcessed++;
console.log(item.url + " scraped");
if(itemsProcessed === jobObj.length){
scrapeFinished();
fs.writeFile('output.json', JSON.stringify(jobs, null, "\t"), function(err){
if(!err){console.log("output.json file written")}
})
}
}
})
})
})
And finally this is what I get on fs.writefile
[
{},
{
"jobDesc": "We are a successful design and digital agency that works with some great clients on a wide range of digital projects.We simply need more developers to join our great team to deliver even more great work.The projects we work on are all php based, typically built using WordPress, Laravel or flat html.We are seen as a premium agency because of the quality and complexity of the work we do.That means you will have to do more that just manipulate a theme - you will have to code. But you will be given the space, time and support to do so.We want you to be proud of the work you do, because the reputation of the agency need you to be.Key skills we will want you to bringCSS (CSS3) & HTMLAt least some knowledge of MySQL and JavaScript.At least some knowledge PHP (seniors will be tested)PhotoshopWhat you will want that we can giveA good place to work with a friendly teamA chance to develop your coding craftA decent range of projects to challenge yourselfA senior developer on hand to coach and adviseA successful company with an optimistic outlook, growth plans and a secure futureExactly how much experience you have can vary, but you must have some. And the more experience you have, the more we will pay you.We are based in offices we own in the centre of Chesterfield will two staff that do the short commute from Sheffield.If you think this job sounds interesting, we would love to hear from you, please apply!(though not agencies please)Job Type: Full-timeSalary: £22,000.00 to £30,000.00 /yearExperience:development: 2 years",
"jobName": "Middleweight web developer"
},
{},
{},
{}
]
forEach run in synchronous fashion so no matter how much time it will get to scrap the webpage, it won't run for the next array item. What can cause trouble here is your code maybe pushing an object in the array before scrapping the webpage. This is why you are getting an empty object. What you do is to push object after your program finishes visiting the URL.
Do something like this
jobObj.forEach(function (item) {
var newJob = new Object;
request(item.url, function (err, res, html) {
// Scrap URL and save values in newJob
});
jobs.push(newJob);
});
if your code still pushes an empty object before completing the request then consider using Async module.
So I'm building a music platform, using firebase, where users can upload their songs. When a user logs in, I save their data on the "usuarios" node. And when they upload a song, I push to the "musicas" node (child of usuarios).
This way, I know who uploaded the song.
But the problem is: I want to get a list of all the last 15 songs uploaded. How do I do that? Did I structure my data wrong?
Here's what I tried:
var musicsRef = firebase.database().ref('usuarios').orderByChild('musicas');
musicsRef.on('value', function(snapshot) {
console.log(snapshot.child('musicas');
});
Here's my database structured:
With Firebase (like with most NoSQL databases) you often end up modeling your data for the way you app wants to consume it. So if you need to find a global list of the most recently uploaded songs, you should keep such a list:
songsByDate
$pushIdOfSong: true
You'll note that we don't keep a real value in this list. It is just the ID of each song and a true value. This is called a secondary index and so command that we explain it in our documentation on creating a scalable data structure.
With that structure in place you can query with:
var recents = ref.child('songsByDate').orderByKey().limitToLast(15);
recents.on('child_added', function(snapshot) {
var key = snapshot.key;
// load the song by its key
});
I highly recommend reading this article on NoSQL data modeling.
I've been trying to work with images in mongodb for the past while now. Every where I keep looking just says use GridFS due to the 16mb size restraint per document. However, each document Id like to store is <16mb. I currently stored it like this
var testChamp = new Champion ({
name: "Aatrox",
img: fs.readFileSync("D:/CounterMe/Aatrox_0.jpg"),
contentType: "jpg"
});
testChamp.save(function() {
console.log("Saved");
})
In my schema:
name: String,
img: Buffer,
contentType: String `
Finding this information in the mongoshell brings a HUGE amount of lines in cmd, so Im not sure if im even storing it correctly. If I am, I have no clue how to insert it into my html. I know how to retrieve the name, pass it in as an object:
app.get("/", function(req, response) {
Champion.find(function(err, champs) {
response.render("index", {
champ: champs[0]
});
});
});
However, once I retrieve champs[0].img, im not sure what to do with it. How am I supposed to pass the img into my HTML? Any help would be appreciated, I am very stumped currently.
You really shouldn't be trying to render in HTML. I've seen approaches and they are all generally bad, and for one good reason.
The browser implements a cache, and as such any "images" it has retrieved before will be held in that cache. But if you are somehow attempting to "embed" that image data ( via Base64 encoding for example ) you are basically throwing away that caching capability and causing that data to be sent with every request.
If you must store images in MongoDB in this way then you should have a seperate endpoint in your API which would return what would be basically just an image. So make all requests "look like" they are just fetching an image from the web server.
First change your data a little to reflect what would be a "filename":
var testChamp = new Champion ({
name: "aatrox.jpg",
img: fs.readFileSync("D:/CounterMe/Aatrox_0.jpg"),
contentType: "image/jpg"
});
Then you would have controller code like this:
app.get("/images/:image", function(req,res) {
Champion.findOne({ "name": req.param.image },function(err,champ) {
res.set("Content-Type", champ.contentType);
res.send( champ.img );
});
});
In your templates you then just refer to the URL of the image as you ordinarily would. You can add to that and possibly even store things like content length and set that in the response header as well. There are modules that you can use to detect mime-type and size from images uploaded through another API point prior to storing them as well. Have a look around for these.
This makes sense and is much more efficient than otherwise sending all of the image data within the HTML.
I asked this question when I was first getting into web development. Coming back to it 5 years later, my approach was definitely wrong. I was trying to store my images themselves in mongodb, when I should have been storing them in a storage service like S3, then just storing the urls to said images in mongodb. Hopefully this helps someone who comes across this!