Best method for clearing broken images from a database. I have 30,000+ image entries in my mongodb database however many have become broken over time. I want to only return images that aren't broken on the server side. What would be the best method to find all the broken images and add a parameter or delete the broken images from the database?
Currently I am returning all the images to the frontend and doing and hiding on the 'onerror' attribute. Should I export and iterate over all the images somehow and get an export of the image ids? Are there better options or what can I use to do this? Just looking for advice.
You could use Blaze template event handlers to catch image load errors, then flag the image as failed, or just delete it. Self cleaning and always running...
Template.ImageLoader.events({
//TODO: possibly add an extra class name on the img tag, to make sure you only catch errors from your collection.
'error img': function(event, template) {
const url = event.currentTarget.src;
//TODO: find the image in your collection, using the url
//TODO: perform cleanup (flag or delete)
alert(`error loading image ${url}`);
}
});
I would suggest adding an error counter to your collection and $inc it each time a loading error occurs. Once the load error count reaches a certain number, delete it.
Related
Basically, I would like to create some optimistic response loader for the images while they are uploading to render the optimistic images and the loader.
So I have this idea: I use Apollo optimistic response to add a base64 image into the cache to render it while the image is uploading but I also want to add a field for this image type like: imageLoading and I could use it in my UI to show some loader, etc. and remove this loader after the real image will be uploaded.
I know the only way to create a custom field in apollo cache: Field policy -> read functin
But the main problem is: how to let the apollo cache know when I should set the imageLoading field as true or false?
Probably it's possible to use the reactiveVar for this imageLoading marker and later in the Field Policy -> read function I should somehow get this value and based on its return true/false for this cache field.
So far I know I could get this value directly from the reactive var in the read function of the field policy by this way:
const cache = new InMemoryCache({
typePolicies: {
Image: {
fields: {
name: {
imageLoading (name) {
const loading = reactiveVarIsLoading();
return loading;
}
}
},
},
},
});
But I'm not sure if it's the correct way to get the reactive var value directly in the cache.
So I'm wondering what is the best way to achieve that or if I'm moving in the totally wrong direction to achieve my goal?
Thanks for any help!
Is there any way to validate or limit what file-type/ extensions could be uploaded before it actually gets uploaded?
I've tried using a couple of custom-hooks but they all just went through.
Hooks tried:
files.create.before
items.create.before
module.exports = function registerHook({ exceptions }) {
const { InvalidPayloadException } = exceptions;
return {
"files.create.before": async function (input) {
console.log(input);
throw new InvalidPayloadException("title");
},
};
};
If you scroll down past either Filter, or Action Events, you will see a small block listing all system collection names where files cannot be intercepted during create/update.
Maybe they're trying to keep it as a file manager which can store every types, then bind it through pivot table.
You could try creating a custom user interface where you limited the choice of uploading file extensions yourself (Not sure if it works).
enter image description here
I'm trying to set up a history command for when a player gets warned with a moderation bot. I have it where it saves it in a JSON file however I don't know how to save multiple cases for one player. Right now it just replaces the last warn.
//--Adds To Logs
let warnhist = JSON.parse(fs.readFileSync("./historywarn.json", "utf8"));
warnhist[wUser.id] = {
Case: `${wUser} was warned by ${message.author} for ${reason}`
};
fs.writeFile("./historywarn.json", JSON.stringify(warnhist), (err) => {
if (err) console.log(err)
});
It saves like this without adding onto it every time:
{"407104392647409664":{"Case":"<#407104392647409664> was warned by <#212770377833775104> for 2nd Warn"}}
I need it to save like this:
{
"407104392647409664":{"Case":"<#407104392647409664> was warned by <#212770377833775104> for 1st Warn", "Case 2":"<#407104392647409664> was warned by <#212770377833775104> for 2nd Warn" }
}
You want an array instead of an object. Try structuring your data a bit more too, so your JSON looks like this:
{ "<user id>": { "warnings": [...] }
Then instead of overriding the history every time with an assignment warnhist[wUser.id] = ... you can use Array.prototype.push to add a new one.
Taking a step back though, I don't think JSON is a good strategy to store this data, especially if it contains all warnings for all users. You need to load the entire JSON and read the file and write the file every single time you add a warning, and there could be conflicts if you have multiple instances of your service doing the same for a large number of users.
For a super similar API and storage format but better performance and consistency, try MongoDB instead of a plain JSON file.
I'm uploading a File with Angular to my Node server. Sometimes the image is not uploaded and I get req.files "undefined" in the server side and I can not figure it out why. I think it has not to do with the weight of the image: the image that is uploaded is around 48KB and the one that isn't 16KB. I attach two snapshot to show the difference in timing (http post) between the one that works and the other (network tab of Chrome developer tool). I can see that the "times" ("blocking", "sending") are not overlapping themselves in the one that is failing. Maybe is that the problem... but I'm not sure about how to fix it.
I got it.
I was using express.multipart in the server side with defer option "express.multipart({defer: true})"
Because of that, sometimes the execution reached the code where I manage the post request to get the image with "req.files.file" before form's "end" event in express.multipart module is fired where the module sets req.files:
form.on('end', function(){
if (done) return;
try {
req.body = qs.parse(data);
**req.files = qs.parse(files);**
if (!options.defer) next();
} catch (err) {
form.emit('error', err);
}
});
So sometimes req.files.file = undefined, simply because is not set yet...
Getting rid of {defer: true} option solves my problem. Nothing to do with one or another image.
Here you are the doc of "defers" option in express.multipart module:
"defers processing and exposes the Formidable form object as req.form.
* next() is called without waiting for the form's "end" event.
* This option is useful if you need to bind to the "progress" event, for example."
I've been trying to work with images in mongodb for the past while now. Every where I keep looking just says use GridFS due to the 16mb size restraint per document. However, each document Id like to store is <16mb. I currently stored it like this
var testChamp = new Champion ({
name: "Aatrox",
img: fs.readFileSync("D:/CounterMe/Aatrox_0.jpg"),
contentType: "jpg"
});
testChamp.save(function() {
console.log("Saved");
})
In my schema:
name: String,
img: Buffer,
contentType: String `
Finding this information in the mongoshell brings a HUGE amount of lines in cmd, so Im not sure if im even storing it correctly. If I am, I have no clue how to insert it into my html. I know how to retrieve the name, pass it in as an object:
app.get("/", function(req, response) {
Champion.find(function(err, champs) {
response.render("index", {
champ: champs[0]
});
});
});
However, once I retrieve champs[0].img, im not sure what to do with it. How am I supposed to pass the img into my HTML? Any help would be appreciated, I am very stumped currently.
You really shouldn't be trying to render in HTML. I've seen approaches and they are all generally bad, and for one good reason.
The browser implements a cache, and as such any "images" it has retrieved before will be held in that cache. But if you are somehow attempting to "embed" that image data ( via Base64 encoding for example ) you are basically throwing away that caching capability and causing that data to be sent with every request.
If you must store images in MongoDB in this way then you should have a seperate endpoint in your API which would return what would be basically just an image. So make all requests "look like" they are just fetching an image from the web server.
First change your data a little to reflect what would be a "filename":
var testChamp = new Champion ({
name: "aatrox.jpg",
img: fs.readFileSync("D:/CounterMe/Aatrox_0.jpg"),
contentType: "image/jpg"
});
Then you would have controller code like this:
app.get("/images/:image", function(req,res) {
Champion.findOne({ "name": req.param.image },function(err,champ) {
res.set("Content-Type", champ.contentType);
res.send( champ.img );
});
});
In your templates you then just refer to the URL of the image as you ordinarily would. You can add to that and possibly even store things like content length and set that in the response header as well. There are modules that you can use to detect mime-type and size from images uploaded through another API point prior to storing them as well. Have a look around for these.
This makes sense and is much more efficient than otherwise sending all of the image data within the HTML.
I asked this question when I was first getting into web development. Coming back to it 5 years later, my approach was definitely wrong. I was trying to store my images themselves in mongodb, when I should have been storing them in a storage service like S3, then just storing the urls to said images in mongodb. Hopefully this helps someone who comes across this!