TL;DR: I want a PouchDB db that acts like Ember Data: fetch from the local store first, and if not found, go to the remote. Replicate only that document in both cases.
I have a single document type called Post in my PouchDB/CouchDB servers. I want PouchDB to look at the local store, and if it has the document, return the document and start replicating. If not, go to the remote CouchDB server, fetch the document, store it in the local PouchDB instance, then start replicating only that document. I don't want to replicate the entire DB in this case, only things the user has already fetched.
I could achieve it by writing something like this:
var local = new PouchDB('local');
var remote = new PouchDB('http://localhost:5984/posts');
function getDocument(id) {
return local.get(id).catch(function(err) {
if (err.status === 404) {
return remote.get(id).then(function(doc) {
return local.put(id);
});
}
throw error;
});
}
This doesn't handle the replication issue either, but it's the general direction of what I want to do.
I can write this code myself I guess, but I'm wondering if there's some built-in way to do this.
Unfortunately what you describe doesn't quite exist (at least as a built-in function). You can definitely fall back from local to remote using the code above (which is perfect BTW :)), but local.put() will give you problems, because the local doc will end up with a different _rev than the remote doc, which could mess with replication later on down the line (it would be interpreted as a conflict).
You should be able to use {revs: true} to fetch the doc with its revision history, then insert with {new_edits: false} to properly replicate the missing doc, while preserving revision history (this is what the replicator does under the hood). That would look like this:
var local = new PouchDB('local');
var remote = new PouchDB('http://localhost:5984/posts');
function getDocument(id) {
return local.get(id).catch(function(err) {
if (err.status === 404) {
// revs: true gives us the critical "_revisions" object,
// which contains the revision history metadata
return remote.get(id, {revs: true}).then(function(doc) {
// new_edits: false inserts the doc while preserving revision
// history, which is equivalent to what replication does
return local.bulkDocs([doc], {new_edits: false});
}).then(function () {
return local.get(id); // finally, return the doc to the user
});
}
throw error;
});
}
That should work! Let me know if that helps.
Related
Thanks for looking at my question. It should be easy for anyone who has used Meteor in production, I am still at the learning stage.
So my meteor setup is I have a bunch of documents with ownedBy _id's reflecting which user owns each document (https://github.com/rgstephens/base/tree/extendDoc is the full github, note that it is the extendDoc branch and not the master branch).
I now want to modify my API such that I can display the real name of each owner of the document. On the server side I can access this with Meteor.users.findOne({ownedBy}) but on the client side I have discovered that I cannot do this due to Meteor security protocols (a user doesnt have access to another user's data).
So I have two options:
somehow modify the result of what I am publishing to include the user's real name on the server side
somehow push the full user data to the clientside and do the mapping of the _id to the real names on the clientside
what is the best practice here? I have tried both and here are my results so far:
I have failed here. This is very 'Node' thinking I know. I can access user data on clientside but Meteor insists that my publications must return cursors and not JSON objects. How do I transform JSON objects into cursors or otherwise circumvent this publish restriction? Google is strangely silent on this topic.
Meteor.publish('documents.listAll', function docPub() {
let documents = Documents.find({}).fetch();
documents = documents.map((x) => {
const userobject = Meteor.users.findOne({ _id: x.ownedBy });
const x2 = x;
if (userobject) {
x2.userobject = userobject.profile;
}
return x2;
});
return documents; //this causes error due to not being a cursor
}
I have succeeded here but I suspect at the cost of a massive security hole. I simply modified my publish to be an array of cursors, as below:
Meteor.publish('documents.listAll', function docPub() {
return [Documents.find({}),
Meteor.users.find({}),
];
});
I would really like to do 1 because I sense there is a big security hole in 2, but please advise on how I should do it? thanks very much.
yes, you are right to not want to publish full user objects to the client. but you can certainly publish a subset of the full user object, using the "fields" on the options, which is the 2nd argument of find(). on my project, i created a "public profile" area on each user; that makes it easy to know what things about a user we can publish to other users.
there are several ways to approach getting this data to the client. you've already found one: returning multiple cursors from a publish.
in the example below, i'm returning all the documents, and a subset of all the user object who own those documents. this example assumes that the user's name, and whatever other info you decide is "public," is in a field called publicInfo that's part of the Meteor.user object:
Meteor.publish('documents.listAll', function() {
let documentCursor = Documents.find({});
let ownerIds = documentCursor.map(function(d) {
return d.ownedBy;
});
let uniqueOwnerIds = _.uniq(ownerIds);
let profileCursor = Meteor.users.find(
{
_id: {$in: uniqueOwnerIds}
},
{
fields: {publicInfo: 1}
});
return [documentCursor, profileCursor];
});
In the MeteorChef slack channel, #distalx responded thusly:
Hi, you are using fetch and fetch return all matching documents as an Array.
I think if you just use find - w/o fetch it will do it.
Meteor.publish('documents.listAll', function docPub() {
let cursor = Documents.find({});
let DocsWithUserObject = cursor.filter((doc) => {
const userobject = Meteor.users.findOne({ _id: doc.ownedBy });
if (userobject) {
doc.userobject = userobject.profile;
return doc
}
});
return DocsWithUserObject;
}
I am going to try this.
I am currently building an api application that checks the status and gets information of various types of dbs(i.e. Mongo, MySQL) using Sailsjs such as users, depending on a user input. Here is a snippet of the code I am working on. The local host is just the test database I am connecting to, but in the future it will be supplied by the user.
var mp = require('mongodb-promise');
var MongoClient = require('mongodb');
mp.MongoClient.connect("mongodb://#localhost:27017/test")
.then(function(db){
db.getUsers().then(function(users){
res.ok(users);
})
})
.fail(function(err) {
console.log(err);
})
I am attempting to use promises for the async issue. The problem I am having is that it doesn't work. It tells me that that Object[object object] has no method 'getUsers'. I have searched and can't seem to find a solution that works.
If I change the function to the below, I get the some data back.
mp.MongoClient.connect("mongodb://#localhost:27017/IMS")
.then(function(db){
db.stats().then(function(stats){
return res.ok(stats);
})
})
.fail(function(err) {
console.log(err);
dbObject.vipUp = false;
})
I am not sure what the issue is or how to solve it.
What you are doing here is using the node native driver methods to connect and inspect the database. There is in fact "no such method" as .getUsers() here in this API or in fact in any other API.
The .getUsers() function is just a "shell helper" that is basically implemented like this:
function (args) {
var cmdObj = {usersInfo: 1};
Object.extend(cmdObj, args);
var res = this.runCommand(cmdObj);
if (!res.ok) {
var authSchemaIncompatibleCode = 69;
if (res.code == authSchemaIncompatibleCode ||
(res.code == null && res.errmsg == "no such cmd: usersInfo")) {
// Working with 2.4 schema user data
return this.system.users.find({}).toArray();
}
throw Error(res.errmsg);
}
return res.users;
}
So what you should be able to see here is that this normally wraps a "command" form, or otherwise falls back for compatibility with MongoDB 2.4 to querying the system.users collection on the current database.
Therefore, instead of calling a method that does not exist, you then need to use the .command() method instead:
mp.MongoClient.connect("mongodb://#localhost:27017/test")
.then(function(db){
db.command({ "usersInfo": 1}).then(function(users){
res.ok(users);
})
})
.fail(function(err) {
console.log(err);
})
Or in the case of connecting to a MongoDB 2.4 instance, then fetch from the .collection():
mp.MongoClient.connect("mongodb://#localhost:27017/test")
.then(function(db){
db.collection('system.users').find().toArray().then(function(users){
res.ok(users);
})
})
.fail(function(err) {
console.log(err);
})
At any rate, you really should be establishing the database connection elsewhere in your application ( or re-using the underlying driver connection from another store ), and then calling methods on the connection already establihed. This is always preferable to creating a connection on the request of the information you want to retrieve.
Also, recent versions of the node native driver support promises right out of the box. So there may be no need to configure in anything else, depending on how you intend to use it.
I would like to know if it is possible to intercept and modify a post data when the url and some of the variables meet some pattern.
for example:
let the login url be: http://www.someonlineprofiles.com
let the post data be:
email: "myemail#gmail.com"
pass: "mypass"
theme: "skyblue"
I would like that if:
url = "http://www.someonlineprofiles.com/ajax/login_action_url" and
email = "myemail#gmail.com"
then theme value be unconditionally changed to: "hotdesert"
Is it possible to create a Firefox add-on for that?, are add-ons powerful enough for that?
I found this link:
modify the post data of a request in firefox extension
Thanks in advance!
[ADDED INFORMATION]
I don't know if it is interesting to know the version of my Firefox: 35.0.1
Your question borders on being too broad, so I will give only an overview on how to do this, but not a copy-paste-ready solution, which would take a while to create, and would also deny you a learning experience.
Observers
First of all, it is possible for add-ons to observe and manipulate HTTP(S) requests before the browser sends the request, you just need to implement and register what is called a http observer.
const {classes: Cc, instances: Ci, utils: Cu} = Components;
Cu.import("resource://gre/modules/Services.jsm"); // for Services
var httpRequestObserver = {
observe: function(channel, topic, data) {
if (topic != "http-on-modify-request") {
return;
}
if (!(channel instanceof Ci.nsIHttpChannel)) {
return; // Not actually a http channel
}
// See nsIChannel, nsIHttpChannel and nsIURI/nsIURL
if (channel.URI.host != "www.someonlineprofiles.com") {
return;
}
doSomething(channel);
},
register: function() {
Services.obs.addObserver(this, "http-on-modify-request", false);
},
unregister: function() {
Services.obs.removeObserver(this, "http-on-modify-request");
}
};
httpObserver.register();
// When your add-on is shut down, don't forget to call httpObserver.unregister();
Do only register the http observer once in your add-on:
If you're using the SDK, then put it into main.js or a dedicated module. You'll also need to rewrite the code a bit and replace the const .. = Components line with a require("chrome").
If you're writing a XUL overlay add-on, put it into a code module.
Rewriting post data
We still need to implement doSomething() and actually rewrite the post data. An http channel usually implements the nsIUploadStream interface, and the upload stream is where the current post data is, if any. It also has a setUploadStream() method, which you can use to replace the upload stream entirely.
function doSomething(channel) {
if (!(channel instanceof Ci.nsIUploadStream)) {
return;
}
// construct new post data
channel.setUploadStream(newStream);
}
Constructing the new post data would be a matter of your actual requirements. I provided a working example in another answer on how you could do it.
If you need to fetch some data from the old upload stream, you'll need to decode the existing channel.uploadStream as multipart/form-data yourself. I suggest you check TamperData and similar add-ons on how they do things there.
Scenario = I am slowly but surely wrapping my head around what is going on with Parse's cloud code features. I just need some help from those who would like to answer some short, relatively simple questions about what is going on in some sample cloud code functions.
The code I will use in this example is below
1) cloud code
Parse.Cloud.define('editUser', function(request, response) {
var userId = request.params.userId,
newColText = request.params.newColText;
var User = Parse.Object.extend('_User'),
user = new User({ objectId: userId });
user.set('new_col', newColText);
Parse.Cloud.useMasterKey();
user.save().then(function(user) {
response.success(user);
}, function(error) {
response.error(error)
});
});
2) called from iOS
[PFCloud callFunction:#"editUser" withParameters:#{
#"userId": #"someuseridhere",
#"newColText": #"new text!"
}];
This code was taken from here
Question 1 =
(request, response)
I am confused by what this is. Is this like typecasting in iOS where I am saying (in the iOS call) I want to pass an NSString into this function ("userId") and inside the cloud code function I'm going to call it "request"? Is that what's going on here?
Question 2 =
Parse.Object.extend('_User')
Is this grabbing the "User" class from the Parse database so that a "PFObject" of sorts can update it by creating a new "user" in the line below it?
Is this like a...
PFObject *userObject = [PFObject objectWithClassName:#"User"]?
Question 3 =
user.set('new_col', newColText)
This obviously 'sets' the values to be saved to the PFUser (~I think). I know that the "newColText" variable is the text that is to be set - but what is 'new_col'? Only thing I can think of is that this sets the name of a new column in the database of whatever type is being passed through the "request"?
Is this like a...
[[PFUser currentUser] setObject: forKey:]
Question 4 =
Parse.Cloud.useMasterKey()
Without getting too technical, is this basically all I have to type before I can edit a "User" object from another User?
Question 5 =
user.save().then(function(user) {
response.success(user);
}
Is this like a...
[user saveInBackgroundWithBlock:]?
and if so, is
function(error) {
response.error(error)
just setting what happens if there is an error in the saveInBackgroundWithBlock?
Please keep in mind, I know iOS - not JavaScript. So try to be as descriptive as possible to someone who understands the Apple realm.
Here's my take on your questions:
The request parameter is for you to access everything that is part of the request/call to your cloud function, it includes the parameters passed (request.params), the User that is authenticated on the client (request.user) and some other things you can learn about in the documentation. The response is for you to send information back to the calling code, you generally call response.success() or response.error() with an optional string/object/etc that gets included in the response, again documentation here.
That's a way of creating an instance of a User, which because it is a special internal class is named _User instead, same with _Role and _Installation. It is creating an instance of the user with an ID, not creating a new one (which wouldn't have an ID until saved). When you create an object this way you can "patch" it by just changing the properties you want updated.
Again, look at the documentation or an example, the first parameter is the column name (it will be created if it doesn't exist), the second value is what you want that column set to.
You have to do Parse.Cloud.useMasterKey() when you need to do something that the user logged into the client doesn't have permission to do. It means "ignore all security, I know what I'm doing".
You're seeing a promise chain, each step in the chain allows you to pass in a "success" handler and an optional "error" handler. There is some great documentation. It is super handy when you want to do a couple of things in order, e.g.
Sample code:
var post = new Parse.Object('Post');
var comment = new Parse.Object('Comment');
// assume we set a bunch of properties on the post and comment here
post.save().then(function() {
// we know the post is saved, so now we can reference it from our comment
comment.set('post', post);
// return the comment save promise, so we can keep chaining
return comment.save();
}).then(function() {
// success!
response.success();
}, function(error) {
// uh oh!
// this catches errors anywhere in the chain
response.error(error);
});
I'm pretty much at the same place as you are, but here are my thoughts:
No, these are the parameters received by the function. When something calls the editUser cloud function, you'll have those two objects to use: request & response. The request is basically what the iOS device sent to the server, and response is what the server will send to the iOS device.
Not quite that. It's like creating a subclass of _User.
Think of Parse objects types as a database table and it's instances as rows. The set will set (derp) the value of 'newColText' to the attribute/column 'new_col'.
Not sure, never used that function as I don't handle User objects. But might be that.
Pretty much that. But it's more sort of like (pseudo-code, mixing JS with Obj-C):
[user saveInBackgroundWithBlock:^(BOOL succeeded, NSError *error){
if(error){
response.error(error); // mark the function as failed and return the error object to the iOS device
}
else{
response.success(user); // mark the function call as successful and return the user object to the iOS device
}
}];
I'm making a temporary fake API and am trying to set up a simple request response script in node using express.js to achieve this. It's very strraightforward, A request comes in, is validated and, if valid, is merged with a .json template file and the result returned, thus giving the impression the user was successfully created.
app.post('/agent/user', function(req, res){
var responseTemplate = new jsonRequired('post_user');
var errorTemplate = new jsonRequired('post_user_error');
var payload = req.body;
var responseData;
var hasErrors = false;
console.log('Creating new user');
//Recursive Merge from http://stackoverflow.com/a/383245/284695
responseData = new mergeRecursive(responseTemplate,payload);
if(!payload.username){
hasErrors = true;
errorTemplate.errors.username.push('A username is required.');
}
if (hasErrors){
res.send(errorTemplate,422);
}else{
res.send(responseData,200);
}
});
The problem I'm having is that data is persisting between calls. So if I define a username and name[first] in 1 request and just a username in the 2nd one, both requests come back with the name[first] property.
I have a feeling it's something to do with js closures. Unfortunately, every tutorial I find seems to be about making closures, not avoiding them.
It should work like this:
The client POST's username=user1&name[first]=joe&name[last]=bloggs
The Server loads a json file containing a prepopulated user object: e.g.
{"username":"demo","name":{"first":"John","last":"Doe"}...}
mergeRecursive() merges the payload from the POST request over the template object and returns the new object as the POST response text.
The problem is that with every new request, the server is using the result of step 3 in step 2 instead of reloading the .json file.
That mergeRecursive function has the same caveat as jQuery.extend: it modifies the first object sent into it. In fact, you don't even need to use its return value.
You didn't show the code of jsonRequired function (it's not even clear why you've used new when invoking it), but it looks like this function doesn't create a new object each time it's called, instead fetching this object from some outer repository. Obviously, mergeRecursive modifications for it won't be lost after that function ends.
The solution is using your data object for merging. Like this:
var responseData = {};
...
mergeRecursive(responseData, responseTemplate);
mergeRecursive(responseData, payload);
Merging two objects will make this for you.
If your responseTemplate has parameter, which actual request did not have, then you will end up having it there.
Check definition of word merge ;)
While this doesn't resolve the issue I had, I have found a workaround using the cloneextend package available via npm:
$ npm install cloneextend
This allows me to use the following js:
var ce = require('cloneextend');
...
ce.extend(responseData, responseTemplate);
ce.extend(responseData, payload);