We Keep Coding

App API design advice specifically around security

I'm building an app and would like some feedback on my approach to building the data sync process and API that supports it. For context, these are the guiding principles for my app/API:
Free: I do not want to charge people at all to use the app/API.
Open source: the source code for both the app and API are available to the public to use as they wish.
Decentralised: the API service that supports the app can be run by anyone on any server, and made available for use to users of the app.
Anonymous: the user should not have to sign up for the service, or submit any personal identifying information that will be stored alongside their data.
Secure: the user's data should be encrypted before being sent to the server, anyone with access to the server should have no ability to read the user's data.
I will implement an instance of the API on a public server which will be selected in the app by default. That way initial users of the app can sync their data straight away without needing to find or set up an instance of the API service. Over time, if the app is popular then users will hopefully set up other instances of the API service either for themselves or to make available to other users of the app should they wish to use a different instance (or if the primary instance runs out of space, goes down, etc). They may even access the API in their own apps. Essentially, I want them to be able to have the choice to be self sufficient and not have to necessarily rely on other's providing an instance on the service for them, for reasons of privacy, resilience, cost-saving, etc. Note: the data in question is not sensitive (i.e. financial, etc), but it is personal.
The user's sync journey works like this:
User downloads the app, and creates their data in the process of using the app.
When the user is ready to initially sync, they enter a "password" in the password field, which is used to create a complex key with which to encrypt their data. Their password is stored locally in plain text but is never sent to the server.
User clicks the "Sync" button, their data is encrypted (using their password) and sent to the specified (or default) API instance and responds by giving them a unique ID which is saved by the app.
For future syncs, their data is encrypted locally using their saved password before being sent to the API along with their unique ID which updates their synced data on the server.
When retrieving synced data, their unique ID is sent to the API which responds with their encrypted data. Their locally stored password is then used to decrypt the data for use by the app.
I've implemented the app in javascript, and the API in Node.js (restify) with MongoDB as a backend, so in practice a sync requests to the server looks like this:
1. Initial sync
POST /api/data
Post body:
{
"data":"DWCx6wR9ggPqPRrhU4O4oLN5P09onApoAULX4Xt+ckxswtFNH/QQ+Y/RgxdU+8+8/muo4jo/jKnHssSezvjq6aPvYK+EAzAoRmXenAgUwHOjbiAXFqF8gScbbuLRlF0MsTKn/puIyFnvJd..."
}
Response:
{
"id":"507f191e810c19729de860ea",
"lastUpdated":"2016-07-06T12:43:16.866Z"
}
2. Get sync data
GET /api/data/507f191e810c19729de860ea
Response:
{
"data":"DWCx6wR9ggPqPRrhU4O4oLN5P09onApoAULX4Xt+ckxswtFNH/QQ+Y/RgxdU+8+8/muo4jo/jKnHssSezvjq6aPvYK+EAzAoRmXenAgUwHOjbiAXFqF8gScbbuLRlF0MsTKn/puIyFnvJd...",
"lastUpdated":"2016-07-06T12:43:16.866Z"
}
3. Update synced data
POST /api/data/507f191e810c19729de860ea
Post body:
{
"data":"DWCx6wR9ggPqPRrhU4O4oLN5P09onApoAULX4Xt+ckxswtFNH/QQ+Y/RgxdU+8+8/muo4jo/jKnHssSezvjq6aPvYK+EAzAoRmXenAgUwHOjbiAXFqF8gScbbuLRlF0MsTKn/puIyFnvJd..."
}
Response:
{
"lastUpdated":"2016-07-06T13:21:23.837Z"
}
Their data in MongoDB will look like this:
{
"id":"507f191e810c19729de860ea",
"data":"DWCx6wR9ggPqPRrhU4O4oLN5P09onApoAULX4Xt+ckxswtFNH/QQ+Y/RgxdU+8+8/muo4jo/jKnHssSezvjq6aPvYK+EAzAoRmXenAgUwHOjbiAXFqF8gScbbuLRlF0MsTKn/puIyFnvJd...",
"lastUpdated":"2016-07-06T13:21:23.837Z"
}
Encryption is currently implemented using CryptoJS's AES implementation. As the app provides the user's password as a passphrase to the AES "encrypt" function, it generates a 256-bit key which which to encrypt the user's data, before being sent to the API.
That about sums up the sync process, it's fairly simple but obviously it needs to be secure and reliable. My concerns are:
As the MongoDB ObjectID is fairly easy to guess, it is possible that a malicious user could request someone else's data (as per step 2. Get sync data) by guessing their ID. However, if they are successful they will only retrieve encrypted data and will not have the key with which to decrypt it. The same applies for anyone who has access to the database on the server.
Given the above, is the CryptoJS AES implementation secure enough so that in the real possibility that a user's encrypted data is retrieved by a malicious user, they will not realistically be able to decrypt the data?
Since the API is open to anyone and doesn't audit or check the submitted data, anyone could potentially submit any data they wish to be stored in the service, for example:
Post body:
{
"data":"This is my anyold data..."
}
Is there anything practical I can do to guard against this whilst adhering to the guiding principles above?
General abuse of the service such as users spamming initial syncs (step 1 above) over and over to fill up the space on the server; or some user's using disproportionately large amounts of server space. I've implemented some features to guard against this, such as logging IPs for initial syncs for one day (not kept any longer than that) in order to limit a single IP to a set number of initial syncs per day. Also I'm limiting the post body size for syncs. These options are configurable in the API however, so if a user doesn't like these limitations on a public API instance, they can host their own instance and tweak the settings to their liking.
So that's it, I would appreciate anyone who has any thoughts or feedback regarding this approach given my guiding principles. I couldn't find any examples where other apps have attempted a similar approach, so if anyone knows of any and can link to them I'd be grateful.

I can't really comment on whether specific AES algorithms/keys are secure or not, but assuming they are (and the keys are generated properly), it should not be a problem if other users can access the encrypted data.
You can maybe protect against abuse, without requiring other accounts, by using captchas or similar guards against automatic usage. If you require a catcha on new accounts, and set limits to all accounts on data volume and call frequency, you should be ok.
To guard against accidental clear-text data, you might generate a secondary key for each account, and then check on the server with the public secondary key whether the messages can be decrypted. Something like this:
data = secondary_key(user_private_key(cleartext))
This way the data will always be encrypted, and in worst case the server will be able to read it, but others wouldn't.
A few comments to your API :) If you're already using HTTP and POST, you don't really need an id. The POST usually returns a URI that points to the created data. You can then GET that URI, or PUT it to change:
POST /api/data
{"data": "..."}
Response:
Location: /api/data/12345
{"data": "...", "lastmodified": "..." }
To change it:
PUT /api/data/12345
{"data": "..."}
You don't have to do it this way, but it might be easier to implement on the client side, and maybe even help with caching and cache invalidation.

Difference between RSAwithSHA256 and SHA256

I need to sign XML SAML message with the SAML 2.0 standardised algorithm (RSAwithSHA256). But my saml plugin (passport-saml) only seems to support sha1 and sha256. The SHA256 sounds pretty close to RSAwithSHA256, but probably is not the same thing? What is the difference, and how could I use RSAwithSHA256 instead? I probably need to edit the passport-saml library, to allow the use of RSAwithSHA256 algorithm?

I try to explain the differences, but not how to solve your issue.
RSA is a Public Key Cryptographic algorithm (Public and Private Key-Pair algorithm) and it assures Confidentiality, Authenticity (includes Identification) and Non-Repudiation.
SHA-256 is a Hashing algorithm, that produce a unique, fixed size 256-bit (32-byte) hash and it assures Message Integrity.
Hashing algorithm employed as follows,
Sender sends message and its hash to receiver. [Hashing employed]
Receiver hash the message to generate new hash. [Hashing employed]
Receiver check whether the new hash is equal to original hash.
If its equal, then message integrity is confirmed and receiver process the message further.
If its unequal, then message is tampered and receiver discard the message.
Here, how receiver confirms that message and its hash are indeed sent by expected sender? There is no authentication or identification of sender by receiver in the above case.
To do that, we have to use both Public Key Cryptography and Hashing Algorithms (like RSAWithSHA256) together to satisfy the above said requirement.
So, when employ Public Key Cryptography and Hashing Algorithms together,
Sender sends message and its encrypted hash (using private-key of sender) to receiver. [Encryption and Hashing employed]
Receiver decrypt the encrypted hash (using public-key of sender). [Decryption and Hashing employed]
Receiver hash the message to generate new hash. [Hashing employed]
Receiver check whether the new hash is equal to decrypted hash.
If its equal, then message integrity, authenticity and
identification of sender is confirmed and receiver process the
message further.
If its unequal, then message is tampered or not sent by intended
sender (since encrypted hash is not generated with private-key of expected sender) and receiver discard the message.

JavaScript message system encryption

I'm trying to create a message system with JavaScript and PHP / MySQL. I have a form with two input elements (recipient id, message content). I'm using MVC (Zend Framework 1). The form post data is send to my controller and stored in the database.
Now I want to encrypt the message before it is sent. I want to keep it user-friendly, so my idea was to use RSA (private / public key). The idea was that a private key was generated on user log in and stored in the cookies, to make sure that the private key is only on the user's machine. The public key could be stored in the user's table, so that any user, who want to send a message to him, can encrypt the data.
It is important that the key-pair is generated by the user's password. If it's random generated, it would not be possible to use multiple systems to log in, because the private key would change everytime. So that would be the mechanism to make sure, that he will always have the same private key, until he is changing his password.
I tried a few JavaScript libraries. cryptico seemed to be the right choice, because it generates private / public key by password. The problem here is, that I can not store the private key and not even look into the value.
They have an example on the website
// The passphrase used to repeatably generate this RSA key.
var PassPhrase = "The Moon is a Harsh Mistress.";
// The length of the RSA key, in bits.
var Bits = 1024;
var MattsRSAkey = cryptico.generateRSAKey(PassPhrase, Bits);
When I try to output MattsRSAkey, I only get [Object object]. It's the same when I store it in the Cookies. I tried to use JSON.stringify. With this function I can store and look inside MattsRSAKey. But when I want to use it later to decrypt the message, I get an error, that I have no valid public key. I think the private key got broken while storing it. When I read the private key from Cookies I use JSON.parse.
Is there any way to solve my problem? I just want to send encrypted messages from multiple users (public key) to one user (private key). My intention is not to have a secure transport but to store the messages encrypted in the database, so that unauthorized persons can not read it. It is important that I do not only have encryption for one-to-one messaging. This would be easy, because both users only would need to share a password for encryption.

There's a couple of things wrong here.
First, you're trying to store a Javascript object directly in a cookie. This won't work: cookies can only store string values. You will need to serialize the key to a string to store it in a cookie; unfortunately, it doesn't appear that the cryptico library exposes any methods to do this, so you will need to either implement a custom serializer, or use another cryptographic library.
Second, you are storing private cryptographic key data in cookies. This is perhaps the worst possible place to store this, as cookies are sent to the web server on every request. Local storage is much more appropriate here, as it is only accessible from Javascript code.

Query a server for the existence of a record without the server knowing exactly what record was being queried for

I've been thinking about services such as pwnedlist.com and shouldichangemypassword.com and the fundamental problem with them - trust.
That is to say the user must trust that these services aren't going to harvest the submitted queries.
Pwnedlist.com offers the option to submit a SHA-512 hash of the users query which is a step forward but still leaks information if the query does exist in the database. That is, a malicious service would know that the given email address was valid (see also: why you should never click unsubscribe links in spam email).
The solution I came up with is as follows:
1) Instead of the user calculating and submitting the hash herself, the hash (I'll use the much simpler md5 in my example) is calculated via client side javascript:
md5("user#example.com") = "b58996c504c5638798eb6b511e6f49af"
2) Now, instead of transmitting the entire hash as a query to the server, only the first N bits are transmitted:
GET http://remotesite.com?query=b58996
3) The server responds with all hashes that exist in it's database that begin with the same N bits:
{
"b58996afe904bc7a211598ff2a9200fe",
"b58996c504c5638798eb6b511e6f49af",
"b58996443fab32c087632f8992af1ecc",
...etc... }
4) The client side javascript compares the list of hashes returned by the server and informs the user whether or not her email address exists in the DB.
Since "b58996c504c5638798eb6b511e6f49af" is present in the server response, the email exists in the database - inform the user!
Now, the obvious problem with this solution is that the user must trust the client side javascript to only transmit what it says it is going to transmit. Sufficiently knowledgable individuals however, would be able to verify that the query isn't being leaked (by observing the queries sent to the server). It's not a perfect solution but it would add to the level of trust if a user could (theoretically) verify that site functions as it says it does.
What does SO think of this solution? Importantly, does anyone know of any existing examples or discussion of this technique?
NOTE: Both pwnedlist.com and shouldichangemypassword.com are apparently run by reputable people/organizations, and I have no reason to believe otherwise. This is more of a thought exercise.

Services like pwnedlist.com are working with public information. By definition everyone has access to this data, so attempting to secure it is a moot point. An attacker will just download it from The Pirate Bay.
However, using a hash function like this is still easy to break because its unsalted and lacks key straighting. In all reality a message digest function like sha-512 just isn't the right tool for the job.
You are much better off with a Bloom Filter. This allows you to create a blacklist of leaked data without any possibility of obtaining the plain-text. This is because a permutation based brute force likely to find collisions than real plain text. Lookups and insertions a cool O(1) complexity, and the table its self takes up much less space, maybe 1/10,000th of the space it would using a traditional sql database, but this value is variable depending on the error rate you specify.

Simple XOR a message (Javascript/Tcl)?

I need the username/password to be scrambled at the client-side before sending it over via HTTP GET/POST. And the server will decode it with Tcl, before the checks against database.
Currently I'm thinking about using JavaScript for the client-side. Java Applet will also do.
Is there any way, that I can easily achieve it, using Simple XOR or any other methods? (Examples would be much appreciated)
I've found the few samples in C/Python/.NET/Java... But not in JavaScript and Tcl.
SSL is not an option to use, sadly.

If ssl is not an option, then I suggest the following scheme, which many sites use instead of SSL:
On the client side, combine the user name and password, then calculate a hash from it (MD5 is a popular choice).
Send the user's name and hash over to the server
On the server side, retrieve the password for that user from the database.
From the user name and password, calculate the hash and compare it with the client's hash. If the two match, then the passwords match.
For added security, add a little random text to the user+password mix. This random text, AKA the "salt", must be known on both the client and server sides.
Here is a suggestion on how to calculate the hash using MD5:
package require md5
proc calculateHash {user password salt} {
return md5:md5 -hex "$user:$salt:$password"
}
How to use it:
set user "johnny"
set password "begood2mama"
set salt "myDog_is_meaner_than_yourDog"
set hash [calculateHash $user $password $salt]

superNobody,
You should consider alternatives to storing plain-text passwords in the database. See:
http://www.codinghorror.com/blog/2007/09/youre-probably-storing-passwords-incorrectly.html
Instead of encoding the password in Javascript, then decoding the password in Tcl to compare with the database, you should consider SHA1 hashing in Javascript, and storing SHA1 hashed values in the database.
There are several available examples of a SHA1 hash function in javascript (just Google 'sha1 javascript'). The tcllib Tcl library has SHA1 support.
As HaiVu mentioned, you should also consider hashing / storing more than just a straight password hash, but instead use something like SHA1( username + websitename + password ). You can calculate this on the client in Javascript, and store it in the db.