We Keep Coding

Regex validation for memo field (client and server side) with few special tags

Have been going over this problem for two days without any real luck. I am using asp.net webapi2 with jquery ajax on client side.
I have an edit box for entering memo text, allowable characters are ^[©a-zA-Z0-9\u0900-\u097f,\.\s\-\'\"!?\(\)\[\]]+$ and two tags <LineBreak/> and <Link attr="value"/> (may be couple of more attributes in Link tag. The problem is that NO other tags are allowable - which means that even a simple <br/> should be prevented. This negative check is proving to be bit complicated.
Requesting help in formulating regex for javascript on client side and c# based DataAnnotation check on the server side.

What you're attempting to do is sanitize user input, however, using JavaScript and Regex is the wrong way to go about it.
Don't concern yourself with validating user input on the front end, at least not yet, the focus should be validating it server side first and the best tool for the job is HtmlSanitizer. In their words:
HtmlSanitizer is a .NET library for cleaning HTML fragments and documents from constructs that can lead to XSS attacks.
HtmlSanitizer can be customized at several levels:
Configure allowed HTML tags through the property AllowedTags.
Configure allowed HTML attributes through the property AllowedAttributes.
Configure allowed CSS property names through the property AllowedCssProperties.
Configure allowed CSS at-rules through the property AllowedAtRules.
Configure allowed URI schemes through the property AllowedSchemes.
Configure HTML attributes that contain URIs (such as "src", "href" etc.)
Provide a base URI that will be used to resolve relative URIs against.
Cancelable events are raised before a tag, attribute, or style is removed.
I've mocked up a demo on dotnetfiddle.net using that library for you to play with
void Main()
{
var allowedTags = new[]{"LineBreak", "Link"};
var allowedAttributes = new[]{"attr"};
var sanitizer = new HtmlSanitizer(allowedTags: allowedTags, allowedAttributes: allowedAttributes);
//sanitizer.
var html = #"<script>alert('xss')</script><div onload=""alert('xss')""" + #"style=""background-color: test"">Test<img src=""test.gif""" + #"style=""background-image: url(javascript:alert('xss')); margin: 10px""></div>
<LineBreak></LineBreak>
<Link attr=""v123""/>";
var sanitized = sanitizer.Sanitize(html);
Console.WriteLine(sanitized);
}
Edit
But would like to know why "regex is the wrong way to go about it".
Regex isn't made for this type of task, you need to be able to parse a html document, meaning parsing its tags, attributes and values within those attributes in a tree like structure to be able to properly sanitize it because there's just too many edge cases that's too difficult to cover with just Regex. Regex is better used for scraping data from a source that's already in a structure that is predictable, user input isn't one of those things.
Even though your use case is simple enough, you're still enabling users to type in HTML that will be re displayed to other users in its raw format so anything that you miss will give you a headache down the line.
Here's the XSS Filter Evasion Cheat Sheet from OWASP, if Regex could cover everything listed here, I would say fine, but it's such a difficult task to achieve that in Regex that it just doesn't make sense.
HtmlSanitizer on the other hand does cover the issues listed on that cheat sheet, it's also actively maintained and is specially built for exactly this sort of application, it's also not bulky by any means, it can handle large sanitization tasks with processing times in the 50-100ms range.

Managed to achieve this by a combination of RegularExpression data annotation which allows angle brackets (thereby custom tags)
[RegularExpression(#"([©a-zA-Z0-9\u0900-\u097f,\.\s\-\'\""!?\(\)\[\]\<\>\/]*)")]
and a ValidationAttribute class which checks for unwanted tags (other than LineBreak and Link)
public class CustomTagValidatorAttribute : ValidationAttribute
{
protected override ValidationResult IsValid(object value, ValidationContext validationContext)
{
Regex re = new Regex(#"(<(?!(LineBreak\s*|Link\s+[\s\w\'\""\=]*)\/?>))", RegexOptions.Multiline);
return re.Match(value.ToString()).Length == 0 ? ValidationResult.Success : new ValidationResult(Resources.ErrorStrings.InvalidValuesInRequest);
}
}
Both attributes are applied to the class property as below -
[CustomTagValidator]
[RegularExpression(#"([©a-zA-Z0-9\u0900-\u097f,\.\s\-\'\""!?\(\)\[\]\<\>\/]*)")]
public string PropertyToValidate { get; set; }
Also added an ActionFilterAttribute to ensure the validation check is performed before the controller action is called -
public class ValidateModelAttribute : ActionFilterAttribute
{
public override void OnActionExecuting(HttpActionContext actionContext)
{
if (actionContext.ModelState.IsValid == false)
{
actionContext.Response = actionContext.Request.CreateErrorResponse(
HttpStatusCode.BadRequest, actionContext.ModelState);
}
}
}
and applied this to relevant controller action as below -
[ValidateModel]
public HttpResponseMessage Post([FromBody] MyModel mm)
Hope this helps someone stuck with similar issues.
Almost forgot, same solution was applied on client side using same regex based javascript validation.

RSA sign in python and verify in JS

I am using pyCrypto PKCS1_v1_5 signature to sign the message. (Original code)
from Crypto.Signature import PKCS1_v1_5
from Crypto.Hash import *
from Crypto.PublicKey import RSA
from Crypto import Random
import base64
import codecs
private_key = """-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQDtmBQTFLfaYOsio48B0Sbw4qMnaXJ7Qk4i1gzI+lgLIOPV0YyU
fWMKgl24iJsGN6W5Vk+4GfDE7jKGkti9ZsswaJ/1Un+zFpXlVCOjEe9CZz3npIEK
G8jJL1rwnBc2qPY3YD8jWBhLaPUgchbxUbe1Q2eKDmlRF7B8BzeNG4aVdQIDAQAB
AoGBAK+x6Q5fIMHVL6vyxRqz3pb9EWdgm664Tm2GWff44EiCbti717gqrKWl18ZR
Bkpnth5CzDq4vAn0ltpbFvmgXRmnUL8WsxigrL6tf2mcx4QAgcqr4B6er7X738lB
UDSaEiSLmT/hr9Rg6H/P8IoP01tLKnj/gYQd5CVLkh/VD+dVAkEA9InBNnM5ViAM
Aq7lL/+RWc+jfKnT9kWo+X0v7K8fD9/188K45G6zK/9X0J6Qj83dk1uni5QGx1i1
VVBXIlUnqwJBAPi7AEl8nmmohrSsq8YHX6lTTy6EybSgQS4Qu3SBcVl2RoG89BLx
kReqyTRGupZz/fNS9VfGFZU9VOHcFCeVl18CQEOiLgwvRjZp2qiLUtw5pSvf3+nE
1tkQXzHRzAV8Ue0EFnR68MRNUcTjdJhAot8DIzt0aByUrmNIR67274KRZs0CQC1X
kZ7T2+Dw+tV24L1x3Kt2Z2nYhRirWhZ2sGV1r18ao5HFC01kwglpddJUznDc5j90
MQt2LbsN+ipOP1JT/8sCQE4qVh+TeYVd8aXSqlJaTsLZNzDMjmREYfuodmyZp5WR
q1R6SaZoxQltHgTtK85QqhDxmmWPkR6jdNLDlIR0Bx4=
-----END RSA PRIVATE KEY-----"""
public_key = """"-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDtmBQTFLfaYOsio48B0Sbw4qMn
aXJ7Qk4i1gzI+lgLIOPV0YyUfWMKgl24iJsGN6W5Vk+4GfDE7jKGkti9ZsswaJ/1
Un+zFpXlVCOjEe9CZz3npIEKG8jJL1rwnBc2qPY3YD8jWBhLaPUgchbxUbe1Q2eK
DmlRF7B8BzeNG4aVdQIDAQAB
-----END PUBLIC KEY-----"""
message = 'To be signed'
key = RSA.generate(1024)
pub_key = key.publickey().exportKey(public_key)
priv_key = key.exportKey(private_key)
h = SHA256.new(message)
print '-----------------'
signer = PKCS1_v1_5.new(key)
signature = signer.sign(h)
print signature
hexlify = codecs.getencoder('hex')
m = hexlify(signature)[0] #<--- I am sending this hex signature to JS.
#--------------Code below is to test the verify in python, n it works !
h = SHA256.new('To be signed')
verifier = PKCS1_v1_5.new(key.publickey())
if verifier.verify(h, signature):
print "The signature is authentic."
else:
print "The signature is not authentic."
for testing to verify in JS, I am using this demo sample page. Its written on the page that Signature value should be hexa decimal encoded 'RSASSA-PKCS1-v1_5' signature, so thats why I am converting my signature to hex value in python, but still the signature doesn't match in JS.
I've noticed one strange thing in python code, that is, when ever I generate a signature with fixed private key, the signature is not the same. Is this normal ? but in JS it gives me fixed signature every time.
Note: There is a little bug in JS DEMO sample page,go to the source code and change rsa.signString to rsa.sign. And also, change rsa.verifyString to rsa.verify. I've raised this issue in his Github, he said he'll gona fix it shortly.

when ever I generate a signature with fixed private key, the signature is not the same. Is this normal ?
Not normal. With PKCS#1 v1.5 the signature should always be the same with the same private key.
Check the following:
The SHA256 hash is exactly the same on both ends
I'm not a Python expert, but it also seems weird that you are calling RSA.generate() when you already have key
Given that your Python code is giving different signature every time, I would concentrate my investigation on that part. If a single byte is different in the input data the produced hash is different - so check with your Python code if the hash is changing with the same input data.

Someone posted an answer, suggesting me to check python code and told that its NOT normal to get different signature for fixed private key. I don't know for what reason he deleted the answer, but his observation was right. Actually the way I was getting the key pairs in python is totally wrong. I am generating new key pair every time and exporting it to local variable pub_key & priv_key. Instead I should do something like this;
#key = RSA.generate(2048)
#pub_key = RSA.publickey().exportKey(public_key)
priv_key = RSA.importKey(private_key)
pub_key = RSA.importKey(public_key)
In this way, I get same signature for this fixed private key, and it really matches the signature of JS code :D ...

how to solve C# Json Serialization parsing in javascript with quoutes?

I am serializing a C# Class AnnouncementClass
public class AnnouncementClass
{
public string application;
public List< AnnoucementsMessages > annoucements = new List<AnnoucementsMessages>();
}
public class AnnoucementsMessages
{
public string message;
public string startdate;
public string priority;
}
using the serializing method
string jsonboj = new JavaScriptSerializer().Serialize(announcementClass);
after that I write the json in the javascript by using the <%# > Tag and perform a page.databind() in the page and later in javascript y I use Json.Parse(jsonObj) to generate the json file
but that's giving me a problem
if in the message part i insert a double quoute it breakes the json.parse, even if i try to scape it.
Is there a proper way to translate the json to the javascript side?
do i need to replace something before the c# serialization ?
Thank you

Solution found, thank you
when parsing , Javascript will reduce all the scapes so, using \" = " and \\="
so i had to replace the \" with ;quot

How to Format OpenSSL Keys to Match Online Sample (JSBN-ECC)

I want to be able to format the public key of Elliptic Curve Diffie-Hellman in OpenSSL - Ruby into something like this online example (link), as I have been using that JS library.
My code below generates an OpenSSL::PKey::EC public and private keys
#Ruby
ec = OpenSSL::PKey::EC.new('secp128r1')
ec.generate_key
ec.private_key
#--> 205607153615223513963863936713567041725
ec.public_key.to_bn
#--> 499599043529551953518354858381998373780459818901085313561109939106744612770290
Try copying the private key above 205607153615223513963863936713567041725 and pasting it on the online (link) as Alices' Private value. But click the secp1284r1 button first to have same curve parameters, and then click Compute Public button.
That will generate a public key from the inputted private key. However, the Ruby OpenSSL documentation isn't really helpful, and I am stuck on figuring out how to convert the generated public key above:
499599043529551953518354858381998373780459818901085313561109939106744612770290
Into something like this (as seen from the online site):
x: 107060165679262225845922473865530329196
y: 109296969851421346147544217212275741170
I've assumed that by properly converting one, it can somehow become equal to the other since they have same curve parameters. Or am I wrong? (And also because the default format of point_conversion_form is :uncompressed, as I just have tested) Please help.
P.S. You might wonder why I need to convert the public key into the other. No, I don't really have to. I just want to learn how to convert it as I'll be using that method to convert something similar. And this is the simplified question for your testing-convenience.

Jay-Ar,
You shouldn't find your solution weird as I think the call to ec.public_key.to_bn might be adhering to rfc5480 section 2.2 re: Subject Public Keys which states:
2.2. Subject Public Key
The first octet of the OCTET STRING indicates whether the key is
compressed or uncompressed. The uncompressed form is indicated
by 0x04 and the compressed form is indicated by either 0x02 or
0x03 (see 2.3.3 in [SEC1]). The public key MUST be rejected if
any other value is included in the first octet.
Assuming this is the case and since you stated the format is :uncompressed, your own answer makes total sense to me. Thanks for posting it! :)

ec.public_key.to_bn
#--> 499599043529551953518354858381998373780459818901085313561109939106744612770290
It would probably be better to print that in hex.
The public key is a point on the curve. That is, its an (x,y) coordinate. So that value probably needs to be split (hence the reason for printing in hex). Taking a guess:
x = 499599043529551953518354858381998373780
y = 459818901085313561109939106744612770290
The public key is a point because its derived from the base point G, which is also a point. G is sometimes expanded as (g_x, g_y). The private exponent is a (or b), which is a scalar or integer. So the public key is A=G^a or A=(g_x, g_y)^a, which is a point.
And also because the default format of point_conversion_form is :uncompressed, as I just have tested
Point conversion/compression is just a presentation layer optimization trick. It omits the y portion of the coordinate because you can solve for it given x. Because its a curve, sometimes you need to send +1 or -1 to specify the quadrant the y coordinate lies in. But again, its just an optimization and only matters for interop'ing. After you read the point into a library, both x and y are available.
You might wonder why I need to convert the public key into the other...
Nope :)
But one thing you might want to be aware of is OpenSSL's "named curve" flag. If you want to load an EC key into an OpenSSL based server, you will want to ensure the private key and resulting certificate have the OPENSSL_EC_NAMED_CURVE flag. Otherwise, you'll get bizarre errors like "no shared ciphers" when trying to connect to the server. See ECDH and Named Curves on the OpenSSL wiki for details.

Finally! I somehow managed to convert it properly but it's somehow weird.
#From above code
c.public_key.to_bn
#--> 499599043529551953518354858381998373780459818901085313561109939106744612770290
#irb:
require 'openssl'
key_int = '499599043529551953518354858381998373780459818901085313561109939106744612770290'
key_bn = OpenSSL::BN.new(key_int, 10) #Convert to OpenSSL::BN (Big Number, with 10=Decimal as base)
key_hex = key_bn.to_s(16) #Convert to Hex String (16=Hexadecimal)
#--> "04508B09B35FA8C21820BE19C16B38486C5239D4A932D081DD56B90F91120551F2"
#I don't really know why, but removing '04' above will finally convert it properly
key_hex = key_hex[2..-1] #Remove first 2 chars: '04'
#--> "508B09B35FA8C21820BE19C16B38486C5239D4A932D081DD56B90F91120551F2"
#Split key_hex into halves
key_hexarr = key_hex.chars.each_slice( (key_hex.length/2.0).round ).map(&:join)
#--> ["508B09B35FA8C21820BE19C16B38486C", "5239D4A932D081DD56B90F91120551F2"]
#Convert first value into BN (input: 16=hexadecimal), then convert to string(output: 10=decimal)
key_x_int = OpenSSL::BN.new(key_hexarr[0], 16).to_s(10)
#--> "107060165679262225845922473865530329196"
#Convert second value into BN (input: 16=hexadecimal), then convert to string(output: 10=decimal)
key_y_int = OpenSSL::BN.new(key_hexarr[1], 16).to_s(10)
#--> "109296969851421346147544217212275741170"
Finally, key_x_int and key_y_int now matches the result from the online link

Base64 HMAC SHA1 String in VBA

I'm trying to convert an ASP/VBScript OAuth library to VBA. One of the challenges is this line of code:
Get_Signature = b64_hmac_sha1(strSecret, strBaseSignature)
This function, b64_hmac_sha1 is actually a function contained in a JavaScript library. It appears to me that calling a JavaScript function from VBA is fairly impractical.
Because I know so little about encryption, it's not even clear to me what this b64_hmac_sha1 function does. Is HMAC SHA1 different from SHA1?
I half suspect I might be able to find some VBA code online to do what I need to do if I just understood what this function is actually doing. If I do not find an existing function, I could possibly write one that would use the .NET Cryptography library (you can actually call the .NET cryptography libraries from VBA if you know how).
I'm not looking for someone to convert this JavaScript to VBA. I'm only trying to understand what it is that this b64_hmac_sha1 function is outputting so I can try to find ways to achieve the same output in VBA if possible.
A copy of this JavaScript library is visible on this website. You'll have to scroll down past the VBScript to the JavaScript section.
http://solstice.washington.edu/solstice/ASP_Signing_REST_Example
Edit1:
OK, so here's the functions I ended up writing and using:
Public Function Base64_HMACSHA1(ByVal sTextToHash As String, ByVal sSharedSecretKey As String)
Dim asc As Object, enc As Object
Dim TextToHash() As Byte
Dim SharedSecretKey() As Byte
Set asc = CreateObject("System.Text.UTF8Encoding")
Set enc = CreateObject("System.Security.Cryptography.HMACSHA1")
TextToHash = asc.Getbytes_4(sTextToHash)
SharedSecretKey = asc.Getbytes_4(sSharedSecretKey)
enc.Key = SharedSecretKey
Dim bytes() As Byte
bytes = enc.ComputeHash_2((TextToHash))
Base64_HMACSHA1 = EncodeBase64(bytes)
Set asc = Nothing
Set enc = Nothing
End Function
Private Function EncodeBase64(ByRef arrData() As Byte) As String
Dim objXML As MSXML2.DOMDocument
Dim objNode As MSXML2.IXMLDOMElement
Set objXML = New MSXML2.DOMDocument
' byte array to base64
Set objNode = objXML.createElement("b64")
objNode.DataType = "bin.base64"
objNode.nodeTypedValue = arrData
EncodeBase64 = objNode.Text
Set objNode = Nothing
Set objXML = Nothing
End Function
Using this function:
Debug.Print Base64_HMACSHA1("abc", "123")
VAsMU9SSWDe9krP3Gr56nXC2dsQ=

HMAC is a construct for turning a hash function, like SHA1, into a Message Authentication Code (MAC).
Normal hash functions don't have any secret data associated with it. This means that anyone can compute the digest, assuming they have the original input. HMAC uses a secret key, so that only those in possession of the key can compute outputs.
Suppose I have a file, file.txt. I want to send this to you, and we need to make sure nobody tampers with it. Sorry, I have no clever way to represent this with just text.
me -> file.txt -> you
me -> SHA1(file.txt) -> you
Then you verify the result by computing your own SHA1 digest, and verifying it matches what I sent you.
Now suppose an attacker was in the middle. Unfortunately, because there is no secret involved, the attacker can modify the file, and compute his own file/digest pair. When you compute your version, it'll match what he sent you, and you'll be none the wiser.
me -> file.txt -> attacker -> modified.txt -> you
me -> SHA1(file.txt) -> attacker -> SHA1(modified.txt) -> you
With HMAC, we add a secret key to the computation.
me -> file.txt -> you
me -> SHA1_HMAC(file.txt, our_secret) -> you
When you compute your version, you apply the secret key as well, and the result matches. The attacker, without knowledge of the key, can't replace the digest.
me -> file.txt -> attacker -> modified.txt -> you
me -> SHA1(file.txt) -> attacker -> SHA1_HMAC(modified.txt, // DOESN'T KNOW KEY) -> you
HMAC is a very specific way of adding the secret key. Unfortunately, simple methods of just concatenating a key to the end of the file, or pre-pending it before hashing, are vulnerable to different attacks (length extension attacks, for example).
The B64 is Base64 encoding the output, to make it pretty.
What this code is ultimately doing is taking some input, and some secret key, and computing a 160-bit digest, and base64 encoding the result.
There is an implementation of SHA1 HMAC in .NET
This looks like an implementation of Base64 for VBA
I hope this answers it well enough, or clear enough. If the text is confusing, please let me know. I tried a couple routes of how to express it, and none of them seemed that clear.

You have written:
It appears to me that calling a JavaScript function from VBA is fairly impractical.
That is a misjudgment.
Javascript can be easily packaged as a Windows Script Component (WSC) and then invokved via COM, from VBA, Perl, VB6, or what-have-you.
Here's an example of packaging Javascript as a WSC and invoking it: https://stackoverflow.com/a/849970/48082
Therefore, your problem should be easily solvable.