We Keep Coding

How to filter out characters that aren't letters, numbers or punctuation

I have a string that will have a lot of formatting things like bullet points or arrows or whatever. I want to clean this string so that it only contains letters, numbers and punctuation. Multiple spaces should be replaced by a single space too.
Allowed punctuation: , . : ; [ ] ( ) / \ ! # # $ % ^ & * + - _ { } < > = ? ~ | "
Basically anything allowed in this ASCII table.
This is what I have so far:
let asciiOnly = y.replace(/[^a-zA-Z0-9\s]+/gm, '')
let withoutSpacing = asciiOnly.replace(/\s{2,}/gm, ' ')
Regex101: https://regex101.com/r/0DC1tz/2
I also tried the [:punct:] tag but apparently it's not supported by javascript. Is there a better way I can clean this string other than regex? A library or something maybe (I didn't find any). If not, how would I do this with regex? Would I have to edit the first regex to add every single character of punctuation?
EDIT: I'm trying to paste an example string in the question but SO just removes characters it doesn't recognize so it looks like a normal string. Heres a paste.
EDIT2: I think this is what I needed:
let asciiOnly = x.replace(/[^\x20-\x7E]+/gm, '')
let withoutSpacing = asciiOnly.replace(/\s{2,}/gm, ' ')
I'm testing it with different cases to make sure.

You can achieve this using below regex, which finds any non-ascii characters (also excludes non-printable ascii characters and excluding extended ascii too) and removes it with empty string.
[^ -~]+
This is assuming you want to retain all printable ASCII characters only, which range from space (ascii value 32) to tilde ~ hence usage of this char set [^ !-~]
And then replaces all one or more white space with a single space
var str = `Determine the values of P∞ and E∞ for each of the following signals: b.
d.
f.
Periodic and aperiodic signals Determine whether or not each of the following signals is periodic:
b.
Determine whether or not each of the following signals is periodic. If a signal is periodic, specify its fundamental period.
b.
d.
Transformation of Independent variables A continuous-time signal x(t) is shown in Figure 1. Sketch and label carefully each of the following signals:
b. c.
d. e. f. Figure 1: Problem Set 1.4
Even and Odd Signals
For each signal given below, determine all the values of the independent variable at which the even part of the signal is guaranteed to be zero.
b.
d. -------------------------`;
console.log(str.replace(/[^ -~]+/g,'').replace(/\s+/g, ' '));
<!-- begin snippet: js hide: false console: true babel: false -->
console.log(str.replace(/[^ !-~]+/g,'').replace(/\s+/g, ' '));
Also, if you just want to allow all alphanumeric characters and mentioned special characters, then you can use this regex to first retain all needed characters using this regex ,
[^ a-zA-Z0-9,.:;[\]()/\!##$%^&*+_{}<>=?~|"-]+
Replace this with empty string and then replace one or more white spaces with just a single space.
var str = `Determine the values of P∞ and E∞ for each of the following signals: b.
d.
f.
Periodic and aperiodic signals Determine whether or not each of the following signals is periodic:
b.
Determine whether or not each of the following signals is periodic. If a signal is periodic, specify its fundamental period.
b.
d.
Transformation of Independent variables A continuous-time signal x(t) is shown in Figure 1. Sketch and label carefully each of the following signals:
b. c.
d. e. f. Figure 1: Problem Set 1.4
Even and Odd Signals
For each signal given below, determine all the values of the independent variable at which the even part of the signal is guaranteed to be zero.
b.
d. -------------------------`;
console.log(str.replace(/[^ a-zA-Z0-9,.:;[\]()/\!##$%^&*+_{}<>=?~|"-]+/g,'').replace(/\s+/g, ' '));

This is how i will do. I will remove the all the non allowed character first and than replace the multiple spaces with a single space.
let str = `Determine the values of P∞ and E∞ for each of the following signals: b.
d.
f.
Periodic and aperiodic signals Determine whether or not each of the following signals is periodic:!!!23
b.
Determine whether or not each of the following signals is periodic. If a signal is periodic, specify its fundamental period.
b.
d.
Transformation of Independent variables A continuous-time signal x(t) is shown in Figure 1. Sketch and label carefully each of the following signals:
b. c.
d. e. f. Figure 1: Problem Set 1.4
Even and Odd Signals
For each signal given below, determine all the values of the independent variable at which the even part of the signal is guaranteed to be zero.
b.
d. ------------------------- `
const op = str.replace(/[^\w,.:;\[\]()/\!##$%^&*+{}<>=?~|" -]/g, '').replace(/\s+/g, " ")
console.log(op)
EDIT : In case you want to keep \n or \t as it is use (\s)\1+, "$1" in second regex.

There probably isn't a better solution than a regex. The under-the-hood implementation of regex actions is usually well optimized by virtue of age and ubiquity.
You may be able to explicitly tell the regex handler to "compile" the regex. This is usually a good idea if you know the regex is going to be used a lot within a program, and may help with performance here. But I don't know if javascript exposes such an option.
The idea of "normal punctuation" doesn't have an excellent foundation. There are some common marks like "90°" that aren't ASCII, and some ASCII marks like "" () that you almost certainly don't want. I would expect you to find similar edge cases with any pre-made list. In any case, just explicitly listing all the punctuation you want to allow is better in general, because then no one will ever have to look up what's in the list you chose.
You may be able to perform both substitutions in a single pass, but it's unclear if that will perform better and it almost certainly won't be clearer to any co-workers (including yourself-from-the-future). There will be a lot of finicky details to work out such as whether " ° " should be replaced with "", " ", or " ".

Javascript - how to use regex process the following complicated string

I have the following string that will occur repeatedly in a larger string:
[SM_g]word[SM_h].[SM_l] "
Notice in this string after the phrase "[SM_g]word[Sm_h]" there are three components:
A period (.) This could also be a comma (,)
[SM_l]
"
Zero to all three of these components will always appear after "[SM_g]word[SM_h]". However, they can also appear in any order after "[SM_g]word[SM_h]". For example, the string could also be:
[SM_g]word[SM_h][SM_l]"
or
[SM_g]word[SM_h]"[SM_l].
or
[SM_g]word[SM_h]".
or
[SM_g]word[SM_h][SM_1].
or
[SM_g]word[SM_h].
or simply just
[SM_g]word[SM_h]
These are just some of the examples. The point is that there are three different components (more if you consider the period can also be a comma) that can appear after "[SM_h]word[SM_g]" where these three components can be in any order and sometimes one, two, or all three of the components will be missing.
Not only that, sometimes there will be up to one space before " and the previous component/[SM_g]word[SM_h].
For example:
[SM_g]word[SM_h] ".
or
[SM_g]word[SM_h][SM_l] ".
etc. etc.
I am trying to process this string by moving each of the three components inside of the core string (and preserving the space, in case there is a space before &\quot; and the previous component/[SM_g]word[SM_h]).
For example, [SM_g]word[SM_h].[SM_l]" would turn into
[SM_g]word.[SM_l]"[SM_h]
or
[SM_g]word[SM_h]"[SM_l]. would turn into
[SM_g]word"[SM_l].[SM_h]
or, to simulate having a space before "
[SM_g]word[SM_h] ".
would turn into
[SM_g]word ".[SM_h]
and so on.
I've tried several combinations of regex expressions, and none of them have worked.
Does anyone have advice?

You need to put each component within an alternation in a grouping construct with maximum match try of 3 if it is necessary:
\[SM_g]word(\[SM_h])((?:\.|\[SM_l]| ?"){0,3})
You may replace word with .*? if it is not a constant or specific keyword.
Then in replacement string you should do:
$1$3$2
var re = /(\[SM_g]word)(\[SM_h])((?:\.|\[SM_l]| ?"){0,3})/g;
var str = `[SM_g]word[SM_h][SM_l] ".`;
console.log(str.replace(re, `$1$3$2`));

This seems applicable for your process, in other word, changing sub-string position.
(\[SM_g])([^[]*)(\[SM_h])((?=([,\.])|(\[SM_l])|( ?&\\?quot;)).*)?
Demo,,, in which all sub-strings are captured to each capture group respectively for your post processing.
[SM_g] is captured to group1, word to group2, [SM_h] to group3, and string of all trailing part is to group4, [,\.] to group5, [SM_l] to group6, " ?&\\?quot;" to group7.
Thus, group1~3 are core part, group4 is trailing part for checking if trailing part exists, and group5~7 are sub-parts of group4 for your post processing.
Therefore, you can get easily matched string's position changed output string in the order of what you want by replacing with captured groups like follows.
\1\2\7\3 or $1$2$7$3 etc..
For replacing in Javascript, please refer to this post. JS Regex, how to replace the captured groups only?
But above regex is not sufficiently precise because it may allow any repeatitions of the sub-part of the trailing string, for example, \1\2\3\5\5\5\5 or \1\2\3\6\7\7\7\7\5\5\5, etc..
To avoid this situation, it needs to adopt condition which accepts only the possible combinations of the sub-parts of the trailing string. Please refer to this example. https://regex101.com/r/6aM4Pv/1/ for the possible combinations in the order.
But if the regex adopts the condition of allowing only possible combinations, the regex will be more complicated so I leave the above simplified regex to help you understand about it. Thank you:-)

Number is different than itself (trimming strange characters)

I've copied the first number from the windows calculator, and typed the second one. In Chrome console I get:
"‭65033‬" == "65033"
//false
65033‬ == 65033
//Uncaught SyntaxError: Invalid or unexpected token
It seems there is an unknown character at the beginning and end of it.
1) Is there a way to trim all "strange" characters without knowing them a priori?
2) Why does the windows calculator puts such chars in the number?
Edit: Was not explicit in the question, but any chars with valid information, such as ã,ü,ç,¢,£ would also be valid. What I don't want is characters that do not carry any information for the human reader.

Edit: after the edit of the original question, this answer no longer offers a bulletproof solution.
var myNumber = 'foo123bar';
var realNumber = window.parseInt(myNumber.replace(/\D*/g, ''), 10);
What this does?
It replaces all the non-digit characters with empty character and then parses the integer out of numbers left in the string.

A quick solution for this case:
eval("65033‬ == 65033".replace(/[^a-zA-Z0-9 =-_.]/, ''))
You can place your copied text in a string, then remove all unnecessary characters (by explicitly listing the ones that should stay there).
These may include non-alphanumerical characters + hyphen, underscore, equality, space et cetera - actual character that need to stay there will depend on your choice and needs.
Alternatively, you may try to remove all non-printable characters, as suggested here.
Finally, evaluate resulting code. Remember this is not necessarily the best idea for production code.

javascript regular expression for DN

I wan a regex to alidate all types of possible DN's
I create one but its not so good.
/([A-z0-9=]{1}[A-z0-9]{1})*[,??]/ and some others by changing it, but in vain.
Posible DN's can be
CN=abcd,CN=abcd,O=abcd,C=us
CN=abcd0520,CN=users,O=abcd,C=us
C=us
etc

I recently had a need for this, so I created one that perfectly follows the LDAPv3 distinguished name syntax at RFC-2253.
Attribute Type
An attributeType can be expressed 2 ways. An alphanumeric string that starts with an alpha, validated using:
[A-Za-z][\w-]*
Or it can be an OID, validated using:
\d+(?:\.\d+)*
So attributeType validates using:
[A-Za-z][\w-]*|\d+(?:\.\d+)*
Attribute Value
An attributeValue can be expressed 3 ways. A hex string, which is a sequence of hex-pairs with a leading #. A hex string validates using:
#(?:[\dA-Fa-f]{2})+
Or an escaped string; each non-special character is expressed "as-is" (validates using [^,=\+<>#;\\"]). Special characters can be expressed with a leading \ (validates using \\[,=\+<>#;\\"]). Finally any character can be expressed as a hex-pair with a leading \ (validates using \\[\dA-Fa-f]{2}). An escaped string validates using:
(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*
Or a quoted-string; the value starts and ends with ", and can contain any character un-escaped except \ and ". Additionally, any of the methods from the escaped string above can be used. A quoted-string validates using:
"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*"
All combined, an attributeValue validates using:
#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*"
Name component
A name-component in BNF is:
name-component = attributeTypeAndValue *("+" attributeTypeAndValue)
attributeTypeAndValue = attributeType "=" attributeValue
In RegEx is:
(?#attributeType)=(?#attributeValue)(?:\+(?#attributeType)=(?#attributeValue))*
Replacing the (?#attributeType) and (?#attributeValue) placeholders with the values above gives us:
(?:[A-Za-z][\w-]*|\d+(?:\.\d+)*)=(?:#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*")(?:\+(?:[A-Za-z][\w-]*|\d+(?:\.\d+)*)=(?:#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*"))*
Which validates a single name-component.
Distinguished name
Finally, the BNF for a distinguished name is:
name-component *("," name-component)
In RegEx is:
(?#name-component)(?:,(?#name-component))*
Replacing the (?#name-component) placeholder with the value above gives us:
^(?:[A-Za-z][\w-]*|\d+(?:\.\d+)*)=(?:#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*")(?:\+(?:[A-Za-z][\w-]*|\d+(?:\.\d+)*)=(?:#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*"))*(?:,(?:[A-Za-z][\w-]*|\d+(?:\.\d+)*)=(?:#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*")(?:\+(?:[A-Za-z][\w-]*|\d+(?:\.\d+)*)=(?:#(?:[\dA-Fa-f]{2})+|(?:[^,=\+<>#;\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*|"(?:[^\\"]|\\[,=\+<>#;\\"]|\\[\dA-Fa-f]{2})*"))*)*$
Test it here

This is not only not possible, it will never work, and should not even be attempted. LDAP data (distinguished name in this case) are not strings. A distinguished name has distinguishedName syntax, which is not a string, and comparisons must be made with using matching rules defined in the directory server schema. For this reason, regular expressions and native-language comparison, relative value, and equality operations like perl's ~~, eq and == and Java's == cannot be used with LDAP data - if a programmer attempts this, unexpected results can occur and the code is brittle, fragile, unpredictable, and does not have repeatable characteristics. Language LDAP APIs that do not support matching rules cannot be used with LDAP where comparison, equality checks, and relative value ordering comparisons are required.
By way of example, the distinguished names "dc=example,dc=com" and "DC=example, DC=COM" are equivalent in every way from an LDAP perspective, but native language equality operators would return false.

This worked for me:
Expression:
^(?<RDN>(?<Key>(?:\\[0-9A-Fa-f]{2}|\\\[^=\,\\]|[^=\,\\]+)+)\=(?<Value>(?:\\[0-9A-Fa-f]{2}|\\\[^=\,\\]|[^=\,\\]+)+))(?:\s*\,\s*(?<RDN>(?<Key>(?:\\[0-9A-Fa-f]{2}|\\\[^=\,\\]|[^=\,\\]+)+)\=(?<Value>(?:\\[0-9A-Fa-f]{2}|\\\[^=\,\\]|[^=\,\\]+)+)))*$
Test:
CN=Test User Delete\0ADEL:c1104f63-0389-4d25-8e03-822a5c3616bc,CN=Deleted Objects,DC=test,DC=domain,DC=local
The expression is already Regex escaped so to avoid having to repeat all the backslashes in C# make sure you prefix the string with the non-escaped literal # sign, i.e.
var dnExpression = #"...";
This will yield four groups, first a copy of the whole string, second a copy of the last RDN, third and fourth the key/value pairs. You can index into each key/value using the Captures collection of each group.
You can also use this to validate a RDN by cutting the expression to the "(?...)" group surrounded by the usual "^...$" to required a whole value (start-end of string).
I've allowed a hex special character escape "\", simple character escape "\" or anything other than ",=\" inside the key/value DN text. I'd guess this expression could be perfected by taking extra time to go through the MSDN AD standard and restrict the allowed characters to match exactly what is or is not allowed. But I believe this is a good start.

I created one. Working great.
^(\w+[=]{1}\w+)([,{1}]\w+[=]{1}\w+)*$

Why is my RegExp construction not accepted by JavaScript?

I'm using a RegExp to validate some user input on an ASP.NET web page. It's meant to enforce the construction of a password (i.e. between 8 and 20 long, at least one upper case character, at least one lower case character, at least one number, at least one of the characters ##!$% and no use of letters L or O (upper or lower) or numbers 0 and 1. This RegExp works fine in my tester (Expresso) and in my C# code.
This is how it looks:
(?-i)^(?=.{8,20})(?=.*[2-9])(?=.*[a-hj-km-np-z])(?=.*[A-HJ-KM-NP-Z])
(?=.*[##!$%])[2-9a-hj-km-np-zA-HJ-KM-NP-Z##!$%]*$
(Line break added for formatting)
However, when I run the code it lives in in IE6 or IE7 (haven't tried other browsers as this is an internal app and we're a Microsoft shop), I get a runtime error saying 'Syntax error in regular expression'. That's it - no further information in the error message aside from the line number.
What is it about this that JavaScript doesn't like?

Well, there are two ways of defining a Regex in Javascript:
a. Through a Regexp object constructor:
var re = new RegExp("pattern","flags");
re.test(myTestString);
b. Using a string literal:
var re = /pattern/flags;
You should also note that JS does not support some of the tenets of Regular Expressions. For a non-comprehensive list of features unsupported in JS, check out the regular-expressions.info site.
Specifically speaking, you appear to be setting some flags on the expression (for example, the case insensitive flag). I would suggest that you use the /i flag (as indicated by the syntax above) instead of using (?-i)
That would make your Regex as follows (Positive Lookahead appears to be supported):
/^(?=.{8,20})(?=.*[2-9])(?=.*[a-hj-km-np-z])(?=.*[A-HJ-KM-NP-Z])(?=.*[##!$%])[2-9a-hj-km-np-zA-HJ-KM-NP-Z##!$%]*$/i;
For a very good article on the subject, check out Regular Expressions in JavaScript.
Edit (after Howard's comment)
If you are simply assigning this Regex pattern to a RegularExpressionValidator control, then you will not have the ability to set Regex options (such as ignore case). Also, you will not be able to use the Regex literal syntax supported by Javascript. Therefore, the only option that remains is to make your pattern intrinsically case insensitive. For example, [a-h] would have to be written as [A-Ha-h]. This would make your Regex quite long-winded, I'm sorry to say.
Here is a solution to this problem, though I cannot vouch for it's legitimacy. Some other options that come to mind may be to turn of Client side validation altogether and validate exclusively on the Server. This will give you access to the full Regex flavour implemented by the System.Text.RegularExpressions.Regex object. Alternatively, use a CustomValidator and create your own JS function which applies the Regex match using the patterns that I (and others) have suggested.

I'm not familiar with C#'s regular expression syntax, but is this (at the start)
(?-i)
meant to turn the case insensitivity pattern modifier on? If so, that's your problem. Javascript doesn't support specifying the pattern modifiers in the expression. There's two ways to do this in javascript
var re = /pattern/i
var re = new RegExp('pattern','i');
Give one of those a try, and your expression should be happy.

As Cerberus mentions, (?-i) is not supported in JavaScript regexps. So, you need to get rid of that and use /i. Something to keep in mind is that there is no standard for regular expression syntax; it is different in each language, so testing in something that uses the .NET regular expression engine is not a valid test of how it will work in JavaScript. Instead, try and look for a reference on JavaScript regular expressions, such as this one.
Your match that looks for 8-20 characters is also invalid. This will ensure that there are at least 8 characters, but it does not limit the string to 20, since the character class with the kleene-closure (* operator) at the end can match as many characters as provided. What you want instead is to replace the * at the end with the {8,20}, and eliminate it from the beginning.
var re = /^(?=.*[2-9])(?=.*[a-hj-km-np-z])(?=.*[A-HJ-KM-NP-Z])(?=.*[##!$%])[2-9a-hj-km-np-zA-HJ-KM-NP-Z##!$%]{8,20}$/i;
On the other hand, I'm not really sure why you would want to restrict the length of passwords, unless there's a hard database limit (which there shouldn't be, since you shouldn't be storing passwords in plain text in the database, but instead hashing them down to something fixed size using a secure hash algorithm with a salt). And as mentioned, I don't see a reason to be so restrictive on the set of characters you allow. I'd recommend something more like this:
var re = /^(?=.*[0-9])(?=.*[a-z])(?=.*[A-Z])(?=.*[##!$%])[a-zA-Z0-9##!$%]{8,}$/i;
Also, why would you forbid 1, 0, L and O from your passwords (and it looks like you're trying to forbid I as well, which you forgot to mention)? This will make it very hard for people to construct good passwords, and since you never see a password as you type it, there's no reason to worry about letters which look confusingly similar. If you want to have a more permissive regexp:
var re = /^(?=.*[0-9])(?=.*[a-z])(?=.*[A-Z])(?=.*[##!$%]).{8,}$/i;

Are you enclosing the regexp in / / characters?
var regexp = /[]/;
return regexp.test();

(?-i)
Doesn't exist in JS Regexp. Flags can be specified as “new RegExp('pattern', 'i')”, or literal syntax “/pattern/i”.
(?=
Exists in modern implementations of JS Regexp, but is dangerously buggy in IE. Lookahead assertions should be avoided in JS for this reason.
between 8 and 20 long, at least one upper case character, at least one lower case character, at least one number, at least one of the characters ##!$% and no use of letters L or O (upper or lower) or numbers 0 and 1.
Do you have to do this in RegExp, and do you have to put all the conditions in one RegExp? Because those are easy conditions to match using multiple RegExps, or even simple string matching:
if (
s.length<8 || s.length>20 ||
s==s.toLowerCase() || s==s.toUpperCase() ||
s.indexOf('0')!=-1 || s.indexOf('1')!=-1 ||
s.toLowerCase().indexOf('l')!=-1 || s.toLowerCase().indexOf('o')!=-1 ||
(s.indexOf('#')==-1 && s.indexOf('#')==-1 && s.indexOf('!')==-1 && s.indexOf('%')==-1 && s.indexOf('%')==-1)
)
alert('Bad password!');
(These are really cruel and unhelpful password rules if meant for end-users BTW!)

I would use this regular expression:
/(?=[^2-9]*[2-9])(?=[^a-hj-km-np-z]*[a-hj-km-np-z])(?=[^A-HJ-KM-NP-Z]*[A-HJ-KM-NP-Z])(?=[^##!$%]*[##!$%])^[2-9a-hj-km-np-zA-HJ-KM-NP-Z##!$%]{8,}$/
The [^a-z]*[a-z] will make sure that the match is made as early as possible instead of expanding the .* and doing backtracking.

(?-i) is supposed to turn case-insensitivity off. Everybody seems to be assuming you're trying to turn it on, but that would be (?i). Anyway, you don't want it to be case-insensitive, since you need to ensure that there are both uppercase and lowercase letters. Since case-sensitive matching is the default, prefacing a regex with (?-i) is pointless even in those flavors (like .NET) that support inline modifiers.