nodejs compare Unicode file names - javascript

We're trying to compare the same file name, one is set by = operator, the other is returned by getting the file from Nodejs server after uploaded, as the following code block:
var name = "tên_đẹp.WAV";
// uploaded_file is the file (tên_đẹp) returned by calling an ajax function
// to get the uploaded file in uploaded_folder of a Nodejs server
ajaxUploadFile(name).done( function(e, uploaded_file) {
if(name == uploaded_file.name) {
return uploaded_file; // This line is never reached
else {
console.log(escape(name)); // t%EAn_%u0111%u1EB9p.WAV
console.log(escape(uploaded_file.name)); // te%u0302n_%u0111e%u0323p.WAV
}
}
As you can see the result of the 2 escape commands are different.
I don't know why they use different unicode format and how can I make them use the same Unicode charset or any solution would be much appreciated?
Thanks.

The issue is that "e\u0302" and "\u00EA" are both visually identical. One is the specific character U+00EA (LATIN SMALL LETTER E WITH CIRCUMFLEX), and the other is e with the combining character U+0302 (COMBINING CIRCUMFLEX ACCENT). You must normalize each string to a standard form first to compare them.
require('unorm');
var name = "tên_đẹp.WAV";
// uploaded_file is the file (tên_đẹp) returned by calling an ajax function
// to get the uploaded file in uploaded_folder of a Nodejs server
ajaxUploadFile(name).done( function(e, uploaded_file) {
if(name.normalize() == uploaded_file.name.normalize()) {
return uploaded_file; // This line is never reached
else {
console.log(escape(name)); // t%EAn_%u0111%u1EB9p.WAV
console.log(escape(uploaded_file.name)); // te%u0302n_%u0111e%u0323p.WAV
}
}
Note that I've loaded the unorm module, which polyfills in the .normalize() method being called on the strings. This method is part of ECMA6, and in future versions of Node you will not need to load unorm at all.
It's impossible to say what introduced the differences there, it could have been your text editor or your browser.

%EA == ê
e%u0302 == e + ^
These are two unicode sequences that look the same, but typed differently. If you're need to compare them, you'll have to do unicode normalization first.

The unicode characters in uploaded_file.name are accents. %u0302 is a diacritical mark COMBINING CIRCUMFLEX ACCENT, %u0323 is a diactritical mark COMBINING DOT BELOW.
On the other hand, %EA (ê) and %u1EB9 (ẹ) are the equivalent characters with accents integrated.
This is something handled by Unicode equivalence (see Wikipedia). The sequence of e%u0302 is said to be canonicaly equivalent to %EA, and similarly for the other pair.
To handle the comparison properly in node.js, you have to normalize the strings into a canonical form (NFC or NFD). This can be achieved with unorm:
var unorm = require('unorm');
var s1 = 'êẹ';
var s2 = 'e\u0302e\u0323';
console.log(s1 == s2); // false
console.log(unorm.nfc(s1) == unorm.nfc(s2)); // true
console.log(unorm.nfd(s1) == unorm.nfd(s2)); // true
The choice between NFC (composed) and NFD (decomposed) should not matter in this case.
Important: Note that canonicalization can sometimes introduce nonobvious exploitable vulnerabilities, especially with filenames, as the OS would likely still see them as different. E.g. see this story of spotify: Creative usernames and Spotify account hijacking.

Related

Capitalise first letter in the one line code [duplicate]

How do I make the first character of a string uppercase if it's a letter, but not change the case of any of the other letters?
For example:
"this is a test" → "This is a test"
"the Eiffel Tower" → "The Eiffel Tower"
"/index.html" → "/index.html"
function capitalizeFirstLetter(string) {
return string.charAt(0).toUpperCase() + string.slice(1);
}
Some other answers modify String.prototype (this answer used to as well), but I would advise against this now due to maintainability (hard to find out where the function is being added to the prototype and could cause conflicts if other code uses the same name/a browser adds a native function with that same name in future).
Edited to add this DISCLAIMER: please read the comments to understand the risks of editing JS basic types.
Here's a more object-oriented approach:
Object.defineProperty(String.prototype, 'capitalize', {
value: function() {
return this.charAt(0).toUpperCase() + this.slice(1);
},
enumerable: false
});
You'd call the function, like this:
"hello, world!".capitalize();
With the expected output being:
"Hello, world!"
In CSS:
p::first-letter {
text-transform:capitalize;
}
Here is a shortened version of the popular answer that gets the first letter by treating the string as an array:
function capitalize(s)
{
return s[0].toUpperCase() + s.slice(1);
}
Update
According to the comments below this doesn't work in IE 7 or below.
Update 2:
To avoid undefined for empty strings (see #njzk2's comment below), you can check for an empty string:
function capitalize(s)
{
return s && s[0].toUpperCase() + s.slice(1);
}
ES version
const capitalize = s => s && s[0].toUpperCase() + s.slice(1)
// to always return type string event when s may be falsy other than empty-string
const capitalize = s => (s && s[0].toUpperCase() + s.slice(1)) || ""
If you're interested in the performance of a few different methods posted:
Here are the fastest methods based on this jsperf test (ordered from fastest to slowest).
As you can see, the first two methods are essentially comparable in terms of performance, whereas altering the String.prototype is by far the slowest in terms of performance.
// 10,889,187 operations/sec
function capitalizeFirstLetter(string) {
return string[0].toUpperCase() + string.slice(1);
}
// 10,875,535 operations/sec
function capitalizeFirstLetter(string) {
return string.charAt(0).toUpperCase() + string.slice(1);
}
// 4,632,536 operations/sec
function capitalizeFirstLetter(string) {
return string.replace(/^./, string[0].toUpperCase());
}
// 1,977,828 operations/sec
String.prototype.capitalizeFirstLetter = function() {
return this.charAt(0).toUpperCase() + this.slice(1);
}
I didn’t see any mention in the existing answers of issues related to astral plane code points or internationalization. “Uppercase” doesn’t mean the same thing in every language using a given script.
Initially I didn’t see any answers addressing issues related to astral plane code points. There is one, but it’s a bit buried (like this one will be, I guess!)
Overview of the hidden problem and various approaches to it
Most of the proposed functions look like this:
function capitalizeFirstLetter(str) {
return str[0].toUpperCase() + str.slice(1);
}
However, some cased characters fall outside the BMP (basic multilingual plane, code points U+0 to U+FFFF). For example take this Deseret text:
capitalizeFirstLetter("𐐶𐐲𐑌𐐼𐐲𐑉"); // "𐐶𐐲𐑌𐐼𐐲𐑉"
The first character here fails to capitalize because the array-indexed properties of strings don’t access “characters” or code points*. They access UTF-16 code units. This is true also when slicing — the index values point at code units.
It happens to be that UTF-16 code units are 1:1 with USV code points within two ranges, U+0 to U+D7FF and U+E000 to U+FFFF inclusive. Most cased characters fall into those two ranges, but not all of them.
From ES2015 on, dealing with this became a bit easier. String.prototype[##iterator] yields strings corresponding to code points**. So for example, we can do this:
function capitalizeFirstLetter([ first='', ...rest ]) {
return [ first.toUpperCase(), ...rest ].join('');
}
capitalizeFirstLetter("𐐶𐐲𐑌𐐼𐐲𐑉") // "𐐎𐐲𐑌𐐼𐐲𐑉"
For longer strings, this is probably not terribly efficient*** — we don’t really need to iterate the remainder. We could use String.prototype.codePointAt to get at that first (possible) letter, but we’d still need to determine where the slice should begin. One way to avoid iterating the remainder would be to test whether the first codepoint is outside the BMP; if it isn’t, the slice begins at 1, and if it is, the slice begins at 2.
function capitalizeFirstLetter(str) {
if (!str) return '';
const firstCP = str.codePointAt(0);
const index = firstCP > 0xFFFF ? 2 : 1;
return String.fromCodePoint(firstCP).toUpperCase() + str.slice(index);
}
capitalizeFirstLetter("𐐶𐐲𐑌𐐼𐐲𐑉") // "𐐎𐐲𐑌𐐼𐐲𐑉"
You could use bitwise math instead of > 0xFFFF there, but it’s probably easier to understand this way and either would achieve the same thing.
We can also make this work in ES5 and below by taking that logic a bit further if necessary. There are no intrinsic methods in ES5 for working with codepoints, so we have to manually test whether the first code unit is a surrogate****:
function capitalizeFirstLetter(str) {
if (!str) return '';
var firstCodeUnit = str[0];
if (firstCodeUnit < '\uD800' || firstCodeUnit > '\uDFFF') {
return str[0].toUpperCase() + str.slice(1);
}
return str.slice(0, 2).toUpperCase() + str.slice(2);
}
capitalizeFirstLetter("𐐶𐐲𐑌𐐼𐐲𐑉") // "𐐎𐐲𐑌𐐼𐐲𐑉"
Deeper into internationalization (whose capitalization?)
At the start I also mentioned internationalization considerations. Some of these are very difficult to account for because they require knowledge not only of what language is being used, but also may require specific knowledge of the words in the language. For example, the Irish digraph "mb" capitalizes as "mB" at the start of a word. Another example, the German eszett, never begins a word (afaik), but still helps illustrate the problem. The lowercase eszett (“ß”) capitalizes to “SS,” but “SS” could lowercase to either “ß” or “ss” — you require out-of-band knowledge of the German language to know which is correct!
The most famous example of these kinds of issues, probably, is Turkish. In Turkish Latin, the capital form of i is İ, while the lowercase form of I is ı — they’re two different letters. Fortunately we do have a way to account for this:
function capitalizeFirstLetter([ first='', ...rest ], locale) {
return [ first.toLocaleUpperCase(locale), ...rest ].join('');
}
capitalizeFirstLetter("italy", "en") // "Italy"
capitalizeFirstLetter("italya", "tr") // "İtalya"
In a browser, the user’s most-preferred language tag is indicated by navigator.language, a list in order of preference is found at navigator.languages, and a given DOM element’s language can be obtained (usually) with Object(element.closest('[lang]')).lang || YOUR_DEFAULT_HERE in multilanguage documents.
In agents which support Unicode property character classes in RegExp, which were introduced in ES2018, we can clean stuff up further by directly expressing what characters we’re interested in:
function capitalizeFirstLetter(str, locale=navigator.language) {
return str.replace(/^\p{CWU}/u, char => char.toLocaleUpperCase(locale));
}
This could be tweaked a bit to also handle capitalizing multiple words in a string with fairly good accuracy for at least some languages, though outlying cases will be hard to avoid completely if doing so no matter what the primary language is.
The CWU or Changes_When_Uppercased character property matches all code points which change when uppercased in the generic case where specific locale data is absent. There are other interesting case-related Unicode character properties that you may wish to play around with. It’s a cool zone to explore but we’d go on all day if we enumerated em all here. Here’s something to get your curiosity going if you’re unfamiliar, though: \p{Lower} is a larger group than \p{LowercaseLetter} (aka \p{Ll}) — conveniently illustrated by the default character set comparison in this tool provided by Unicode. (NB: not everything you can reference there is also available in ES regular expressions, but most of the stuff you’re likely to want is).
Alternatives to case-mapping in JS (Firefox & CSS love the Dutch!)
If digraphs with unique locale/language/orthography capitalization rules happen to have a single-codepoint “composed” representation in Unicode, these might be used to make one’s capitalization expectations explicit even in the absence of locale data. For example, we could prefer the composed i-j digraph, ij / U+133, associated with Dutch, to ensure a case-mapping to uppercase IJ / U+132:
capitalizeFirstLetter('ijsselmeer'); // "IJsselmeer"
On the other hand, precomposed digraphs and similar are sometimes deprecated (like that one, it seems!) and may be undesirable in interchanged text regardless due to the potential copypaste nuisance if that’s not the normal way folks type the sequence in practice. Unfortunately, in the absence of the precomposition “hint,” an explicit locale won’t help here (at least as far as I know). If we spell ijsselmeer with an ordinary i + j, capitalizeFirstLetter will produce the wrong result even if we explicitly indicate nl as the locale:
capitalizeFirstLetter('ijsselmeer', 'nl'); // "Ijsselmeer" :(
(I’m not entirely sure whether there are some such cases where the behavior comes down to ICU data availability — perhaps someone else could say.)
If the point of the transformation is to display textual content in a web browser, though, you have an entirely different option available that will likely be your best bet: leveraging features of the web platform’s other core languages, HTML and CSS. Armed with HTML’s lang=... and CSS’s text-transform:..., you’ve got a (pseudo-)declarative solution that leaves extra room for the user agent to be “smart.” A JS API needs to have predictable outcomes across all browsers (generally) and isn’t free to experiment with heuristics. The user-agent itself is obligated only to its user, though, and heuristic solutions are fair game when the output is for a human being. If we tell it “this text is Dutch, but please display it capitalized,” the particular outcome might now vary between browsers, but it’s likely going to be the best each of them could do. Let’s see:
<!DOCTYPE html>
<dl>
<dt>Untransformed
<dd>ijsselmeer
<dt>Capitalized with CSS and <code>lang=en</code>
<dd lang="en" style="text-transform: capitalize">ijsselmeer
<dt>Capitalized with CSS and <code>lang=nl</code>
<dd lang="nl" style="text-transform: capitalize">ijsselmeer
In Chromium at the time of writing, both the English and Dutch lines come out as Ijsselmeer — so it does no better than JS. But try it in current Firefox! The element that we told the browser contains Dutch will be correctly rendered as IJsselmeer there.
This solution is purpose-specific (it’s not gonna help you in Node, anyway) but it was silly of me not to draw attention to it previously given some folks might not realize they’re googling the wrong question. Thanks #paul23 for clarifying more about the nature of the IJ digraph in practice and prompting further investigation!
As of January 2021, all major engines have implemented the Unicode property character class feature, but depending on your target support range you may not be able to use it safely yet. The last browser to introduce support was Firefox (78; June 30, 2020). You can check for support of this feature with the Kangax compat table. Babel can be used to compile RegExp literals with property references to equivalent patterns without them, but be aware that the resulting code can sometimes be enormous. You probably would not want to do this unless you’re certain the tradeoff is justified for your use case.
In all likelihood, people asking this question will not be concerned with Deseret capitalization or internationalization. But it’s good to be aware of these issues because there’s a good chance you’ll encounter them eventually even if they aren’t concerns presently. They’re not “edge” cases, or rather, they’re not by-definition edge cases — there’s a whole country where most people speak Turkish, anyway, and conflating code units with codepoints is a fairly common source of bugs (especially with regard to emoji). Both strings and language are pretty complicated!
* The code units of UTF-16 / UCS2 are also Unicode code points in the sense that e.g. U+D800 is technically a code point, but that’s not what it “means” here ... sort of ... though it gets pretty fuzzy. What the surrogates definitely are not, though, is USVs (Unicode scalar values).
** Though if a surrogate code unit is “orphaned” — i.e., not part of a logical pair — you could still get surrogates here, too.
*** maybe. I haven’t tested it. Unless you have determined capitalization is a meaningful bottleneck, I probably wouldn’t sweat it — choose whatever you believe is most clear and readable.
**** such a function might wish to test both the first and second code units instead of just the first, since it’s possible that the first unit is an orphaned surrogate. For example the input "\uD800x" would capitalize the X as-is, which may or may not be expected.
For another case I need it to capitalize the first letter and lowercase the rest. The following cases made me change this function:
//es5
function capitalize(string) {
return string.charAt(0).toUpperCase() + string.slice(1).toLowerCase();
}
capitalize("alfredo") // => "Alfredo"
capitalize("Alejandro")// => "Alejandro
capitalize("ALBERTO") // => "Alberto"
capitalize("ArMaNdO") // => "Armando"
// es6 using destructuring
const capitalize = ([first,...rest]) => first.toUpperCase() + rest.join('').toLowerCase();
This is the 2018 ECMAScript 6+ Solution:
const str = 'the Eiffel Tower';
const newStr = `${str[0].toUpperCase()}${str.slice(1)}`;
console.log('Original String:', str); // the Eiffel Tower
console.log('New String:', newStr); // The Eiffel Tower
If you're already (or considering) using Lodash, the solution is easy:
_.upperFirst('fred');
// => 'Fred'
_.upperFirst('FRED');
// => 'FRED'
_.capitalize('fred') //=> 'Fred'
See their documentation: https://lodash.com/docs#capitalize
_.camelCase('Foo Bar'); //=> 'fooBar'
https://lodash.com/docs/4.15.0#camelCase
_.lowerFirst('Fred');
// => 'fred'
_.lowerFirst('FRED');
// => 'fRED'
_.snakeCase('Foo Bar');
// => 'foo_bar'
Vanilla JavaScript for first upper case:
function upperCaseFirst(str){
return str.charAt(0).toUpperCase() + str.substring(1);
}
There is a very simple way to implement it by replace. For ECMAScript 6:
'foo'.replace(/^./, str => str.toUpperCase())
Result:
'Foo'
Capitalize the first letter of all words in a string:
function ucFirstAllWords( str )
{
var pieces = str.split(" ");
for ( var i = 0; i < pieces.length; i++ )
{
var j = pieces[i].charAt(0).toUpperCase();
pieces[i] = j + pieces[i].substr(1);
}
return pieces.join(" ");
}
CSS only
If the transformation is needed only for displaying on a web page:
p::first-letter {
text-transform: uppercase;
}
Despite being called "::first-letter", it applies to the first character, i.e. in case of string %a, this selector would apply to % and as such a would not be capitalized.
In IE9+ or IE5.5+ it's supported in legacy notation with only one colon (:first-letter).
ES2015 one-liner
const capitalizeFirstChar = str => str.charAt(0).toUpperCase() + str.substring(1);
Remarks
In the benchmark I performed, there was no significant difference between string.charAt(0) and string[0]. Note however, that string[0] would be undefined for an empty string, so the function would have to be rewritten to use "string && string[0]", which is way too verbose, compared to the alternative.
string.substring(1) is faster than string.slice(1).
Benchmark between substring() and slice()
The difference is rather minuscule nowadays (run the test yourself):
21,580,613.15 ops/s ±1.6% for substring(),
21,096,394.34 ops/s ±1.8% (2.24% slower) for slice().
It's always better to handle these kinds of stuff using CSS first, in general, if you can solve something using CSS, go for that first, then try JavaScript to solve your problems, so in this case try using :first-letter in CSS and apply text-transform:capitalize;
So try creating a class for that, so you can use it globally, for example: .first-letter-uppercase and add something like below in your CSS:
.first-letter-uppercase:first-letter {
text-transform:capitalize;
}
Also the alternative option is JavaScript, so the best gonna be something like this:
function capitalizeTxt(txt) {
return txt.charAt(0).toUpperCase() + txt.slice(1); //or if you want lowercase the rest txt.slice(1).toLowerCase();
}
and call it like:
capitalizeTxt('this is a test'); // return 'This is a test'
capitalizeTxt('the Eiffel Tower'); // return 'The Eiffel Tower'
capitalizeTxt('/index.html'); // return '/index.html'
capitalizeTxt('alireza'); // return 'Alireza'
capitalizeTxt('dezfoolian'); // return 'Dezfoolian'
If you want to reuse it over and over, it's better attach it to javascript native String, so something like below:
String.prototype.capitalizeTxt = String.prototype.capitalizeTxt || function() {
return this.charAt(0).toUpperCase() + this.slice(1);
}
and call it as below:
'this is a test'.capitalizeTxt(); // return 'This is a test'
'the Eiffel Tower'.capitalizeTxt(); // return 'The Eiffel Tower'
'/index.html'.capitalizeTxt(); // return '/index.html'
'alireza'.capitalizeTxt(); // return 'Alireza'
String.prototype.capitalize = function(allWords) {
return (allWords) ? // If all words
this.split(' ').map(word => word.capitalize()).join(' ') : // Break down the phrase to words and then recursive
// calls until capitalizing all words
this.charAt(0).toUpperCase() + this.slice(1); // If allWords is undefined, capitalize only the first word,
// meaning the first character of the whole string
}
And then:
"capitalize just the first word".capitalize(); ==> "Capitalize just the first word"
"capitalize all words".capitalize(true); ==> "Capitalize All Words"
Update November 2016 (ES6), just for fun:
const capitalize = (string = '') => [...string].map( // Convert to array with each item is a char of
// string by using spread operator (...)
(char, index) => index ? char : char.toUpperCase() // Index true means not equal 0, so (!index) is
// the first character which is capitalized by
// the `toUpperCase()` method
).join('') // Return back to string
then capitalize("hello") // Hello
SHORTEST 3 solutions, 1 and 2 handle cases when s string is "", null and undefined:
s&&s[0].toUpperCase()+s.slice(1) // 32 char
s&&s.replace(/./,s[0].toUpperCase()) // 36 char - using regexp
'foo'.replace(/./,x=>x.toUpperCase()) // 31 char - direct on string, ES6
let s='foo bar';
console.log( s&&s[0].toUpperCase()+s.slice(1) );
console.log( s&&s.replace(/./,s[0].toUpperCase()) );
console.log( 'foo bar'.replace(/./,x=>x.toUpperCase()) );
We could get the first character with one of my favorite RegExp, looks like a cute smiley: /^./
String.prototype.capitalize = function () {
return this.replace(/^./, function (match) {
return match.toUpperCase();
});
};
And for all coffee-junkies:
String::capitalize = ->
#replace /^./, (match) ->
match.toUpperCase()
...and for all guys who think that there's a better way of doing this, without extending native prototypes:
var capitalize = function (input) {
return input.replace(/^./, function (match) {
return match.toUpperCase();
});
};
Here is a function called ucfirst()(short for "upper case first letter"):
function ucfirst(str) {
var firstLetter = str.substr(0, 1);
return firstLetter.toUpperCase() + str.substr(1);
}
You can capitalise a string by calling ucfirst("some string") -- for example,
ucfirst("this is a test") --> "This is a test"
It works by splitting the string into two pieces. On the first line it pulls out firstLetter and then on the second line it capitalises firstLetter by calling firstLetter.toUpperCase() and joins it with the rest of the string, which is found by calling str.substr(1).
You might think this would fail for an empty string, and indeed in a language like C you would have to cater for this. However in JavaScript, when you take a substring of an empty string, you just get an empty string back.
Use:
var str = "ruby java";
console.log(str.charAt(0).toUpperCase() + str.substring(1));
It will output "Ruby java" to the console.
If you use Underscore.js or Lodash, the underscore.string library provides string extensions, including capitalize:
_.capitalize(string) Converts first letter of the string to
uppercase.
Example:
_.capitalize("foo bar") == "Foo bar"
If you're ok with capitalizing the first letter of every word, and your usecase is in HTML, you can use the following CSS:
<style type="text/css">
p.capitalize {text-transform:capitalize;}
</style>
<p class="capitalize">This is some text.</p>
This is from CSS text-transform Property (at W3Schools).
var capitalized = yourstring[0].toUpperCase() + yourstring.substr(1);
If you are wanting to reformat all-caps text, you might want to modify the other examples as such:
function capitalize (text) {
return text.charAt(0).toUpperCase() + text.slice(1).toLowerCase();
}
This will ensure that the following text is changed:
TEST => Test
This Is A TeST => This is a test
String.prototype.capitalize = function(){
return this.replace(/(^|\s)([a-z])/g,
function(m, p1, p2) {
return p1 + p2.toUpperCase();
});
};
Usage:
capitalizedString = someString.capitalize();
This is a text string => This Is A Text String
function capitalize(s) {
// returns the first letter capitalized + the string from index 1 and out aka. the rest of the string
return s[0].toUpperCase() + s.substr(1);
}
// examples
capitalize('this is a test');
=> 'This is a test'
capitalize('the Eiffel Tower');
=> 'The Eiffel Tower'
capitalize('/index.html');
=> '/index.html'
yourString.replace(/\w/, c => c.toUpperCase())
I found this arrow function easiest. Replace matches the first letter character (\w) of your string and converts it to uppercase. Nothing fancier is necessary.
var str = "test string";
str = str.substring(0,1).toUpperCase() + str.substring(1);
𝗔 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗧𝗵𝗮𝘁 𝗪𝗼𝗿𝗸𝘀 𝗙𝗼𝗿 𝗔𝗹𝗹 𝗨𝗻𝗶𝗰𝗼𝗱𝗲 𝗖𝗵𝗮𝗿𝗮𝗰𝘁𝗲𝗿𝘀
57 81 different answers for this question, some off-topic, and yet none of them raise the important issue that none of the solutions listed will work with Asian characters, emoji's, and other high Unicode-point-value characters in many browsers. Here is a solution that will:
const consistantCapitalizeFirstLetter = "\uD852\uDF62".length === 1 ?
function(S) {
"use-strict"; // Hooray! The browser uses UTF-32!
return S.charAt(0).toUpperCase() + S.substring(1);
} : function(S) {
"use-strict";
// The browser is using UCS16 to store UTF-16
var code = S.charCodeAt(0)|0;
return (
code >= 0xD800 && code <= 0xDBFF ? // Detect surrogate pair
S.slice(0,2).toUpperCase() + S.substring(2) :
S.charAt(0).toUpperCase() + S.substring(1)
);
};
const prettyCapitalizeFirstLetter = "\uD852\uDF62".length === 1 ?
function(S) {
"use-strict"; // Hooray! The browser uses UTF-32!
return S.charAt(0).toLocaleUpperCase() + S.substring(1);
} : function(S) {
"use-strict";
// The browser is using UCS16 to store UTF-16
var code = S.charCodeAt(0)|0;
return (
code >= 0xD800 && code <= 0xDBFF ? // Detect surrogate pair
S.slice(0,2).toLocaleUpperCase() + S.substring(2) :
S.charAt(0).toLocaleUpperCase() + S.substring(1)
);
};
Do note that the above solution tries to account for UTF-32. However, the specification officially states that browsers are required to do everything in UTF-16 mapped into UCS2. Nevertheless, if we all come together, do our part, and start preparing for UTF32, then there is a chance that the TC39 may allow browsers to start using UTF-32 (like how Python uses 24-bits for each character of the string). This must seem silly to an English speaker: no one who uses only latin-1 has ever had to deal with Mojibake because Latin-I is supported by all character encodings. But, users in other countries (such as China, Japan, Indonesia, etc.) are not so fortunate. They constantly struggle with encoding problems not just from the webpage, but also from the JavaScript: many Chinese/Japanese characters are treated as two letters by JavaScript and thus may be broken apart in the middle, resulting in � and � (two question-marks that make no sense to the end user). If we could start getting ready for UTF-32, then the TC39 might just allow browsers do what Python did many years ago which had made Python very popular for working with high Unicode characters: using UTF-32.
consistantCapitalizeFirstLetter works correctly in Internet Explorer 3+ (when the const is changed to var). prettyCapitalizeFirstLetter requires Internet Explorer 5.5+ (see the top of page 250 of this document). However, these fact are more of just jokes because it is very likely that the rest of the code on your webpage will not even work in Internet Explorer 8 - because of all the DOM and JScript bugs and lack of features in these older browsers. Further, no one uses Internet Explorer 3 or Internet Explorer 5.5 any more.
Check out this solution:
var stringVal = 'master';
stringVal.replace(/^./, stringVal[0].toUpperCase()); // Returns Master
Only because this is really a one-liner I will include this answer. It's an ES6-based interpolated string one-liner.
let setStringName = 'the Eiffel Tower';
setStringName = `${setStringName[0].toUpperCase()}${setStringName.substring(1)}`;
with arrow function
let fLCapital = s => s.replace(/./, c => c.toUpperCase())
fLCapital('this is a test') // "This is a test"
with arrow function, another solution
let fLCapital = s => s = s.charAt(0).toUpperCase() + s.slice(1);
fLCapital('this is a test') // "This is a test"
with array and map()
let namesCapital = names => names.map(name => name.replace(/./, c => c.toUpperCase()))
namesCapital(['james', 'robert', 'mary']) // ["James", "Robert", "Mary"]

Javascript RegEx causes message loop and crashes browser when set to anything other than /mg

I've created a script (my first) that accepts input text and then runs about 30 regular expressions on the text, outputting discovered values into an HTML table. The purpose is for me to be able to paste in text containing specification information for products and have them uniformly outputted so I can paste them into a spreadsheet.
I've had it working really well and I've been tuning the regexes as I've pasted data with different variations/layouts in. However, I've hit an impasse and need some assistance:
One of the regular expressions searches for the product part number (sku) and returns the value in a column. Some of the source data includes more than one match because there are regional variations to the products. In all cases the first match is the only one that I want. I've tested the RegEx on RegEx101 and it returns the first match only with the 'global' flag switched off. However, the same RegEx running in my script causes it to return console messages infinitely before crashing. It's immediately unresponsive so I can't see any error messages.
Here's a sample of the regex section in my script. sku being the one that's causing problems:
let wallMountable = /(?<wallmountable>[Ww]all [Mm]ount)/mg;
let sku = /^.*\((?<sku>\d\d\w\w[\d\w]+?).+?\).*$/m;
function parseData() {
// Execute the Regular Expressions on the text in 'specSheet'
let specSheet = document.getElementById("specSheet").value;
let wallMountableToken = wallMountable.exec(specSheet);
let skuToken = sku.exec(specSheet);
do {
// If Token isn't null, then test to see if the regex group value is undefined. If either are true, do nothing, otherwise write the value to the document.
if (wallMountableToken !== null) {
if (wallMountableToken.groups.wallmountable !== undefined)
{document.write(`${wallMountableToken.groups.wallmountable}`);}
else {}
}
else {}
if (skuToken !== null) {
if (skuToken.groups.sku !== undefined)
{document.write(`${skuToken.groups.sku}`);}
else {}
}
else {}
}
// Loop through the script until a condition is met.
while (
(wallMountableToken = wallMountable.exec(specSheet)) !== null,
(skuToken = sku.exec(specSheet)) !== null
);
}
The while loop may not be necessary and in truth I'm not entirely sure what purpose it serves here, but it seemed to consistently appear in the reference material I was studying. I have included it here because it's part of the script, but please note that the script works if I change the second regex to /mg instead of /m, however, it returns multiple values and I only want it to return the first capture.
I know there's a lot wrong with the script, but this particular question is about why the regex is causing an infinite loop, so any help toward that goal is much appreciated.

Sync storage, max quota bytes per item and chunked data storage

I'm using a modified version of this code (Update: that answer has since been updated to use correct code, but this question still carries value since it contains relevant test cases and discussions for this problem) to store a single object after stringification in chunked keys inside of sync storage.
Note that sync storage has a maximum quota size per item. So, I have those maxLengthPerItem and maxValueLength variables.
function lengthInUtf8Bytes(str) {
// by: https://stackoverflow.com/a/5515960/2675672
// Matches only the 10.. bytes that are non-initial characters in a multi-byte sequence.
var m = encodeURIComponent(str).match(/%[89ABab]/g);
return str.length + (m ? m.length : 0);
}
function syncStore(key, objectToStore, callback) {
var jsonstr = JSON.stringify(objectToStore), i = 0, storageObj = {},
// (note: QUOTA_BYTES_PER_ITEM only on sync storage)
// subtract two for the quotes added by stringification
// extra -5 to err on the safe side
maxBytesPerItem = chrome.storage.sync.QUOTA_BYTES_PER_ITEM - NUMBER,
// since the key uses up some per-item quota, use
// "maxValueBytes" to see how much is left for the value
maxValueBytes, index, segment, counter;
console.log("jsonstr length is " + lengthInUtf8Bytes(jsonstr));
// split jsonstr into chunks and store them in an object indexed by `key_i`
while(jsonstr.length > 0) {
index = key + "_" + i++;
maxValueBytes = maxBytesPerItem - lengthInUtf8Bytes(index);
counter = maxValueBytes;
segment = jsonstr.substr(0, counter);
while(lengthInUtf8Bytes(segment) > maxValueBytes)
segment = jsonstr.substr(0, --counter);
storageObj[index] = segment;
jsonstr = jsonstr.substr(counter);
}
// later used by retriever function
storageObj[key] = i;
console.log((i + 1) + " keys used (= key + key_i)");
// say user saves till chunk 20 in case I
// in case II, user deletes several snippets and brings down
// total no. of "required" chunks to 15; however, the previous chunks
// (16-20) remain in memory unless they are "clear"ed.
chrome.storage.sync.clear(function(){
console.log(storageObj);
console.log(chrome.storage.sync);
chrome.storage.sync.set(storageObj, callback);
});
}
The problem is in this line:
maxLengthPerItem = chrome.storage.sync.QUOTA_BYTES_PER_ITEM - NUMBER,
The problem is that 5 is the minimum NUMBER for which there's no error. Here's the sample code you can use to test my theory:
var len = 102000,
string = [...new Array(len)].map(x => 1).join(""),
Data = {
"my_text": string
},
key = "key";
syncStore(key, Data, function(){
console.log(chrome.runtime.lastError && chrome.runtime.lastError.message);
});
Using 4 yields MAX_QUOTA_BYTES_PER_ITEM exceed error. You can yourself adjust the value of len (to 20000, 60000 < 102000, etc.) to check my theory.
Question:
Why is the current method requiring exactly 5 as the minimum value? I know there's two quotes for stringification, but what about the other 3 characters? Where'd they come from?
Additionally, I've noticed that in textual Data like this one,
even 5 does not work. In the specific case above, minimum NUMBER required is 6.
Clarification:
The point of my question is not what are the other means to store data in sync.
The point of my question is why is the current method requiring exactly 5 (And why that textual data requires a 6.) Imho, my question is very specific and surely does not deserve a close vote.
Update: I've added new code which stores data based on measurement of length of UTF-8 bytes, but it still does not provide desirable results. I've also added code to more easily test my theory.
The problem is that Chrome applies JSON.stringify to each string chunk before storing it, which adds three \ characters to the first string (which, added to the known 2 for outer quotes, makes a full 5). This behavior is noted in the Chromium source code: Calculate the setting size based on its JSON serialization size (and the implementation does indeed compute size based on key.size() + value_as_json.size()).
That is, the value in key_0 is the string
{"my_text":"11111111...
But it is stored as
"{\"my_text\":\"11111111..."
The reason you need to account for the two outer quotes is the same reason you need to account for added slashes. Both are indicative of the output of JSON.stringify operating on a string input.
You can confirm that escape-slashes are the issue by doing
var jsonstr = JSON.stringify(objectToStore).replace(/"/g,"Z")
And observing that the required NUMBER offset is 2 instead of 5, because {Zmy_textZ:Z11111... does not have extra slashes.
I haven't looked closely, but the Lorem text contains a newline and a tab (see: id faucibus diam.\), which your JSON.stringify (correctly) turns into \n\t but then Chrome's additional stringify further expands to \\n\\t, for an extra 2 bytes you do not account for. If that gets chunked with two other quotes or other escapable characters, it could cause a chunk with 4 unaccounted-for bytes.
The solution here is to account for the escaping that Chrome will do upon storage. I'd suggest applying JSON.stringify to each segment when evaluating if it's too big, so that the correct number of bytes will be consumed by the chunking algorithm. Then, once you decide on a size that will not cause problems, even after being double-stringifed, consume that many bytes from the regular string. Something like:
while(lengthInUtf8Bytes(JSON.stringify(segment)) > maxValueBytes)
...
Note that this will automatically account for the two bytes from outer quotes, so there's no need to even have a QUOTA_BYTES_PER_ITEM - NUMBER computation. In the terms you've presented it, with this approach, the NUMBER is 0.
For some reason, the technique only works when we do this:
while(lengthInUtf8Bytes(JSON.stringify(JSON.stringify(segment))) > maxValueBytes)
here's a paste containing data that you can use to compare both this and #apsiller's original approach (and verify the fact that only the above approach works).
Here's the code I used to test all this stuff
I am not accepting either answer yet since neither of them provides an acceptable logic as to why only the above approach is working.
After carefully reading through this thread I finally was able to understand where the extra bytes come from. apsillers actually reference the part from the chromium code that holds the answer:
key.size() + value_as_json.size()
You have to account for the side of the key as well. So the working accurate check is:
while((lengthInUtf8Bytes(JSON.stringify(segment)) + key.length) > maxValueBytes)

NodeJS (General JS) comparing IP address

I'm using the following code to compare returned IP address (Using node-restify which is similar to express):
var checkIP = function (config, req) {
var ip = req.connection.remoteAddress.split('.'),
curIP,
b,
block = [];
for (var i=0, z=config.ips.length-1; i<=z; i++) {
curIP = config.ips[i].split('.');
b = 0;
// Compare each block
while (b<=3) {
(curIP[b]===ip[b] || curIP[b]==='*') ? block[b] = true : block[b] = false;
b++;
}
// Check all blocks
if (block[0] && block[1] && block[2] && block[3]) {
return true;
}
}
return false;
};
config.ips contains an array which (as should be obvious from the code) can be specific or wildcarded IPs.
This works, but it seems like there is a more efficient way to do this. Just curious if anyone has any suggestions on a way to simplify this or make it more efficient. My request time nearly doubled when I introduced this and I'd like to squeeze out some load time if possible.
If my intuition is correct, you might be doing a bunch of extra work right now:
For each IP expression in your config.ips array, your code is parsing and comparing:
if (block[0] && block[1] && block[2] && block[3]) {
return true;
}
^^^ Note that you have already done work to get all 4 blocks in the iterations of calculating this expression 4 times per IP: (curIP[b]===ip[b] || curIP[b]==='*'), so the ANDing above is not preventing the overhead of the work that is already happening regardless.
I have 2 ideas for you:
Since IP addresses are strings anyways, the * notation lends itself to be suitable for a Regex to do the work, instead of your splitting and comparing? So maybe as a next step you could look into implementing a Regex to do the work, instead of .split() and compare, and test the performance of that?
Or maybe figure out how to avoid that overhead associated with comparing the parts all the time, and compare the wholes when you can? And then fall back into comparing the parts only when necessity requires it.
If you want to read some C code, here's how Apache does IP blacklisting behind the scenes. Look at the function named in_domain for some inspiration.
Good luck, hope this helps!

Optional character in input mask

How do I specify an optional character in an input mask?
I found this masked input plugin http://digitalbush.com/projects/masked-input-plugin/ and these mask definitions.
$.mask.definitions['g']="[ ]";
$.mask.definitions['h']="[aApP]";
$.mask.definitions['i']="[mM]";
$.mask.definitions['2']="[0-1]";
$.mask.definitions['6']="[0-5]";
new_mask = "29:69";
$("#txtTime").mask(new_mask);
This defines an input mask for a 12 hour time format, e.g. 11:00. I'd like to allow the user to specify only one digit for the "hour". Instead of having to type 01:00, the user should be able to type 1:00. How do I do that?
You need to use ? for optional characters so in your case you should use "2?9:69".
Quote and example from the page you linked to in your question:
You can have part of your mask be optional. Anything listed after '?'
within the mask is considered optional user input. The common example
for this is phone number + optional extension.
jQuery(function($){
$("#phone").mask("(999) 999-9999? x99999");
});
You can use $.mask.definitions['g']="[0-9 ]";
I have a similar problem I am trying to solve for variable currency amounts ($0.00 - $10000.00), and I don't really understand how the plugin is working.
On the one hand, the $.mask.definitions['symbol'] = '???' bit looks like it's employing a regex fragment, based on the examples in the API reference, and somewhat confirmed by this part in the source, within the mask function's code:
$.each(mask.split(""), function (i, c) {
if (c == '?') {
len--;
partialPosition = i;
} else if (defs[c]) {
tests.push(new RegExp(defs[c]));
if (firstNonMaskPos == null)
firstNonMaskPos = tests.length - 1;
} else {
tests.push(null);
}
});
...On the other, a mask definition of []|[0-9] does not work (I am attempting to do empty string or 0-9, for those not literate in regex.) For reference, I am attempting to build a mask to fulfill the 0.00-10000.00 condition like oooo9.99, where o is []|[0-9].
Since the code confirms that this is based upon regexes, the null or range pattern should be working but aren't; this is a more obscure bug in the mask framework. Unless I am the one trying to do something incorrectly, which is also possible...

Categories