It seems + is not the right operator to handle the concatenation of strings in JavaScript. what are some alternatives to handle the both the ltr and rtl cases?
The problem is, + is not right operator to concatenate strings at all. Or maybe it is, but concatenating string is an Internationalization bug.
Instead of simply concatenating them, one should actually format them. So what you should actually do, is use placeholders:
var somePattern = "This language is written {0}.";
var someMessage = somePattern.format("LTR");
This way, the translator would be able to re-order the sentence, including word order. And I believe it solves your problem.
For formatting function, let me quote this excellent answer:
String.prototype.format = function() {
var args = arguments;
return this.replace(/\{(\d+)\}/g, function() {
return args[arguments[1]];
});
};
EDIT: Adding information about directionality marks.
Sometimes, when you have multiple placeholders you may lose the control of string direction, i.e. {0}/{1} would still be shown as first/second instead of desired second/last. To fix this, you would add Strong Directionality Mark to the pattern, i.e. {0}/{1}. is an HTML entity that resolves to Unicode code point U+200F, that is right-to-left strong directionality mark.
Actually, assuming both string are localized and you want the string on the right to be displayed logically after the string on the left, then + sometimes works fine. Strings in languages such as Arabic should be displayed RTL (right to left) on the screen, but the character ordering is still meant to be LTR (left to right) in memory. So + operator is logically consistent to use for generating an 'ordered list' of terms in any language.
But there are also scenarios where + does not solve the problem correctly. There are scenarios where the correct solution is to follow the grammar of the containing language. For instance, are you really embedding an English word in an Arabic sentence? Or vice versa? Regardless, the solution here is to do string formatting, where the containing sentence localized has a placeholder for the foreign term, like {0}.
The third case is what if there is no grammatical relationship because it is just two separate sentences? In this case there is no correct ordering. E.g. if you have an English sentence displayed in front of an Arabic sentence. An English speaker will probably read the sentences LTR (left sentence first, then right). An Arabic speaker will probably read the sentences RTL. Either way it's unclear to everyone which order the author intended the sentences to be read in. :)
Related
I have an application which uses a Javascript-based rules engine. I need a way to convert regular straight quotes into curly (or smart) quotes. It’d be easy to just do a string.replace for ["], only this will only insert one case of the curly quote.
The best way I could think of was to replace the first occurrence of a quote with a left curly quote and every other one following with a left, and the rest right curly.
Is there a way to accomplish this using Javascript?
You could replace all that preceed a word character with the left quote, and all that follow a word character with a right quote.
str = str.replace(/"(?=\w|$)/g, "“");
str = str.replace(/(?<=\w|^)"/g, "”"); // IF the language supports look-
// behind. Otherwise, see below.
As pointed out in the comments below, this doesn't take punctuation into account, but easily can:
/(?<=[\w,.?!\)]|^)"/g
[Edit:] For languages that don't support look-behind, like Javascript, as long as you replace all the front-facing ones first, you have two options:
str = str.replace(/"/g, "”"); // Replace the rest with right curly quotes
// or...
str = str.replace(/\b"/g, "”"); // Replace any quotes after a word
// boundary with right curly quotes
(I've left the original solution above in case this is helpful to someone using a language that does support look-behind)
You might want to look at what Pandoc does—apparently with the --smart option, it handles quotes properly in all cases (including e.g. ’tis and ’twere).
I recently wrote a Javascript typography prettification engine that does, among other things, quote replacement; I wound up using basically the algorithm suggested by Renesis, but there’s currently a failing test up waiting for a smarter solution.
If you’re interested in cribbing my code (and/or submitting a patch based on work you’ve done), check it out: jsPrettify. jsprettify.prettifyStr does what you’re looking for. If you don’t want to deal with the Closure dependency, there’s an older version that runs on its own—it even works in Rhino.
'foo "foo bar" "bar"'.replace(/"([-a-zA-Z0-9 ]+)"/g, function(wholeMatch, m1){
return "“" + m1 + "”";
});
The following just changes every quote by alternating (this specific example however would leave out the orphaned quotes).
str.replace(/\"([^\"]*)\"/gi,"“$1”");
Works perfectly, as long as the text you're texturizing isn't already screwed up with improper use of the double quote. In English, quotes are never nested.
I don't think something like that in general is easy at all, because you'd have to interpret exactly what each double-quote character in your content means. That said, what I'd do is collect all the text nodes I was interested in, and then go through and keep track of the "on/off" (or "odd/even"; whatever) nature of each double quote instance. Then you can know which replacement entity to use.
I didn't find the logic I wanted here, so here's what I ended up going with.
value = value.replace(/(^|\s)(")/g, "$1“"); // replace quotes that start a line or follow spaces
value = value.replace(/"/g, "”"); // replace rest of quotes with the back smart quote
I have a small textarea that I need to replace straight quotes with curly (smart) quotes. I'm just executing this logic on keyup. I tried to make it behave like Microsoft Word.
Posting for posterity.
As suggested by #Steven Dee, I went to Pandoc.
I try to use a mature and tested tool whenever I can versus baking my own regex. Hand built regex's can be overly greedy, or not greedy enough, and they may not be sensitive to word boundaries and commas etc. Pandoc accounts for most this and more.
From the command line (the --smart parameter turns on smart quotes):
pandoc --smart --standalone -o output.html input.html
..and I know a command line script may or may not fit OP's requirement of using Javascript. (related: How to execute shell command in Javascript)
I am working on a project in which i have to extract text data from a PDF.
I am able to extract text from the PDF, but extracted text sometimes contains lines which i would like to strip off from it.
Here's and example of unwanted lines -
ISBN 0-7225-3293-8. = CONTENTS = Part One Part Two Epilogue
Page 1 / 94
And, here's an example of good line (which i'd like to keep) -
Dusk was falling as the boy arrived with his herd at an abandoned church.
I wanted to sleep a little longer, he thought. He had had the same dream that night as a week ago
Different PDFs can give out different unwanted lines.
How can i detect them ?
Option 1 - Give the computer a rule: If you are able to narrow down what content it is that you would like to keep, the obvious criteria that sticks out to me is the exclusion of special characters, then you can filter your results based on this.
So let's say you agree that all "good lines" will be without special characters ('/', '-', and '=') for example, if a line DOES contain one of these items, you know you can remove it from the content you are keeping. This could be done in a for loop containing an if-then condition that looks something like this..
var lineArray = //code needed to make each line of the file an element of the array
For (cnt = 0; cnt < totalLines; cnt++)
{
var line = lineArray[cnt];
if (line.contains("/") || line.contains("-") || line.contains("="))
lineArray[cnt] = "";
}
At the end of this code you could simply get all the text within the array and it would no longer contain the unwanted lines. If there are unwanted lines however, that are virtually indistinguishable by characters, length, positioning etc. the previous approach begins to break down on some of the trickier lines.
This is because there is no rule you can give the computer to distinguish between the good and the bad without giving it a brain such as yours that recognizes parts of speech and sentence structure. In which case you might consider option 2, which is just that.
Option 2- Give the computer a brain: Given that the text you want to remove will more or less be incoherent documentation based on what you have shown us, an open source (or purchased) natural language processor may be what you are looking for.
I found a good beginner's intro at http://myreaders.info/10_Natural_Language_Processing.pdf with some information that might be of use to you. From the source,
"Linguistics is the science of language. Its study includes:
sounds (phonology),
word formation (morphology),
sentence structure (syntax),
meaning (semantics), and understanding (pragmatics) etc.
Syntactic Analysis : Here the analysis is of words in a sentence to know the grammatical structure of the sentence. The words are transformed into structures that show how the words relate to each others. Some word sequences may be rejected if they violate the rules of the language for how words may be combined. Example: An English syntactic analyzer would reject the sentence say : 'Boy the go the to store.' "
Using some sort of NLP, you can discover whether a given section of text contains a sentence or some incoherent rambling. This test could then be used as a filter in your program for what you would like to keep or remove.
Side note- As it appears your sample text is not just sentences but literature, sometimes characters will speak in sentence fragments as part of their nature given by the author. In this case, you could add a separate condition that if the text is contained within two quotations and has no special characters, you want to keep the text regardless.
In the end NLP may be more work than you require or that you want to do, in which case Option 1 is likely going to be your best bet. On the other hand, it may be just the thing you are looking for. Whatever the case or if you decide you need some combination of the two, best of luck! I hope this answer helps.
Background
I have burned myself out looking for this answer. The closest code I could find that works was from Stack Edit specifically the Markdown.Converter.js script; copied below. This is a pretty heavy hitting regular expression though, my regex for finding ** for example happens in almost 1/5 of the steps and I don't need this much extra support.
function _DoItalicsAndBold(text) {
// <strong> must go first:
text = text.replace(/([\W_]|^)(\*\*|__)(?=\S)([^\r]*?\S[\*_]*)\2([\W_]|$)/g,"$1<strong>$3</strong>$4");
text = text.replace(/([\W_]|^)(\*|_)(?=\S)([^\r\*_]*?\S)\2([\W_]|$)/g,"$1<em>$3</em>$4");
return text;
}
Question
I'm trying to make my own very simple markdown script that makes these transformations:
* ---> Italics
** ---> Bold
__ ---> Underline
So far I can find all uses of ** (two stars, bold text) with this regex:
/(\*\*)(?:(?=(\\?))\2.)*?\1/g
However I can not for the life of me figure out how to match only * (single star, italicized text) with one regular expression. If I decide to go further I may have to distinguish between _ and __ as well.
Can someone point me in the right direction on how to properly write the regular expressions that will do this?
Update / Clarifty of OP's Question
I am aware of parser's and I am afraid that this question is going to be derailed from the point. I am not asking for parser help (but I do welcome and appreciate it) I am looking specifically for regular expression help. If this helps people get away from parser answers here is another example. Lets say I have an app that looks for strings inside double quotes and pulls them out to make tags or something. I want to avoid troll users trying to mess things up or sneak things by me so if they use double double quotes I should just ignore it and not bother making a tag out of it. Example:
In this "sentence" my regex would match "sentence" and use other code I'm not showing you to pull out only the word: sentence.
Now if someone does double double quotes I just ignore it because no match was found. Meaning the inner word should not be found as a match in this instance.
In this ""sentence"" I have two double quotes around the word sentence and it should be completely ignored now. I don't even care about ignoring the outer double quotes and matching on the inner ones. I want no match in this case.
I have a large valid JavaScript file (utf-8), from which I need to extract all text strings automatically.
For simplicity, the file doesn't contain any comment blocks in it, only valid ES6 JavaScript code.
Once I find an occurrence of ' or " or `, I'm supposed to scan for the end of the text block, is where I got stuck, given all the possible variations, like "'", '"', "\'", '\"', '", `\``, etc.
Is there a known and/or reusable algorithm for detecting the end of a valid ES6 JavaScript text block?
UPDATE-1: My JavaScript file isn't just large, I also have to process it as a stream, in chunks, so Regex is absolutely not usable. I didn't want to complicate my question, mentioning joint chunks of code, I will figure that out myself, If I have an algorithm that can work for a single piece of code that's in memory.
UPDATE-2: I got this working initially, thanks to the many advises given here, but then I got stuck again, because of the Regular Expressions.
Examples of Regular Expressions that break any of the text detection techniques suggested so far:
/'/
/"/
/\`/
Having studied the matter closer, by reading this: How does JavaScript detect regular expressions?, I'm afraid that detecting regular expressions in JavaScript is a whole new ball game, worth a separate question, or else it gets too complicated. But I appreciate very much if somebody can point me in the right direction with this issue...
UPDATE-3: After much research I found with regret that I cannot come up with an algorithm that would work in my case, because presence of Regular Expressions makes the task incredibly more complicated than was initially thought. According to the following: When parsing Javascript, what determines the meaning of a slash?, determining the beginning and end of regular expressions in JavaScript is one of the most complex and convoluted tasks. And without it we cannot figure out when symbols ', '"' and ` are opening a text block or whether they are inside a regular expression.
The only way to parse JavaScript is with a JavaScript parser. Even if you were able to use regular expressions, at the end of the day they are not powerful enough to do what you are trying to do here.
You could either use one of several existing parsers, that are very easy to use, or you could write your own, simplified to focus on the string extraction problem. I hardly imagine you want to write your own parser, even a simplified one. You will spend much more time writing it and maintaining it than you might think.
For instance, an existing parser will handle something like the following without breaking a sweat.
`foo${"bar"+`baz`}`
The obvious candidates for parsers to use are esprima and babel.
By the way, what are you planning to do with these strings once you extract them?
If you only need an approximate answer, or if you want to get the string literals exactly as they appear in the source code, then a regular expression can do the job.
Given the string literal "\n", do you expect a single-character string containing a newline or the two characters backslash and n?
In the former case you need to interpret escape sequences exactly like a JavaScript interpreter does. What you need is a lexer for JavaScript, and many people have already programmed this piece of code.
In the latter case the regular expression has to recognize escape sequences like \x40 and \u2026, so even in that case you should copy the code from an existing JavaScript lexer.
See https://github.com/douglascrockford/JSLint/blob/master/jslint.js, function tokenize.
Try code below:
txt = "var z,b \n;z=10;\n b='321`1123`321321';\n c='321`321`312`3123`';"
function fetchStrings(txt, breaker){
var result = [];
for (var i=0; i < txt.length; i++){
// Define possible string starts characters
if ((txt[i] == "'")||(txt[i] == "`")){
// Get our text string;
textString = txt.slice(i+1, i + 1 + txt.slice(i+1).indexOf(txt[i]));
result.push(textString)
// Jump to end of fetched string;
i = i + textString.length + 1;
}
}
return result;
};
console.log(fetchStrings(txt));
ok i do have this following data in my div
<div id="mydiv">
<!--
what is your present
<code>alert("this is my present");</code>
where?
<code>alert("here at my left hand");</code>
oh thank you! i love you!! hehe
<code>alert("welcome my honey ^^");</code>
-->
</div>
well what i need to do there is to get the all the scripts inside the <code> blocks and the html codes text nodes without removing the html comments inside. well its a homework given by my professor and i can't modify that div block..
I need to use regular expressions for this and this is what i did
var block = $.trim($("div#mydiv").html()).replace("<!--","").replace("-->","");
var htmlRegex = new RegExp(""); //I don't know what to do here
var codeRegex = new RegExp("^<code(*n)</code>$","igm");
var code = codeRegex.exec(block);
var html = "";
it really doesn't work... please don't give the exact answer.. please teach me.. thank you
I need to have the following blocks for the variable code
alert("this is my present");
alert("here at my left hand");
alert("welcome my honey ^^");
and this is the blocks i need for variable html
what is your present
where?
oh thank you! i love you!! hehe
my question is what is the regex pattern to get the results above?
Parsing HTML with a regular expression is not something you should do.
I'm sure your professor thinks he/she was really clever and that there's no way to access the DOM API and can wave a banner around and justify some minor corner-case for using regex to parse the DOM and that sometimes it's okay.
Well, no, it isn't. If you have complex code in there, what happens? Your regex breaks, and perhaps becomes a security exploit if this is ever in production.
So, here:
http://jsfiddle.net/zfp6D/
Walk the dom, get the nodeType 8 (comment) text value out of the node.
Invoke the HTML parser (that thing that browsers use to parse HTML, rather than regex, why you wouldn't use the HTML parser to parse HTML is totally beyond me, it's like saying "Yeah, I could nail in this nail with a hammer, but I think I'm going to just stomp on the nail with my foot until it goes in").
Find all the CODE elements in the newly parsed HTML.
Log them to console, or whatever you want to do with them.
First of all, you should be aware that because HTML is not a regular language, you cannot do generic parsing using regular expressions that will work for all valid inputs (generic nesting in particular cannot be expressed with regular expressions). Many parsers do use regular expressions to match individual tokens, but other algorithms need to be built around them
However, for a fixed input such as this, it's just a case of working through the structure you have (though it's still often easier to use different parsing methods than just regular expressions).
First lets get all the code:
var code = '', match = [];
var regex = new RegExp("<code>(.*?)</code>", "g");
while (match = regex.exec(content)) {
code += match[1] + "\n";
}
I assume content contains the content of the div that you've already extracted. Here the "g" flag says this is for "global" matching, so we can reuse the regex to find every match. The brackets indicate a capturing group, . means any character, * means repeated 0 or more times, and ? means "non-greedy" (see what happens without it to see what it does).
Now we can do a similar thing to get all the other bits, but this time the regex is slightly more complicated:
new RegExp("(<!--|</code>)(.*?)(-->|<code>)", "g")
Here | means "or". So this matches all the bits that start with either "start comment" or "end code" and end with "end comment" or "start code". Note also that we now have 3 sets of brackets, so the part we want to extract is match[2] (the second set).
You're doing a lot of unnecessary stuff. .html() gives you the inner contents as a string. You should be able to use regEx to grab exactly what you need from there. Also, try to stick with regEx literals (e.g. /^regexstring$/). You have to escape escape characters using new RegExp which gets really messy. You generally only want to use new RegExp when you need to put a string var into a regEx.
The match function of strings accepts regEx and returns a collection of every match when you add the global flag (e.g. /^regexstring$/g <-- note the 'g'). I would do something like this:
var block = $('#mydiv').html(), //you can set multiple vars in one statement w/commas
matches = block.match(/<code>[^<]*<\/code>/g);
//[^<]* <-- 0 or more characters that aren't '<' - google 'negative character class'
matches.join('_') //lazy way of avoiding a loop - join into a string with a safe character
.replace(/<\/*code>/g,'') //\/* 0 or more forward slashes
.split('_');//return the matches string back to array
//Now do what you want with matches. Eval (ew) or append in a script tag (ew).
//You have no control over the 'ew'. I just prefer data to scripts in strings