Related
I decided to try and make a language tokenizer (don't even know if that's a real word) and made around 4 tokens that successfully tokenized a full program with line breaks and multiple spaces etc, but I just started from scratch and am running into a problem; I have two tokens currently, int and variableSet. The program being read has the content of 1 sv 1 2 as just a test, and the tokenizer returns an array of int, int, int, int with sv having a value of 1.
const code = `1 sv 1 2`
var validTokens = require("./tokens"); // just an object with the structure tokenName: RegExp object
function reverseTokenSearch(regex){
for (const [index, [key, value]] of Object.entries(Object.entries(validTokens))) {
if (value === regex){
return key;
}
}
return false;
}
function throughTokens (code,lastidx=0) {
for (const tokentype in validTokens){ // loop through all of the valid tokens
validTokens[tokentype].lastIndex = lastidx;
const searchresult = validTokens[tokentype]
const tokenresult = searchresult.exec(code.toString());
if (tokenresult) {
return [searchresult, tokenresult[0], tokenresult.index, lastidx+tokenresult[0].length+1, tokenresult.groups]
}
}
}
function resetIndexes (){
for (const tt in validTokens){
validTokens[tt].lastidx = 0;
}
}
resetIndexes();
var lst = 0
var tokens = []
var res = 1;
console.log("\ntokenizer; original input:\n"+code+"\n");
while (lst !== undefined && lst !== null){
if (lst > code.length){
console.error("Fatal error: tokenizer over-reached program length.")
process.exit(1)
}
const res = throughTokens(code,lst);
if(res){
console.log(res,lst)
const current = []
current[0] = reverseTokenSearch(res[0])
current[1] = res[1]
const currentidx = 2
for (const x in res[4]) {
current[currentidx] = x;
}
tokens.push(current)
lst = res[3]
} else {
lst = null
}
}
console.log(tokens)
// What outputs:
/*
tokenizer; original input:
1 sv 1 2
[ /\d+/g { lastidx: 0 }, '1', 0, 2, undefined ] 0
[ /\d+/g { lastidx: 0 }, '1', 5, 4, undefined ] 2
[ /\d+/g { lastidx: 0 }, '1', 5, 6, undefined ] 4
[ /\d+/g { lastidx: 0 }, '2', 7, 8, undefined ] 6
[ [ 'int', '1' ], [ 'int', '1' ], [ 'int', '1' ], [ 'int', '2' ] ]
*/
I think it's because of the order of the array but I have no idea where to start fixing it and would greatly appreciate a push in the right direction.
(edit): I tried removing the "g" flag on the RegExp object and all it did was broke the program into an infinite loop.
The problem is that you are silently assuming that every match found by the regex will start at lastidx which is not always the case. If you log tokenresult and lastidx before returning from throughTokens, you will see:
0
[ '1', index: 0, input: '1 sv 1 2', groups: undefined ]
2
[ '1', index: 5, input: '1 sv 1 2', groups: undefined ]
4
[ '1', index: 5, input: '1 sv 1 2', groups: undefined ]
6
[ '2', index: 7, input: '1 sv 1 2', groups: undefined ]
In the second iteration, the match is at index 5, but you assume it to be at index 2, which it is not (whereby you also incorrectly increment lastidx to 4). You also at the end of throughTokens assume that every match is followed by a space, which is also incorrect for the last token.
Simplest way to fix this code is to replace
//if (tokenresult) { // replace in throughTokens with below
if (tokenresult && tokenresult.index === lastidx) {
to be sure that you're matching at the right place and then in the main loop
//while (lst !== undefined && lst !== null){ // replace with below
while (lst !== undefined && lst !== null && lst < code.length){
to handle the end of the input correctly.
With these changes, the printouts that we added earlier will be
0
[ '1', index: 0, input: '1 sv 1 2', groups: undefined ]
2
[ 'sv', index: 2, input: '1 sv 1 2', groups: undefined ]
5
[ '1', index: 5, input: '1 sv 1 2', groups: undefined ]
7
[ '2', index: 7, input: '1 sv 1 2', groups: undefined ]
which is correct and the output would be
[
[ 'int', '1' ],
[ 'variableSet', 'sv' ],
[ 'int', '1' ],
[ 'int', '2' ]
]
Recommendations
There are a lot of other logical and programmatical problems with this code which I will not go into but my advice is to go through every piece of the code and understand what it does and whether it could be done in a simpler way.
On a general level instead of returning an array with data [d1, d2, d3, ...] return an object with named properties { result: d1, index: d2, ... }. Then it is much easier for someone else to understand your code. Also go through naming of methods.
As far as this approach is concerned, if you know that there will be a space after each token, then extract only the current token and send to throughToken. Then you can make that function both more efficient and robust against errors.
While I was thinking in someway to compare between two arrays, accidentally for the first time something like this happens with me.. something worked with me for the first time without showing me any errors!
This is a very simplified visualization for an application I'm working on currently.
I have two arrays, one called options, the other is called correct.
so the options are options of some question, the chances are the chances of this question.
in this example two of the options are correct answers..
so what I have to do is to compare between this two arrays to check if it returns true (later) or if it's returning false (not included in the code).
could you please explain for me how this actually worked?
const options = ['facebook', 'twitter', 'tango', 'skype'];
const correct = ['twitter', 'skype'];
const trueIndexes = [];
options.forEach((cur, index) => {
correct.forEach((cur1, index1) => {
if (cur === cur1) {
trueIndexes.push(index);
}
});
});
console.log(trueIndexes);
There are nested forEach loops for each array, and during each inner loop there is a conditional test to ascertain if one element exists in the other. If so, the index of where the matching item exists in the other array is pushed to the new trueIndexes array.
Another way to write this is:
const options = ['facebook', 'twitter', 'tango', 'skype']
const correct = ['twitter', 'skype']
const trueIndices = correct.reduce((_trueIndices, correctItem) => {
let correctItemIndex = options.indexOf(correctItem)
if(correctItemIndex !== -1) _trueIndices.push(correctItemIndex)
return _trueIndices
}, [])
console.log(trueIndices)
or
const options = ['facebook', 'twitter', 'tango', 'skype']
const correct = ['twitter', 'skype']
const trueIndices = []
correct.forEach((correctItem) => {
let correctItemIndex = options.indexOf(correctItem)
if(correctItem !== -1) trueIndices.push(correctItem)
})
console.log(trueIndices)
Both of these alternate solutions should be faster.
Your code goes through each option and compares it to both correct values.
If they match the index of the option is added to the true indexes list.
So the comparisons are like follows:
'facebook' = 'twitter'? no
'facebook' = 'skype'? no
'twitter' = 'twitter'? YES -> add index to list
'twitter' = 'skype'? no
'tango' = 'twitter'? no
'tango' = 'skype'? no
'skype' = 'twitter'? no
'skype' = 'skype'? YES -> add index to list
As explained by others, you just iterate over all the combinations of these 2 arrays and collect the indexes where an option is in both arrays(in other words, if an option is true)
I just wanted to add that you should perhaps change your data structure so that options are stored with the questions...and correct answers are indicated by a flag so they don't need to be looked up.
For example:
const questions = [
{
id: 'social_1',
question: 'Which of these are popular social media sites?',
answers: [
{id: 'fb', label: 'Facebook', correct: true},
{id: 'skype', label: 'Skype', correct: false},
{id: 'twitter', label: 'Twitter', correct: true},
{id: 'yt', label: 'YouTube', correct: false},
]
}, {
id: 'cars_1',
question: 'Which of these are car brands?',
answers: [
{id: 'tesla', label: 'Tesla', correct: true},
{id: 'bmw', label: 'BMW', correct: true},
{id: 'twitter', label: 'Twitter', correct: false},
{id: 'yt', label: 'YouTube', correct: false},
]
}
];
https://jsfiddle.net/rainerpl/L438qjms/26/
I was unable to find a solution that is identical with the issue I am trying to solve. If is a duplicate kindly provide a link to a solution.
Using vanilla Javascript, merge array of objects that has a common value. I have a JSON file and it will create the following javascript object (I am unable to change the original structure). Notice that each object will have different nested name and nested value. However the common value is the name[0]
var data = [
{
name: [
'Data 1', // common value
'1 Jan, 2019', // same value therefore not overwrite/merge
'hotfix dec'
],
value: [
'hotfix1.fsfix',
'hotfix1.txt'
]
},
{
name: [
'Data 1', // common value
'1 Jan, 2019' // same value therefore not overwrite/merge
],
value: 'data1.jar'
},
{
name: [
'Data 2',
'1 Feb, 2019'
],
value: 'data2.fsfix'
},
{
name: [
'Data 2',
'1 Feb, 2019'
],
value: 'data2.jar'
},
{
name: [
'Data 3',
'1 Mar, 2018'
],
value: 'data3.fsfix'
}
]
The desire output will be merging the nested object that has the same name[0].
var data = [
{
name: [
'Data 1', // common value
'1 Jan, 2019', // same value therefore not overwrite/merge
'hotfix dec'
],
value: [
'data1.fsfix',
'data1.txt',
'data1.jar' // This was added after the merge
]
},
{
name: [
'Data 2',
'1 Feb, 2019'
],
value: [
'data2.fsfix',
'data2.jar' // This was added after the merge
]
},
{
name: [
'Data 3',
'1 Mar, 2018'
],
value: 'data3.fsfix'
}
]
Using this new merged structure, I would create a function to loop each array set. Thanks in advance
You could key the data by the first name entry using a Map. Then populate the data into the value property within the corresponding Map value, also collect all the extra values in the name arrays (beyond the first two entries), and finally extract the Map values.
This works with linear time complexity:
var data = [{name: ['Data 1', '1 Jan, 2019', 'hotfix dec'],value: ['hotfix1.fsfix','hotfix1.txt']},{name: ['Data 1', '1 Jan, 2019'],value: 'data1.jar'},{name: ['Data 2','1 Feb, 2019'],value: 'data2.fsfix'},{name: ['Data 2','1 Feb, 2019'],value: 'data2.jar'},{name: ['Data 3','1 Mar, 2018'],value: 'data3.fsfix'}];
const map = new Map(data.map(o =>
[o.name[0], { name: o.name.slice(0,2), value: [] }]));
data.forEach(o => map.get(o.name[0]).value.push(...[].concat(o.value)));
data.forEach(o => map.get(o.name[0]).name.push(...o.name.slice(2)));
const result = Array.from(map.values(), o => o.value.length === 1
? { ...o, value: o.value[0] } : o);
console.log(result);
The callback function that is passed to Array.from is a map-function. It is only needed to turn value arrays with only one string into that string only. If this is not required, you can omit that callback and just do Array.from(map.values()).
With some Array methods, I think you can make this work (untested code below, but I think it should do what you want):
// Merge matching entries by pushing values from matching objects to each other
data = data.map(entry => {
let matchingObjects = data.filter(match => {
return match.name[0] === entry.name[0];
});
matchingObjects.forEach(match => {
if (match !== entry) {
var flatValue = [entry.value].flat();
entry.value = flatValue.push.apply(flatValue, [match.value].flat());
}
});
});
// Remove duplicates by filtering out all but the first entry of each name[0]
data = data.filter((entry, index) => {
return index === data.findIndex(match => {
return match.name[0] === entry.name[0];
});
});
You can also use reduce and map to create your desired outcome. I think it is quite powerful combination in general.
const data = [{name: ['Data 1', '1 Jan, 2019', 'hotfix dec'],value: ['hotfix1.fsfix','hotfix1.txt']},{name: ['Data 1', '1 Jan, 2019'],value: 'data1.jar'},{name: ['Data 2','1 Feb, 2019'],value: 'data2.fsfix'},{name: ['Data 2','1 Feb, 2019'],value: 'data2.jar'},{name: ['Data 3','1 Mar, 2018'],value: 'data3.fsfix'}];
const dataMap = data.reduce((acc, curr)=>{
id = curr.name[0];
acc[id] = acc[id] || {
name: [],
value: []
};
const value = Array.isArray(curr.value) ? curr.value : [curr.value]
acc[id].name = [...acc[id].name, ...curr.name.filter((i)=>acc[id].name.indexOf(i)===-1)]
acc[id].value = [...acc[id].value, ...value.filter((i)=>acc[id].value.indexOf(i)===-1)]
return acc
},{})
const result = Object.keys(dataMap).map(key=> {
const d = dataMap[key];
d.value = d.value.length===1 ? d.value[0] : d.value
return d;
})
My if/else statement goes straight to else and I cant seem to figure out why. Here is my code:
var sentiment = require ('sentiment');
var twitterSentiment;
var geoColor;
var results;
var twit = new twitter({
consumer_key: credentials.consumer_key,
consumer_secret: credentials.consumer_secret,
access_token_key: credentials.access_token_key,
access_token_secret: credentials.access_token_secret
});
twit.stream(
'statuses/filter',
{ 'locations': location },
function(stream) {
stream.on('data', function(tweet) {
console.log(tweet.text);
results = sentiment (tweet.text);
twitterSentiment = results;
//Comparison of Sentiment Scores
if (twitterSentiment == 0) {
geoColor = '#B5B5B5';
}
else if (twitterSentiment < 0) {
geoColor = '#FC0828';
}
else {
geoColor = '#00DE1E';
}
console.log (geoColor);
});
});
This is an example output:
omg yes!!
#00DE1E
Do you think will actually understand? I want to ask mine the same question. Just not sure I'm ready to have that conversation.
#00DE1E
A thing of beauty by:
#youtube
#00DE1E
do you still do this??
#00DE1E
As you can see all the tweets are being identified by only one color; almost as if my if/else statement is not implemented correctly?
When I change console.log (geoColor); to console.log (results); This is my output:
omg yes!!
{ score: 1,
comparative: 0.25,
tokens: [ 'omg', 'yes' ],
words: [ 'yes' ],
positive: [ 'yes' ],
negative: [] }
Do you think will actually understand? I want to ask mine the same question. Just not sure I'm ready to have that conversation.
{ score: 1,
comparative: 0.041666666666666664,
tokens:
[
'do',
'you',
'think',
'will',
'actually',
'understand',
'i',
'want',
'to',
'ask',
'mine',
'the',
'same',
'question',
'just',
'not',
'sure',
'i\'m',
'ready',
'to',
'have',
'that',
'conversation' ],
words: [ 'want' ],
positive: [ 'want' ],
negative: [] }
A thing of beauty by:
#youtube
{ score: 3,
comparative: 0.25,
tokens:
[ 'a',
'thing',
'of',
'beauty',
'by',
'youtube', ],
words: [ 'beauty' ],
positive: [ 'beauty' ],
negative: [] }
do you still do this??
{ score: 0,
comparative: 0,
tokens:
[
'do',
'you',
'still',
'do',
'this' ],
words: [],
positive: [],
negative: [] }
As you can see each tweet has their individual sentiment score of respectively 1,1,3,0 So why is my if/else statement disregarding those numbers?
What can I change in my code so that my if/else statement correctly implements and considers the sentiment score of the tweets? My goal is to output the appropriate color for each tweet.
Your var results is an object, that contains many other attributes. In javascript the "if" clause will always return "true" for non-null object instance. When comparing an object to a number - any instance of a non-primitive object will return 1.
As you said, you want to compare the value of your score attribute, so what you need to do is reference your results.score in your if clause.
Changing to
twitterSentiment = results.score;
Should fix your issue.
You are setting twitterSentiment to the result object and comparing the whole object instead of just the score. Change your code to:
if (twitterSentiment.score == 0) {
geoColor = '#B5B5B5';
}
else if (twitterSentiment.score < 0) {
geoColor = '#FC0828';
}
else {
geoColor = '#00DE1E';
}
I'm using lodash mapKeys to take my array of objects and convert it to a mapped object using the id property. That's simple enough, but the problem is that it's sorting the new object by id.
For example if I had three objects in my array:
let myArray = [
{
id: 3,
name: 'Number Three'
},
{
id: 1,
name: 'Number One'
},
{
id: 2,
name: 'Number Two'
}
];
Then I map the keys by id:
_.mapKeys(myArray, 'id')
It returns the following:
{
1: {
id: 1,
name: 'Number One'
},
2: {
id: 2,
name: 'Number Two'
},
3: {
id: 3,
name: 'Number Three'
}
}
My server returns the array in a specific order, so I would like the objects to remain the same, so that when I loop over the object properties, they are in the correct order.
Is that possible with this method? If not, is there a possible alternative to achieve the results?
Use a Map because each item has a custom key (like objects), but the order of insertion will be the order of iteration (like arrays):
const myArray = [
{
id: 3,
name: 'Number Three'
},
{
id: 1,
name: 'Number One'
},
{
id: 2,
name: 'Number Two'
}
];
const map = myArray.reduce((map, item) => map.set(item.id, item), new Map());
map.forEach((item) => console.log(item));
As pointed out in the comments, looping over an object doesn't guarantee order. If you want an ordered list, you need an array.
However, you could apply the iterator pattern. In this pattern, it's up to you to decide what “next” element is. So, you could have a set with the objects (in order to get them in constant time) and an array to store the order. To iterate, you'd use the iterator.
This code could be used as example.
Hope it helps.
let myArray = [{
id: 3,
name: 'Number Three'
}, {
id: 1,
name: 'Number One'
}, {
id: 2,
name: 'Number Two'
}];
let myIterator = ((arr) => {
let mySet = _.mapKeys(arr, 'id'),
index = 0,
myOrder = _.map(arr, _.property('id'));
return {
getObjById: (id) => mySet[id],
next: () => mySet[myOrder[index++]],
hasNext: () => index < myOrder.length
};
})(myArray);
// Access elements by id in constant time.
console.log(myIterator.getObjById(1));
// Preserve the order that you got from your server.
while (myIterator.hasNext()) {
console.log(myIterator.next());
}
<script src="https://cdn.jsdelivr.net/lodash/4.16.6/lodash.min.js"></script>
Like mentioned in the comments, the best would be to keep the object references both in an array to keep the order and in a hash to ease updating.
Backbone's collection (source) works like this. It keeps objects in an array (models), but automatically updates a hash (_byId) when adding and removing models (objects) or when a model's id changes.
Here's a simple implementation of the concept. You could make your own implementation or check for a collection lib.
// a little setup
var array = [];
var hash = {};
var addObject = function addObject(obj) {
hash[obj.id] = obj;
array.push(obj);
}
// Create/insert the objects once
addObject({ id: 3, name: 'Number Three' });
addObject({ id: 1, name: 'Number One' });
addObject({ id: 2, name: 'Number Two' });
// Easy access by id
console.log("by id with hash", hash['1']);
// updating is persistent with the object in the array
hash['1'].name += " test";
// keeps the original ordering
for (var i = 0; i < 3; ++i) {
console.log("iterating", i, array[i]);
}