CSV data with 2 different delimiters for Python 2/Ignition - javascript

I am developing the code in Ignition software using Jython/Python 2 scripts. We need to read data from csv file that has two delimiters "," in header and "\t" in data. The code we use are:
file_path = r'T:\test1.csv'
csvData = csv.reader(open(file_path, 'r'))
header = csvData.next() # Skip the fist row
dataset = system.dataset.toDataSet(header,list(csvData))
calcwindow.rootContainer.getComponent('Power Table').data = dataset
After applying this code we get this:
Power Table
Question are how can we separate the data so that all rows and columns match with csv.reader as ignition do not support panda or re :(
Update the code and now it separate data correctly:
csvData = csv.reader(open(file_path, 'r'),delimiter=',')
header = csvData.next()# Skip the fist row
for line in csvData:
str1 = "".join(line) #removes commas
#print str1
parts = str1.split("\t")
print parts
dataset = system.dataset.toDataSet(header,list(parts))
calcwindow.rootContainer.getComponent('Power Table').data = dataset
, but the error code came up:
Row 0 doesn't have the same number of columns as header list.
Any suggestions??
Thanks
Igor

I figure it out myself.
Here is the code:
file_path = r'T:\test1.csv'
try:
file = open(file_path)
csvData = csv.reader(file,delimiter=',') # open the file with comma delimiter
header = csvData.next()# Skip the fist row
csvData1 = list(csvData) # create list from data
lstLine = csvData1[-1] # selects last line added
str1 = "".join(lstLine) #removes commas and create string
parts = str1.split("\t") #split string back into list
dataset = system.dataset.toDataSet(header,[parts])
calcwindow.rootContainer.getComponent('Power Table').data = dataset
file.close()
except:
print "CSV busy exporting from TIA software"
Hope it will help anyone.

Related

Why is .setValue() in for-loop skipping columns?

I have this web app in Appscript that receives info from a POST request from Siri Shortcuts.
function doPost(e) {
//Get active spreadsheet and input database tab name.
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('Test02');
if(typeof e !== 'undefined')
//Retrieve data received and make a string.
var data = JSON.stringify(e);
//1st print. Print string on new row.
sheet.getRange(sheet.getLastRow()+1,1).setValue(data);
//Eliminate all special characters coming from Siri Shortcuts POST request.
data = data.replace(new RegExp(["\\\\"], 'g'), "");
data = data.replace(new RegExp('"', 'g'), "");
data = data.replace(new RegExp("{", 'g'), "");
data = data.replace(new RegExp("}", 'g'), "");
data = data.replace(new RegExp(":", 'g'), "");
//Split all info on the left side of the first parameter and on the right side of the second.
data = data.split('Log')[1];
data = data.split("name")[0];
//2nd print. Print clean string on a new row.
sheet.getRange(sheet.getLastRow()+1,1).setValue(data);
//3rd print. Print all array element on row. End goal is to have them start from column A and keep adding 1 column every loop.
//Split string into array.
var dataArray = data.split(",");
var lastRow = sheet.getLastRow()+1;
for(var i = 0; i < dataArray.length; i++) {
//3rd print, first part. This tests the real data.
sheet.getRange(lastRow,1+[i]).setValue(dataArray[i]);
//3rd print, second part. This tests the index number to try and find why row are being skipped.
sheet.getRange(lastRow+1,1+[i]).setValue([i]);
}
//Use Mail app to send mail after completion.
MailApp.sendEmail("abc123#gmail.com","InningLogger - Test", data);
return;
}
Attached is the result I am getting in the spreadsheet.
I have two problems with rows 3 and 7:
Why is the data not starting from column A?
Why is row 7 skipping a ton
of columns (I hid them for the screenshot) to print the last 3
values?
Thanks!
As #Pointy mentioned, the problem lied within the 1+[i] in the for loop. "Because i is a number in your code, say 3, 1+[i] will be the string "13", not a number. Changing this to [i+1] solved the problem completely.

Storing multi line html encoded string into sql

I want to store html code into sql database which I've encoded by using encodeURI() but it showing me multiple errors as below
I'm using dataType as <CLOB> also tried using NVARCHAR(MAX) but is showing same error.
Sharing my encoded html code below in string format tobe store into sql.
%3Cp%3Ethis%20kind%20of%20text%20i'm%20storing%20into%20database%3C/p%3E%3Cpre%20class=%22code-pre%22%3Evar uri%20= %22my%20test.asp?name=st%C3%A5le&car=saab%22;%0Avar enc%20=%20encodeURI(uri);%0Avar dec%20=%20decodeURI(enc);%0Avar res%20=%20enc%20+ %22<br>%22 +%20dec;%0A%3C/pre%3E
INSERT INTO "Mytable" VALUES(
8/*ID <INTEGER>*/,
'Return matching objects from array of objects'/*QUESTION <NVARCHAR(200)>*/,
'%3Cp%3Ethis%20kind%20of%20text%20i'm%20storing%20into%20database%3C/p%3E%3C pre%20class=%22code-pre%22%3Evar uri%20= %22my%20test.asp ?
name=st%C3%A5le&car=saab%22;%0Avar enc%20=%20encodeURI(uri);%0Avar
dec%20=%20decodeURI(enc);%0Avar res%20=%20enc%20+ %22<br>%22
+%20dec;%0A%3C/pre%3E'/*QUESTION_DESC <CLOB>*/,
'20170508'/*CREATED <TIMESTAMP>*/,
0/*USERID <INTEGER>*/,
1/*TAGID <INTEGER>*/
);
Above command i'm using for pushing data to db. QUESTION_DESC string i've encoded.original string is
<p>this kind of text i'm storing into database</p><pre class="code-
pre">var uri = "my test.asp?name=ståle&car=saab";
var enc = encodeURI(uri);
var dec = decodeURI(enc);
var res = enc + "<br>" + dec;
</pre>
Help will be appriciated
This was quite simple when i tried posting html code from middle ware.
The problem is when i tried to post an html code to database it was showing error because of some random double quotes. So while sending that html code from middle ware i just replaced the double quotes to ignore double quotes. i did like
my html code to be stored in db
var htmlCodeToBeStored =
"<p>this kind of text i'm storing into database</p><pre class="code-
pre">var uri = "my test.asp?name=ståle&car=saab";
var enc = encodeURI(uri);
var dec = decodeURI(enc);
var res = enc + "<br>" + dec;
</pre>"
I replaced above string with as below
htmlCodeToBeStored = htmlCodeToBeStored.replace(/"/g, "\"")
with that simple change i'm able to store my ans into data base.

how to propely assemble csv file to avoid column shift

in my node app, I'm trying to clean up a csv file.
first, I split it into separate lines
then I replace unwanted characters in the first line (the column headers)
then I re-assemble the file by pushing individual lines into a new array, and writing that array to a new .csv file
For some reason, all my rows ending up being shifted by 1 position with respect to the header row.
I have opened the resulting file in a vu editor, and can see, that all rows somehow acquired a "," character at the besieging
I know I'm doing something incorrectly, but can not see what that is.
Here is my code:
var XLSX = require('xlsx');
var fs = require('fs');
var csv = require("fast-csv");
var workbook = XLSX.readFile('Lineitems.xls');
var worksheet = workbook.Sheets['Sheet1'];
var csv_conversion = XLSX.utils.sheet_to_csv(worksheet);
var csv_lines = csv_conversion.split('\n');
var dirtyHeaderLine = csv_lines[0];
var cleanHeaderLine = dirtyHeaderLine.replace(/\./g,"")
.replace(/"'"/g,"")
.replace(/","/g,"")
.replace(/"\/"/g,"")
.replace(/"#"/g,"");
cleanHeaderLine = cleanHeaderLine.replace(/,+$/, "");
console.log(cleanHeaderLine);
csv_lines[0] = cleanHeaderLine;
var newCsvLines = [];
csv_lines.forEach(function(line){
newCsvLines.push(line + "\n");
});
fs.writeFile('clean_file.csv', newCsvLines, function(err) {
if (err) throw err;
console.log('clean file saved');
});
I don't see any glaring errors here (maybe something with your regex? Not an expert on those) but this will solve your issue.
if (line.charAt(0) == ',') { line = line.substring(1); }
Adjust your variables accordingly. I don't think I have the same case as you.
EDIT: Here's a JSBin of it working.
http://jsbin.com/mirocedagi/1/edit?html,js,output

javascript to parse csv excel file

I am trying to parse a CSV file I made in Excel. I want to use it to update my Google map. This Google map is in a mobile app that I am developing with Eclipse for Android.
Honestly, I am not sure how to write the JavaScript. Any help will be greatly appreciated. I would be happy to credit your work.
I just want some JavaScript to run when the user hits a button that does the following:
Locates users current location (I have already done this part!)
Locate nearby locations as entered in the .CSV excel file by parsing the .CSV
Display a small link inside every locations notification bubble that says "Navigate" that when the user clicks it, opens google maps app and starts navigating the user to that location from the users current location (Geolocation).
This is the ONLY part I need to finish this application. So once again, any help at all will be greatly appreciated. Thanks everyone!
Honestly, I've been round and round with this problem. The CSV format is not made for easy parsing and even with complicated RegEx it is difficult to parse.
Honestly, the best thing to do is import it into an FormSite or PHPMyAdmin, then re-export the document with a custom separator that is easier to parse than ",". I often use "%%" as the field delimiter and everything works like a charm.
Dont know if this will help but see http://www.randomactsofsentience.com/2012/04/csv-handling-in-javascript.html if it helps...
Additional:
On top of the solution linked to above (my preference) I also used a shed load of stacked regular expressions to token a CSV but it's not as easy to modify for custom error states...
Looks heavy but still only takes milliseconds:
function csvSplit(csv){
csv = csv.replace(/\r\n/g,'\n')
var rows = csv.split("\n");
for (var i=0; i<rows.length; i++){
var row = rows[i];
rows[i] = new Array();
row = row.replace(/&/g, "&");
row = row.replace(/\\\\/g, "\");
row = row.replace(/\\"/g, """);
row = row.replace(/\\'/g, "'");
row = row.replace(/\\,/g, ",");
row = row.replace(/#/g, "#");
row = row.replace(/\?/g, "?");
row = row.replace(/"([^"]*)"/g, "#$1\?");
while (row.match(/#([^\?]*),([^\?]*)\?/)){
row = row.replace(/#([^\?]*),([^\?]*)\?/g, "#$1,$2?");
}
row = row.replace(/[\?#]/g, "");
row = row.replace(/\'([^\']*)\'/g, "#$1\?");
while (row.match(/#([^\?]*),([^\?]*)\?/)){
row = row.replace(/#([^\?]*),([^\?]*)\?/g, "#$1,$2?");
}
row = row.replace(/[\?#]/g, "");
row = row.split(",")
for (var j=0; j<row.length; j++){
col = row[j];
col = col.replace(/?/g, "\?");
col = col.replace(/#/g, "#");
col = col.replace(/,/g, ",");
col = col.replace(/'/g, '\'');
col = col.replace(/"/g, '\"');
col = col.replace(/\/g, '\\');
col = col.replace(/&/g, "&");
row[j]=col;
}
rows[i] = row;
}
return rows;
}
I had this problem which is why I had to come up with this answer, I found on npm a something called masala parser which is indeed a parser combinator. However it didn't run on browsers yet, which is why I am using this fork, the code remains unchanged. Please read it's documentation to understand the Parser-side of the code.
import ('https://cdn.statically.io/gh/kreijstal-contributions/masala-parser/Kreijstal-patch-1/src/lib/index.js').then(({
C,
N,
F,
Streams
}) => {
var CSV = (delimeter, eol) => {
//parses anything beween a string converts "" into "
var innerstring = F.try(C.string('""').returns("\"")).or(C.notChar("\"")).rep().map(a => a.value.join(''));
//allow a string or any token except line delimeter or tabulator delimeter
var attempth = F.try(C.char('"').drop().then(innerstring).then(C.char('"').drop())).or(C.charNotIn(eol[0] + delimeter))
//this is merely just a CSV header entry or the last value of a CSV line (newlines not allowed)
var wordh = attempth.optrep().map(a => (a.value.join('')));
//This parses the whole header
var header = wordh.then(C.char(delimeter).drop().then(wordh).optrep()).map(x => {
x.header = x.value;
return x
})
//allow a string or any token except a tabulator delimeter, the reason why we allow newlines is because we already know how many columns there is, so if there is a newline, it is part of the value.
var attempt = F.try(C.char('"').drop().then(innerstring.opt().map(a=>(a.value.__MASALA_EMPTY__?{value:""}:a))).then(C.char('"').drop())).or(C.notChar(delimeter))
//this is merely just a CSV entry
var word = attempt.optrep().map(a => (a.value[0]?.value??a.value[0]));
//This parses a CSV "line" it will skip newlines if they're enclosed with doublequotation marks
var line = i => C.string(eol).drop().then(word.then(C.char(delimeter).drop().then(word).occurrence(i - 1).then(C.char(delimeter).drop().then(wordh)))).map(a => a.value);
return header.flatMap(a => line(a.header.length - 1).rep().map(b => {
b.header = a.header;
return b
}))
};
var m = {
'tab': '\t',
"comma": ",",
"space": " ",
"semicolon": ";"
}
document.getElementById('button').addEventListener('click', function() {
var val = document.getElementById('csv').value;
var parsedCSV = CSV(m[document.getElementById('delimeter').value], '\n').parse(Streams.ofString(val)).value;
console.log(parsedCSV);
})
})
Type some csv<br>
<textarea id="csv"></textarea>
<label for="delimeter">Choose a delimeter:</label>
<select name="delimeter" id="delimeter">
<option value="comma">,</option>
<option value="tab">\t</option>
<option value="space"> </option>
<option value="semicolon">;</option>
</select>
<button id="button">parse</button>
I would suggest stripping the newlines and the end of the file. Because it might get confused.
This appears to work. You may want to translate the Japanese, but it is very straight-forward to use:
http://code.google.com/p/csvdatajs/

Javascript ADO recordset open method not working

I have the following code in an html page in a Javascript tag:
var adOpenDynamic = 2
var adLockOptimistic = 3
var conn_str = 'Provider=Microsoft.Jet.OLEDB.4.0; Data Source=G:/path_to_myDB.mdb'
var conn = new ActiveXObject("ADODB.Connection")
conn.open(conn_str)
and this is a the beginning of a function that is called from the onload event of the html :
var PassNbrAppel = new Array();
var i=1
var rsPass = new ActiveXObject("ADODB.Recordset")
SQLpass = 'SELECT Avis.[Numéro Passerelle], Count(Avis.[Numéro Passerelle]) AS [CompteDeNuméro Passerelle] FROM Avis WHERE (((Avis.[Date Appel])>#10/19/2011# And (Avis.[Date Appel])<#11/07/2011#) AND (Avis.[Numéro Passerelle] IS NOT NULL)) GROUP BY Avis.[Numéro Passerelle] ORDER BY Val(Avis.[Numéro Passerelle]);'
rsPass.open(SQLpass, conn, adOpenDynamic, adLockOptimistic)
rs2arr(rsPass,arrPass)
rs.close()
I get the following error message (translated from french): "no value given for one or more of the required parameters" and the line number is pointing to rsPass.open(SQLpass, conn, adOpenDynamic, adLockOptimistic)
I keep on re-checking to see if there is a mistake in the code but I can't seem to find anything wrong...
I took bits of code from here
The problem was the special characters in my SQL statement. Instead of trying to make it work with the "é" I changed the feild names so they dont have special characters. So much for French pride...

Categories