As the title says, I currently have a CSV file created from SharePoint list data and in order to display this information as a spreadsheet, I want to convert it to an Excel XLSX file. I prefer to do this without relying on a third-party library. At first, I started to use ActiveX objects to try to recreate and/or save the CSV as XLSX, but there's a limitation with that since I can't really use it in other browsers besides IE. I was thinking using Blob to somehow convert it? That's where I'm stuck.
function createCsv(data) {
var result = "";
if (data == null || data.length == 0) {
return;
}
var columnDelimiter = ',';
var lineDelimiter = '\n';
var keys = Object.keys(data[0]);
// spreadsheet header
result += keys.join(columnDelimiter);
result += lineDelimiter;
// spreadsheet data
data.forEach(function (obj) {
var count = 0;
keys.forEach(function (key) {
if (count > 0) {
result += columnDelimiter;
}
result += obj[key];
count++;
});
result += lineDelimiter;
});
return result;
}
function downloadCsv(csv) {
if (csv == null) {
return;
}
var filename = "test.csv";
csv = "data:text/csv;charset=utf-8," + csv;
var data = encodeURI(csv);
console.log(data);
var link = document.getElementById('csv');
link.setAttribute('href', data);
link.setAttribute('download', filename);
console.log(link);
//displayCsv(csv);
}
function displayCsv() {
// using test csv here
var message = "data:text/csv;charset=utf-8, yo, hey, lol";
//var fileType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
var fileType = "application/msexcel";
var csvFile = new Blob([message], {type: fileType});
var csvUrl = URL.createObjectURL(csvFile);
console.log(csvFile);
console.log(csvUrl);
}
CSV works fine with using the spreadsheet (by downloading and opening it in Excel), but I really need a way to display it as a spreadsheet on a webpage and not as text, so that's why I'm looking to convert it over. Since I'm using this within SharePoint then I can use a Excel web part to display the XLSX - it won't open CSV files like this though. Thanks in advance.
It would be quite the undertaking to try to manually try to do this without libraries. While OpenXML files are XML based at their core, they are also bundled/zipped.
I would recommend take a look at SheetJS. https://sheetjs.com/
You can take CSV as input, and write it back out immediately as XSLX.
I'm not sure that this will solve your issues but if a xls file will suffice you can create a xls file simply by adding a separator tag to the first line of the csv and rename it to xls.
Quotes around the values has also been important.
Eg:
"sep=,"
"Service","Reported","Total","%"
"a service","23","70","32.86%"
"yet_a_service","27","70","38.57%"
"more_services","20","70","28.57%"
If you are fine with using a third-party library (which I strongly recommend considering the complexity involved in conversion ), this solution will suit your needs if it needs to be done in nodejs.
If you want to use it in the browser, convertCsvToExcel function needs to be modified to transform the buffer to a blob object, then converting that blob to an XLS file.
// Convert a CSV string to XLSX buffer
// change from xlsx/xls and other formats by going through sheetsjs documentation.
import * as XLSX from 'xlsx';
export const convertCsvToExcelBuffer = (csvString: string) => {
const arrayOfArrayCsv = csvString.split("\n").map((row: string) => {
return row.split(",")
});
const wb = XLSX.utils.book_new();
const newWs = XLSX.utils.aoa_to_sheet(arrayOfArrayCsv);
XLSX.utils.book_append_sheet(wb, newWs);
const rawExcel = XLSX.write(wb, { type: 'base64' })
return rawExcel
}
// Express request handler for sending the excel buffer to response.
export const convertCsvToExcel = async (req: express.Request, res: express.Response) => {
const csvFileTxt = fileBuffer.toString()
const excelBuffer = convertCsvToExcelBuffer(csvFileTxt)
res.setHeader('Content-Type', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
res.status(200).send(Buffer.from(excelBuffer, 'base64'))
}
Related
I am getting a JSON back from an API and want to add this data as a new column to an Excel file that already has some data. I wanted to ask that is this possible using just frontend Javascript (without involving Node.js)? If yes, then how?
Yes, you can do it using the library exceljs
Github: https://github.com/exceljs/exceljs
NPM: https://www.npmjs.com/package/exceljs
CDN: https://cdn.jsdelivr.net/npm/exceljs#1.13.0/dist/exceljs.min.js
<input type="file" onchange="parseChoosenExcelFile(this)">
function parseChoosenExcelFile(inputElement) {
var files = inputElement.files || [];
if (!files.length) return;
var file = files[0];
console.time();
var reader = new FileReader();
reader.onloadend = function(event) {
var arrayBuffer = reader.result;
// var buffer = Buffer.from(arrayBuffer)
// debugger
var workbook = new ExcelJS.Workbook();
// workbook.xlsx.read(buffer)
workbook.xlsx.load(arrayBuffer).then(function(workbook) {
console.timeEnd();
var result = ''
workbook.worksheets.forEach(function (sheet) {
sheet.eachRow(function (row, rowNumber) {
result += row.values + ' | \n'
})
})
// Output somewhere your result file
// result2.innerHTML = result
});
};
reader.readAsArrayBuffer(file);
}
I am looking for a way to read the feather files via GoLang or Javascript, or some other languages that does not require users to do some other extra installation.
My goal is to provide a User-interface to read a feather csv file and convert it back to a human-readable csv. However I can't find much resources on how to work it out.
Currently I have a test feather file generated by below.
import pandas as pd
import datetime
import numpy as np
import pyarrow.feather as feather
# Create a dummy dataframe
todays_date = datetime.datetime.now().date()
index = pd.date_range(todays_date-datetime.timedelta(10), periods=10, freq='D')
columns = ['A','B', 'C']
df = pd.DataFrame(index=index, columns=columns)
df = df.fillna(0) # with 0s rather than NaNs
feather.write_feather(df, 'test_feather.csv')
Thanks in advance.
The Javascript package apache-arrow comes with a script that does exactly this. You can find the source for the script here: https://github.com/apache/arrow/blob/master/js/bin/arrow2csv.js
If it is not doing exactly what you want the script should serve as an example of how to use the API to read in a feather file.
Thanks for the hints from #Pace. Turns out I found that I can simply use the arrow.Table.from([arrow]) function to convert .feather file to csv.
For those people encountered same issue, you may find the code below for reference.
const apArrow = require('apache-arrow');
const fs = require('fs');
const outputDir = 'output/feather';
const writeIntoFile = (data) => {
fs.appendFileSync(`${outputDir}/test_feather.csv`, data, function (err) {
if (err) return console.log(err);
});
};
const readDataFromRow = (fields, row) => {
return fields
.map((f) => row.get(f))
.join(',');
};
const arrowReader = (filePath) => {
console.log('filePath', filePath);
const arrow = fs.readFileSync(filePath);
const table = apArrow.Table.from([arrow]);
const columns = table.schema.fields.map((f) => f.name);
let buf = columns.join(',') + '\n';
for (let i = 0; i < table.count(); i++) {
const rowData = readDataFromRow(columns, table.get(i));
buf += `${rowData}\n`;
// export to csv every 10000 rows
if (i % 10000 === 0) {
writeIntoFile(buf);
buf = '';
if (i > 0) {
break;
}
}
}
writeIntoFile(buf);
};
I have an angular app where user can export a table to CSV. Although it works fine in desktop, when trying to open it in mobile I receive the following error:
"Unable to open the document spreadsheet appears to be corrupted".
I used this site
https://csvlint.io
to check the CSV, and received this warning:
"Context problem: Incorrect Encoding
Your CSV appears to be encoded in ASCII-8BIT. We recommend you use UTF-8."
After searching a little for solution, I found many issues about it, but my code already appear to have the solution (The BOM), so I don't know what else is missing.
This is my code ( it is a angular app)
protected exportCSV(items: any, fileName: string): void {
const isItemsExists = Array.isArray(items) && items.length > 0;
if (!isItemsExists) {
this.alertService.showTranslatedMessage('dataExporterService.errors.noItemToExport', '', MessageSeverity.error);
return;
}
const replacer = (key, value) => value === null ? '' : value; // specify how you want to handle null values here
const header = Object.keys(items[0]);
const csv = items.map(row => header.map(fieldName => JSON.stringify(row[fieldName], replacer)).join(','));
csv.unshift(header.join(','));
const csvArray = csv.join('\r\n');
const BOM = '\uFEFF'; // Use this and utf8 to support hebrew
const blob = new Blob([BOM + csvArray], {
type: 'data:text/csv;charset=utf-8'
});
this.fileSaver.save(blob, fileName);
}
I have two CSV files, one with routing steps and one with a list of ids. I need to add each id to the start of each routing step. I'm using Node.js.
var routeNumFile = '/routing_numbers.csv';
var routeStepFile = '/routing_steps.csv';
const csvToJson = require('csvtojson');
const jsonToCsv = require('json2csv').parse;
const fs = require('fs');
var routeNumArray;
var routeStepArray;
try {
routeNumArray = await csvToJson().fromFile(routeNumFile);
} catch (err) {
console.log("error in reading csv file")
}
try {
routeStepArray = await csvToJson().fromFile(routeStepFile);
} catch (err) {
console.log("error in reading csv file")
}
var outputArray = new Array;
var outputPath = '/gitlab/BSI_Create_Csv_Import/finalOutput.csv';
if (routeNumArray != null && routeStepArray != null) {
Object.keys(routeNumArray).forEach(function (key1) {
Object.keys(routeStepArray).forEach(function (key2) {
var comboObj = Object.assign(routeNumArray[key1], routeStepArray[key2]);
console.log(comboObj);
outputArray.push(comboObj);
});
});
}
console.log(outputArray);
var csv = jsonToCsv(outputArray);
fs.writeFileSync(outputPath, csv);
The output from console.log(comboObj) is what I want. However, when I put that into an array, I just get the very last entry in the routing steps CSV over and over. If I write it using a stream directly where the comboObj is created, it mostly works. If I do that I convert the object to CSV first and the output leaves me with the headings duplicated at the end of every line. It writes much cleaner from an array of JSON objects to a CSV.
So, I'm trying to get all of the combined objects together in an array so I can convert it all to CSV and write it to a file. Can anyone tell me what's going wrong with my method?
You should create a new object and assign properties to it, like this:
var comboObj = Object.assign({}, routeNumArray[key1], routeStepArray[key2]);
When a video on my local storage—let's say it's currently located at file:///home/user/video.m4v—is opened by dragging it into a new tab in Chrome, how can I calculate the SHA-1 checksum for the file using JavaScript?
Purpose:
I am planning to write a Chrome extension which will store the calculated checksum of videos (files with extensions matching a pattern) as localStorage objects in order to save the playback position of video upon tab close and then restore it when the file is loaded again, even if the location or filename of the video is changed.
You need a crypto library for this. A well known one is Google CryptoJS.
I found this as an specific example for your task: https://gist.github.com/npcode/11282867
After including the crypto-js source:
function sha1sum() {
var oFile = document.getElementById('uploadFile').files[0];
var sha1 = CryptoJS.algo.SHA1.create();
var read = 0;
var unit = 1024 * 1024;
var blob;
var reader = new FileReader();
reader.readAsArrayBuffer(oFile.slice(read, read + unit));
reader.onload = function(e) {
var bytes = CryptoJS.lib.WordArray.create(e.target.result);
sha1.update(bytes);
read += unit;
if (read < oFile.size) {
blob = oFile.slice(read, read + unit);
reader.readAsArrayBuffer(blob);
} else {
var hash = sha1.finalize();
console.log(hash.toString(CryptoJS.enc.Hex)); // print the result
}
}
}
I wouldn't recommend to calculate a hash over the whole video file as it can be pretty resource consuming depending on the file size. Maybe you can use just the meta information or reconsider about the filename and filepath again?
Web APIs have progressed considerably since I asked this question. Calculating a hex digest is now possible using the built-in SubtleCrypto.digest().
TS Playground link
function u8ToHex (u8: number): string {
return u8.toString(16).padStart(2, '0');
}
/** Ref: https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#supported_algorithms */
const supportedAlgorithms = [
'SHA-1',
'SHA-256',
'SHA-384',
'SHA-512',
] as const;
type SupportedAlgorithm = typeof supportedAlgorithms[number];
type Message = string | Blob | BufferSource;
async function hexDigest (
algorithm: SupportedAlgorithm,
message: Message,
): Promise<string> {
let buf: BufferSource;
if (typeof message === 'string') buf = new TextEncoder().encode(message);
else if (message instanceof Blob) buf = await message.arrayBuffer();
else buf = message;
const hash = await crypto.subtle.digest(algorithm, buf);
return [...new Uint8Array(hash)].map(u8ToHex).join('');
}