At the moment I'm implementing a pipeline with streams for CSV files to write its lines to a db with a certain model. I started out with writing everything to stdout. That let to unexpected behavior quickly, when I attached my custom (Transform) product map stream.
At first I started out using only the fs readstream, piped that to the csv transformer (based on the npm package csv-streamify) and wrote to process.stdout. Everything was flowing like normal.
But then I connected my custom transformer and there it starts to act weird. Because as soon as I apply any operation (JSON.parse, typeof chunk) in to the chunk in the transform, the data will not flow directly to stdout but, as it appears, only when the readstream is done.
Does anybody know why this occurs?
My pipeline:
const filePath = path.join(__dirname, 'file1.csv');
//From NPM module: csv-streamify
const csvTransformer = csv({objectMode: false, columns: true});
const mapper = new CsvMapper();
const productMapStream = ProductMapStream.create(mapper);
const writeStream = ProductWriteStream.create(this.connectionPool);
fs.createReadStream(filePath)
.pipe(csvTransformer)
.pipe(csvProductMapStream)
.pipe(process.stdout)
.on('finish', () => resolve('Ok'))
My custom transform stream:
export class ProductMapStream {
static create(mapper: ProductMappable) {
return new Transform({
transform(chunk: any, enc, callback) {
try {
const chunkAsString = chunk.toString('utf8');
// This already prevents continuous flowing
const newString = '' + (typeof chunkAsString);
this.push(newString);
callback(null);
} catch (e) {
callback(e);
}
}
}).once('error', console.log)
}
}
EDIT:
After some experimentation based on the comments of #PatrickDillon I've found out that this problem only occurs when it's run inside a Docker container. I've tried different node versions based on Docker node images. I started out with node:8.12.0-alpine and I've also tried node:10.11.0-jessie but no difference in behavior unfortunately. Does anybody know of special behavior of when using Docker with fs or streams or anything that might seem related?
Related
When using child_process.spawn in Node, it spawns a child process and automatically create stdin, stdout and stderr streams to interact with the child.
const child = require('child_process');
const subProcess = child.spawn("python", ["myPythonScript.py"])
subProcess.stdout.on('data', function(data) {
console.log('stdout: ' + data);
});
I thus imlemented this in my project but the thing is that the subprocess actually write on the output stream only when the buffer reach a certain size. And not when the buffer is set with data (whatever the size of the data).
Indeed, i'd like to receive the subprocess output stream directly when it writes it on the output stream, and not when it has filled the whole buffer. any solution ?
EDIT: As pointed out by t.888, it should actually be working as i expect. And it actually does if I spawn another subprocess. A c++ one this time. But I don't know why it does not work when I spawn my python script. Actually, the python script sends only big chunks of messages via stdout (probably when the buffer is full)
I think that you need readline instead.
const fs = require('fs');
const readline = require('readline');
async function processLineByLine() {
const fileStream = fs.createReadStream('input.txt');
const rl = readline.createInterface({
input: fileStream,
crlfDelay: Infinity
});
// Note: we use the crlfDelay option to recognize all instances of CR LF
// ('\r\n') in input.txt as a single line break.
for await (const line of rl) {
// Each line in input.txt will be successively available here as `line`.
console.log(`Line from file: ${line}`);
}
}
processLineByLine();
From https://nodejs.org/api/readline.html#readline_example_read_file_stream_line_by_line
I solved my problem yesterday. It was actually due to python itself and not child_process function.
I have to do
const subProcess = child.spawn("python", ["-u", "myPythonScript.py"])
instead of
const subProcess = child.spawn("python", ["myPythonScript.py"])
indeed, -u argument tells python to flush data as soon as possible.
I am using the cypress cy.task() method/ function to copy a csv file from one directory to another directory. Below is my cy.task('copycsvFile') and relevant code written in support/index.js file. While running it throws following error;
CypressError: cy.task('copycsvFile') failed with the following error:
The 'task' event has not been registered in the plugins file. You must register it before using cy.task() Any idea why this is not recognised ?
Node version : v10.15.3,
Cypress version : 3.1.5
//sample-spec.js file
cy.task('copycsvFile');
Below is my index.js file
// support/index.js file
const fs = require('fs');
module.exports = (on) => {
on('task', {
copycsvFile: (Obj)=>{
var fs = require('fs');
fs.createReadStream('C:/Users/SomeName/Downloads/Export_Survey_CSV.csv').pipe(fs.createWriteStream('C:/User/Client/Client - Cypress Web UI Tests/cypress/fixtures/Export_Survey_CSV.csv'));
}
});
};
At last, I have figured out the answer.
I have added the below code in the wrong location and that was the reason it fails and returned the above error.
Now I have corrected the location and added under plugins/index.js and works perfectly.
I've also made a small change, ie I added return null since I didn't have anything to return in my case.
// In my spec file;
cy.task('copycsvFile');
// added below code under.. /plugins/index.js
const fs = require('fs');
module.exports = (on) => {
on("task", {
copycsvFile: (Obj) => {
var fs = require("fs");
fs.createReadStream(
"C:/Users/SomeName/Downloads/Export_Survey_CSV.csv"
).pipe(
fs.createWriteStream(
"C:/User/Client/Client - Cypress Web UI Tests/cypress/fixtures/Export_Survey_CSV.csv"
)
);
return null;
},
});
};
Had the same problem, different solution for me though.
Since upgrading to Cypress 10, this approach(support/index.js) is no longer supported, see: https://docs.cypress.io/guides/tooling/plugins-guide#cy-task
I'm quite confused with topic.
I develop a some kind of lazy module assembler using webpack-dev-server. It finally works but sometimes we need more assurance. That's why I need some tests. The task is to make them a kind of autotests.
For now the code of server start looks like this (I omit excessive params).
import webpack from "webpack";
import webpackConfig from "../webpack.config.js";
import webpackCompileConfig from "../webpack-compiler.config.mjs";
import WebpackDevServer from "webpack-dev-server";
webpack(webpackConfig(mode, dirname, masterPath)).run(function(err) {
if (err) throw err;
const compileOpt = {
// pack of compiler options
};
const compiler = webpack(webpackCompileConfig(compileOpt));
const server = new WebpackDevServer(compiler, {
// pack of server options
});
server.listen(port, "0.0.0.0", err => {
if (err) throw err;
console.log(`Starting root server on 0.0.0.0:${port}`);
});
});
It starts and works properly: gets some file requests, bundles necessary modules with webpack and sends them to requester.Simple test I want to start with are to check are there files after assembling or not.
Supposed logic is:
Run some command inside this project, e.g. npm run test
It starts server and sends a pack of requests with different logic I want to test (parallel, simultaneous requests etc.)
It tests file existence and send me results in console or smth. of that sort
The problem is my very little expirience in any kind of testing so I'll appreciate your help.
===The way I use it now (spoiler: manually)
The only thing the Internet helps me about.
Server starts as usual
There is another fully off-site test module (code below)
Run mocha
See listed test results
test-server.js
var chai = require("chai");
var chaiHttp = require("chai-http");
const should = chai.should();
chai.use(chaiHttp);
describe("Test file existence", function() {
it("units", done => {
chai
.request("http://localhost:9000")
.get("/units/awesome-project/index.js")
.end((err, res) => {
res.should.have.status(200);
done();
});
});
// another 'it()'s to test other files
});
Yes, it works. But I want more automatisation. Just
Run server
Send requests
Get test results
I'm ready for dialog.
Well.. just sad that nobody ask.
Anyway I've found the answer by myself. And I even wonder how it was freaking easy. It seems that all I need is written here: https://github.com/mochajs/mocha/wiki/Using-mocha-programmatically
So the final result is similar to this:
import fs from "fs";
import path from "path";
import Mocha from "mocha";
const mocha = new Mocha();
const testDir = `${config.dirname}/test/tests`;
fs.readdirSync(testDir)
.filter(file => file.match(/\.js$/))
.forEach(file => {
mocha.addFile(path.join(testDir, file));
});
// somewhere before the devserver start
const runner = mocha.timeout(30000).run();
runner.on("end", () => {
process.exit();
});
I found it at the day I posted this question but there was a hope for someone else to answer (to compare solutions).
Best regards,
Nick Rimer
I want to read a symlink, and get the details of the link itself, not the contents of the linked file. How do I do that in Node, in a cross-platform way?
I can detect symlinks easily using lstat, no problem. Once I know the path of the file, and that it is a symlink though, how can I read it? fs.readFile always reads the target file, or throws an error for reading a directory for links to directories.
There is a fs.constants.O_SYMLINK constant, which in theory solves this on OSX, but it seems to be undefined on both Ubuntu & Windows 10.
If you have determined that the file is a symlink try this:
fs.readlink("./mysimlink", function (err, linkString) {
// .. do some error handling here ..
console.log(linkString)
});
Confirmed as working on Linux.
You could then use fs.realpath() to turn it into a full path. Be aware though that linkString can be just a filename or relative path as well as a fully qualified path so you may have to get fs.realpath() for the symlink, determine its directory part and prefix it to linkString before using fs.realpath() on it.
I've just faced the same issue: sometimes fs.readlink returns a relative path, sometimes it returns an absolute path.
(proper error handling not implemented to keep things simple)
const fs = require('fs');
const pathPckg = require('path');
async function getTarLinkOfSymLink(path){
return new Promise((resolve, reject)=>{
fs.readlink(path, (err, tarPath)=>{
if(err){
console.log(err.message);
return resolve('');
}
const baseSrcPath = pathPckg.dirname(path);
return resolve( pathPckg.resolve(baseSrcPath, tarPath) );
});
});
}
// usage:
const path = '/example/symbolic/link/path';
const tarPath = await getTarLinkOfSymLink(path);
The code works if the symbolic link is either a file or a directory/folder - tested on Linux
Here is my problem, I want to create a CLI that automatically runs a test. Without the CLI, I'm able to run everything perfectly with the node command:
node test.js
Basically, I want to do the exact same thing as the command before, so I googled for a technique that does this. I found this:
#!/usr/bin/env node
'use strict';
const options = process.argv;
const { execFile } = require('child_process');
const child = execFile('node', ['../dist/test.js'], (error, stdout, stderr) => {
if (error) {
throw error;
}
console.log(stdout);
});
This method doesn't work for me because, in the test.js file, I'm using the ora package. And because this package is making real-time animations, it doesn't come in stdout.
Is there any way of executing in real time (without subprocess) my test.js using Node? I'm open to other methods, but I want to publish the CLI on NPM, so keep in mind that it has to be in JavaScript 😊.
You can find every file that I've talked here on GitHub. Normally, you wouldn't need this link, but I'm giving it to you if you need to have a closer look.
You should simply call your test() function from your CLI code, after requiring the module that defines it. Have a look at mocha and jasmine: you will see that while both tools provide a CLI, they also provide instructions for invoking the test frameworks from arbitrary JS code.
I can't think of a way without a sub-process. but this may help.
The child process exec will not work with the continuous output commands as it buffers the output the process will halt when that buffer is full.
The suitable solution is spwan :
var spwan = require('child_process').spwan
var child = spwan('node', ['../dist/test.js'])
child.stdout.on('data', function(data) {
console.log(data)
})
child.stderr.on('data', function(data) {
console.log(data)
})
Here is my solution, you can use the fs library to get the code of the file, and then, you simply use eval to execute in the same process.
const fs = require("fs");
function run(file) {
fs.readFile(file, (err, data) => {
eval(data.toString('utf8'))
})
}