I am trying to plug individual id's into a function to pull log outputs for each id. These ids are inside objects inside a larger object. I cannot figure out how to call these id's individually in order to print out the log report for each object.
objects
(2) [{…}, {…}]
0: {id: "60844b1f5ce50a40c1adfc32", rdbmsJobId: null, sqoopJobId: "30471e65-c9b9-40e0-b27f-472769480f0c", status: "FINISHED", progress: 100, …}
1: {id: "60844b1f5ce50a40c1adfc33", rdbmsJobId: null, sqoopJobId: "1e74fb62-4a61-420b-9c78-e005007eb17e", status: "FINISHED", progress: 100, …}
length: 2
__proto__: Array(0)
Function
getLogDataById(id: string){
this.logService.getLogById(id).subscribe(logRes => {
this.logReport = logRes
console.log("Log report is: " + this.logReport)
})
}
Expected Output
{
"id": "60844b1f5ce50a40c1adfc34",
"jobId": "60844b1f5ce50a40c1adfc32",
"log": "2021-04-24 16:45:20,180 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7\n2021-04-24 16:45:21,084 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.\nWarning: /opt/sqoop/../hbase does not exist! HBase imports will fail.\nPlease set $HBASE_HOME to the root of your HBase installation.\nWarning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.\nPlease set $HCAT_HOME to the root of your HCatalog installation.\nWarning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.\nPlease set $ACCUMULO_HOME to the root of your Accumulo installation.\nWarning: /opt/sqoop/../zookeeper does not exist! Accumulo imports will fail.\nPlease set $ZOOKEEPER_HOME to the root of your Zookeeper installation.\nJob Created and Saved to Sqoop.\nSqoop Job Submitted to Yarn\n2021-04-24 16:46:27,894 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n2021-04-24 16:46:28,051 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032\nContainer: container_1618958577187_0219_01_000001 on ip-172-31-50-48.us-west-2.compute.internal_43829\nLogAggregationType: AGGREGATED\n=====================================================================================================\nLogType:stderr\nLogLastModifiedTime:Sat Apr 24 16:46:26 +0000 2021\nLogLength:1722\nLogContents:\nApr 24, 2021 4:46:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register\nINFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class\nApr 24, 2021 4:46:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register\nINFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class\nApr 24, 2021 4:46:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register\nINFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class\nApr 24, 2021 4:46:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate\nINFO: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'\nApr 24, 2021 4:46:03 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider\nINFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope \"Singleton\"\nApr 24, 2021 4:46:04 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider\nINFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope \"Singleton\"\nApr 24, 2021 4:46:06 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider\nINFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope \"PerRequest\"\nlog4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).\nlog4j:WARN Please initialize the log4j system properly.\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\n\nEnd of LogType:stderr\n***********************************************************************\n\n\nEnd of LogType:stdout\n***********************************************************************\n\nContainer: container_1618958577187_0219_01_000003 on ip-172-31-50-48.us-west-2.compute.internal_43829\nLogAggregationType: AGGREGATED\n=====================================================================================================\nLogType:stderr\nLogLastModifiedTime:Sat Apr 24 16:46:26 +0000 2021\nLogLength:227\nLogContents:\nLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.\n\nEnd of LogType:stderr\n***********************************************************************\n\n\nEnd of LogType:stdout\n***********************************************************************\n\nContainer: container_1618958577187_0219_01_000002 on ip-172-31-50-48.us-west-2.compute.internal_43829\nLogAggregationType: AGGREGATED\n=====================================================================================================\nLogType:stderr\nLogLastModifiedTime:Sat Apr 24 16:46:26 +0000 2021\nLogLength:227\nLogContents:\nLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.\n\nEnd of LogType:stderr\n***********************************************************************\n\n\nEnd of LogType:stdout\n***********************************************************************\n\nIngestion Finished"
}
Related
I have excluded some parts of my original code for readability, sorry if it causes any confusion!
I have a state variable in App.js defined as such
const [tasks, setTasks] = useState([])
From App.js, I pass into Tasks.js the state variable as a
prop.
Tasks.js receives the prop and sorts it as below (when a certain button is clicked)
const Tasks = function ({tasks, setTasks}) {
setTasks(tasks.sort((a, b) => {
if ((moment(a.isoDay).unix()) < (moment(b.isoDay).unix())) return -1
else {
return 0
}
}))
}
This causes the state variable in App.js to be updated, and this state variable is
passed into Calendar.js (from App.js) as a prop as well, as seen below
const Calendar = ({tasks}) => {
function TaskToCalendar(tasks) {
console.log(tasks)
console.log(tasks[0])
}
}
However, I am not accurately capturing the value of tasks.
For example, console.log(tasks) yields this before it is
sorted in Tasks.js
(2) [{…}, {…}]
0: {text: "Test0", day: "Tue Jun 01 2021"}
1: {text: "Test1", day: "Wed Jun 02 2021"}
and yields this after it is sorted in Tasks.js
(2) [{…}, {…}]
0: {text: "Test1", day: "Wed Jun 02 2021}
1: {text: "Test0", day: "Tue Jun 01 2021}
But, console.log(tasks[0]) yields this before and
after it is sorted
{text: "Test0", day: "Tue Jun 01 2021"}
I have to click on another button somewhere in my UI (that runs an unrelated function), for console.log(tasks[0]) to yield {text: "Test1", day: "Wed Jun 02 2021} after sorting.
How would I go about correctly passing the value of tasks, such that accessing the array indices gives me the correct values?
The SetTask function is working asynchronously. hence, this behavior.
For logging the value you can use the useeffect Hook
I have create a mock data and performed a batch update with firestore.batch(). The problem is that location is saved as a number and time as a string and not as Geopoint and Timestamp instances respectively, causing my React app to crash.
I could see that only firestore is accessible through window.firestore, but not Firebase. Since Firebase isn't exported as window.firebase, I cannot create either Geopoint or Timestamp instance.
So, how to create a Timestamp and Geopoint instance in Firebase Emulator from browser console?
Here's a type of doc I'm adding to firestore
const doc = {
"company": "company-1",
"location": [
-72.3623, // number
79.5748 // but, want to convert to Geopoint instance
],
"time": "Fri Sep 10 1976 07:42:23 GMT+0530 (India Standard Time)", // string
"createdAt": "Mon Apr 28 2014 13:30:16 GMT+0530 (India Standard Time)", // want to convert to Timestamp
}
A more dependable accessor path than .Gf:
new firestore.app.firebase_.firestore.GeoPoint(lat, lng)
There's a firebase accessible through window.firestore.
So I figured a way to create Geopoint and Timestamp instance through Firestore.
You can access it through window.firestore.Gf.firebase_, through which you can create those both instance.
const raw = // pasting from clipboard
const batch = firestore.batch()
const firebase = firestore.Gf.firebase_
const Timestamp = firebase.firestore.FieldValue().Timestamp
const GeoPoint = firebase.firestore.FieldValue().GeoPoint
raw.forEach(doc => {
const docRef = firestore
.collection('user')
.doc('user-1')
.collection('interviews')
.doc()
doc = {
...doc,
time: Timestamp.fromDate(new Date(doc.time)),
createdAt: Timestamp.fromDate(new Date(doc.createdAt)),
location: new GeoPoint(doc.location[0], doc.location[1])
}
batch.set(docRef, doc)
})
batch.commit()
I'm trying to test out an idea I had but using this XML to JSON script:
https://github.com/sergeyt/jQuery-xml2json
But then take the JSON data and then output that into HTML with jQuery.
So far, I've been able to load up the data correctly and I can console.log out the JSON data which turns out to be something like this:
releases:
$: {}
matching_count: "698"
returned_count: "50"
latestModified: "1537876805"
release: Array(50)
0:
$: {}
id: "713"
headline: "Eiger BioPharmaceuticals to Participate in Investor Conferences"
released: "1537876800"
releaseDate: "Tue, 25 Sep 2018 08:00:00 -0400"
modified: "1537876805"
modifiedDate: "Tue, 25 Sep 2018 08:00:05 -0400"
1:
$: {}
id: "712"
headline: "Communications industry innovator to speak at AMEC Global Summit"
released: "1491400800"
releaseDate: "Wed, 05 Apr 2017 10:00:00 -0400"
modified: "1491400806"
modifiedDate: "Wed, 05 Apr 2017 10:00:06 -0400"
...
My JS code that I have so far is this:
var ul = $("<ul>").appendTo("body");
$.ajax({
url: "https://s3-us-west-2.amazonaws.com/s.cdpn.io/8689/list.xml",
dataType: "xml",
success: function(response) {
json = $.xml2json(response);
console.log(json);
$(json).each(function(index, headline) {
ul.append($(document.createElement("li")).text(headline));
});
}
});
The issue is that when it renders out, all that comes back is [object Object] in a single li.
So, I know that I'm doing something incorrectly, but I'm not sure where and at what point I need to change over to get the data so I can render out the headline, releaseDate, and so on.
I've created a Codepen here:
https://codepen.io/ultraloveninja/pen/YzPRjRw
I've read that I might need to use jQuery.parseJSON but after testing that a bit, I get some errors since it's still getting in that object.Object again.
I feel that I need to drill down a bit more to get within the releases.release somehow since that's where all the data is at to be rendered out.
The response you try to loop is an object representing the xml document.
To access the releases array you have to use json["#document"].releases.release
var ul = $("<ul>").appendTo("body");
$.ajax({
url: "https://s3-us-west-2.amazonaws.com/s.cdpn.io/8689/list.xml",
dataType: "xml",
success: function(response) {
json = $.xml2json(response);
let release = json["#document"].releases.release
$(release).each(function(index, headline) {
ul.append($(document.createElement("li")).text(headline.headline));
});
}
});
Example
I'm using a snapshot test in my project and came across a weird problem when running this specific test on a CI server: it displays the timezone name instead of the GMT code, causing the test failure.
I have tried using "moment-timezone" and Date.UTC() to normalize the dates, the result shown was the correct date with the same issue as above.
I've also tried to stub the global.Date object, but the components complained about prop incompatibility.
it('should render with props', () => {
const order = {
merchant: { logo: 'abc', name: 'Pizza Hut' },
bag: {
items: [{ name: 'Corn & Bacon' }],
total: {
valueWithDiscount: 99.99,
},
},
delivery: {
deliversAt: new Date('2019-05-21 13:00'),
},
payment: {
mode: 'online',
},
lastStatus: API_STATUSES.cancelled,
createdAt: new Date('2019-05-21 12:00'),
details: {},
};
const wrapper = shallowMount(Order, {
...commons,
propsData: { order },
});
expect(wrapper).toMatchSnapshot();
});
See that the expected date is the same as the received one, but syntactic differences:
<div class="order__details">
- <orderdetails-stub paymentmode="online" deliverytime="Fri Jun 21 2019 10:00:00 GMT-0300 (GMT-03:00)" value="99.99" laststatus="cancelled"></orderdetails-stub>
+ <orderdetails-stub paymentmode="online" deliverytime="Fri Jun 21 2019 10:00:00 GMT-0300 (Brasilia Standard Time)" value="99.99" laststatus="cancelled"></orderdetails-stub>
Using Date strings as props like this is hazardous and likely to lead to the sort of problem you're encountering.
Best practice for tests in my experience is to use Date.getTime() so values are numbers of milliseconds without any locale information.
Alternatively, you can use moment-timezone as described in this article:
import moment from 'moment-timezone';
it('renders without crashing', () => {
moment.tz.setDefault('EST');
let props = {
currentDay: moment("2017-09-15 09:30:00").format("MMM Do YYYY h:mm:ss a")
};
const tree = renderer.create(<App {...props} />).toJSON();
expect(tree).toMatchSnapshot();
});
I am trying to run group command via mongo shell
db.contract.group({
key:{id: 1},
initial: {v: []},
reduce: function(obj, prev){
prev.v.push(obj.name)
}
});
and see the following error on client
Thu Nov 17 12:12:49 uncaught exception: group command failed: {
"errmsg" : "exception: JS_NewObject failed: toJSObject2",
"code" : 13072,
"ok" : 0
}
and mongod logs says the following
Thu Nov 17 12:12:17 [initandlisten] connection accepted from 127.0.0.1:58509 #1
Thu Nov 17 12:12:49 [conn1] JS Error: out of memory
Thu Nov 17 12:12:49 [conn1] Assertion: 13072:JS_NewObject failed: toJSObject2
0x10008de9b 0x1001565bd 0x100156c9e 0x10037011e 0x10037204c 0x10034c4d6 0x10034d877 0x100180cc4 0x100184649 0x1002b9e89 0x1002c3f18 0x100433888 0x100446d74 0x7fff86e00fd6 0x7fff86e00e89
0 mongod 0x000000010008de9b _ZN5mongo11msgassertedEiPKc + 315
1 mongod 0x00000001001565bd _ZN5mongo9Convertor10toJSObjectEPKNS_7BSONObjEb + 1229
2 mongod 0x0000000100156c9e _ZN5mongo7SMScope9setObjectEPKcRKNS_7BSONObjEb + 78
3 mongod 0x000000010037011e _ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE + 2110
4 mongod 0x000000010037204c _ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 3676
5 mongod 0x000000010034c4d6 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 1350
6 mongod 0x000000010034d877 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 2151
7 mongod 0x0000000100180cc4 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 52
8 mongod 0x0000000100184649 _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 10585
9 mongod 0x00000001002b9e89 _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 569
10 mongod 0x00000001002c3f18 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 1528
11 mongod 0x0000000100433888 _ZN5mongo10connThreadEPNS_13MessagingPortE + 616
12 mongod 0x0000000100446d74 thread_proxy + 132
13 libSystem.B.dylib 0x00007fff86e00fd6 _pthread_start + 331
14 libSystem.B.dylib 0x00007fff86e00e89 thread_start + 13
Thu Nov 17 12:12:49 [conn1] query staging.$cmd ntoreturn:1 command: { group: { key: { asset_id: 1.0 }, initial: { v: {} }, ns: "contract", $reduce: function (obj, prev) {
prev.v.push(obj.name);
} } } reslen:119 21013ms
I tried checking if the virtual memory is unlimited and it is
bash-3.2$ ulimit -a | egrep virtual\|open
open files (-n) 256
virtual memory (kbytes, -v) unlimited
so I am not sure how to fix this problem
Thank you
Mongo is grouping in memory, so if you have a large database with many different values for id things might get too big.
Also you are piling up all names for a given id in one array. There is also a limit for the maximum document size in Mongo which might be the reason for the out of memory error if you have many docs for a given id.
I guess the solution is to use map/reduce instead.