We Keep Coding

NodeJS: How can I scrape two different tables, that are visually part of the same table, into one JSON Object?

Here's an example of the table of data I'm scraping:
The elements in red are in the <th> tags while the elements in green are in a <td> tag, the <tr> tag can be displayed according to how they're grouped (i.e. '1' is in it's own <tr>; HTML snippet:
EDIT: I forgot to add the surrounding div
<div class="table-cont">
<table class="tg-1">
<thead>
<tr>
<th class="tg-phtq">ID</td>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0pky">1</td>
<td class="tg-0pky">2</td>
<td class="tg-0pky">3</td>
</tr>
</tbody>
</table>
<table class="tg-2">
<thead>
<tr>
<th class="tg-phtq">Sample1</td>
<th class="tg-phtq">Sample2</td>
<...the rest of the table code matches the pattern...>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0pky">Swimm</td>
<td class="tg-dvpl">1:30</td>
<...>
</tr>
</tbody>
<...the rest of the table code...>
</table>
</div>
As you can see, in the HTML they're actually two different tables while they're displayed in the above example as only one. I want to generate a JSON object where the keys and values include the data from the two tables as if they were one, and output a single JSON Object.
How I'm scraping it right now is a bit of modified javascript code I found on a tutorial:
EDIT: In the below, I've been trying to find a way to select all relevant <th> tags from both tables and insert them into the same array as the rest of the <th> tag array and do the same for <tr> in the table body; I'm fairly sure for the th I can just insert the element separately before the rest but only because there's a single one - I've been having problems figuring out how to do that for both arrays and make sure all the items in the two arrays map correctly to each other
EDIT 2: Possible solution? I tried using XPath Selectors and I can use them in devTools to select everything I want, but page.evaluate doesn't accept them and page.$x('XPath') returns JSHandle#node since I'm trying to make an array, but I don't know where to go from there
let scrapeMemberTable = async (page) => {
await page.evaluate(() => {
let ths = Array.from(document.querySelectorAll('div.table-cont > table.tg-2 > thead > tr > th'));
let trs = Array.from(document.querySelectorAll('div.table-cont > table.tg-2 > tbody > tr'));
// the above two lines of code are the main problem area- I haven't been
//able to select all the head/body elements I want in just those two lines of code
// just removig the table id "tg-2" seems to deselect the whole thing
const headers = ths.map(th => th.textContent);
let results = [];
trs.forEach(tr => {
let r = {};
let tds = Array.from(tr.querySelectorAll('td')).map(td => td.textContent);
headers.forEach((k,i) => r[k] = tds[i]);
results.push(r);
});
return results; //results is OBJ in JSON format
}
}
...
results = results.concat( //merge into one array OBJ
await scrapeMemberTable(page)
);
...
Intended Result:
[
{
"ID": "1", <-- this is the goal
"Sample1": "Swimm",
"Sample2": "1:30",
"Sample3": "2:05",
"Sample4": "1:15",
"Sample5": "1:41"
}
]
Actual Result:
[
{
"Sample1": "Swimm",
"Sample2": "1:30",
"Sample3": "2:05",
"Sample4": "1:15",
"Sample5": "1:41"
}
]

How to make this table design in angular with css

I data in this format in my angular controller. These are dummy datas and will be fetched from some sort of services later.
$scope.attendanceLog =
{
attendances:
[
{
date:'12.12.17',
entries:
[
{
time:'12PM',
device:'1212',
location:'katabon'
},
{
time:'1PM',
device:'1212',
location:'katabon'
},
{
time:'2PM',
device:'1321',
location:'katabon'
}
]
},
{
date:'13.12.17',
entries:
[
{
time:'12PM',
device:'1212',
location:'katabon'
},
{
time:'1PM',
device:'1212',
location:'katabon'
},
{
time:'2PM',
device:'1321',
location:'katabon'
},
{
time:'5PM',
device:'1321',
location:'katabon'
}
]
}
]
};
Now I designed the table to view this data like this. Here is the html code
for the table
<table class="table table-bordered">
<thead>
<th>Date</th>
<th class="text-center">Time</th>
<th class="text-center">Device</th>
<th class="text-center">Location</th>
</thead>
<tbody>
<tr ng-repeat-start="attendance in attendanceLog.attendances">
<td rowspan="{{attendance.entries.length}}" class="date">{{attendance.date}}</td>
<td>{{attendance.entries[0].time}}</td>
<td>{{attendance.entries[0].device}}</td>
<td>{{attendance.entries[0].location}}</td>
</tr>
<tr ng-repeat-end ng-repeat="entries in attendance.entries" ng-if="$index>0">
<td>{{entries.time}}</td>
<td>{{entries.device}}</td>
<td>{{entries.location}}</td>
</tr>
</tbody>
I want to make every other instance of the highlighted sections' background a diffrent color.Here is the reference image.
So if there are 5 dates then the 1st, 3rd and 5th date cell and all the other cells on their right side would have a different color.
Now is there any way to do this with angular. I am sorry if its a stupid question. I am new to front end development.

You could change it to have one expression for the table entries and use ng-class-odd and ng-class-even:
<tbody>
<tr ng-repeat="..."
ng-class-odd="'someOddClass'" ng-class-even="'someEvenClass'">
</tr>
</tbody>
Then you'd just need to change your styling.

Instead of printing new rows in the table I created a new table for every date. And then I applied the css class on every odd numbered table.
<table class="table table-bordered" ng-repeat="...." class =
ng-class ="$index % 2 == 0 ? 'table table-bordered striped':'table table-bordered'" >
.......
.......
</table>
Here striped is the class I used to highlight the odd numbered records background a different color

DataTables footerCallback and contenteditable

I have a DataTable that sums over each column using footerCallback. This works perfectly given the data in each column, but I also want to add the ability to change each cell's value that is being summed. I've tried adding "contenteditable" to those cells, but making a change does not affect the sum in the footer.
Here is a simple jsfiddle showing the behavior I'm experiencing: https://jsfiddle.net/rantoun/552y9j90/6/
HTML:
<table id="table1">
<thead>
<tr>
<th>Fruit</th>
<th># Eaten</th>
<th># Remaining</th>
</tr>
</thead>
<tfoot>
<tr>
<th align="right">Count</th>
<th align="left"></th>
<th align="left"></th>
</tr>
</tfoot>
<tbody>
<tr>
<td>Apples</td>
<td contenteditable>3</td>
<td contenteditable>8</td>
</tr>
<tr>
<td>Oranges</td>
<td contenteditable>6</td>
<td contenteditable>5</td>
</tr>
<tr>
<td>Bananas</td>
<td contenteditable>2</td>
<td contenteditable>9</td>
</tr>
</tbody>
</table>
jQuery:
$("#table1").DataTable({
"paging": false,
"searching": false,
"info": false,
"footerCallback": function ( row, data, start, end, display ) {
var columns = [1, 2];
var api = this.api();
_.each(columns, function(idx) {
var total = api
.column(idx)
.data()
.reduce(function (a, b) {
return parseInt(a) + parseInt(b);
}, 0)
$('tr:eq(0) th:eq('+idx+')', api.table().footer()).html(total);
})
}
});
I have also found the DataTables editor (https://editor.datatables.net/examples/inline-editing/simple), which would be perfect for this situation - but it is not open source. Any ideas on how to mimic this inline editing functionality is welcome. I would like to avoid doing this with modals. Any help is appreciated!

Here is an answer that allows you to edit in place with contenteditable.
Note that this requires the dataTables KeyTable plugin
Working Fiddle here
/* Note - requires datatable.keys plugin */
var table = $("#table1").DataTable({
"keys": true,
"paging": false,
"searching": false,
"info": false,
"footerCallback": function ( row, data, start, end, display ) {
var columns = [1, 2];
var api = this.api();
_.each(columns, function(idx) {
var total = api
.column(idx)
.data()
.reduce(function (a, b) {
return parseInt(a) + parseInt(b);
}, 0)
$('tr:eq(0) th:eq('+idx+')', api.table().footer()).html(total);
})
}
});
// keys introduces the key-blur event where we can detect moving off a cell
table
.on( 'key-blur', function ( e, datatable, cell ) {
// The magic part - using the cell object, get the HTML node value,
// put it into the dt cell cache and redraw the table to invoke the
// footer function.
cell.data( $(cell.node()).html() ).draw()
} );
Note - I can foresee that you may get non-numeric data entered. You will have to police that in the key-blur event where you can test the dom node value and act accordingly.

Adding select menu to multiple tables with jquery data tables pluglin

I've got two tables (table1,table2) that are structurally the same, but fed from different sources. Both tables share the class 'display' which I use to initialise the plugin for both tables. This works great for the most part, however the column filters select menu for table2 is duplicated on table1.
I've tried to fix this by using 'this' keyword to indicate a particular instance of the toolbar each of the filters need applying to, but not had much success.
The code that works is:
HTML:
<table id="table1" class="display table-striped table-borded table-hover" cellspacing="0" width="100%">
<thead>
<tr>
<th>Report Name</th>
<th>Year</th>
<th>Montly Snapshot</th>
</tr>
</thead>
<tfoot>
<tr>
<th>Date of Last Refresh --var from warehouse-- </th>
</tr>
</tfoot>
</table>
<table id="table2" class="display table-striped table-borded table-hover" cellspacing="0" width="100%">
<thead>
<tr>
<th>Report Name</th>
<th>Year</th>
<th>Montly Snapshot</th>
</tr>
</thead>
<tfoot>
<tr>
<th>Date of Last Refresh --var from warehouse-- </th>
</tr>
</tfoot>
</table>
JS:
//initialise data tables plugin and apply custom settings.
var reportstable = $('table.display').DataTable({
"sDom": '<"toolbar">Bfrtlp',
"searchCols": [
null, {
'sSearch': 'Current'
}
],
language: {
search: "_INPUT_",
searchPlaceholder: "Search for a Report..."
},
// create select form controls
initComplete: function() {
this.api().columns([1, 2]).every(function() {
var column = this;
var select = $('<select><option value=""></option></select>')
.appendTo($('.toolbar'))
.on('change', function() {
var val = $.fn.dataTable.util.escapeRegex(
$(this).val()
);
column
.search(val ? '^' + val + '$' : '', true, false)
.draw();
});
column.data().unique().sort().each(function(d, j) {
select.append('<option value="' + d + '">' + d + '</option>')
if ($('#info').css('display') == 'block') {
$("#reports_wrapper").css('display', 'none');
}
// add bootstrap classes and placeholder property to form controls to add style
$('.toolbar select').addClass('selectpicker');
$('.dt-button').addClass('btn btn-danger').removeClass('dt-button');
$('.toolbar select').attr('id', 'year');
$('#year').prop('title', 'Financial Year');
$("#year option:contains('Current')").remove();
$('.toolbar select:nth-of-type(2)').attr('id', 'month');
$('#month').prop('title', 'Monthy Snapshot');
$("#month option:contains('undefined')").remove();
});
});
},
// create clear filter button
buttons: [
{
text: 'Clear Filters',
action: function(e, dt, node, config) {
$('select').val('').selectpicker('refresh');
// set Current Year as default filter
reportstable
.search('')
.columns([1]).search('Current')
.columns([2]).search('')
.draw();
}
}
],
//hide specified columns
"columnDefs": [
{
"targets": [1],
"visible": false,
"searchable": true
}, {
"targets": [2],
"visible": false,
"searchable": true
}
]
});

I've updates your jsfiddle and created a new jsfiddle. It's now appending 2 select menus in both tables's wrapper div. I've found why it's appending 4 select menus instead of 2 on the table1's wrapper div. It's because of this code: .appendTo($('.toolbar')). When initComplete callback function is called for table1 there is only one div with class=toolbar, and when the initComplete callback function is called for table2 there is two div with class=toolbar, one in table1's wrapper div and one in table2's wrapper div. that's why on table2's initComplete callback function it append select menus to all divs with class=toolbar. We should rather append to the current table wrapper's div with class=toolbar.

Angular custom filter logs out as working with some matches but filters out all results

I think I may have a syntax error in here somewhere. The idea is that the user selects currencies that they are interested in. Subsequently I only want to print out rows of the currencies table that have a matching currency. I am using a custom filter to achieve this.
I want to either fix my custom filter syntax or look at a way to use a built in filter to achieve the same thing.
//directive html
<table class="table table-striped">
<thead>
<tr>
<th>Currency</th>
<th>Gram Price</th>
<th>Ounce Price</th>
<th>Kilo Price</th>
<th>24 hr Change</th>
<th>1 yr Change</th>
<th>10 yr Change</th>
</tr>
</thead>
<tbody>
<tr ng-repeat="quote in vm.quotes | selectedCurrencies:vm.selectedCurrencies()">
<th scope="row">{{quote.currency}}</th>
<td>{{quote.price}}</td>
<td>{{quote.ouncePrice}}</td>
<td>{{quote.kiloPrice}}</td>
<td>{{quote.change.d1}}</td>
<td>{{quote.change.y1}}</td>
<td>{{quote.change.y10}}</td>
</tr>
</tbody>
</table>
//filter
angular.module('bgAngularTest').filter('selectedCurrencies', function($log) {
return function(quotes, selectedCurrencies) {
if(!quotes.length){
return;
}
return quotes.filter(function(quote) {
selectedCurrencies.forEach(function(cur){
$log.info(quote.currency +' ==== '+ cur + ' ?');
if(cur === quote.currency){
$log.info('match');
return true;
}
})
return false;
});
};
});
//example of quote data
{
"quotes": [{
"currency": "USD",
"price": 40.55,
"symbol": "$",
"change": {
"d1": 0.0116,
"y1": 0.0425,
"y10": 1.1029
}
}, ...]
}
//example of currency list
["RUB","USD","EUR"]

Your selectedCurrencies.forEach doesn't actually return anything, because the callback is internal to the forEach:
if(cur === quote.currency){
$log.info('match');
return true;
}
You return false all the time, and all your items are filtered out. Try this instead:
return quotes.filter(function(quote) {
return selectedCurrencies.some(function(cur){ // if at least one currency matches return true
return cur === quote.currency;
});
});