We Keep Coding

NodeJS: How can I scrape two different tables, that are visually part of the same table, into one JSON Object?

Here's an example of the table of data I'm scraping:
The elements in red are in the <th> tags while the elements in green are in a <td> tag, the <tr> tag can be displayed according to how they're grouped (i.e. '1' is in it's own <tr>; HTML snippet:
EDIT: I forgot to add the surrounding div
<div class="table-cont">
<table class="tg-1">
<thead>
<tr>
<th class="tg-phtq">ID</td>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0pky">1</td>
<td class="tg-0pky">2</td>
<td class="tg-0pky">3</td>
</tr>
</tbody>
</table>
<table class="tg-2">
<thead>
<tr>
<th class="tg-phtq">Sample1</td>
<th class="tg-phtq">Sample2</td>
<...the rest of the table code matches the pattern...>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0pky">Swimm</td>
<td class="tg-dvpl">1:30</td>
<...>
</tr>
</tbody>
<...the rest of the table code...>
</table>
</div>
As you can see, in the HTML they're actually two different tables while they're displayed in the above example as only one. I want to generate a JSON object where the keys and values include the data from the two tables as if they were one, and output a single JSON Object.
How I'm scraping it right now is a bit of modified javascript code I found on a tutorial:
EDIT: In the below, I've been trying to find a way to select all relevant <th> tags from both tables and insert them into the same array as the rest of the <th> tag array and do the same for <tr> in the table body; I'm fairly sure for the th I can just insert the element separately before the rest but only because there's a single one - I've been having problems figuring out how to do that for both arrays and make sure all the items in the two arrays map correctly to each other
EDIT 2: Possible solution? I tried using XPath Selectors and I can use them in devTools to select everything I want, but page.evaluate doesn't accept them and page.$x('XPath') returns JSHandle#node since I'm trying to make an array, but I don't know where to go from there
let scrapeMemberTable = async (page) => {
await page.evaluate(() => {
let ths = Array.from(document.querySelectorAll('div.table-cont > table.tg-2 > thead > tr > th'));
let trs = Array.from(document.querySelectorAll('div.table-cont > table.tg-2 > tbody > tr'));
// the above two lines of code are the main problem area- I haven't been
//able to select all the head/body elements I want in just those two lines of code
// just removig the table id "tg-2" seems to deselect the whole thing
const headers = ths.map(th => th.textContent);
let results = [];
trs.forEach(tr => {
let r = {};
let tds = Array.from(tr.querySelectorAll('td')).map(td => td.textContent);
headers.forEach((k,i) => r[k] = tds[i]);
results.push(r);
});
return results; //results is OBJ in JSON format
}
}
...
results = results.concat( //merge into one array OBJ
await scrapeMemberTable(page)
);
...
Intended Result:
[
{
"ID": "1", <-- this is the goal
"Sample1": "Swimm",
"Sample2": "1:30",
"Sample3": "2:05",
"Sample4": "1:15",
"Sample5": "1:41"
}
]
Actual Result:
[
{
"Sample1": "Swimm",
"Sample2": "1:30",
"Sample3": "2:05",
"Sample4": "1:15",
"Sample5": "1:41"
}
]

Changing <td> <i> CSS based on the value

I have been asked to implement a small validation on values and if the values are greater or less than 0 i need to change or add/remove the css for the td and i tag
My table looks something like this
<table class="table table-hover" id="studentweek">
<thead>
<tr>
<th>Year</th>
<th">Weeks</th>
</tr>
</thead>
<tbody>
<tr>
<td>VAR (%)</td>
<td class="text-warning"> <i class="classname">-10.65%</i></td>
</tr>
<tr>
<td>VAR (diff)</td>
<td class="text-warning"> <i class="classname">-13,953</i></td>
</tr>
<tr>
<td>VAR (%)</td>
<td class="text-navy"> <i class="classname">8.81%</i></td>
</tr>
<tr>
<td>VAR (diff)</td>
<td class="text-navy"> <i class="classname">11,320</i></td>
</tr>
</tbody>
</table>
currently i am hard coding the css but i would like to be able to dynamicly change these as the values change automatically, can someone suggest the best way to archive this?
i was thinking in my Ajax request to do something like this:
var sdlyvar = $(parseFloat(".classname").text());
if (sdlyvar < 0){
$('.classname').removeClass(".classname").addClass("fa-level-down");
} else {
$('.classname').removeClass(".classname").addClass("fa-level-up");
}

Use JavaScript parseFloat for parsing percentage (http://www.w3schools.com/jsref/jsref_parsefloat.asp).
var percent = $('#sdlyvar').text();
var result = parseFloat(percent) / 100.0;
if (result < 0){
$('#sdlyvar').removeClass("fa-level-up");
$('#sdlyvar').addClass("fa-level-down")
} else {
$('#sdlyvar').removeClass("fa-level-down");
$('#sdlyvar').addClass("fa-level-up")
}

Your first problem is that you can't compare a string like "-10.95%" with an integer, because of the final % symbol. You have to use parseFloat on tha value:
var sdlyvar = parseFloat($('#sdlyvar').text());
It will take care of all the non-numeric stuff after the number.
Then, you'd probably want to remove the opposite class when updating:
if (sdlyvar < 0){
$('#sdlyvar').removeClass("fa-level-up").addClass("fa-level-down");
} else {
$('#sdlyvar').removeClass("fa-level-down").addClass("fa-level-up");
}
A few random suggestions:
Make clear what's wrong in your code when posting on StackOverflow
When referring an element more than once with jQuery, consider putting the selection in a variable, like var $sdlyvar = $("sdlyvar");: faster to type and execute.
Save us some whitespaces when posting code :/

Here .slice will remove the % sign in this code and the rest of the code will compare the value and assign or remove class
var sdlyvar = $('#sdlyvar').text();
if (sdlyvar.slice(0,-1) < 0){
$('#sdlyvar').removeClass("fa-level-up");
$('#sdlyvar').addClass("fa-level-down");
} else {
$('#sdlyvar').removeClass("fa-level-down");
$('#sdlyvar').addClass("fa-level-up");
}

var lis=document.querySelectorAll("tr td i");
for(var i in lis){
if(parseInt(lis[i].innerHTML)<0){
lis[i].className+=" fa-level-down";
}
else{
lis[i].className+=" fa-level-up";
}
}

how to convert/transform an HTML table tbody (with rowspans) TO json?

I have an HTML table with combined row td's, or how to say, I don't know how to express myself (I am not so good at English), so I show it! This is my table:
<table border="1">
<thead>
<tr>
<th>line</th>
<th>value1</th>
<th>value2</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2">1</td>
<td>1.1</td>
<td>1.2</td>
</tr>
<tr>
<td>1.3</td>
<td>1.4</td>
</tr>
<tr>
<td rowspan="2">2</td>
<td>2.1</td>
<td>2.2</td>
</tr>
<tr>
<td>2.3</td>
<td>2.4</td>
</tr>
</tbody>
</table>
(you can check it here)
I want to convert this table to a JSON variable by jquery or javascript.
How should it look like, and how should I do it? Thank you, if you can help me!

if you want to convert only text use this one :
var array = [];
$('table').find('thead tr').each(function(){
$(this).children('th').each(function(){
array.push($(this).text());
})
}).end().find('tbody tr').each(function(){
$(this).children('td').each(function(){
array.push($(this).text());
})
})
var json = JSON.stringify(array);

To make a somehow representation of your table made no problem to me, but the problem is how to parse it back to HTML! Here a JSON with the first 6 tags:
{"table":{"border":1,"thead":{"th":{"textContent":"line","tr":"textContent":"value1",...}}}}}...
OR for better understanding:
{"tag":"table","border":1,"child":{"tag":"thead","child":{"tag":"th","textContent":"line",
"child":{"tag":"tr","textContent":"value1","child":...}}}}...
Closing tags are included.
For further explanations I need to know whether your table is a string or part of the DOM.

I belive this is what you want:
var jsonTable = {};
// add a new array property named: "columns"
$('table').find('thead tr').each(function() {
jsonTable.columns = $(this).find('th').text();
};
// now add a new array property which contains your rows: "rows"
$('table').find('tbody tr').each(function() {
var row = {};
// add data by colum names derived from "tbody"
for(var i = 0; i < jsonTable.columnsl.length; i++) {
row[ col ] = $(this).find('td').eq( i ).text();
}
// push it all to the results..
jsonTable.rows.push( row );
};
alert(JSON.stringify(jsonTable));
I think there should be some corrections, but this is it I think.

jQuery - selector inside another selector - passed in variable loses its value and becomes an indexer?

I'm somewhat new to jQuery and am just wondering how I go about passing in a string value rather than what appears to be a reference to a jQuery item from a selector? I'm having a hard time explaining so here's a sample demo. Don't even know what to title this so please have at editing the title if you can think of a better one.
At the line where I do $("td").filter(function(str){ the str that is passed in becomes an index position of which TD I'm in. So while debugging the first time in it's a 0 the next time a 1 and so on. I tried google but I'm not even sure what to search for, any documentation/code help would be much appreciated
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<script type="text/javascript">
$(document).ready(function(){
$("select[name='showTeam']").change(function () {
$("select[name='showTeam'] option:selected").each(function () {
var str = $(this).val().toLowerCase();
//str = what it was set to up there
//alert(str);
$("td").filter(function(str) {
//str = becomes a number = to position of TD.. ie for 5th TD match STR = 4 (starts at index 0)
return $(this).text().toLowerCase().indexOf(str) != -1;
}).css('background','red');
});
})
});
</script>
Show Team: <select id="showTeam" name="showTeam">
<option>All</option>
<option>Chelsea</option>
</select>
<div id="games">
<table border="1">
<tbody>
<tr>
<th>ID</th>
<th>Game date</th>
<th>Field</th>
<th>Home team</th>
<th>Home team score</th>
<th>Away team</th>
<th>Away team score</th>
<th>Game type</th>
</tr>
<tr class="odd_line" id="game_460">
<td>459</td>
<td>03 Nov 19:00</td>
<td>Field 2</td>
<td>Madrid </td>
<td>3</td>
<td>Bayern Munich </td>
<td>1</td>
<td>Season</td>
</tr>
<tr class="odd_line" id="game_461">
<td>460</td>
<td>03 Nov 19:00</td>
<td>Field 3</td>
<td>chelsea</td>
<td>5</td>
<td>arsenal</td>
<td>4</td>
<td>Season</td>
</tr>
</div>

$(document).ready(function(){
$("#showTeam").change(function () {
var searchFor = $(this).val().toLowerCase();
$("#games table tbody tr td:contains('" + searchFor + "')").parent().css('background','red');
})
});
Demo

Well, yes. The first parameter will refer to the index of the element in the set of matched elements. Just do:
...
$("select[name='showTeam'] option:selected").each(function() {
var str = $(this).val().toLowerCase();
$("td").filter(function() {
return $(this).text().toLowerCase().indexOf(str) != -1;
}).css('background', 'red');​
...
since str will already be available within the scope of the filter callback function.
From the docs:
.filter( function(index) )
function(index)A function used as a test for each element in the set.
this is the current DOM element.

$(document).ready(function(){
$("#showTeam").change(function() {
var target = $("#showTeam").val();
$("#games td:contains(" + target + ")").css('background','red');
});
});
I've made a jsfiddle to demonstrate this.
http://jsfiddle.net/Zf5dA/
Notes:
:contains() is case sensitive so I had to make "Chelsea" capitalized in the table.
I simplified the selector on the select element - it has an id, so I selected that. Faster and simpler.
This will find td cells that contain the text, but they can also contain other text. This will get you started.

jQuery Table Row Filtering by Column

I'm trying to filter table rows in an intelligent way (as opposed to just tons of code that get the job done eventually) but a rather dry of inspiration.
I have 5 columns in my table. At the top of each there is either a dropdown or a textbox with which the user may filter the table data (basically hide the rows that don't apply)
There are plenty of table filtering plugins for jQuery but none that work quite like this, and thats the complicated part :|

Here is a basic filter example http://jsfiddle.net/urf6P/3/
It uses the jquery selector :contains('some text') and :not(:contains('some text')) to decide if each row should be shown or hidden. This might get you going in a direction.
EDITED to include the HTML and javascript from the jsfiddle:
$(function() {
$('#filter1').change(function() {
$("#table td.col1:contains('" + $(this).val() + "')").parent().show();
$("#table td.col1:not(:contains('" + $(this).val() + "'))").parent().hide();
});
});

Slightly enhancing the accepted solution posted by Jeff Treuting, filtering capability can be extended to make it case insensitive. I take no credit for the original solution or even the enhancement. The idea of enhancement was lifted from a solution posted on a different SO post offered by Highway of Life.
Here it goes:
// Define a custom selector icontains instead of overriding the existing expression contains
// A global js asset file will be a good place to put this code
$.expr[':'].icontains = function(a, i, m) {
return $(a).text().toUpperCase()
.indexOf(m[3].toUpperCase()) >= 0;
};
// Now perform the filtering as suggested by #jeff
$(function() {
$('#filter1').on('keyup', function() { // changed 'change' event to 'keyup'. Add a delay if you prefer
$("#table td.col1:icontains('" + $(this).val() + "')").parent().show(); // Use our new selector icontains
$("#table td.col1:not(:icontains('" + $(this).val() + "'))").parent().hide(); // Use our new selector icontains
});
});

This may not be the best way to do it, and I'm not sure about the performance, but an option would be to tag each column (in each row) with an id starting with a column identifier and then a unique number like a record identifier.
For example, if you had a column Produce Name, and the record ID was 763, I would do something like the following:
​​<table id="table1">
<thead>
<tr>
<th>Artist</th>
<th>Album</th>
<th>Genre</th>
<th>Price</th>
</tr>
</thead>
<tbody>
<tr>
<td id="artist-127">Red Hot Chili Peppers</td>
<td id="album-195">Californication</td>
<td id="genre-1">Rock</td>
<td id="price-195">$8.99</td>
</tr>
<tr>
<td id="artist-59">Santana</td>
<td id="album-198">Santana Live</td>
<td id="genre-1">Rock</td>
<td id="price-198">$8.99</td>
</tr>
<tr>
<td id="artist-120">Pink Floyd</td>
<td id="album-183">Dark Side Of The Moon</td>
<td id="genre-1">Rock</td>
<td id="price-183">$8.99</td>
</tr>
</tbody>
</table>
You could then use jQuery to filter based on the start of the id.
For example, if you wanted to filter by the Artist column:
var regex = /Hot/;
$('#table1').find('tbody').find('[id^=artist]').each(function() {
if (!regex.test(this.innerHTML)) {
this.parentNode.style.backgroundColor = '#ff0000';
}
});

You can filter specific column by just adding children[column number] to JQuery filter. Normally, JQuery looks for the keyword from all the columns in every row. If we wanted to filter only ColumnB on below table, we need to add childern[1] to filter as in the script below. IndexOf value -1 means search couldn't match. Anything above -1 will make the whole row visible.
ColumnA | ColumnB | ColumnC
John Doe 1968
Jane Doe 1975
Mike Nike 1990
$("#myInput").on("change", function () {
var value = $(this).val().toLowerCase();
$("#myTable tbody tr").filter(function () {
$(this).toggle($(this.children[1]).text().toLowerCase().indexOf(value) > -1)
});
});

step:1 write the following in .html file
<input type="text" id="myInput" onkeyup="myFunction()" placeholder="Search for names..">
<table id="myTable">
<tr class="header">
<th style="width:60%;">Name</th>
<th style="width:40%;">Country</th>
</tr>
<tr>
<td>Alfreds Futterkiste</td>
<td>Germany</td>
</tr>
<tr>
<td>Berglunds snabbkop</td>
<td>Sweden</td>
</tr>
<tr>
<td>Island Trading</td>
<td>UK</td>
</tr>
<tr>
<td>Koniglich Essen</td>
<td>Germany</td>
</tr>
</table>
step:2 write the following in .js file
function myFunction() {
// Declare variables
var input, filter, table, tr, td, i;
input = document.getElementById("myInput");
filter = input.value.toUpperCase();
table = document.getElementById("myTable");
tr = table.getElementsByTagName("tr");
// Loop through all table rows, and hide those who don't match the search query
for (i = 0; i < tr.length; i++) {
td = tr[i].getElementsByTagName("td")[0];
if (td) {
if (td.innerHTML.toUpperCase().indexOf(filter) > -1) {
tr[i].style.display = "";
} else {
tr[i].style.display = "none";
}
}
}
}