Initial data:
The server has html with clogged source data for which jscript using jquery requests information from the site for a number of criteria via the provided API.

Data is requested by ID sequentially (from the API side there is a limit on the number of requests per second, but this is not the case), for example, 1-10000 or 25000000-30000000.
The answer received in json parsed and displayed on the same page.

Html source code:

  <html> <head> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width,initial-scale=1.0"> <title> API test</title> </head> <body ng-app="Test_App"> <div id="bodyMain"> Filter Greater Than Last Battle: <input type="date" id="lastDate" /> Search By ID (FROM): <input type="number" id="fromID" value="x"/> Search By ID (TO): <input type="number" id="toID" value="xxxxx" /> <br> <br> WinRate From %: <input value="xx" type="number" id="winRateFrom" /> To <input value="xx" type="number" id="winRateTo" /> <br> <!-- default item for winrate filter --> Exp: <input type="number" id="_avg_xp_Default" value="xxx" /> <br> Rating : <input type="number" id="_global_rating_Default" value="xxx" /> <br> battles : <input type="number" id="_clan_battles_Default" value="xxx" /> <br> Attack <input type="number" id="_str_A_battles_Default" value="xxx" /> <br> Defence : <input type="number" id="_str_D_battles_Default" value="xxx" /> <br> <input type="button" value="Search" onclick="getFunction(true)" /> <br> ID Received : <span id="pendingID">0</span> of <span id="toPendingID">0</span> &nbsp;<span id="alertMsg"></span> <br> <input type="button" onclick="exportExcel(event)" id="btnExport" value="Export as Excel File" /> <table id="table_wrapp" style="width: 1200px;"> <thead> <th>Account ID</th> <th>Nickname</th> <th>Last Battle</th> <th>Created at</th> <th>Global Rating</th> <th>Average XP</th> <th>Clan battles All </th> <th>Stronghold Atack</th> <th>Stronghold Defense</th> <th>Win</th> <th>Loses</th> <th>Win %</th> </tr> </thead> <tbody> </tbody> </table> </div> <script src="jquery-3.2.1.js" defer="defer"></script> <script src="jquery.excel.js" defer="defer"></script> <script src="main.js?v2" defer="defer"></script> </body> </html>

Script source code:

 var accountFields = 'statistics.all.losses%2Cstatistics.all.wins%2Cstatistics.all.battles%2Cclan_id%2Cprivate.gold%2Caccount_id%2Clast_battle_time%2Ccreated_at%2Cglobal_rating%2Cstatistics.all.battle_avg_xp%2Cstatistics.clan.battles%2Cstatistics.globalmap_absolute.battles%2Cstatistics.globalmap_champion.battles%2Cstatistics.stronghold_defense.battles%2Cstatistics.stronghold_skirmish.battles%2Cnickname%2Cclient_language%2Cglobal_rating' //Separted by , > "%2C" without quote var applicationID = '********'; var reqPerMilSecond = 40 ; //Request Per Milisecond. Will send request every 40 Milisec (1000 = 1sec) var counter = 1; var counter2 = 1; var max = 100000000; var html = [] function getFunction(flag){ if(flag){ $('#alertMsg').text('Please Wait') counter = parseInt($('#fromID').val()); max = parseInt($('#toID').val()); $('table>tbody').html('<tr><td colspan="5">Waiting For Response</td></tr>'); $('#toPendingID').text(max); if(counter > max){ alert('To ID must be greater than From') return; } } if($('#lastDate').val() == ''){ alert('Please Input Date') return; } var query = getIds(); main(query,function(data){ var html = []; data = $.parseJSON(data); var data = data.data var date = new Date(document.getElementById("lastDate").value); var winT = parseInt($('#winRateTo').val()) var winF = parseInt($('#winRateFrom').val()) var _avg_xp_D = parseInt($('#_avg_xp_Default').val()) var _global_rating_D = parseInt($('#_global_rating_Default').val()) var _clan_battles_D = parseInt($('#_clan_battles_Default').val()) var _str_A_battles_D = parseInt($('#_str_A_battles_Default').val()) var _str_D_battles_D = parseInt($('#_str_D_battles_Default').val()) for(attr in data){ if(data[attr] && !data[attr]['clan_id']){ //This will check if there's a valid Player (data[attr]) var _date = new Date(parseInt(data[attr]['last_battle_time'])*1000); var _battles = parseInt(data[attr]['statistics']['all']['battles']) || 0; var _wins = parseInt(data[attr]['statistics']['all']['wins']) || 0; var _winRate = (_wins/_battles)*100; var _avg_xp = parseInt(data[attr]['statistics']['all']['battle_avg_xp']) || 0; var _global_rating = parseInt(data[attr]['global_rating']) || 0; var _clan_battles_all = parseInt(data[attr]['statistics']['clan']['battles']) || 0; var _str_A_battles = parseInt(data[attr]['statistics']['stronghold_skirmish']['battles']) || 0; var _str_D_battles = parseInt(data[attr]['statistics']['stronghold_defense']['battles']) || 0; if(_date.getTime() >= date.getTime()){ //Filter player time if (winF<=_winRate && winT>=_winRate && _avg_xp>=_avg_xp_D && _global_rating>=_global_rating_D && _clan_battles_all>=_clan_battles_D && _str_A_battles>=_str_A_battles_D && _str_D_battles>=_str_D_battles_D) { var tempHTML = '<tr>'; var _lastBattle = getCurrDate(new Date(parseInt(data[attr]['last_battle_time']) * 1000)); var tempHTML = '<tr>'; var _lastBattle = getCurrDate(new Date(parseInt(data[attr]['last_battle_time']) * 1000)); var _loses = parseInt(data[attr]['statistics']['all']['losses']) || 0; var _created_at = getCurrDate(new Date(parseInt(data[attr]['created_at']) * 1000)); var tempHTML = '<tr>'; tempHTML += '<td>'+data[attr]['account_id']+'</td>'; tempHTML += '<td>'+data[attr]['nickname']+'</td>'; tempHTML += '<td>'+_lastBattle+'</td>'; tempHTML += '<td>'+_created_at+'</td>'; tempHTML += '<td>'+_global_rating+'</td>'; tempHTML += '<td>'+_avg_xp+'</td>'; tempHTML += '<td>'+_clan_battles_all+'</td>'; tempHTML += '<td>'+_str_A_battles+'</td>'; tempHTML += '<td>'+_str_D_battles+'</td>'; tempHTML += '<td>'+_wins+'</td>'; tempHTML += '<td>'+_loses+'</td>'; tempHTML += '<td>'+parseFloat(_winRate).toFixed(2)+'</td>'; tempHTML += '</tr>' html.push(tempHTML); } } } } if(tempHTML!=''){ $('table>tbody').append(html.join()); } counter2 += 100; if(counter2 >= max) { $('#alertMsg').text('Done') } $('#pendingID').text(counter2); },function() { }) if(counter >= max){ }else{ setTimeout(function(){counter += 100; getFunction();}, reqPerMilSecond); } } function getIds(){ var maxID = counter +100; var html2 = []; for(var x = counter;x<maxID;x++){ html2.push(x) } return html2.join('%2C'); } return html2.join('%2C'); } main = function(query,callback,errorCallback){ var contentType ="application/x-www-form-urlencoded; charset=utf-8"; var url = 'https://*****adress_syte*****/account/info/?application_id='+applicationID+'&'; url += 'account_id='+query+'&'; url += 'fields='+accountFields query = query; var dataSend = query $.ajax({ url :url, data : { 'data': dataSend }, dataType: 'html', contentType:contentType, method:'POST', success: function(datares){ datares = $.trim(datares); var mydata = datares.replace(/'/g,'"'); callback(mydata); },error: function(xhr){ } }); } getCurrDate = function(d){ var mon = (d.getMonth()+1) var day = ( d.getDate() ) if(mon.toString().length == 1){ mon = "0"+mon.toString(); } if(day.toString().length == 1){ day = "0"+day.toString(); } var strDate = d.getFullYear() + "-" + mon + "-" + day; var hr = d.getHours(); var min = d.getMinutes(); return strDate+' '+hr+':'+min; }

If you request a small range like 1-100000 and choose a small request rate - everything is ok.
But if you request a polling of the ID range of several millions, for example 30,000,000-35,000,000, then an error gets out in the chrome logs (it is established experimentally that even at a low polling rate):

jquery-3.2.1.js: 9566 POST https: // **** query address **** net :: ERR_INSUFFICIENT_RESOURCES

line 9566 itself:

// Do send the request (this may raise an exception) xhr.send (options.hasContent && options.data || null);

I, as a zero-level programmer, suspect that this either the server itself does not pull (it does not have enough sockets, xs, I don’t know), or somewhere in the code some kind of memory leak or something like that.

If necessary, I can lay out the timeline and profiles of Chrome.

  • Maybe even hint who, in which direction to dig? - Jora_S
  • It is interesting that if you redirect the output to a file, and not to update the html, then no errors occur. - Jora_S
  • but not ... the same thing ... although according to subjective feelings the survey range lasts longer - Jora_S
  • Maybe so the server does not have time to accept all the answers? and how can this be traced? - Jora_S

1 answer 1

If someone has a similar problem, the search algorithm was something like this:

  1. Make an alternative output (to the log / file / database) and compare the speed / time of processing.

  2. To trace requests in the browser

  3. In my case, it was clear that the requests were coming, and the recording was late (I did not clarify the exact reason)

  4. I reconfigured a number of parameters in /etc/sysctl.conf responsible for the number of detachable sockets, files, several timeouts, etc. - one of these parameters worked. Errors in the trace disappeared and all responses to requests are processed.