How to improve performance of ngRepeat over a huge dataset (angular.js)?

I have a huge dataset of several thousand rows with around 10 fields each, about 2MBs of data. I need to display it in the browser. Most straightforward approach (fetch data, put it into $scope, let ng-repeat="" do its job) works fine, but it freezes the browser for about half of a minute when it starts inserting nodes into DOM. How should I approach this problem?

One option is to append rows to $scope incrementally and wait for ngRepeat to finish inserting one chunk into DOM before moving to the next one. But AFAIK ngRepeat does not report back when it finishes "repeating", so it's going to be ugly.

Other option is to split data on the server into pages and fetch them in multiple requests, but that's even uglier.

I looked through Angular documentation in search of something like ng-repeat="data in dataset" ng-repeat-steps="500", but found nothing. I am fairly new to Angular ways, so it is possible that I am missing the point completely. What are the best practices at this?

I agree with @AndreM96 that the best approach is to display only a limited amount of rows, faster and better UX, this could be done with a pagination or with an infinite scroll.

Infinite scroll with Angular is really simple with limitTo filter. You just have to set the initial limit and when the user asks for more data (I am using a button for simplicity) you increment the limit.

<table>
    <tr ng-repeat="d in data | limitTo:totalDisplayed"><td>{{d}}</td></tr>
</table>
<button class="btn" ng-click="loadMore()">Load more</button>

//the controller
$scope.totalDisplayed = 20;

$scope.loadMore = function () {
  $scope.totalDisplayed += 20;  
};

$scope.data = data;

Here is a JsBin.

This approach could be a problem for phones because usually they lag when scrolling a lot of data, so in this case I think a pagination fits better.

To do it you will need the limitTo filter and also a custom filter to define the starting point of the data being displayed.

Here is a JSBin with a pagination.

The hottest - and arguably most scalable - approach to overcoming these challenges with large datasets is embodied by the approach of Ionic's collectionRepeat directive and of other implementations like it. A fancy term for this is 'occlusion culling', but you can sum it up as: don't just limit the count of rendered DOM elements to an arbitrary (but still high) paginated number like 50, 100, 500... instead, limit only to as many elements as the user can see.

If you do something like what's commonly known as "infinite scrolling", you're reducing the initial DOM count somewhat, but it bloats quickly after a couple refreshes, because all those new elements are just tacked on at the bottom. Scrolling comes to a crawl, because scrolling is all about element count. There's nothing infinite about it.

Whereas, the collectionRepeat approach is to use only as many elements as will fit in viewport, and then recycle them. As one element rotates out of view, it's detached from the render tree, refilled with data for a new item in the list, then reattached to the render tree at the other end of the list. This is the fastest way known to man to get new information in and out of the DOM, making use of a limited set of existing elements, rather than the traditional cycle of create/destroy... create/destroy. Using this approach, you can truly implement an infinite scroll.

Note that you don't have to use Ionic to use/hack/adapt collectionRepeat, or any other tool like it. That's why they call it open-source. :-) (That said, the Ionic team is doing some pretty ingenious things, worthy of your attention.)

---

There's at least one excellent example of doing something very similar in React. Only instead of recycling the elements with updated content, you're simply choosing not to render anything in the tree that's not in view. It's blazing fast on 5000 items, although their very simple POC implementation allows a bit of flicker...

---

Also... to echo some of the other posts, using track by is seriously helpful, even with smaller datasets. Consider it mandatory.

I recommend to see this:

• http://blog.scalyr.com/2013/10/angularjs-1200ms-to-35ms/

Optimizing AngularJS: 1200ms to 35ms

they made a new directive by optimizing ng-repeat at 4 parts:

> Optimization#1: Cache DOM elements > > Optimization#2: Aggregate watchers > > Optimization#3: Defer element creation > > Optimization#4: Bypass watchers for hidden elements

the project is here on github:

• https://github.com/scalyr/angular

Usage:

1- include these files in your single-page app:

• core.js
• scalyr.js
• slyEvaluate.js
• slyRepeat.js

2- add module dependency:

var app = angular.module("app", ['sly']);

3- replace ng-repeat

<tr sly-repeat="m in rows"> .....<tr>

ENjoY!

Beside all the above hints like track by and smaller loops, this one also helped me a lot

<span ng-bind="::stock.name"></span>

this piece of code would print the name once it has been loaded, and stop watching it after that. Similarly, for ng-repeats, it could be used as

<div ng-repeat="stock in ::ctrl.stocks">{{::stock.name}}</div>

however it only works for AngularJS version 1.3 and higher. From http://www.befundoo.com/blog/optimizing-ng-repeat-in-angularjs/

You can use "track by" to increase the performance:

<div ng-repeat="a in arr track by a.trackingKey">

Faster than:

<div ng-repeat="a in arr">

ref:https://www.airpair.com/angularjs/posts/angularjs-performance-large-applications

If all your rows have equal height, you should definitely take a look at the virtualizing ng-repeat: http://kamilkp.github.io/angular-vs-repeat/

This demo looks very promising (and it supports inertial scrolling)

Virtual scrolling is another way to improve scrolling performance when dealing with huge lists and large dataset.

One way to implement this is by using Angular Material md-virtual-repeat as it is demonstrated on this Demo with 50,000 items

Taken straight from the documentation of virtual repeat:

> Virtual repeat is a limited substitute for ng-repeat that renders only enough dom nodes to fill the container and recycling them as the user scrolls.

Rule No.1: Never let the user wait for anything.

That in mind a life growing page that needs 10seconds appears way faster than waiting 3 seconds on before a blank screen and get all at once.

So instead of make the page fast, just let the page appear to be fast, even if the final result is slower:

function applyItemlist(items){
	var item = items.shift();
	if(item){
		$timeout(function(){
			$scope.items.push(item);
			applyItemlist(items);
		}, 0); // <-- try a little gap of 10ms
	}
}

The code above let appear the list to be growing row-by-row, and is always slower than render all at once. But for the user it appears to be faster.

Another version @Steffomio

Instead of adding each item individually we can add items by chunks.

// chunks function from here: 
// http://stackoverflow.com/questions/8495687/split-array-into-chunks#11764168
var chunks = chunk(folders, 100);

//immediate display of our first set of items
$scope.items = chunks[0];

var delay = 100;
angular.forEach(chunks, function(value, index) {
	delay += 100;

    // skip the first chuck
	if( index > 0 ) {
		$timeout(function() {
			Array.prototype.push.apply($scope.items,value);
		}, delay);
	}		
});

Sometimes what happened,you get the data from server (or back-end) in few ms (for example I'm I am assuming it 100ms) but it takes more time to display in our web page (let's say it is taking 900ms to display).

So, What is happening here is 800ms It is taking just to render web page.

What I have done in my web application is, I have used pagination (or you can use infinite scrolling also) to display list of data. Let's say I am showing 50 data/page.

So I will not load render all the data at once,only 50 data I am loading initially which takes only 50ms (I'm assuming here).

so total time here decreased from 900ms to 150ms, once user request next page then display next 50 data and so on.

Hope this will help you to improve the performance. All the best

Created a directive (ng-repeat with lazy loading) 

which loads data when it reaches to bottom of the page and remove half of the previously loaded data and when it reaches to top of the div again previous data(depending upon on page number) will be loaded removing half of the current data So on DOM at a time only limited data is present which may leads to better performance instead of rendering whole data on load.

HTML CODE:

<!DOCTYPE html>
<html ng-app="plunker">

  <head>
    <meta charset="utf-8" />
    <title>AngularJS Plunker</title>
    <script>document.write('<base href="' + document.location + '" />');</script>
    <link rel="stylesheet" href="style.css" />
    <script src="https://code.jquery.com/jquery-2.2.4.min.js" integrity="sha256-BbhdlvQf/xTY9gja0Dq3HiwQF8LaCRTXxZKRutelT44=" crossorigin="anonymous"></script>
    <script data-require="[email protected]" src="https://code.angularjs.org/1.3.20/angular.js" data-semver="1.3.20"></script>
    <script src="app.js"></script>
  </head>

  <body ng-controller="ListController">
  <div class="row customScroll" id="customTable" datafilter pagenumber="pageNumber" data="rowData" searchdata="searchdata" itemsPerPage="{{itemsPerPage}}"  totaldata="totalData"   selectedrow="onRowSelected(row,row.index)"  style="height:300px;overflow-y: auto;padding-top: 5px">
   
    <!--<div class="col-md-12 col-xs-12 col-sm-12 assign-list" ng-repeat="row in CRGC.rowData track by $index | orderBy:sortField:sortReverse | filter:searchFish">-->
    <div class="col-md-12 col-xs-12 col-sm-12 pdl0 assign-list" style="padding:10px" ng-repeat="row in rowData" ng-hide="row[CRGC.columns[0].id]=='' && row[CRGC.columns[1].id]==''">
        <!--col1-->

        <div ng-click ="onRowSelected(row,row.index)"> <span>{{row["sno"]}}</span> <span>{{row["id"]}}</span> <span>{{row["name"]}}</span></div>
      <!--   <div class="border_opacity"></div> -->
    </div>

</div>

  </body>

</html>

Angular CODE:

var app = angular.module('plunker', []);
var x;
ListController.$inject = ['$scope', '$timeout', '$q', '$templateCache'];

function ListController($scope, $timeout, $q, $templateCache) {
  $scope.itemsPerPage = 40;
  $scope.lastPage = 0;
  $scope.maxPage = 100;
  $scope.data = [];
  $scope.pageNumber = 0;


  $scope.makeid = function() {
    var text = "";
    var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";

    for (var i = 0; i < 5; i++)
      text += possible.charAt(Math.floor(Math.random() * possible.length));

    return text;
  }
 
 
  $scope.DataFormFunction = function() {
      var arrayObj = [];
      for (var i = 0; i < $scope.itemsPerPage*$scope.maxPage; i++) {
          arrayObj.push({
              sno: i + 1,
              id: Math.random() * 100,
              name: $scope.makeid()
          });
      }
      $scope.totalData = arrayObj;
      $scope.totalData = $scope.totalData.filter(function(a,i){ a.index = i; return true; })
      $scope.rowData = $scope.totalData.slice(0, $scope.itemsperpage);
    }
  $scope.DataFormFunction();

  $scope.onRowSelected = function(row,index){
    console.log(row,index);
  }

}

angular.module('plunker').controller('ListController', ListController).directive('datafilter', function($compile) {
  return {
    restrict: 'EAC',
    scope: {
      data: '=',
      totalData: '=totaldata',
      pageNumber: '=pagenumber',
      searchdata: '=',
      defaultinput: '=',
      selectedrow: '&',
      filterflag: '=',
      totalFilterData: '='
    },
    link: function(scope, elem, attr) {
      //scope.pageNumber = 0;
      var tempData = angular.copy(scope.totalData);
      scope.totalPageLength = Math.ceil(scope.totalData.length / +attr.itemsperpage);
      console.log(scope.totalData);
      scope.data = scope.totalData.slice(0, attr.itemsperpage);
      elem.on('scroll', function(event) {
        event.preventDefault();
      //  var scrollHeight = angular.element('#customTable').scrollTop();
      var scrollHeight = document.getElementById("customTable").scrollTop
        /*if(scope.filterflag && scope.pageNumber != 0){
        scope.data = scope.totalFilterData;
        scope.pageNumber = 0;
        angular.element('#customTable').scrollTop(0);
        }*/
        if (scrollHeight < 100) {
          if (!scope.filterflag) {
            scope.scrollUp();
          }
        }
        if (angular.element(this).scrollTop() + angular.element(this).innerHeight() >= angular.element(this)[0].scrollHeight) {
          console.log("scroll bottom reached");
          if (!scope.filterflag) {
            scope.scrollDown();
          }
        }
        scope.$apply(scope.data);

      });
      
      /*
       * Scroll down data append function
       */
      scope.scrollDown = function() {
          if (scope.defaultinput == undefined || scope.defaultinput == "") { //filter data append condition on scroll
            scope.totalDataCompare = scope.totalData;
          } else {
            scope.totalDataCompare = scope.totalFilterData;
          }
          scope.totalPageLength = Math.ceil(scope.totalDataCompare.length / +attr.itemsperpage);
          if (scope.pageNumber < scope.totalPageLength - 1) {
            scope.pageNumber++;
            scope.lastaddedData = scope.totalDataCompare.slice(scope.pageNumber * attr.itemsperpage, (+attr.itemsperpage) + (+scope.pageNumber * attr.itemsperpage));
            scope.data = scope.totalDataCompare.slice(scope.pageNumber * attr.itemsperpage - 0.5 * (+attr.itemsperpage), scope.pageNumber * attr.itemsperpage);
            scope.data = scope.data.concat(scope.lastaddedData);
            scope.$apply(scope.data);
            if (scope.pageNumber < scope.totalPageLength) {
              var divHeight = $('.assign-list').outerHeight();
              if (!scope.moveToPositionFlag) {
                angular.element('#customTable').scrollTop(divHeight * 0.5 * (+attr.itemsperpage));
              } else {
                scope.moveToPositionFlag = false;
              }
            }


          }
        }
        /*
         * Scroll up data append function
         */
      scope.scrollUp = function() {
          if (scope.defaultinput == undefined || scope.defaultinput == "") { //filter data append condition on scroll
            scope.totalDataCompare = scope.totalData;
          } else {
            scope.totalDataCompare = scope.totalFilterData;
          }
          scope.totalPageLength = Math.ceil(scope.totalDataCompare.length / +attr.itemsperpage);
          if (scope.pageNumber > 0) {
            this.positionData = scope.data[0];
            scope.data = scope.totalDataCompare.slice(scope.pageNumber * attr.itemsperpage - 0.5 * (+attr.itemsperpage), scope.pageNumber * attr.itemsperpage);
            var position = +attr.itemsperpage * scope.pageNumber - 1.5 * (+attr.itemsperpage);
            if (position < 0) {
              position = 0;
            }
            scope.TopAddData = scope.totalDataCompare.slice(position, (+attr.itemsperpage) + position);
            scope.pageNumber--;
            var divHeight = $('.assign-list').outerHeight();
            if (position != 0) {
              scope.data = scope.TopAddData.concat(scope.data);
              scope.$apply(scope.data);
              angular.element('#customTable').scrollTop(divHeight * 1 * (+attr.itemsperpage));
            } else {
              scope.data = scope.TopAddData;
              scope.$apply(scope.data);
              angular.element('#customTable').scrollTop(divHeight * 0.5 * (+attr.itemsperpage));
            }
          }
        }
    }
  };
});

Demo with directive

Another Solution: If you using UI-grid in the project then  same implementation is there in UI grid with infinite-scroll.

Depending upon height of the division it loads the data and upon scroll new data will be append and previous data will be removed.

HTML Code:

<!DOCTYPE html>
<html ng-app="plunker">

  <head>
    <meta charset="utf-8" />
    <title>AngularJS Plunker</title>
    <script>document.write('<base href="' + document.location + '" />');</script>
    <link rel="stylesheet" href="style.css" />
    <link rel="stylesheet" href="https://cdn.rawgit.com/angular-ui/bower-ui-grid/master/ui-grid.min.css" type="text/css" />
    <script data-require="[email protected]" src="https://code.angularjs.org/1.3.20/angular.js" data-semver="1.3.20"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/angular-ui-grid/4.0.6/ui-grid.js"></script>
    <script src="app.js"></script>
  </head>

  <body ng-controller="ListController">
     <div class="input-group" style="margin-bottom: 15px">
      <div class="input-group-btn">
        <button class='btn btn-primary' ng-click="resetList()">RESET</button>
      </div>
      <input class="form-control" ng-model="search" ng-change="abc()">
    </div>
    
    <div data-ui-grid="gridOptions" class="grid" ui-grid-selection  data-ui-grid-infinite-scroll style="height :400px"></div>
    
    <button ng-click="getProductList()">Submit</button>
  </body>

</html>

Angular Code:

var app = angular.module('plunker', ['ui.grid', 'ui.grid.infiniteScroll', 'ui.grid.selection']);
var x;
angular.module('plunker').controller('ListController', ListController);
ListController.$inject = ['$scope', '$timeout', '$q', '$templateCache'];

function ListController($scope, $timeout, $q, $templateCache) {
    $scope.itemsPerPage = 200;
    $scope.lastPage = 0;
    $scope.maxPage = 5;
    $scope.data = [];

    var request = {
        "startAt": "1",
        "noOfRecords": $scope.itemsPerPage
    };
    $templateCache.put('ui-grid/selectionRowHeaderButtons',
        "<div class=\"ui-grid-selection-row-header-buttons \" ng-class=\"{'ui-grid-row-selected': row.isSelected}\" ><input style=\"margin: 0; vertical-align: middle\" type=\"checkbox\" ng-model=\"row.isSelected\" ng-click=\"row.isSelected=!row.isSelected;selectButtonClick(row, $event)\">&nbsp;</div>"
    );


    $templateCache.put('ui-grid/selectionSelectAllButtons',
        "<div class=\"ui-grid-selection-row-header-buttons \" ng-class=\"{'ui-grid-all-selected': grid.selection.selectAll}\" ng-if=\"grid.options.enableSelectAll\"><input style=\"margin: 0; vertical-align: middle\" type=\"checkbox\" ng-model=\"grid.selection.selectAll\" ng-click=\"grid.selection.selectAll=!grid.selection.selectAll;headerButtonClick($event)\"></div>"
    );

    $scope.gridOptions = {
        infiniteScrollDown: true,
        enableSorting: false,
        enableRowSelection: true,
        enableSelectAll: true,
        //enableFullRowSelection: true,
        columnDefs: [{
            field: 'sno',
            name: 'sno'
        }, {
            field: 'id',
            name: 'ID'
        }, {
            field: 'name',
            name: 'My Name'
        }],
        data: 'data',
        onRegisterApi: function(gridApi) {
            gridApi.infiniteScroll.on.needLoadMoreData($scope, $scope.loadMoreData);
            $scope.gridApi = gridApi;
        }
    };
    $scope.gridOptions.multiSelect = true;
    $scope.makeid = function() {
        var text = "";
        var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";

        for (var i = 0; i < 5; i++)
            text += possible.charAt(Math.floor(Math.random() * possible.length));

        return text;
    }
    $scope.abc = function() {
        var a = $scope.search;
        x = $scope.searchData;
        $scope.data = x.filter(function(arr, y) {
            return arr.name.indexOf(a) > -1
        })
        console.log($scope.data);
        if ($scope.gridApi.grid.selection.selectAll)
            $timeout(function() {
                $scope.gridApi.selection.selectAllRows();
            }, 100);
    }


    $scope.loadMoreData = function() {
        var promise = $q.defer();
        if ($scope.lastPage < $scope.maxPage) {
            $timeout(function() {
                var arrayObj = [];
                for (var i = 0; i < $scope.itemsPerPage; i++) {
                    arrayObj.push({
                        sno: i + 1,
                        id: Math.random() * 100,
                        name: $scope.makeid()
                    });
                }

                if (!$scope.search) {
                    $scope.lastPage++;
                    $scope.data = $scope.data.concat(arrayObj);
                    $scope.gridApi.infiniteScroll.dataLoaded();
                    console.log($scope.data);
                    $scope.searchData = $scope.data;
                    // $scope.data = $scope.searchData;
                    promise.resolve();
                    if ($scope.gridApi.grid.selection.selectAll)
                        $timeout(function() {
                            $scope.gridApi.selection.selectAllRows();
                        }, 100);
                }


            }, Math.random() * 1000);
        } else {
            $scope.gridApi.infiniteScroll.dataLoaded();
            promise.resolve();
        }
        return promise.promise;
    };

    $scope.loadMoreData();

    $scope.getProductList = function() {

        if ($scope.gridApi.selection.getSelectedRows().length > 0) {
            $scope.gridOptions.data = $scope.resultSimulatedData;
            $scope.mySelectedRows = $scope.gridApi.selection.getSelectedRows(); //<--Property undefined error here
            console.log($scope.mySelectedRows);
            //alert('Selected Row: ' + $scope.mySelectedRows[0].id + ', ' + $scope.mySelectedRows[0].name + '.');
        } else {
            alert('Select a row first');
        }
    }
    $scope.getSelectedRows = function() {
        $scope.mySelectedRows = $scope.gridApi.selection.getSelectedRows();
    }
    $scope.headerButtonClick = function() {

        $scope.selectAll = $scope.grid.selection.selectAll;

    }
}
  

Demo with UI grid with infinite-scroll Demo

for large data set and multiple value drop down it is better to use ng-options rather than ng-repeat.

ng-repeat is slow because it loops over all coming values but ng-options simply display to the select option.

ng-options='state.StateCode as state.StateName for state in States'>

much much faster than

<option ng-repeat="state in States" value="{{state.StateCode}}">
    {{state.StateName }}
</option>