The next task is: there are a million records in the database, you need to select all the records. They said that sampling just a million records at once is bad for memory.

Need implementation spring-data-jpa pagination without the parameters used. That is, in my Service , already filtered data will be transferred to the controller, but in order to filter it is necessary to iterate through all the records, you need to do a search for 10-20 thousand records.

This is what my Service looks like now. In pageable , the page size is 5, just for example.

 @Transactional public List<Contact> getAllContactByFilter(String filter) { List<Contact> result = new ArrayList<>(); Page<Contact> contactList = contactRepository.findAll(createPageRequest()); logger.info("contList size: " + contactList.getSize()); Pattern p = Pattern.compile(filter); Matcher matcher = null; logger.info("get all contacts by filter"); for (Iterator<Contact> iterator = contactList.iterator(); iterator.hasNext();) { Contact currentContact = iterator.next(); matcher = p.matcher(currentContact.getName()); if (matcher.matches()) { iterator.remove(); } } return result; } private Pageable createPageRequest() { PageRequest request = new PageRequest(0, 5); return request; }
  • What is your question? - Slava Semushin
  • In this implementation, there is only one request to the database and the first 5 records are selected. It is necessary that, depending on what size of the page I specified, send inquiries to the database, pull out the records, check the matcher and add them to the result list. Ie was the passage of all records in the database, but not at one time. - Igor Golyak
  • Somewhere in the documentation Spring-Data-JPA something was like that, but look for laziness for you. Have you already searched there? - Slava Semushin

1 answer 1

If I understand you correctly, it will look something like this:

 public List<Contact> getAllContactByFilter(int pageSize, String filter){ List<Contact> result = new ArrayList<>(); PageRequest request = new PageRequest(0, pageSize); Page<Contact> page = contactRepository.findAll(request); for (int i = 0; i <= page.getTotalPages(); i++) { List<Contact> listPage = contactRepository.findAll(request.next()).getContent(); //фильтрация result.addAll(listPage); } return result; }

those. You will go through all the records with the number of these records you choose per page.