Hello.

The idea is to create a range based on the file size, each range on its own stream, i.e. if for example the file size is 100,000 bytes, and we divide this size into several streams, then we get ranges of 0-1000, 1001-2000,2001-3000, etc. I wrote my own algorithm.

public List<String> getThreadsDiapason(int startBytes, int endBytes, int c_thread) { int step = (int)(endBytes / c_thread); List<Integer> temp_start = new ArrayList<>(); List<Integer> temp_end = new ArrayList<>(); temp_start.add(startBytes); temp_end.add(step); try { for(int i = 1; i < c_thread; i++) { temp_start.add(i,temp_end.get(i - 1) + 1); temp_end.add(i,temp_end.get(i - 1) + step); } } catch (Exception ex) { Log.e("getThreadsDiapason", ex.getMessage()); } int last_value = temp_end.get(c_thread-1) + (endBytes - temp_end.get(c_thread-1)); temp_end.set(c_thread-1,last_value); List<String> finalList = new ArrayList<>(); try { for(int i = 0; i < c_thread; i++) { finalList.add(temp_start.get(i)+"-"+temp_end.get(i)); } } catch (Exception ex) { Log.e("getThreadsDiapason",ex.getMessage()); } return finalList; }

After it is necessary to run this list, create threads with parameters equal to each of the cells of the list. Wrote the method described below:

  public void threadedDownload(final String url,final String path,final int size) throws Exception { int threads_count = 32; final int stock_buffer = 4096 * threads_count; final List<String> threadsDiap = getThreadsDiapason(0,size,threads_count); final List<Thread> threadList = new ArrayList<>(); for(int i = 0; i < threads_count; i++) { final int diap_i = i; Thread t = new Thread(new Runnable() { @Override public void run() { byte[] buffer = new byte[stock_buffer]; int bytesRead = -1; try { System.out.println("Thread "+diap_i+" Started : " + threadsDiap.get(diap_i)); HttpURLConnection conn = null; FileOutputStream fileOutputStream = null; try { conn = (HttpURLConnection)new URL(url).openConnection(); } catch (IOException e) { Log.e("threadedDownload",e.getMessage()); } conn.setDoInput(true); conn.setConnectTimeout(100000); conn.setRequestProperty("Range", "bytes="+threadsDiap.get(diap_i)); fileOutputStream = new FileOutputStream(path ,true); conn.connect(); InputStream inputStream = conn.getInputStream(); while ((bytesRead = inputStream.read(buffer)) != -1) { fileOutputStream.write(buffer, Integer.parseInt(threadsDiap.get(diap_i).split("-")[0]), bytesRead); } fileOutputStream.flush(); fileOutputStream.close(); System.out.println("Thread "+diap_i+" Ended : " + threadsDiap.get(diap_i)); }catch (Exception ex) { Log.e("Error",ex.getMessage()); } } }); threadList.add(t); } final List<Thread> finalThreadList = threadList; new Thread(new Runnable() { @Override public void run() { for(int i = 0; i < finalThreadList.size(); i++) { finalThreadList.get(i).start(); } } }).start(); }

But when I start, I get errors:

 E/Error: length=131072; regionStart=84024925; regionLength=1360 E/Error: length=131072; regionStart=33609970; regionLength=894 E/Error: length=131072; regionStart=201659820; regionLength=1360 E/Error: length=131072; regionStart=235269790; regionLength=1580 E/Error: length=131072; regionStart=100829910; regionLength=2048 E/Error: length=131072; regionStart=168049850; regionLength=2048 E/Error: length=131072; regionStart=117634895; regionLength=2048 E/Error: length=131072; regionStart=50414955; regionLength=1582 E/Error: length=131072; regionStart=184854835; regionLength=1580 E/Error: length=131072; regionStart=16804985; regionLength=1589 E/Error: length=131072; regionStart=319294715; regionLength=1580 E/Error: length=131072; regionStart=302489730; regionLength=2048 E/Error: length=131072; regionStart=285684745; regionLength=1580 E/Error: length=131072; regionStart=268879760; regionLength=2048 E/Error: length=131072; regionStart=369709670; regionLength=1360 E/Error: length=131072; regionStart=470539580; regionLength=1580 E/Error: length=131072; regionStart=252074775; regionLength=1580 E/Error: length=131072; regionStart=67219940; regionLength=2048 E/Error: length=131072; regionStart=420124625; regionLength=1580 E/Error: length=131072; regionStart=386514655; regionLength=1580 E/Error: length=131072; regionStart=453734595; regionLength=2048 E/Error: length=131072; regionStart=520954535; regionLength=1580 E/Error: length=131072; regionStart=436929610; regionLength=1580 E/Error: length=131072; regionStart=487344565; regionLength=2048 E/Error: length=131072; regionStart=336099700; regionLength=2048 E/Error: length=131072; regionStart=403319640; regionLength=1580 E/Error: length=131072; regionStart=537759529; regionLength=1580 E/Error: length=131072; regionStart=504149550; regionLength=1580 E/Error: length=131072; regionStart=151244865; regionLength=2048 E/Error: length=131072; regionStart=352904685; regionLength=1580 E/Error: length=131072; regionStart=218464805; regionLength=1580 E/Error: length=131072; regionStart=134439880; regionLength=1580

I tried different ways, the latter is probably the most correct, specify the offset and write to the fileOutputStream.write(buffer, Integer.parseInt(threadsDiap.get(diap_i).split("-")[0]), bytesRead); However, it does not work correctly.

Guide me on the right path.

  • Thanks, I'll try a little later, i.e. use RandomAccessFile.seek (<specify start of record in bytes>)? - just_koala
  • I suppose that some kind of laboratory work is being solved, and not the task of a really working System? After all, if a file is downloaded from one source to one consumer, then at least download it in one stream, even in multiple streams - the channel between them will not increase. That is, the speed will remain the same, but the reliability of the functional will noticeably decrease, not to mention the readability of the source code ... - bobzer
  • @bobzer can and so to say, laboratory work, all possible files that will be downloaded using this method will be downloaded from Chinese servers, with normal, single-threaded downloading, the speed from them reaches unbelievable 40kb per second, sometimes more, with streams the speed grows to 5 MB per second, which significantly speeds up the download of large files - just_koala

1 answer 1

At the prompt @zRrr, the following happened:

The Range class which deals with the creation of a list of ranges with different data output

  class Range { public List<String> rangeList = new ArrayList<>(); Range(int startBytes, int endBytes, int c_thread) { rangeList = getThreadsDiapason(startBytes,endBytes,c_thread); } public String getRangeString(int rangeId) { if(rangeList.size() < rangeId) return null; else return rangeList.get(rangeId); } public int getFirstRangeInt(int rangeId) { if(rangeList.size() < rangeId) return 0; else return Integer.parseInt(rangeList.get(rangeId).split("-")[0]); } public int getLastRangeInt(int rangeId) { if(rangeList.size() < rangeId) return 0; else return Integer.parseInt(rangeList.get(rangeId).split("-")[1]); } private List<String> getThreadsDiapason(int startBytes, int endBytes, int c_thread) { int step = (int)(endBytes / c_thread); List<Integer> temp_start = new ArrayList<>(); List<Integer> temp_end = new ArrayList<>(); temp_start.add(startBytes); temp_end.add(step); try { for(int i = 1; i < c_thread; i++) { temp_start.add(i,temp_end.get(i - 1) + 1); temp_end.add(i,temp_end.get(i - 1) + step); } } catch (Exception ex) { Log.e("getThreadsDiapason", ex.getMessage()); } int last_value = temp_end.get(c_thread-1) + (endBytes - temp_end.get(c_thread-1)); temp_end.set(c_thread-1,last_value); List<String> finalList = new ArrayList<>(); try { for(int i = 0; i < c_thread; i++) { finalList.add(temp_start.get(i)+"-"+temp_end.get(i)); } } catch (Exception ex) { Log.e("getThreadsDiapason",ex.getMessage()); } return finalList; } }

And the threadedDownload method, using RandomAccessFile:

  public void threadedDownload(final String url,final String path,final int size, int threads_count) throws Exception { final int stock_buffer = 4096 * threads_count; final Range range = new Range(0,size,threads_count); final List<Thread> threadList = new ArrayList<>(); for(int i = 0; i < threads_count; i++) { final int diap_i = i; Thread t = new Thread(new Runnable() { @Override public void run() { byte[] buffer = new byte[stock_buffer]; int bytesRead = -1; try { System.out.println("Thread "+diap_i+" Started : " + range.getRangeString(diap_i)); HttpURLConnection conn = null; RandomAccessFile randomAccessFile = null; try { conn = (HttpURLConnection)new URL(url).openConnection(); } catch (IOException e) { Log.e("threadedDownload",e.getMessage()); } conn.setDoInput(true); conn.setConnectTimeout(100000); conn.setRequestProperty("Range", "bytes="+range.getRangeString(diap_i)); randomAccessFile = new RandomAccessFile(path, "rw"); randomAccessFile.seek(range.getFirstRangeInt(diap_i)); conn.connect(); InputStream inputStream = conn.getInputStream(); while ((bytesRead = inputStream.read(buffer)) != -1) { randomAccessFile.write(buffer, 0, bytesRead); } randomAccessFile.close(); System.out.println("Thread "+diap_i+" Ended : " + range.getRangeString(diap_i)); }catch (Exception ex) { Log.e("Error",ex.getMessage()); } } }); threadList.add(t); } final List<Thread> finalThreadList = threadList; new Thread(new Runnable() { @Override public void run() { for(int i = 0; i < finalThreadList.size(); i++) { finalThreadList.get(i).start(); } } }).start(); }
  • I added an external variable, in which I add bytesRead from all the threads, at the output I get an amount not equal to the file size, what could be the problem? - just_koala 5:59