即日起在codingBlog上分享您的技术经验即可获得积分,积分可兑换现金哦。

Java语言多线程多任务开发下载

编程语言 DamonM 13℃ 0评论

在日常的工作中,经常会遇到大文件下载问题,如果采用单线程的话,可能会耗时比较长,效率不高,因此,使用多线程分段处理来联合下载统一资源。

核心思路:分而治之

  • 分段切割:HTTP1.1协议定义了断点续传相关的Range和Content-Range字段,通过HttpUrlConnection类的setRequestProperty(“Range”, “bytes=” + start + “-” + end)方法来实现内容分段切割;
  • 分段传输:HttpUrlConnection类的getInputStream()方法,以流的方式传输目标数据;
  • 分段保存:RandomAccessFile类提供了一系列随机访问和读取文件的方法,通过seek(int start)方法便可轻松的在指定位置写入数据。

Java核心代码demo如下:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.*;
import java.net.URL;
import java.net.URLConnection;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 * DownloadUtil
 * 下载工具类(多线程下载)
 *
 * @author damon
 * @date 2017/4/19
 */
public final class DownloadUtil {

    private static final Logger LOGGER = LoggerFactory.getLogger(DownloadUtil.class);

    // 获取线程数大小
    private static final int THREAD_SIZE = Runtime.getRuntime().availableProcessors() + 1;

    // 定义线程池
    private static final ExecutorService DOWNLOAD_THREAD_POOL = Executors.newFixedThreadPool(THREAD_SIZE);

    private DownloadUtil() {
        throw new Error("Do not instantiate!");
    }

    /**
     * 下载文件
     *
     * @param downloadUrl 文件地址
     * @param destPath 文件存储地址
     */
    public static void download(String downloadUrl, String destPath){
        try {
            // 创建连接,并获取大小
            URL url = new URL(downloadUrl);
            URLConnection conn = url.openConnection();
            long contentLength = conn.getContentLength();

            File destFile = new File(destPath);
            if (!destFile.exists()) {
                destFile.createNewFile();
            }

            // 设置每个线程下载大小
            long subLength = contentLength / THREAD_SIZE;
            for (int i = 0; i < THREAD_SIZE; i++) {
                long startPos = subLength * i;
                long endPos = subLength * (i + 1) - 1;
                DownloadThread downloadThread = new DownloadThread(url, startPos, endPos, destFile);
                DOWNLOAD_THREAD_POOL.execute(downloadThread);
            }

            LOGGER.info("Download successful, from url:{}, to:{}", downloadUrl, destPath);
        } catch (Exception e) {
            LOGGER.error("Download failed, from url:{}, to:{}, details:{}", downloadUrl, destPath, e.getMessage(), e);
        }
    }
}

class DownloadThread implements Runnable {

    private static final int SIZE_BYTE = 1024; // 每次下载大小

    private static final String FILE_MODE = "rw"; // 读写式文件

    private URL url; // 下载url
    private long currentPos; // 当前下载位置
    private long startPos; // 下载开始位置
    private long endPos; // 下载结束位置
    private File file; // 存储文件

    public DownloadThread(URL url, long startPos, long endPos, File file) {
        this.url = url;
        this.currentPos = startPos;
        this.startPos = startPos;
        this.endPos = endPos;
        this.file = file;
    }

    @Override
    public void run() {
        RandomAccessFile randomAccessFile = null;
        BufferedInputStream bufferedInputStream = null;
        URLConnection connection = null;
        byte[] buff = new byte[SIZE_BYTE];

        try {
            // 创建连接,并设置开始和结束位置
            connection = url.openConnection();
            connection.setAllowUserInteraction(true);
            connection.setRequestProperty("Range", "bytes=" + startPos + "-" + endPos);

            // 创建随机读取文件
            randomAccessFile = new RandomAccessFile(file, FILE_MODE);
            // 设置起始位置
            randomAccessFile.seek(startPos);

            // 读取网络流数据,并写入文件
            bufferedInputStream = new BufferedInputStream(connection.getInputStream());
            while (currentPos < endPos) {
                int len = bufferedInputStream.read(buff, 0, SIZE_BYTE);
                if (len == -1) {
                    break;
                }

                randomAccessFile.write(buff, 0, len);
                currentPos += len;
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            // 关闭buffer文件
            try {
                if (null != bufferedInputStream) {
                    bufferedInputStream.close();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }

            // 关闭random文件
            try {
                if (null != randomAccessFile) {
                    randomAccessFile.close();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

转载请注明:CodingBlog » Java语言多线程多任务开发下载

喜欢 (0)or分享 (0)
发表我的评论
取消评论

*

表情