澳门新萄京官方网站-www.8455.com-澳门新萄京赌场网址

网络爬虫,多线程网络爬虫

2019-06-08 作者:www.8455.com   |   浏览(80)

上次做了贰个帮公司妹子做了爬虫,不是很精美,本次公司项目里要选取,于是有做了壹番修改,功用增加了网址图片采撷,下载,线程管理分界面网站图片下载等。

C#网络爬虫--二十四线程管理强化版,

上次做了四个帮集团妹子做了爬虫,不是很精密,此次公司项目里要运用,于是有做了1番修改,效能增添了网址图片搜聚,下载,线程管理分界面网站图片下载等。

说说思路:首相获取开端网站的装有内容 在开始网站收集图片 去早先网站搜聚链接 把募集到的链接放入队列 继续搜聚图片,然后继续采访链接,Infiniti循环

或然上海体育场所片大家看一下,在上代码!

澳门新萄京官方网站 1

 

拍卖网页内容抓取跟网页网址爬取都做了立异,上边照旧大家来看望代码,有不足之处,还请之处!

网页内容抓取HtmlCodeRequest,

网页网站爬取GetHttpLinks,用正则去筛选html中的Links

图片抓取GetHtmlImageUrlList,用正则去筛选html中的Img

都写进了一个封装类里面 HttpHelper

    /// <summary>   
        /// 取得HTML中所有图片的 URL。   
        /// </summary>   
        /// <param name="sHtmlText">HTML代码</param>   
        /// <returns>图片的URL列表</returns> 
public static string HtmlCodeRequest(string Url)
        {
            if (string.IsNullOrEmpty(Url))
            {
                return "";
            }
            try
            {
                //创建一个请求
                HttpWebRequest httprequst = (HttpWebRequest)WebRequest.Create(Url);
                //不建立持久性链接
                httprequst.KeepAlive = true;
                //设置请求的方法
                httprequst.Method = "GET";
                //设置标头值
                httprequst.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
                httprequst.Accept = "*/*";
                httprequst.Headers.Add("Accept-Language", "zh-cn,en-us;q=0.5");
                httprequst.ServicePoint.Expect100Continue = false;
                httprequst.Timeout = 5000;
                httprequst.AllowAutoRedirect = true;//是否允许302
                ServicePointManager.DefaultConnectionLimit = 30;
                //获取响应
                HttpWebResponse webRes = (HttpWebResponse)httprequst.GetResponse();
                //获取响应的文本流
                string content = string.Empty;
                using (System.IO.Stream stream = webRes.GetResponseStream())
                {
                    using (System.IO.StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding("utf-8")))
                    {
                        content = reader.ReadToEnd();
                    }
                }
                //取消请求
                httprequst.Abort();
                //返回数据内容
                return content;
            }
            catch (Exception)
            {

                return "";
            }
        }
/// <summary>
        /// 提取页面链接
        /// </summary>
        /// <param name="html"></param>
        /// <returns></returns>
public static List<string> GetHtmlImageUrlList(string url)
        {
            string html = HttpHelper.HtmlCodeRequest(url);
            if (string.IsNullOrEmpty(html))
            {
                return new List<string>();
            }
            // 定义正则表达式用来匹配 img 标签   
            Regex regImg = new Regex(@"<imgb[^<>]*?bsrc[strn]*=[strn]*[""']?[strn]*(?<imgUrl>[^strn""'<>]*)[^<>]*?/?[strn]*>", RegexOptions.IgnoreCase);

            // 搜索匹配的字符串   
            MatchCollection matches = regImg.Matches(html);
            List<string> sUrlList = new List<string>();

            // 取得匹配项列表   
            foreach (Match match in matches)
                sUrlList.Add(match.Groups["imgUrl"].Value);
            return sUrlList;
        }


        /// <summary>
        /// 提取页面链接
        /// </summary>
        /// <param name="html"></param>
        /// <returns></returns>
        public static List<string> GetHttpLinks(string url)
        {
            //获取网址内容
            string html = HttpHelper.HtmlCodeRequest(url);
            if (string.IsNullOrEmpty(html))
            {
                return new List<string>();
            }
            //匹配http链接
            const string pattern2 = @"http(s)?://([w-] .) [w-] (/[w- ./?%&=]*)?";
            Regex r2 = new Regex(pattern2, RegexOptions.IgnoreCase);
            //获得匹配结果
            MatchCollection m2 = r2.Matches(html);
            List<string> links = new List<string>();
            foreach (Match url2 in m2)
            {
                if (StringHelper.CheckUrlIsLegal(url2.ToString()) || !StringHelper.IsPureUrl(url2.ToString()) || links.Contains(url2.ToString()))
                    continue;
                links.Add(url2.ToString());
            }
            //匹配href里面的链接
            const string pattern = @"(?i)<as[^>]*?href=(['""]?)(?!javascript|__doPostBack)(?<url>[^'""s*#<>] )[^>]*>"; ;
            Regex r = new Regex(pattern, RegexOptions.IgnoreCase);
            //获得匹配结果
            MatchCollection m = r.Matches(html);
            foreach (Match url1 in m)
            {
                string href1 = url1.Groups["url"].Value;
                if (!href1.Contains("http"))
                {
                    href1 = Global.WebUrl   href1;
                }
                if (!StringHelper.IsPureUrl(href1) || links.Contains(href1)) continue;
                links.Add(href1);
            }
            return links;
        }    

原文 C#创制多线程管理强化版网络爬虫

上次做了一个帮公司妹子做了爬虫,不是很精细,此次公司项目里要运用,于是有做了壹番修改,成效加多了网站图片收罗,下载,线程管理分界面网站图片下载等。

说说思路:首相获取初步网址的装有内容 在开班网站搜聚图片 去早先网站搜罗链接 把采访到的链接放入队列 继续收集图片,然后继续搜聚链接,Infiniti循环

那边下载图片有个职责条数限制,限制是200条。若是超过的电话线程等待5秒,这里下载图片是异步调用的寄托

public string DownLoadimg(string url)
        {
            if (!string.IsNullOrEmpty(url))
            {
                try
                {
                    if (!url.Contains("http"))
                    {
                        url = Global.WebUrl   url;
                    }
                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
                    request.Timeout = 2000;
                    request.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
                    //是否允许302
                    request.AllowAutoRedirect = true;
                    WebResponse response = request.GetResponse();
                    Stream reader = response.GetResponseStream();
                    //文件名
                    string aFirstName = Guid.NewGuid().ToString();
                    //扩展名
                    string aLastName = url.Substring(url.LastIndexOf(".")   1, (url.Length - url.LastIndexOf(".") - 1));
                    FileStream writer = new FileStream(Global.FloderUrl   aFirstName   "."   aLastName, FileMode.OpenOrCreate, FileAccess.Write);
                    byte[] buff = new byte[512];
                    //实际读取的字节数
                    int c = 0;
                    while ((c = reader.Read(buff, 0, buff.Length)) > 0)
                    {
                        writer.Write(buff, 0, c);
                    }
                    writer.Close();
                    writer.Dispose();
                    reader.Close();
                    reader.Dispose();
                    response.Close();
                    return (aFirstName   "."   aLastName);
                }
                catch (Exception)
                {
                    return "错误:地址"   url;
                }
            }
            return "错误:地址为空";
        }

 

话非常少说,更加多的内需我们温馨去改良咯!招待读者来与楼主实行交换。即使本文对您有参谋价值,迎接帮博主点下小说下方的引入,感激

上面源码送上:嘿嘿要分的啊!

 

澳门新萄京官方网站, 上次做了贰个帮公司妹子做了爬虫,不是很精美,这一次集团项目里要采用,于是有做了一番修改,功用添...

网络爬虫,多线程网络爬虫。上次做了3个帮企业妹子做了爬虫,不是很精妙,此番公司项目里要使用,于是有做了1番修改,成效增加了网站图片搜聚,下载,线程管理分界面网站图片下载等。

说说思路:首相获取开端网站的保有剧情 在初叶网站收罗图片 去初步网站采撷链接 把采访到的链接放入队列 继续采访图片,然后继续搜集链接,Infiniti循环

要么上海体育场合片大家看一下,在上代码!

说说思路:首相获取开头网站的具有内容 在初阶网站搜集图片 去发轫网站搜聚链接 把采访到的链接放入队列 继续收罗图片,然后继续搜聚链接,Infiniti循环

抑或上海教室片大家看一下:

澳门新萄京官方网站 2

或然上图片我们看一下:

 

澳门新萄京官方网站 3

管理网页内容抓取跟网页网站爬取都做了创新,上面照旧大家来探望代码,有不足之处,还请之处!

拍卖网页内容抓取跟网页网站爬取都做了改正,上面照旧大家来看望代码,有不足之处,还请之处!

管理网页内容抓取跟网页网站爬取都做了革新,上面依旧咱们来探视代码,有不足之处,还请之处!

网页内容抓取HtmlCodeRequest,

网页内容抓取HtmlCodeRequest,

网页内容抓取HtmlCodeRequest,

网页网站爬取GetHttpLinks,用正则去筛选html中的Links

网页网站爬取GetHttpLinks,用正则去筛选html中的Links

网页网站爬取GetHttpLinks,用正则去筛选html中的Links

图表抓取GetHtmlImageUrlList,用正则去筛选html中的Img

图表抓取GetHtmlImageUrlList,用正则去筛选html中的Img

图片抓取GetHtmlImageUrlList,用正则去筛选html中的Img

都写进了四个封装类里面 HttpHelper

都写进了一个封装类里面 HttpHelper

都写进了三个封装类里面 HttpHelper

澳门新萄京官方网站 4

    /// <summary>   
        /// 取得HTML中所有图片的 URL。   
        /// </summary>   
        /// <param name="sHtmlText">HTML代码</param>   
        /// <returns>图片的URL列表</returns> 
public static string HtmlCodeRequest(string Url)
        {
            if (string.IsNullOrEmpty(Url))
            {
                return "";
            }
            try
            {
                //创建一个请求
                HttpWebRequest httprequst = (HttpWebRequest)WebRequest.Create(Url);
                //不建立持久性链接
                httprequst.KeepAlive = true;
                //设置请求的方法
                httprequst.Method = "GET";
                //设置标头值
                httprequst.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
                httprequst.Accept = "*/*";
                httprequst.Headers.Add("Accept-Language", "zh-cn,en-us;q=0.5");
                httprequst.ServicePoint.Expect100Continue = false;
                httprequst.Timeout = 5000;
                httprequst.AllowAutoRedirect = true;//是否允许302
                ServicePointManager.DefaultConnectionLimit = 30;
                //获取响应
                HttpWebResponse webRes = (HttpWebResponse)httprequst.GetResponse();
                //获取响应的文本流
                string content = string.Empty;
                using (System.IO.Stream stream = webRes.GetResponseStream())
                {
                    using (System.IO.StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding("utf-8")))
                    {
                        content = reader.ReadToEnd();
                    }
                }
                //取消请求
                httprequst.Abort();
                //返回数据内容
                return content;
            }
            catch (Exception)
            {

                return "";
            }
        }
/// <summary>
        /// 提取页面链接
        /// </summary>
        /// <param name="html"></param>
        /// <returns></returns>
public static List<string> GetHtmlImageUrlList(string url)
        {
            string html = HttpHelper.HtmlCodeRequest(url);
            if (string.IsNullOrEmpty(html))
            {
                return new List<string>();
            }
            // 定义正则表达式用来匹配 img 标签   
            Regex regImg = new Regex(@"<imgb[^<>]*?bsrc[strn]*=[strn]*[""']?[strn]*(?<imgUrl>[^strn""'<>]*)[^<>]*?/?[strn]*>", RegexOptions.IgnoreCase);

            // 搜索匹配的字符串   
            MatchCollection matches = regImg.Matches(html);
            List<string> sUrlList = new List<string>();

            // 取得匹配项列表   
            foreach (Match match in matches)
                sUrlList.Add(match.Groups["imgUrl"].Value);
            return sUrlList;
        }


        /// <summary>
        /// 提取页面链接
        /// </summary>
        /// <param name="html"></param>
        /// <returns></returns>
        public static List<string> GetHttpLinks(string url)
        {
            //获取网址内容
            string html = HttpHelper.HtmlCodeRequest(url);
            if (string.IsNullOrEmpty(html))
            {
                return new List<string>();
            }
            //匹配http链接
            const string pattern2 = @"http(s)?://([w-] .) [w-] (/[w- ./?%&=]*)?";
            Regex r2 = new Regex(pattern2, RegexOptions.IgnoreCase);
            //获得匹配结果
            MatchCollection m2 = r2.Matches(html);
            List<string> links = new List<string>();
            foreach (Match url2 in m2)
            {
                if (StringHelper.CheckUrlIsLegal(url2.ToString()) || !StringHelper.IsPureUrl(url2.ToString()) || links.Contains(url2.ToString()))
                    continue;
                links.Add(url2.ToString());
            }
            //匹配href里面的链接
            const string pattern = @"(?i)<as[^>]*?href=(['""]?)(?!javascript|__doPostBack)(?<url>[^'""s*#<>] )[^>]*>"; ;
            Regex r = new Regex(pattern, RegexOptions.IgnoreCase);
            //获得匹配结果
            MatchCollection m = r.Matches(html);
            foreach (Match url1 in m)
            {
                string href1 = url1.Groups["url"].Value;
                if (!href1.Contains("http"))
                {
                    href1 = Global.WebUrl   href1;
                }
                if (!StringHelper.IsPureUrl(href1) || links.Contains(href1)) continue;
                links.Add(href1);
            }
            return links;
        }    
 /// <summary>  
    /// 取得HTML中所有图片的 URL。  
    /// </summary>  
    /// <param name="sHtmlText">HTML代码</param>  
    /// <returns>图片的URL列表</returns> 
public static string HtmlCodeRequest(string Url)
    {
      if (string.IsNullOrEmpty(Url))
      {
        return "";
      }
      try
      {
        //创建一个请求
        HttpWebRequest httprequst = (HttpWebRequest)WebRequest.Create(Url);
        //不建立持久性链接
        httprequst.KeepAlive = true;
        //设置请求的方法
        httprequst.Method = "GET";
        //设置标头值
        httprequst.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
        httprequst.Accept = "*/*";
        httprequst.Headers.Add("Accept-Language", "zh-cn,en-us;q=0.5");
        httprequst.ServicePoint.Expect100Continue = false;
        httprequst.Timeout = 5000;
        httprequst.AllowAutoRedirect = true;//是否允许302
        ServicePointManager.DefaultConnectionLimit = 30;
        //获取响应
        HttpWebResponse webRes = (HttpWebResponse)httprequst.GetResponse();
        //获取响应的文本流
        string content = string.Empty;
        using (System.IO.Stream stream = webRes.GetResponseStream())
        {
          using (System.IO.StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding("utf-8")))
          {
            content = reader.ReadToEnd();
          }
        }
        //取消请求
        httprequst.Abort();
        //返回数据内容
        return content;
      }
      catch (Exception)
      {

        return "";
      }
    }
/// <summary>
    /// 提取页面链接
    /// </summary>
    /// <param name="html"></param>
    /// <returns></returns>
public static List<string> GetHtmlImageUrlList(string url)
    {
      string html = HttpHelper.HtmlCodeRequest(url);
      if (string.IsNullOrEmpty(html))
      {
        return new List<string>();
      }
      // 定义正则表达式用来匹配 img 标签  
      Regex regImg = new Regex(@"<imgb[^<>]*?bsrc[strn]*=[strn]*[""']?[strn]*(?<imgUrl>[^strn""'<>]*)[^<>]*?/?[strn]*>", RegexOptions.IgnoreCase);

      // 搜索匹配的字符串  
      MatchCollection matches = regImg.Matches(html);
      List<string> sUrlList = new List<string>();

      // 取得匹配项列表  
      foreach (Match match in matches)
        sUrlList.Add(match.Groups["imgUrl"].Value);
      return sUrlList;
    }


    /// <summary>
    /// 提取页面链接
    /// </summary>
    /// <param name="html"></param>
    /// <returns></returns>
    public static List<string> GetHttpLinks(string url)
    {
      //获取网址内容
      string html = HttpHelper.HtmlCodeRequest(url);
      if (string.IsNullOrEmpty(html))
      {
        return new List<string>();
      }
      //匹配http链接
      const string pattern2 = @"http(s)?://([w-] .) [w-] (/[w- ./?%&=]*)?";
      Regex r2 = new Regex(pattern2, RegexOptions.IgnoreCase);
      //获得匹配结果
      MatchCollection m2 = r2.Matches(html);
      List<string> links = new List<string>();
      foreach (Match url2 in m2)
      {
        if (StringHelper.CheckUrlIsLegal(url2.ToString()) || !StringHelper.IsPureUrl(url2.ToString()) || links.Contains(url2.ToString()))
          continue;
        links.Add(url2.ToString());
      }
      //匹配href里面的链接
      const string pattern = @"(?i)<as[^>]*?href=(['""]?)(?!javascript|__doPostBack)(?<url>[^'""s*#<>] )[^>]*>"; ;
      Regex r = new Regex(pattern, RegexOptions.IgnoreCase);
      //获得匹配结果
      MatchCollection m = r.Matches(html);
      foreach (Match url1 in m)
      {
        string href1 = url1.Groups["url"].Value;
        if (!href1.Contains("http"))
        {
          href1 = Global.WebUrl   href1;
        }
        if (!StringHelper.IsPureUrl(href1) || links.Contains(href1)) continue;
        links.Add(href1);
      }
      return links;
    }  
 /// <summary>  
    /// 取得HTML中所有图片的 URL。  
    /// </summary>  
    /// <param name="sHtmlText">HTML代码</param>  
    /// <returns>图片的URL列表</returns> 
public static string HtmlCodeRequest(string Url)
    {
      if (string.IsNullOrEmpty(Url))
      {
        return "";
      }
      try
      {
        //创建一个请求
        HttpWebRequest httprequst = (HttpWebRequest)WebRequest.Create(Url);
        //不建立持久性链接
        httprequst.KeepAlive = true;
        //设置请求的方法
        httprequst.Method = "GET";
        //设置标头值
        httprequst.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
        httprequst.Accept = "*/*";
        httprequst.Headers.Add("Accept-Language", "zh-cn,en-us;q=0.5");
        httprequst.ServicePoint.Expect100Continue = false;
        httprequst.Timeout = 5000;
        httprequst.AllowAutoRedirect = true;//是否允许302
        ServicePointManager.DefaultConnectionLimit = 30;
        //获取响应
        HttpWebResponse webRes = (HttpWebResponse)httprequst.GetResponse();
        //获取响应的文本流
        string content = string.Empty;
        using (System.IO.Stream stream = webRes.GetResponseStream())
        {
          using (System.IO.StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding("utf-8")))
          {
            content = reader.ReadToEnd();
          }
        }
        //取消请求
        httprequst.Abort();
        //返回数据内容
        return content;
      }
      catch (Exception)
      {

        return "";
      }
    }
/// <summary>
    /// 提取页面链接
    /// </summary>
    /// <param name="html"></param>
    /// <returns></returns>
public static List<string> GetHtmlImageUrlList(string url)
    {
      string html = HttpHelper.HtmlCodeRequest(url);
      if (string.IsNullOrEmpty(html))
      {
        return new List<string>();
      }
      // 定义正则表达式用来匹配 img 标签  
      Regex regImg = new Regex(@"<imgb[^<>]*?bsrc[strn]*=[strn]*[""']?[strn]*(?<imgUrl>[^strn""'<>]*)[^<>]*?/?[strn]*>", RegexOptions.IgnoreCase);

      // 搜索匹配的字符串  
      MatchCollection matches = regImg.Matches(html);
      List<string> sUrlList = new List<string>();

      // 取得匹配项列表  
      foreach (Match match in matches)
        sUrlList.Add(match.Groups["imgUrl"].Value);
      return sUrlList;
    }


    /// <summary>
    /// 提取页面链接
    /// </summary>
    /// <param name="html"></param>
    /// <returns></returns>
    public static List<string> GetHttpLinks(string url)
    {
      //获取网址内容
      string html = HttpHelper.HtmlCodeRequest(url);
      if (string.IsNullOrEmpty(html))
      {
        return new List<string>();
      }
      //匹配http链接
      const string pattern2 = @"http(s)?://([w-] .) [w-] (/[w- ./?%&=]*)?";
      Regex r2 = new Regex(pattern2, RegexOptions.IgnoreCase);
      //获得匹配结果
      MatchCollection m2 = r2.Matches(html);
      List<string> links = new List<string>();
      foreach (Match url2 in m2)
      {
        if (StringHelper.CheckUrlIsLegal(url2.ToString()) || !StringHelper.IsPureUrl(url2.ToString()) || links.Contains(url2.ToString()))
          continue;
        links.Add(url2.ToString());
      }
      //匹配href里面的链接
      const string pattern = @"(?i)<as[^>]*?href=(['""]?)(?!javascript|__doPostBack)(?<url>[^'""s*#<>] )[^>]*>"; ;
      Regex r = new Regex(pattern, RegexOptions.IgnoreCase);
      //获得匹配结果
      MatchCollection m = r.Matches(html);
      foreach (Match url1 in m)
      {
        string href1 = url1.Groups["url"].Value;
        if (!href1.Contains("http"))
        {
          href1 = Global.WebUrl   href1;
        }
        if (!StringHelper.IsPureUrl(href1) || links.Contains(href1)) continue;
        links.Add(href1);
      }
      return links;
    }  

那边下载图片有个职务条数限制,限制是200条。如若超过的电话线程等待5秒,这里下载图片是异步调用的委托

public string DownLoadimg(string url)
        {
            if (!string.IsNullOrEmpty(url))
            {
                try
                {
                    if (!url.Contains("http"))
                    {
                        url = Global.WebUrl   url;
                    }
                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
                    request.Timeout = 2000;
                    request.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
                    //是否允许302
                    request.AllowAutoRedirect = true;
                    WebResponse response = request.GetResponse();
                    Stream reader = response.GetResponseStream();
                    //文件名
                    string aFirstName = Guid.NewGuid().ToString();
                    //扩展名
                    string aLastName = url.Substring(url.LastIndexOf(".")   1, (url.Length - url.LastIndexOf(".") - 1));
                    FileStream writer = new FileStream(Global.FloderUrl   aFirstName   "."   aLastName, FileMode.OpenOrCreate, FileAccess.Write);
                    byte[] buff = new byte[512];
                    //实际读取的字节数
                    int c = 0;
                    while ((c = reader.Read(buff, 0, buff.Length)) > 0)
                    {
                        writer.Write(buff, 0, c);
                    }
                    writer.Close();
                    writer.Dispose();
                    reader.Close();
                    reader.Dispose();
                    response.Close();
                    return (aFirstName   "."   aLastName);
                }
                catch (Exception)
                {
                    return "错误:地址"   url;
                }
            }
            return "错误:地址为空";
        }

 

话非常的少说,越多的必要大家本身去革新咯!招待读者来与楼主实行沟通。若是本文对你有参谋价值,应接帮博主点下小说下方的推介,感激

有意思味可进入企鹅群一齐发展:4951045九叁

上面源码送上:嘿嘿要分的哦!

 

那边下载图片有个职务条数限制,限制是200条。即使赶上的电话线程等待5秒,这里下载图片是异步调用的寄托

澳门新萄京官方网站 5

public string DownLoadimg(string url)
    {
      if (!string.IsNullOrEmpty(url))
      {
        try
        {
          if (!url.Contains("http"))
          {
            url = Global.WebUrl   url;
          }
          HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
          request.Timeout = 2000;
          request.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
          //是否允许302
          request.AllowAutoRedirect = true;
          WebResponse response = request.GetResponse();
          Stream reader = response.GetResponseStream();
          //文件名
          string aFirstName = Guid.NewGuid().ToString();
          //扩展名
          string aLastName = url.Substring(url.LastIndexOf(".")   1, (url.Length - url.LastIndexOf(".") - 1));
          FileStream writer = new FileStream(Global.FloderUrl   aFirstName   "."   aLastName, FileMode.OpenOrCreate, FileAccess.Write);
          byte[] buff = new byte[512];
          //实际读取的字节数
          int c = 0;
          while ((c = reader.Read(buff, 0, buff.Length)) > 0)
          {
            writer.Write(buff, 0, c);
          }
          writer.Close();
          writer.Dispose();
          reader.Close();
          reader.Dispose();
          response.Close();
          return (aFirstName   "."   aLastName);
        }
        catch (Exception)
        {
          return "错误:地址"   url;
        }
      }
      return "错误:地址为空";
    }

那边下载图片有个职责条数限制,限制是200条。假诺超越的电话线程等待5秒,这里下载图片是异步调用的嘱托

话十分少说,愈来愈多的内需我们本身去改良咯!

澳门新萄京官方网站 6

 

public string DownLoadimg(string url)
    {
      if (!string.IsNullOrEmpty(url))
      {
        try
        {
          if (!url.Contains("http"))
          {
            url = Global.WebUrl   url;
          }
          HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
          request.Timeout = 2000;
          request.UserAgent = "User-Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705";
          //是否允许302
          request.AllowAutoRedirect = true;
          WebResponse response = request.GetResponse();
          Stream reader = response.GetResponseStream();
          //文件名
          string aFirstName = Guid.NewGuid().ToString();
          //扩展名
          string aLastName = url.Substring(url.LastIndexOf(".")   1, (url.Length - url.LastIndexOf(".") - 1));
          FileStream writer = new FileStream(Global.FloderUrl   aFirstName   "."   aLastName, FileMode.OpenOrCreate, FileAccess.Write);
          byte[] buff = new byte[512];
          //实际读取的字节数
          int c = 0;
          while ((c = reader.Read(buff, 0, buff.Length)) > 0)
          {
            writer.Write(buff, 0, c);
          }
          writer.Close();
          writer.Dispose();
          reader.Close();
          reader.Dispose();
          response.Close();
          return (aFirstName   "."   aLastName);
        }
        catch (Exception)
        {
          return "错误:地址"   url;
        }
      }
      return "错误:地址为空";
    }

澳门新萄京官方网站 7

话非常的少说,更加的多的供给大家温馨去改良咯!

 

未有整理与综合的学识,一钱不值!中度回顾与梳理的文化,才是团结实在的知识与技能。 永世不要让投机的自由、好奇、充满创制力的主张被具体的框架所束缚,让创制力自由成长吧! 多花时间,关怀她(她)人,正如人家所关心你的。理想的飙升与落实,未有外人的帮助与协助,是万万无法的。

本文由澳门新萄京官方网站发布于www.8455.com,转载请注明出处:网络爬虫,多线程网络爬虫

关键词: