【记录】用Python的Scrapy去爬取Youtube中Humf的字幕

对于之前的手动操作去找小毛怪 Humf 的字幕：

但是找到个好（工具）网站，支持提取youtube中视频的字幕的，具体步骤：

针对于Humf的youtube官网的每个系列：

Humf – Official Channel – YouTube – YouTube

点击进去后：

https://www.youtube.com/watch?v=eDaHu2vnvlM&list=PLHOR8x-IicVIHXV3Dkro99d4KyBAuVx9_

即对于每个系列中的每个视频：

的页面地址，粘贴到：

http://www.yousubtitles.com

中，然后点击Download后，即可看到有很多语言的srt字幕，点击English的下载即可得到srt文件。

其余的100多个视频的字幕：

要么我抽空写爬虫脚本自动化去下载

现在希望用scrapy去实现

之前已经折腾了一点scrapy了：

【记录】用Python的Scrapy去爬取cbeebies.com

现在去试试上面这个爬取。

所以入口是：

Humf – Official Channel – YouTube – YouTube

去创建项目：

➜ scrapy scrapy startproject youtubeSubtitle

New Scrapy project ‘youtubeSubtitle’, using template directory ‘/usr/local/lib/python2.7/site-packages/scrapy/templates/project’, created in:

/Users/crifan/dev/dev_root/company/naturling/projects/scrapy/youtubeSubtitle

You can start your first spider with:

cd youtubeSubtitle

scrapy genspider example example.com

➜ scrapy cd youtubeSubtitle

然后看到提示，所以去看看命令：

➜ youtubeSubtitle scrapy –help

Scrapy 1.4.0 – project: youtubeSubtitle

Usage:

scrapy <command> [options] [args]

Available commands:

bench Run quick benchmark test

check Check spider contracts

crawl Run a spider

edit Edit spider

fetch Fetch a URL using the Scrapy downloader

genspider Generate new spider using pre-defined templates

list List available spiders

parse Parse URL (using its spider) and print the results

runspider Run a self-contained spider (without creating a project)

settings Get settings values

shell Interactive scraping console

startproject Create new project

version Print Scrapy version

view Open URL in browser, as seen by Scrapy

Use “scrapy <command> -h” to see more info about a command

果然genspider是用来生成默认的爬虫的，然后就去创建：

➜ youtubeSubtitle scrapy genspider YoutubeSubtitle youtube.com

Created spider ‘YoutubeSubtitle’ using template ‘basic’ in module:

youtubeSubtitle.spiders.YoutubeSubtitle

自动生成的内容是：

# -*- coding: utf-8 -*-

import scrapy

class YoutubesubtitleSpider(scrapy.Spider):

name = ‘YoutubeSubtitle’

allowed_domains = [‘youtube.com’]

start_urls = [‘http://youtube.com/’]

def parse(self, response):

pass

然后继续添加内容：

# -*- coding: utf-8 -*-

import scrapy

class YoutubesubtitleSpider(scrapy.Spider):

name = ‘YoutubeSubtitle’

allowed_domains = [‘youtube.com’, “yousubtitles.com”]

start_urls = [

“https://www.youtube.com/user/theofficialhumf/playlists”

]

def parse(self, response):

respUrl = response.url

print “respUrl=%s”%(respUrl)

filename = respUrl.split(“/”)[-2] + ‘.html’

with open(filename, ‘wb’) as f:

f.write(response.body)

然后去执行：

➜ youtubeSubtitle scrapy crawl YoutubeSubtitle

2018-01-13 10:49:45 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: youtubeSubtitle)

2018-01-13 10:49:45 [scrapy.utils.log] INFO: Overridden settings: {‘NEWSPIDER_MODULE’: ‘youtubeSubtitle.spiders’, ‘SPIDER_MODULES’: [‘youtubeSubtitle.spiders’], ‘ROBOTSTXT_OBEY’: True, ‘BOT_NAME’: ‘youtubeSubtitle’}

2018-01-13 10:49:45 [scrapy.middleware] INFO: Enabled extensions:

[‘scrapy.extensions.memusage.MemoryUsage’,

‘scrapy.extensions.logstats.LogStats’,

‘scrapy.extensions.telnet.TelnetConsole’,

‘scrapy.extensions.corestats.CoreStats’]

2018-01-13 10:49:45 [scrapy.middleware] INFO: Enabled downloader middlewares:

[‘scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware’,

‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware’,

‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware’,

‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware’,

‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware’,

‘scrapy.downloadermiddlewares.retry.RetryMiddleware’,

‘scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware’,

‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware’,

‘scrapy.downloadermiddlewares.redirect.RedirectMiddleware’,

‘scrapy.downloadermiddlewares.cookies.CookiesMiddleware’,

‘scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware’,

‘scrapy.downloadermiddlewares.stats.DownloaderStats’]

2018-01-13 10:49:45 [scrapy.middleware] INFO: Enabled spider middlewares:

[‘scrapy.spidermiddlewares.httperror.HttpErrorMiddleware’,

‘scrapy.spidermiddlewares.offsite.OffsiteMiddleware’,

‘scrapy.spidermiddlewares.referer.RefererMiddleware’,

‘scrapy.spidermiddlewares.urllength.UrlLengthMiddleware’,

‘scrapy.spidermiddlewares.depth.DepthMiddleware’]

2018-01-13 10:49:45 [scrapy.middleware] INFO: Enabled item pipelines:

[]

2018-01-13 10:49:45 [scrapy.core.engine] INFO: Spider opened

2018-01-13 10:49:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:49:45 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

好像卡死了

难道是因为还没翻墙导致的？

果然超时了：

2018-01-13 10:49:45 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

2018-01-13 10:50:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:51:00 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.youtube.com/robots.txt> (failed 1 times): TCP connection timed out: 60: Operation timed out.

2018-01-13 10:51:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:52:16 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.youtube.com/robots.txt> (failed 2 times): TCP connection timed out: 60: Operation timed out.

2018-01-13 10:52:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:53:31 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.youtube.com/robots.txt> (failed 3 times): TCP connection timed out: 60: Operation timed out.

2018-01-13 10:53:31 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET https://www.youtube.com/robots.txt>: TCP connection timed out: 60: Operation timed out.

TCPTimedOutError: TCP connection timed out: 60: Operation timed out.

2018-01-13 10:53:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:54:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:54:48 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.youtube.com/user/theofficialhumf/playlists> (failed 1 times): TCP connection timed out: 60: Operation timed out.

2018-01-13 10:55:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:56:04 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.youtube.com/user/theofficialhumf/playlists> (failed 2 times): TCP connection timed out: 60: Operation timed out.

2018-01-13 10:56:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-01-13 10:57:20 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.youtube.com/user/theofficialhumf/playlists> (failed 3 times): TCP connection timed out: 60: Operation timed out.

2018-01-13 10:57:20 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.youtube.com/user/theofficialhumf/playlists>: TCP connection timed out: 60: Operation timed out.

2018-01-13 10:57:20 [scrapy.core.engine] INFO: Closing spider (finished)

2018-01-13 10:57:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

{‘downloader/exception_count’: 6,

‘downloader/exception_type_count/twisted.internet.error.TCPTimedOutError’: 6,

‘downloader/request_bytes’: 1398,

‘downloader/request_count’: 6,

‘downloader/request_method_count/GET’: 6,

‘finish_reason’: ‘finished’,

‘finish_time’: datetime.datetime(2018, 1, 13, 2, 57, 20, 551337),

‘log_count/DEBUG’: 7,

‘log_count/ERROR’: 2,

‘log_count/INFO’: 14,

‘memusage/max’: 50995200,

‘memusage/startup’: 50167808,

‘retry/count’: 4,

‘retry/max_reached’: 2,

‘retry/reason_count/twisted.internet.error.TCPTimedOutError’: 4,

‘scheduler/dequeued’: 3,

‘scheduler/dequeued/memory’: 3,

‘scheduler/enqueued’: 3,

‘scheduler/enqueued/memory’: 3,

‘start_time’: datetime.datetime(2018, 1, 13, 2, 49, 45, 656662)}

2018-01-13 10:57:20 [scrapy.core.engine] INFO: Spider closed (finished)

所以要去：

【已解决】Scrapy如何添加本地socks代理以便能打开Youtube网页

然后从网页中提取出多个系列的地址

保存出来的页面，希望提取的是：

‘//div[@class=“yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’

去参考：

【记录】尝试Scrapy shell去提取cbeebies.com页面中的子url

中的scrapy shell去试试xpath能不能找到我们要的内容

先去：

➜ youtubeSubtitle scrapy shell

2018-01-13 21:05:57 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: youtubeSubtitle)

2018-01-13 21:05:57 [scrapy.utils.log] INFO: Overridden settings: {‘NEWSPIDER_MODULE’: ‘youtubeSubtitle.spiders’, ‘ROBOTSTXT_OBEY’: True, ‘DUPEFILTER_CLASS’: ‘scrapy.dupefilters.BaseDupeFilter’, ‘SPIDER_MODULES’: [‘youtubeSubtitle.spiders’], ‘BOT_NAME’: ‘youtubeSubtitle’, ‘LOGSTATS_INTERVAL’: 0}

2018-01-13 21:05:57 [scrapy.middleware] INFO: Enabled extensions:

[‘scrapy.extensions.memusage.MemoryUsage’,

‘scrapy.extensions.telnet.TelnetConsole’,

‘scrapy.extensions.corestats.CoreStats’]

2018-01-13 21:05:57 [scrapy.middleware] INFO: Enabled downloader middlewares:

[‘youtubeSubtitle.middlewares.ProxyMiddleware’,

‘scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware’,

‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware’,

‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware’,

‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware’,

‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware’,

‘scrapy.downloadermiddlewares.retry.RetryMiddleware’,

‘scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware’,

‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware’,

‘scrapy.downloadermiddlewares.redirect.RedirectMiddleware’,

‘scrapy.downloadermiddlewares.cookies.CookiesMiddleware’,

‘scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware’,

‘scrapy.downloadermiddlewares.stats.DownloaderStats’]

2018-01-13 21:05:57 [scrapy.middleware] INFO: Enabled spider middlewares:

[‘scrapy.spidermiddlewares.httperror.HttpErrorMiddleware’,

‘scrapy.spidermiddlewares.offsite.OffsiteMiddleware’,

‘scrapy.spidermiddlewares.referer.RefererMiddleware’,

‘scrapy.spidermiddlewares.urllength.UrlLengthMiddleware’,

‘scrapy.spidermiddlewares.depth.DepthMiddleware’]

2018-01-13 21:05:57 [scrapy.middleware] INFO: Enabled item pipelines:

[]

2018-01-13 21:05:57 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

[s] Available Scrapy objects:

[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)

[s] crawler <scrapy.crawler.Crawler object at 0x111cd2b90>

[s] item {}

[s] settings <scrapy.settings.Settings object at 0x111cd2b10>

[s] Useful shortcuts:

[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)

[s] fetch(req) Fetch a scrapy.Request and update local objects

[s] shelp() Shell help (print this help)

[s] view(response) View response in a browser

研究了下，最后是：

>>> fetch(“https://www.youtube.com/user/theofficialhumf/playlists“)

2018-01-13 21:07:09 [scrapy.core.engine] INFO: Spider opened

2018-01-13 21:07:09 [default] INFO: YoutubesubtitleSpiderMiddleware process_request: request=<GET https://www.youtube.com/user/theofficialhumf/playlists>, spider=<DefaultSpider ‘default’ at 0x111f751d0>

2018-01-13 21:07:09 [default] INFO: request.meta{‘handle_httpstatus_list’: <scrapy.utils.datatypes.SequenceExclude object at 0x111f75090>, ‘proxy’: ‘http://127.0.0.1:1087’}

2018-01-13 21:07:09 [default] INFO: YoutubesubtitleSpiderMiddleware process_request: request=<GET https://www.youtube.com/robots.txt>, spider=<DefaultSpider ‘default’ at 0x111f751d0>

2018-01-13 21:07:09 [default] INFO: request.meta{‘dont_obey_robotstxt’: True, ‘proxy’: ‘http://127.0.0.1:1087’}

2018-01-13 21:07:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.youtube.com/robots.txt> (referer: None)

2018-01-13 21:07:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.youtube.com/user/theofficialhumf/playlists> (referer: None)

fetch(“your_url”)

得到了结果

无意间用：

>>> view(response)

True

调用了浏览器打开结果：

然后继续response.body可以得到结果

这样就可以接续试试xpath了

但是上面view出来的网页中，想要的地址是：

https://www.youtube.com/watch?v=eDaHu2vnvlM&list=PLHOR8x-IicVIHXV3Dkro99d4KyBAuVx9_

但是此处的：

>>> response.xpath(‘//div[@class=”yt-lockup-thumbnail”]’)

[<Selector xpath=’//div[@class=”yt-lockup-thumbnail”]’ data=u'<div class=”yt-lockup-thumbnail”>\n ‘>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]’ data=u'<div class=”yt-lockup-thumbnail”>\n ‘>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]’ data=u'<div class=”yt-lockup-thumbnail”>\n ‘>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]’ data=u'<div class=”yt-lockup-thumbnail”>\n ‘>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]’ data=u'<div class=”yt-lockup-thumbnail”>\n ‘>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]’ data=u'<div class=”yt-lockup-thumbnail”>\n ‘>]

>>> response.xpath(‘//div[@class=”yt-lockup-thumbnail”]/a’)

[<Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a’ data=u'<a href=”/watch?v=23t1f8d2ISs&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a’ data=u'<a href=”/watch?v=eDaHu2vnvlM&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a’ data=u'<a href=”/watch?v=Ensz3fh8608&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a’ data=u'<a href=”/watch?v=I7xxF6iTCTc&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a’ data=u'<a href=”/watch?v=gpwWkfIyOao&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a’ data=u'<a href=”/watch?v=gpwWkfIyOao&list=P’>]

>>> response.xpath(‘//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’)

[<Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’ data=u'<a href=”/watch?v=23t1f8d2ISs&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’ data=u'<a href=”/watch?v=eDaHu2vnvlM&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’ data=u'<a href=”/watch?v=Ensz3fh8608&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’ data=u'<a href=”/watch?v=I7xxF6iTCTc&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’ data=u'<a href=”/watch?v=gpwWkfIyOao&list=P’>, <Selector xpath=’//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’ data=u'<a href=”/watch?v=gpwWkfIyOao&list=P’>]

地址貌似只是：

/watch?v=23t1f8d2ISs&list=P

后面地址不全的感觉

或许是此处调试期间的后续数据没有完全显示？

>>> response.xpath(‘//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’)[0]

继续调试看看

试了试a属性，结果没有：

>>> response.xpath(‘//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’)[0].a

Traceback (most recent call last):

File “<console>”, line 1, in <module>

AttributeError: ‘Selector’ object has no attribute ‘a’

无意间试了试extract，结果可以看到希望的全部内容了：

>>> response.xpath(‘//div[@class=”yt-lockup-thumbnail”]/a[starts-with(@href, “/watch”)]’)[0].extract()

u'<a href=”/watch?v=23t1f8d2ISs&list=PLHOR8x-IicVJDAmJWZmJ-IMu1x3lTAld5″ class=”yt-pl-thumb-link yt-uix-sessionlink spf-link ” data-sessionlink=”ei=eQRaWs-OIsK9qQHosZ3IBA&feature=c4-videos&ved=CDcQlx4iEwiPr5_2gNXYAhXCXioKHehYB0komxw”>\n <span class=” yt-pl-thumb”>\n \n <span class=”video-thumb yt-thumb yt-thumb-196″>\n <span class=”yt-thumb-default”>\n <span class=”yt-thumb-clip”>\n \n <img width=”196″ data-ytimg=”1″ src=”https://i.ytimg.com/vi/23t1f8d2ISs/hqdefault.jpg?sqp=-oaymwEWCMQBEG5IWvKriqkDCQgBFQAAiEIYAQ==&;rs=AOn4CLAlYOJF6LvCxIQsdzkOtRxyA3l1pw” onload=”;window.__ytRIL && __ytRIL(this)” alt=”” aria-hidden=”true”>\n\n <span class=”vertical-align”></span>\n </span>\n </span>\n </span>\n\n\n <span class=”sidebar”>\n <span class=”yt-pl-sidebar-content yt-valign”>\n <span class=”yt-valign-container”>\n <span class=”formatted-video-count-label”>\n <b>66</b> videos\n </span>\n\n <span class=”yt-pl-icon yt-pl-icon-reg yt-sprite”></span>\n </span>\n </span>\n </span>\n\n <span class=”yt-pl-thumb-overlay”>\n <span class=”yt-pl-thumb-overlay-content”>\n <span class=”play-icon yt-sprite”></span>\n <span class=”yt-pl-thumb-overlay-text”>\nPlay all\n </span>\n </span>\n </span>\n\n <span class=”yt-pl-watch-queue-overlay”>\n <span class=”thumb-menu dark-overflow-action-menu video-actions”>\n <button aria-expanded=”false” aria-haspopup=”true” onclick=”;return false;” class=”yt-uix-button-reverse flip addto-watch-queue-menu spf-nolink hide-until-delayloaded yt-uix-button yt-uix-button-dark-overflow-action-menu yt-uix-button-size-default yt-uix-button-has-icon no-icon-markup yt-uix-button-empty” type=”button”><span class=”yt-uix-button-arrow yt-sprite”></span><ul class=”watch-queue-thumb-menu yt-uix-button-menu yt-uix-button-menu-dark-overflow-action-menu hid”><li role=”menuitem” class=”overflow-menu-choice addto-watch-queue-menu-choice addto-watch-queue-play-now yt-uix-button-menu-item” data-action=”play-now” onclick=”;return false;” data-list-id=”PLHOR8x-IicVJDAmJWZmJ-IMu1x3lTAld5″><span class=”addto-watch-queue-menu-text”>Play now</span></li></ul></button>\n </span>\n\n <button class=”yt-uix-button yt-uix-button-size-small yt-uix-button-default yt-uix-button-empty yt-uix-button-has-icon no-icon-markup addto-button addto-queue-button video-actions spf-nolink hide-until-delayloaded addto-tv-queue-button yt-uix-tooltip” type=”button” onclick=”;return false;” title=”Queue” data-style=”tv-queue” data-list-id=”PLHOR8x-IicVJDAmJWZmJ-IMu1x3lTAld5″></button>\n\n </span>\n\n </span>\n\n </a>’

其中的内容，就是之前从正常的网页中拷贝出来的html：