【记录】安装Scrapy

1.参考：

http://scrapy.org/download/

去cmd中通过pip安装，结果出错：

E:\Dev_Tools\python\Scrapy>pip install Scrapy
Downloading/unpacking Scrapy
  Downloading Scrapy-0.16.2.tar.gz (675kB): 675kB downloaded
  Running setup.py egg_info for package Scrapy

    warning: no files found matching 'license.txt' under directory 'scrapy'
    no previously-included directories found matching 'docs\build'
Downloading/unpacking Twisted>=8.0 (from Scrapy)
  Downloading Twisted-12.2.0.tar.bz2 (2.9MB): 2.9MB downloaded
  Running setup.py egg_info for package Twisted

Downloading/unpacking w3lib>=1.2 (from Scrapy)
  Downloading w3lib-1.2.tar.gz
  Running setup.py egg_info for package w3lib

Downloading/unpacking lxml (from Scrapy)
  Downloading lxml-3.0.1.tar.gz (3.2MB): 3.2MB downloaded
  Running setup.py egg_info for package lxml
    Building lxml version 3.0.1.
    Building without Cython.
    ERROR: 'xslt-config' 不是内部或外部命令，也不是可运行的程序
    或批处理文件。

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt

    warning: no files found matching 'lxml.etree.c' under directory 'src\lxml'
    warning: no files found matching 'lxml.objectify.c' under directory 'src\lxml'
    warning: no files found matching 'lxml.etree.h' under directory 'src\lxml'
    warning: no files found matching 'lxml.etree_api.h' under directory 'src\lxml'
    warning: no files found matching 'etree_defs.h' under directory 'src\lxml'
    warning: no files found matching '*.txt' under directory 'src\lxml\tests'
    warning: no files found matching 'pubkey.asc' under directory 'doc'
    warning: no files found matching 'tagpython*.png' under directory 'doc'
    warning: no files found matching 'Makefile' under directory 'doc'
Downloading/unpacking pyOpenSSL (from Scrapy)
  Downloading pyOpenSSL-0.13.tar.gz (250kB): 250kB downloaded
  Running setup.py egg_info for package pyOpenSSL

    error: Could not find 'openssl.exe'
    Complete output from command python setup.py egg_info:
    running egg_info

creating pip-egg-info\pyOpenSSL.egg-info

writing pip-egg-info\pyOpenSSL.egg-info\PKG-INFO

writing top-level names to pip-egg-info\pyOpenSSL.egg-info\top_level.txt

writing dependency_links to pip-egg-info\pyOpenSSL.egg-info\dependency_links.txt

writing manifest file 'pip-egg-info\pyOpenSSL.egg-info\SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



error: Could not find 'openssl.exe'

----------------------------------------
Command python setup.py egg_info failed with error code 1 in c:\users\admini~1\appdata\local\temp\pip-build\pyOpenSSL
Storing complete log in C:\Users\Administrator\pip\pip.log

2。【已解决】安装Scrapy过程中出错：ERROR: ‘xslt-config’ 不是内部或外部命令，也不是可运行的程序 或批处理文件。

3.【已解决】安装Scrapy过程中出错：error: Could not find ‘openssl.exe’

4. 解决了openssl的问题，结果又出现

error: Unable to find vcvarsall.bat

的问题：

【已解决】安装Scrapy过程中出错：error: Unable to find vcvarsall.bat

5.最后终于安装成功：

E:\Dev_Tools\python\Scrapy>pip install Scrapy
Requirement already satisfied (use --upgrade to upgrade): Scrapy in e:\dev_install_root\python27\lib\site-packages
Requirement already satisfied (use --upgrade to upgrade): Twisted>=8.0 in e:\dev_install_root\python27\lib\site-packages
 (from Scrapy)
Downloading/unpacking w3lib>=1.2 (from Scrapy)
  Running setup.py egg_info for package w3lib

Requirement already satisfied (use --upgrade to upgrade): lxml in e:\dev_install_root\python27\lib\site-packages (from S
crapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in e:\dev_install_root\python27\lib\site-packages (f
rom Scrapy)
Downloading/unpacking zope.interface (from Twisted>=8.0->Scrapy)
  Running setup.py egg_info for package zope.interface

Requirement already satisfied (use --upgrade to upgrade): distribute in e:\dev_install_root\python27\lib\site-packages\d
istribute-0.6.28-py2.7.egg (from zope.interface->Twisted>=8.0->Scrapy)
Installing collected packages: w3lib, zope.interface
  Running setup.py install for w3lib

  Running setup.py install for zope.interface

    building 'zope.interface._zope_interface_coptimizations' extension
    ********************************************************************************
    WARNING:

            An optional code optimization (C extension) could not be compiled.

            Optimizations for this package will not be available!
    ()
    Unable to find vcvarsall.bat
    ********************************************************************************
    Skipping installation of E:\dev_install_root\Python27\Lib\site-packages\zope\__init__.py (namespace package)
    Installing E:\dev_install_root\Python27\Lib\site-packages\zope.interface-4.0.1-py2.7-nspkg.pth
Successfully installed w3lib zope.interface
Cleaning up...

6.然后再去参考：

Steps to install Scrapy

去测试安装是否成功：

Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import zope.interface
>>> import twisted
>>> import lxml
>>> import OpenSSL

Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    import OpenSSL
  File "E:\dev_install_root\Python27\lib\site-packages\OpenSSL\__init__.py", line 13, in <module>
    from OpenSSL import crypto
ImportError: DLL load failed: %1 不是有效的 Win32 应用程序。
>>> import openssl

Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    import openssl
ImportError: No module named openssl
>>> import scrapy
>>> print twisted.version
[twisted, version 10.2.0]
>>> print zope.version

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    print zope.version
AttributeError: 'module' object has no attribute 'version'
>>> print lxml.version

Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    print lxml.version
AttributeError: 'module' object has no attribute 'version'
>>> print OpenSSL.version

Traceback (most recent call last):
  File "<pyshell#9>", line 1, in <module>
    print OpenSSL.version
NameError: name 'OpenSSL' is not defined
>>> print scrapy.version

Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    print scrapy.version
AttributeError: 'module' object has no attribute 'version'
>>> 

很明显，此处貌似OpenSSL好像没哟安装成功啊。

但是，前面的显示是：

Requirement already satisfied (use –upgrade to upgrade): pyOpenSSL in e:\dev_install_root\python27\lib\site-packages (from Scrapy)

说明是安装成功的才对啊。

先不管了，等以后遇到问题再说。

7.不过，后来参考了官网的安装教程：

http://doc.scrapy.org/en/latest/intro/install.html#intro-install

->

http://doc.scrapy.org/en/latest/intro/install.html#intro-install-platform-notes

去

Win32 OpenSSL page

下载

Visual C++ 2008 Redistributables (x64)

得到vcredist_x64.exe并安装。

再下载Win64 OpenSSL v1.0.0j得到Win64OpenSSL-1_0_0j.exe并安装。

期间竟然提示找不到VC++ 2008：

不过不管，继续安装至完毕。

然后把

E:\dev_install_root\OpenSSL-Win64\bin

添加到环境变量PATH中。

【总结】

想要安装Scrapy，但是其依赖一堆的其他库，而本身pip或者Easy_install时，又不难很好的自动解决这些依赖的库的问题（比如twisted，是twisted本身的问题），导致只是安装Scrapy，都这么麻烦。无语了。

转载请注明：在路上 » 【记录】安装Scrapy