折腾：
期间，在
pipenv install
去安装PySpider之后，进入虚拟环境运行，结果报错：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | pyspider -c config.json phantomjs fetcher running on port 25555[I 190327 16:32:30 result_worker:49] result_worker starting...[I 190327 16:32:30 processor:211] processor starting...[I 190327 16:32:31 tornado_fetcher:638] fetcher starting...[I 190327 16:32:31 scheduler:647] scheduler starting...[I 190327 16:32:31 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333[I 190327 16:32:31 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0[I 190327 16:32:31 app:84] webui exiting...Traceback (most recent call last): File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/bin/pyspider", line 10, in <module> sys.exit(main()) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 754, in main cli() File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 1114, in invoke return Command.invoke(self, ctx) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli ctx.invoke(all) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 497, in all ctx.invoke(webui, **webui_config) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui app.run(host=host, port=port) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run from .webdav import dav_app File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 216, in <module> dav_app = WsgiDAVApp(config) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 135, in __init__ _check_config(config) File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 119, in _check_config raise ValueError("Invalid configuration:\n - " + "\n - ".join(errors))ValueError: Invalid configuration: - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead. |
http_authenticator.domain_controller
结果没有wsgidav：
1 2 3 4 5 | pipenv uninstall wsgidavUninstalling wsgidav...No package wsgidav to remove from Pipfile.Locking [dev-packages] dependencies...Locking [packages] dependencies... |
pyspider Deprecated option domaincontroller use http_authenticator.domain_controller instead
“WsgiDAV released a new version — pre-release 3.x
‘dir_browser.enable’ DEPRECATED with 2.4.0
and ‘domaincontroller’ updated to ‘http_authenticator.HTTPAuthenticator’
reference：https://wsgidav.readthedocs.io/en/latest/addons-ntdc.html
You can change pyspider/webui/webdav.py line 203：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | config = DEFAULT_CONFIG.copy() config.update({ 'mount_path': '/dav', 'provider_mapping': { '/': ScriptProvider(app) }, #'domaincontroller': NeedAuthController(app), 'http_authenticator': { 'HTTPAuthenticator':NeedAuthController(app), }, 'verbose': 1 if app.debug else 0, 'dir_browser': {'davmount': False, #'enable': True, 'msmount': False, 'response_trailer': ''},})dav_app = WsgiDAVApp(config) |
“
去修改一下
不过打算直接先去安装2.4.1的wsgidav
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | pipenv reinstall wsgidav==2.4.1Usage: pipenv [OPTIONS] COMMAND [ARGS]...Try "pipenv --help" for help.Error: No such command "reinstall".Did you mean one of these? install uninstall➜ crawler_xiaohuasheng_app git:(master) ✗ pipenv install wsgidav==2.4.1 Installing wsgidav==2.4.1...Adding wsgidav to Pipfile's [packages]...<img draggable="false" role="img" class="emoji" alt=" src="https://s.w.org/images/core/emoji/14.0.0/svg/2714.svg"> Installation Succeeded Pipfile.lock not found, creating...Locking [dev-packages] dependencies...Locking [packages] dependencies... |
"结果：
就可以了：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | pyspider -c config.json Error: Could not create web server listening on port 25555[I 190327 16:40:32 result_worker:49] result_worker starting...[I 190327 16:40:33 processor:211] processor starting...[I 190327 16:40:33 tornado_fetcher:638] fetcher starting...[I 190327 16:40:33 scheduler:647] scheduler starting...[I 190327 16:40:33 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333[I 190327 16:40:33 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0[I 190327 16:40:33 app:76] webui running on 0.0.0.0:7700Error: Could not create web server listening on port 25555Error: Could not create web server listening on port 25555Error: Could not create web server listening on port 25555Error: Could not create web server listening on port 25555^C[I 190327 16:40:43 app:84] webui exiting...[I 190327 16:40:43 scheduler:663] scheduler exiting...[I 190327 16:40:43 tornado_fetcher:671] fetcher exiting...[I 190327 16:40:43 processor:229] processor exiting...[I 190327 16:40:43 result_worker:66] result_worker exiting...Aborted! |
而此处估计是之前的端口被占用了，去看看：
1 2 3 4 5 | lsof -i:25555COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEphantomjs 62688 crifan 11u IPv4 0xc72d8eb51fef7cb9 0t0 TCP *:25555 (LISTEN)➜ crawler_xiaohuasheng_app git:(master) ✗ kill -9 62688➜ crawler_xiaohuasheng_app git:(master) ✗ lsof -i:25555 |
即可正常运行了：
1 2 3 4 5 6 7 8 9 | pyspider -c config.jsonphantomjs fetcher running on port 25555[I 190327 16:42:07 result_worker:49] result_worker starting...[I 190327 16:42:07 processor:211] processor starting...[I 190327 16:42:08 tornado_fetcher:638] fetcher starting...[I 190327 16:42:08 scheduler:647] scheduler starting...[I 190327 16:42:08 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333[I 190327 16:42:08 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0[I 190327 16:42:08 app:76] webui running on 0.0.0.0:7700 |
【总结】
此处运行PySpider会出错：
1 2 3 4 | lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 119, in _check_config raise ValueError("Invalid configuration:\n - " + "\n - ".join(errors))ValueError: Invalid configuration: - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead. |
原因是：
wsgidav库升级到了最新版，导出不兼容。
解决办法：
安装旧版本：
1 | pipenv install wsgidav==2.4.1 |
或：
手动改代码为：
pyspider/webui/webdav.py 203行：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | config = DEFAULT_CONFIG.copy() config.update({ 'mount_path': '/dav', 'provider_mapping': { '/': ScriptProvider(app) }, #'domaincontroller': NeedAuthController(app), 'http_authenticator': { 'HTTPAuthenticator':NeedAuthController(app), }, 'verbose': 1 if app.debug else 0, 'dir_browser': {'davmount': False, #'enable': True, 'msmount': False, 'response_trailer': ''},})dav_app = WsgiDAVApp(config) |
转载请注明：在路上 » 【已解决】mac中PySpider运行出错：Deprecated option domaincontroller use http_authenticator.domain_controller instead