折腾：
期间，在
pipenv install
去安装PySpider之后，进入虚拟环境运行，结果报错：
pyspider -c config.json
phantomjs fetcher running on port 25555
[I 190327 16:32:30 result_worker:49] result_worker starting...
[I 190327 16:32:30 processor:211] processor starting...
[I 190327 16:32:31 tornado_fetcher:638] fetcher starting...
[I 190327 16:32:31 scheduler:647] scheduler starting...
[I 190327 16:32:31 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 190327 16:32:31 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 190327 16:32:31 app:84] webui exiting...
Traceback (most recent call last):
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/bin/pyspider", line 10, in <module>
sys.exit(main())
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 754, in main
cli()
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 1114, in invoke
return Command.invoke(self, ctx)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli
ctx.invoke(all)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 497, in all
ctx.invoke(webui, **webui_config)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui
app.run(host=host, port=port)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run
from .webdav import dav_app
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 216, in <module>
dav_app = WsgiDAVApp(config)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 135, in __init__
_check_config(config)
File "/Users/crifan/.local/share/virtualenvs/crawler_xiaohuasheng_app-zY4wnqo9/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 119, in _check_config
raise ValueError("Invalid configuration:\n - " + "\n - ".join(errors))
ValueError: Invalid configuration:
- Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.
http_authenticator.domain_controller
结果没有wsgidav：
pipenv uninstall wsgidav Uninstalling wsgidav... No package wsgidav to remove from Pipfile. Locking [dev-packages] dependencies... Locking [packages] dependencies...
pyspider Deprecated option domaincontroller use http_authenticator.domain_controller instead
“WsgiDAV released a new version — pre-release 3.x
‘dir_browser.enable’ DEPRECATED with 2.4.0
and ‘domaincontroller’ updated to ‘http_authenticator.HTTPAuthenticator’
reference：https://wsgidav.readthedocs.io/en/latest/addons-ntdc.html
You can change pyspider/webui/webdav.py line 203：
config = DEFAULT_CONFIG.copy()
config.update({
'mount_path': '/dav',
'provider_mapping': {
'/': ScriptProvider(app)
},
#'domaincontroller': NeedAuthController(app),
'http_authenticator': {
'HTTPAuthenticator':NeedAuthController(app),
},
'verbose': 1 if app.debug else 0,
'dir_browser': {'davmount': False,
#'enable': True,
'msmount': False,
'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)“
去修改一下
不过打算直接先去安装2.4.1的wsgidav
pipenv reinstall wsgidav==2.4.1 Usage: pipenv [OPTIONS] COMMAND [ARGS]... Try "pipenv --help" for help. Error: No such command "reinstall". Did you mean one of these? install uninstall ➜ crawler_xiaohuasheng_app git:(master) ✗ pipenv install wsgidav==2.4.1 Installing wsgidav==2.4.1... Adding wsgidav to Pipfile's [packages]... ✔ Installation Succeeded Pipfile.lock not found, creating... Locking [dev-packages] dependencies... Locking [packages] dependencies...
结果：
就可以了：
pyspider -c config.json Error: Could not create web server listening on port 25555 [I 190327 16:40:32 result_worker:49] result_worker starting... [I 190327 16:40:33 processor:211] processor starting... [I 190327 16:40:33 tornado_fetcher:638] fetcher starting... [I 190327 16:40:33 scheduler:647] scheduler starting... [I 190327 16:40:33 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333 [I 190327 16:40:33 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0 [I 190327 16:40:33 app:76] webui running on 0.0.0.0:7700 Error: Could not create web server listening on port 25555 Error: Could not create web server listening on port 25555 Error: Could not create web server listening on port 25555 Error: Could not create web server listening on port 25555 ^C[I 190327 16:40:43 app:84] webui exiting... [I 190327 16:40:43 scheduler:663] scheduler exiting... [I 190327 16:40:43 tornado_fetcher:671] fetcher exiting... [I 190327 16:40:43 processor:229] processor exiting... [I 190327 16:40:43 result_worker:66] result_worker exiting... Aborted!
而此处估计是之前的端口被占用了，去看看：
lsof -i:25555 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME phantomjs 62688 crifan 11u IPv4 0xc72d8eb51fef7cb9 0t0 TCP *:25555 (LISTEN) ➜ crawler_xiaohuasheng_app git:(master) ✗ kill -9 62688 ➜ crawler_xiaohuasheng_app git:(master) ✗ lsof -i:25555
即可正常运行了：
pyspider -c config.json phantomjs fetcher running on port 25555 [I 190327 16:42:07 result_worker:49] result_worker starting... [I 190327 16:42:07 processor:211] processor starting... [I 190327 16:42:08 tornado_fetcher:638] fetcher starting... [I 190327 16:42:08 scheduler:647] scheduler starting... [I 190327 16:42:08 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333 [I 190327 16:42:08 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0 [I 190327 16:42:08 app:76] webui running on 0.0.0.0:7700
【总结】
此处运行PySpider会出错：
lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 119, in _check_config
raise ValueError("Invalid configuration:\n - " + "\n - ".join(errors))
ValueError: Invalid configuration:
- Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.原因是：
wsgidav库升级到了最新版，导出不兼容。
解决办法：
安装旧版本：
pipenv install wsgidav==2.4.1
或：
手动改代码为：
pyspider/webui/webdav.py 203行：
config = DEFAULT_CONFIG.copy()
config.update({
'mount_path': '/dav',
'provider_mapping': {
'/': ScriptProvider(app)
},
#'domaincontroller': NeedAuthController(app),
'http_authenticator': {
'HTTPAuthenticator':NeedAuthController(app),
},
'verbose': 1 if app.debug else 0,
'dir_browser': {'davmount': False,
#'enable': True,
'msmount': False,
'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)转载请注明：在路上 » 【已解决】mac中PySpider运行出错：Deprecated option domaincontroller use http_authenticator.domain_controller instead