折腾：
期间，已经本地调试完毕了，现在需要上传并部署到线上环境：
其中：
去 删除之前的虚拟环境，停止之前的服务：
<code>(robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# pipenv --rm Removing virtualenv (/root/.local/share/virtualenvs/robotDemo-dwdcgdaG)… (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# ll /root/.local/share/virtualenvs/ total 12 drwxr-xr-x 5 root root 4096 Jul 17 17:16 BatchImportScript-vVMBoBec drwxr-xr-x 5 root root 4096 Jun 6 17:45 debug_nlp-wDGPvi3o drwxr-xr-x 5 root root 4096 Jun 7 13:53 xxx-VJ297lRu (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# ll /root/.virtualenvs/ total 12 drwxr-xr-x 5 root root 4096 Jul 17 17:16 BatchImportScript-vVMBoBec drwxr-xr-x 5 root root 4096 Jun 6 17:45 debug_nlp-wDGPvi3o drwxr-xr-x 5 root root 4096 Jun 7 13:53 xxx-VJ297lRu (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# pwd /root/xxx/server/robotDemo (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# ll total 104 drwxr-xr-x 5 root root 4096 May 10 14:57 ai -rw-r--r-- 1 root root 41549 Jul 25 11:12 app.py -rw-r--r-- 1 root root 6 Jun 25 11:04 celerybeat.pid -rw-r--r-- 1 root root 2252 Jul 3 17:13 config.py -rw-r--r-- 1 root root 2005 Apr 24 16:36 gunicorn_config.py drwxr-xr-x 2 root root 4096 Aug 25 06:11 logs -rw-r--r-- 1 root root 303 May 11 17:22 Pipfile -rw-r--r-- 1 root root 12454 May 11 17:24 Pipfile.lock drwxr-xr-x 2 root root 4096 Jul 25 11:19 __pycache__ -rw-r--r-- 1 root root 2102 May 2 16:10 README.md drwxr-xr-x 2 root root 4096 Aug 27 17:14 runtime -rw-r--r-- 1 root root 2447 May 25 17:34 supervisord_server.conf drwxr-xr-x 3 root root 4096 May 14 16:17 tmp (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# pwd /root/xxx/server/robotDemo (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# supervisorctl status gunicorn RUNNING pid 16426, uptime 16 days, 3:57:45 redis RUNNING pid 21685, uptime 63 days, 6:20:12 robotDemo RUNNING pid 19658, uptime 33 days, 6:05:03 robotDemo_CeleryBeat RUNNING pid 21684, uptime 63 days, 6:20:12 robotDemo_CeleryWorker RUNNING pid 21687, uptime 63 days, 6:20:12 (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# supervisorctl stop robotDemo robotDemo_CeleryBeat robotDemo_CeleryWorker redis robotDemo: stopped robotDemo_CeleryBeat: stopped robotDemo_CeleryWorker: stopped redis: stopped (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# supervisorctl status gunicorn RUNNING pid 16426, uptime 16 days, 3:58:37 redis STOPPED Aug 27 05:24 PM robotDemo STOPPED Aug 27 05:24 PM robotDemo_CeleryBeat STOPPED Aug 27 05:24 PM robotDemo_CeleryWorker STOPPED Aug 27 05:24 PM (robotDemo-dwdcgdaG) [root@xxx-general-01 robotDemo]# </code>
然后把之前代码改名留后续删除：
<code>(robotDemo-dwdcgdaG) [root@xxx-general-01 server]# ll -lha total 20K drwxr-xr-x 5 root root 4.0K Aug 11 13:26 . drwxr-xr-x 5 root root 4.0K Jul 17 17:14 .. drwxr-xr-x 2 root root 4.0K Jul 25 09:53 for_bacup_mysql drwxr-xr-x 8 root root 4.0K Jul 31 11:36 xxxCmsServer srwxrwxrwx 1 root root 0 Aug 11 13:26 xxxCmsServer.sock drwxr-xr-x 7 root root 4.0K Aug 27 17:24 robotDemo (robotDemo-dwdcgdaG) [root@xxx-general-01 server]# mv robotDemo/ robotDemo_toDelete (robotDemo-dwdcgdaG) [root@xxx-general-01 server]# ll -lha total 20K drwxr-xr-x 5 root root 4.0K Aug 27 17:57 . drwxr-xr-x 5 root root 4.0K Jul 17 17:14 .. drwxr-xr-x 2 root root 4.0K Jul 25 09:53 for_bacup_mysql drwxr-xr-x 8 root root 4.0K Jul 31 11:36 xxxCmsServer srwxrwxrwx 1 root root 0 Aug 11 13:26 xxxCmsServer.sock drwxr-xr-x 7 root root 4.0K Aug 27 17:24 robotDemo_toDelete (robotDemo-dwdcgdaG) [root@xxx-general-01 server]# mkdir robotDemo (robotDemo-dwdcgdaG) [root@xxx-general-01 server]# ll -lha total 24K drwxr-xr-x 6 root root 4.0K Aug 27 17:58 . drwxr-xr-x 5 root root 4.0K Jul 17 17:14 .. drwxr-xr-x 2 root root 4.0K Jul 25 09:53 for_bacup_mysql drwxr-xr-x 8 root root 4.0K Jul 31 11:36 xxxCmsServer srwxrwxrwx 1 root root 0 Aug 11 13:26 xxxCmsServer.sock drwxr-xr-x 2 root root 4.0K Aug 27 17:58 robotDemo drwxr-xr-x 7 root root 4.0K Aug 27 17:24 robotDemo_toDelete (robotDemo-dwdcgdaG) [root@xxx-general-01 server]# pwd /root/xxx/web/server </code>
然后重新用Fabric：
/Users/crifan/dev/dev_root/xxx/deploy/fabfile.py
<code>from invoke import task
from fabric import Connection
from patchwork.transfers import rsync
import os
RemoteHost = '47.x.x.x'
RemoteUser = 'xxx'
RemotePathRoot = '/root/xxx/robotDemo'
def seperatorLine(seperatorChar='-', count=80):
print(seperatorChar * count)
def fabLoadedPath():
from fabric.main import program
return program.collection.loaded_from
@task
def upload(context):
print("upload: context=%s", context)
seperatorLine()
fabFilePath = fabLoadedPath()
print("fabFilePath=%s" % fabFilePath)
localProjectRootPath = os.path.join(fabFilePath, "..")
print("localProjectRootPath=%s" % localProjectRootPath)
seperatorLine()
print("Local environment:")
context.run("uname -a")
context.run("pwd")
context.run("ls -lha")
seperatorLine()
remoteConn = Connection(host=RemoteHost, user=RemoteUser)
print(remoteConn)
seperatorLine()
print("Remote Server:")
remoteConn.run('uname -a')
remoteConn.run('pwd')
print("remote path: %s" % RemotePathRoot)
remoteConn.run('ls -lha %s' % (RemotePathRoot))
# remoteConn.run('cd %s && pwd && ls -lha' % RemotePathRoot)
# putFileResult = remoteConn.put('fabfile.py', remote=RemotePathRoot)
# print("Uploaded {0.local} to {0.remote}".format(putFileResult))
# syncSource = "../deploy"
syncSource = localProjectRootPath
syncTarget = RemotePathRoot
# syncExclude = [".DS_Store", "data/", "processData/", "*.log", "*.pyc", "__pycache__"]
syncExclude = [
".DS_Store", ".git/", ".idea/", "*.pyc", "__pycache__",
"dump.rdb", "debug/", "logs/", "runtime/", "tmp/"]
syncResp = rsync(
remoteConn,
source=syncSource,
target=syncTarget,
# Note: be careful to add `delete`, to void to delete unexpected files
# delete=True,
exclude=syncExclude)
print("Sync lcoal %s to remote %s while exclude %s -> return %s" %
(syncSource, syncTarget, syncExclude, syncResp))
</code>上传本地代码：
<code>Sync lcoal /Users/xxx/deploy/.. to remote /root/x/web/server/robotDemo while exclude ['.DS_Store', '.git/', '.idea/', '*.pyc', '__pycache__', 'dump.rdb', 'debug/', 'logs/', 'runtime/', 'tmp/'] -> return Command exited with status 0. === stdout === building file list ... done ./ celerybeat.pid deploy/ deploy/fabfile.py sent 1.29K bytes received 100 bytes 926.00 bytes/sec total size is 77.13K speedup is 55.53 (no stderr) ➜ xx git:(master) ✗ fab -r deploy upload </code>
在线就可以看到代码了，并且也新建了对应的几个原本忽略掉的文件夹：
<code>(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pwd /root/x/web/server/robotDemo (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# mkdir logs (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# mkdir runtime (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# mkdir tmp (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# ll -lha total 68K drwxr-xr-x 9 root root 4.0K Aug 27 14:19 . drwxr-xr-x 6 root root 4.0K Aug 27 17:58 .. -rw-r--r-- 1 root root 1.5K Aug 27 10:26 app.py -rw-r--r-- 1 root root 6 Aug 27 14:19 celerybeat.pid drwxr-xr-x 2 root root 4.0K Aug 27 17:19 common drwxr-xr-x 5 root root 4.0K Aug 22 14:50 conf drwxr-xr-x 2 root root 4.0K Aug 27 18:03 deploy -rw-r--r-- 1 root root 6.6K Aug 25 15:07 factory.py -rw-r--r-- 1 root root 162 May 24 11:52 .gitignore drwxr-xr-x 2 root root 4.0K Aug 27 18:02 logs -rw-r--r-- 1 root root 495 Aug 27 10:35 Pipfile -rw-r--r-- 1 root root 6.2K Aug 27 14:03 README.md drwxr-xr-x 2 root root 4.0K Aug 27 17:05 resources drwxr-xr-x 2 root root 4.0K Aug 27 18:02 runtime drwxr-xr-x 2 root root 4.0K Aug 27 18:02 tmp (robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# </code>
重新创建pipenv的虚拟环境：
<code>(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pipenv --rm
No virtualenv has been created for this project yet!
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# rm /root/.local/share/virtualenvs/
BatchImportScript-vVMBoBec/ debug_nlp-wDGPvi3o/ x-VJ297lRu/
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# rm /root/.local/share/virtualenvs/^C
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# ll /root/.virtualenvs/
BatchImportScript-vVMBoBec/ debug_nlp-wDGPvi3o/ x-VJ297lRu/
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# cat Pipfile
[[source]]
#url = "https://pypi.python.org/simple"
url = "https://pypi.tuna.tsinghua.edu.cn/simple"
verify_ssl = true
name = "pypi"
[packages]
flask = "*"
flask-restful = "*"
flask-pymongo = "*"
gunicorn = "*"
pymysql = "*"
openpyxl = "*"
flask-cors = "*"
requests = "*"
celery = {extras = ["redis"]}
python-dotenv = "*"
numpy = "*"
schedule = "*"
pylru = "*"
jieba = "*"
pycnnum = "*"
solrclient = "*"
gensim = "*"
patchwork = "*"
tqdm = "*"
[dev-packages]
[requires]
python_version = "3.6"
(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pipenv install
Creating a virtualenv for this project…
Using /root/miniconda3/bin/python3.6m (3.6.5) to create virtualenv…
⠋Running virtualenv with interpreter /root/miniconda3/bin/python3.6m
Using base prefix '/root/miniconda3'
New python executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python3.6m
Also creating executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python
Please make sure you remove any previous custom paths from your /root/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
Virtualenv location: /root/.local/share/virtualenvs/robotDemo-dwdcgdaG
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
...
</code>感觉死掉了：
又卡死了。
重新试试：
<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --rm Removing virtualenv (/root/.local/share/virtualenvs/robotDemo-dwdcgdaG)… (robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv install ... </code>
终于开始安装库了：
<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv install Creating a virtualenv for this project… Using /root/miniconda3/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /root/miniconda3/bin/python3.6m Using base prefix '/root/miniconda3' New python executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python3.6m Also creating executable in /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python Please make sure you remove any previous custom paths from your /root/.pydistutils.cfg file. Installing setuptools, pip, wheel...done. Virtualenv location: /root/.local/share/virtualenvs/robotDemo-dwdcgdaG Pipfile.lock not found, creating… Locking [dev-packages] dependencies… Locking [packages] dependencies… Updated Pipfile.lock (ab35a7)! Installing dependencies from Pipfile.lock (ab35a7)… ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 60/60 — 00:00:28 (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# pipenv graph celery==4.2.1 - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.4] - kombu [required: <5.0,>=4.2.0, installed: 4.2.1] - amqp [required: <3.0,>=2.1.4, installed: 2.3.2] - vine [required: >=1.1.3, installed: 1.1.4] - pytz [required: >dev, installed: 2018.5] Flask-Cors==3.0.6 - Flask [required: >=0.9, installed: 1.0.2] - click [required: >=5.1, installed: 6.7] - itsdangerous [required: >=0.24, installed: 0.24] - Jinja2 [required: >=2.10, installed: 2.10] - MarkupSafe [required: >=0.23, installed: 1.0] - Werkzeug [required: >=0.14, installed: 0.14.1] - Six [required: Any, installed: 1.11.0] Flask-PyMongo==2.1.0 - Flask [required: >=0.11, installed: 1.0.2] - click [required: >=5.1, installed: 6.7] - itsdangerous [required: >=0.24, installed: 0.24] - Jinja2 [required: >=2.10, installed: 2.10] - MarkupSafe [required: >=0.23, installed: 1.0] - Werkzeug [required: >=0.14, installed: 0.14.1] - PyMongo [required: >=3.0, installed: 3.7.1] Flask-RESTful==0.3.6 - aniso8601 [required: >=0.82, installed: 3.0.2] - Flask [required: >=0.8, installed: 1.0.2] - click [required: >=5.1, installed: 6.7] - itsdangerous [required: >=0.24, installed: 0.24] - Jinja2 [required: >=2.10, installed: 2.10] - MarkupSafe [required: >=0.23, installed: 1.0] - Werkzeug [required: >=0.14, installed: 0.14.1] - pytz [required: Any, installed: 2018.5] - six [required: >=1.3.0, installed: 1.11.0] gensim==3.5.0 - numpy [required: >=1.11.3, installed: 1.15.1] - scipy [required: >=0.18.1, installed: 1.1.0] - numpy [required: >=1.8.2, installed: 1.15.1] - six [required: >=1.5.0, installed: 1.11.0] - smart-open [required: >=1.2.1, installed: 1.6.0] - boto [required: >=2.32, installed: 2.49.0] - boto3 [required: Any, installed: 1.8.1] - botocore [required: >=1.11.1,<1.12.0, installed: 1.11.1] - docutils [required: >=0.10, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3] - six [required: >=1.5, installed: 1.11.0] - urllib3 [required: <1.24,>=1.20, installed: 1.23] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3] - s3transfer [required: <0.2.0,>=0.1.10, installed: 0.1.13] - botocore [required: <2.0.0,>=1.3.0, installed: 1.11.1] - docutils [required: >=0.10, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3] - six [required: >=1.5, installed: 1.11.0] - urllib3 [required: <1.24,>=1.20, installed: 1.23] - bz2file [required: Any, installed: 0.98] - requests [required: Any, installed: 2.19.1] - certifi [required: >=2017.4.17, installed: 2018.8.24] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: <2.8,>=2.5, installed: 2.7] - urllib3 [required: <1.24,>=1.21.1, installed: 1.23] gunicorn==19.9.0 jieba==0.39 openpyxl==2.5.5 - et-xmlfile [required: Any, installed: 1.0.1] - jdcal [required: Any, installed: 1.4] patchwork==1.0.1 - fabric [required: <3.0,>=2.0, installed: 2.3.1] - cryptography [required: >=1.1, installed: 2.3.1] - asn1crypto [required: >=0.21.0, installed: 0.24.0] - cffi [required: !=1.11.3,>=1.7, installed: 1.11.5] - pycparser [required: Any, installed: 2.18] - idna [required: >=2.1, installed: 2.7] - six [required: >=1.4.1, installed: 1.11.0] - invoke [required: <2.0,>=1.1, installed: 1.1.1] - paramiko [required: >=2.4, installed: 2.4.1] - bcrypt [required: >=3.1.3, installed: 3.1.4] - cffi [required: >=1.1, installed: 1.11.5] - pycparser [required: Any, installed: 2.18] - six [required: >=1.4.1, installed: 1.11.0] - cryptography [required: >=1.5, installed: 2.3.1] - asn1crypto [required: >=0.21.0, installed: 0.24.0] - cffi [required: !=1.11.3,>=1.7, installed: 1.11.5] - pycparser [required: Any, installed: 2.18] - idna [required: >=2.1, installed: 2.7] - six [required: >=1.4.1, installed: 1.11.0] - pyasn1 [required: >=0.1.7, installed: 0.4.4] - pynacl [required: >=1.0.1, installed: 1.2.1] - cffi [required: >=1.4.1, installed: 1.11.5] - pycparser [required: Any, installed: 2.18] - six [required: Any, installed: 1.11.0] pycnnum==1.0.1 pylru==1.1.0 PyMySQL==0.9.2 - cryptography [required: Any, installed: 2.3.1] - asn1crypto [required: >=0.21.0, installed: 0.24.0] - cffi [required: !=1.11.3,>=1.7, installed: 1.11.5] - pycparser [required: Any, installed: 2.18] - idna [required: >=2.1, installed: 2.7] - six [required: >=1.4.1, installed: 1.11.0] python-dotenv==0.9.1 redis==2.10.6 schedule==0.5.0 SolrClient==0.2.1 - kazoo [required: ==2.2.1, installed: 2.2.1] - six [required: Any, installed: 1.11.0] - requests [required: >=2.2.1, installed: 2.19.1] - certifi [required: >=2017.4.17, installed: 2018.8.24] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: <2.8,>=2.5, installed: 2.7] - urllib3 [required: <1.24,>=1.21.1, installed: 1.23] tqdm==4.25.0 </code>
看看虚拟环境信息：
<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --where /root/xx/web/server/robotDemo (robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --py /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python (robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --envs The following environment variables can be set, to do various things: - PIPENV_DEFAULT_PYTHON_VERSION - PIPENV_PYTHON - PIPENV_NOSPIN - PIPENV_SHELL - PIPENV_USE_SYSTEM - PIPENV_MAX_DEPTH - PIPENV_PIPFILE - PIPENV_SKIP_VALIDATION - PIPENV_VIRTUALENV - PIPENV_MAX_SUBPROCESS - PIPENV_NO_INHERIT - PIPENV_DONT_USE_PYENV - PIPENV_COLORBLIND - PIPENV_DONT_LOAD_ENV - PIPENV_TIMEOUT - PIPENV_HIDE_EMOJIS - PIPENV_TEST_INDEX - PIPENV_VENV_IN_PROJECT - PIPENV_USE_HASHES - PIPENV_MAX_ROUNDS - PIPENV_SHELL_FANCY - PIPENV_INSTALL_TIMEOUT - PIPENV_CACHE_DIR - PIPENV_DOTENV_LOCATION - PIPENV_YES You can learn more at: http://docs.pipenv.org/advanced/#configuration-with-environment-variables (robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --venv /root/.local/share/virtualenvs/robotDemo-dwdcgdaG (robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# pipenv --version pipenv, version 11.10.0 </code>
其中的Python解析器位置是：
<code>/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/python </code>
对应的：
<code>(robotDemo-dwdcgdaG) [root@xx-general-01 robotDemo]# ll -lha /root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/ total 3.9M drwxr-xr-x 3 root root 4.0K Aug 27 18:13 . drwxr-xr-x 6 root root 4.0K Aug 27 18:12 .. -rw-r--r-- 1 root root 2.1K Aug 27 18:12 activate -rw-r--r-- 1 root root 1.1K Aug 27 18:12 activate.csh -rw-r--r-- 1 root root 2.2K Aug 27 18:12 activate.fish -rw-r--r-- 1 root root 1.2K Aug 27 18:12 activate_this.py -rwxr-xr-x 1 root root 12K Aug 27 18:13 asadmin -rwxr-xr-x 1 root root 1.6K Aug 27 18:13 bundle_image -rwxr-xr-x 1 root root 266 Aug 27 18:12 celery -rwxr-xr-x 1 root root 3.5K Aug 27 18:13 cfadmin -rwxr-xr-x 1 root root 273 Aug 27 18:13 chardetect -rwxr-xr-x 1 root root 3.1K Aug 27 18:13 cq -rwxr-xr-x 1 root root 5.0K Aug 27 18:13 cwutil -rwxr-xr-x 1 root root 259 Aug 27 18:13 dotenv -rwxr-xr-x 1 root root 2.2K Aug 27 18:13 dynamodb_dump -rwxr-xr-x 1 root root 3.5K Aug 27 18:13 dynamodb_load -rwxr-xr-x 1 root root 282 Aug 27 18:12 easy_install -rwxr-xr-x 1 root root 282 Aug 27 18:12 easy_install-3.6 -rwxr-xr-x 1 root root 9.6K Aug 27 18:13 elbadmin -rwxr-xr-x 1 root root 806 Aug 27 18:13 f2py -rwxr-xr-x 1 root root 272 Aug 27 18:13 fab -rwxr-xr-x 1 root root 1.9K Aug 27 18:13 fetch_file -rwxr-xr-x 1 root root 260 Aug 27 18:13 flask -rwxr-xr-x 1 root root 5.2K Aug 27 18:13 glacier -rwxr-xr-x 1 root root 269 Aug 27 18:13 gunicorn -rwxr-xr-x 1 root root 271 Aug 27 18:13 gunicorn_paster -rwxr-xr-x 1 root root 5.7K Aug 27 18:13 instance_events -rwxr-xr-x 1 root root 272 Aug 27 18:12 inv -rwxr-xr-x 1 root root 272 Aug 27 18:12 invoke -rwxr-xr-x 1 root root 1.7K Aug 27 18:12 jp.py -rwxr-xr-x 1 root root 970 Aug 27 18:13 kill_instance -rwxr-xr-x 1 root root 11K Aug 27 18:13 launch_instance -rwxr-xr-x 1 root root 3.1K Aug 27 18:13 list_instances -rwxr-xr-x 1 root root 3.5K Aug 27 18:13 lss3 -rwxr-xr-x 1 root root 19K Aug 27 18:13 mturk -rwxr-xr-x 1 root root 264 Aug 27 18:12 pip -rwxr-xr-x 1 root root 264 Aug 27 18:12 pip3 -rwxr-xr-x 1 root root 264 Aug 27 18:12 pip3.6 -rwxr-xr-x 1 root root 2.7K Aug 27 18:13 pyami_sendmail drwxr-xr-x 2 root root 4.0K Aug 27 18:13 __pycache__ lrwxrwxrwx 1 root root 10 Aug 27 18:12 python -> python3.6m lrwxrwxrwx 1 root root 10 Aug 27 18:12 python3 -> python3.6m lrwxrwxrwx 1 root root 10 Aug 27 18:12 python3.6 -> python3.6m -rwxr-xr-x 1 root root 3.6M Aug 27 18:12 python3.6m -rwxr-xr-x 1 root root 2.4K Aug 27 18:12 python-config -rwxr-xr-x 1 root root 8.9K Aug 27 18:13 route53 -rwxr-xr-x 1 root root 762 Aug 27 18:13 rst2html4.py -rwxr-xr-x 1 root root 1.2K Aug 27 18:13 rst2html5.py -rwxr-xr-x 1 root root 642 Aug 27 18:13 rst2html.py -rwxr-xr-x 1 root root 839 Aug 27 18:13 rst2latex.py -rwxr-xr-x 1 root root 648 Aug 27 18:13 rst2man.py -rwxr-xr-x 1 root root 1.8K Aug 27 18:13 rst2odt_prepstyles.py -rwxr-xr-x 1 root root 812 Aug 27 18:13 rst2odt.py -rwxr-xr-x 1 root root 649 Aug 27 18:13 rst2pseudoxml.py -rwxr-xr-x 1 root root 685 Aug 27 18:13 rst2s5.py -rwxr-xr-x 1 root root 919 Aug 27 18:13 rst2xetex.py -rwxr-xr-x 1 root root 650 Aug 27 18:13 rst2xml.py -rwxr-xr-x 1 root root 718 Aug 27 18:13 rstpep2html.py -rwxr-xr-x 1 root root 17K Aug 27 18:13 s3put -rwxr-xr-x 1 root root 6.9K Aug 27 18:13 sdbadmin -rwxr-xr-x 1 root root 3.8K Aug 27 18:13 taskadmin -rwxr-xr-x 1 root root 261 Aug 27 18:12 tqdm -rwxr-xr-x 1 root root 261 Aug 27 18:12 wheel </code>
也有要的：celery和gunicorn
也先去用Fabric：
deploy/fabfile.py
<code>from invoke import task
from fabric import Connection
from patchwork.transfers import rsync
import os
RemoteHost = '47.x.x.x'
RemoteUser = 'xxx'
# RemotePathRoot = '/root/xx/nlp/test_fabric'
RemotePathRoot = '/root/xx'
def seperatorLine(seperatorChar='-', count=80):
print(seperatorChar * count)
def fabLoadedPath():
from fabric.main import program
return program.collection.loaded_from
@task
def upload(context):
print("upload: context=%s", context)
seperatorLine()
fabFilePath = fabLoadedPath()
print("fabFilePath=%s" % fabFilePath)
localProjectRootPath = os.path.join(fabFilePath, "..")
print("localProjectRootPath=%s" % localProjectRootPath)
seperatorLine()
print("Local environment:")
context.run("uname -a")
context.run("pwd")
context.run("ls -lha")
seperatorLine()
remoteConn = Connection(host=RemoteHost, user=RemoteUser)
print(remoteConn)
seperatorLine()
print("Remote Server:")
remoteConn.run('uname -a')
remoteConn.run('pwd')
print("remote path: %s" % RemotePathRoot)
remoteConn.run('ls -lha %s' % (RemotePathRoot))
# remoteConn.run('cd %s && pwd && ls -lha' % RemotePathRoot)
# putFileResult = remoteConn.put('fabfile.py', remote=RemotePathRoot)
# print("Uploaded {0.local} to {0.remote}".format(putFileResult))
# syncSource = "../deploy"
syncSource = localProjectRootPath
syncTarget = RemotePathRoot
# syncExclude = [".DS_Store", "data/", "processData/", "*.log", "*.pyc", "__pycache__"]
syncExclude = [".DS_Store", "data/", "processData/mysqlQa/output_summary/", "*.log", "*.pyc", "__pycache__"]
syncResp = rsync(
remoteConn,
source=syncSource,
target=syncTarget,
# Note: be careful to add `delete`, to void to delete unexpected files
# delete=True,
exclude=syncExclude)
print("Sync lcoal %s to remote %s while exclude %s -> return %s" %
(syncSource, syncTarget, syncExclude, syncResp))
</code>上传x的代码（其中包含用到的search）：
<code>cd /Users/crifan/dev/dev_root/xxx fab -r deploy upload </code>
再去拷贝glove.6B数据到data中：
<code>(robotDemo-dwdcgdaG) [root@xx-general-01 xx]# cd /opt/word2vec/glove.6B (robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# ll total 3378124 -rwxrwxrwx 1 root root 347116733 Jul 18 21:52 glove.6B.100d.txt -rwxrwxrwx 1 root root 693432828 Jul 18 21:57 glove.6B.200d.txt -rwxrwxrwx 1 root root 1037962819 Jul 18 21:43 glove.6B.300d.txt -rwxrwxrwx 1 root root 1037962830 Jul 18 21:50 glove.6B.300d.w2vformat.txt -rwxrwxrwx 1 root root 171350079 Jul 18 21:53 glove.6B.50d.txt -rwxrwxrwx 1 root root 171350089 Jul 18 21:44 glove.6B.50d.w2vformat.txt (robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# ls -lha total 3.3G drwxrwxrwx 2 root root 4.0K Jul 18 21:53 . drwxrwxrwx 3 root root 4.0K Jul 18 21:37 .. -rwxrwxrwx 1 root root 332M Jul 18 21:52 glove.6B.100d.txt -rwxrwxrwx 1 root root 662M Jul 18 21:57 glove.6B.200d.txt -rwxrwxrwx 1 root root 990M Jul 18 21:43 glove.6B.300d.txt -rwxrwxrwx 1 root root 990M Jul 18 21:50 glove.6B.300d.w2vformat.txt -rwxrwxrwx 1 root root 164M Jul 18 21:53 glove.6B.50d.txt -rwxrwxrwx 1 root root 164M Jul 18 21:44 glove.6B.50d.w2vformat.txt (robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# cp /opt/word2vec/glove.6B/* /root/x/data/^C (robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# pwd /opt/word2vec/glove.6B (robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# cd /root/x/data (robotDemo-dwdcgdaG) [root@x-general-01 data]# ll total 0 (robotDemo-dwdcgdaG) [root@x-general-01 data]# mkdir glove.6B (robotDemo-dwdcgdaG) [root@x-general-01 data]# cp /opt/word2vec/glove.6B/* /root/x/data/glove.6B/ (robotDemo-dwdcgdaG) [root@x-general-01 data]# pwd /root/x/data (robotDemo-dwdcgdaG) [root@x-general-01 data]# cd glove.6B/ (robotDemo-dwdcgdaG) [root@x-general-01 glove.6B]# ll -lha total 3.3G drwxr-xr-x 2 root root 4.0K Aug 27 17:43 . drwxr-xr-x 3 root root 4.0K Aug 27 17:43 .. -rwxr-xr-x 1 root root 332M Aug 27 17:43 glove.6B.100d.txt -rwxr-xr-x 1 root root 662M Aug 27 17:43 glove.6B.200d.txt -rwxr-xr-x 1 root root 990M Aug 27 17:43 glove.6B.300d.txt -rwxr-xr-x 1 root root 990M Aug 27 17:43 glove.6B.300d.w2vformat.txt -rwxr-xr-x 1 root root 164M Aug 27 17:43 glove.6B.50d.txt -rwxr-xr-x 1 root root 164M Aug 27 17:43 glove.6B.50d.w2vformat.txt </code>
然后再去更新相关的
gunicorn和supervisor的配置：
更新了配置文件后：
conf/gunicorn/gunicorn_config.py
<code>import multiprocessing
import os
import sys
# sys.path.append(".")
# from config import BaseConfig
curPath = os.path.dirname(os.path.abspath(__file__))
print("curPath=%s" % curPath)
confPath = os.path.join(curPath, "..")
print("confPath=%s" % confPath)
sys.path.append(confPath)
from conf.app import settings
print("settings=%s" % settings)
appRootPath = os.path.join(confPath, "..")
print("appRootPath=%s" % appRootPath)
#appRootPath = "/Users/crifan/dev/dev_root/xxx/robotDemo"
# appRootPath = os.getcwd()
# print("appRootPath=%s" % appRootPath)
# flaskHost = BaseConfig.FLASK_HOST
# flaskPort = BaseConfig.FLASK_PORT
flaskHost = settings.FLASK_HOST
flaskPort = settings.FLASK_PORT
print("flaskHost=%s, flaskPort=%s" % (flaskHost, flaskPort))
reload = True #当代码改变时自动重启服务
#bind = '127.0.0.1:32851' #绑定ip和端口号
# bind = '0.0.0.0:32851' #绑定ip和端口号
bind = ("%s:%s" % (flaskHost, flaskPort))
backlog = 512 #监听队列
chdir = appRootPath #gunicorn要切换到的目的工作目录
timeout = 30 #超时
worker_class = 'sync' #默认的是sync模式
workers = multiprocessing.cpu_count() * 2 + 1 #进程数
threads = 2 #指定每个进程开启的线程数
loglevel = 'info' #日志级别，这个日志级别指的是错误日志的级别，而访问日志的级别无法设置
access_log_format = '%(t)s %(p)s %(h)s "%(r)s" %(s)s %(L)s %(b)s %(f)s" "%(a)s"' #设置gunicorn访问日志格式，错误日志无法设置
"""
其每个选项的含义如下：
h remote address
l '-'
u currently '-', may be user name in future releases
t date of the request
r status line (e.g. ``GET / HTTP/1.1``)
s status
b response length or '-'
f referer
a user agent
T request time in seconds
D request time in microseconds
L request time in decimal seconds
p process ID
"""
accesslog = appRootPath + "/logs/gunicorn_access.log" #访问日志文件
errorlog = appRootPath + "/logs/gunicorn_error.log" #错误日志文件
</code>和：
conf/supervisor/supervisord_server.conf
<code>;ROBOTDEMO_PRJOECT_ROOT=/root/x/web/server/robotDemo ;ROBOTDEMO_VIRTUALENV_ROOT=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG [program:redis] directory=/root/x/web/server/robotDemo ;command=/usr/bin/redis-server command=/usr/bin/redis-server /etc/redis.conf --dir /root/x/x/server/robotDemo/runtime autostart=true autorestart=true stdout_logfile=/root/x/web/server/robotDemo/logs/redis-%(program_name)s-stdout.log stdout_logfile_maxbytes=2MB stdout_logfile_backups=10 stderr_logfile=/root/x/web/server/robotDemo/logs/redis-%(program_name)s-stderr.log stderr_logfile_maxbytes=2MB stderr_logfile_backups=10 [program:robotDemo_CeleryWorker] command=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/celery worker -A resources.tasks.celery directory=/root/x/web/server/robotDemo autostart=true autorestart=true stdout_logfile=/root/x/web/server/robotDemo/logs/celery-worker-%(program_name)s-stdout.log stdout_logfile_maxbytes=2MB stdout_logfile_backups=10 stderr_logfile=/root/x/web/server/robotDemo/logs/celery-worker-%(program_name)s-stderr.log stderr_logfile_maxbytes=2MB stderr_logfile_backups=10 [program:robotDemo_CeleryBeat] command=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/celery beat -A resources.tasks.celery -s /root/x/web/server/robotDemo/runtime/celerybeat-schedule directory=/root/x/web/server/robotDemo autostart=true autorestart=true stdout_logfile=/root/x/web/server/robotDemo/logs/celery-beat-%(program_name)s-stdout.log stdout_logfile_maxbytes=2MB stdout_logfile_backups=10 stderr_logfile=/root/x/web/server/robotDemo/logs/celery-beat-%(program_name)s-stderr.log stderr_logfile_maxbytes=2MB stderr_logfile_backups=10 [program:robotDemo] command=/root/.local/share/virtualenvs/robotDemo-dwdcgdaG/bin/gunicorn -c conf/gunicorn/gunicorn_config.py app:app environment = FLASK_ENV="production" directory=/root/x/web/server/robotDemo startsecs=0 stopwaitsecs=0 autostart=true autorestart=true killasgroup=true stopasgroup=true stdout_logfile=/root/x/web/server/robotDemo/logs/supervisord-%(program_name)s-stdout.log stdout_logfile_maxbytes=2MB stdout_logfile_backups=10 stderr_logfile=/root/x/web/server/robotDemo/logs/supervisord-%(program_name)s-stderr.log stderr_logfile_maxbytes=2MB stderr_logfile_backups=10 </code>
用fabric部署更新文件到服务器，然后再去服务器中：
<code>(robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# cp -f conf/supervisor/supervisord_server.conf /etc/supervisord.d/supervisord_server.conf cp: overwrite ‘/etc/supervisord.d/supervisord_server.conf’? y (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl reload Restarted supervisord (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl status gunicorn STOPPED Not started redis RUNNING pid 15818, uptime 0:00:02 robotDemo RUNNING pid 15816, uptime 0:00:02 robotDemo_CeleryBeat RUNNING pid 15817, uptime 0:00:02 robotDemo_CeleryWorker RUNNING pid 15819, uptime 0:00:02 (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl start redis gunicorn STOPPED Not started^C (robotDemo-dwdcgdaG) [root@x-general-01 robotDemo]# supervisorctl start redis robotDemo robotDemo_CeleryBeat robotDemo_CeleryWorker redis: ERROR (already started) robotDemo: ERROR (already started) robotDemo_CeleryBeat: ERROR (already started) robotDemo_CeleryWorker: ERROR (already started) </code>
结果在线测试失败：
睿小麟英语智能机器人演示
http://xx/robotDemo/
需要找找原因。
<code>[root@xx-general-01 robotDemo]# supervisorctl status gunicorn STOPPED Not started redis RUNNING pid 15818, uptime 14:38:53 robotDemo RUNNING pid 9723, uptime 0:00:00 robotDemo_CeleryBeat RUNNING pid 15817, uptime 14:38:53 robotDemo_CeleryWorker RUNNING pid 15819, uptime 14:38:53 [root@x-general-01 robotDemo]# supervisorctl reload Restarted supervisord [root@x-general-01 robotDemo]# supervisorctl status error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 794 [root@xx-general-01 robotDemo]# supervisorctl status gunicorn STOPPED Not started redis STARTING robotDemo RUNNING pid 9731, uptime 0:00:00 robotDemo_CeleryBeat STARTING robotDemo_CeleryWorker STARTING [root@xx-general-01 robotDemo]# supervisorctl status gunicorn STOPPED Not started redis RUNNING pid 9733, uptime 0:00:02 robotDemo RUNNING pid 9731, uptime 0:00:02 robotDemo_CeleryBeat RUNNING pid 9732, uptime 0:00:02 robotDemo_CeleryWorker RUNNING pid 9734, uptime 0:00:02 </code>
看到状态有些奇怪：
gunicorn是另外一个项目，状态是stopped
而之前没有改动这个项目，应该是正常运行才对
去重启：
<code>[root@x-general-01 robotDemo]# supervisorctl restart all robotDemo: stopped robotDemo_CeleryBeat: stopped redis: stopped robotDemo_CeleryWorker: stopped robotDemo: started robotDemo_CeleryBeat: started redis: started gunicorn: started robotDemo_CeleryWorker: started [root@x-general-01 robotDemo]# supervisorctl status gunicorn RUNNING pid 9796, uptime 0:00:05 redis RUNNING pid 9795, uptime 0:00:05 robotDemo RUNNING pid 9793, uptime 0:00:05 robotDemo_CeleryBeat RUNNING pid 9794, uptime 0:00:05 robotDemo_CeleryWorker RUNNING pid 9797, uptime 0:00:05 </code>
然后删除log再去重启：
<code>[root@x-general-01 robotDemo]# pwd /root/x/web/server/robotDemo [root@xx-general-01 robotDemo]# ll total 92 -rw-r--r-- 1 root root 1448 Aug 27 10:26 app.py drwxr-xr-x 3 root root 4096 Aug 27 18:25 common drwxr-xr-x 6 root root 4096 Aug 27 18:25 conf drwxr-xr-x 2 root root 4096 Aug 27 18:03 deploy -rw-r--r-- 1 root root 6707 Aug 25 15:07 factory.py drwxr-xr-x 2 root root 4096 Aug 28 04:18 logs -rw-r--r-- 1 root root 495 Aug 27 10:35 Pipfile -rw-r--r-- 1 root root 35942 Aug 27 18:12 Pipfile.lock drwxr-xr-x 2 root root 4096 Aug 27 18:25 __pycache__ -rw-r--r-- 1 root root 6267 Aug 27 14:03 README.md drwxr-xr-x 3 root root 4096 Aug 27 18:25 resources drwxr-xr-x 2 root root 4096 Aug 28 09:06 runtime drwxr-xr-x 3 root root 4096 Aug 27 18:25 tmp [root@x-general-01 robotDemo]# rm -rf logs [root@x-general-01 robotDemo]# mkdir logs [root@x-general-01 robotDemo]# supervisorctl restart all robotDemo: started robotDemo_CeleryBeat: started redis: started gunicorn: started robotDemo_CeleryWorker: started [root@x-general-01 robotDemo]# supervisorctl status gunicorn RUNNING pid 9843, uptime 0:00:07 redis RUNNING pid 9842, uptime 0:00:07 robotDemo RUNNING pid 9870, uptime 0:00:01 robotDemo_CeleryBeat RUNNING pid 9841, uptime 0:00:07 robotDemo_CeleryWorker RUNNING pid 9844, uptime 0:00:07 </code>
方便去查看启动的log是否正常。
【未解决】在线环境中用gunicorn部署的产品demo无法正常初始化运行
转载请注明：在路上 » 【记录】把合并了基于搜索的兜底对话的产品demo部署到在线环境中