折腾:
【已解决】python的solr运行出错:SolrClient.exceptions.ConnectionError: 404
目的是Mac中本地运行起来,加上合适的配置的solr的server
但是缺少对应的配置。
而线上的CentOS中有个正在运行的solr,现在希望去搞清楚,线上那个的正在运行的solr的配置是什么,比如什么配置文件之类的。
然后再去想办法复制到本地,使得本地Mac中可以正常运行代码中所需要的solr的server。
how check running solr config
running solr how get config
solr server配置
Running Solr | Apache Solr Reference Guide 6.6
通过之前的:
<code>[root@xxx-general-01 xxx]# ps aux | grep solr dev 5424 0.0 3.8 5819128 633084 ? Sl Jul18 38:45 java -server -Xms2g -Xmx2g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/dev/nlp/qa/data/solr-7.4.0/server/logs/sol_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.log.dir=/home/dev/nlp/qa/data/solr-7.4.0/server/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/home/dev/nlp/qa/data/solr-7.4.0/server -Dsolr.solr.home=/home/dev/nlp/qa/data/solr-7.4.0/server/solr -Dsolr.data.home= -Dsolr.install.dir=/home/dev/nlp/qa/data/solr-7.4.0 -Dsolr.default.confdir=/home/dev/nlp/qa/data/solr-7.4.0/server/solr/configsets/_default/conf -Xss256k -Dsolr.jetty.https.port=8983 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/home/dev/nlp/qa/data/solr-7.4.0/bin/oom_solr.sh 8983 /home/dev/nlp/qa/data/solr-7.4.0/server/logs -jar start.jar --module=http root 24991 0.0 0.0 112664 976 pts/0 S+ 14:53 0:00 grep --color=auto solr </code>
而找到server的路径。
经过自己的简单查看:
<code>[root@xx-xx-01 qa]# cd /home/dev/nlp/qa/data/solr-7.4.0/server/ [root@xx-xx-01 server]# ll total 184 drwxrwxrwx 2 root root 4096 Jul 18 22:02 contexts drwxrwxrwx 2 root root 4096 Jul 18 22:02 etc drwxrwxrwx 3 root root 4096 Jul 18 22:04 lib drwxrwxrwx 2 root root 4096 Jul 18 22:15 logs drwxrwxrwx 2 root root 4096 Jul 18 22:04 modules -rwxrwxrwx 1 root root 4074 Jul 18 22:04 README.txt drwxrwxrwx 2 root root 4096 Jul 18 22:02 resources drwxrwxrwx 3 root root 4096 Jul 18 22:04 scripts drwxrwxrwx 5 root root 4096 Jul 18 22:03 solr drwxrwxrwx 3 root root 4096 Jul 18 22:04 solr-webapp -rwxrwxrwx 1 root root 147435 Jul 18 22:04 start.jar [root@xx-general-01 server]# cd solr [root@xx-general-01 solr]# l -bash: l: command not found [root@xx-general-01 solr]# ll total 24 drwxrwxrwx 4 root root 4096 Jul 18 22:03 configsets drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa_fast -rwxrwxrwx 1 root root 3095 Jul 18 22:03 README.txt -rwxrwxrwx 1 root root 2170 Jul 18 22:03 solr.xml -rwxrwxrwx 1 root root 1006 Jul 18 22:03 zoo.cfg [root@xx-general-01 solr]# cat solr.xml <?xml version="1.0" encoding="UTF-8" ?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <!-- This is an example of a simple "solr.xml" file for configuring one or more Solr Cores, as well as allowing Cores to be added, removed, and reloaded via HTTP requests. More information about options available in this configuration file, and Solr Core administration can be found online: http://wiki.apache.org/solr/CoreAdmin --> <solr> <solrcloud> <str name="host">${host:}</str> <int name="hostPort">${jetty.port:8983}</int> <str name="hostContext">${hostContext:solr}</str> <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool> <int name="zkClientTimeout">${zkClientTimeout:30000}</int> <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int> <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int> <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str> <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str> </solrcloud> <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory"> <int name="socketTimeout">${socketTimeout:600000}</int> <int name="connTimeout">${connTimeout:60000}</int> </shardHandlerFactory> </solr> [root@xx-general-01 solr]# ll total 24 drwxrwxrwx 4 root root 4096 Jul 18 22:03 configsets drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa_fast -rwxrwxrwx 1 root root 3095 Jul 18 22:03 README.txt -rwxrwxrwx 1 root root 2170 Jul 18 22:03 solr.xml -rwxrwxrwx 1 root root 1006 Jul 18 22:03 zoo.cfg [root@xx-general-01 solr]# cat zoo.cfg # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # dataDir=/opt/zookeeper/data # NOTE: Solr defaults the dataDir to <solrHome>/zoo_data # the port at which the clients will connect # clientPort=2181 # NOTE: Solr sets this based on zkRun / zkHost params # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 [root@xx-general-01 solr]# cd qa [root@xx-general-01 qa]# ll total 12 drwxrwxrwx 3 root root 4096 Jul 18 22:03 conf -rwxrwxrwx 1 root root 72 Jul 18 22:02 core.properties drwxrwxrwx 5 root root 4096 Jul 18 22:02 data [root@xx-general-01 qa]# cat core.properties #Written by CorePropertiesLocator #Wed Jan 24 05:44:00 UTC 2018 name=qa [root@xx-general-01 qa]# cd conf/ [root@xx-general-01 conf]# ll total 108 drwxrwxrwx 2 root root 4096 Jul 18 22:03 lang -rwxrwxrwx 1 root root 30999 Jul 18 22:03 managed-schema -rwxrwxrwx 1 root root 308 Jul 18 22:03 params.json -rwxrwxrwx 1 root root 873 Jul 18 22:03 protwords.txt -rwxrwxrwx 1 root root 54029 Jul 18 22:03 solrconfig.xml -rwxrwxrwx 1 root root 781 Jul 18 22:03 stopwords.txt -rwxrwxrwx 1 root root 1123 Jul 18 22:03 synonyms.txt [root@xx-general-01 conf]# cat params.json {"params":{ "query":{ "defType":"edismax", "q.alt":"*:*", "rows":"10", "fl":"*,score", "":{"v":0} }, "facets":{ "facet":"on", "facet.mincount": "1", "":{"v":0} }, "velocity":{ "wt": "velocity", "v.template":"browse", "v.layout": "layout", "":{"v":0} } </code>
感觉是此处server下面的qa,要么是动态生成的,要么是最开始的配置,也就是我们要找的。
所以继续研究看看。
想办法找找此处CentOS中
solr status
中的solr在哪里
参考:
https://lucene.apache.org/solr/guide/6_6/running-solr.html
去线上试试:
<code>[root@xx-xx-01 conf]# curl http://localhost:8983/solr/qa/select?q=question_str%3A%22sing+a++song%22&fq=%2A%3A%2A+AND+scene%3Aqa&rows=1&fl=question%2Canswer%2Cid&wt=json&indent=false [1] 25086 [2] 25087 [3] 25088 [4] 25089 [5] 25090 [2] Done fq=%2A%3A%2A+AND+scene%3Aqa [3] Done rows=1 [x@x-general-01 conf]# { "responseHeader":{ "status":0, "QTime":0, "params":{ "q":"question_str:\"sing a song\""}}, "response":{"numFound":0,"start":0,"docs":[] }} [1] Done curl http://localhost:8983/solr/qa/select?q=question_str%3A%22sing+a++song%22 [4]- Done fl=question%2Canswer%2Cid [5]+ Done wt=json [root@xx-general-01 conf]# </code>
Config API | Apache Solr Reference Guide 6.6
Solr Configuration Files | Apache Solr Reference Guide 7.0
默认home目录是:
server/solr
所以去看看服务器中的
从:
<code>[root@xx-general-01 qa]# pwd /home/dev/nlp/qa/data/solr-7.4.0/server/solr/qa [root@xx-general-01 qa]# cd .. [root@xx-general-01 solr]# ll total 24 drwxrwxrwx 4 root root 4096 Jul 18 22:03 configsets drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa_fast -rwxrwxrwx 1 root root 3095 Jul 18 22:03 README.txt -rwxrwxrwx 1 root root 2170 Jul 18 22:03 solr.xml -rwxrwxrwx 1 root root 1006 Jul 18 22:03 zoo.cfg [root@xx-general-01 solr]# ll configsets/ total 8 drwxrwxrwx 3 root root 4096 Jul 18 22:03 _default drwxrwxrwx 3 root root 4096 Jul 18 22:03 sample_techproducts_configs [root@xx-general-01 solr]# cd qa_fast/ [root@xx-general-01 qa_fast]# ll total 12 drwxrwxrwx 3 root root 4096 Jul 18 22:03 conf -rwxrwxrwx 1 root root 77 Jul 18 22:03 core.properties drwxrwxrwx 5 root root 4096 Jul 18 22:03 data [root@xx-general-01 qa_fast]# cd ../qa [root@xx-general-01 qa]# ll total 12 drwxrwxrwx 3 root root 4096 Aug 16 15:49 conf -rwxrwxrwx 1 root root 72 Jul 18 22:02 core.properties drwxrwxrwx 5 root root 4096 Jul 18 22:02 data [root@xx-general-01 qa]# pwd /home/dev/nlp/qa/data/solr-7.4.0/server/solr/qa </code>
可以看出是:
Standalone Mode
再去:
<code>[root@xx-general-01 solr]# ll total 24 drwxrwxrwx 4 root root 4096 Jul 18 22:03 configsets drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa drwxrwxrwx 4 root root 4096 Jul 18 22:03 qa_fast -rwxrwxrwx 1 root root 3095 Jul 18 22:03 README.txt -rwxrwxrwx 1 root root 2170 Jul 18 22:03 solr.xml -rwxrwxrwx 1 root root 1006 Jul 18 22:03 zoo.cfg [root@xx-general-01 solr]# du -sh qa 90M qa [root@xx-general-01 solr]# du -sh qa_fast/ 1.4M qa_fast/ </code>
像是:
qa是我们要的配置
而qa_fast是暂时没用到的?
https://www.jianshu.com/p/9a42afdd7bb3
也看到了,默认都有solr.xml和zoo.cfg
自己此处:
<code>[root@xx-general-01 solr]# cd qa [root@xx-general-01 qa]# ll total 12 drwxrwxrwx 3 root root 4096 Aug 16 15:49 conf -rwxrwxrwx 1 root root 72 Jul 18 22:02 core.properties drwxrwxrwx 5 root root 4096 Jul 18 22:02 data [root@xx-general-01 qa]# ll conf total 108 drwxrwxrwx 2 root root 4096 Jul 18 22:03 lang -rwxrwxrwx 1 root root 30999 Jul 18 22:03 managed-schema -rwxrwxrwx 1 root root 308 Jul 18 22:03 params.json -rwxrwxrwx 1 root root 873 Jul 18 22:03 protwords.txt -rwxrwxrwx 1 root root 54029 Jul 18 22:03 solrconfig.xml -rwxrwxrwx 1 root root 781 Jul 18 22:03 stopwords.txt -rwxrwxrwx 1 root root 1123 Jul 18 22:03 synonyms.txt [root@xx-general-01 qa]# ll data total 12 drwxrwxrwx 2 root root 4096 Jul 18 22:03 index drwxrwxrwx 2 root root 4096 Jul 18 22:02 snapshot_metadata drwxrwxrwx 2 root root 4096 Jul 18 22:02 tlog [root@xx-general-01 qa]# </code>
而https://www.jianshu.com/p/9a42afdd7bb3是
基本上都是一致的。
-》感觉自己直接把服务器中的qa目录下载下来,或许就可以了?
不过先去看看此处Mac中solr的server目录下有啥
<code>➜ solr git:(master) ✗ pwd /usr/local/Cellar/solr/7.2.1/server/solr ➜ solr git:(master) ✗ ll total 24 -rw-r--r-- 1 crifan admin 2.9K 1 10 2018 README.txt drwxr-xr-x 4 crifan admin 128B 1 10 2018 configsets -rw-r--r-- 1 crifan admin 2.1K 1 10 2018 solr.xml -rw-r--r-- 1 crifan admin 975B 1 10 2018 zoo.cfg ➜ solr git:(master) ✗ ll configsets total 0 drwxr-xr-x 3 crifan admin 96B 1 10 2018 _default drwxr-xr-x 3 crifan admin 96B 1 10 2018 sample_techproducts_configs ➜ solr git:(master) ✗ </code>
看来直接拷贝qa整个目录过来或许就可以了?
提示我了:
中可以看到JVM配置参数,
去看看自己本地的
<code>JVM RuntimeOracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_112 25.112-b16 Processors4 Args-DSTOP.KEY=solrrocks -DSTOP.PORT=7983 -Djetty.home=/usr/local/Cellar/solr/7.2.1/server -Djetty.port=8983 -Dsolr.data.home= -Dsolr.default.confdir=/usr/local/Cellar/solr/7.2.1/server/solr/configsets/_default/conf -Dsolr.install.dir=/usr/local/Cellar/solr/7.2.1 -Dsolr.jetty.https.port=8983 -Dsolr.log.dir=/usr/local/Cellar/solr/7.2.1/server/logs -Dsolr.log.muteconsole -Dsolr.solr.home=/usr/local/Cellar/solr/7.2.1/server/solr -Duser.timezone=UTC -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:+ParallelRefProcEnabled -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:ConcGCThreads=4 -XX:GCLogFileSize=20M -XX:MaxTenuringThreshold=8 -XX:NewRatio=3 -XX:NumberOfGCLogFiles=9 -XX:OnOutOfMemoryError=/usr/local/Cellar/solr/7.2.1/bin/oom_solr.sh 8983 /usr/local/Cellar/solr/7.2.1/server/logs -XX:ParallelGCThreads=4 -XX:PretenureSizeThreshold=64m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -Xloggc:/usr/local/Cellar/solr/7.2.1/server/logs/solr_gc.log -Xms512m -Xmx512m -Xss256k -verbose:gc </code>
再去看看线上的:
http://xxx:pp/solr/
->估计是阿里云的安全组限制了端口访问,导致此处无法打开
去开通端口:
Format of solr.xml | Apache Solr Reference Guide 6.6
The Well-Configured Solr Instance | Apache Solr Reference Guide 6.6
创建了core,相当于api入口?
重启后即可在Core admin中看到
那说明在线环境中的core admin中会有qa
而自己本地这里没有是因为没有创建。
另外对比了:
在线和本地的:
<code>➜ solr git:(master) ✗ ll total 24 -rw-r--r-- 1 crifan admin 2.9K 1 10 2018 README.txt drwxr-xr-x 4 crifan admin 128B 1 10 2018 configsets -rw-r--r-- 1 crifan admin 2.1K 1 10 2018 solr.xml -rw-r--r-- 1 crifan admin 975B 1 10 2018 zoo.cfg </code>
中的:
solr.xml
内容是一样的
去看看在线的web页信息:
http://xxx/solr/
可以打开,貌似很慢
-XX:OnOutOfMemoryError=/home/dev/nlp/qa/data/solr-7.4.0/bin/oom_solr.sh 8983 /home/dev/nlp/qa/data/solr-7.4.0/server/logs
果然有2个core:
是时候去下载在线的qa到本地了,再重启本地的solr的server
突然想到:难道后续也要把qa_fast下载下来?暂时不去下,后面如果发现qa有问题再说。
重启solr
<code> solr git:(master) ✗ pwd /usr/local/Cellar/solr/7.2.1/server/solr ➜ solr git:(master) ✗ solr status Found 1 Solr nodes: Solr process 83817 running on port 8983 { "solr_home":"/usr/local/Cellar/solr/7.2.1/server/solr", "version":"7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:54:21", "startTime":"2018-08-16T06:24:08.798Z", "uptime":"0 days, 2 hours, 42 minutes, 8 seconds", "memory":"43 MB (%8.8) of 490.7 MB"} ➜ solr git:(master) ✗ solr stop Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 83817 to stop gracefully. ➜ solr git:(master) ✗ solr start Waiting up to 180 seconds to see Solr running on port 8983 [\] </code>
然后Core Admin中出错:
<code>SolrCore Initialization Failures qa:org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher Please check your logs for more information </code>
<code>Log4j (org.slf4j.impl.Log4jLoggerFactory) Time (Local) Level Core Logger Message 2018/8/16 下午5:07:03 WARN false SystemInfoHandler Resolving canonical hostname for local host took 30.171 seconds,​ possible DNS misconfiguration. Set the 'solr.dns.prevent.reverse.lookup' sysprop to true on startup to prevent future lookups if DNS can not be fixed. 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/contrib/extraction/lib filtered by .*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/contrib/extraction/lib 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/dist filtered by solr-cell-\d.*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/dist 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/contrib/clustering/lib filtered by .*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/contrib/clustering/lib 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/dist filtered by solr-clustering-\d.*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/dist 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/contrib/langid/lib filtered by .*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/contrib/langid/lib 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/dist filtered by solr-langid-\d.*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/dist 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/contrib/velocity/lib filtered by .*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/contrib/velocity/lib 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/dist filtered by solr-velocity-\d.*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/dist 2018/8/16 下午5:07:03 WARN false SolrConfig Couldn't add files from /usr/local/Cellar/solr/7.2.1/dist filtered by .*\.jar to classpath: /usr/local/Cellar/solr/7.2.1/dist 2018/8/16 下午5:07:37 WARN false SystemInfoHandler Resolving canonical hostname for local host took 30.004 seconds,​ possible DNS misconfiguration. Set the 'solr.dns.prevent.reverse.lookup' sysprop to true on startup to prevent future lookups if DNS can not be fixed. 2018/8/16 下午5:07:37 ERROR true CoreContainer </code>
看起来错误和:
/usr/local/Cellar/solr/7.2.1/server/solr/qa/conf/solrconfig.xml
有关
去添加一个core,叫qa,试试
instanceDir and dataDir need to exist before you can create the core
结果失败:
<code>Error CREATEing SolrCore 'qa': Could not create a new core in /usr/local/Cellar/solr/7.2.1/server/solr/qaas another core is already defined there </code>
问题依旧。
去问问别人,在线的slor的server是如何搭建出来的
【总结】
至此,基本上算是大概明白了对应的逻辑了:
其实在线的solr的服务中的qa,想要本地模拟的话,直接拷贝过来,貌似是出错的。
所以此处,对于想要在线的环境完全模拟一遍的话,貌似通过直接复制的办法,搞不定。
以后有机会再去研究这条路。
现在去用正常的新建并导入数据的步骤:
【记录】Mac本地Solr中新建qa的collection再运行脚本导入数据
转载请注明:在路上 » 【无需解决】CentOS中正在运行的Solr如何得到其所有的配置