最新消息:20210816 当前crifan.com域名已被污染,为防止失联,请关注(页面右下角的)公众号

【已解决】PySpider中self.crawl的POST的json参数如何不被encode编码而保持json原始字符串格式

JSON crifan 1301浏览 0评论
折腾:
【已解决】PySpider模拟请求小花生api接口出错:requests.exceptions.HTTPError HTTP 500 Internal Server Error
期间,PySpider去POST调用某个url,始终500出错。
调试期间发现:
Postman中调用正常,
其中json是能看到数据的字符串
而在PySpider中,却被编码成:
然后接口调用出500的错。
怀疑是json的问题。
尝试去看看:
PySpider中self.crawl中,如何指定json格式的data,而保证data不被字符编码
pyspider self.crawl pass json
pyspider self.crawl post json
pyspider – 使用self.crawl post提交application/json数据结果出错。如何转为json对象而不是字符串? – SegmentFault 思否
试试这个json.dumps
        jcJsonDict = {"J":"{\"userId\":\"1134723\",\"fieldName\":\"\",\"fieldValue\":\"全部类别\",\"theStageOfTheChild\":\"\",\"parentalEnglishLevel\":\"\",\"supportingResources\":\"\",\"offset\":0,\"limit\":10}","C":0}

        self.crawl(ParentChildReadingUrl,
            method="POST",
            # data=jcJsonDict,
            data= json.dumps(jcJsonDict),
            callback=self.getParentChildReadingCallback,
            save=parentChildReadingParamDict
        )
结果:
真的可以正常调用接口了。
【总结】
把代码从:
        jcJsonDict = {"J":"{\"userId\":\"1134723\",\"fieldName\":\"\",\"fieldValue\":\"全部类别\",\"theStageOfTheChild\":\"\",\"parentalEnglishLevel\":\"\",\"supportingResources\":\"\",\"offset\":0,\"limit\":10}","C":0}


        self.crawl(ParentChildReadingUrl,
            method="POST",
            data=jcJsonDict,
            callback=self.getParentChildReadingCallback,
            save=parentChildReadingParamDict
        )
改为:
        jcJsonDict = {"J":"{\"userId\":\"1134723\",\"fieldName\":\"\",\"fieldValue\":\"全部类别\",\"theStageOfTheChild\":\"\",\"parentalEnglishLevel\":\"\",\"supportingResources\":\"\",\"offset\":0,\"limit\":10}","C":0}


        self.crawl(ParentChildReadingUrl,
            method="POST",
            # data=jcJsonDict,
            data= json.dumps(jcJsonDict),
            callback=self.getParentChildReadingCallback,
            save=parentChildReadingParamDict
        )
就可以使得PySpider的接口调用期间的data中的json从:
  "fetch": {
    "data": "J=%7B%22userId%22%3A%221134723%22%2C%22fieldName%22%3A%22%22%2C%22fieldValue%22%3A%22%E5%85%A8%E9%83%A8%E7%B1%BB%E5%88%AB%22%2C%22theStageOfTheChild%22%3A%22%22%2C%22parentalEnglishLevel%22%3A%22%22%2C%22supportingResources%22%3A%22%22%2C%22offset%22%3A0%2C%22limit%22%3A10%7D&C=0",
变成:
  "fetch": {
    "data": "{\"J\": \"{\\\"userId\\\":\\\"1134723\\\",\\\"fieldName\\\":\\\"\\\",\\\"fieldValue\\\":\\\"\\u5168\\u90e8\\u7c7b\\u522b\\\",\\\"theStageOfTheChild\\\":\\\"\\\",\\\"parentalEnglishLevel\\\":\\\"\\\",\\\"supportingResources\\\":\\\"\\\",\\\"offset\\\":0,\\\"limit\\\":10}\", \"C\": 0}",
从而使得此处接口调用不出现之前的错误:
    requests.exceptions.HTTPError: HTTP 500: Internal Server Error
而可以正常返回期望结果:

转载请注明:在路上 » 【已解决】PySpider中self.crawl的POST的json参数如何不被encode编码而保持json原始字符串格式

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
82 queries in 0.168 seconds, using 22.17MB memory