之前通过代码批量把一堆的mp3和pdf等文件保存到在线的mongo中了。现在手上有3个mp3文件:
现在本地的fs.files中单个文件是:
<code>> db.fs.files.findOne() { "_id" : ObjectId("5b20d5ec7f4d384006e91f18"), "contentType" : "audio/mpeg", "chunkSize" : 261120, "metadata" : { "song" : { "singers" : [ ] }, "fitAgeStart" : 3, "topics" : [ "Wisdom", "Animals", "Villain" ], "storybook" : { "publisher" : "Mathew Price", "isFiction" : "是", "lexileIndex" : "", "awards" : "", "authors" : [ "Atsuko Morozumi" ], "foreignCountry" : "英国" }, "keywords" : { "fromName" : [ "go away Mr Wolf", "Mr wolf", "Go away", "wolf" ], "other" : [ ], "fromContent" : [ "knock!", "go away", "Mr Wolf", "quickly shut the door" ] }, "name" : "Go away, Mr Wolf", "resourceType" : "storybook", "mainActors" : [ "three little pigs", "Mr.Wolf" ], "contentAbstract" : "一个教小朋友警惕坏人的故事,狼先生扮成各种样子前来敲小猪家的们,3只小猪识别了狼的真面目,把门关上不让狼进来,一直到爸爸回来把发生的事情告诉爸爸听。", "isSeries" : false, "series" : { "number" : 0, "name" : "" }, "fitAgeEnd" : 4, "fileInfo" : { "isAudio" : true, "contentType" : "audio/mpeg", "name" : "Go away, Mr Wolf.mp3", "suffix" : "mp3" } }, "filename" : "Go away, Mr Wolf.mp3", "length" : 1613790, "uploadDate" : ISODate("2018-06-13T08:29:32.493Z"), "md5" : "3fa28e2ef5e7e56e15f797cd169845cc" } > </code>
需要保存mp3到mongo中,且把相关的信息都保存进去。
mongo save file with meta
mongo save file with metadata
mongofiles save with metadata
貌似没有metadata的选项啊
去看看Mongo Compass中是否有插入文件,且设置metadata的功能
貌似没有插入文件的选项?
去搜:
自己看了看:
貌似导入的是json
但是貌似没发指定mp3文件?
[JAVA-1013] Saved extra fields for Gridfs does not go in metadata sub-document – MongoDB
mongo 保存 metadata
mongo 保存 GridFS metadata
难道是只能和之前一样,去写python代码去保存file,同时传递metadata了?
mongo compass gridfs
mongo compass support gridfs
Creating Compass Plugins — MongoDB Compass stable
Visualizing your data with MongoDB Compass | MongoDB
【未解决】尝试用Mongo Management Studio去实现导入文件到Mongo的gridfs且带metadata信息
看到:
https://www.guru99.com/top-20-mongodb-tools.html
“15) NoSQL Manager:
* File Manager Tool to work with GridFS”
去看看
NoSQL Manager – GUI tool with intelligent Shell certified by MongoDB
貌似也没有gridfs上传文件的选项,更别说metadata了。
“File Manager tool to work with GridFS”
是支持gridfs上传文件的
不清楚是否支持修改metadata?
去试试
下载之前也先去看看功能:
Managing GridFS
File Manager tool to work with GridFS
free版本都支持。
Downloads – NoSQL Manager – GUI tool with intelligent Shell certified by MongoDB
下载:
NoSQL Manager for MongoDB Freeware
发现是exe版本的:
才注意到:
“System Requirements
* Microsoft Windows Vista or higher
* Microsoft .Net Framework 4.5”
而此处是Mac,用不了。
再去确认:
<code>mongofiles put --help </code>
和:
<code>mongofiles --help </code>
输出一样:
都不支持put时传递metadata
看来真的只能去写python代码去保存了。
【已解决】mongo的shell中find返回多个有限个数的结果
【总结】
最后是写代码实现:
把mp3文件存入进去后,且还带着metadata的:
<code># -*- coding: utf-8 -*- import pymongo from pymongo import MongoClient import gridfs # from pymongo.objectid import ObjectId # from pymongo import objectid from bson.objectid import ObjectId from gridfs import GridFS # import pprint import os import errno import logging import sys sys.path.append("libs/crifan") import crifanLib import re import time import mime from urllib import quote_plus ################################################################################ # Global Config/Setting ################################################################################ ########## Mongo Settings ########## # mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]] MongoHostRemote = "x.x.x.x" # MongoHostLocal = "localhost" # MongoHostLocal = "127.0.0.1" # MongoHost = MongoHostLocal MongoHost = MongoHostRemote MongoPort = 27017 </code>
MongoUseAuth = True
# MongoUseAuth = False
# with auth
MongoUsername = “gridfs”
MongoPassword = “P@wd”
MongoAuthenticationDatabase = “gridfs”
mongodbUri = “”
if MongoUseAuth :
mongodbUri = “mongodb://%s:%s@%s:%s/%s” % (
quote_plus(MongoUsername), \
quote_plus(MongoPassword), \
MongoHost, \
MongoPort, \
MongoAuthenticationDatabase \
)
#’mongodb://userName:[email protected]:27017/gridfs’
else:
mongodbUri = “mongodb://%s:%s” % (
MongoHost, \
MongoPort
)
# print “mongodbUri=”,mongodbUri
StorybookResourceType = “storybook”
SongResourceType = “song”
################################################################################
# Global Value
################################################################################
ToImportFileFolder = “files/to_import”
FileInfoList = [
{
“filename”: “Finger Family.mp3”,
“metadata”: {
“song”: {
“singers”: [
“ChuChu TV”
]
},
“fitAgeStart”: 3,
“topics”: [
],
“storybook”: {
“publisher”: “”,
“isFiction”: “未知”,
“lexileIndex”: “”,
“awards”: “”,
“authors”: [],
“foreignCountry”: “”
},
“keywords”: {
“fromName”: [
“Finger”,
“Family”
],
“other”: [],
“fromContent”: [
“Daddy Finger”
]
},
“name”: “Finger Family”,
“resourceType”: “song”,
“mainActors”: [],
“contentAbstract”: “”,
“isSeries”: False,
“series”: {
“number”: 0,
“name”: “”
},
“fitAgeEnd”: 6,
“fileInfo”: {
“isAudio”: True,
“contentType”: “audio/mpeg”,
“name”: “Finger Family.mp3”,
“suffix”: “mp3”
}
},
},
{
“filename”: “Finger Family Collection.mp3”,
“metadata”: {
“song”: {
“singers”: []
},
“fitAgeStart”: 3,
“topics”: [
],
“storybook”: {
“publisher”: “”,
“isFiction”: “未知”,
“lexileIndex”: “”,
“awards”: “”,
“authors”: [],
“foreignCountry”: “”
},
“keywords”: {
“fromName”: [
“Finger”,
“Family”,
“Collection”
],
“other”: [],
“fromContent”: [
“Daddy Finger Collection”
]
},
“name”: “Finger Family”,
“resourceType”: “song”,
“mainActors”: [],
“contentAbstract”: “”,
“isSeries”: False,
“series”: {
“number”: 0,
“name”: “”
},
“fitAgeEnd”: 6,
“fileInfo”: {
“isAudio”: True,
“contentType”: “audio/mpeg”,
“name”: “Finger Family Collection.mp3”,
“suffix”: “mp3”
}
},
},
{
“filename”: “Mr. Sun, Sun, Mr. Golden Sun.mp3”,
“metadata”: {
“song”: {
“singers”: [
“Super Simple Songs”
]
},
“fitAgeStart”: 3,
“topics”: [
],
“storybook”: {
“publisher”: “”,
“isFiction”: “未知”,
“lexileIndex”: “”,
“awards”: “”,
“authors”: [],
“foreignCountry”: “”
},
“keywords”: {
“fromName”: [
“Sun”,
“Golden Sun”
],
“other”: [],
“fromContent”: []
},
“name”: “Mr. Sun, Sun, Mr. Golden Sun”,
“resourceType”: “song”,
“mainActors”: [],
“contentAbstract”: “”,
“isSeries”: False,
“series”: {
“number”: 0,
“name”: “”
},
“fitAgeEnd”: 6,
“fileInfo”: {
“isAudio”: True,
“contentType”: “audio/mpeg”,
“name”: “Mr. Sun, Sun, Mr. Golden Sun.mp3”,
“suffix”: “mp3”
}
},
},
]
################################################################################
# Local Function
################################################################################
def initLogging():
“””
init logging
:return: log file name
“””
global gCfg
# init logging
filenameNoSufx = crifanLib.getInputFileBasenameNoSuffix()
logFilename = filenameNoSufx + “.log”
crifanLib.loggingInit(logFilename)
return logFilename
def saveFileToDb(curFullFilename, metadataDict):
“””
get more file meta info then save (text or audio) file into mongo db
:param curFullFilename:
:param metadataDict:
:return:
“””
resultDict = {
“hasSaved”: False,
“errMsg”: “Unknown error”
}
if not os.path.isfile(curFullFilename):
errMsg = “Not exist file for %s” % curFullFilename
logging.error(errMsg)
resultDict[“hasSaved”] = False
resultDict[“errMsg”] = errMsg
return resultDict
curFilename = crifanLib.getBasename(curFullFilename) #’Finger Family.mp3′
logging.info(“curFilename=%s”, curFilename)
metadataDict[“fileInfo”][“name”] = curFilename
fileSuffix = crifanLib.getFileSuffix(curFullFilename)
logging.info(“fileSuffix=%s”, fileSuffix)
metadataDict[“fileInfo”][“suffix”] = fileSuffix
# extract MIME
curFileMimeType = mime.Types.of(curFilename)[0].content_type
logging.info(“curFileMimeType=%s”, curFileMimeType) #’audio/mpeg’, ‘application/pdf’
metadataDict[“fileInfo”][“contentType”] = curFileMimeType
if re.match(“^audio/.+”, curFileMimeType):
metadataDict[“fileInfo”][“isAudio”] = True
with open(curFullFilename) as curFp:
curFileObjectId = fsCollection.put(
curFp,
filename=curFilename,
content_type=curFileMimeType,
metadata=metadataDict)
logging.info(“curFileObjectId=%s”, curFileObjectId)
# readOutFile = fsCollection.get(curFileObjectId)
# logging.info(“readOutFile=%s”, readOutFile)
# fileMedata = readOutFile.metadata
# logging.info(“fileMedata=%s”, fileMedata)
fileIdStr = str(curFileObjectId) #’5ac31bd2a4bc7156fd727d59′
resultDict[“hasSaved”] = True
resultDict[“errMsg”] = “”
logging.info(“+++ Saved file: id=%s, name=%s”, fileIdStr, curFilename)
return resultDict
def processFiles():
“””proces each file”””
curPath = os.getcwd()
for curIdx, curFileInfo in enumerate(FileInfoList):
curNum = curIdx + 1
curFilename = curFileInfo[“filename”]
logging.info(“-“*30 + ” [%d] %s ” + “-“*30, curNum, curFilename)
metadataDict = curFileInfo[“metadata”]
logging.info(“metadataDict=%s”, metadataDict)
curFullFilename = os.path.join(curPath, ToImportFileFolder, curFilename)
resultDict = saveFileToDb(curFullFilename, metadataDict)
logging.info(“resultDict=%s”, resultDict)
################################################################################
# Main Part
################################################################################
initLogging()
# init mongodb
mongoClient = MongoClient(mongodbUri)
logging.info(“mongoClient=%s”, mongoClient)
gridfsDb = mongoClient.gridfs
logging.info(“gridfsDb=%s”, gridfsDb)
fsCollection = GridFS(gridfsDb)
logging.info(“fsCollection=%s”, fsCollection)
processFiles()
效果:
<code>2018/07/23 02:19:25 LINE 375 INFO ------------------------------ [3] Mr. Sun, Sun, Mr. Golden Sun.mp3 ------------------------------ 2018/07/23 02:19:25 LINE 378 INFO metadataDict={'song': {'singers': ['Super Simple Songs']}, 'series': {'number': 0, 'name': ''}, 'topics': [], 'storybook': {'publisher': '', 'isFiction': '\xe6\x9c\xaa\xe7\x9f\xa5', 'lexileIndex': '', 'awards': '', 'authors': [], 'foreignCountry': ''}, 'keywords': {'fromName': ['Sun', 'Golden Sun'], 'other': [], 'fromContent': []}, 'name': 'Mr. Sun, Sun, Mr. Golden Sun', 'resourceType': 'song', 'mainActors': [], 'contentAbstract': '', 'isSeries': False, 'fitAgeStart': 3, 'fitAgeEnd': 6, 'fileInfo': {'isAudio': True, 'contentType': 'audio/mpeg', 'name': 'Mr. Sun, Sun, Mr. Golden Sun.mp3', 'suffix': 'mp3'}} 2018/07/23 02:19:25 LINE 330 INFO curFilename=Mr. Sun, Sun, Mr. Golden Sun.mp3 2018/07/23 02:19:25 LINE 334 INFO fileSuffix=mp3 2018/07/23 02:19:25 LINE 339 INFO curFileMimeType=audio/mpeg 2018/07/23 02:19:25 LINE 351 INFO curFileObjectId=5b55736d7f4d38be4e02496c 2018/07/23 02:19:25 LINE 362 INFO +++ Saved file: id=5b55736d7f4d38be4e02496c, name=Mr. Sun, Sun, Mr. Golden Sun.mp3 2018/07/23 02:19:25 LINE 383 INFO resultDict={'hasSaved': True, 'errMsg': ''} </code>
系统中就可以选择了:
后续再去:
【未解决】Mongo中更新gridfs中的mp3文件的metadata信息且尽量保持id不变
转载请注明:在路上 » 【已解决】把本地mp3文件存入在线Mongo中且填写meta信息