最新消息:20210816 当前crifan.com域名已被污染,为防止失联,请关注(页面右下角的)公众号

【已解决】把本地mp3文件存入在线Mongo中且填写meta信息

MongoDB crifan 4460浏览 0评论

之前通过代码批量把一堆的mp3和pdf等文件保存到在线的mongo中了。现在手上有3个mp3文件:

现在本地的fs.files中单个文件是:

<code>&gt; db.fs.files.findOne()
{
        "_id" : ObjectId("5b20d5ec7f4d384006e91f18"),
        "contentType" : "audio/mpeg",
        "chunkSize" : 261120,
        "metadata" : {
                "song" : {
                        "singers" : [ ]
                },
                "fitAgeStart" : 3,
                "topics" : [
                        "Wisdom",
                        "Animals",
                        "Villain"
                ],
                "storybook" : {
                        "publisher" : "Mathew Price",
                        "isFiction" : "是",
                        "lexileIndex" : "",
                        "awards" : "",
                        "authors" : [
                                "Atsuko Morozumi"
                        ],
                        "foreignCountry" : "英国"
                },
                "keywords" : {
                        "fromName" : [
                                "go away Mr Wolf",
                                "Mr wolf",
                                "Go away",
                                "wolf"
                        ],
                        "other" : [ ],
                        "fromContent" : [
                                "knock!",
                                "go away",
                                "Mr Wolf",
                                "quickly shut the door"
                        ]
                },
                "name" : "Go away, Mr Wolf",
                "resourceType" : "storybook",
                "mainActors" : [
                        "three little pigs",
                        "Mr.Wolf"
                ],
                "contentAbstract" : "一个教小朋友警惕坏人的故事,狼先生扮成各种样子前来敲小猪家的们,3只小猪识别了狼的真面目,把门关上不让狼进来,一直到爸爸回来把发生的事情告诉爸爸听。",
                "isSeries" : false,
                "series" : {
                        "number" : 0,
                        "name" : ""
                },
                "fitAgeEnd" : 4,
                "fileInfo" : {
                        "isAudio" : true,
                        "contentType" : "audio/mpeg",
                        "name" : "Go away, Mr Wolf.mp3",
                        "suffix" : "mp3"
                }
        },
        "filename" : "Go away, Mr Wolf.mp3",
        "length" : 1613790,
        "uploadDate" : ISODate("2018-06-13T08:29:32.493Z"),
        "md5" : "3fa28e2ef5e7e56e15f797cd169845cc"
}
&gt;
</code>

需要保存mp3到mongo中,且把相关的信息都保存进去。

mongo save file with meta

$meta — MongoDB Manual

Consider MongoDB and GridFS for meta, thumbnails, transcoded video etc · Issue #46 · jansmolders86/mediacenterjs

Adding a field to the JSON of a PDF in MongoDB => NullPointerException for the river · Issue #91 · richardwilly98/elasticsearch-river-mongodb

mongo save file with metadata

File: GridFS

mongofiles save with metadata

mongofiles — MongoDB Manual

貌似没有metadata的选项啊

去看看Mongo Compass中是否有插入文件,且设置metadata的功能

貌似没有插入文件的选项?

去搜:

自己看了看:

貌似导入的是json

但是貌似没发指定mp3文件?

[JAVA-1013] Saved extra fields for Gridfs does not go in metadata sub-document – MongoDB

mongo 保存 metadata

mongo 保存 GridFS  metadata

mongodb开发—GridFS文件存储 – CSDN博客

难道是只能和之前一样,去写python代码去保存file,同时传递metadata了?

mongo compass gridfs

mongo compass support gridfs

Creating Compass Plugins — MongoDB Compass stable

Visualizing your data with MongoDB Compass | MongoDB

【未解决】尝试用Mongo Management Studio去实现导入文件到Mongo的gridfs且带metadata信息

看到:

https://www.guru99.com/top-20-mongodb-tools.html

“15) NoSQL Manager:

* File Manager Tool to work with GridFS”

去看看

NoSQL Manager – GUI tool with intelligent Shell certified by MongoDB

貌似也没有gridfs上传文件的选项,更别说metadata了。

“File Manager tool to work with GridFS”

是支持gridfs上传文件的

不清楚是否支持修改metadata?

去试试

下载之前也先去看看功能:

NoSQL Manager for MongoDB – Features – NoSQL Manager – GUI tool with intelligent Shell certified by MongoDB

Managing GridFS

File Manager tool to work with GridFS

free版本都支持。

Downloads – NoSQL Manager – GUI tool with intelligent Shell certified by MongoDB

下载:

NoSQL Manager for MongoDB Freeware

发现是exe版本的:

才注意到:

“System Requirements

* Microsoft Windows Vista or higher

* Microsoft .Net Framework 4.5”

而此处是Mac,用不了。

再去确认:

<code>mongofiles put --help
</code>

和:

<code>mongofiles --help
</code>

输出一样:

都不支持put时传递metadata

看来真的只能去写python代码去保存了。

【已解决】mongo的shell中find返回多个有限个数的结果

【总结】

最后是写代码实现:

把mp3文件存入进去后,且还带着metadata的:

<code># -*- coding: utf-8 -*-

import pymongo
from pymongo import MongoClient
import gridfs

# from pymongo.objectid import ObjectId
# from pymongo import objectid
from bson.objectid import ObjectId

from gridfs import GridFS
# import pprint
import os
import errno
import logging
import sys
sys.path.append("libs/crifan")
import crifanLib
import re
import time
import mime
from urllib import quote_plus


################################################################################
# Global Config/Setting
################################################################################

########## Mongo Settings ##########
# mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]

MongoHostRemote = "x.x.x.x"
# MongoHostLocal = "localhost"
# MongoHostLocal = "127.0.0.1"
# MongoHost = MongoHostLocal
MongoHost = MongoHostRemote

MongoPort = 27017

</code>

MongoUseAuth = True

# MongoUseAuth = False

# with auth

MongoUsername = “gridfs”

MongoPassword = “P@wd”

MongoAuthenticationDatabase = “gridfs”

mongodbUri = “”

if MongoUseAuth :

    mongodbUri = “mongodb://%s:%s@%s:%s/%s” % (

        quote_plus(MongoUsername), \

        quote_plus(MongoPassword), \

        MongoHost, \

        MongoPort, \

        MongoAuthenticationDatabase \

    )

    #’mongodb://userName:[email protected]:27017/gridfs’

else:

    mongodbUri = “mongodb://%s:%s” % (

        MongoHost, \

        MongoPort

    )

    #’mongodb://localhost:27017

    #’mongodb://x.x.x.x:27017

# print “mongodbUri=”,mongodbUri

StorybookResourceType = “storybook”

SongResourceType = “song”

################################################################################

# Global Value

################################################################################

ToImportFileFolder = “files/to_import”

FileInfoList = [

    {

        “filename”: “Finger Family.mp3”,

        “metadata”: {

            “song”: {

                “singers”: [

                    “ChuChu TV”

                ]

            },

            “fitAgeStart”: 3,

            “topics”: [

            ],

            “storybook”: {

                “publisher”: “”,

                “isFiction”: “未知”,

                “lexileIndex”: “”,

                “awards”: “”,

                “authors”: [],

                “foreignCountry”: “”

            },

            “keywords”: {

                “fromName”: [

                    “Finger”,

                    “Family”

                ],

                “other”: [],

                “fromContent”: [

                    “Daddy Finger”

                ]

            },

            “name”: “Finger Family”,

            “resourceType”: “song”,

            “mainActors”: [],

            “contentAbstract”: “”,

            “isSeries”: False,

            “series”: {

                “number”: 0,

                “name”: “”

            },

            “fitAgeEnd”: 6,

            “fileInfo”: {

                “isAudio”: True,

                “contentType”: “audio/mpeg”,

                “name”: “Finger Family.mp3”,

                “suffix”: “mp3”

            }

        },

    },

    {

        “filename”: “Finger Family Collection.mp3”,

        “metadata”: {

            “song”: {

                “singers”: []

            },

            “fitAgeStart”: 3,

            “topics”: [

            ],

            “storybook”: {

                “publisher”: “”,

                “isFiction”: “未知”,

                “lexileIndex”: “”,

                “awards”: “”,

                “authors”: [],

                “foreignCountry”: “”

            },

            “keywords”: {

                “fromName”: [

                    “Finger”,

                    “Family”,

                    “Collection”

                ],

                “other”: [],

                “fromContent”: [

                    “Daddy Finger Collection”

                ]

            },

            “name”: “Finger Family”,

            “resourceType”: “song”,

            “mainActors”: [],

            “contentAbstract”: “”,

            “isSeries”: False,

            “series”: {

                “number”: 0,

                “name”: “”

            },

            “fitAgeEnd”: 6,

            “fileInfo”: {

                “isAudio”: True,

                “contentType”: “audio/mpeg”,

                “name”: “Finger Family Collection.mp3”,

                “suffix”: “mp3”

            }

        },

    },

    {

        “filename”: “Mr. Sun, Sun, Mr. Golden Sun.mp3”,

        “metadata”: {

            “song”: {

                “singers”: [

                    “Super Simple Songs”

                ]

            },

            “fitAgeStart”: 3,

            “topics”: [

            ],

            “storybook”: {

                “publisher”: “”,

                “isFiction”: “未知”,

                “lexileIndex”: “”,

                “awards”: “”,

                “authors”: [],

                “foreignCountry”: “”

            },

            “keywords”: {

                “fromName”: [

                    “Sun”,

                    “Golden Sun”

                ],

                “other”: [],

                “fromContent”: []

            },

            “name”: “Mr. Sun, Sun, Mr. Golden Sun”,

            “resourceType”: “song”,

            “mainActors”: [],

            “contentAbstract”: “”,

            “isSeries”: False,

            “series”: {

                “number”: 0,

                “name”: “”

            },

            “fitAgeEnd”: 6,

            “fileInfo”: {

                “isAudio”: True,

                “contentType”: “audio/mpeg”,

                “name”: “Mr. Sun, Sun, Mr. Golden Sun.mp3”,

                “suffix”: “mp3”

            }

        },

    },

]

################################################################################

# Local Function

################################################################################

def initLogging():

    “””

    init logging

    :return: log file name

    “””

    global gCfg

    # init logging

    filenameNoSufx = crifanLib.getInputFileBasenameNoSuffix()

    logFilename = filenameNoSufx + “.log”

    crifanLib.loggingInit(logFilename)

    return logFilename

def saveFileToDb(curFullFilename, metadataDict):

    “””

    get more file meta info then save (text or audio) file into mongo db

    :param curFullFilename:

    :param metadataDict:

    :return:

    “””

    resultDict = {

        “hasSaved”: False,

        “errMsg”: “Unknown error”

    }

    if not os.path.isfile(curFullFilename):

        errMsg = “Not exist file for %s” % curFullFilename

        logging.error(errMsg)

        resultDict[“hasSaved”] = False

        resultDict[“errMsg”] = errMsg

        return resultDict

    curFilename = crifanLib.getBasename(curFullFilename) #’Finger Family.mp3′

    logging.info(“curFilename=%s”, curFilename)

    metadataDict[“fileInfo”][“name”] = curFilename

    fileSuffix = crifanLib.getFileSuffix(curFullFilename)

    logging.info(“fileSuffix=%s”, fileSuffix)

    metadataDict[“fileInfo”][“suffix”] = fileSuffix

    # extract MIME

    curFileMimeType = mime.Types.of(curFilename)[0].content_type

    logging.info(“curFileMimeType=%s”, curFileMimeType) #’audio/mpeg’, ‘application/pdf’

    metadataDict[“fileInfo”][“contentType”] = curFileMimeType

    if re.match(“^audio/.+”, curFileMimeType):

        metadataDict[“fileInfo”][“isAudio”] = True

    with open(curFullFilename) as curFp:

        curFileObjectId = fsCollection.put(

            curFp,

            filename=curFilename,

            content_type=curFileMimeType,

            metadata=metadataDict)

        logging.info(“curFileObjectId=%s”, curFileObjectId)

        # readOutFile = fsCollection.get(curFileObjectId)

        # logging.info(“readOutFile=%s”, readOutFile)

        # fileMedata = readOutFile.metadata

        # logging.info(“fileMedata=%s”, fileMedata)

        fileIdStr = str(curFileObjectId) #’5ac31bd2a4bc7156fd727d59′

        resultDict[“hasSaved”] = True

        resultDict[“errMsg”] = “”

        logging.info(“+++ Saved file: id=%s, name=%s”, fileIdStr, curFilename)

    return resultDict

def processFiles():

    “””proces each file”””

    curPath = os.getcwd()

    for curIdx, curFileInfo in enumerate(FileInfoList):

        curNum = curIdx + 1

        curFilename = curFileInfo[“filename”]

        logging.info(“-“*30 + ” [%d] %s ” + “-“*30, curNum, curFilename)

        metadataDict = curFileInfo[“metadata”]

        logging.info(“metadataDict=%s”, metadataDict)

        curFullFilename = os.path.join(curPath, ToImportFileFolder, curFilename)

        resultDict = saveFileToDb(curFullFilename, metadataDict)

        logging.info(“resultDict=%s”, resultDict)

################################################################################

# Main Part

################################################################################

initLogging()

# init mongodb

mongoClient = MongoClient(mongodbUri)

logging.info(“mongoClient=%s”, mongoClient)

gridfsDb = mongoClient.gridfs

logging.info(“gridfsDb=%s”, gridfsDb)

fsCollection = GridFS(gridfsDb)

logging.info(“fsCollection=%s”, fsCollection)

processFiles()

效果:

<code>2018/07/23 02:19:25 LINE 375  INFO    ------------------------------ [3] Mr. Sun, Sun, Mr. Golden Sun.mp3 ------------------------------
2018/07/23 02:19:25 LINE 378  INFO    metadataDict={'song': {'singers': ['Super Simple Songs']}, 'series': {'number': 0, 'name': ''}, 'topics': [], 'storybook': {'publisher': '', 'isFiction': '\xe6\x9c\xaa\xe7\x9f\xa5', 'lexileIndex': '', 'awards': '', 'authors': [], 'foreignCountry': ''}, 'keywords': {'fromName': ['Sun', 'Golden Sun'], 'other': [], 'fromContent': []}, 'name': 'Mr. Sun, Sun, Mr. Golden Sun', 'resourceType': 'song', 'mainActors': [], 'contentAbstract': '', 'isSeries': False, 'fitAgeStart': 3, 'fitAgeEnd': 6, 'fileInfo': {'isAudio': True, 'contentType': 'audio/mpeg', 'name': 'Mr. Sun, Sun, Mr. Golden Sun.mp3', 'suffix': 'mp3'}}
2018/07/23 02:19:25 LINE 330  INFO    curFilename=Mr. Sun, Sun, Mr. Golden Sun.mp3
2018/07/23 02:19:25 LINE 334  INFO    fileSuffix=mp3
2018/07/23 02:19:25 LINE 339  INFO    curFileMimeType=audio/mpeg
2018/07/23 02:19:25 LINE 351  INFO    curFileObjectId=5b55736d7f4d38be4e02496c
2018/07/23 02:19:25 LINE 362  INFO    +++ Saved file: id=5b55736d7f4d38be4e02496c, name=Mr. Sun, Sun, Mr. Golden Sun.mp3
2018/07/23 02:19:25 LINE 383  INFO    resultDict={'hasSaved': True, 'errMsg': ''}
</code>

系统中就可以选择了:

后续再去:

【未解决】Mongo中更新gridfs中的mp3文件的metadata信息且尽量保持id不变

转载请注明:在路上 » 【已解决】把本地mp3文件存入在线Mongo中且填写meta信息

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
89 queries in 0.191 seconds, using 22.19MB memory