In most cases, databases makes the management of data quite convenient. In this article, we would scrape data using the code we discussed before but write data into MongoDB.

For installation of MongoDB, please refer to the official documentation.

The Code

To write data to MongoDB using Node.js, we choose the package mongojs, which provides almost exactly the standard MongoDB syntax.

To install mongojs,

npm i mongojs --save

Here is a module that can write data to MongoDB. We create a file named dao.js and copy pase the following code into it.

// use mongojs
const mongojs = require('mongojs')
// connect to the database 'simple_spider' in MongoDB and use collection 'test'
const localdb = mongojs('simple_spider', ['test'])
// a function that saves data to MongoDB
const saveData = (data,cb) => {
	localdb.test.save(data, (err, res)=>{
	 	cb && cb(err, res)
	})
}
// a function that prints out the data to console
const printAllData = (cb) => {
	localdb.test.find({},(err, docs)=>{
		console.log(docs)
		cb && cb(err, docs)
	})
}
// close connection to MongoDB in case of memory leak
const closeMongo = () => localdb.close()
// expose the functions in this module to the program
module.exports = {
	saveData,printAllData,closeMongo
}

() =>

Three functions are provided in this module, saveData, printAllData and closeMongo. We will grab our previous code and call the functions from this module. We would like to modify the index.js file to make it look like the following.

const superagent = require('superagent')
const fs = require('fs')
const dao = require('./dao')
superagent
	.get('https://api.bilibili.com/x/web-interface/archive/stat')
	.query({ aid:26763233 })
	.then(res => {
    // obtain the data to be saved
		const data = res.body.data
		dao.saveData(data, (err,res)=>{
			console.log('saved data')
			// save data to MongoDB and close connection
			dao.printAllData( () => dao.closeMongo() )
		})
	})
	.catch(err => console.error(err))

Run the code with the command

node index

and we obtaint he following output:

saved data
[ { _id: 5b41d634f3df89032c834d5b,
    aid: 26186448,
    view: 619751,
    danmaku: 12053,
    reply: 3500,
    favorite: 14961,
    coin: 40699,
    share: 1676,
    like: 20751,
    now_rank: 0,
    his_rank: 1,
    no_reprint: 1,
    copyright: 1 } ]

With this print out, we would have already have out data stored in our MongoDB.