automation

Unexpected Automation: Quickly creating test data in MongoDB

Generating test data with JavaScript is as easy as pie!

Mark Winteringham

Mark Winteringham

— 9 min read
Unexpected Automation: Quickly creating test data in MongoDB
A picture of shipping containers stacked on top of one another and next to one another

Part of my workshop 'Understanding and testing RESTful Web serivces' relies on attendees testing my demo web service restful-booker and when restful-booker starts up it generates a group of records for the attendees to use.

I thought I would shared how I achieved this as I find a lot of time I am creating little scripts like the one we shall see in order to quickly create test data. So therefore in this post I'll go into how I create randomised data so that you can use this idea to help you create your own randomisation script that can easily be reused for automated checks, exploratory testing or performance testing.

The post assumes you have a basic understanding of JavaScript and Mongo.

Getting started

Before we start generating the data we will need the following applications installed and set up:

Head to the link, install Mongo and startup the Mongo service and we'll begin.

The problem

Restful-booker uses MongoDB to store the records for each booking. The model for each booking record is as follows:

{
  "_id": "554a49813fbbaece35502fb6",
  "firstname": "Sally",
  "lastname": "Brown",
  "totalprice": 111,
  "depositpaid": true,
  "additionalneeds": "Breakfast",
  "bookingdates": {
    "checkin": "2013-02-23T00:00:00.000Z",
    "checkout": "2014-10-23T00:00:00.000Z"
  }
}

The _id field is generated by Mongo but we need to generate the rest of the data in the model and do it in a way that each record we create contains randomised data.

Building the randomised data

Firstly, lets start by building the model and outputting the contents using Node.

var booking = {
  "firstname": "Sally",
  "lastname": "Brown",
  "totalprice": 111,
  "depositpaid": true,
  "additionalneeds": "Breakfast",
  "bookingdates": {
    "checkin": new Date(),
    "checkout": new Date()
  }
}

console.log(booking);

Running the script will output the object you've just created but we still need to randomise the data within the model so we'll start by creating a set of first names and then randomly choosing one:

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]

var randomiseNumber = function(from, to)
  return Math.floor(Math.random() * (to - from + 1) + from);
}

Firstly we add in the array of first names we want to select from and then create the randomiseNumber function. The randomiseNumber function allows us to put a minimum and maximum integer and it will use the native Math functions to generate an integer within your desired range. The reason we use this function is so we can easily increase or decrease the amount of entries in the first name array and leave the rest of the code unchanged.

You can read more on how the Math function works here.

So with the name array and the randomiseNumber function set in place we can now use them to randomise the firstname parameter.

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]

var randomiseNumber = function(from, to){
return Math.floor(Math.random() * (to - from + 1) + from);
}

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": "Brown",
  "totalprice": 111,
  "depositpaid": true,
  "additionalneeds": "Breakfast",
  "bookingdates": {
    "checkin": new Date(),
    "checkout": new Date()
  }
}

console.log(booking);

Now in place of static text for firstname we can use the name array and pass it a random integer using randomiseNumber. Since arrays are zero indexed we give it a minimum range of 0 and a maximum of the length of the array minus one to make the maximum zero indexed too. With that in place we can run the script a few times and now see the firstname parameter changing each time.

With that in place we can copy the same practise over to the lastname parameter like so:


var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1) + from);
}

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": 111,
  "depositpaid": true,
  "additionalneeds": "Breakfast",
  "bookingdates": {
    "checkin": new Date(),
    "checkout": new Date()
  }
}

console.log(booking);

With the firstname and lastname now randomised we can start on the totalprice parameter. Since we want to create a random integer for the total price we can simply reuse randomiseNumber again and give it a large range such as 100 to 1000:

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1) + from);
}

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": randomiseNumber(100,1000),
  "depositpaid": true,
  "additionalneeds": "Breakfast",
  "bookingdates": {
    "checkin": new Date(),
    "checkout": new Date()
  }
}

console.log(booking);

Now onto depositpaid. For this we can use a little trick with JavaScript and use Math.random() >= 0.5. Since Math.random() will generate a float between 0 and 1 we have an even chance of getting a true or false and using that little piece of code within an object will actually return us a boolean which will be added to the object. So add it in as shown below and give it a try:

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1) + from);
}

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": randomiseNumber(100,1000),
  "depositpaid": Math.random() >= 0.5,
  "additionalneeds": "Breakfast",
  "bookingdates": {
    "checkin": new Date(),
    "checkout": new Date()
  }
}

console.log(booking);

You'll notice now that the boolean for depositpaid will randomly switch now.

Now, in restful-booker the additionalneeds feed is optional so rather than randomise it's content we are going to randomise it's appearance. We can use the same randomised boolean trick we used for depositpaid like so:

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1)) + from;
}

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": randomiseNumber(100,1000),
  "depositpaid": Math.random() >= 0.5,
  "bookingdates": {
    "checkin": new Date(),
    "checkout": new Date()
  }
}

if(Math.random() >= 0.5){
  booking.additionalneeds = "Breakfast";
}

console.log(booking);

Finally we want to create randomised dates within a set range as we will always want the checkin date to be before the checkout date and we can do this by creating the following:

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1)) + from;
}

var randomiseDate = function(start, end) {
  return new Date(start.getTime() + Math.random() * (end.getTime() - start.getTime()));
}

var latestDate = new Date()
var checkInDate = randomiseDate(new Date(2013, 1, 1),latestDate)

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": randomiseNumber(100,1000),
  "depositpaid": Math.random() >;= 0.5,
  "bookingdates": {
    "checkin": checkInDate,
    "checkout": new Date(),
  }
}

if(Math.random() >= 0.5){
  booking.additionalneeds = "Breakfast";
}

console.log(booking);

The randomiseDate function much like the randomiseNumber function will take two date objects and create a date between the two ranges. So, as shown in the example above, we can create a checkin date between 1st January 2013 by creating two new date objects (one being specifically set to the 1st January 2013) and passing them torandomiseDate.

The date randomisation code was taken from here.

You may also notice that the checkInDate is created outside of the object and that's because we will use the checkInDate and the latestDate with randomiseDate to create the checkout date like so:

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1)) + from;
}

var randomiseDate = function(start, end) {
  return new Date(start.getTime() + Math.random() * (end.getTime() - start.getTime()));
}

var latestDate = new Date()
var checkInDate = randomiseDate(new Date(2013, 1, 1),latestDate)

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": randomiseNumber(100,1000),
  "depositpaid": Math.random() >= 0.5,
  "bookingdates": {
    "checkin": checkInDate,
    "checkout": randomiseDate(checkInDate,latestDate)
  }
}

if(Math.random() >= 0.5){
  booking.additionalneeds = "Breakfast";
}

console.log(booking);

So now we have the script in place to create randomised data we now need to get it into Mongo, which we can do in a couple of ways.

Getting the data into Mongo - Using the Mongo shell

The Mongo shell is one of the reasons why I love using Mongo.  The Mongo shell allows you to execute JavaScript within Mongo itself and it makes it very easy to run something like our script inside Mongo really easily and all it requires is replacing the console.log line of code with the following:

use test
db.bookings.insert(booking)

The use test line allows us to switch to the database we want to use and since Mongo comes with a test database out of the box that seems sensible to use. Then we call Mongo's library to connect to the db and insert our booking into a collection called bookings.

With those two lines in you can start up your mongo service and then simply call

mongo < name_of_file.js

Which will result in output similar to this:

MongoDB shell version: 2.6.5
connecting to: test
switched to db test
WriteResult({ "nInserted" : 1 })
bye

A quick look in Mongo in the test database will reveal a bookings collection with our record created. So if we want to create more than one record in one go we simply put our randomisation code into a loop with each loop ending with an insert into Mongo like so (Note that use test is kept at the top of the script now because we don't need to keep calling that each loop.)

use test

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1)) + from;
}

var randomiseDate = function(start, end) {
return new Date(start.getTime() + Math.random() * (end.getTime() - start.getTime()));
}

var latestDate = new Date()
var checkInDate = randomiseDate(new Date(2013, 1, 1),latestDate)

for(i = 0; i < 100; i++){ 
  var booking = {
    "firstname": name[randomiseNumber(0,name.length - 1)], "lastname": surname[randomiseNumber(0,surname.length - 1)], "totalprice": randomiseNumber(100,1000), "depositpaid": Math.random() >= 0.5,
    "bookingdates": {
      "checkin": checkInDate,
      "checkout": randomiseDate(checkInDate,latestDate)
    }
  }

  if(Math.random() >= 0.5){
    booking.additionalneeds = "Breakfast";
  }

  db.bookings.insert(booking)
}

And there you go, running the script again will produce you 100 randomised records. If you want more details on how to use the Mongo shell you can view them <a href="https://docs.mongodb.org/getting-started/shell/client/">here</a>.

But what if you can't access the Mongo shell well there is another way we can use our script.

Getting the data into Mongo - Remotely connecting to Mongo

Unless you are working with Mongo on a local system it fairly typical that you won't have access to the Mongo shell but there is another way in which you can create content via the MongoDB npm package for Node which allows us to remotely connect to Mongo and insert our data.

Before we change anything we will need to install the npm package mongodb. So run the following in the directory of your script:

npm install mongodb

With the mongodb package installed we update our script to the following:

var MongoClient = require('mongodb').MongoClient;

MongoClient.connect("mongodb://localhost:27017/test", function(err, db) {
var collection = db.collection('bookings');

var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

var randomiseNumber = function(from, to){
  return Math.floor(Math.random() * (to - from + 1)) + from;
}

var randomiseDate = function(start, end) {
  return new Date(start.getTime() + Math.random() * (end.getTime() - start.getTime()));
}

var latestDate = new Date()
var checkInDate = randomiseDate(new Date(2013, 1, 1),latestDate)

var booking = {
  "firstname": name[randomiseNumber(0,name.length - 1)],
  "lastname": surname[randomiseNumber(0,surname.length - 1)],
  "totalprice": randomiseNumber(100,1000),
  "depositpaid": Math.random() >= 0.5,
  "bookingdates": {
    "checkin": checkInDate,
    "checkout": randomiseDate(checkInDate,latestDate)
  }
}

if(Math.random() >= 0.5){
  booking.additionalneeds = "Breakfast";
}

collection.insert(booking);

db.close();
});

There are a few new parts to this that I'll explain

  1. Firstly we pull in the MongoClient library with the require which we will use to make the connection to Mongo
  2. With the library in place we make the connection to mongodb://localhost:27017/test.  Mongo runs on port 27017 as default but it may be updated so if connect to mongo on a different IP it may be the port is different. Finally the /test indicates which database we want to work with and just like the previous section we will use test.
  3. Once the connection is open we need to determine which collection within Mongo we want to use so for this example we assign the collection 'bookings' to the variable collection.
  4. With the collection now set we can generate our object and then finally call collection.insert to send our object to Mongo where it will be created.
  5. And to conclude we close the DB down to let the script exit.

Give the script a run using:

node .js

Once run you should see your randomised data appear in Mongo. And with that we finish the script by adding the for loop:

var MongoClient = require('mongodb').MongoClient;

MongoClient.connect("mongodb://localhost:27017/test", function(err, db) {
  var collection = db.collection('bookingshttp');

  var name = ["Mark","Mary","Sally","Jim","Eric","Susan"]
  var surname = ["Jones","Wilson","Jackson","Brown","Smith","Ericsson"]

  var randomiseNumber = function(from, to){
    return Math.floor(Math.random() * (to - from + 1)) + from;
  }

  var randomiseDate = function(start, end) {
    return new Date(start.getTime() + Math.random() * (end.getTime() - start.getTime()));
  }

  var latestDate = new Date()
  var checkInDate = randomiseDate(new Date(2013, 1, 1),latestDate)

  for(var i = 0; i < 100; i++){
    var booking = {
      "firstname": name[randomiseNumber(0,name.length - 1)],  
      "lastname": surname[randomiseNumber(0,surname.length - 1)],
      "totalprice": randomiseNumber(100,1000), "depositpaid": Math.random() >= 0.5,
      "bookingdates": {
        "checkin": checkInDate,
        "checkout": randomiseDate(checkInDate,latestDate)
      }
    }

    if(Math.random() >= 0.5){
      booking.additionalneeds = "Breakfast";
    }

    collection.insert(booking);
  }

  db.close();
});

And once again we have a script that can create multiple randomised bookings but remotely rather than within the shell.

Wrapping it all up

So we have looked at using JavaScript to create randomised data in a JSON structure and then looked at how we can use this to populate Mongo with data via the Mongo shell and remotely. But there is more! Not everyone has MongoDB in their technology stack so it maybe that this isn't entirely relevant, however in the next blog post we'll look about how you can use the randomisation script we've created here to create data over HTTP and learn about the scripts wider application.

Read more