Updated March 2017 Since this post, other MEAN & MERN stack posts have been written: The Modern Application Stack by Andrew Morgan.
In this first part, we will describe the basic mechanics of our application and undertake data modeling. In the second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application.
Let’s begin by defining the MEAN stack.
What is the MEAN stack?
The MEAN stack can be summarized as follows:
- M = MongoDB/Mongoose.js: the popular database, and an elegant ODM for node.js.
- E = Express.js: a lightweight web application framework.
The MEAN stack is a modern replacement for the LAMP (Linux, Apache, MySQL, PHP/Python) stack that became the popular way for building web applications in the late 1990s.
In our application, we won’t be using Angular.js, as we are not building an HTML user interface. Instead, we are building a REST API which has no user interface, but could instead serve as the basis for any kind of interface, such as a website, an Android application, or an iOS application. You might say we are building our REST API on the ME(a)N stack, but we have no idea how to pronounce that!
What is a REST API?
REST stands for Representational State Transfer. It is a lighter weight alternative to SOAP and WSDL XML-based API protocols.
The CRUD acronym is often used to describe database operations. CRUD stands for CREATE, READ, UPDATE, and DELETE. These database operations map very nicely to the HTTP verbs, as follows:
- POST: A client wants to insert or create an object.
- GET: A client wants to read an object.
- PUT: A client wants to update an object.
- DELETE: A client wants to delete an object.
These operations will become clear later when define our API.
Some of the common HTTP result codes that are often used inside REST APIs are as follows:
- 200 - “OK”.
- 201 - “Created” (Used with POST).
- 400 - “Bad Request” (Perhaps missing required parameters).
- 401 - “Unauthorized” (Missing authentication parameters).
- 403 - “Forbidden” (You were authenticated but lacking required privileges).
- 404 - “Not Found”.
A complete description can be found in the RFC document, listed in the resources section at the end of this blog. We will use these result codes in our application and you will see some examples shortly.
Why Are We Starting with a REST API?
Developing a REST API enables us to create a foundation upon which we can build all other applications. As previously mentioned, these applications may be web-based or designed for specific platforms, such as Android or iOS.
Today, there are also many companies that are building applications that do not use an HTTP or web interface, such as Uber, WhatsApp, Postmates, and Wash.io. A REST API also makes it easy to implement other interfaces or applications over time, turning the initial project from a single application into a powerful platform.
Creating our REST API
The application that we will be building will be an RSS Aggregator, similar to Google Reader. Our application will have two main components:
- The REST API
- Feed Grabber (similar to Google Reader)
In this blog series we will focus on building the REST API, and we will not cover the intricacies of RSS feeds. However, code for Feed Grabber is available in a github repository, listed in the resources section of this blog.
Let’s now describe the process we will follow in building our API. We will begin by defining the data model for the following requirements:
- Store user information in user accounts
- Track RSS feeds that need to be monitored
- Pull feed entries into the database
- Track user feed subscriptions
- Track which feed entry a user has already read
Users will need to be able to do the following:
- Create an account
- Subscribe/unsubscribe to feeds
- Read feed entries
- Mark feeds/entries as read or unread
Modeling Our Data
An in-depth discussion on data modeling in MongoDB is beyond the scope of this article, so see the references section for good resources on this topic.
We will need 4 collections to manage this information:
- Feed collection
- Feed entry collection
- User collection
- User-feed-entry mapping collection
Let’s take a closer look at each of these collections.
Lets now look at some code. To model a feed collection, we can use the following JSON document:
If you are familiar with relational database technology, then you will know about databases, tables, rows and columns. In MongoDB, there is a mapping to most of these Relational concepts. At the highest level, a MongoDB deployment supports one or more databases. A database contains one or more collections, which are the similar to tables in a relational database. Collections hold documents. Each document in a collection is, at a highest level, similar to a row in a relational table. However, documents do not follow a fixed schema with pre-defined columns of simple values. Instead, each document consists of one or more key-value pairs where the value can be simple (e.g., a date), or more sophisticated (e.g., an array of address objects).
Our JSON document above is an example of one RSS feed for the Eater Blog, which tracks information about restaurants in New York City. We can see that there are a number of different fields but the key ones that our client application may be interested in include the
URL of the feed and the
feed description. The description is important so that if we create a mobile application, it would show a nice summary of the feed.
The remaining fields in our JSON document are for internal use. A very important field is
_id. In MongoDB, every document must have a field called
_id. If you create a document without this field, at the point where you save the document, MongoDB will create it for you. In MongoDB, this field is a primary key and MongoDB will guarantee that within a collection, this value is unique.
Feed Entry Collection
After feeds, we want to track feed entries. Here is an example of a document in the feed entry collection:
Again, we can see that there is a
_id field. There are also some other fields, such as
summary. For the content field, note that we are using an array, and the array is also storing a document. MongoDB allows us to store sub-documents in this way and this can be very useful in some situations, where we want to hold all information together. The
entryID field uses the tag format to avoid duplicate feed entries. Notice also the
feedID field that is of type
ObjectId - the value is the
_id of the Eater Blog document, described earlier. This provides a referential model, similar to a foreign key in a relational database. So, if we were interested to see the feed document associated with this
ObjectId, we could take the value
523b1153a2aa6a3233a913f8 and query the feed collection on
_id, and it would return the Eater Blog document.
Here is the document we could use to keep track of users:
A user has an
first name and
last name. There is also an
sp_api_key_secret - we will use these later with Stormpath, a user management API. The last field, called subs, is a subscription array. The subs field tells us which feeds this user is subscribed-to.
User-Feed-Entry Mapping Collection
The last collection allows us to map users to feeds and to track which feeds have been read.
We use a Boolean (true/false) to mark the feed as read or unread.
Functional Requirements for the REST API
As previously mentioned, users need to be able to do the following:
- Create an account.
- Subscribe/unsubscribe to feeds.
- Read feed entries.
- Mark feeds/entries as read or unread.
Additionally, a user should be able to reset their password.
The following table shows how these operations can be mapped to HTTP routes and verbs.
||Register a new user
||Get feed subscriptions for each user with description and unread count
||Subscribe to a new feed
||Get all entries for feeds the user is subscribed to
||Get all entries for a specific feed
||Mark all entries for a specific feed as read or unread
||read = <true | false>
||Mark a specific entry as either read or unread
||read = <true | false>
||Unsubscribe from this particular feed
In a production environment, the use of secure HTTP (HTTPS) would be the standard approach when sending sensitive details, such as passwords.
Real World Authentication with Stormpath
In robust real-world applications it is important to provide user authentication. We need a secure approach to manage users, passwords, and password resets.
There are a number of ways we could authenticate users for our application. One possibility is to use Node.js with the Passport Plugin, which could be useful if we wanted to authenticate with social media accounts, such as Facebook or Twitter. However, another possibility is to use Stormpath. Stormpath provides User Management as a Service and supports authentication and authorization through API keys. Basically, Stormpath maintains a database of user details and passwords and a client application REST API would call the Stormpath REST API to perform user authentication.
The following diagram shows the flow of requests and responses using Stormpath.
In detail, Stormpath will provide a secret key for each “Application” that is defined with their service. For example, we could define an application as “Reader Production” or “Reader Test”. This could be very useful when we are still developing and testing our application, as we may be frequently adding and deleting test users. Stormpath will also provide an
API Key Properties file.
Stormpath also allows us to define password strength requirements for each application, such as:
- Must have >= 8 characters.
- Must include lowercase and uppercase.
- Must include a number.
- Must include a non-alphabetic character
Stormpath keeps track of all of our users and assigns them API keys, which we can use for our REST API authentication. This greatly simplifies the task of building our application, as we don’t have to focus on writing code for authenticating users.
Node.js applications are built using many library modules and there is a very rich ecosystem of libraries available, some of which we will use to build our application.
To start using Node.js, we need to define a
package.json file describing our application and all of its library dependencies.
The Node.js Package Manager installs copies of the libraries in a subdirectory, called
node_modules/, in the application directory. This has benefits, as it isolates the library versions for each application and so avoids code compatibility problems if the libraries were to be installed in a standard system location, such as
/usr/lib, for example.
npm install will create the node_modules/ directory, with all of the required libraries.
Our application is called reader-api. The main file is called
server.js. Then we have a list of the dependent libraries and their versions. Some of these libraries are designed for parsing the HTTP queries. The test harness we will use is called
jasmine-nodeis used to run frisby scripts.
One library that is particularly important is
async. If you have never used node.js, it is important to understand that node.js is designed to be asynchronous. So, any function which does blocking input/output (I/O), such as reading from a socket or querying a database, will take a callback function as the last parameter, and then continue with the control flow, only returning to that callback function once the blocking operation has completed. Let’s look at the following simple example to demonstrate this.
In the above example, we may think that the output would be:
but in fact it might be:
because the line that prints “one” might happen later, asynchronously, in the callback. We say “might” because if conditions are just right, “one” might print before “two”. This element of uncertainty in asynchronous programming is called non-deterministic execution. For many programming tasks, this is actually desirable and permits high performance, but clearly there are times when we want to execute functions in a particular order. The following example shows how we could use the async library to achieve the desired result of printing the numbers in the correct order:
In the above code, we are guaranteed that
function two will only be called after
function one has completed.
Wrapping Up Part 1
Now that we have seen the basic mechanics of node.js and async function setup, we are ready to move on. Rather than move into creating the application, we will instead start by creating tests that validate the behavior of the application. This approach is called test-driven development and has two very good features:
- It helps the developer really understand how data and functions are consumed and often exposes subtle needs like the ability to return 2 or more things in an array instead of just one thing.
- By writing tests before building the application, the paradigm becomes “broken / unimplemented until proven tested OK” instead of “assumed to be working until a test fails.” The former is a “safer” way to keep the code healthy.
About the Author - Norberto Leite
Norberto Leite is Technical Evangelist at MongoDB. Norberto has been working for the last 5 years on large scalable and distributable application environments, both as advisor and engineer. Prior to MongoDB Norberto served as a Big Data Engineer at Telefonica.About the Author - Norbert