Monthly Archives: July 2015

ELK tutorial: part 3

Installing and configuring Logstash-Forwarder

On the central server, create a folder called ‘/etc/pki/tls/certs’ and ‘/etc/pki/tls/private’:

ubuntu@elk: sudo mkdir -p /etc/pki/tls/certs
ubuntu@elk: sudo mkdir /etc/pki/tls/private

Because logstash-forwarder is using an SSL connection to the ELK server, we will need to create a certificate. To do so, add the following to your ‘/etc/ssl/openssl.cfg’ file (most likely the v3_ca tag is already in the file):

[ v3_ca ]
subjectAltName =IP:

Then let’s go ahead and create the certificate as follows:

ubuntu@elk:/etc/pki/tls$ sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

You will see that a certificate is created and copied to ‘etc/pki/tls/certs’. Then obviously we will need to copy this certificate to the webserver that will communicate with our ELK server. Do this as follows:

ubuntu@elk:/etc/pki/tls/certs$ sudo scp -i keypair_ccs.pem logstash-forwarder.crt ubuntu@:/tmp

On each new server, we need to copy the certificate from the ‘/tmp’ folder to the ‘/etc/pki/tls/certs’ folder:

ubuntu@webserver:/tmp$ cp logstash-forwarder.crt /etc/pki/tls/certs

We also need to install logstash-forwarder of course:

ubuntu@webserver:/etc/pki/tls/certs$ echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list
deb http://packages.elasticsearch.org/logstashforwarder/debian stable main
ubuntu@webserver:/etc/pki/tls/certs$ wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
--2015-07-17 10:53:20--  http://packages.elasticsearch.org/GPG-KEY-elasticsearch
Resolving packages.elasticsearch.org (packages.elasticsearch.org)... 50.16.229.96, 23.23.156.102, 174.129.21.167, ...
Connecting to packages.elasticsearch.org (packages.elasticsearch.org)|50.16.229.96|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1768 (1.7K) [binary/octet-stream]
Saving to: 'STDOUT'

100%[==================================================================================================================>] 1,768       --.-K/s   in 0s      

2015-07-17 10:53:20 (391 MB/s) - written to stdout [1768/1768]

OK
ubuntu@webserver:/etc/pki/tls/certs$ sudo apt-get update
ubuntu@webserver:/etc/pki/tls/certs$ sudo apt-get install logstash-forwarder

...
Preparing to unpack .../logstash-forwarder_0.4.0_amd64.deb ...
Unpacking logstash-forwarder (0.4.0) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up logstash-forwarder (0.4.0) ...
 Adding system startup for /etc/init.d/logstash-forwarder ...
   /etc/rc0.d/K20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc1.d/K20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc6.d/K20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc2.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc3.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc4.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc5.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
Logs for logstash-forwarder will be in /var/log/logstash-forwarder/

We then need to modify the configuration file for logstash-forwarder:

ubuntu@webserver:/etc/pki/tls/certs$ sudo nano /etc/logstash-forwarder.conf

and copy/paste the following:

{
  "network": {
    "servers": [ "" ],
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/apache2/access.log"
      ]
    ,
      "fields": { "type": "apache-remote2" }
    }
   ]
}

Finally, let’s check if our logstash-forwarder has established a secure connection to our logstash server:

ubuntu@apachewebserver2:/var/www$ cd /var/log/logstash-forwarder/
ubuntu@apachewebserver2:/var/log/logstash-forwarder$ ls
logstash-forwarder.err  logstash-forwarder.log
ubuntu@apachewebserver2:/var/log/logstash-forwarder$ tail logstash-forwarder.err 


2015/07/17 10:59:00.213359 Waiting for 1 prospectors to initialise
2015/07/17 10:59:00.213415 Launching harvester on new file: /var/log/apache2/access.log
2015/07/17 10:59:00.213467 harvest: "/var/log/apache2/access.log" (offset snapshot:0)
2015/07/17 10:59:00.213512 All prospectors initialised with 0 states to persist
2015/07/17 10:59:00.213808 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/07/17 10:59:00.214301 Connecting to []:5000 () 
2015/07/17 10:59:00.309486 Connected to 
2015/07/17 10:59:05.250653 Registrar: processing 3 events
{
        "message" => "173.38.209.6 - - [17/Jul/2015:11:17:52 +0000] \"GET / HTTP/1.1\" 200 447 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:17:52.000Z",
           "type" => "apache-remote1",
           "file" => "/var/log/apache2/access.log",
           "host" => "apache-webserver1",
         "offset" => "627275",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:17:52 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "447",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""
}
{
        "message" => "173.38.209.6 - - [17/Jul/2015:11:17:56 +0000] \"GET / HTTP/1.1\" 200 448 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:17:56.000Z",
           "type" => "apache-remote2",
           "file" => "/var/log/apache2/access.log",
           "host" => "apachewebserver2",
         "offset" => "828",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:17:56 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "448",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""
}

When you now look to Kibana, you will …

Kibana

ELK tutorial: part 2

Introduction

In part 1, we installed the ELK stack. This part will focus on the configuration of it. We’ll keep it rather simple for now. We will process a local Apache access log file and visualize it using Kibana. Let’s get started!

Configuring Logstash

Logstash needs a configuration file.

input { stdin { } }
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

In the input folder, you will see that we define a stdin() function, which will allow Logstash to parse content supplied via the command line. As we want to parse Apache access log entries, we will also use a filter that is able to parse the Apache log format. Luckily this comes out of the box with a Grok filter ‘COMBINEDAPACHELOG’. You can read here what it exactly does:

Next, we need to run Logstash using this configuration file. You do this as follows:

ubuntu@elk:~/logstash-1.5.2$ sudo bin/logstash -f logstash-stdin.conf 
...
Logstash startup completed

If all goes well, you should see the ‘Logstash startup completed’ message appear. If there is an error in your configuration file, Logstash will not hesitate to inform you.

If you now supply the following text to Logstash:

173.38.209.6 - - [17/Jul/2015:06:33:11 +0000] "GET / HTTP/1.1" 200 448 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36"

you will see that Logstash filters it and creates the following:

{
        "message" => "173.38.209.6 - - [17/Jul/2015:06:33:11 +0000] \"GET / HTTP/1.1\" 200 448 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T06:33:11.000Z",
           "host" => "elk",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:06:33:11 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "448",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""

This indicates that everything is working just fine. The only problem is that this is not very useful right? Let’s continue to parse a real Apache access log file. As you could have figured out probably, we can achieve this by changing the configuration file for Logstash:

input {
  file {
    path => "/var/log/apache2/access.log"
    type => "apache-access"
    start_position => "beginning"
  }
}
filter {
  if [type] == "apache-access" {
    grok {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch {
    host => localhost
  }
  stdout { codec => rubydebug }
}
ubuntu@elk:~/logstash-1.5.2$

Here you can see that we monitor the Apache access log file in the folder ‘/var/log/apache2/access.log’. We expect that each time we retrieve an HTML file (or object in general), we will this updated in Kibana. Let’s see what happens…

Don’t forget to run Logstash with this new configuration file:

ubuntu@elk:~/logstash-1.5.2$ sudo bin/logstash -f logstash-access.conf 

The moment Logstash is started, you will see messages appearing in your console:

{
        "message" => "187.102.35.62 - - [17/Jul/2015:11:23:07 +0000] \"GET / HTTP/1.1\" 200 11819 \"-\" \"x00_-gawa.sa.pilipinas.2015\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:23:07.000Z",
           "host" => "elk",
           "path" => "/var/log/apache2/access.log",
           "type" => "apache-access",
       "clientip" => "187.102.35.62",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:23:07 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "11819",
       "referrer" => "\"-\"",
          "agent" => "\"x00_-gawa.sa.pilipinas.2015\""
}
{
        "message" => "173.38.209.6 - - [17/Jul/2015:11:48:22 +0000] \"GET / HTTP/1.1\" 200 3594 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:48:22.000Z",
           "host" => "elk",
           "path" => "/var/log/apache2/access.log",
           "type" => "apache-access",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:48:22 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "3594",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""

Showing something in Kibana

First things first, we left off part 1 with a default Kibana install but it didn’t do a lot yet.

kibana-default

Just take the default settings (as shown in previous picture) for now to create the index. You will get the following:

Kibana-index
And index has now been configured. If you want, you could then go and hit the ‘Discover’ tab. It should not show a lot yet.

When I hit the webpage, I found following entry in my console:

{
        "message" => "173.38.209.6 - - [17/Jul/2015:12:30:20 +0000] \"GET /icons/ubuntu-logo.png HTTP/1.1\" 304 179 \"http://173.39.243.147/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T12:30:20.000Z",
           "host" => "elk",
           "path" => "/var/log/apache2/access.log",
           "type" => "apache-access",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:12:30:20 +0000",
           "verb" => "GET",
        "request" => "/icons/ubuntu-logo.png",
    "httpversion" => "1.1",
       "response" => "304",
          "bytes" => "179",
       "referrer" => "\"http://173.39.243.147/\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""
}

Let’s now see if this also appears in Kibana. If you doubt this is real, check the date and time of the JSON message above and validate it with the picture below 🙂
Kibana_apache_access

This concludes part 2.

ELK tutorial: part 1

[keen]

Introduction

I have recently been playing around with Elastisearch, Logstash and Kibana, often referred to as the ELK stack.

Installing ELK

Installing ELK is rather simple. To do so, go to the Elastic website (here) and download each package. As I’m running on Ubuntu (fresh install on Openstack), I’m downloading the equivalent packages for each service:

ubuntu@elk:~$ sudo apt-get update
ubuntu@elk:~$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.tar.gz
ubuntu@elk:~$ wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
ubuntu@elk:~$ wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz

Next, move ahead on untar these packages:

ubuntu@elk:~$ tar xvzf elasticsearch-1.6.0.tar.gz
ubuntu@elk:~$ tar xvzf logstash-1.5.2.tar.gz
ubuntu@elk:~$ tar xvzf kibana-4.1.1-linux-x64.tar.gz 

When all is well, you’ll see the packages in your directory.

ubuntu@elk:~$ ls -l
total 16
drwxrwxr-x 7 ubuntu ubuntu 4096 Jul 16 15:59 elasticsearch-1.6.0
drwxrwxr-x 7 ubuntu ubuntu 4096 Jun 29 18:07 kibana-4.1.1-linux-x64
drwxrwxr-x 5 ubuntu ubuntu 4096 Jul 17 10:21 logstash-1.5.2

Don’t forget to also install Java in case you haven’t done so:

ubuntu@elk:~$ sudo add-apt-repository -y ppa:webupd8team/java
ubuntu@elk:~$ sudo apt-get update
ubuntu@elk:~$ sudo apt-get -y install oracle-java8-installer

In order to make things a bit easier later on, let’s also create a startup script:

ubuntu@elk:~$ cat startup.sh 
echo "----Starting ElasticSearch----"
cd ~/elasticsearch-1.6.0/
./bin/elasticsearch &

echo "----Starting Kibana----"
cd ~/kibana-4.1.1-linux-x64/
./bin/kibana &

echo "----Starting Logstash----"
cd ~/logstash-1.5.2/
./bin/logstash -f logstash-access.conf 

echo "----Finished booting the ELK stack"

The script obviously is not strictly necessary and can probably be improved dramatically, but for now it does the trick quite well.

If all goes well, you will see messages appearing like below:

ubuntu@elk:~/elasticsearch-1.6.0$ sudo ./bin/elasticsearch
[2015-07-17 11:46:42,569][INFO ][node                     ] [Mindworm] version[1.6.0], pid[3797], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-07-17 11:46:42,577][INFO ][node                     ] [Mindworm] initializing ...
[2015-07-17 11:46:42,581][INFO ][plugins                  ] [Mindworm] loaded [], sites []
[2015-07-17 11:46:42,664][INFO ][env                      ] [Mindworm] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [45.5gb], net total_space [49.1gb], types [ext4]
[2015-07-17 11:46:46,591][INFO ][node                     ] [Mindworm] initialized
[2015-07-17 11:46:46,592][INFO ][node                     ] [Mindworm] starting ...
[2015-07-17 11:46:46,876][INFO ][transport                ] [Mindworm] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/173.39.243.147:9300]}
[2015-07-17 11:46:46,929][INFO ][discovery                ] [Mindworm] elasticsearch/pQUGKeJCT4CbNxjWZK33Jw
[2015-07-17 11:46:50,745][INFO ][cluster.service          ] [Mindworm] new_master [Mindworm][pQUGKeJCT4CbNxjWZK33Jw][elk][inet[/173.39.243.147:9300]], reason: zen-disco-join (elected_as_master)
[2015-07-17 11:46:50,811][INFO ][http                     ] [Mindworm] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/173.39.243.147:9200]}
....

Let’s now also test if ElasticSearch is really running and available. Go to http://:9200. You will see something similar like below picture:

ElasticSearch

In order to reach Kibana, you’ll go to http://:5601. You will find a webpage with the Kibana logo which means you’re good to go! Note that you will have to configure an index first, but we’ll cover that in part 2 of this series.

kibana-default

You have now a full ELK stack up and running. In part 2 of this tutorial, we are going to configure ELK to monitor apache access log files and visualize then using Kibana.

REST API Server with Node (Express) and Mongodb on Heroku

In the previous post, we created a simple API to add, retrieve, update and delete todo items to a Mongodb database. The issue is that everything is running on localhost, which is good for development purposes but doesn’t bring us far. It would be good to have the server API running somewhere in the cloud. Hence, in this post, I will basically use the application we created back then and run it from Heroku.

To follow along, I suggest to use git clone command to clone the app in your own directory and start from there.

wim@ubuntu:~/Blog$ git clone https://github.com/wiwa1978/blog-express-todo-mongo-api Express_Todo_Mongo_API_Heroku

‘Express_Todo_Mongo_API_Heroku’ is just the directory to which we will clone the repository.

In order to run the application on Heroku, we need to add a Procfile with the following content:

web: node bin/www

Next, we need to create an Heroku application. We choose the name ‘express-todo-api-server’, so our app will be available on https://express-todo-api-server.herokuapp.com

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ heroku create express-todo-api-server
Creating express-todo-api-server... done, stack is cedar-14
https://express-todo-api-server.herokuapp.com/ | https://git.heroku.com/express-todo-api-server.git

Next, we also will need to have a Mongo database in the cloud. Heroku works with Mongolabs, which is very easy to setup and use. First, add it to the Heroku application.

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ heroku addons:add mongolab --app express-todo-api-server
Adding mongolab on express-todo-api-server... done, v3 (free)
Welcome to MongoLab.  Your new subscription is being created and will be available shortly.  Please consult the MongoLab Add-on Admin UI to check on its progress.
Use `heroku addons:docs mongolab` to view documentation.

Before we can push the code to Heroku, we need to ensure that we are using this Mongolab connection rather than our local Mongodb installation. In order to do see, change the connection settings in the config/database.js file from:

module.exports = { url : ‘mongodb://localhost/todo’ }

to

module.exports = {
	url : process.env.MONGOLAB_URI
}

Note: don’t put quotes (‘ ‘) around the environment variable!!

I struggled a lot with the default view engine. As we want to create an API service only, we should not define a view engine like Jade. In the app.js file, there is a section like below:

app.use(function(err, req, res, next) {
  res.status(err.status || 500);
  res.render('error', {
    message: err.message,
    error: {}
  });
});

Change this to:

app.use(function(err, req, res, next){
    res.status(err.status || 500);
    res.send({
        message: err.message,
        error: err
    });
    return;
});

What the above does is actually quite simple. Instead of rendering the error view, it will just send back a json message with the error message. I have to admit that I’m still not satified completely as the ideal would be that I don’t have to specify a default template engine at all. In the end, it does not make sense to specify a view template if all we want to do is to expose a REST API. I’ll have a look into this in the future. I consider it less relevant for now as it works perfectly well now…

Deploying to Heroku

I always post the code to Github and then also use Git to deploy to heroku. I’ll give an example on how to do this in below snippet. First I will cover how to upload the code to Github:

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ echo "# Express_Todo_Mongo_API_Heroku" >> README.md
wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ git init
wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ git add .
wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ git commit -m "First commit"
wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ git remote add origin https://github.com/wiwa1978/blog-express-todo-mongo-api-heroku
wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Heroku$ git push -u origin master

Next, I’ll show how to upload the code to Heroku:

wim@ubuntu:~/Dropbox/Programming/Blog/Express_Todo_Mongo_API_Heroku$ heroku git:remote -a express-todo-api-server
wim@ubuntu:~/Dropbox/Programming/Blog/Express_Todo_Mongo_API_Heroku$ git push heroku master
Counting objects: 1370, done.
Compressing objects: 100% (1251/1251), done.
Writing objects: 100% (1370/1370), 1.99 MiB | 234.00 KiB/s, done.
Total 1370 (delta 152), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Node.js app detected
remote: 
remote: -----> Creating runtime environment
remote:        
remote:        NPM_CONFIG_LOGLEVEL=error
....
remote: -----> Build succeeded!
remote:        ├── body-parser@1.12.4
remote:        ├── cookie-parser@1.3.5
remote:        ├── debug@2.2.0
remote:        ├── express@4.12.4
remote:        ├── jade@1.9.2
remote:        ├── method-override@2.3.3
remote:        ├── mongoose@3.8.31
remote:        ├── morgan@1.5.3
remote:        └── serve-favicon@2.2.1
remote:        
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing... done, 11.6MB
remote: -----> Launching... done, v4
remote:        https://express-todo-api-server.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy.... done.
To https://git.heroku.com/express-todo-api-server.git
 * [new branch]      master -> master

I always check the heroku logs to see if my application is up or if it has crashed.

2015-07-03T15:24:21.530494+00:00 heroku[api]: Release v12 created by wauterw@gmail.com
2015-07-03T15:24:21.840320+00:00 heroku[web.1]: State changed from up to starting
2015-07-03T15:24:23.016169+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2015-07-03T15:24:23.940442+00:00 heroku[web.1]: Process exited with status 143
2015-07-03T15:24:24.279515+00:00 heroku[web.1]: Starting process with command `node bin/www`
2015-07-03T15:24:26.453511+00:00 app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
2015-07-03T15:24:26.453541+00:00 app[web.1]: Recommending WEB_CONCURRENCY=1
2015-07-03T15:24:27.619823+00:00 app[web.1]: Listening on port 8148
2015-07-03T15:24:28.159017+00:00 heroku[web.1]: State changed from starting to up

The last sentence is what we need to have. Good, seems like everything is running fine. Let’s verify it with POSTMAN, the ultimate REST client.

First, let’s see all todo items. Obviously, we will receive an empty Json body, indicated with []:

1.GetAllItems

Then, let us add a todo item:

2.PostItem

Let’s ensure it has been added successfully:

3.GetAllItens

Why not update the item:

4.Updated

Check if it has been updated successfully:

5.GetItemAfterUpdate

And finally, delete the item again:

6.Deleted

Full source code as it is deployed to Heroku can be found on Github. Should be ready for you guys to clone and test it. Obviously, use your own Heroku environment to test out things. I’m using https://express-todo-api-server.herokuapp.com/ on Heroku.

Consuming Express (NodeJS) REST API with Angular

In our previous post, we had developed a simple Jade application to consume the REST API service we developed in this post.

Right now, we will basically do the same but we will use Angular instead of Jade. To get started, we will first clone the Github repository that contains the server code.

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Angular$ git clone https://github.com/wiwa1978/blog-express-todo-mongo-api Express_Todo_Mongo_API_Angular

Now, go to the directory that contains the code, in our case the directory is called ‘Express_Todo_Mongo_API_Angular’.

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Angular$ cd Express_Todo_Mongo_API_Angular && npm install

If all went well, you should be able to go to http://localhost:3000 and see a welcome page. That indicates that our server is running. So far, we did exactly the same as what we did in this post (the Jade variant).

Let’s now add the Angular specific code…
By default, Express is using Jade templates to create client code. When using Angular, we want to prevent that the Jade code is running. So therefore do the following in the app.js file:

var app = express();
// view engine setup
//app.set('views', path.join(__dirname, 'views'));
//app.set('view engine', 'jade');

As you can see, we have simply uncommented the lines that informs Express to use the Jade template. Also, Express won’t look for Jade files in the ‘views’ directory.
We will put the Angular client files in a folder called ‘public’. Obviously you are free to choose your own folder name as long as you inform Express about it. In the app.js file, you can do it as follows:

app.use(express.static(path.join(__dirname, 'public')));

Everything is now ready for the actual client code. As Angular is a client framework to build single page applications, we will put all code in a file called index.html in the ‘public’ folder we created.

The following snippet is adding Angular to the HTML code, refer to the ng-app=’MyApp’ reference which is part of the html tag. We have also included the reference to the Angular javascript file as well as a reference to a controller.js file which will contain all of our ‘todo item intelligence’. As I’m a big fan of Bootstrap, obviously I’ve included the stylesheet reference as well.

<!DOCTYPE>
<html ng-app='myApp'>
<head>
  <!-- Latest compiled and minified CSS -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">

  <!-- Optional theme -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap-theme.min.css">
  <title>Todo application</title>
</head>
<body>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.3.16/angular.min.js"></script>
<script src="controllers/controller.js"></script>

</body>
</html>

Let’s now have a look at the ‘‘ part. Essentially, we want to display a table that contains all todo items retrieved from the server.

<div class="container" ng-controller="AppCtrl">
  <h1>Todo Application</h1>
  <table class="table">
    <thead>
      <tr>
        <th>Content</th>
        <th>Description</th>
        <th>Status</th>
        <th>Created</th>
        <th>Updated</th>
        <th>Action</th>
      </tr>
    </thead>
    <tbody>
      <tr>
    
              <td><input class="form-control" ng-model="todo.content"></td>
              <td><input class="form-control" ng-model="todo.description"></td>
              <td>
                <div ng-if="todo.edit === true">
            <div class="form-group">
             <div class="radio">
               <label>
                 <input type="radio" name="true" value="true" ng-model="todo.completed">
                            Completed
               </label>
              </div>
              <div class="radio">
                 <label>
                   <input type="radio" name="false" value="false" ng-model="todo.completed">
                            Pending
                 </label>
               </div>
             </div>
        </div>  
            </td>
              <td></td>
              <td></td>
              <td>
                <div ng-if="todo.edit === true">
              <button class="btn btn-primary" ng-click="updateTodo()">Update Todo</button>
          </div>
          <div ng-if="todo.edit !== true">
              <button class="btn btn-primary" ng-click="addTodo()">Add Todo</button>
          </div>
              </td>
          </tr>
          <tr ng-repeat="todo in todolist">
        <td>{{todo.content}}</td>
        <td>{{todo.description}}</td>
        <td>
        <div ng-if="todo.completed === true">
              <span class="label label-success">Completed</span>
        </div>
        <div ng-if="todo.completed !== true">
              <span class="label label-warning">Pending</span>
        </div>
        </td>
        <td>{{todo.created_at| date:'dd-MM-yyyy -- HH:mm:ss'}}</td>
        <td>{{todo.updated_at| date:'dd-MM-yyyy -- HH:mm:ss'}}</td>
        <td>
          <button class="btn btn-success" ng-click="editTodo(todo._id)">Edit</button>
          <button class="btn btn-info" ng-click="showTodo(todo._id)">Show</button>
          <button class="btn btn-danger" ng-click="removeTodo(todo._id)">Remove</button>
        </td>
      </tr>
    </tbody>  
  </table>
</div>

In the above snippet, you see that ‘ng-controller=”AppCtrl”‘ binds the code to the controller. We also use an Angular ‘ng-repeat’ statement to loop over all the items and we use the {{}} directives to effectively display the content on screen. In the body part of the table, you can also see that we define two input fields that refer to an ‘ng-model’ and two buttons that refer to an ‘ng-click’ event, respectively addTodo() and updateTodo()

Let’s then have a look at the controller.js file.

var myApp = angular.module('myApp', []);
myApp.controller('AppCtrl', ['$scope', '$http', function($scope, $http) {
   
url = 'http://localhost:3000/todos/'

var refresh = function() {
  $http.get(url).success(function(response) {
    $scope.todolist = response;
    $scope.todo = "";
    $scope.todo.edit = false;
  });
};

In the above code, you’ll see that the myApp is defined in the controller file and that the controller ‘AppCtrl’ is also defined. We also define the $scope and $http variables. We need $scope to be able to use the info in the HTML file and the $http to be able to make the REST calls to the server.

We can also see that we do a HTTP GET call to the URL http://localhost:3000/todos, which is effectively our server (as we run everything on localhost for now). The JSON response from the server is stored in the $scope.todolist which is used in the ‘ng-repeat’ statement (cfr: ng-repeat=”todo in todolist”).

Let’s look at the addTodo(), updateTodo() and deleteTodo() functions:

scope.addTodo = function() {
  console.log($scope.todo);
  $http.post(url, $scope.todo).success(function(response) {
    console.log(response);
    refresh();
  });
};

$scope.removeTodo = function(id) {
  $http.delete(url + id).success(function(response) {
    console.log("deleting: " + response);
    refresh();
  });
};

$scope.updateTodo = function() {
  console.log("Completed" + $scope.todo.completed);
  $http.put(url + $scope.todo._id, $scope.todo).success(function(response) {
     console.log("new updated: " + response.updated_at);
    refresh();
 	})
};

In the addTodo() function, we see that we create an HTTP POST and pass it the $scope.todo, which essentially contains all the info from the input fields (in the todo.content and todo.description).

In the UpdateTodo(), we create an HTTP PUT request to the URL and pass it again the $scope.todo parameter, which contains the updated todo items.

Finally, the removeTodo() is using an HTTP delete request to the URL which we pass the id of the todo item.

You can also see that each time the refresh function is called each time. The refresh function is wrapped around the HTTP GET method that retrieves all todo items, so in fact each time we add, update or delete a todo item, we retrieve the whole list of todo items again so the page is updated automatically.

Let’s test out couple of things:

Below screenshot shows how we add a todo item:

1.AddItem

Let’s verify if it has been added properly:
2.ItemAdded

And update the item we just added:
3.UpdateItem

And verify once again if the update was executed successfully:
4.ItemUpdated
Let’s add couple of items more:
5.AddingMultipleItems
And then delete Todo item 2 using the delete button:
6.RemoveItem2
OK perfect. Looks like this Angular code is running just fine!

The code can be found on Github as usual.

Consuming Express (NodeJS) REST API with Jade

In this post, we created a REST API server that allows a client to add, update, delete and show todo items. One could use a REST client, such as POSTMAN, to consume the REST API services exposed by the REST server developed in Express (NodeJS).

In this post, we are going to clone the repository from Github and modify it to include some user interfaces by using the Jade template which comes as part of the Express framework. Let’s get started….

wim@ubuntu:~/Blog$ git clone https://github.com/wiwa1978/blog-express-todo-mongo-api.git Express_Todo_Mongo_API_Jade

The github repository is now cloned in a folder called ‘Express_Todo_Mongo_API_Jade’. To ensure everything is OK, let’s try to run the application.

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Jade$ node bin/www 

Going to http://localhost:3000 will show you the welcome page of the Express generator.

Let’s now start the real work and add some Jade templates to be able to view, create, update and delete todo items. To do so, we will create a subdirectory called ‘todos’ in the ‘views’ directory. As we want a really nice layout, we will first ensure that we can use the Bootstrap framework. To do this, add a reference to the Bootstrap framework in the layout.jade file. The complete content of this file is in below snippet.

doctype html
html
    title Express Todo
    link(rel="stylesheet", href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css")
body
    block content

Then, in the newly created todos folder, we will create a page called ‘index.jade’. This file will show all todo items currently in our Mongo database.

extends ../layout

block content
  div.container
    h2 Todo Application
    div.content
      table.table
        a.btn.btn-primary( href="/todos/new" ) Add Todo
        thead
          tr
            th Content
            th Description
            th Status
            th Created
            th Updated
            th Actions
        tbody
          - each todo, i in todos  
            tr
              td #{todo.content}
              td #{todo.description}
              #{todo.completed}
              if todo.completed
                td Completed
              else
                td Pending
              td #{todo.created_at)
              td #{todo.updated_at}
              td 
                a.btn.btn-success( href="/todos/#{todo._id}/edit" ) Edit  
                   
                a.btn.btn-info( href="/todos/#{todo._id}/show" ) Show
                   
                form(action='/todos/#{todo._id}/delete',method='post',enctype='application/x-www-form-urlencoded')
                input(type='hidden',value='DELETE',name='_method')
                button.btn.btn-danger(type='submit') Delete
                p 

In the above snippet, you can see that we fill a Bootstrap table by looping over all todo-items using a for each loop. For each todo item, we print the ‘content’, ‘description’, ‘completed status’, ‘created_at’ and ‘updated_at’ date. Here and there, we applied some more Bootstrap functionality (cfr the span class to indicate a ‘completed’ or ‘pending’ state). You can view the output by going to http://localhost:3000/todos.

It looks as follows:

Todo_Index

You’ll see that the ‘created_at’ and ‘updated_at’ dates are looking ugly. We will use moment.js to format them to an appropriate format.

wim@ubuntu:~/Blog/Express_Todo_Mongo_API_Jade$ sudo npm install moment --save
moment@2.10.3 node_modules/moment

Then we need to ensure that our Express app knows about the momentjs library.

var moment = require('moment');
app.locals.moment = require('moment');

Once this is done, we can use the momentjs functionality to give the dates a more usable format. To do this, replace the following code in ‘todos/index.js’

  td #{todo.created_at}
  td #{todo.updated_at}

by this code:

td #{moment(todo.created_at).format('DD/MM/YYYY -- HH:mm:ss')}
td #{moment(todo.updated_at).format('DD/MM/YYYY -- HH:mm:ss')}

The final result now looks like:

Momentjs

Let’s now continue to add some more functionality, e.g add todo items to the Mongo database:

extends ../layout
block content
  div.container
    h2 Add Todo item

    div.content
    form#formAddTodo(name="addtodo",method="post",action="/todos")
        p
            input#content.form-control(type='content', name='content', placeholder='Content')
        p
            input#description.form-control(type='description', name='description', placeholder='Description')
        p
            .checkbox
                label
                    input(type='checkbox', name='completed')
                    |  Completed
        p
            button.btn.btn-primary(type="submit") Add Todo

Easy enough…you add a form, provide the text input fields for the content, the description and a checkbox for the completed item and a button that effectively adds the item to the Mongo database. Looking behind the curtains, you will hit the server with an HTTP POST to http://localhost:3000/todos.

I refer to the Github repository to see the Jade files for the update and delete functionality.

We will now test the webapplication a little bit using POSTMAN and the web interface:
We will start with an empty todo list as shown in the following screenshot:

1.EmptyTodoList

Then we will add a single Todo item using the webinterface:

2.AddItem

Using the webinterface, we ensure that the item has been properly added to the Mongo database:

3.OverviewAllItems

Obviously, also using POSTMAN will give us the same Todo item:

4.Postman_AllItems

Then, let’s try to update the todo item:

5.UpdateBox

Again, we verify using the webapplication if it was updated successfully:

6.Updated_AllItems

This is obviously also reflected in POSTMAN:

8.Postman_AllItemsAfterUpdate

Deleting a todo item using the webapplication also works. To verify the deletion was successful, you could do again a POSTMAN query as indicated in the below screenshot:

9.Postman_afterDelete

Please refer to the Github repository for the full source code.

Have fun!!

REST API Server with Node (Express) and Mongo

In previous posts, I have been using Sinatra and Flask to create a REST API for a simple todo application. Nodejs has been on my todo list for quite some time so I decided to get to create the same todo-application again in NodeJS (with the Express JS framework). Have fun following this tutorial.

Luckily, the Express framework allows us to use an Express generator, which basically will create a working Express application. Use the following command:

wim@ubuntu:~/Blog$ express Express_Todo_Mongo_API

   create : Express_Todo_Mongo_API
   create : Express_Todo_Mongo_API/package.json
   create : Express_Todo_Mongo_API/app.js
   create : Express_Todo_Mongo_API/public
   create : Express_Todo_Mongo_API/public/javascripts
   create : Express_Todo_Mongo_API/public/images
   create : Express_Todo_Mongo_API/public/stylesheets
   create : Express_Todo_Mongo_API/public/stylesheets/style.css
   create : Express_Todo_Mongo_API/routes
   create : Express_Todo_Mongo_API/routes/index.js
   create : Express_Todo_Mongo_API/routes/users.js
   create : Express_Todo_Mongo_API/views
   create : Express_Todo_Mongo_API/views/index.jade
   create : Express_Todo_Mongo_API/views/layout.jade
   create : Express_Todo_Mongo_API/views/error.jade
   create : Express_Todo_Mongo_API/bin
   create : Express_Todo_Mongo_API/bin/www

   install dependencies:
     $ cd Express_Todo_Mongo_API && npm install

   run the app:
     $ DEBUG=Express_Todo_Mongo_API:* npm start

wim@ubuntu:~/Blog$ cd Express_Todo_Mongo_API && npm install
cookie-parser@1.3.5 node_modules/cookie-parser
├── cookie@0.1.3
└── cookie-signature@1.0.6

debug@2.2.0 node_modules/debug
└── ms@0.7.1

serve-favicon@2.2.1 node_modules/serve-favicon
├── fresh@0.2.4
├── ms@0.7.1
├── parseurl@1.3.0
└── etag@1.6.0 (crc@3.2.1)

morgan@1.5.3 node_modules/morgan
├── basic-auth@1.0.3
├── depd@1.0.1
└── on-finished@2.2.1 (ee-first@1.1.0)

body-parser@1.12.4 node_modules/body-parser
├── content-type@1.0.1
├── bytes@1.0.0
├── depd@1.0.1
├── raw-body@2.0.2 (bytes@2.1.0)
├── qs@2.4.2
├── on-finished@2.2.1 (ee-first@1.1.0)
├── type-is@1.6.4 (media-typer@0.3.0, mime-types@2.1.2)
└── iconv-lite@0.4.8

express@4.12.4 node_modules/express
├── merge-descriptors@1.0.0
├── utils-merge@1.0.0
├── cookie-signature@1.0.6
├── methods@1.1.1
├── fresh@0.2.4
├── cookie@0.1.2
├── escape-html@1.0.1
├── range-parser@1.0.2
├── content-type@1.0.1
├── finalhandler@0.3.6
├── vary@1.0.0
├── parseurl@1.3.0
├── serve-static@1.9.3
├── content-disposition@0.5.0
├── path-to-regexp@0.1.3
├── depd@1.0.1
├── on-finished@2.2.1 (ee-first@1.1.0)
├── qs@2.4.2
├── etag@1.6.0 (crc@3.2.1)
├── proxy-addr@1.0.8 (forwarded@0.1.0, ipaddr.js@1.0.1)
├── send@0.12.3 (destroy@1.0.3, ms@0.7.1, mime@1.3.4)
├── type-is@1.6.4 (media-typer@0.3.0, mime-types@2.1.2)
└── accepts@1.2.10 (negotiator@0.5.3, mime-types@2.1.2)

jade@1.9.2 node_modules/jade
├── character-parser@1.2.1
├── void-elements@2.0.1
├── commander@2.6.0
├── mkdirp@0.5.1 (minimist@0.0.8)
├── with@4.0.3 (acorn-globals@1.0.4, acorn@1.2.2)
├── constantinople@3.0.1 (acorn-globals@1.0.4)
└── transformers@2.1.0 (css@1.0.8, promise@2.0.0, uglify-js@2.2.5)

We can then start the node app generated by the Express generator:

wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ node bin/www

Going to http://localhost:3000 will show you a welcome message from the Express installer. Perfect, we have generated an Express application upon which we can continue to build further.

Let’s continue with the database section. As the database we will use is Mongo, we will also need to install a package to be able to work with Mongo. My preference goes to Mongoose, which is an object modeling tool which allows us to easily interact with Mongo from our Express code.. Let’s install the mongoose package using ‘npm install mongoose’.

Let’s now go ahead and change some code. You’ll see that the boilerplate code is referring to a users route file. You can see this by looking at the app.js file where it mentions ‘var users = require(‘./routes/users’);’. As we’d rather use todo items instead of users, we will make some changes:

We will add some references in the app.js file to ensure our Express app uses the Mongoose model and the correct Mongo database:

var routes = require('./routes/index');
var users = require('./routes/users');

app.use('/', routes);
app.use('/users', users);

to

var index = require('./routes/index');
var todo = require('./routes/todo');

app.use('/', index);
app.use('/todos', todo);

We will rename the users.js route file in the ‘routes’ folder to ‘todos.js’. In the app.js file we will need to add a variable todos that refers to the ‘./routes/todos’ file. Just to be on the safe side, let’s ensure that the app is still working by launching it:

wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ node bin/www
GET / 304 786.113 ms - -
GET /stylesheets/style.css 200 30.333 ms - 110

OK, so everything still works. Good! Let’s continue…

We will now add the Mongo/Mongoose references to app.js.

var mongoose = require('mongoose'); 
var database = require('./config/database'); 
mongoose.connect(database.url); 

In the database.js file under config we just place the URL to our database. The contents of the database.js file is

module.exports = {
	url : 'mongodb://localhost/todo'
}

OK, we are now ready with the basic changes to the boilerplate code. Let’s know focus on adding the functionality to our todo app.

The first thing to do is to define the Schema that we will be using. Our app basically needs to be told what the fields are to be used in the Mongo database. To achieve that, create a folder called ‘models’ in the root of your application and make a file called ‘todo.js’ in that folder.

var mongoose = require('mongoose');

// todo model
var todoSchema = new mongoose.Schema({
    content: String,
    description:String,
    completed: { type: Boolean, default: false },
    created_at: { type: Date, default: Date.now },
    updated_at: { type: Date, default: Date.now }
});

module.exports = mongoose.model('Todo', todoSchema);

In the above snippet, you can see that we want to store the content, description to the Mongo database, but also want to be able to indicate if a todo item has been completed or not.

Next, we will add the routes to the ‘routes/todo.js’ file. We will explain the example to get the list of all todo items. Note: don’t forget to refer to the database model from the route file. In other words, ensure you are putting the following line at the top of the ‘route/todo.js file’:

var Todo = require('../models/todo');

Here is the code to define a route to retrieve all todo items:

router.route('/')
    //GET all todos
    .get(function(req, res, next) {
        Todo.find({}, function (err, todos) {
              if (err) {
                  return console.error(err);
              } else {
              		console.log("Showing all todo items");
              		res.format({
                    html: function(){
                        res.render('todos/index', {
                            title: 'All my todos', 
                            "todos" : todos
                        });
                    },
                    json: function(){
                        res.json(todos);
                    }
                });
              }     
        });
    })

Here you can see that we define an HTTP GET function that will retrieve all todo items from the Mongo database. We also return an HTML section and a JSON section. Strictly speaking, given the fact we are only developing the REST API server, we don’t need to return the HTML section. Having said that, we’ll leave it in for future use.

In the previous snippet, we have only shown the ‘easy’ part, which was listing all todo items from the database. To avoid that this post is going to be extremely long, I have added -just for informational purposes- the entire content of the todo.js file:

var express = require('express');
var router = express.Router();
var Todo = require('../models/todo');

// GET New Todo page
router.get('/new', function(req, res) {
	console.log("Show page to create new todo item");
    res.render('todos/new', { title: 'Add New Todo' });
});

// route middleware to validate :id
router.param('id', function(req, res, next, id) {
    Todo.findById(id, function (err, todo) {
        //if it isn't found, we are going to repond with 404
        if (err) {
            console.log("Todo item with " + id + " was not found in the database");
            res.status(404)
            var err = new Error('Not Found');
            err.status = 404;
            res.format({
                html: function(){
                    next(err);
                },
                json: function(){
                    res.json({message : err.status  + " " + err});
                }
            });
        //if it is found we continue on
        } else {
            console.log(todo);
            req.id = id;
            next(); 
        } 
    });
});

// -----------------------------------------------------------------------------------------
// Matches routes without identifiers

router.route('/')
    //GET all todos
    .get(function(req, res, next) {
        Todo.find({}, function (err, todos) {
              if (err) {
                  return console.error(err);
              } else {
              		console.log("Showing all todo items");
              		res.format({
                    html: function(){
                        res.render('todos/index', {
                            title: 'All my todos', 
                            "todos" : todos
                        });
                    },
                    json: function(){
                        res.json(todos);
                    }
                });
              }     
        });
    })

	//POST a new todo item
    .post(function(req, res) {
		var content = req.body.content;
        var completed = false;
        var description = req.body.description;


        Todo.create({
            content : content,
            completed : completed,
            description : description,
           
        }, function (err, todo) {
            if (err) {
            	res.send("Todo item was not created succesfully");
                console.log("Todo item was not created succesfully");
            } 
            else {
                console.log('Created new todo item: ' + todo);
                res.format({
					html: function(){
                        res.location("todos");
                        res.redirect("/todos");
                    },
                    json: function(){
                        res.json(todo);
                    }
                });
              }
        })
    });

// -----------------------------------------------------------------------------------------
// Matches routes with identifiers

/* SHOW single todo item */
router.route('/:id/show')
	.get(function(req, res) {
    	Todo.findById(req.id, function (err, todo) {
	      	if (err) {
	        	console.log('Todo item with id ' + todo._id + ' could not be found ' + err);
	        	res.send('Todo item with id ' + todo._id + ' could not be found ' + err);
	      	} 
	      	else {
	        	console.log('Show todo item with id ' + todo._id);
	        	res.format({
		        	html: function(){
		             	res.render('todos/show', {
		                	"todo" : todo
		              	});
		          	},
		          	json: function(){
		              	res.json(todo);
		          	}
	        	});
	      	}
    	});
  	});

/* EDIT single todo item */
router.route('/:id/')
	//GET single todo item
	.get(function(req, res) {
	    Todo.findById(req.id, function (err, todo) {
	        if (err) {
	            console.log('Todo item could not be found ' + err);
	        } 
	        else {
	        	console.log('Edit todo item with id ' + todo._id);
              	res.format({
	                html: function(){
	                    res.render('todos/edit', {
	                        title: 'Todo' + todo._id,
                            "todo" : todo
	                    });
	                },
	                json: function(){
	                    res.json(todo);
	                }
	            });
	        }
	    });
	})

	//PUT to update todo item
	.put(function(req, res) {
	    var content = req.body.content;
	    var description = req.body.description;
	    var completed = req.body.completed;
	    var updated_at = new Date();
	    console.log("New time is " + updated_at);
	   
	    Todo.findById(req.id, function (err, todo) {
	        todo.update({
	            content : content,
	            description : description,
	            completed : completed,
	            updated_at : updated_at,
	           
	        }, function (err, todoID) {
	        	if (err) {
	        		console.log('Todo item could not be found ' + err);
	            	res.send('Todo item could not be found ' + err);
	          	} 
	          	else {
	          		console.log('Updated todo item with id ' + todo._id);
	                res.format({
	                    html: function(){
	                        res.redirect("/todos");
	                    },
	                    //JSON responds showing the updated values
	                    json: function(){
	                        res.json(todo);
	                    }
	                });
	           	}
	        })
	    });
	})

	//DELETE a todo item
	.delete(function (req, res){
	    Todo.findById(req.id, function (err, todo) {
	        if (err) {
	            console.log('Todo item could not be found ' + err);
	            res.send('Todo item could not be found ' + err);
	        } 
	        else {
	            todo.remove(function (err, todo) {
	                if (err) {
	                    console.log('Todo item could not be deleted ' + err);
	            		res.send('Todo item could not be deleted ' + err);
	                } 
	                else {
	                    console.log('Deleted todo item with id ' + todo._id);
	                    res.format({
	                        html: function(){
	                            res.redirect("/todos");
	                        },
	                        //JSON returns the item with the message that is has been deleted
	                        json: function(){
	                            res.json({message : 'deleted',
	                                item : todo
	                            });
	                        }
	                    });
	                }
	            });
	        }
	    });
	});


module.exports = router;

We have now created a complete CRUD application that exposes itself as a REST Server that will return Json objects. We are now completely ready for testing the API.

First off, we want to show all items currently in the Mongo database. Obviously, this should be empty as we’ve been developing from scratch:
GetAllItems_empty

Let’s add a todo item to the database:

2.Post_Todo_item

Let’s then be really sure it was added properly:

3.RetrieveAllItems

You will see that the todo item was properly added to the database. In order to only view that single item, refer to the following screenshot:

5.ShowItem

Let’s also update the item now:

4.UpdateItem

And finally also delete it:
6.DeleteItem

Let’s ensure the item was properly deleted:
8.getitems

Adding the source code to Github

Let’s now add everything to Github. First create a new repository on Github. I typically do this through the webinterface. Then, continue to add the source code to the repository using following steps.

wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ git init
wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ echo "# Express_Todo_Mongo_API" >> README.md
wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ git add .
wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ git commit -m "First Commit"
wim@ubuntu:~/Blog/Express_Todo_Mongo_API$ git remote add origin https://github.com/wiwa1978/blog-express_todo_mongo_api.git
wim@ubuntu:~/Dropbox/Programming/Blog/Express_Todo_Mongo_API$ git push -u origin master

Please refer to the link here to see the final result.