GlusterFS installation and configuration on CentOS 7 / RHEL 7

In this tutorial, We will see GlusterFS installation and configuration. How to configure GlusterFS server and GlusterFS client? How to test GlusterFS High availability? These all points we will cover in this tutorial, we will also proceed step by step installation and configuration of all these points.

What is GlusterFS?

GlusterFS is an open-source, scalable network file system suitable for high data-intensive workloads such as media streaming, cloud storage, and CDN (Content Delivery Network). In gluster, client server can access the storage as like local storage. Whenever user creates the data on gluster storage, then data will be mirrored and distributed to other storage nodes. The primary purpose of GlusterFS is data high availability accessible to the applications or users.

For more about GlusterFS installation and configuration you can Click Here

Terminologies:-

These are the important terminologies that we are going to use throughout this article.

Brick:– Brick is basic storage (directory) on a server in the trusted storage pool. Like /test/glusterfs

Volume:- Volume is a logical collection of bricks.

Cluster:- Cluster is a group of computers which are linked between all computers, and working together as a single computer. If one computer goes down then second computer will handle all services.

Distributed File System:- A file system in which the data is spread across the multiple storage nodes and allows the clients to access it over a network.

Client:- Client is a machine where our gluster file system will be mounts.

Server:- Server is a machine where the actual file system is hosted and all data will be stored.

Replicate:- Making multiple copies of data to achieve high redundancy.

Fuse:- Fuse is a loadable kernel module that lets non-privileged users create their own file systems without editing kernel code.

Glusterd:- Glusterd is a daemon that runs on all servers in the trusted storage pool.

Volumes:-

Volume is a collection of bricks, and most of the gluster operations such as reading and writing happen on the volume. Different types of volumes supported by GlusterFS based on the requirements.

In this article, we are going to configure a replicated GlusterFS volume on CentOS 7 / RHEL 7.

Replicated Glusterfs Volume is like a RAID 1, and volume maintains exact copies of the data on all bricks. We can decide the number of replicas while creating the volume, so we would need to have at least two bricks to create a volume with two replicas or three bricks to create a volume of 3 replicas.

Pre Request:-

Here, we are going to configure GlusterFS volume with two replicas. First of all you have to make sure you have two 64bit systems (either Virtual Machine or Physical Server) with minimum 1GB of memory, and one spare hard disk on each server.

Host Name IP Address OS   Memory Disk Purpose
urgluster1 192.168.43.20 CentOS 7   1GB /dev/sdb (2GB)   Storage Node 1
urgluster2 192.168.43.21 CentOS 7   1GB /dev/sdb (2GB)   Storage Node 2
urclient 192.168.43.22 CentOS 7    N/A N/A   Client Machine

Configure DNS:-

GlusterFS components are use DNS for name resolutions, so configure either DNS or we can set up a hosts entry like below. If we do not have a DNS on our environment. Here I am going to use /etc/hosts file to name resolution between our gluster server and gluster client machine.

[root@urgluster1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.43.20 urgluster1
192.168.43.21 urgluster2
192.168.43.22 urclient
[root@urgluster1 ~]#

Add GlusterFS Repository:-

First of all we need to configure GlusterFS repository on both storage nodes to install glusterfs packages. We can follow this instruction to add the repository to our system.

On RHEL 7:-

Add Gluster repository on RHEL 7.

[root@urgluster1 ~]# vi /etc/yum.repos.d/Gluster.repo

[gluster38]
name=Gluster 3.8
baseurl=http://mirror.centos.org/centos/7/storage/$basearch/gluster-3.8/
gpgcheck=0
enabled=1

On CentOS 7:-

Install centos-release-gluster package, it will provides us automatically required YUM repository files. This RPM is available from CentOS Extras.

[root@urgluster1 ~]# yum install -y centos-release-gluster

GlusterFS Installation on server side:-

Once we have added the repository on our systems, then we can go for GlusterFS installation. We can install GlusterFS package using below command.

[root@urgluster1 ~]# yum install -y glusterfs-server

Start the glusterd service on all gluster nodes after installation.

[root@urgluster1 ~]# systemctl start glusterd

Verify that the glusterfs service is running fine or not using below command.

[root@urgluster1 ~]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-09-08 19:12:48 CEST; 46s ago
  Process: 2628 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2629 (glusterd)
   CGroup: /system.slice/glusterd.service
           └─2629 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Sep 08 19:12:48 urgluster1 systemd[1]: Starting GlusterFS, a clustered file-system server...
Sep 08 19:12:48 urgluster1 systemd[1]: Started GlusterFS, a clustered file-system server.
[root@urgluster1 ~]#

Enable glusterd service to start automatically on system boot.

[root@urgluster1 ~]# systemctl enable glusterd
Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
[root@urgluster1 ~]#

Configure Firewall:-

We need to configure firewall to access properly our glusterfs services between server and client. We can either disable the firewall or configure the firewall to allow all connections within a cluster.

By default, glusterd will listen on TCP/24007 but opening that port is not enough on the gluster nodes. Each time we will add a brick, it will open a new port (that we will be able to see with “gluster volumes status”)

# Disable FirewallD
[root@urgluster1 ~]# systemctl stop firewalld
[root@urgluster1 ~]# systemctl disable firewalld

OR

We can run this below command on a node in which we want to accept all traffics coming from the source IP.

[root@urgluster1 ~]# firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.43.21" accept'
success
[root@urgluster1 ~]# firewall-cmd --reload
success
[root@urgluster1 ~]#

Add Storage:-

I am assuming that we have one spare hard disk on our machine, /dev/sdb is the one, which we will use for a brick. We need to create a single partition on the spare hard disk as like below.

We need to perform the below steps on both nodes.

[root@urgluster1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x5b54b761.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-4188133, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4188133, default 4188133):
Using default value 4188133
Partition 1 of type Linux and of size 2 GiB is set

Command (m for help): p

Disk /dev/sdb: 2144 MB, 2144324608 bytes, 4188134 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x5b54b761

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     4188133     2093043   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@urgluster1 ~]#

Format the created partition with the file system of your choice.

[root@urgluster1 ~]# mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
130816 inodes, 523260 blocks
26163 blocks (5.00%) reserved for the super user
First data block=0
Maximum file system blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and file system accounting information: done

[root@urgluster1 ~]#

Now we can mount the disk on a directory like /test/glusterfs.

[root@urgluster1 ~]# mkdir -p /test/glusterfs
[root@urgluster1 ~]# mount /dev/sdb1 /test/glusterfs
[root@urgluster1 ~]#

Now we need to add an entry into /etc/fstab for keeping the mount persistent across reboot this directory.

[root@urgluster1 ~]# echo "/dev/sdb1 /test/glusterfs ext4 defaults 0 0" | tee --append /etc/fstab

GlusterFS configuration on CentOS 7:-

Before creating a volume, we need to create trusted storage pool by adding urgluster2. We can run GlusterFS configuration commands on any one server in the cluster will execute the same command on all other servers.

Here we will run all GlusterFS commands in gluster1 node.

[root@urgluster1 ~]# gluster peer probe urgluster2
peer probe: success.
[root@urgluster1 ~]#

Now we can verify the status of the trusted storage pool using below commands.

[root@urgluster1 ~]# gluster peer status
Number of Peers: 1

Hostname: urgluster2
Uuid: ceed9138-a3f3-40ed-94df-37b57b17de4a
State: Peer in Cluster (Connected)
[root@urgluster1 ~]#

We can display list the storage pool using below command.

[root@urgluster1 ~]# gluster pool list
UUID                                    Hostname        State
ceed9138-a3f3-40ed-94df-37b57b17de4a    urgluster2      Disconnected
269e06ee-5ef2-40cf-ad87-34b8eebe6d71    localhost       Connected
[root@urgluster1 ~]#

Setup GlusterFS Volume:-

Now we need to create a brick (directory) called “urclouds1” in the mounted file system on both nodes.

[root@urgluster1 ~]# mkdir -p /test/glusterfs/urclouds1
[root@urgluster1 ~]#

As we know, we are going to use replicated volume, so we need to create the volume named “urclouds1” with two replicas.

[root@urgluster1 ~]# gluster volume create urclouds1 replica 2 urgluster1:/test/glusterfs/urclouds1 urgluster2:/test/glusterfs/urclouds1
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
 (y/n) y
volume create: urclouds1: success: please start the volume to access data
[root@urgluster1 ~]#

Now we can start the volume using below command.

[root@urgluster1 ~]# gluster volume start urclouds1
volume start: urclouds1: success
[root@urgluster1 ~]#

We can check the status of the created volume using below command.

[root@urgluster1 ~]# gluster volume info urclouds1

Volume Name: urclouds1
Type: Replicate
Volume ID: 5c5f8450-9621-459d-9ae5-4b57713b61b5
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: urgluster1:/test/glusterfs/urclouds1
Brick2: urgluster2:/test/glusterfs/urclouds1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@urgluster1 ~]#

GlusterFS Installation and configuration on Client:-

We can install glusterfs-client package to support the mounting of GlusterFS file systems. We need to run all commands as root user.

First of all we need to install repository package for install glusterfs-client package.

[root@urgluster1 ~]# yum install -y centos-release-gluster

On CentOS 7 and RHEL 7 we can use below command to install glusterfs-client package.

[root@urclient ~]# yum install -y glusterfs-client

Now we need to create a directory to mount the GlusterFS file system on client server.

[root@urclient ~]# mkdir -p /client/glusterfs

Now, mount the GlusterFS file system to /client/glusterfs using the following command.

[root@urclient ~]# mount -t glusterfs urgluster1:/urclouds1 /client/glusterfs
[root@urclient ~]#

If you get any error like below.

WARNING: getfattr not found, certain checks will be skipped..
Mount failed. Please check the log file for more details.

Consider adding Firewall rules for client machine (client) to allow connections on the gluster nodes (gluster1 and urgluster2). Run the below command on both gluster nodes.

[root@urgluster1 ~]# firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.43.22" accept'
success
[root@urgluster1 ~]# firewall-cmd --reload
success
[root@urgluster1 ~]#

You can also use urgluster2 instead of gluster1 in the above command.

We can verify the mounted GlusterFS file system using below command.

[root@urclient ~]# df -h /client/glusterfs
Filesystem             Size  Used Avail Use% Mounted on
urgluster1:/urclouds1  2.0G   26M  1.9G   2% /client/glusterfs
[root@urclient ~]#

We can also use below command to verify the GlusterFS file system.

[root@urclient ~]# cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=493100k,nr_inodes=123275,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/centos-root / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/sda1 /boot xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=101688k,mode=700,uid=1000,gid=1000 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=101688k,mode=700 0 0
urgluster1:/urclouds1 /client/glusterfs fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
[root@urclient ~]#

Add below entry in /etc/fstab for automatically mounting during system boot.

[root@urclient ~]# echo "urgluster1:/urclouds1 /client/glusterfs glusterfs  defaults,_netdev 0 0" | tee --append /etc/fstab
urgluster1:/urclouds1 /client/glusterfs glusterfs  defaults,_netdev 0 0
[root@urclient ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Sep 26 12:44:35 2017
#
# Accessible file systems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=9b1b6c8c-a702-4654-8b65-3ea79c368a84 /boot                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0
urgluster1:/urclouds1 /client/glusterfs glusterfs  defaults,_netdev 0 0
[root@urclient ~]#

Test GlusterFS Replication and High-Availability:-

GlusterFS Server Side:-

To check the replication, we need to mount the created GlusterFS volume on the same storage node.

First of all we need to create one directory on both glusterfs server like /testHA

[root@urgluster1 ~]# mkdir /testHA
[root@urgluster1 ~]#
[root@urgluster2 ~]# mkdir /testHA
[root@urgluster2 ~]#
[root@urgluster1 ~]# mount -t glusterfs urgluster2:/urclouds1 /testHA
[root@urgluster1 ~]# 
[root@urgluster2 ~]# mount -t glusterfs urgluster1:/urclouds1 /testHA
[root@urgluster2 ~]#

Data inside the /testHA directory of both nodes will always be same (replication).

GlusterFS Client Side:-

Let’s create some files on the mounted file system on the client.

[root@urclient ~]# touch /client/glusterfs/file1
[root@urclient ~]# touch /client/glusterfs/file2
[root@urclient ~]#

We can verify the created files using below command.

[root@urclient ~]# ls -l /client/glusterfs/
total 0
-rw-r--r--. 1 root root 0 Sep  9 06:56 file1
-rw-r--r--. 1 root root 0 Sep  9 06:56 file2
[root@urclient ~]#

Test the both GlusterFS nodes whether they have same data inside /testHA.

[root@urgluster1 ~]# ls -l /testHA/
total 0
-rw-r--r-- 1 root root 0 Sep  9 06:56 file1
-rw-r--r-- 1 root root 0 Sep  9 06:56 file2
[root@urgluster1 ~]#
[root@urgluster2 ~]# ls -l /testHA/
total 0
-rw-r--r--. 1 root root 0 Sep  9 06:56 file1
-rw-r--r--. 1 root root 0 Sep  9 06:56 file2
[root@urgluster2 ~]#

As you know, we have mounted GlusterFS volume from urgluster1 on client, now it is time to test the high-availability of the volume by shutting down the node.

[root@urgluster1 ~]# poweroff

Now we need to test the availability of the files, we can see files that we created recently even though the node is down.

[root@urclient ~]# ls -l /client/glusterfs/
total 0
-rw-r--r--. 1 root root 0 Sep  9 06:56 file1
-rw-r--r--. 1 root root 0 Sep  9 06:56 file2
[root@urclient ~]#

Create some more files on the GlusterFS file system to check the replication.

[root@urclient ~]# touch /client/glusterfs/file3
[root@urclient ~]# touch /client/glusterfs/file4
[root@urclient ~]#

Verify the files count after created.

[root@urclient ~]# ls -l /client/glusterfs/
total 0
-rw-r--r--. 1 root root 0 Sep  9 06:56 file1
-rw-r--r--. 1 root root 0 Sep  9 06:56 file2
-rw-r--r--. 1 root root 0 Sep  9 07:41 file3
-rw-r--r--. 1 root root 0 Sep  9 07:41 file4
[root@urclient ~]#

Since the gluster1 is down, all your data’s are now written on urgluster2 due to High Availability. Now we need to power on the node1 gluster1.

Check the /testHA on the gluster1, we can see all four files in the directory, this confirms that our file replication is working as expected.

[root@urgluster1 ~]# mount -t glusterfs urgluster1:/urclouds1 /testHA
[root@urgluster1 ~]# ls -l /testHA/
total 0
-rw-r--r-- 1 root root 0 Sep  9 06:56 file1
-rw-r--r-- 1 root root 0 Sep  9 06:56 file2
-rw-r--r-- 1 root root 0 Sep  9 07:41 file3
-rw-r--r-- 1 root root 0 Sep  9 07:41 file4
[root@urgluster1 ~]#

That’s all. We have completed GlusterFS server and GlusterFS client installation and configuration. We have tested GlusterFS High Availability in this tutorial.

This Post Has 43 Comments

  1. keonhacai

    Hello to every body, it’s my first pay a quick visit of this webpage; this webpage consists of remarkable and
    really fine stuff in favor of visitors.

  2. land rover umbau wohnmobil

    Way cool! Some very valid points! I appreciate you writing this post and also the rest of the website is
    really good.

  3. land rover umbau wohnmobil

    Way cool! Some very valid points! I appreciate you writing
    this post.

  4. Rapid Results Ketones

    Pretty nice post. I just stumbled upon your blog and wanted to say that I have
    truly enjoyed surfing around your blog posts. In any case
    I’ll be subscribing to your rss feed and I hope you write again very soon!

  5. Rapid Results Ketones

    Pretty nice post. I just stumbled upon your blog and wanted to say that I have truly enjoyed surfing around
    your blog posts. In any case I’ll be subscribing to your rss
    feed and I hope you write again very soon!

  6. Rapid Results Keto Review

    Thank you for the auspicious writeup. It in truth was a entertainment account it.
    Glance complex to more introduced agreeable from you!

  7. Rapid Results Keto Review

    Thank you for the auspicious writeup. It in truth was a
    entertainment account it. Glance complex to more introduced agreeable from you!

  8. stop snoring

    I’m not sure why but this website is loading extremely slow for me.
    Is anyone else having this issue or is it a problem on my end?

    I’ll check back later on and see if the problem still exists.

  9. Body Sculpt Forskolin Review

    Great blog right here! Also your website quite a bit up fast!
    What web host are you the usage of? Can I get your affiliate hyperlink to your host?
    I want my web site loaded up as quickly as yours.

  10. Marketstrom.Gr κρεβατια

    Hello, i think that i saw you visited my site thus i
    came to “return the favor”.I’m trying to find
    things to improve my website!I suppose its ok
    to use a few of your ideas!!

  11. Radiance Cream

    I really appreciate this post. I have been looking all
    over for this! Thank goodness I found it on Bing. You have made my day!
    Thanks again!

  12. Radiance Cream

    I really appreciate this post. I have been looking all over for this!
    Thank goodness I found it on Bing. You have made my
    day! Thanks again!

  13. 3d models

    Hi there! This is my first visit to your blog! We are a collection of volunteers and starting a new
    project in a community in the same niche. Your blog provided us valuable information to work on. You have done a extraordinary job!

  14. Marina

    I blog quite often and I genuinely appreciate your
    information. This great article has truly peaked my interest.
    I will bookmark your site and keep checking for new details about once a week.
    I opted in for your Feed as well.

  15. sushi winter park fl

    Piece of writing writing is also a fun, if you be familiar with afterward you
    can write otherwise it is complex to write.

  16. camera ip không dây

    These are truly wonderful ideas in concerning blogging.
    You have touched some nice things here. Any way keep up wrinting.

  17. camera wifi giá rẻ

    This is very fascinating, You are an overly skilled blogger.
    I’ve joined your feed and stay up for searching for extra of your excellent post.
    Also, I’ve shared your website in my social networks

  18. test

    I think this is one of the most significant information for me.
    And i am glad reading your article. But wanna remark on some general things,
    The site style is wonderful, the articles is really excellent : D.
    Good job, cheers

  19. monitor genuine blood pressure

    I’m really impressed with your writing skills as well as with the layout on your weblog.
    Is this a paid theme or did you customize it yourself?

    Either way keep up the excellent quality writing, it is rare to see a great blog like this
    one today.

  20. buy

    Excellent blog you have here but I was wanting to know if you knew of any
    discussion boards that cover the same topics discussed in this article?
    I’d really love to be a part of group where I can get suggestions from
    other experienced people that share the same interest.
    If you have any recommendations, please let me know. Thank you!

  21. carer services singapore

    I love what you guys are up too. Such clever work and coverage!
    Keep up the fantastic works guys I’ve incorporated you guys to blogroll.

  22. wheelchair services in singapore

    Wow that was odd. I just wrote an incredibly long comment but
    after I clicked submit my comment didn’t appear. Grrrr…
    well I’m not writing all that over again. Anyways, just
    wanted to say fantastic blog!

  23. Cory Brann

    Thanks for another fantastic article. The place else may anybodyget that kind of information in such a perfect approach of writing?I have a presentation next week, and I am on the look for suchinfo.

  24. minecraft

    No matter if some one searches for his required thing, so
    he/she needs to be available that in detail, therefore that thing is maintained over
    here.

  25. minecraft

    I’m gone to say to my little brother, that he should also
    pay a visit this webpage on regular basis to obtain updated from hottest gossip.

  26. http://tinyurl.com/y3hvbvpr

    Hello, this weekend is fastidious in support of me, as this point in time i am reading this enormous
    educational piece of writing here at my home.

  27. minecraft

    Saved as a favorite, I like your site!

  28. minecraft

    I got this web site from my pal who shared
    with me about this website and at the moment this time I am visiting this
    web page and reading very informative articles here.

  29. gamefly free trial

    You can definitely see your skills in the work you write.
    The world hopes for more passionate writers such as you who are not afraid to say
    how they believe. Always follow your heart.

  30. apex legends mobile download

    If you wish for to obtain a great deal from this article then you have to apply
    such techniques to your won weblog.

  31. Daftar Domino Qiu Qiu

    This website certainly has all of the info

  32. how to get help in windows 10

    Spot on with this write-up, I seriously feel this amazing site needs
    a lot more attention. I’ll probably be returning to read through more, thanks for the info!

  33. bandarqq

    It’s hard to find educated people on this topic, however, you seem like you know what you’re talking about!
    Thanks

  34. online loans no paperwork bad credit

    This is better information than I have found anywhere else.

  35. aduq 365

    It’s going to be end of mine day, except before ending I am reading
    this wonderful piece of writing to improve my experience.

  36. john spencer ellis videos on marketing

    Hi, i think that i saw you visited my weblog so i came to “return the favor”.I am
    trying to find things to improve my web site!I suppose its ok to use some of your ideas!!

  37. daftar bandarq

    Hello There. I discovered your blog the usage of msn. That is a really well
    written article. I’ll be sure to bookmark it and come back to read extra of your useful information. Thank you for the post.

  38. download domino qiu qiu

    Everyone loves what you guys are usually up too. This kind of clever work and exposure!

    Keep up the wonderful works guys.

  39. Verlene

    Quality content is the crucial to be a focus for
    the viewers to go to see the website, that’s what this web site is providing.

  40. Lauren

    This site was… how do I say it? Relevant!! Finally I
    have found something which helped me. Cheers!

  41. Silke

    Thanks for sharing your thoughts. I really appreciate your efforts and I am waiting for
    your further post thank you once again.

  42. watch

    This is the perfect webpage for anybody who wants to find out about this topic. You understand a whole lot its almost tough to argue with you (not that I personally would want to…HaHa). You certainly put a fresh spin on a topic that has been discussed for a long time. Wonderful stuff, just great!

  43. Christa

    Appreciate the recommendation. Will try it out.

Leave a Reply