mbox data storage management

mbox holds different type of database depending which operating modes:

    • HotSpot Gateway (radius database, holding user accounts and data analytics data)
    • HotSpot Gateway (CMS portals)
    • mlog server (Syslog database, holding all raw syslog data from local source or external source)
    • mfusion server (mfusion database, holding all monitoring and configuration data)
    • mbox admin data (mboxadmin database, storing all users/hosts/entities accessing to mbox local services)

There are three ways to store and backup mbox data:

    • store & backup to local PC from mbox GUI
    • store on externally attached storage, either USB drive or additional disk
    • archive to external FTP server (eg. NAS running FTP service)

By default, all data is stored in the same disk that shares with mboxOS, which has limited storage space. For mlog and mfusion, where there's large amount of data to be stored, it's usually not possible to store on mbox OS disk. So we need to slot in additional HDD (USB drive or built-in SSD/SATA/SAS etc), so that data is stored in the large disk, without impacting OS operational functions.


BACKUP TO LOCAL PC

Login to mbox portal, eg. http://<mboxip>, login as mboxadmin, then password. Once login, go to ADMIN -- > Settings -- Config Backups.

NOTE: by default the backup only saves config related data, such as user accounts, profiles, or syslog alarm rules and mfusion configuration and mbox user/hosts/entities data etc. You can optionally tick to save user access records, or mfusion monitoring data etc.

Once the backup files are created, you can download offline to store on your PC. In case of recovery, you can upload it back and restore your backups.

Please refer to video demo for more comprehensive mbox disaster recovery options.

STORE ON USB OR ADDITIONAL HDD STORAGE

In this example, we have an 1TB USB drive added to mbox (either built-in HDD or USB), so that we can store large amount of Syslog/radius/mfusion data using external disk. In the case of mlog, it already comes with a built-in additional SSD storage HDD, 1TB or 2TB depending on the models. But we should go through the same steps to mount the additional HDD.

PART 1: discover and mount USB storage (or new disk)

there are 3 steps in this part:

    1. discover new disk
    2. format the disk
    3. mount USB/HDD disk to a local folder

1. When a USB/HDD drive is slotted into mbox USB port, it will be automatically discovered and mounted. Check what’s the drive name.

mfusion#mfusion# disk show Disk /dev/sda: 32.0 GB, 32000442368 bytes255 heads, 63 sectors/track, 3890 cylinders, total 62500864 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x000213e1 Device Boot Start End Blocks Id System/dev/sda1 * 2048 59895807 29946880 83 Linux/dev/sda2 59897854 62531583 1316865 5 Extended/dev/sda5 59897856 62531583 1316864 82 Linux swap / SolarisDisk /dev/sdb: 1024.2 GB, 1024209543168 bytes <--this is the USB storage........

In above output,

    • /dev/sda (32GB) is the primary disk where mbox OS sits, DO NOT TOUCH THIS!
    • /dev/sdb is the new USB/HDD drive

Sometimes, we may have another disk within mbox for special purposes, so the new USB/HDD storage will be detected as /dev/sdc, similarly if you have more disks, they will be detected as /dev/sde, /dev/sdf, etc et.

2. Once you identify the drive name for your USB/HDD storage, create and format the disk. Enter below command and press enter to finish at the end.

mfusion#mfusion# disk create /dev/sdball data on disk /dev/sdb will be lost. do you want to continue "y" or "n": yCalling ioctl() to re-read partition table.Syncing disks.mke2fs 1.42.5 (29-Jul-2012)Filesystem label=OS type: Linux..........Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): doneWriting superblocks and filesystem accounting information: doneplease map new partition /dev/sdb1 to a folder.mfusion#

And issue “disk show” again, you should see the new partition added.

mfusion#mfusion# disk showDisk /dev/sdb: 1000.2 GB, 1000170586112 bytes11 heads, 16 sectors/track, 11099194 cylinders, total 1953458176 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x1d084add Device Boot Start End Blocks Id System/dev/sdb1 2048 1953458175 976728064 83 Linux <--new partition addedmfusion#

3. Now the new disk will appear as a local partition, then we mount it to a local folder so we can use it to store data. (use the partition name from above "disk show", sometimes could be /dev/sdb1 or /dev/dbc1)

mfusion#mfusion# disk mount /data /dev/sdb1creating a new local folder /data/data is mounted to /dev/sdc1 successfully.

To verify the disk is mounted to local folder and ready for storing data, use “show system disk”

mfusion#mfusion# show system disk Filesystem Size Used Avail Use% Mounted onrootfs 28G 4.4G 23G 17% /udev 10M 0 10M 0% /devtmpfs 1.6G 224K 1.6G 1% /runtmpfs 5.0M 0 5.0M 0% /run/locktmpfs 3.2G 4.0K 3.2G 1% /run/shmtmpfs 256M 35M 222M 14% /var/log/dev/sdb1 917G 285M 870G 1% /datamfusion#

PART 2: store SQL data to mounted USB/HDD storage

Now the USB/HDD storage is mounted as a local disk and mount to a local folder, ready to use. We configure mysql server to use the mounted folder to store data.

!mfusion mysql-server data-path /data start!


ARCHIVE TO EXTERNAL NAS STORAGE (FTP)

We can optionally backup data to external NAS storage for data archival purposes (configure NAS as a FTP server). This is particularly useful for mlog because it usually expects large amount of log data. Even the additional HDD may not be sufficient to store all data, especially for long-term log retention requirements.


!security log-server log-input 10 accept all data-lifetime 5 ftp ftpuser ftppass <ftpserverip> <folder on ftp, eg. /backup> start!

In above configuration:

    1. "data-lifetime 5": this keeps latest 5 days data locally (if using additional drive, it will be on /data), and older data is archived to ftp server on daily.
    2. “ftp”: we use the ftp option, so that mbox will mount external ftp server folder to a temp folder on mbox, and nightly backup/push previous day's data to the ftp folder
    3. Use below commands to verify that data is backed up to the ftp folder

mbox# show system backup ftptotal 12K4.0K drwxr-xr-x 2 root root 4.0K Apr 2 13:33 .4.0K drwxr-xr-x 28 root root 4.0K Apr 2 21:27 ..4.0K -rw-rw-rw- 1 mysql mysql 53 Apr 2 13:33 Syslog-SystemEvents-20150401.csv.gz