Manage logs

When HSG/mlog is configured as a log collector, it comes with intuitive GUI for administrators to view live logs, search historical records and archive logs for compliance or future forensic investigation purposes.

SECURITY LOGGING --> Logs

There're a few tabs under this menu

  • "Live" shows the latest incoming raw logs.
    • It's auto-refreshed/updated every 5 seconds by default, however It's possible to change/adjust the refresh interval manually,
    • Click on "Pause" to freeze the fresh for investigation purpose.
    • Filter by different contents to only see the target "interesting" logs.
    • Mouseover or click on a message line to view full message details
  • "Search" allows administrator to search historical logs based on various filtering criterion, and it's possible to export and print the search results into csv. The depth of searchable logs depends on how much raw logs you configured to keep in the SQL database (refer to "Keep raw logs locally" value in next tab).
  • "Archive" allows administrator to retain historical logs for compliance or forensic investigations. NOTE, if you've configured CLI for log archival, the GUI setting will superseded CLI setting (eg. CLI config won't take effect once GUI setting is configured). It's recommended to use GUI to configure log archival.
    • "Archive log data into daily or hourly files (run nightly)". This option defines how raw logs are archived. By default the raw logs are collected and stored in local SQL database, viewable & searchable from GUI. When the logs are archived, they are exported out from SQL database into compressed csv files. The primary purpose of "archive" is to reduce storage space. when the same log contents are exported/converted from SQL to zipped csv files, typically the storage space will be reduced by 20 times. You can click to switch export in "Daily Files" or "Hourly Files". If the archival file sizes are expected to be very huge, then it's better to choose "Hourly Files". For example, an mlog can collect more than 20GB logs per day, and the compressed csv file can be 1GB in size. It will be very hard to download and unzip the archived files later when it comes to investigation. However, in most case, "Daily Files" will be just enough. For a typical network with up to 2000 users, the daily file won't be more than 50MB, which is still quite manageable.
    • Keep raw logs locally (recommend 1 day)* This option defines how much raw logs to store in SQL database. The raw logs can be viewed and searched using GUI for immediate investigation purposes. However raw logs usually consume large storage, and this is dangerous for HSG which has limited disk space. So It's highly recommended to minimize this value. However, mlog would have large local storage (with additional HDD) so you can slightly put this value higher.

NOTE, all raw logs older than the configured x day here will be purged from SQL database. If you need to investigate incidents happened earlier than the configured x day, you would refer to archival files. So please make sure you set the right values to "Keep archived files locally", in order to meet compliance requirements.

    • Keep archived files locally (optional)* This setting defines how long (or how many) archived files (the compressed csv files) you want to keep in the local storage, and the archival files are listed/shown in GUI so you can download and unzip to view the raw logs when it comes to investigation or reporting. Please be sure to set this value to compliant to local cyber security regulations. Many countries require to keep minimum 90 days of firewall/URL access logs for public Internet access.

NOTE, HSG default storage disk space can store data up to 20GB (including user accounts, profiles, session records, firewall access logs, etc). For a large mall or F&B chain with 150 outlets, the typical daily archival file size is 50MB, therefore 90 days of archival storage is about 5GB. So the default disk size would be sufficient. However, extra disk space can be customized upon request.

    • Backup archived files to external FTP Server (runs nightly). This optional features give you the flexibility to store archival files into external FTP server. You can combine above setting (keep archived files locally) together, eg. store x days of local archival files so that you can easily download from GUI and at the same time store all archival files to FTP server so that you can keep for as long as needed. This option is extremely important if you have large amount of logs and must minimize local archival files. Exporting to external FTP eliminates all the storage constraints (potentially unlimited, eg. if you use a huge & cheap NAS storage).

SECURITY LOGGING --> Alarms

HSG/mlogs comes with a log analyzer engine that tracks all incoming live logs, compares each log against the predefined rules, and alarms once a rule is triggered (log pattern is matched to a rule). The engine runs every one minute to analyze latest raw logs. The alarms are shown in GUI and sent out to emails at the same time.

  • "Engine" allows administrator to turn on/off engine. Note turning on log analyzer enginer with large amount of rules and logs will impact system performance.
  • "Alarms" shows abnormal events triggered by alarms rules. It means certain log patterns are matched, eg. access to prohibited websites or destinations.
  • "Rules" defines the patterns that you want to match certain conditions.
    • "Logical Operator, and/or" If you have multiple matching Criteria, and means ALL criteria must be matched to trigger an alarm; or means EITHER one of the criteria matched will trigger an alarm.
    • "Criteria" sets what fields/contents of the raw log patterns to match. You can match by time, host, or certain contents of the log message (see above screenshot).
    • "Action" allows you to add emails to send alarms


View local logs from CLI

mbox is by default enabled with local logging, to view locally generated logs (usually for troubleshooting purpose), issue below command:

mbox# show logging system

Info: showing system local logs. use CTL+C to stop

Oct 14 23:22:06 zydev kernel: [3964398.563219] mboxfw-permit:IN=eth0 OUT= MAC=00:0c:29:44:8b:f8:00:0c:29:f2:fd:c6:08:00 SRC=10.99.1.3 DST=10.65.19.9 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=14154 DF PROTO=TCP SPT=50467 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0 Oct 14 23:22:31 zydev kernel: [3964422.848630] mboxfw-permit:IN=eth0 OUT= MAC=00:0c:29:44:8b:f8:00:0c:29:f2:fd:c6:08:00 SRC=10.99.1.3 DST=10.65.19.9 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=19492 DF PROTO=TCP SPT=50468 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0 Oct 14 23:28:15 zydev kernel: [3964766.693619] mboxfw-permit:IN=eth0 OUT= MAC=00:0c:29:44:8b:f8:00:0c:29:f2:fd:c6:08:00 SRC=10.99.1.3 DST=10.65.19.9 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=40692 DF PROTO=TCP SPT=50470 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0

NOTE:

  • This command only shows real-time logs, for troubleshooting purposes. mbox doesn't keep any historical local logs. If we need historical logs, we have to export logs to an external log server/collector, or enable local "security log-server"
  • if we want to view firewall logs etc from local console/ssh, we still need to enable access logging, discussed in this section. But don't output/export to an external server.
  • if an mbox is configured as a log client (export logs out to external log collector), the exported logs will not appear in this command. We need to view exported logs from the log server/collector.