{"id":9986,"date":"2024-10-01T16:12:53","date_gmt":"2024-10-01T16:12:53","guid":{"rendered":"https:\/\/cheapwindowsvps.com\/blog\/unlocking-insights-centralized-log-collection-and-analysis-with-graylog\/"},"modified":"2025-01-20T11:44:17","modified_gmt":"2025-01-20T11:44:17","slug":"unlocking-insights-centralized-log-collection-and-analysis-with-graylog","status":"publish","type":"post","link":"https:\/\/cheapwindowsvps.com\/blog\/unlocking-insights-centralized-log-collection-and-analysis-with-graylog\/","title":{"rendered":"Unlocking Insights: Centralized Log Collection and Analysis with Graylog"},"content":{"rendered":"<p><strong>Graylog<\/strong> serves as a comprehensive open-source platform dedicated to centralized log collection, storage, visualization, filtering, searching, and analysis. In addition, it is capable of performing monitoring functions and generating alerts. Graylog is designed to gather logs and events from a vast array of network devices, which include Linux servers, Windows servers, and various network appliances or hardware. As a robust solution, Graylog has the capacity to manage terabytes of log data while enabling rapid search capabilities. This guide discusses the deployment of the Graylog stack along with its fundamental functionalities for logging tasks.<\/p>\n<p>Components that constitute the Graylog stack include:<\/p>\n<ul>\n<li>Graylog Server \u2013 the web interface for configuration and visualization.<\/li>\n<li>MongoDB \u2013 the database responsible for storing metadata.<\/li>\n<li>ElasticSearch or its alternative OpenSearch \u2013 utilized for storing and performing full-text searches on both structured and unstructured logs.<\/li>\n<li>Java (OpenJDK) \u2013 serves as the runtime for OpenSearch (or ElasticSearch).<\/li>\n<\/ul>\n<p>For log storage, we will implement OpenSearch, which is a free and open-source alternative to the ELK stack (Elasticsearch + Logstash + Kibana).<\/p>\n<h2>How to Install the Graylog Stack on Linux Systems<\/h2>\n<p>The process of installing the Graylog stack varies significantly based on the version you are using. Consequently, we highly recommend visiting the <a href=\"https:\/\/go2docs.graylog.org\/current\/home.htm\" target=\"_blank\" rel=\"nofollow noopener\">official Graylog documentation<\/a> to manually select your specific Linux distribution and Graylog version. This guide is based on Debian 12 and Graylog 5.2.<\/p>\n<div>\n<p>The Graylog stack can be deployed using docker-compose; however, this guide will focus on a complete deployment of all components.<\/p>\n<\/div>\n<p>In this guide, we will utilize:<\/p>\n<ul>\n<li><strong>OpenJDK 17<\/strong><\/li>\n<li><strong>OpenSearch<\/strong> 2.x (or <strong>Elasticsearch<\/strong> 7.10.2 \u2013 the only version compatible with Graylog 5.2)<\/li>\n<li><strong>MongoDB<\/strong> 5.x or 6.x (Deployment of MongoDB 7 is currently not advised)<\/li>\n<li>For minimal installations, Graylog Server requires at least 2GB of RAM.<\/li>\n<\/ul>\n<p>Begin by installing the necessary prerequisites and adding the MongoDB 6 repository:<\/p>\n<p>$ sudo apt-get install gnupg curl<br \/>\n$ curl -fsSL https:\/\/www.mongodb.org\/static\/pgp\/server-6.0.asc | sudo gpg -o \/usr\/share\/keyrings\/mongodb-server-6.0.gpg &#8211;dearmor<br \/>\n$ echo &#8220;deb [ signed-by=\/usr\/share\/keyrings\/mongodb-server-6.0.gpg] http:\/\/repo.mongodb.org\/apt\/debian bullseye\/mongodb-org\/6.0 main&#8221; | sudo tee \/etc\/apt\/sources.list.d\/mongodb-org-6.0.list<br \/>\n$ sudo apt-get update<\/p>\n<p>Next, proceed to install the MongoDB package:<\/p>\n<p><code>$ sudo apt-get install -y mongodb-org<\/code><\/p>\n<div>\n<p>In my experience, trying to install Mongodb on Debian 12 led to an issue:<\/p>\n<pre>The following packages have unmet dependencies:<\/pre>\n<p>mongodb-org-mongos: Depends: libssl1.1 (&gt;= 1.1.1) but it is not installable<\/p>\n<p>mongodb-org-server: Depends: libssl1.1 (&gt;= 1.1.1) but it is not installable<\/p>\n<p>I had to install an additional libssl package:<\/p>\n<p>Resolve this by installing the libssl package.<\/p>\n<p><code>$ sudo wget http:\/\/archive.ubuntu.com\/ubuntu\/pool\/main\/o\/openssl\/libssl1.1_1.1.1f-1ubuntu2_amd64.deb<br \/>\n$ sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb<\/code><\/p>\n<\/div>\n<p>To start the service:<\/p>\n<p><code>$ sudo systemctl daemon-reload<br \/>\n$ sudo systemctl enable mongod.service<br \/>\n$ sudo systemctl restart mongod.service<br \/>\n$ sudo systemctl --type=service --state=active | grep mongod<\/code><\/p>\n<p>Next, continue with the installation of OpenSearch.<\/p>\n<p><code>$ sudo apt-get update &amp;&amp; sudo apt-get -y install lsb-release ca-certificates curl gnupg2<\/code><\/p>\n<p>To add the repository, execute the following commands:<\/p>\n<p><code>$ curl -o- https:\/\/artifacts.opensearch.org\/publickeys\/opensearch.pgp | sudo gpg --dearmor --batch --yes -o \/usr\/share\/keyrings\/opensearch-keyring<br \/>\n$ echo \"deb [signed-by=\/usr\/share\/keyrings\/opensearch-keyring] https:\/\/artifacts.opensearch.org\/releases\/bundle\/opensearch\/2.x\/apt stable main\" | sudo tee \/etc\/apt\/sources.list.d\/opensearch-2.x.list<br \/>\n$ sudo apt update<\/code><\/p>\n<p>Beginning with OpenSearch version 2.12, it is mandatory to create an administrator password when installing:<\/p>\n<p><code>$ sudo OPENSEARCH_INITIAL_ADMIN_PASSWORD=$(tr -dc A-Z-a-z-0-9_@#%^-_=+ &lt; \/dev\/urandom | head -c${1:-32}) apt-get install opensearch<\/code><\/p>\n<p>Then, proceed to adjust the OpenSearch settings:<\/p>\n<p><code>$ sudo nano \/etc\/opensearch\/opensearch.yml<\/code><\/p>\n<p>To set up a basic single-node configuration, you only need to adjust the following parameters:<\/p>\n<pre>cluster.name: graylog<\/pre>\n<p>node.name: ${HOSTNAME}<\/p>\n<p>path.data: \/var\/lib\/opensearch<\/p>\n<p>path.logs: \/var\/log\/opensearch<\/p>\n<p>discovery.type: single-node<\/p>\n<p>network.host: 0.0.0.0<\/p>\n<p>action.auto_create_index: false<\/p>\n<p>plugins.security.disabled: true<\/p>\n<p>indices.query.bool.max_clause_count: 32768<\/p>\n<p>Next, you should configure the SMTP server settings to facilitate sending email notifications:<\/p>\n<pre>transport_email_enabled = true<\/pre>\n<p>transport_email_hostname = smtp.gmail.com<\/p>\n<p>transport_email_port = 465<\/p>\n<p>transport_email_use_auth = true<\/p>\n<p>transport_email_use_tls = false<\/p>\n<p>transport_email_use_ssl = true<\/p>\n<p>transport_email_auth_username =<\/p>\n<p>transport_email_auth_password =<\/p>\n<p>transport_email_subject_prefix = [graylog]<\/p>\n<p>transport_email_from_email =<\/p>\n<p>transport_email_web_interface_url =<\/p>\n<p>Next, you&#8217;ll want to modify the default Java configuration settings:<\/p>\n<p><code>$ sudo nano \/etc\/opensearch\/jvm.options<\/code><\/p>\n<p>Utilize the Xms and Xmx options to define the maximum memory allocation for the Java Virtual Machine. A good rule of thumb is to allocate half of the available RAM on your server. For instance, if your system is equipped with 8GB of RAM, you should allocate 4GB as follows:<\/p>\n<pre>-Xms4g<\/pre>\n<p>-Xmx4g<\/p>\n<p>To adjust the virtual memory settings, you can use the following commands:<\/p>\n<p><code>$ sudo sysctl -w vm.max_map_count=262144<br \/>\n$ sudo echo 'vm.max_map_count=262144' &gt;&gt; \/etc\/sysctl.conf<\/code><\/p>\n<p>Next, initiate the OpenSearch service with this command:<\/p>\n<p><code>$ sudo systemctl enable --now opensearch<\/code><\/p>\n<p>Next, you will want to <strong>install the Graylog Server<\/strong>. There are two available versions of Graylog: the free <strong>Graylog Open<\/strong> and the subscription-based enterprise version known as <strong>Graylog Operations<\/strong>.<\/p>\n<p>To install Graylog Open, follow these commands:<\/p>\n<p>$ wget https:\/\/packages.graylog2.org\/repo\/packages\/graylog-5.2-repository_latest.deb<br \/>\n$ sudo dpkg -i graylog-5.2-repository_latest.deb<br \/>\n$ sudo apt-get update &amp;&amp; sudo apt-get install graylog-server<\/p>\n<p>Make sure to create passwords for the variables <code>password_secret<\/code> and <code>root_password_sha2<\/code>. The Graylog server will not initiate without these passwords.<\/p>\n<p>The password <strong>password_secret<\/strong> should have a minimum length of 64 characters:<\/p>\n<p><code>$ pwgen -N 1 -s 96<\/code><\/p>\n<p>Next, retrieve the hash for the Graylog administrator password. Use the command below to create a hash of the specified password:<\/p>\n<p><code>$ echo -n \"Enter Password: \" &amp;&amp; head -1 &lt; \/dev\/stdin | tr -d '' | sha256sum | cut -d\" \" -f1<\/code><\/p>\n<p>Insert the values for <strong>password_secret<\/strong> and <strong>root_password_sha2<\/strong> into the configuration file located at <strong>\/etc\/graylog\/server\/server.conf<\/strong>.<\/p>\n<p>Configure the IP address and port for the Graylog web server by adjusting the <code>http_bind_address<\/code> setting.<\/p>\n<p>To begin using Graylog, execute the following command:<\/p>\n<p><code>$ sudo systemctl enable --now graylog-server<\/code><\/p>\n<h2>Initiating Your Graylog Configuration<\/h2>\n<p>Begin by accessing the Graylog web interface with the admin username and the password hash you set in the configuration file. Be aware that this password will not be recognized during your initial login attempt.<\/p>\n<p>Examining the Graylog server logs will reveal a noteworthy message:<\/p>\n<p><code>$ cat \/var\/log\/graylog-server\/server.log<\/code><\/p>\n<pre>It appears that you are launching Graylog for the first time. A setup interface has been initiated for a fresh installation. To perform the initial configuration and proceed, you must log in. The initial configuration can be accessed at 0.0.0.0:9000, using the username 'admin' and the password 'eDLaYAlaeX'. You can click on http:\/\/admin:<a href=\"\/cdn-cgi\/l\/email-protection\" target=\"_blank\" rel=\"nofollow noopener\">[email\u00a0protected]<\/a>:9000<\/pre>\n<p>Utilize the temporary password from the log file for your first sign-in. Follow the straightforward initial configuration wizard. The <em>Graylog data node<\/em> mode is utilized to set up a multi-node OpenSearch cluster. If you&#8217;re configuring a simpler setup with a single log host, this step can be bypassed.<\/p>\n<p>Next, access the Graylog web interface with the <strong>admin<\/strong> user and the provided password:<\/p>\n<p>After logging in, create your Graylog user by navigating to <strong>System<\/strong> -&gt; <strong>Users and Teams<\/strong>. Certain settings of the built-in Admin account are not customizable; therefore, it is advisable to establish a separate user account. Assign this user the Admin role and provide an email address.<\/p>\n<p>It&#8217;s time to set up the <strong>Inputs<\/strong> (data collectors). Ideally, each input should be dedicated to a specific device category, such as Linux servers, networking devices, Windows hosts, and so forth.<\/p>\n<p>We will establish a <strong>Syslog UDP<\/strong> input specifically for Linux hosts. You&#8217;ll need to provide a name along with the port number that the Graylog server will use to receive data. You can keep the remaining settings at their default values.<\/p>\n<p>Next, navigate to <strong>System<\/strong> -&gt; <strong>Indices<\/strong> and set up a distinct index for the Linux device category.<\/p>\n<p>You&#8217;ll need to indicate a name, a description, and a prefix (for example, <code>linux_indx<\/code>). Additionally, configure how many days you want to retain older logs, the timeline for deleting obsolete indexes, and the maximum size for each index.<\/p>\n<p>Establish a distinct <strong>Stream for each input.<\/strong> This method helps in categorizing logs from various device classes into separate indexes. Navigate to <strong>Stream<\/strong> -&gt; Create Stream -&gt; designate a name for the stream and choose the index you prefer to utilize.<\/p>\n<p>Within the Stream settings, introduce a new rule that defines the type of logs allocated to this stream. For instance, opt for <strong>Match Input<\/strong> -&gt; select your <strong>Linux Input.<\/strong><\/p>\n<p>Subsequently, activate the stream.<\/p>\n<h2>How to Transfer Logs from Linux to Graylog<\/h2>\n<p>Next, set up your client devices to forward events and logs to Graylog. Graylog is capable of receiving data from multiple sources including Filebeat, Winlogbeat, Nxlog, Syslog, and Rsyslog.<\/p>\n<p>For collecting logs and transmitting them to Graylog, you can utilize <strong>rsyslog<\/strong> on Linux servers.<\/p>\n<p>To install rsyslog, run the following command:<\/p>\n<p><code>$ sudo apt install rsyslog<\/code><\/p>\n<p>To check the status of rsyslog, use this command:<\/p>\n<p><code>$ sudo systemctl status rsyslog<\/code><\/p>\n<p>When the rsyslog service is operational, it&#8217;s time to set up the logs that will be directed to Graylog.<\/p>\n<p><code>$ sudo nano \/etc\/rsyslog.d\/60-graylog.conf<\/code><\/p>\n<p>Insert the subsequent line into the file:<\/p>\n<pre>*.*@192.168.14.146:20514;RSYSLOG_SyslogProtocol23Format<\/pre>\n<p>In this scenario, we will forward all logs in syslog format to the Graylog server located at 192.168.14.146, using the designated input port of 20514.<\/p>\n<p>To apply the changes, restart the rsyslog service:<\/p>\n<p><code>$ sudo systemctl restart rsyslog<\/code><\/p>\n<h2>Searching and Analyzing Logs in Graylog<\/h2>\n<p>Next, navigate to the <strong>Search Graylog<\/strong> section, and choose the \u2018Linux\u2019 stream. This will display all events that have been collected from your hosts over the last 5 minutes (you can adjust the time frame as needed).<\/p>\n<p>You can apply straightforward filters to pinpoint specific events. The Graylog search bar is designed to be user-friendly. In it, you can select various event properties and pick from the provided options.<\/p>\n<p>For instance, to locate all events pertaining to unsuccessful SSH connection attempts to a host, you can use the following search filter:<\/p>\n<p><code>source:srv-ubun01 AND application_name:sshd AND message:Fail*<\/code><\/p>\n<p>This query retrieves all logs that fulfill all three criteria (using the <code>AND<\/code> parameter). In the absence of the AND specification, Graylog defaults to searching for events in <code>OR<\/code> mode.<\/p>\n<p>The resulting table will display events that align with your search parameters. You can expand and delve into the specifics of each log detail. The search filters in Graylog are straightforward and user-friendly, allowing for easy exploration. You can always expand any event to view its available fields (event attributes) and use those values to refine your search for particular events.<\/p>\n<p>Graylog offers the capability to create and tailor various dashboards that showcase information about events of interest (which may include counts of events, hosts, grouped items, visual graphs, or maps). Dashboards can be customized using widgets.<\/p>\n<h2>Setting Up Graylog Alerts for Event Occurrences<\/h2>\n<p>In certain situations, Graylog has the capability to send automatic notifications, commonly referred to as alerts. For instance, I would like to set up Graylog to notify me via email whenever someone accesses one of my servers through SSH.<\/p>\n<p>To begin, navigate to <strong>Alerts<\/strong> -&gt; <strong>Alerts and Events<\/strong>. Within the Notification tab, you can enter the email addresses that will receive these alerts.<\/p>\n<div>Graylog offers pre-defined templates for alert notifications integrated with MS Teams and Slack. Additionally, with the help of extra plugins, you can configure notifications to be sent to WhatsApp or Telegram.<\/div>\n<p>Next, you will want to create a query to identify the specific events that you wish to be alerted about in the Event Definition. If you are interested in monitoring SSH logins to Linux systems, select the relevant stream and provide a suitable search query:<\/p>\n<p><code>application_name:sshd AND message:Accepted password*<\/code><\/p>\n<p>A list of events that match your search criteria will show up in the right preview pane. Make sure that all the events you intended to find are present. If any are missing, feel free to modify your search query.<\/p>\n<p>Next, you need to choose your notification type -&gt; <strong>email<\/strong>. When a specific event occurs in the logs, Graylog will automatically send out an email notification.<\/p>\n<p>This article has provided an overview of deploying and using Graylog for the purposes of centralized log collection and analysis. In the upcoming article, we will explore how to use Graylog for the centralized collection and search of logs from Windows servers.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Graylog serves as a comprehensive open-source platform dedicated to centralized log collection, storage, visualization, filtering, searching, and analysis. In addition, it is capable of performing monitoring functions and generating alerts. Graylog is designed to gather logs and events from a vast array of network devices, which include Linux servers, Windows servers, and various network appliances [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":9987,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[116],"tags":[],"class_list":["post-9986","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/posts\/9986","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/comments?post=9986"}],"version-history":[{"count":1,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/posts\/9986\/revisions"}],"predecessor-version":[{"id":10476,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/posts\/9986\/revisions\/10476"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/media\/9987"}],"wp:attachment":[{"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/media?parent=9986"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/categories?post=9986"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cheapwindowsvps.com\/blog\/wp-json\/wp\/v2\/tags?post=9986"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}