Security logging and monitoring are two of the most essential security controls recommended by security standards and guidelines. And there’s a reason for that.
In this article, you'll understand what security logging and monitoring are, why they matter, and the key challenges involved. Finally, you'll get to know how ThreatKey integrates with your security logging to enhance your cybersecurity.
What Are Logging and Monitoring?
To understand logging and monitoring in general, you should first understand their basic concepts.
Let's first understand what an event is, as it's fundamental to logging and monitoring. An event is a notable occurrence in the operation of a software component. An event may describe what a user did, a change in the component's internal state or external environment, or a failure to handle a situation.
Events come from every level of software architecture. Events from device drivers, an operating system, and system services are called system events. Those from server software, user applications, and application libraries are called application events. Events can also be categorized by their sources — user devices like smartphones, internet of things (IoT) devices, network routers, workstations, web servers, database servers, and so on.
A log — or event log — is a record of a sequence of events. It stores the order of events, timestamps, source details, and other event-specific information for the long term. Depending on the use case, log data can be stored as log files on disk, as records in a database, or as items in persistent queues.
There are different types of logs based on the type of events they contain and their purposes. For example, there are system logs, application logs, security logs, and performance logs.
3. Logging and Monitoring
Logging refers to the tasks of generating, collecting, and storing logs.
Log management covers all the aspects and policies necessary for logs to satisfy the business goals they support. These aspects include log generation, collection, storage, archiving, data transfer, analysis, search, retention, deletion, and security.
Log monitoring refers to analyzing, searching, understanding, visualizing, and responding to logs. Visualization and analytics provide infrastructure-wide overviews and metrics on graphical dashboards but also support looking at the logs and metrics of a single system or device.
Together, logging, log monitoring, and log management help you observe your infrastructure in real time. In this way, they support your business goals and decision-making in areas like operations, security, and performance.
Logging vs. Tracing
Many people, including software engineers, confuse logging and tracing because the names of tracing libraries often contain "log".
Tracing helps show the logic and flows inside a software component. It's mostly meant to help internal software engineering teams uncover bugs in their code and verify that their software logic is behaving as expected. Tracing is usually not meant to be seen by the software's customers or users (with some rare exceptions).
In contrast, logging records notable events that are relevant to the software's customers and users. It allows observation of the software that you purchase or deploy in production as a customer. It supports your operations and security teams in their respective business goals.
What Are Security Logging and Monitoring?
As you may have guessed, security logging and monitoring focus on security events and logs. Security logging covers the generation, collection, and storage of security event logs. Security monitoring refers to the analysis and visualization of security logs. They include:
- Audit logs that record audit trails of authentication attempts from user accounts or system accounts and the security policy decisions for them
- Device logs that record device-specific security events, like firewall logs
- System logs from the operating system or system services with events that impact security, like security policy changes
- Application logs from servers and applications (web, desktop, smartphone, or software-as-a-service) that record security-impacting events like configuration changes and application programming interface (API) calls
Security Logging vs. General Logging
General and security logging may both be managed by the same centralized log management system. But once the logs are collected, security logging and monitoring differ from general logging and monitoring:
- Security logging and monitoring are handled by your security organization consisting of a security operations center, security analysts and engineers, threat intelligence specialists, and similar roles. In contrast, general logging and monitoring are done by operations and software engineering teams.
- Log management, retention periods, and security policies for security logs are often different from those for operational or performance logs.
Security Log Examples
Security log entries contain information like:
- The source of an action or event, like its IP address
- The timestamp of the event
- The log message with useful descriptive information about the event
- Priority and category of the event
Why Security Logging Is Key
Let's understand three of the main benefits of security logging.
1. Cyberattack Detection and Prevention
The most important benefits of security logging are security incident detection and prevention. Log analysis supports the following capabilities:
- Intrusion detection: Finds malicious attempts to enter your network perimeter
- Malware detection: Finds malicious software installed or embedded in your systems
- User and entity behavior analysis: Detects suspicious activities of users and software
- Anomaly detection: Establishes baseline security behaviors so that security systems can judge deviations from them as suspicious anomalies that warrant investigation
- Authentication alerts: Notifies security teams about events like repeated failed login attempts
- Access control: Flags attempts by unauthorized users to access protected data or perform protected actions
- Incident response: Guides incident response actions according to the events recorded in logs
2. Cyberattack Forensic Investigations
Despite your best efforts, you may experience a cyberattack like a data breach and discover it only months later. Security logging becomes critical to understand when and how it happened.
3. Regulatory Compliance
Security logging and monitoring are essential components recommended by all regulatory standards, cybersecurity, and application security guidelines. A few examples:
- The Health Insurance Portability and Accountability Act (HIPAA) mandates log management and monitoring procedures for healthcare companies.
- The National Institute of Standards and Technology (NIST) cybersecurity framework provides guidelines related to logging for cyber risk management and security assessments.
- The Payment Card Industry Data Security Standards (PCI DSS) requirements on audit logs (PDF) say, "retain audit trail history for at least one year, with a minimum of three months immediately available for analysis (for example, online, archived, or restorable from backup)."
5 Drawbacks and Security Challenges of Security Logging
Security logs are just another type of data, and like any data, they face data security risks. Security logging and log management procedures can bring additional security challenges to your organization. Here’s how:
1. Log Volume Renders Manual Analysis Impractical
When your company is facing a cyberattack, your security teams don't get much time to detect the threats and respond. They have to go through the logs to understand what's happening and design effective countermeasures.
But the sheer volume of infrastructure logging makes manual analysis rather impractical. Consider these numbers:
- WePay, a mid-market online payments service, can handle up to 80,000 events every second.
- Twitter logged 600,000 events every second — a whopping 42 terabytes per day — from each data center in 2021.
These numbers are typical for companies of their sizes. Even if a mere 10% of logs are on security information, it's still a huge volume of data to analyze.
That's why cybersecurity is becoming increasingly automated. Basic security logging first evolved into security incident and event management (SIEM). Then came security orchestration, automation, and response (SOAR) systems. More evolution gave us extended detection and response (XDR) systems that use techniques like real-time pattern recognition to automatically detect cyberthreats in terabytes of logs. Once detected, they send notifications to the security teams for follow-up.
2. Insufficient Logging
At the other end of the scale is the problem of insufficient logging. What information gets logged is generally under the discretion of software development teams. Teams that aren't knowledgeable about security operations may omit events that are useful to security analysts. Involving security engineers in code reviews and system tests helps avoid this problem.
3. Need to Maintain Log Confidentiality, Integrity, and Availability
Security logs are like any other data. They should satisfy the primary security goals of:
- Confidentiality: The logs should have access control policies and follow the principle of least privilege.
- Integrity: Log integrity ensures that malicious attackers or insiders can't tamper with current and archived logs to hide their activities.
- Availability: Attackers can't stop the availability and transfer of logs to the log management system.
These apply even to petabyte-scale log data, requiring well-thought-out operational and security procedures for log retention and deletion. Without them, you risk leakage of business and infrastructure details that might help attackers access your confidential data.
4. Sensitive Information Leakage
Logs should not store sensitive information like passwords, credit card numbers, or social security numbers. Where it's necessary to log them (for example, in a payment gateway to detect fraud), strict access control and confidentiality policies should be enforced.
5. Risk of Log Injection Attacks
A log injection attack introduces malicious data or malware into a software system through its logging components. It relies on the common behavior of recording user-supplied data in log entries. Such data, if carefully crafted, can exploit an existing vulnerability or help open up new vulnerabilities in the software being logged. The 2021 Log4shell attack used this technique to cause widespread havoc.
How ThreatKey Supports Security Logging
Security logging and monitoring are key for a healthy cybersecurity strategy. They can help you detect and prevent cyberattacks, understand how these attacks happen, and adhere to regulations. However, keep in mind that large volumes of data, not enough logging, and leak risks are among some of the security logging drawbacks.
ThreatKey is a security automation SaaS that's both a consumer and producer of critical security logs.
As a consumer of security logs, ThreatKey analyzes logs and configurations from the cloud and third-party services you use every day — Google Workspace, Microsoft 365, Slack, Box, GitHub, Salesforce, Zoom, and more — to find and fix any vulnerabilities in real-time. It searches logs to rapidly identify and remediate security issues in these SaaS applications without using your security analyst’s time.
As a producer of security logs, the vulnerabilities we find are reported to your security logging and monitoring infrastructure for tracking and follow-up.
Our service gives you safe remediation and recommendations to reduce your security risks right away. Try ThreatKey for free today.
Skip the intro call and get started now.
No time for an introductory call? We get it. That's why we have a simple, no-pressure way to get started with ThreatKey.
Just sign up for a free account and you can start using our platform immediately. No credit card required.