Understanding Log Levels and Rotation: DEBUG, INFO, WARNING, ERROR Explained

In a logging framework a “log level” is a numeric threshold that determines which messages are emitted to a handler. The level is attached to each log call (e.g., logger.debug()) and the logger compares the call’s numeric value with the configured threshold; only records whose value is greater than or equal to the threshold are passed through.

• DEBUG 10 – fine‑grained diagnostic data useful for tracing code paths.
• INFO 20 – normal operational milestones such as start‑up, shutdown, or successful completion of a task.
• WARNING 30 – recoverable anomalies that do not stop processing (e.g., missing optional configuration).
• ERROR 40 – an operation failed and the application must handle the failure.
• CRITICAL 50 – a fatal condition that prevents the application from continuing.

The numeric values are not arbitrary tags; they act as a filter. Setting logger.setLevel(logging.INFO) (or the equivalent in a configuration file) raises the threshold to 20, so DEBUG‑level messages (value 10) are silently dropped while INFO, WARNING, ERROR, and CRITICAL are recorded. Lowering the threshold to DEBUG during an incident instantly reveals the detailed trace without code changes.

import logging, logging.handlers

rotating = logging.handlers.RotatingFileHandler(
    "app.log", maxBytes=10*1024*1024, backupCount=5, encoding="utf-8"
)
stream = logging.StreamHandler()
logging.basicConfig(level=logging.INFO, handlers=[rotating, stream])

In the snippet above the logger is configured for INFO‑level output. All logger.debug() calls remain invisible during normal operation, yet they stay in the source code ready for activation when a developer temporarily lowers the level to DEBUG. This pattern lets teams embed exhaustive diagnostics without polluting production logs.

When designing log‑level policies for an enterprise system, consider the following steps:

• Define the business impact of each level and map it to compliance requirements (e.g., SOC 2 or ISO 27001 audit trails should capture at least WARNING and above).
• Configure production loggers at INFO or WARNING to limit noise and storage costs.
• Enable a dynamic mechanism (environment variable, feature flag, or admin UI) to raise the threshold to DEBUG on demand.
• Pair level filtering with handler configuration (rotating files, remote syslog, or in‑memory capture) to control both the vertical (detail) and horizontal (audience) flow of log data.

In a Python‑based service such as maintenance_agent.py, log levels act as a numeric filter that determines which messages survive the logger’s threshold. The configuration sets logging.INFO as the base level, so only records with a numeric value of 20 (INFO) or higher are emitted to the rotating file and the console. This design lets developers embed detailed DEBUG statements (19 calls in the codebase) without cluttering production logs, and later lower the threshold for troubleshooting.

Practical usage patterns emerge from the observed call distribution:

• INFO (135 calls) – records normal progress. Each site processed by the maintenance run logs steps such as “starting backup” or “updating plugins”, providing a chronological trace that operators can follow without overwhelming the log.
• WARNING (114 calls) – signals recoverable anomalies. For example, send_alert_email() logs a warning when SMTP settings are incomplete. The operation is skipped, but the overall maintenance workflow continues, so the condition is noteworthy but not fatal.
• ERROR (28 calls) – denotes genuine failures. Situations like a WP‑CLI command returning a non‑zero exit code or an exception during mail delivery are logged as errors because the specific task cannot be completed, yet the agent can still move on to the next site.
• CRITICAL (0 calls) – absent by design. The agent processes sites independently; a failure in one site triggers a rollback for that site only. There is no scenario where the entire application must abort, so the highest severity level is unnecessary. Urgent user notifications are handled through dedicated channels (e.g., email) rather than through a log level.

Because log levels are vertical filters, handler configuration provides the horizontal filter. The RotatingFileHandler caps the log size at 10 MB with five backups, ensuring that even a long‑running process cannot exhaust disk space while preserving recent history for post‑mortem analysis. An additional in‑memory handler (_SiteLogCapture) captures a single site’s log output for inclusion in a report, illustrating how the same log records can serve different audiences.

When designing log level usage for enterprise applications, follow these steps:

1. Define INFO as the default progress marker for every major business operation.
2. Reserve WARNING for conditions that are unexpected but do not halt the transaction flow.
3. Log ERROR only when an operation cannot be completed and corrective action is required.
4. Introduce CRITICAL only if the application has a single point of failure that forces a full shutdown.

Adhering to this hierarchy keeps log files concise, aligns severity with actual impact, and simplifies compliance with standards such as ISO 27001 or NIST, which require clear audit trails without unnecessary noise.

Python’s logging module uses numeric thresholds to decide which messages are emitted. The built‑in levels are DEBUG (10), INFO (20), WARNING (30), ERROR (40) and CRITICAL (50). Setting a logger’s level establishes a filter: only records with a numeric value equal to or higher than the threshold are passed to attached handlers.

In most production services the default threshold is logging.INFO. This hides the fine‑grained DEBUG statements that would otherwise clutter log files and consoles, while still recording normal progress, recoverable anomalies, and failures.

Typical configuration

import logging
from logging.handlers import RotatingFileHandler

# Create handlers
rotating_handler = RotatingFileHandler(
    "app.log", maxBytes=10*1024*1024, backupCount=5, encoding="utf-8"
)
stream_handler = logging.StreamHandler()

# Apply a global configuration
logging.basicConfig(
    level=logging.INFO,               # filter threshold
    handlers=[rotating_handler, stream_handler],
    format="%(asctime)s %(levelname)s %(name)s: %(message)s"
)

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)        # explicit, reinforces the threshold

With the level=logging.INFO argument, any call such as logger.debug("cache miss") is silently dropped. The same code base can retain dozens of logger.debug() statements for future investigations without affecting normal operation.

Why DEBUG is hidden by default

• Signal‑to‑noise reduction: Production logs focus on events that matter to operators (INFO‑CRITICAL).
• Performance: Formatting and I/O for low‑level messages are avoided when they are filtered out.
• Compliance: Standards such as ISO 27001 and NIST recommend limiting exposure of detailed internal state to reduce attack surface.

Enabling DEBUG for investigations

When an incident occurs, the threshold can be lowered without code changes:

# Temporary increase of detail
logger.setLevel(logging.DEBUG)
# or, for a single run
logging.getLogger().setLevel(logging.DEBUG)

After the investigation, the level is restored to INFO to resume normal logging volume. This dynamic adjustment is the practical payoff of level filtering: detailed diagnostic statements stay in the source permanently, remain invisible during routine operation, and become visible on demand.

The RotatingFileHandler from Python’s logging.handlers module is a simple yet reliable way to bound log growth on long‑running services. The handler is instantiated with two key parameters:

logging.handlers.RotatingFileHandler(
    "maintenance.log",
    maxBytes=10 * 1024 * 1024,   # 10 MB per file
    backupCount=5,
    encoding="utf-8"
)

How rotation works:

• When the active file maintenance.log reaches maxBytes (10 MB), the handler closes it and renames it to maintenance.log.1.
• A new empty maintenance.log is opened for subsequent writes.
• On the next overflow, the existing series are shifted: maintenance.log.1 becomes maintenance.log.2, maintenance.log.2 becomes maintenance.log.3, and so on.
• When a new generation would exceed backupCount (i.e., a .6 file), the oldest file is deleted before the rename occurs.

This renaming scheme preserves a linear history of the most recent log files while guaranteeing that no more than backupCount + 1 files exist at any time.

Disk‑usage bound:

• Each file is limited to 10 MB.
• The handler retains the active file plus five backups.
• Total possible footprint = 10 MB × (1 + 5) = 60 MB.

Because the size limit is enforced before any rename, the disk consumption never exceeds this bound, regardless of how long the process runs. This property is essential for compliance frameworks such as SOC 2 or ISO 27001, which require controls that prevent uncontrolled growth of audit artifacts.

Practical example for an enterprise service:

1. Configure the handler as shown above and attach it to the root logger.
2. Set the logger level to logging.INFO so that only informational, warning, error, and critical messages are persisted under normal operation.
3. During an incident, temporarily lower the level to logging.DEBUG to capture fine‑grained traces; the rotation mechanism will still enforce the 60 MB ceiling.

By combining level filtering with a bounded rotating file handler, engineers obtain both a manageable log volume and a retained window of recent activity useful for debugging, forensic analysis, and compliance reporting.

The _SiteLogCapture class is a lightweight subclass of logging.Handler whose sole purpose is to keep every log record generated during a single site‑maintenance run in an in‑memory list. Its implementation is intentionally minimal:

class _SiteLogCapture(logging.Handler):
    """Temporarily captures the log output for a single site's run."""
    def __init__(self):
        super().__init__()
        self.lines = []

    def emit(self, record):
        self.lines.append(self.format(record))

When a site’s workflow starts, the application attaches an instance of this handler to the root logger. All messages—regardless of level—are formatted and appended to self.lines. After the run finishes, the collected strings are inserted into the “execution log” section of a white‑label report or notification email, and the handler is removed, allowing the list to be garbage‑collected.

Vertical vs. horizontal filtering

• Vertical filtering (log level): The numeric level (DEBUG = 10, INFO = 20, etc.) acts as a threshold. Setting logger.setLevel(logging.INFO) discards DEBUG records for the entire logger hierarchy, reducing noise while preserving the ability to raise the threshold later.
• Horizontal filtering (handler selection): Each attached handler decides independently what to do with the same stream of records. The rotating file handler persists every record that passes the level filter to disk, whereas _SiteLogCapture keeps a transient copy for a specific downstream consumer.

Because the two filters operate orthogonally, engineers can fine‑tune both the granularity of information (vertical) and its audience or lifespan (horizontal). For example, a maintenance run might keep the global logger at INFO but temporarily lower the level to DEBUG while the _SiteLogCapture instance is active, ensuring the report contains full traceability without flooding the rotating log files.

Practical usage pattern

1. Create the capture handler and attach it:

   capture = _SiteLogCapture()
   logger.addHandler(capture)

2. Optionally lower the logger level for the duration:

   original_level = logger.level
   logger.setLevel(logging.DEBUG)

3. Run the site‑specific logic; all formatted messages are stored in capture.lines.
4. Restore the original level and detach the handler:

   logger.setLevel(original_level)
   logger.removeHandler(capture)

5. Insert "\n".join(capture.lines) into the report payload.

This pattern isolates per‑site diagnostics from the long‑term audit trail, satisfies compliance requirements such as SOC 2 or ISO 27001 (by keeping persistent logs immutable), and avoids unnecessary disk growth because the in‑memory buffer is discarded after each run.