
This blog breaks down how log levels serve as filtering thresholds, why applications choose specific levels, and how log rotation keeps files from growing without bound.
What Are Log Levels and Why They Matter
In a logging framework a âlog levelâ is a numeric threshold that determines which messages are emitted to a handler. The level is attached to each log call (e.g., logger.debug()) and the logger compares the callâs numeric value with the configured threshold; only records whose value is greater than or equal to the threshold are passed through.
- DEBUGâŻ10 â fineâgrained diagnostic data useful for tracing code paths.
- INFOâŻ20 â normal operational milestones such as startâup, shutdown, or successful completion of a task.
- WARNINGâŻ30 â recoverable anomalies that do not stop processing (e.g., missing optional configuration).
- ERRORâŻ40 â an operation failed and the application must handle the failure.
- CRITICALâŻ50 â a fatal condition that prevents the application from continuing.
The numeric values are not arbitrary tags; they act as a filter. Setting logger.setLevel(logging.INFO) (or the equivalent in a configuration file) raises the threshold toâŻ20, so DEBUGâlevel messages (valueâŻ10) are silently dropped while INFO, WARNING, ERROR, and CRITICAL are recorded. Lowering the threshold to DEBUG during an incident instantly reveals the detailed trace without code changes.
import logging, logging.handlers
rotating = logging.handlers.RotatingFileHandler(
"app.log", maxBytes=10*1024*1024, backupCount=5, encoding="utf-8"
)
stream = logging.StreamHandler()
logging.basicConfig(level=logging.INFO, handlers=[rotating, stream])
In the snippet above the logger is configured for INFOâlevel output. All logger.debug() calls remain invisible during normal operation, yet they stay in the source code ready for activation when a developer temporarily lowers the level to DEBUG. This pattern lets teams embed exhaustive diagnostics without polluting production logs.
When designing logâlevel policies for an enterprise system, consider the following steps:
- Define the business impact of each level and map it to compliance requirements (e.g., SOCâŻ2 or ISOâŻ27001 audit trails should capture at least WARNING and above).
- Configure production loggers at INFO or WARNING to limit noise and storage costs.
- Enable a dynamic mechanism (environment variable, feature flag, or admin UI) to raise the threshold to DEBUG on demand.
- Pair level filtering with handler configuration (rotating files, remote syslog, or inâmemory capture) to control both the vertical (detail) and horizontal (audience) flow of log data.
Designing Log Level Usage in Real Applications
In a Pythonâbased service such as maintenance_agent.py, log levels act as a numeric filter that determines which messages survive the loggerâs threshold. The configuration sets logging.INFO as the base level, so only records with a numeric value of 20 (INFO) or higher are emitted to the rotating file and the console. This design lets developers embed detailed DEBUG statements (19 calls in the codebase) without cluttering production logs, and later lower the threshold for troubleshooting.
Practical usage patterns emerge from the observed call distribution:
- INFO (135 calls) â records normal progress. Each site processed by the maintenance run logs steps such as âstarting backupâ or âupdating pluginsâ, providing a chronological trace that operators can follow without overwhelming the log.
- WARNING (114 calls) â signals recoverable anomalies. For example,
send_alert_email()logs a warning when SMTP settings are incomplete. The operation is skipped, but the overall maintenance workflow continues, so the condition is noteworthy but not fatal. - ERROR (28 calls) â denotes genuine failures. Situations like a WPâCLI command returning a nonâzero exit code or an exception during mail delivery are logged as errors because the specific task cannot be completed, yet the agent can still move on to the next site.
- CRITICAL (0 calls) â absent by design. The agent processes sites independently; a failure in one site triggers a rollback for that site only. There is no scenario where the entire application must abort, so the highest severity level is unnecessary. Urgent user notifications are handled through dedicated channels (e.g., email) rather than through a log level.
Because log levels are vertical filters, handler configuration provides the horizontal filter. The RotatingFileHandler caps the log size at 10âŻMB with five backups, ensuring that even a longârunning process cannot exhaust disk space while preserving recent history for postâmortem analysis. An additional inâmemory handler (_SiteLogCapture) captures a single siteâs log output for inclusion in a report, illustrating how the same log records can serve different audiences.
When designing log level usage for enterprise applications, follow these steps:
- Define
INFOas the default progress marker for every major business operation. - Reserve
WARNINGfor conditions that are unexpected but do not halt the transaction flow. - Log
ERRORonly when an operation cannot be completed and corrective action is required. - Introduce
CRITICALonly if the application has a single point of failure that forces a full shutdown.
Adhering to this hierarchy keeps log files concise, aligns severity with actual impact, and simplifies compliance with standards such as ISOâŻ27001 or NIST, which require clear audit trails without unnecessary noise.
Implementing Log Level Filtering in Python
Pythonâs logging module uses numeric thresholds to decide which messages are emitted. The builtâin levels are DEBUG (10), INFO (20), WARNING (30), ERROR (40) and CRITICAL (50). Setting a loggerâs level establishes a filter: only records with a numeric value equal to or higher than the threshold are passed to attached handlers.
In most production services the default threshold is logging.INFO. This hides the fineâgrained DEBUG statements that would otherwise clutter log files and consoles, while still recording normal progress, recoverable anomalies, and failures.
Typical configuration
import logging
from logging.handlers import RotatingFileHandler
# Create handlers
rotating_handler = RotatingFileHandler(
"app.log", maxBytes=10*1024*1024, backupCount=5, encoding="utf-8"
)
stream_handler = logging.StreamHandler()
# Apply a global configuration
logging.basicConfig(
level=logging.INFO, # filter threshold
handlers=[rotating_handler, stream_handler],
format="%(asctime)s %(levelname)s %(name)s: %(message)s"
)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO) # explicit, reinforces the threshold
With the level=logging.INFO argument, any call such as logger.debug("cache miss") is silently dropped. The same code base can retain dozens of logger.debug() statements for future investigations without affecting normal operation.
Why DEBUG is hidden by default
- Signalâtoânoise reduction: Production logs focus on events that matter to operators (INFOâCRITICAL).
- Performance: Formatting and I/O for lowâlevel messages are avoided when they are filtered out.
- Compliance: Standards such as ISOâŻ27001 and NIST recommend limiting exposure of detailed internal state to reduce attack surface.
Enabling DEBUG for investigations
When an incident occurs, the threshold can be lowered without code changes:
# Temporary increase of detail
logger.setLevel(logging.DEBUG)
# or, for a single run
logging.getLogger().setLevel(logging.DEBUG)
After the investigation, the level is restored to INFO to resume normal logging volume. This dynamic adjustment is the practical payoff of level filtering: detailed diagnostic statements stay in the source permanently, remain invisible during routine operation, and become visible on demand.
Log Rotation: Keeping Logs Bounded
The RotatingFileHandler from Pythonâs logging.handlers module is a simple yet reliable way to bound log growth on longârunning services. The handler is instantiated with two key parameters:
logging.handlers.RotatingFileHandler(
"maintenance.log",
maxBytes=10 * 1024 * 1024, # 10âŻMB per file
backupCount=5,
encoding="utf-8"
)
How rotation works:
- When the active file
maintenance.logreachesmaxBytes(10âŻMB), the handler closes it and renames it tomaintenance.log.1. - A new empty
maintenance.logis opened for subsequent writes. - On the next overflow, the existing series are shifted:
maintenance.log.1becomesmaintenance.log.2,maintenance.log.2becomesmaintenance.log.3, and so on. - When a new generation would exceed
backupCount(i.e., a.6file), the oldest file is deleted before the rename occurs.
This renaming scheme preserves a linear history of the most recent log files while guaranteeing that no more than backupCount + 1 files exist at any time.
Diskâusage bound:
- Each file is limited to 10âŻMB.
- The handler retains the active file plus five backups.
- Total possible footprint = 10âŻMB Ă (1âŻ+âŻ5) = 60âŻMB.
Because the size limit is enforced before any rename, the disk consumption never exceeds this bound, regardless of how long the process runs. This property is essential for compliance frameworks such as SOCâŻ2 or ISOâŻ27001, which require controls that prevent uncontrolled growth of audit artifacts.
Practical example for an enterprise service:
- Configure the handler as shown above and attach it to the root logger.
- Set the logger level to
logging.INFOso that only informational, warning, error, and critical messages are persisted under normal operation. - During an incident, temporarily lower the level to
logging.DEBUGto capture fineâgrained traces; the rotation mechanism will still enforce the 60âŻMB ceiling.
By combining level filtering with a bounded rotating file handler, engineers obtain both a manageable log volume and a retained window of recent activity useful for debugging, forensic analysis, and compliance reporting.
Advanced Handlers: Capturing Logs for Specific Purposes
The _SiteLogCapture class is a lightweight subclass of logging.Handler whose sole purpose is to keep every log record generated during a single siteâmaintenance run in an inâmemory list. Its implementation is intentionally minimal:
class _SiteLogCapture(logging.Handler):
"""Temporarily captures the log output for a single site's run."""
def __init__(self):
super().__init__()
self.lines = []
def emit(self, record):
self.lines.append(self.format(record))
When a siteâs workflow starts, the application attaches an instance of this handler to the root logger. All messagesâregardless of levelâare formatted and appended to self.lines. After the run finishes, the collected strings are inserted into the âexecution logâ section of a whiteâlabel report or notification email, and the handler is removed, allowing the list to be garbageâcollected.
Vertical vs. horizontal filtering
- Vertical filtering (log level): The numeric level (DEBUGâŻ=âŻ10, INFOâŻ=âŻ20, etc.) acts as a threshold. Setting
logger.setLevel(logging.INFO)discards DEBUG records for the entire logger hierarchy, reducing noise while preserving the ability to raise the threshold later. - Horizontal filtering (handler selection): Each attached handler decides independently what to do with the same stream of records. The rotating file handler persists every record that passes the level filter to disk, whereas
_SiteLogCapturekeeps a transient copy for a specific downstream consumer.
Because the two filters operate orthogonally, engineers can fineâtune both the granularity of information (vertical) and its audience or lifespan (horizontal). For example, a maintenance run might keep the global logger at INFO but temporarily lower the level to DEBUG while the _SiteLogCapture instance is active, ensuring the report contains full traceability without flooding the rotating log files.
Practical usage pattern
- Create the capture handler and attach it:
capture = _SiteLogCapture() logger.addHandler(capture) - Optionally lower the logger level for the duration:
original_level = logger.level logger.setLevel(logging.DEBUG) - Run the siteâspecific logic; all formatted messages are stored in
capture.lines. - Restore the original level and detach the handler:
logger.setLevel(original_level) logger.removeHandler(capture) - Insert
"\n".join(capture.lines)into the report payload.
This pattern isolates perâsite diagnostics from the longâterm audit trail, satisfies compliance requirements such as SOCâŻ2 or ISOâŻ27001 (by keeping persistent logs immutable), and avoids unnecessary disk growth because the inâmemory buffer is discarded after each run.
Looking for Custom Software, SaaS, or Dedicated Developers?
Appworks Technologies designs, builds, and scales production enterprise platforms, microservices, and multi-tenant SaaS products tailored to your business goals.
Editorial Policy & Research Methodology
Our findings are based on rigorous internal research, verified industry benchmarks, and direct technical implementation experience from our enterprise client projects. All statistics and technical claims are reviewed by senior engineers before publication to ensure accuracy, transparency, and helpfulness for our readers.
