AWS Glue 6.0: 30% Lower Cost & Full Apache Iceberg v3 Support

AWS Glue 6.0 introduces a modernized, serverless runtime environment based on Apache Spark 4.1, Python 3.12, and Scala 2.13. This architectural shift prioritizes execution efficiency and simplifies complex data pipeline development. Notably, the release provides a 30% reduction in pricing compared to previous versions, impacting the cost structure for extract, transform, and load (ETL) workloads.

The runtime integrates full support for the Apache Iceberg v3 specification, powered by Iceberg 1.11.0. This implementation focuses on optimizing the storage and retrieval of semi-structured data through several key mechanisms:

• VARIANT Data Type with Shredding: Enables storage and querying of semi-structured formats (e.g., JSON or logs) without requiring flattened schemas, effectively reducing data duplication and parsing overhead.
• Geospatial Data Types: Native support for Geometry and Geography types to facilitate direct spatial processing within Spark.
• Nanosecond-Precision Timestamps: Supports high-frequency workloads, such as IoT telemetry or financial event streams, requiring sub-millisecond precision.
• Unknown Type Handling: Provides schema resilience by allowing pipelines to process and adapt to evolving or unexpected upstream data structures.

Engineers can leverage the modern runtime to address performance bottlenecks via Spark Declarative Pipelines and Arrow-native execution. Declarative pipelines allow users to define desired state transformations, offloading execution planning and optimization to the engine. Furthermore, the integration of Arrow-native Python User-Defined Functions (UDFs) and User-Defined Table Functions (UDTFs) removes serialization latency between the Python process and the JVM, yielding significant performance gains in custom transformation logic.

For stateless streaming requirements, AWS Glue 6.0 introduces a real-time streaming mode built on Spark 4.1. By utilizing Glue-optimized execution, this mode achieves single-digit millisecond latency, suitable for time-sensitive data routing and event processing. Implementation does not require API modifications; developers can transition existing jobs by updating the --glue-version parameter to 6.0 via the AWS CLI, SDK, or the Glue Studio console.

AWS Glue 6.0 is released with a pricing structure that is 30 percent lower than the rates applied to earlier Glue versions. The reduction applies uniformly across the core compute services—crawlers and ETL jobs—because the underlying runtime (Spark 4.1) and language stacks have been modernized, allowing the same workload to execute with fewer compute‑second charges.

Both crawlers (which scan data stores to infer schemas) and ETL jobs (which transform and load data) are billed on an hourly basis, but the measurement granularity is per second. This model eliminates the “rounded‑up‑to‑the‑hour” effect that can inflate costs for short‑lived tasks. The billing formula is:

• Cost = Rate × (Execution seconds ÷ 3 600)
• Rate is the per‑hour price for the selected Glue version (30 % lower for version 6.0).
• Execution seconds are accumulated across all parallel workers used by the job or crawler.

For example, a crawler that runs for 7 minutes (420 seconds) on a single DPU (Data Processing Unit) with a pre‑6.0 rate of $0.44 per hour would have costed $0.032. After the 30 % reduction, the same run costs $0.022, yielding a $0.010 saving per execution. Scaling to 100 daily crawls reduces the monthly spend from roughly $96 to $67, illustrating how the per‑second billing compounds savings at scale.

The AWS Glue Data Catalog is charged via a simplified monthly fee that covers metadata storage and access. The first 1 million objects stored and the first 1 million metadata accesses each month are free; beyond those thresholds a flat per‑object and per‑access charge applies. This model decouples catalog costs from compute, allowing teams to predict monthly expenses based on catalog growth rather than job runtime.

Key cost‑management points for engineers:

• Adopt Glue 6.0 to capture the 30 % price reduction without code changes (set --glue-version=6.0 in job definitions).
• Design crawlers and jobs to complete within the smallest practical duration to benefit from per‑second billing.
• Monitor catalog object count and access patterns; stay within the free‑tier thresholds when possible.
• Use AWS Cost Explorer or detailed billing reports to validate that the expected per‑second and per‑object rates are applied.

Apache Iceberg v3, integrated via the 1.11.0 specification, introduces architectural enhancements designed to handle evolving semi-structured data and high-fidelity temporal requirements. By moving beyond traditional column-level storage for complex objects, this specification optimizes retrieval for data-intensive workloads.

Key advancements in this release include:

• VARIANT Shredding: This feature enables the storage of semi-structured formats, such as JSON or event logs, by decomposing the data into shred-ready components. This eliminates the need for manual schema flattening and custom parsing logic. Unlike storing JSON as raw strings, shredding allows the engine to optimize query performance by accessing specific fields directly, reducing duplicate data and improving scan efficiency.
• Geometry & Geography Types: Native support for spatial data types facilitates direct geospatial processing and location-intelligence analytics. These types are optimized for spatial indexing, enabling engineers to execute complex geographic queries within the Spark runtime without external transformation layers.
• Nanosecond-Precision Timestamps: The specification expands temporal fidelity beyond the legacy millisecond standard. This is critical for high-frequency financial modeling, IoT sensor telemetry, and scientific computing workloads that require sub-microsecond event ordering and precision.
• Unknown Type Handling: To mitigate pipeline breakage caused by upstream schema drift, the specification includes robust handling for unrecognized data types. This allows ingestion pipelines to remain resilient, effectively isolating schema evolution from downstream query failures.

For implementation, engineers should leverage these types to replace opaque string-based columns. For example, migrating existing event-based string columns to the VARIANT type allows for immediate performance gains in filtering and projection. Similarly, transitioning from millisecond-based Epoch timestamps to nanosecond-precision types ensures alignment with modern high-frequency datasets, provided the downstream consuming applications are configured to interpret the extended precision scale. These updates provide a formalized mechanism for schema evolution, reducing the maintenance overhead traditionally associated with static data contracts.

Apache Spark 4.1 introduces a declarative pipeline model that separates the “what” of a transformation from the “how” of execution. Data engineers describe the desired schema and logical operations; the engine then constructs an optimal physical plan, reducing manual orchestration and allowing the optimizer to apply cost‑based decisions such as predicate push‑down and adaptive query execution. This model directly improves ETL throughput by minimizing unnecessary shuffles and by re‑using cached intermediate results.

Python 3.12, the default interpreter for the runtime, brings a more efficient bytecode compiler and reduced interpreter overhead. In the context of PySpark, these interpreter gains combine with Spark 4.1’s Arrow‑native execution path. Arrow eliminates the Java‑Python serialization step that previously dominated the latency of user‑defined functions (UDFs) and user‑defined table functions (UDTFs). The result is a measurable reduction in per‑record processing time for complex Python logic.

Scala 2.13 upgrades the language’s collection library, offering immutable data structures with lower allocation costs and more expressive pattern‑matching syntax. Since Spark’s core engine is written in Scala, the newer compiler produces tighter bytecode, which translates into lower JVM pause times and more predictable garbage‑collection behavior—critical for long‑running batch jobs.

• Performance impact: Declarative pipelines + Arrow‑native UDFs reduce end‑to‑end job latency.
• Resource efficiency: Scala 2.13’s collection improvements lower heap pressure, allowing higher task parallelism per executor.
• Streaming capability: Spark 4.1’s real‑time mode delivers single‑digit millisecond latency for stateless streams, enabling low‑latency event routing without external brokers.

Practical example – an Arrow‑native Python UDF that parses a JSON column and extracts a nested field:

from pyspark.sql import functions as F
import pyarrow as pa

@F.udf(returnType="string", useArrow=True)
def extract_user_id(json_str):
    # Arrow handles zero‑copy conversion from JVM to Python
    table = pa.json.read_json(pa.BufferReader(json_str))
    return table.column("user").field("id").to_pylist()[0]
    
df = spark.read.format("iceberg").load("catalog.db.events")
result = df.withColumn("user_id", extract_user_id(F.col("payload")))

In this snippet, the useArrow=True flag activates the Arrow‑native path, avoiding the costly Java‑Python marshaling step described in the runtime documentation. When combined with Spark 4.1’s optimizer, the transformation is pushed down to the executor level, preserving the low‑latency guarantees required for real‑time pipelines.

Performance in modern ETL workflows is often bottlenecked by data serialization between the Python runtime and the JVM. AWS Glue 6.0 addresses this through Arrow-native execution for Python User-Defined Functions (UDFs) and User-Defined Table Functions (UDTFs). By utilizing the Apache Arrow memory format, the engine eliminates the traditional serialization overhead, allowing complex transformations to execute with higher throughput directly on the Spark executor memory space.

To further reduce developer friction, Spark Declarative Pipelines shift the focus from imperative task management to declarative transformation logic. Instead of manually orchestrating the execution graph, engineers define the target state of the data. The underlying engine then performs:

• Automated execution order determination.
• Dynamic optimization of the physical plan.
• Reduced manual orchestration overhead, minimizing potential for pipeline logic errors.

For workloads requiring high-frequency processing, AWS Glue 6.0 introduces a real-time streaming mode built on Spark 4.1. This mode leverages Glue-optimized execution to achieve single-digit millisecond latency, making it suitable for stateless event processing and time-sensitive routing. This is particularly effective for:

• Low-latency transformation: Processing event streams without the overhead of micro-batching.
• Data routing: Immediate redirection of IoT or telemetry data based on real-time payload analysis.

Engineers can implement these features by selecting 6.0 as the --glue-version parameter. The transition to Arrow-native UDFs requires no API changes, provided the environment supports the necessary Arrow dependencies. By abstracting the execution plan through Declarative Pipelines and minimizing latency via the new streaming mode, Glue 6.0 allows for more predictable performance profiles in large-scale Spark 4.1 deployments. When migrating existing jobs, utilizing the Spark upgrade agent within Glue Studio is recommended to ensure compatibility with these modernized runtime components.