Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43273

Support lz4raw compression codec for Parquet

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0
    • 3.5.0
    • SQL
    • None

    Description

      hadoop-parquet version should be updated to 1.3.0 (together with other parquet-mr libs)

      java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1.0 (TID 3) (f2b63fdfa0a6 executor driver): java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZ4_RAW
          at java.base/java.lang.Enum.valueOf(Enum.java:273)
          at org.apache.parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:26)
          at org.apache.parquet.format.converter.ParquetMetadataConverter.fromFormatCodec(ParquetMetadataConverter.java:636)
      ... 

      Attachments

        Issue Links

          Activity

            People

              yumwang Yuming Wang
              ei-grad Andrew Grigorev
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: