[SOLVED] segmentio/parquet fails to read files written with Julia and/or Python

Issue

This Content is from Stack Overflow. Question asked by Juanbuntu

H Folks,

The code below, which was taken almost verbatim from the segmentio/arrow Github.com site, fails to read .parquet files written with Python and/or Julia libs. When the code returns from the call to parquet.ReadFile(“file”) the rows contain 0 values for int64 or “” for strings. The reading fails with codec = {ZSTD, GZIP, or SNAPPY}

Thank you for your help…

type FiRowType struct{ x1, x2, x3 int64 }
func RdFiFile() {
    rows, err := parquet.ReadFile[FiRowType]("fileName_ZSTD.parquet")
    if err != nil {
        log.Fatal(err)
    }
    for _, c := range rows {
        fmt.Printf("%+vn", c)
    }
}

type FsRowType struct{ x1, x2, x3 string }
func RdFsFile() {
    rows, err := parquet.ReadFile[FsRowType]("fileName_ZSTD.parquet")
    if err != nil {
        log.Fatal(err)
    }
    for _, c := range rows {
        fmt.Printf("%+vn", c)
    }
}



Solution

In Go variables that start with a lower case character not exported so cannot be updated from other packages (e.g. segmentio/parquet-go). Try the below:

package main

import (
    "fmt"
    "log"

    "github.com/segmentio/parquet-go"
)

type FiRowType struct {
    X1 int64 `parquet:"x1,optional"`
    X2 int64 `parquet:"x2,optional"`
    X3 int64 `parquet:"x3,optional"`
}

func RdFiFile() {
    rows, err := parquet.ReadFile[FiRowType]("fi_ZSTD.parquet")
    if err != nil {
        log.Fatal(err)
    }
    for i, c := range rows {
        fmt.Printf("%d %+v\n", i, c)
    }
}

func main() {
    RdFiFile()
}


This Question was asked in StackOverflow by Juanbuntu and Answered by Brits It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?