Issue

This Content is from Stack Overflow. Question asked by Daniel Tan

I’d like to get create a random sub-sample of my data.

Spark’s sample function (link) is the API I’d like to use. Particularly, because it allows me to toggle if the sampling is done with or without replacement. However, executing this function takes a long time. Based on the answers from this question Spark sample is too slow, it seems like sample requires a full table scan.
TABLESAMPLE seems like a faster alternative, albeit, the ability to toggle with and without replacement is lost.

I’d like to understand how sample and TABLESAMPLE are different, and why does TABLESAMPLE execute faster than sample. Could it be that TABLESAMPLE does not require a full table scan?

Solution

Limit is definitely worth removing, but the real problem is that sampling requires a full data scan. No matter how low is the fraction, the time complexity is still O(N)*.

If you don’t require good statistical properties, you can try to limit amount of data you’ve loaded in the first place by sampling data files first, and then subsampling from the reduced dataset. This might work reasonably well, if data is uniformly distributed.

Otherwise there is not much you can do about it, other than scaling your cluster.

* How do simple random sampling and dataframe SAMPLE function work in Apache Spark (Scala)?

This Question was asked in StackOverflow by fmv1992 and Answered by zero323 It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.