Simple Random Sampling produces samples of a specific size, where each item has the same probability of being chosen. DataFu has scalable implementations of this that will generate samples …

Sampling: simple random sample with/without replacement, weighted sample, sample by keys Hashing: SHA and MD5 Link Analysis: PageRank Assorted Macros: deduplication of tables, human-readable …

Sampling Simple random sampling with or without replacement, weighted sampling. Link Analysis Run PageRank on a graph represented by a bag of nodes and edges. More Other useful methods like …

It takes a bag of n items and a sampling probability p as the inputs, and outputs a simple random sample of size exactly ceil (p*n) in a bag, with probability at least 99.99%.

It takes a sampling probability p as input and outputs a simple random sample of size exactly ceil (p*n) with probability at least 99.99%, where $n$ is the size of the population.

Sampling UDFs, including weighted sample, reservoir sampling, sampling by key, etc.

The method of sampling is to convert the key to a hash, derive a double value from this, and then test this against a supplied probability. The double value derived from a key is uniformly distributed …

datafu.pig.sampling. ReservoirSample (implements org.apache.pig.Algebraic) datafu.pig.sampling.WeightedReservoirSample datafu.pig.sampling. WeightedReservoirSample …

java.lang.Object org.apache.pig.EvalFunc<T> org.apache.pig.AccumulatorEvalFunc<org.apache.pig.data.DataBag> …

datafu.pig.sampling - package datafu.pig.sampling Sampling UDFs, including weighted sample, reservoir sampling, sampling by key, etc. datafu.pig.sessions - package datafu.pig.sessions