python教程—如何使用Spark查找中位数和分位数-Python实用宝典

# python教程—如何使用Spark查找中位数和分位数

1. 首先，我想做myrdd。sortBy(λx, x) ?
2. 接下来，我将找到rdd (rdd.count())的长度。
3. 最后，我想找到位于rdd中心的元素或两个元素。我也需要这个方法的帮助。

### 火花2.0 +:

Python <强> < /强>:

```df.approxQuantile("x", [0.5], 0.25)
```

Scala <强> < /强>:

```df.stat.approxQuantile("x", Array(0.5), 0.25)
```

```df.approxQuantile(["x", "y", "z"], [0.5], 0.25)
```

```df.approxQuantile(Array("x", "y", "z"), Array(0.5), 0.25)
```

### 火花& lt;2.0

Python <强> < /强>

```import numpy as np

np.random.seed(323)
rdd = sc.parallelize(np.random.randint(1000000, size=700000))

%time np.median(rdd.collect())
np.array(rdd.collect()).nbytes
```

```from numpy import floor
import time

def quantile(rdd, p, sample=None, seed=None):
"""Compute a quantile of order p ∈ [0, 1]
:rdd a numeric rdd
:p quantile(between 0 and 1)
:sample fraction of and rdd to use. If not provided we use a whole dataset
:seed random number generator seed to be used with sample
"""
assert 0 <= p <= 1
assert sample is None or 0 < sample <= 1

seed = seed if seed is not None else time.time()
rdd = rdd if sample is None else rdd.sample(False, sample, seed)

rddSortedWithIndex = (rdd.
sortBy(lambda x: x).
zipWithIndex().
map(lambda (x, i): (i, x)).
cache())

n = rddSortedWithIndex.count()
h = (n - 1) * p

rddX, rddXPlusOne = (
rddSortedWithIndex.lookup(x)[0]
for x in int(floor(h)) + np.array([0L, 1L]))

return rddX + (h - floor(h)) * (rddXPlusOne - rddX)
```

```np.median(rdd.collect()), quantile(rdd, 0.5)
## (500184.5, 500184.5)
np.percentile(rdd.collect(), 25), quantile(rdd, 0.25)
## (250506.75, 250506.75)
np.percentile(rdd.collect(), 75), quantile(rdd, 0.75)
(750069.25, 750069.25)
```

```from functools import partial
median = partial(quantile, p=0.5)
```

<强>语言无关 (Hive UDAF):

```rdd.map(lambda x: (float(x), )).toDF(["x"]).registerTempTable("df")

sqlContext.sql("SELECT percentile_approx(x, 0.5) FROM df")
```

```sqlContext.sql("SELECT percentile(x, 0.5) FROM df")
```

​Python实用宝典 (pythondict.com)