Is there a way to select random rows from a DataFrame in Pandas.

In R, using the car package, there is a useful function some(x, n) which is similar to head but selects, in this example, 10 rows at random from x.

I have also looked at the slicing documentation and there seems to be nothing equivalent.

Update

Now using version 20. There is a sample method.

df.sample(n)

Something like this?

import random

def some(x, n):
    return x.ix[random.sample(x.index, n)]

Note: As of Pandas v0.20.0, ix has been deprecated in favour of loc for label based indexing.

With pandas version 0.16.1 and up, there is now a DataFrame.sample method built-in:

import pandas

df = pandas.DataFrame(pandas.np.random.random(100))

# Randomly sample 70% of your dataframe
df_percent = df.sample(frac=0.7)

# Randomly sample 7 elements from your dataframe
df_elements = df.sample(n=7)

For either approach above, you can get the rest of the rows by doing:

df_rest = df.loc[~df.index.isin(df_percent.index)]

sample

As of v0.20.0, you can use pd.DataFrame.sample, which can be used to return a random sample of a fixed number rows, or a percentage of rows:

df = df.sample(n=k)     # k rows
df = df.sample(frac=k)  # int(len(df.index) * k) rows

For reproducibility, you can specify an integer random_state, equivalent to using np.ramdom.seed. So, instead of setting, for example, np.random.seed = 0, you can:

df = df.sample(n=k, random_state=0)

The best way to do this is with the sample function from the random module,

import numpy as np
import pandas as pd
from random import sample

# given data frame df

# create random index
rindex =  np.array(sample(xrange(len(df)), 10))

# get 10 random rows from df
dfr = df.ix[rindex]

Actually this will give you repeated indices np.random.random_integers(0, len(df), N) where N is a large number.

Below line will randomly select n number of rows out of the total existing row numbers from the dataframe df without replacement.

df=df.take(np.random.permutation(len(df))[:n])

I am trying to determine whether there is an entry in a Pandas column that has a particular value. I tried to do this with if x in df['id']. I thought this was working, except when I fed it a value that I knew was not in the column 43 in df['id'] it still returned True. When I subset to a data frame only containing entries matching the missing id df[df['id'] == 43] there are, obviously, no entries in it. How to I determine if a column in a Pandas data frame contains a particular value and why doesn’t my current method work? (FYI, I have the same problem when I use the implementation in this answer to a similar question).

in of a Series checks whether the value is in the index:

In [11]: s = pd.Series(list('abc'))

In [12]: s
Out[12]: 
0    a
1    b
2    c
dtype: object

In [13]: 1 in s
Out[13]: True

In [14]: 'a' in s
Out[14]: False

One option is to see if it’s in unique values:

In [21]: s.unique()
Out[21]: array(['a', 'b', 'c'], dtype=object)

In [22]: 'a' in s.unique()
Out[22]: True

or a python set:

In [23]: set(s)
Out[23]: {'a', 'b', 'c'}

In [24]: 'a' in set(s)
Out[24]: True

As pointed out by @DSM, it may be more efficient (especially if you’re just doing this for one value) to just use in directly on the values:

In [31]: s.values
Out[31]: array(['a', 'b', 'c'], dtype=object)

In [32]: 'a' in s.values
Out[32]: True

You can also use pandas.Series.isin although it’s a little bit longer than 'a' in s.values:

In [2]: s = pd.Series(list('abc'))

In [3]: s
Out[3]: 
0    a
1    b
2    c
dtype: object

In [3]: s.isin(['a'])
Out[3]: 
0    True
1    False
2    False
dtype: bool

In [4]: s[s.isin(['a'])].empty
Out[4]: False

In [5]: s[s.isin(['z'])].empty
Out[5]: True

But this approach can be more flexible if you need to match multiple values at once for a DataFrame (see DataFrame.isin)

>>> df = DataFrame({'A': [1, 2, 3], 'B': [1, 4, 7]})
>>> df.isin({'A': [1, 3], 'B': [4, 7, 12]})
       A      B
0   True  False  # Note that B didn't match 1 here.
1  False   True
2   True   True

found = df[df['Column'].str.contains('Text_to_search')]
print(found.count())

the found.count() will contains number of matches

And if it is 0 then means string was not found in the Column.

I did a few simple tests:

In [10]: x = pd.Series(range(1000000))

In [13]: timeit 999999 in x.values
567 µs ± 25.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [15]: timeit x.isin([999999]).any()
9.54 ms ± 291 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [16]: timeit (x == 999999).any()
6.86 ms ± 107 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [17]: timeit 999999 in set(x)
79.8 ms ± 1.98 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [21]: timeit x.eq(999999).any()
7.03 ms ± 33.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [22]: timeit x.eq(9).any()
7.04 ms ± 60 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [24]: timeit 9 in x.values
666 µs ± 15.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Interestingly it doesn’t matter if you look up 9 or 999999, it seems like it takes about the same amount of time using the in syntax (must be using binary search)

In [24]: timeit 9 in x.values
666 µs ± 15.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [25]: timeit 9999 in x.values
647 µs ± 5.21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [26]: timeit 999999 in x.values
642 µs ± 2.11 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [27]: timeit 99199 in x.values
644 µs ± 5.31 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [28]: timeit 1 in x.values
667 µs ± 20.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Seems like using x.values is the fastest, but maybe there is a more elegant way in pandas?

Or use Series.tolist or Series.any:

>>> s = pd.Series(list('abc'))
>>> s
0    a
1    b
2    c
dtype: object
>>> 'a' in s.tolist()
True
>>> (s=='a').any()
True

Series.tolist makes a list about of a Series, and the other one i am just getting a boolean Series from a regular Series, then checking if there are any Trues in the boolean Series.

Simple condition:

if any(str(elem) in ['a','b'] for elem in df['column'].tolist()):

Use

df[df['id']==x].index.tolist()

If x is present in id then it’ll return the list of indices where it is present, else it gives an empty list.

I don’t suggest to use “value in series”, which can lead many errors. Please see this answer for detail: Using in operator with Pandas series

Suppose you dataframe looks like :

Now you want to check if filename “80900026941984” is present in the dataframe or not.

You can simply write :

if sum(df["filename"].astype("str").str.contains("80900026941984")) > 0:
    print("found")

I can’t get the average or mean of a column in pandas. A have a dataframe. Neither of things I tried below gives me the average of the column weight

>>> allDF 
         ID           birthyear  weight
0        619040       1962       0.1231231
1        600161       1963       0.981742
2      25602033       1963       1.3123124     
3        624870       1987       0.94212

The following returns several values, not one:

allDF[['weight']].mean(axis=1)

So does this:

allDF.groupby('weight').mean()

If you only want the mean of the weight column, select the column (which is a Series) and call .mean():

In [479]: df
Out[479]: 
         ID  birthyear    weight
0    619040       1962  0.123123
1    600161       1963  0.981742
2  25602033       1963  1.312312
3    624870       1987  0.942120

In [480]: df["weight"].mean()
Out[480]: 0.83982437500000007

Try df.mean(axis=0) , axis=0 argument calculates the column wise mean of the dataframe so the result will be axis=1 is row wise mean so you are getting multiple values.

Do try to give print (df.describe()) a shot. I hope it will be very helpful to get an overall description of your dataframe.

you can use

df.describe() 

you will get basic statistics of the dataframe and to get mean of specific column you can use

df["columnname"].mean()

You can also access a column using the dot notation (also called attribute access) and then calculate its mean:

df.your_column_name.mean()

Mean for each column in df :

    A   B   C
0   5   3   8
1   5   3   9
2   8   4   9

df.mean()

A    6.000000
B    3.333333
C    8.666667
dtype: float64

and if you want average of all columns:

df.stack().mean()
6.0

Additionally if you want to get the round value after finding the mean.

#Create a DataFrame
df1 = {
    'Subject':['semester1','semester2','semester3','semester4','semester1',
               'semester2','semester3'],
   'Score':[62.73,47.76,55.61,74.67,31.55,77.31,85.47]}
df1 = pd.DataFrame(df1,columns=['Subject','Score'])

rounded_mean = round(df1['Score'].mean()) # specified nothing as decimal place
print(rounded_mean) # 62

rounded_mean_decimal_0 = round(df1['Score'].mean(), 0) # specified decimal place as 0
print(rounded_mean_decimal_0) # 62.0

rounded_mean_decimal_1 = round(df1['Score'].mean(), 1) # specified decimal place as 1
print(rounded_mean_decimal_1) # 62.2

You can use either of the two statements below:

numpy.mean(df['col_name'])
# or
df['col_name'].mean()

You can easily followthe following code
    `import pandas as pd 
    import numpy as np 

    classxii = {'Name':['Karan','Ishan','Aditya','Anant','Ronit'],
        'Subject':['Accounts','Economics','Accounts','Economics','Accounts'],
        'Score':[87,64,58,74,87],
        'Grade':['A1','B2','C1','B1','A2']}
    df = pd.DataFrame(classxii,index = ['a','b','c','d','e'],columns=['Name','Subject','Score','Grade'])
    print(df)
    #use the below for mean if you already have a dataframe
print('mean of score is:')
print(df[['Score']].mean())

You can simply go for: df.describe() that will provide you with all the relevant details you need, but to find the min, max or average value of a particular column (say ‘weights’ in your case), use:

    df['weights'].mean(): For average value
    df['weights'].max(): For maximum value
    df['weights'].min(): For minimum value