[Solved] Can Pandas DataFrame efficiently calculate PMI (Pointwise Mutual Information)?

I’ve looked around and surprisingly haven’t found an easy use of framework or existing code for the calculation of Pointwise Mutual Information (Wiki PMI) despite libraries like Scikit-learn offering a metric for overall Mutual Information (by histogram). This is in the context of Python and Pandas!

My problem:

I have a DataFrame with a series of [x,y] examples in each row and wish to calculate a series of PMI values as per the formula (or a simpler one):

PMI(x, y) = log( p(x,y) / p(x) * p(y) )

So far my approach is:

def pmi_func(df, x, y):
    df['freq_x'] = df.groupby(x).transform('count')
    df['freq_y'] = df.groupby(y).transform('count')
    df['freq_x_y'] = df.groupby([x, y]).transform('count')
    df['pmi'] = np.log( df['freq_x_y'] / (df['freq_x'] * df['freq_y']) )

Would this give a valid and/or efficient computation?

Sample I/O:

x  y  PMI
0  0  0.176
0  0  0.176
0  1  0
Enquirer: jfive

||
Solution #1:

I would add three bits.

def pmi(dff, x, y):
    df = dff.copy()
    df['f_x'] = df.groupby(x)[x].transform('count')
    df['f_y'] = df.groupby(y)[y].transform('count')
    df['f_xy'] = df.groupby([x, y])[x].transform('count')
    df['pmi'] = np.log(len(df.index) * df['f_xy'] / (df['f_x'] * df['f_y']) )
    return df
  1. df.groupby(x)[x].transform('count') and df.groupby(y)[y].transform('count') should be used so that only
    count is retured.
  2. np.log(len(df.index) * df['f_xy'] / (df['f_x'] * df['f_y']) probabilities to be used.
  3. work on copy of dataframe, rather than modifying input dataframe.
Respondent: jfive
Solution #2:

Solution (with SKlearn KDE alternative as well):

Please comment for review

from sklearn.neighbors.kde import KernelDensity

# pmi function 
def pmi_func(df, x, y):
    freq_x = df.groupby(x).transform('count')
    freq_y = df.groupby(y).transform('count')
    freq_x_y = df.groupby([x, y]).transform('count')
    df['pmi'] = np.log( len(df.index) *  (freq_x_y / (freq_x * freq_y)) )

# pmi with kernel density estimation    
def kernel_pmi_func(df, x, y):
    # reshape data
    x = np.array(df[x])
    y = np.array(df[y])
    x_y = np.stack((x, y), axis=-1)

    # kernel density estimation
    kde_x = KernelDensity(kernel='gaussian', bandwidth=0.1).fit(x[:, np.newaxis])
    kde_y = KernelDensity(kernel='gaussian', bandwidth=0.1).fit(y[:, np.newaxis])
    kde_x_y = KernelDensity(kernel='gaussian', bandwidth=0.1).fit(x_y)

    # score
    p_x = pd.Series(np.exp(kde_x.score_samples(x[:, np.newaxis])))
    p_y = pd.Series(np.exp(kde_y.score_samples(y[:, np.newaxis])))
    p_x_y = pd.Series(np.exp(kde_x_y.score_samples(x_y)))   

    df['pmi'] = np.log( p_x_y / (p_x * p_y) )
Respondent: Zero
The answers/resolutions are collected from stackoverflow, are licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0 .

Leave a Reply

Your email address will not be published.