Each Answer to this Q is separated by one/two green lines.
I want to use the Pandas dataframe to breakdown the variance in one variable.
For example, if I have a column called ‘Degrees’, and I have this indexed for various dates, cities, and night vs. day, I want to find out what fraction of the variation in this series is coming from cross-sectional city variation, how much is coming from time series variation, and how much is coming from night vs. day.
In Stata I would use Fixed effects and look at the R^2. Hopefully my question makes sense.
Basically, what I want to do, is find the ANOVA breakdown of “Degrees” by three other columns.
I set up a direct comparison to test them, found that their assumptions can differ slightly , got a hint from a statistician, and here is an example of ANOVA on a pandas dataframe matching R’s results:
import pandas as pd import statsmodels.api as sm from statsmodels.formula.api import ols # R code on R sample dataset #> anova(with(ChickWeight, lm(weight ~ Time + Diet))) #Analysis of Variance Table # #Response: weight # Df Sum Sq Mean Sq F value Pr(>F) #Time 1 2042344 2042344 1576.460 < 2.2e-16 *** #Diet 3 129876 43292 33.417 < 2.2e-16 *** #Residuals 573 742336 1296 #write.csv(file="ChickWeight.csv", x=ChickWeight, row.names=F) cw = pd.read_csv('ChickWeight.csv') cw_lm=ols('weight ~ Time + C(Diet)', data=cw).fit() #Specify C for Categorical print(sm.stats.anova_lm(cw_lm, typ=2)) # sum_sq df F PR(>F) #C(Diet) 129876.056995 3 33.416570 6.473189e-20 #Time 2016357.148493 1 1556.400956 1.803038e-165 #Residual 742336.119560 573 NaN NaN