I want to convert a DataFrame to SparseDataFrame before pivoting it (when it gets really sparse, see also this discussion ). I have a textual key, which I need to keep ("chr"):
df = pd.DataFrame( list(zip([3,2,4,1,5,3,2],
["chr1", "chr1", "chr1", "chr1", "chr2", "chr2", "chr3"],
[100,100, 100, 200, 1,3,1],
[True, True, True, False, True, False, True],
[-1,0,1,3, 0,2,1])) ,
columns = ["counts", "chr", "pos", "strand", "distance"])
df.iloc[:,1:].dtypes
Out[]:
chr object
pos int64
strand bool
distance int64
dtype: object
For this small table it works well with regular DataFrame:
pd.pivot_table(df, index= [ "chr", "pos"], columns= ["strand","distance"], values= "counts").fillna(0)
strand False True
distance 2 3 -1 0 1
chr pos
chr1 100 0 0 3 2 4
200 0 1 0 0 0
chr2 1 0 0 0 5 0
3 3 0 0 0 0
chr3 1 0 0 0 0 2
But I need to do it on much larger matrices. So I tried to do following trick:
dfpiv = pd.pivot_table(pd.SparseDataFrame(df), index= [ "chr", "pos"], columns= ["strand","distance"], values= "counts")
but I am getting:
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
Are there any plans to include a functionality option into pivot function for automatic conversion into SparseDataFrame?
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4