Tried an hour to search some related contents and understand functions of pandas.
import os
data_file = '../data/results11.csv'
import pandas as pd
data = pd.read_csv(data_file)
print(data.head())
# calculate the max of the NaN numbers of all columns
m = max(data.isnull().sum(axis = 0))
data_dropmaxnan = data.dropna(axis = 1, thresh = len(data)+1-m)
thresh: Keep only the rows with at least len(data)+1-m non-NA values.
The colomn that has max NaN value has len(data)-m non-NA values.
We don’t want it, so plus 1.
For more:
How to get it by myself?
The loop of dataframe’s colomns is hard to write…
How to read the source code of pandas’ functions?
For example:dataframe.isnull().sum
I wanna figure out how to loop by dataframe’s colomns.
I also heard ‘apply’ function from https://towardsdatascience.com/how-to-make-your-pandas-loop-71-803-times-faster-805030df4f06.
Maybe I will try it next time.
I’m a Chinese student. If I have wrong expressions, please forgive me.
Because new users only can add two links.
Check answer to 2:
Hi @StevenJokes, I am confused by your question. Could you use 3 sentences the describe your question?
0.I have already done by pytorch’s api.
1.The best way to read pytorch’s source code?Please give me some tips.
2.how to loop by dataframe’s colomns?I’m trying to use loop to calculate data.isnull().sum().
Hi @StevenJokes,
1.The best way to read pytorch’s source code?Please give me some tips.
Here are some official API documents that may be helpful.
https://pytorch.org/tutorials/beginner/ptcheat.html
https://pytorch.org/docs/stable/index.html#
2. how to loop by dataframe’s colomns?I’m trying to use loop to calculate data.isnull().sum().
There are a vast amount of tutorials for pandas. You can just search online. Here is the official guide.
https://pandas.pydata.org/docs/user_guide/index.html#user-guide
Thanks.
I will read them later.
Now I have some issues about d2l/pytorch.py
I have renamed it as impytorch.py to avoid same name with package.
$ /usr/bin/env python "d:\onedrive\文档\read\d2l\d2l\imtorch.py"
Traceback (most recent call last):
File "d:\onedrive\文档\read\d2l\d2l\imtorch.py", line 22, in <module>
import torch
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
ctypes.CDLL(dll)
File "C:\ProgramData\Anaconda3\lib\ctypes\__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found
my conda info
active environment : pytorch
active env location : C:\Users\a8679\.conda\envs\pytorch
shell level : 1
user config file : C:\Users\a8679\.condarc
populated config files : C:\Users\a8679\.condarc
conda version : 4.8.3
conda-build version : 3.18.9
python version : 3.7.4.final.0
virtual packages :
base environment : C:\ProgramData\Anaconda3 (read only)
channel URLs : https://conda.anaconda.org/conda-forge/win-64
https://conda.anaconda.org/conda-forge/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud//pytorch/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud//pytorch/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/win-64
https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
package cache : C:\ProgramData\Anaconda3\pkgs
C:\Users\a8679\.conda\pkgs
C:\Users\a8679\AppData\Local\conda\conda\pkgs
envs directories : C:\Users\a8679\.conda\envs
C:\ProgramData\Anaconda3\envs
C:\Users\a8679\AppData\Local\conda\conda\envs
platform : win-64
user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.18362
administrator : False
netrc file : None
offline mode : False
Then I tried “conda install python” and “conda update --all”
I’m still waiting to updating.
Hope everything goes well.
After running pip install -U d2l -f https://d2l.ai/whl.html
,
I can directly run from d2l import torch as d2l
Thanks
But I’m confused about the bug when I directly run the imtorch.py(rename from d2l/torch.py)
$ /usr/bin/env python "d:\onedrive\文档\read\d2l\d2l\imtorch.py"
Traceback (most recent call last):
File "d:\onedrive\文档\read\d2l\d2l\imtorch.py", line 22, in <module>
import torch
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
ctypes.CDLL(dll)
File "C:\ProgramData\Anaconda3\lib\ctypes\__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found
import pandas as pd
data = pd.read_csv(data_file)
print(data)
Thresh=max(data.isnull().sum(axis=0))
print(Thresh)
pro_data=data.dropna(axis=1,thresh=data.shape[0]-Thresh+1)
print(pro_data)
PS: If you want to delete the ROW with most missing values, make changes listed:
Thresh=max(data.isnull().sum(axis=1))
pro_data=data.dropna(axis=0,thresh=data.shape[1]-Thresh+1)
Thanks for your answers which resolved my questions.
result_data = data.dropna(axis=1, thresh=min(data.count(axis=0))+1)
- Delete the column with the most missing values.
data = data.dropna(axis=1, how=any, thresh= len(data) -max(data.isnull().sum(axis=0))+1)
- Convert the preprocessed dataset to the tensor format.
inputs, outputs = data.iloc[:, 0:-1], data.iloc[:, -1]
inputs = inputs.fillna(inputs.mean())
X, y = torch.tensor(inputs.values), torch.tensor(outputs.values)
Here is another answer, I deal with the inputs because we can’t delete outputs anyway.
c = inputs.isna().sum().idxmax()
del inputs[c]
data2 = data2.iloc[:, data2.isna().sum().values < data2.isna().sum().max()]
So I have a question regarding the input data preprocessing.
If I had two biological sequences instead of NumRooms and Alley (as input, and no missing values), how would I convert them to tensors?
data.drop(data.columns[data.isnull().sum(axis=0).argmax()], axis=1) # delete the column with largest number of missing values
data.drop(data.index[data.isnull().sum(axis=1).argmax()], axis=0) # delete the row with largest number of missing values
both of the commands above will delete one column or one row even though there are some columns or rows that have the largest number of NAs
data.drop([pd.isnull(data).sum().idxmax()],axis=1)
In section 2.2.3. Conversion to the Tensor Format, the code uses the .values() method, but I believe (at least according to the pandas documentation) that .to_numpy method is now preferred.
The author should consider updating the code to use the pathlib API
import pathlib
dir_out = pathlib.Path().cwd()/'data'
dir_out.mkdir(parents=True, exist_ok=True)
file_new = dir_out / 'tiny.csv'
list_rows = [
'NumRooms,Alley,Price', # Column names / Header
'NA,Pave,127500', # Each row represents a data example
'2,NA,106000',
'4,NA,178100',
'NA,NA,140000',
]
with file_new.open(mode="w") as file:
for row in list_rows:
file.write(row + '\n')