最近在学习新的模型中遇到读取pkl文件的问题

最近在github上学习新模型时遇到了作者写的:

def prepare_dataloader(opt):
“”" Load data and prepare dataloader. “”"

def load_data(name, dict_name):
    with open(name, 'rb') as f:
        data = pickle.load(f, encoding='latin-1')
        num_types = data['dim_process']
        data = data[dict_name]
        return data, int(num_types)

print('[Info] Loading train data...')
train_data, num_types = load_data(opt.data + 'train.pkl', 'train')
print('[Info] Loading dev data...')
dev_data, _ = load_data(opt.data + 'dev.pkl', 'dev')
print('[Info] Loading test data...')
test_data, _ = load_data(opt.data + 'test.pkl', 'test')

trainloader = get_dataloader(train_data, opt.batch_size, shuffle=True)
testloader = get_dataloader(test_data, opt.batch_size, shuffle=False)
return trainloader, testloader, num_types

我反复确认作者给的数据是CSV文件,但是作者的源码写的是读取pkl文件,我也尝试直接将csv文件直接导入进去是行不通,请问各位是我理解错了吗,还是有什么方法可以改变。

1 Like

I also face the same problem.
TypeError: cannot pickle ‘weakref’ object
train_iter, test_iter = load_data_fashion_mnist(32, resize=64)

想问你下代码里面的data[‘dim_process’]与data[dict_name] 这两个字段都是什么?