Recurrent Neural Network Implementation from Scratch

astonzhang · September 17, 2020, 3:23am

https://d2l.ai/chapter_recurrent-neural-networks/rnn-scratch.html

rongyu_wenju · November 20, 2020, 1:55pm

so, where is the code which have the function of detaching the gradient

astonzhang · November 20, 2020, 6:35pm

@terrytangyuan, in TF do we need to use https://www.tensorflow.org/guide/advanced_autodiff#stop_gradient ?

zhangjiekui · November 23, 2020, 9:08am

Why tensorflow version’s PPL keeps so high and bumpy? even sets lr=0.0001 and uses Adam optimizer?
Something goes wrong?

zhangjiekui · November 23, 2020, 2:27pm

I have fixed the bug, Just transpose Y accordingly (because we have transposed X):

Then the training result is normal! (perplexity =1.0)
see:

github.com

zhangjiekui/myNotes/blob/master/d2l/8.5. Implementation of Recurrent Neural Networks from Scratch_tensorflow_original.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from d2l import tensorflow as d2l\n",
    "import math\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "batch_size, num_steps = 32, 35\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "code",

This file has been truncated. show original

StevenJokess · November 26, 2020, 1:01pm

great. PR please: http://preview.d2l.ai/d2l-en/master/chapter_appendix-tools-for-deep-learning/contributing.html
https://github.com/d2l-ai/d2l-en/edit/master/chapter_recurrent-neural-networks/rnn-scratch.md
@zhangjiekui

zhangjiekui · December 1, 2020, 12:51pm

PR:

Quang_Tran · December 8, 2020, 7:11am

It was a nightmare when I read this chapter’s source code. Why did you make things so complicated?

miwl · July 26, 2022, 7:01am

I guess you missed a “$” sign in the 9.5.2.1. One-Hot Encoding section: It says “$5%”. (You might delete my comment afterwards, just wanted to let you know )