There is the following issue on this page: https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html.
There is no non-linearity wrapper while passing the hidden state onto the next iteration, and on output layer that predicts distribution. Results are coming relatively fine though, maybe because nn.LogSoftmax is implicitly acting as non-linearity. But for learning hidden states, its bad, $h_t$ is literally linear combination of $h_{t-1}$, all past $x$ s and $Cat$.
Simple Fix:
hidden = torch.tanh(self.i2h(input_combined))
output = self.i2o(input_combined)
output_combined = torch.cat((hidden, output), 1)
output = torch.tanh(self.o2o(output_combined))
Thank you.
There is the following issue on this page: https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html.
There is no non-linearity wrapper while passing the hidden state onto the next iteration, and on output layer that predicts distribution. Results are coming relatively fine though, maybe because$h_t$ is literally linear combination of $h_{t-1}$ , all past $x$ s and $Cat$ .
nn.LogSoftmaxis implicitly acting as non-linearity. But for learning hidden states, its bad,Simple Fix:
Thank you.