Notes

Based on the provided output, it appears that the issue is related to tokens that are not present in the vocabulary you've created. Specifically, the tokens "com" and "fault" in the first question and "incrementally" in the second question are not recognized in the vocabulary.

To resolve this issue, you need to make sure that your vocabulary includes all the tokens from your questions and answers. You can do this by carefully constructing the vocabulary to contain all unique words from both questions and answers, as well as the special tokens like `[START]`, `[PAD]`, and `[END]`.

Here are the steps to address this:

1. Update the Vocabulary Building:
- Modify the `build_vocabulary` function to include tokens from both questions and answers.

```python
def build_vocabulary(data):
vocab = set()
for sentence in data:
tokens = nltk.word_tokenize(sentence)
vocab.update(tokens)
vocab = list(vocab)
vocab.insert(0, START_TOKEN)
vocab.append(PADDING_TOKEN)
vocab.append(END_TOKEN)
return vocab

# Combine vocabularies of questions and answers
combined_vocabulary = build_vocabulary(questions + answers)
```

2. Update the Sentences to Indices Function:
- Modify the `sentences_to_indices` function to use the combined vocabulary and vocabulary size.

```python
def sentences_to_indices(sentences, vocabulary, max_length):
return [[vocabulary.get(token, vocabulary[PADDING_TOKEN]) for token in nltk.word_tokenize(sentence)][:max_length] for sentence in sentences]
```

3. Update Vocabulary Dictionaries and Vocabulary Size:
- Update the vocabulary size using the length of the combined vocabulary.
- Update the vocabulary-to-index and index-to-vocabulary dictionaries accordingly.

```python
vocab_size = len(combined_vocabulary)
vocabulary_to_index = {word: idx for idx, word in enumerate(combined_vocabulary)}
index_to_vocabulary = {idx: word for idx, word in enumerate(combined_vocabulary)}
```

By following these steps, you should be able to ensure that all tokens from your questions and answers are included in the vocabulary, which will help you avoid the "token not found in vocabulary" issue during training.