Notes

Sure, let's add some debugging code to help you identify the problematic token causing the error. Modify your code as follows:

1. **Check Vocabulary Size:** Make sure your vocabulary size includes all unique tokens from both questions and answers.

```python
# Build vocabulary
def build_vocabulary(data):
vocab = set()
for sentence in data:
tokens = nltk.word_tokenize(sentence)
vocab.update(tokens)
vocab = list(vocab)
vocab.insert(0, START_TOKEN)
vocab.append(PADDING_TOKEN)
vocab.append(END_TOKEN)
return vocab

# Combine vocabularies of questions and answers
combined_vocabulary = build_vocabulary(questions + answers)

# Update vocabulary size
vocab_size = len(combined_vocabulary)

# Update vocabulary dictionaries
vocabulary_to_index = {word: idx for idx, word in enumerate(combined_vocabulary)}
index_to_vocabulary = {idx: word for idx, word in enumerate(combined_vocabulary)}
```

2. **Update Sentences to Indices Function:** Modify the `sentences_to_indices` function to use the combined vocabulary and vocabulary size.

```python
def sentences_to_indices(sentences, vocabulary, max_length):
return [[vocabulary.get(token, vocabulary[PADDING_TOKEN]) for token in sentence.split()][:max_length] for sentence in sentences]

questions_indices = sentences_to_indices(questions, vocabulary_to_index, max_sequence_length)
answers_indices = sentences_to_indices(answers, vocabulary_to_index, max_sequence_length)
```

3. **Check Data:** Print out the sentences that correspond to the problematic indices to identify the issue.

```python
# Find the problematic indices causing the error
problematic_indices = [103, 926] # Add any other problematic indices here

# Print corresponding sentences for debugging
for idx in problematic_indices:
print(f"Problematic index: {idx}")
print(f"Question: {questions[idx]}")
print(f"Answer: {answers[idx]}n")
```

This debugging code will help you identify which tokens are causing the error and allow you to see the corresponding sentences. This way, you can track down the problematic tokens and update your vocabulary accordingly.