Notes

Sure, if you have your dataset in a DataFrame, you can adapt the code accordingly. Here's how you can modify the code to work with a DataFrame:

```python
import torch
from torch.utils.data import DataLoader, Dataset
import pandas as pd

# Start, padding, and end tokens
START_TOKEN = '[START]'
PADDING_TOKEN = '[PAD]'
END_TOKEN = '[END]'

# Load your dataset into a DataFrame
# Replace 'data.csv' with the actual path to your dataset file
data = pd.read_csv('data.csv')

questions = data['questions'].tolist()
answers = data['answers'].tolist()

# Build vocabulary
def build_vocabulary(data):
vocab = set()
for sentence in data:
vocab.update(sentence.split()) # Split sentences into words and add to vocabulary
vocab = list(vocab)
vocab.insert(0, START_TOKEN)
vocab.append(PADDING_TOKEN)
vocab.append(END_TOKEN)
return vocab

question_vocabulary = build_vocabulary(questions)
answer_vocabulary = build_vocabulary(answers)

question_to_index = {word: idx for idx, word in enumerate(question_vocabulary)}
index_to_question = {idx: word for idx, word in enumerate(question_vocabulary)}

answer_to_index = {word: idx for idx, word in enumerate(answer_vocabulary)}
index_to_answer = {idx: word for idx, word in enumerate(answer_vocabulary)}

# Convert sentences to indices
def sentences_to_indices(sentences, vocabulary):
return [[vocabulary[token] for token in sentence.split()] for sentence in sentences]

questions_indices = sentences_to_indices(questions, question_to_index)
answers_indices = sentences_to_indices(answers, answer_to_index)

# Define your Dataset class to handle question-answer pairs
class QADataset(Dataset):
def __init__(self, questions_indices, answers_indices):
self.questions_indices = questions_indices
self.answers_indices = answers_indices

def __len__(self):
return len(self.questions_indices)

def __getitem__(self, idx):
return {
'input_question': self.questions_indices[idx],
'target_answer': self.answers_indices[idx]
}

# Create your Dataset and DataLoader
train_dataset = QADataset(questions_indices, answers_indices)
batch_size = 32 # Set your desired batch size
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

# Define your Transformer model
class Transformer(...):
# Your model definition

# Training loop, validation loop, inference will follow as before
```

Replace `'data.csv'` with the actual path to your DataFrame file. The code provided here outlines how to preprocess your question and answer pairs stored in a DataFrame, build a vocabulary, and convert sentences into indices for feeding into your Transformer model. The rest of the code, such as the training loop, validation loop, and inference, should remain mostly the same as before, with adjustments to work with question-answer pairs stored in a DataFrame.