I decided to implement the dual encoder using Keras and to give further detail about my code here. One thing that motivated me to write this code is that the available implementations are in Tensorflow or Theano and I found that both are hard to understand (not intuitive). So let’s start !
The dual encoder architecture was used in this paper, where the task was defined as follows. Given a history (called context) of conversation between two users, we want to predict the response (called next utterance) by ranking a set of candidate responses. This is a ranking task, which is different from generating a response word by word. In the paper above, researchers presented the Ubuntu Dialogue Corpus, which is a large dataset of two-user conversations issued from IRC (ubuntu channel). Please refer to the paper for further informations.
Dataset description
The Ubuntu Dialogue Corpus is a large dataset of about 1 million of multi-turn dialogues (dialogues with at least 3 turns between two users) containing 7 million of utterances and 100 million words. The corpus can be downloaded from here with different possible preprocessing including lemmatization, tokenization .. etc. The dataset is separated into training, dev and test subsets with 1000000, 195600 and 189200 samples respectively. Each sample is composed of a context, a response (utterance) and a label.
The training set contains contains 50% of positive and 50% of negative samples. Positive ones are the context with the good next utterance. The negatives ones are obtained by selecting randomly a response from the corpus which does not match the good response of the context. Here are some training examples from the dataset extracted from this interesting blog post.
The characters __eot__ and __eou__ denote the end of turn and end of utterance respectively. Each sample of the dev and test datasets contains a context with 10 candidate responses: 1 ground truth response and 9 distractors randomly taken from the corpus as shown in the picture below.
Model architecture and implementation
The model presented by the authors of the paper is based on a dual encoder architecture. The context and the response have variable lengths, we use an encoder to have a fixed size vector that represents the context and the response. The context encoder has as input the words of the context represented with embedding vectors. In the same way, the response encoder receives as input the word embeddings of the response as mentioned below:
Each time a word embedding is fed into the context or the response encoders, they learn a vector representation of the entire text by updating each time their hidden layer. At the end of the encoding process, vectors c and r in the illustration above represent the context and the response respectively as a fixed size vectors. Let’s see the implementation of the first part (data loading and encoders architectures) using Keras.
First we will load the word embeddings from Glove, we used this method to reproduce the state of the art results, but you can use word2vec or your customized embedding vectors. Note that I tried to train word2vec using Gensim on the training set but this does not improve scores.
print("Start building model...")
# first, build index mapping words in the embeddings set
# to their embedding vector
print('Indexing word vectors.')
embeddings_index = {}
f = open(args.embedding_file, 'r')
for line in f:
values = line.split()
word = values[0]
try:
coefs = np.asarray(values[1:], dtype='float32')
except ValueError:
continue
embeddings_index[word] = coefs
f.close()
MAX_SEQUENCE_LENGTH, MAX_NB_WORDS, word_index = pickle.load(open(args.input_dir + 'params.pkl', 'rb'))
print("MAX_SEQUENCE_LENGTH: {}".format(MAX_SEQUENCE_LENGTH))
print("MAX_NB_WORDS: {}".format(MAX_NB_WORDS))
print("Now loading embedding matrix...")
num_words = min(MAX_NB_WORDS, len(word_index)) + 1
embedding_matrix = np.zeros((num_words , args.emb_dim))
for word, i in word_index.items():
if i > = MAX_NB_WORDS:
continue
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
# words not found in embedding index will be all-zeros.
embedding_matrix[i] = embedding_vector
We followed the state of the art implementation and we chose embedding dimension = 300. We used Glove pretrained word vectors to initialize the embedding_matrix. This variable regroups all vocab words embedding vectors into one single variable. Now let’s do the most important part of the code, which is building the model.
print("Now building dual encoder lstm model...")
# define lstm for sentence1
encoder = Sequential()
encoder.add(Embedding(output_dim=args.emb_dim,
input_dim=MAX_NB_WORDS,
input_length=MAX_SEQUENCE_LENGTH,
weights=[embedding_matrix],
mask_zero=True,
trainable=True))
encoder.add(LSTM(units=args.hidden_size))
You can see how much it is easy to implement an encoder using Keras 😉 We define a sequential model and we add a first layer which is Embedding layer that is initialized with the word embedding matrix loaded previously. We set trainable to true which means that the word vectors are fine-tuned during training. At the end we add an LSTM layer which will encode the hole input (context or response) into one vector of size args.hidden_size. Now we have our encoder that we will duplicate to have separate encoder for context and response.
context_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
response_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
context_branch = encoder(context_input)
response_branch = encoder(response_input)
concatenated = merge([context_branch, response_branch], mode='mul')
out = Dense((1), activation = "sigmoid") (concatenated)
dual_encoder = Model([context_input, response_input], out)
dual_encoder.compile(loss='binary_crossentropy',
optimizer=args.optimizer)
Well here we defined simply the context and the response inputs: context_input and response_input. Then, we encode separately the context and the response into context_branch and response_branch. We merge these two vectors using multiplication which will compute a similarity vector between the context and the response. Finally we add a Dense layer of size 1 with a sigmoid activation to transform the vector into a similarity probability. We built the mode in only few lines thanks to Keras. Now we will simply load the data and feed it into our dual encoder model.
print("Now loading UDC data...")
train_c, train_r, train_l = pickle.load(open(args.input_dir + 'train.pkl', 'rb'))
test_c, test_r, test_l = pickle.load(open(args.input_dir + 'test.pkl', 'rb'))
dev_c, dev_r, dev_l = pickle.load(open(args.input_dir + 'dev.pkl', 'rb'))
print('Found %s training samples.' % len(train_c))
print('Found %s dev samples.' % len(dev_c))
print('Found %s test samples.' % len(test_c))
print("Now training the dual_encoder...")
histories = my_callbacks.Histories()
bestAcc = 0.0
patience = 0
print("\tbatch_size={}, nb_epoch={}".format(args.batch_size, args.n_epochs))
for ep in range(1, args.n_epochs):
dual_encoder.fit([train_c, train_r], train_l,
batch_size=args.batch_size, epochs=1, callbacks=[histories],
validation_data=([dev_c, dev_r], dev_l), verbose=1)
curAcc = histories.accs[0]
if curAcc >= bestAcc:
bestAcc = curAcc
patience = 0
else:
patience = patience + 1
y_pred = dual_encoder.predict([test_c, test_r])
print("Perform on test set after Epoch: " + str(ep) + "...!")
recall_k = compute_recall_ks(y_pred[:,0])
#stop the dual_encoder whch patience = 10
if patience > 10:
print("Early stopping at epoch: "+ str(ep))
break
# saving the model
if args.save_dual_encoder:
print("Now saving the dual_encoder... at {}".format(args.dual_encoder_fname))
dual_encoder.save(args.dual_encoder_fname)
I have already preprocessed data and saved it in .pkl format. The train, test and dev files can be downloaded from here. In case you want to do the preprocessing step by yourself, I also shared the preprocessing script on my github repository so you can download raw data from the corpus repository and do whatever you want using the script. Once data loaded on RAM, we will simply feed it into the network at each epoch. We follow the baseline and we use Recall@k as an evaluation metric. I implemented early stopping based on the evaluation metric. If the recall@1 does not increase during 10 successive epochs we stop training and we save the best model.
That’s all you need to implement a dual encoder for the Next Utterance Ranking Task using Keras. I hope that you enjoyed reading this post and it will be useful for you. At the end please feel free to fork, send pull request, comment … etc.
Materials
Basma 