Notes

https://www.kaggle.com/therohk/million-headlines
Semantic Elasticsearch with Sentence Transformers. We will use the power of Elastic and the magic of BERT to index a million articles and perform lexical and semantic search on them.

The purpose is to provide an ease-of-use way of setting up your own Elasticsearch with near state of the art capabilities of contextual embeddings / semantic search using NLP transformers.
The above setup works as follows

Set up an Elasticsearch server with Dockers
Collect the dataset
Use sentence-transformers to index them onto Elastic (takes about 3 hrs on 4 CPU cores)
Look at some comparison examples between lexical and semantic search

The repo introduces the ElasiticTransformers class. Utilities which help create, index and query Elasticsearch indices which include embeddings

Initiate the connection links as well as (optionally) the name of the index to work with
create_index_spec define mapping for the index. Lists of relevant fields can be provided for keyword search or semantic (dense vector) search. It also has parameters for the size of the dense vector as those can vary create_index - uses the spec created earlier to create an index ready for search

write_large_csv - breaks up a large csv file into chunks and iteratively uses a predefined embedding utility to create the embeddings list for each chunk and subsequently feed results to the index
search - allows to select either keyword (‘match’ in Elastic) or semantic (dense in Elastic) search. Notably it requires the same embedding function used in write_large_csv

Automated Threat Report Classification Over Multi-Source Data
This creates the sentence transformer object as well as small helper function which simplifies the embedding call and helps lading data into elastic easier
https://www.kaggle.com/aashita/nyt-comments?select=CommentsFeb2018.csv
https://github.com/md-experiments/elastic_transformers/blob/master/notebooks/Setting_up_ElasticTransformers.ipynb
https://aws.amazon.com/blogs/machine-learning/powering-a-search-engine-with-amazon-sagemaker/
https://github.com/aws-samples/amazon-sagemaker-nlu-search
https://reposhub.com/python/natural-language-processing/md-experiments-elastic_transformers.html
https://xplordat.com/2019/10/28/semantics-at-scale-bert-elasticsearch/
https://github.com/Hironsan/bertsearch
https://repositorio-aberto.up.pt/bitstream/10216/128539/2/412375.pdf
https://eventregistry.org/
https://www.kaggle.com/therohk/examine-the-examiner
https://github.com/Hironsan/bertsearch
https://trec.nist.gov/data/reuters/reuters.html
https://github.com/ChubaOraka/data-science-on-aws-workshop-2020-09-26
https://github.com/marcelpetrick/KubeFlow_BERT_GPU_TensorFlow_Keras_SageMaker_Workshop
https://towardsdatascience.com/speeding-up-bert-search-in-elasticsearch-750f1f34f455
https://medium.com/analytics-vidhya/elasticbert-information-retrieval-using-bert-and-elasticsearch-51fef465b9ae