BERT is an AI language model that Google uses within its algorithm to help provide relevant search results by better understanding the context of the searcher's query.

About the models BERT BERT (bidirectional encoder representations from transformers) employs the transformer’s encoder mechanism to understand the context around each word in a sentence.

ModernBERT, like BERT, is an encoder-only model. Encoder-only models have the characteristic of outputting a 'list of numbers (embedding vector)', which means that they literally encode human ...

This means that BERT is pre-trained on a large corpus of unlabelled text including the entire Wikipedia (that's 2,500 million words!) and Book Corpus (800 million words).