Stuyvesant Machine Learning Club announces an upcoming talk by a Google researcher. The details of the talk are below:
Date and time: Thursday, November 19, at 5pm (via Google Meet)
Title: "Big Bird: Transformers for Longer Sequences"
Presenter: Dr. Guru Guruganesh, a Machine Learning Research Scientist at Google.
To sign up: Fill out the form at https://forms.gle/zojJ3ofiLCthnmma7
Talk description:
Google's BERT model has been highly successful in Natural Language Processing tasks such as language translation, question answering, and text summarization. However, the cost of BERT's attention mechanism grows quadratically on the length of the input sequence, so it doesn't scale up beyond several hundred input tokens. In this talk, Dr. Guruganesh will present Big Bird, a new state-of-the art language model that uses sparse attention mechanism to reduce BERT's quadratic dependency on input length to linear. Thanks to sparse attention, Big Bird can handle sequences of length up to 8x of what was previously possible with BERT. Thanks to the ability to handle longer inputs, Big Bird drastically improves performance on various NLP tasks such as question answering and summarization. It also enables novel applications to genomics data, such as promoter region prediction and chromatin-profile prediction on DNA sequences.
