Natural Language Processing

Natural Language processing

AI3202 Introduction to Deep Learning

3 hours One requisite

1 Introduction: / What is NLP? Explain the intersection of linguistics, computer science, and AI. / Key applications of NLP including: / Machine Translation: Examples include Google Translate. / Information Retrieval: Search engines like Google. / Question Answering Systems: Mention IBM Watson's Jeopardy win. / Sentiment Analysis: Understanding opinions in product reviews or social media. / Chatbots and Conversational AI: Applications like ChatGPT. / Summarization: Tools that condense lengthy texts, like academic papers or news articles. / Challenges in NLP: / Syntactic and semantic ambiguities, Non-standard English, idioms, neologisms, entity name complexity, and world knowledge (contextual understanding required for language processing). / List statistical models, algorithms & deep learning models that will be taught in the course. / Ethical and social responsibilities associated with designing and deploying NLP systems / Programming assignment: Readings on ethical AI principles: / Assign academic papers, blog posts, or guidelines related to AI ethics, fairness, and NLP applications. / Tasks include: Write a report of 2 to 3 pages. Summarize the main ethical issues discussed in the paper and explain why these issues are important in real-world NLP applications. Reflect on issues like dataset bias, transparency in NLP models, or inclusivity in multilingual NLP systems. / Submission Deadline: one week

2 Basic Text Processing: / Regular expressions. / Tokenizing (segmenting) words. / Normalizing word formats: case folding (lowercase), stemming, lemmatization. / Segmenting sentences. / Programming assignment: / Demonstrate the usage of the “re” and “NLTK” packages to perform a preprocessing pipeline for a given dataset (e.g. Twitter datasets from NLTK). / Learn to use regular expressions for basic text pattern matching. / Implement word tokenization using libraries / Practice text normalization techniques like case folding, stemming, and lemmatization. / Perform sentence segmentation to break text into sentences

3 Edit distance with application to autocorrect / What is Edit Distance and Minimum Edit Distance? / Dynamic Programming for Minimum Edit Distance / Autocorrect misspelled words model / Identify a misspelled word / Find strings n edit distance away / Filter candidates / Calculate word probabilities / Programming assignment: / implement a simple and effective autocorrect system / Students will be provided with Jupyter Notebook that include guided skeleton code with missing components to perform the following: / Get a word count given a corpus / Get a word probability in the corpus / Manipulate strings using insert, delete, switch, replace operations / Filter strings / Implement Minimum edit distance with dynamic programming to compare strings and to help find the optimal path for the edits. / Submission Deadline: One week

4 Text classification: / Introduction to Text Classification / Supervised Machine Learning for Text Classification / Logistic Regression for Sentiment analysis / Naive Bayes for Sentiment analysis / Programming assignment: / implementing binary Naive Bayes for sentiment analysis on tweets: / Students will be provided with guided skeleton code with missing components / Tasks include calculating prior probabilities, likelihoods, and making predictions using Bayes' theorem. / Submission Deadline: One week

5 N-gram Language Models: / What is a language model and its applications / N-grams and probabilities: probability of a sentence, conditional probabilities, chain rule, Markov Assumption. / Estimating N-gram Probabilities. / Fix missing information: / Handle unseen words with special <UNK> token / Handle missing N-Grams in the corpus with smoothing, Backoff, Interpolation. / Language Model Evaluation with Perplexity metric. / Programming assignment: build an auto-complete system / Students will be provided with guided skeleton code with missing components / Tasks include: / Load and preprocess data / Load and tokenize data. / Split the sentences into train and test sets. / Replace words with a low frequency by an unknown marker <unk>. / Develop N-gram based language models / Compute the count of n-grams from a given data set. / Estimate the conditional probability of a next word with k-smoothing. / Evaluate the N-gram models by computing the perplexity score. / Use the model to suggest an upcoming word given a sentence. / Submission Deadline: one week

6 Vector Space models: / Word Meaning: Synonymy, Antonymy, Similarity, Relatedness, Connotation. / Vector Semantics. / Vectors Representation: / Using co-occurrence matrices to construct term-document matrix and term-term matrix. / Comparisons between vector representations using the cosine similarity. / TF-IDF: Weighing terms in the vector / Introduction to word embeddings / Manipulate word embeddings using arithmetic operations and visualize their relationships in two dimensions using PCA. / Machine translation and document search: / How to learn word vectors linear transformation using gradient decent to translate English word embedding to another language. / Approximate K-nearest neighbors using locality sensitive hashing. / Learn word embeddings with neural network / Basic word representation: one hot vectors and their limitations. / Word embeddings: How to encode semantic meaning of words in n-dimensional coordinates, general properties of word embeddings. / Specific properties of basic word embedding methods: word2vec (uses CBOW & Continuous skip-gram models), GloVe, fastText. / Specific properties of advanced contextual embeddings: BERT, ELMo, GPT-2. / How to create word embeddings: / Explain word embeddings creation process as self-supervised learning that require corpus as the context and an embedding method. / Explain Continuous Bag-of-Words (CBOW) model. / Programming assignment 1: / Explore and use trained word vectors (word2vec): / Students will be provided with guided skeleton code with missing components / Tasks include: / Write a function to predict analogies between words. / Compare word embeddings by using a similarity measure (the cosine similarity). / Test the function on the dataset and check the accuracy of the model / Use PCA to reduce the dimensionality of the word embeddings and plot them in two dimensions. / Submission Deadline: One week for each assignment. / Programming assignment 2: / Implement machine translation using pretrained word embeddings: / Students will be provided with guided skeleton code with missing components / Submission Deadline: One week for each assignment. / Programming assignment 3: / Implement the continuous bag of words (CBOW) model to compute word embeddings. / Students will be provided with guided skeleton code with missing components / Submission Deadline: One week for each assignment

7 Sequence models: / Introduction: Emphasize how previous learning equips students with foundational skills for advanced applications: / Sentiment Analysis: Transition from Naïve Bayes classifiers to deep neural networks for robust sentiment prediction. / Sequence Prediction: Move from n-gram language models to recurrent neural networks (RNNs) for text generation. / Real-World NLP Applications: Apply these advanced techniques to develop practical and robust NLP systems. / (self-study) Provide students with “Introduction to TensorFlow” notebook to know about the TensorFlow framework and learn about some of its basic building blocks. / (self-study) Provide students with a practice programming assignment(ungraded): Sentiment with Deep Neural Networks. / Recurrent neural networks and text generation: / Explain limitations of N-gram modeling and how RNNs outperform N-grams in language generation tasks. / Revision for basic RNN structure / Types of RNN Architectures and their applications in NLP tasks: one to many (caption generating), many to one (sentiment analysis), many to many (machine translation). / Revision for Vanilla RNN: structure, math, cost function for RNNs, explanation of how tf.scan() function abstraction in TensorFlow mimics RNNs and used to implement RNNs to allow for much faster computation. / Gated Recurrent Units: GRU structure, comparison between GRUs and Vanilla RNNs. / Deep and Bi-directional RNNs. / LSTMs and named entity recognition (NER). / Revision for LSTMs architecture, math and their advantages over RNNs. / Applications of LSTMs in NLP tasks: Next-character prediction, Chatbots, Music composition, image captioning, speech recognition. / Introduction to information extraction and named entity recognition and applications of NER systems. / How to train NER system and evaluate it. / Programming assignment 1: Explore RNN and text generation / Students will be provided with guided skeleton code with missing components / Tasks include: / Convert a line of text into a tensor / Create a Tensorflow dataset / Define a GRU model using TensorFlow / Train the model using TensorFlow / Compute the accuracy of the model using the perplexity metric / Generate text using the model / Submission Deadline: One week for each assignment. / Programming assignment 2: Named Entity Recognition / Students will be provided with guided skeleton code with missing components / Tasks include: / Design the architecture of a neural network, train it, and test it. / Process features and represents them / Understand word padding / Implement LSTMs / Test with a given sentence / Submission Deadline: One week for each assignment

8 Attention models: / Neural Machine Translation model with Attention. / Transformer architecture and text summarization. / Transfer learning with state-of-the-art models like T5 and BERT and building question answering system. / Model Alignment, Prompting, and In-Context Learning. / Programming assignment 1: Build neural machine translation model using LSTM networks with attention. / Students will be provided with guided skeleton code with missing components / Submission Deadline: one week for each assignment. / Programming assignment 2: Build a transformer to summarize text. / Students will be provided with guided skeleton code with missing components / Submission Deadline: one week for each assignment. / Programming assignment 3: Implement the "Text to Text Transfer from Transformers” (T5) and fine tune the T5 model for Question answering. / Students will be provided with guided skeleton code with missing components / Submission Deadline: one week for each assignment

9 Sequence Labeling for Parts of Speech: Hidden Markov models, the Viterbi algorithm. / Programming assignment: / Students will be provided with guided skeleton code with missing components. / Tasks include: / Load and preprocess data / Compute the transition matrix in a Hidden Markov Model / Compute the emission matrix in a Hidden Markov Model / Compute the Viterbi algorithm / Compute the accuracy of the model / Submission Deadline: one week

10 Syntactic Parsing. / Programming assignment: / Students will be provided with guided skeleton code with missing components / Submission Deadline: one week

1.1 Mapped to: K1

Demonstrate knowledge of the fundamental principles of natural language processing