Literature Review

How to Train LLM? - From Data Parallel To Fully Sharded Data Parallel

How to Inference Big LLM? - Using Accelerate Library

Pyspark - How to preprocess Large Scale Data with Python

Democratizing Large Language Models : From 175B to 7B

Llama3 - Tokenizer

Reflections on Optimizer and LM parameter values

DeepSpeed - Sharding Optimizer, Gradients, Parameters, and Reducing Activations for Efficient Training

KL Divergence

Grouped Query Attention (Llama3)

Python Study & Review

Model Context Protocol (MCP) - provided by Antrophic

Flash attention

Mixed Precision & QLORA & Gradient Checkpointing

NCCL Operation

LLAMA1 Simple Takeways

Docker

(Basic Web Service Structure) & FastAPI

네트워크 장치의 구조

Java (On Going…)

HDFS + YARN + SPARK

SELECTION-INFERENCE: EXPLOITING LARGE LANGUAGE MODELS FOR INTERPRETABLE LOGICAL REASONING

Do Prompt-Based Models Really Understand the Meaning of Their Prompts?

GUESS THE INSTRUCTION! FLIPPED LEARNING MAKES LANGUAGE MODELS STRONGER ZERO-SHOT LEARNERS

Measuring Association Between Labels and Free-Text Rationales

Mutual Information Alleviates Hallucinations in Abstractive SummarizationType of the Paper

Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

Survey of Hallucination in Natural Language Generation

Mitigating Label Biases for In-context Learning

Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts

The False Promise of Imitating Proprietary LLMs

LIMA: Less Is More Alignment

Rethinking the role of the demonstrations

Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations

Explanation-based Finetuning Makes Models More Robust to Spurious Cues

Instruction Mining: High-Quality Instruction Data Selection for Large Language Models

What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning

Self-Alignment with Instruction Backtranslation

ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data

Faithful Low-Resource Data-to-Text Generation through Cycle Training

REPLUG: Retrieval-Augmented Black-Box Language Models

LongLORA: EFFICIENT FINE-TUNING OF LONGCONTEXT LARGE LANGUAGE MODELS

Are Emergent Abilities of Large Language Models a Mirage?

We’re Afraid Language Model Aren’t Modeling Ambiguity

Mistral 7B & Mixtral (Mixtral of Experts)

Can we edit Factual Knowledge by In-Context Learning?

Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning

SILO LANGUAGE MODELS: ISOLATING LEGAL RISK IN A NONPARAMETRIC DATASTORE

LLM Augmented LLMs: Expanding Capabilities through Composition

Rethinking Interpretability in the Era of Large Language Models

Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning (Reasoning을 위한 Prompting)

SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures

Chain-Of-Thought

OVERTHINKING THE TRUTH: UNDERSTANDING HOW LANGUAGE MODELS PROCESS FALSE DEMONSTRATIONS

Beyond Memorization: Violating Privacy Via Inferencing With LLMs

Retrieval-Augmented Data Augmentation For Low-Resource Domain Task

MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning

Preference-free Alignment Learning with Regularized Relevance Reward

Emergent Abilities of Large Language Models

Crafting In-context Examples according to LMs’ Parametric Knowledge

Understanding Emergent Abilities of Language Models from the Loss Perspective

Rethinking Data Selection for Supervised Fine-Tuning

Does Fine Tuning LLMs on News Knowledge Encourage Hallucinations?

Physics of Language Models: Part 2.1 - Grade-School Math and the Hidden Reasoning Process

Reasoning in Flux: Enhancing Large Language Models Reasoning through Uncertainty-aware Adaptive Guidance

SmolLM2: When Smol Goes Big Data-Centric Training of a Small Language Model

Fine-grained Hallucination Detection and Editing for Language Models

Titans: Learning to Memorize at Test Time

LATENT ACTION PRETRAINING FROM VIDEOS

ENCRYPTION-FRIENDLY LLM ARCHITECTURE

LANGBRIDGE: Multilingual Reasoning Without Multilingual Supervision

Glove- Global vectors for word representation

Sequence to Sequence Learning with Neural Networks

Attenton is all you need (Transformer)

ADAM- A METHOD FOR STOCHASTIC OPTIMIZATION (2014 ICLR)`

A Robustly Optimized BERT Pretraining Approach (RoBERTa) & A Lite BERT for Self-supervised Learning of Language Representations (ALBERT)

Improving Language Understanding by Generative Pre-Training (GPT)

Meet Your Favorite Character- Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances (2022 NAACL)

Contrastive Decoding: Open-ended Text Generation as Optimization

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020 NIPS)

Transformer-XL- Attentive Language Models Beyond a Fixed-Length Context (2019 ACL)

XLNet-Generalized Autoregressive Pretraining for Language Understanding (2019 NIPS)

Longformer- The Long-Document Transformer (2020)

Don't Stop Pretraining- Adapt Language Models to Domains and Tasks (2020 ACL)

Bertscore- Evaluating text generation with bert (2020 ICLR)

BART- Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020 ACL)

Improving neural networks by preventing co-adaptation of feature detectors

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (FiD)

Learning to Repair- Repairing model output errors after deployment using a dynamic memory of feedback (2022 NAACL)

FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020 ACL)

Focus Attention- Promoting Faithfulness and Diversity in Summarization (2021 ACL)

Long-Span Summarization via Local Attention and Content Selection (2020 ACL)

Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning (2021 ACL)

MuSeM- Detecting Incongruent News Headlines using Mutual Attentive Semantic Matching (2020 ICMLA)

*****

samples