Tag

• Diffusion 17
• Interpretability 17
• Summary 14
• Hallucination 14
• RL 12
• Contrastive Learning 11
• Analysis 11
• Generation 10
• Contrastive Decoding 10
• Chain-of-Thought 9
• Visual Grounding 9
• Segmentation 8
• Attention 8
• Linear Algebra 7
• Efficiency 6
• In-context Learning 5
• Instruction Tuning 5
• Adapter 5
• Visual Encoder 5
• RLHF 5
• PEFT 5
• Text2Image 4
• Foundation Model 4
• Referring Segmentation 4
• Prefix Tuning 4
• Semantic Segmentation 4
• Training-free 4
• Reasoning Segmentation 3
• Benchmark 3
• Detection 3
• Multimodal 3
• Decoding 3
• SAM 2
• PPO 2
• DPO 2
• Universal Segmentation 2
• Prompt Tuning 2
• LoRA 2
• Bias 2
• Dataset 2
• Visual Perception 2
• ViT 2
• Evaluation 2
• Referring Image Segmentation 2
• HTML5 1
• AR 1
• MLE 1
• VAE 1
• FBM 1
• GAN 1
• EBM 1
• Video 1
• SBM 1
• Fine-tuning 1
• Multi-task Learning 1
• Sioyek 1
• Meta-Prompting 1
• RLAIF 1
• Few-Shot Segmentation 1
• Self-Rewarding 1
• Segmentation Dataset 1
• Adapter Tuning 1
• Joint Training 1
• Localization 1
• RAG 1
• Local Attention 1
• Panoptic Segmentation 1
• Self-supervised Learning 1
• Style Transfer 1
• In-Context Learning 1
• Few-shot Learning 1
• Unified Segmentation 1
• Pruning 1
• Perception 1
• Alignment 1

Diffusion

[23' ICCV] DiT: Scalable Diffusion Models with Transformers  2024.05.06 Generation Diffusion

[22' CVPR] Stable Diffusion: High-Resolution Image Synthesis with Latent Diffusion Models  2024.05.05 Generation Diffusion

[21' ICLR] Score-Based Generative Modeling through Stochastic Differential Equations  2024.04.13 Generation Diffusion SBM

[22' NIPS] Imagen: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding  2024.04.06 Generation Diffusion Text2Image

[22'] DALL-E 2: Hierarchical Text-Conditional Image Generation with CLIP Latents  2024.04.05 Generation Diffusion Text2Image

[22' ICLR-WS] Video Diffusion Models  2024.04.03 Generation Diffusion Text2Image Video

[22' CVPR] RePaint: Inpainting using Denoising Diffusion Probabilistic Models  2024.04.03 Generation Diffusion

[22' PMLR] GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models  2024.04.02 Generation Diffusion Text2Image

[22' JMLR] CDM: Cascaded Diffusion Models for High Fidelity Image Generation  2024.04.01 Generation Diffusion

[22' PAML] SR3: Image Super-Resolution via Iterative Refinement  2024.04.01 Generation Diffusion

[21' NIPS-WS] CFG: Classifier-free Diffusion Guidance  2024.03.30 Generation Diffusion

[22' NIPS] CG: Diffusion Models Beat GANs on Image Synthesis  2024.03.29 Generation Diffusion

[21' PMLR] IDDPM: Improved Denoising Diffusion Probabilistic Models  2024.03.28 Generation Diffusion

[21' ICLR] DDIM: Denoising Diffusion Implicit Models  2024.03.26 Generation Diffusion

[21' ICLR] DDPM: Denoising Diffusion Probabilistic Models  2024.03.19 Generation Diffusion

[CS236n] 9. Score Based Models (2)  2024.03.19 CS236n Diffusion Generation

[CS236n] 8. Score Based Models (1)  2024.03.18 CS236n Diffusion Generation

맨 위로 이동 ↑

Interpretability

[24' EMNLP] MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model  2024.10.14 Multimodal Interpretability

[24'] Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations  2024.10.11 Multimodal Interpretability

[24'] Towards Interpreting Visual Information Processing in Vision-Language Models  2024.10.11 Multimodal Interpretability

[24'] Intriguing Properties of Large Language and Vision Models  2024.10.11 Multimodal Interpretability

[24' ACL] Spectral Filters, Dark Signals, and Attention Sinks  2024.09.04 Language Attention Interpretability

[Mechanistic Interpretability] Token-wise Attribution Analysis (History of Kobayashi)  2024.08.24 Language Interpretability Summary

[24' ACL Findings] Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers  2024.08.23 Multimodal Interpretability

[24' ICLR-WS] A Concept-Based Explainability Framework for Large Multimodal Models  2024.08.19 Multimodal Interpretability

[24'] HIO: Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization  2024.08.19 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[24'] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding  2024.08.17 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[24' ICLR] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models  2024.08.14 Language Contrastive Decoding Interpretability

[23' EMNLP] Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning  2024.08.13 Language Interpretability

[24'] Not All Layers of LLMs Are Necessary During Inference  2024.08.13 Language Decoding Interpretability

[24' CVPR] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation  2024.07.30 Multimodal Hallucination Interpretability

[24' ICML-WS] Transformers need glasses! Information over-squashing in language tasks  2024.07.26 Language Hallucination Interpretability

[24'] The Remarkable Robustness of LLMs: Stages of Inference?  2024.07.26 Language Interpretability

[24' ICLR] Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models  2024.07.26 Language Hallucination Interpretability

맨 위로 이동 ↑

Summary

[24'] Hallucination of Multimodal Large Language Models: A Survey  2024.08.24 Multimodal Hallucination Summary

[Mechanistic Interpretability] Token-wise Attribution Analysis (History of Kobayashi)  2024.08.24 Language Interpretability Summary

[Summary] Multimodal Contrastive Decoding Variants (2)  2024.08.19 Multimodal Contrastive Decoding Hallucination Summary

[24'] HIO: Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization  2024.08.19 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[Summary] Multimodal Contrastive Decoding Variants (1)  2024.08.17 Multimodal Contrastive Decoding Hallucination Summary

[24'] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding  2024.08.17 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[Summary] Recent Contrastive Decoding Variants (2)  2024.08.16 Language Contrastive Decoding Summary

[Summary] Recent Contrastive Decoding Variants (1)  2024.08.15 Language Contrastive Decoding Summary

[Summary] List of Pre-trained CLIP Models for Medical Domain  2024.06.27 Medical Contrastive Learning Summary

[Summary] Analysis of Parameter-Efficient Fine-Tuning Techniques for Large Language Models  2024.06.24 Language Analysis PEFT Summary

[Summary] Brief Summary of Parameter-Efficient Fine-Tuning for Language Models  2024.06.23 Language Adapter Tuning LoRA PEFT Prefix Tuning Prompt Tuning Summary

[24' CVPR] Universal Segmentation at Arbitrary Granularity with Language Instruction  2024.06.18 Vision Summary Universal Segmentation

[24'] A Survey on Multimodal Large Language Models  2024.05.14 Multimodal Summary

[Summary] SAM(Segment Anything) Models in Medical Domain  2024.05.04 Medical SAM Segmentation Summary

맨 위로 이동 ↑

Hallucination

[24'] Hallucination of Multimodal Large Language Models: A Survey  2024.08.24 Multimodal Hallucination Summary

[Summary] Multimodal Contrastive Decoding Variants (2)  2024.08.19 Multimodal Contrastive Decoding Hallucination Summary

[24'] HIO: Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization  2024.08.19 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[Summary] Multimodal Contrastive Decoding Variants (1)  2024.08.17 Multimodal Contrastive Decoding Hallucination Summary

[24'] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding  2024.08.17 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[24' CVPR] M3ID: Multi-Modal Hallucination Control by Visual Information Grounding  2024.08.17 Multimodal Contrastive Decoding Hallucination

[24' CVPR] VCD: Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding  2024.08.16 Multimodal Contrastive Decoding Hallucination

[24' ICLR-WS] Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models  2024.08.14 Multimodal Decoding Hallucination

[24'] VDGD: Mitigating LVLM Hallucinations in Cognitive Prompts by Bridging the Visual Perception Gap  2024.08.13 Multimodal Decoding Hallucination

[24' CVPR] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation  2024.07.30 Multimodal Hallucination Interpretability

[24' CVPR] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models  2024.07.30 Multimodal Benchmark Hallucination

[24' ICML-WS] Transformers need glasses! Information over-squashing in language tasks  2024.07.26 Language Hallucination Interpretability

[24' TACL] Lost in the Middle: How Language Models Use Long Contexts  2024.07.26 Language Hallucination

[24' ICLR] Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models  2024.07.26 Language Hallucination Interpretability

맨 위로 이동 ↑

RL

[CS285] 13. Exploration (Part 1)  2024.06.18 CS285 RL

[CS285] 12. Model-Based Policy Learning  2024.05.28 CS285 RL

[CS285] 11. Model-Based Reinforcement Learning  2024.05.27 CS285 RL

[CS285] 10. Optimal Control and Planning  2024.05.04 CS285 RL

[CS285] 9. Advanced Policy Gradients  2024.05.04 CS285 RL

[CS285] 8. Deep RL with Q-Functions  2024.04.24 CS285 RL

[CS285] 7. Value Function Methods  2024.04.23 CS285 RL

[CS285] 6. Actor-Critic Algorithms  2024.04.14 CS285 RL

[CS285] 5. Policy Gradients  2024.04.07 CS285 RL

[CS285] 4. Introduction to Reinforcement Learning  2024.03.28 CS285 RL

[CS285] 2. Supervised Learning of Behaviors  2024.03.21 CS285 RL

[CS285] 1. Introduction  2024.03.15 CS285 RL

맨 위로 이동 ↑

Contrastive Learning

[22' ECCV] MaskCLIP: Extract Free Dense Labels from CLIP  2024.07.10 Vision Contrastive Learning Semantic Segmentation Training-free

[24' ICML] Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation  2024.07.09 Vision Contrastive Learning Semantic Segmentation

[23' CVPR] ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation  2024.07.08 Vision Contrastive Learning Semantic Segmentation

[22' ECCV] BioViL: Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing  2024.06.28 Medical Contrastive Learning

[23'] BiomedCLIP: A Multimodal Biomedical Foundation Model Pretrained from Fifteen Million Scientific Image-text Pairs  2024.06.28 Medical Contrastive Learning

[21'] PubMedCLIP: Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?  2024.06.27 Medical Contrastive Learning Dataset

[22' EMNLP] MedCLIP: Contrastive Learning from Unpaired Medical Images and Text  2024.06.27 Medical Contrastive Learning

[Summary] List of Pre-trained CLIP Models for Medical Domain  2024.06.27 Medical Contrastive Learning Summary

[21' ICCV] GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-Efficient Medical Image Recognition  2024.06.27 Medical Contrastive Learning Local Attention

[22' PMLR] ConVIRT: Contrastive Learning of Medical Visual Representations from Paired Images and Text  2024.06.27 Medical Contrastive Learning

[21' ICML] CLIP: Learning Transferable Visual Models From Natural Language Supervision  2024.05.26 Vision Contrastive Learning

맨 위로 이동 ↑

Analysis

[23' CVPR] Teaching Matters: Investigating the Role of Supervision in Vision Transformers  2024.07.08 Vision Analysis ViT

[22' CVPR] Splicing ViT Features for Semantic Appearance Transfer  2024.07.08 Vision Analysis Self-supervised Learning Style Transfer

[21' ICCV] Understanding Robustness of Transformers for Image Classification  2024.07.08 Vision Analysis ViT

[24' ICML] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models  2024.07.05 Multimodal Analysis Instruction Tuning Visual Encoder

[24'] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs  2024.07.05 Multimodal Analysis Visual Encoder

[24' CVPR] PerceptionGPT: Effectively Fusing Visual Perception into LLM  2024.07.02 Multimodal Analysis Detection Segmentation Visual Perception

[Summary] Analysis of Parameter-Efficient Fine-Tuning Techniques for Large Language Models  2024.06.24 Language Analysis PEFT Summary

[24' ICML] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study  2024.06.11 Language Analysis DPO PPO RLHF

[24' CVPR] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs  2024.06.07 Multimodal Analysis Visual Encoder

[24' CVPR] Honeybee: Locality-enhanced Projector for Multimodal LLM  2024.06.04 Multimodal Adapter Analysis

[24'] MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training  2024.05.29 Multimodal Analysis In-context Learning

맨 위로 이동 ↑

Generation

[CS236n] 10. Evaluating Generative Models  2024.03.20 CS236n Generation

[CS236n] 9. Score Based Models (2)  2024.03.19 CS236n Diffusion Generation

[CS236n] 8. Score Based Models (1)  2024.03.18 CS236n Diffusion Generation

[CS236n] 7. Energy Based Models  2024.03.17 CS236n EBM Generation

[CS236n] 6. Generative Adversarial Networks  2024.03.16 CS236n GAN Generation

[CS236n] 5. Normalizing Flow Models  2024.03.15 CS236n FBM Generation

[CS236n] 4. Latent Variable Models  2024.03.10 CS236n Generation VAE

[CS236n] 3. Maximum Likelihood Learning  2024.03.09 CS236n Generation MLE

[CS236n] 2. Autoregressive Models  2024.03.08 CS236n AR Generation

[CS236n] 1. Introduction to Generative Models  2024.03.08 CS236n Generation

맨 위로 이동 ↑

Contrastive Decoding

[Summary] Multimodal Contrastive Decoding Variants (2)  2024.08.19 Multimodal Contrastive Decoding Hallucination Summary

[24'] HIO: Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization  2024.08.19 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[Summary] Multimodal Contrastive Decoding Variants (1)  2024.08.17 Multimodal Contrastive Decoding Hallucination Summary

[24'] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding  2024.08.17 Multimodal Contrastive Decoding Hallucination Interpretability Summary

[24' CVPR] M3ID: Multi-Modal Hallucination Control by Visual Information Grounding  2024.08.17 Multimodal Contrastive Decoding Hallucination

[24' CVPR] VCD: Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding  2024.08.16 Multimodal Contrastive Decoding Hallucination

[Summary] Recent Contrastive Decoding Variants (2)  2024.08.16 Language Contrastive Decoding Summary

[Summary] Recent Contrastive Decoding Variants (1)  2024.08.15 Language Contrastive Decoding Summary

[24' ICLR] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models  2024.08.14 Language Contrastive Decoding Interpretability

[23' ACL] Contrastive Decoding: Open-ended Text Generation as Optimization  2024.08.14 Language Contrastive Decoding

맨 위로 이동 ↑

Chain-of-Thought

[24' CVPR] Compositional Chain-of-Thought Prompting for Large Multimodal Models  2024.07.03 Multimodal Chain-of-Thought

[24'] F-LMM: Grounding Frozen Large Multimodal Models  2024.07.02 Multimodal Chain-of-Thought Segmentation Visual Grounding

[23' NIPS] KOSMOS-1: Language Is Not All You Need: Aligning Perception with Language Models  2024.07.01 Multimodal Chain-of-Thought Foundation Model In-context Learning

[23'] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic  2024.06.26 Multimodal Chain-of-Thought Detection Visual Grounding

[24'] Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models  2024.06.25 Multimodal Chain-of-Thought

[24' TMLR] Multimodal Chain-of-Thought Reasoning in Language Models  2024.06.22 Multimodal Chain-of-Thought

[23' ICLR] Auto-CoT: Automatic Chain of Thought Prompting in Large Language Models  2024.06.20 Language Chain-of-Thought

[22' NIPS] Zero-shot-CoT: Large Language Models are Zero-Shot Reasoners  2024.06.19 Language Chain-of-Thought

[22' NIPS] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models  2024.06.19 Language Chain-of-Thought

맨 위로 이동 ↑

Visual Grounding

[24' CVPR] AnyRef: Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception  2024.07.03 Multimodal Referring Segmentation Visual Grounding

[23'] LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models  2024.07.02 Multimodal Segmentation Visual Grounding

[24'] F-LMM: Grounding Frozen Large Multimodal Models  2024.07.02 Multimodal Chain-of-Thought Segmentation Visual Grounding

[24' ICLR] KOSMOS-2: Grounding Multimodal Large Language Models to the World  2024.07.01 Multimodal Visual Grounding

[24' CVPR] GROUNDHOG: Grounding Large Language Models to Holistic Segmentation  2024.07.01 Multimodal Panoptic Segmentation Segmentation Visual Grounding

[24'] GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest  2024.07.01 Multimodal Instruction Tuning Visual Grounding

[24' ICLR] Ferret: Refer and Ground Anything Anywhere at Any Granularity  2024.07.01 Multimodal Detection Visual Grounding

[23'] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic  2024.06.26 Multimodal Chain-of-Thought Detection Visual Grounding

[24' CVPR] GLaMM: Pixel Grounding Large Multimodal Model  2024.06.21 Multimodal Segmentation Dataset Visual Grounding

맨 위로 이동 ↑

Segmentation

[24'] M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models  2024.07.03 Medical Multimodal Segmentation

[24' CVPR] PerceptionGPT: Effectively Fusing Visual Perception into LLM  2024.07.02 Multimodal Analysis Detection Segmentation Visual Perception

[23'] LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models  2024.07.02 Multimodal Segmentation Visual Grounding

[24'] F-LMM: Grounding Frozen Large Multimodal Models  2024.07.02 Multimodal Chain-of-Thought Segmentation Visual Grounding

[24' CVPR] GROUNDHOG: Grounding Large Language Models to Holistic Segmentation  2024.07.01 Multimodal Panoptic Segmentation Segmentation Visual Grounding

[24' CVPR] Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation  2024.06.21 Medical Prompt Tuning Segmentation Universal Segmentation

[23'] Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation  2024.05.06 Medical Fine-tuning SAM Segmentation

[Summary] SAM(Segment Anything) Models in Medical Domain  2024.05.04 Medical SAM Segmentation Summary

맨 위로 이동 ↑

Attention

[24'] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models  2024.09.04 Multimodal Attention Efficiency

[24' ECCV] FlexAttention for Efficient High-Resolution Vision-Language Models  2024.09.04 Multimodal Attention Efficiency

[24' ACL] Spectral Filters, Dark Signals, and Attention Sinks  2024.09.04 Language Attention Interpretability

[24' ECCV] FastV: An Image is Worth 1/2 Tokens After Layer 2  2024.09.03 Multimodal Attention Efficiency

[24' ICLR] StreamingLLM: Efficient Streaming Language Models with Attention Sinks  2024.08.29 Language Attention Efficiency

[24' ICLR] PASTA: Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs  2024.08.29 Language Attention

[24' ICML] ACT: Unveiling and Harnessing Hidden Attention Sinks  2024.08.29 Language Attention

[24'] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments  2024.08.27 Multimodal Attention Efficiency

맨 위로 이동 ↑

Linear Algebra

[MIT 18.065] 1.7. Symmetric Positive Definite Matrices  2024.03.21 MIT 18.065 Linear Algebra

[MIT 18.065] 1.6. Eigenvalues and Eigenvectors  2024.03.20 MIT 18.065 Linear Algebra

[MIT 18.065] 1.5. Orthogonal Matrices and Subspaces  2024.03.19 MIT 18.065 Linear Algebra

[MIT 18.065] 1.4. Elimination and A = LU  2024.03.17 MIT 18.065 Linear Algebra

[MIT 18.065] 1.3. The Four Fundamental Subspaces  2024.03.16 MIT 18.065 Linear Algebra

[MIT 18.065] 1.2. Matrix-Matrix Multiplication AB  2024.03.14 MIT 18.065 Linear Algebra

[MIT 18.065] 1.1. Multiplication Ax Using Columns of A  2024.03.14 MIT 18.065 Linear Algebra

맨 위로 이동 ↑

Efficiency

[24'] Quadratic Is Not What You Need For Multimodal Large Language Models  2024.10.11 Multimodal Efficiency Pruning

[24'] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models  2024.09.04 Multimodal Attention Efficiency

[24' ECCV] FlexAttention for Efficient High-Resolution Vision-Language Models  2024.09.04 Multimodal Attention Efficiency

[24' ECCV] FastV: An Image is Worth 1/2 Tokens After Layer 2  2024.09.03 Multimodal Attention Efficiency

[24' ICLR] StreamingLLM: Efficient Streaming Language Models with Attention Sinks  2024.08.29 Language Attention Efficiency

[24'] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments  2024.08.27 Multimodal Attention Efficiency

맨 위로 이동 ↑

In-context Learning

[23' NIPS] KOSMOS-1: Language Is Not All You Need: Aligning Perception with Language Models  2024.07.01 Multimodal Chain-of-Thought Foundation Model In-context Learning

[23' PMLR] Med-Flamingo: a Multimodal Medical Few-shot Learner  2024.06.26 Medical In-context Learning Multimodal

[24' CVPR] LLaFS: When Large Language Models Meet Few-Shot Segmentation  2024.06.11 Multimodal Few-Shot Segmentation In-context Learning

[24'] MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training  2024.05.29 Multimodal Analysis In-context Learning

[22' NIPS] Flamingo: a Visual Language Model for Few-Shot Learning  2024.05.27 Multimodal Foundation Model In-context Learning

맨 위로 이동 ↑

Instruction Tuning

[24' ICML] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models  2024.07.05 Multimodal Analysis Instruction Tuning Visual Encoder

[24' CVPR] Osprey: Pixel Understanding with Visual Instruction Tuning  2024.07.05 Multimodal Instruction Tuning Visual Encoder Visual Perception

[24'] GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest  2024.07.01 Multimodal Instruction Tuning Visual Grounding

[24' CVPR] LLaVA-1.5: Improved Baselines with Visual Instruction Tuning  2024.06.10 Multimodal Adapter Instruction Tuning

[23' NIPS] LLaVA: Visual Instruction Tuning  2024.05.28 Multimodal Instruction Tuning

맨 위로 이동 ↑

Adapter

[24' EMNLP] TroL: Traversal of Layers for Large Language and Vision Models  2024.10.14 Multimodal Adapter

[24' CVPR] VIVL: Towards Better Vision-Inspired Vision-Language Models  2024.06.25 Multimodal Adapter Prefix Tuning

[22' ICLR] Towards a Unified View of Parameter-Efficient Transfer Learning  2024.06.24 Language Adapter PEFT

[24' CVPR] LLaVA-1.5: Improved Baselines with Visual Instruction Tuning  2024.06.10 Multimodal Adapter Instruction Tuning

[24' CVPR] Honeybee: Locality-enhanced Projector for Multimodal LLM  2024.06.04 Multimodal Adapter Analysis

맨 위로 이동 ↑

Visual Encoder

[24' ICML] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models  2024.07.05 Multimodal Analysis Instruction Tuning Visual Encoder

[24' CVPR] Osprey: Pixel Understanding with Visual Instruction Tuning  2024.07.05 Multimodal Instruction Tuning Visual Encoder Visual Perception

[24'] ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models  2024.07.05 Multimodal Visual Encoder

[24'] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs  2024.07.05 Multimodal Analysis Visual Encoder

[24' CVPR] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs  2024.06.07 Multimodal Analysis Visual Encoder

맨 위로 이동 ↑

RLHF

[24' ICML] Self-Rewarding Language Models  2024.06.17 Language RLHF Self-Rewarding

[23'] RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback  2024.06.11 Language RLAIF RLHF

[24' ICML] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study  2024.06.11 Language Analysis DPO PPO RLHF

[23' NIPS] Direct Preference Optimization: Your Language Model is Secretly a Reward Model  2024.06.11 Language DPO RLHF

[19’] RLHF: Fine-Tuning Language Models from Human Preferences  2024.06.10 Language PPO RLHF

맨 위로 이동 ↑

PEFT

[Summary] Analysis of Parameter-Efficient Fine-Tuning Techniques for Large Language Models  2024.06.24 Language Analysis PEFT Summary

[22' ICLR] Towards a Unified View of Parameter-Efficient Transfer Learning  2024.06.24 Language Adapter PEFT

[Summary] Brief Summary of Parameter-Efficient Fine-Tuning for Language Models  2024.06.23 Language Adapter Tuning LoRA PEFT Prefix Tuning Prompt Tuning Summary

[22' ICLR] LoRA: Low-Rank Adaptation of Large Language Models  2024.06.23 Language LoRA PEFT

[24' ICLR] LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention  2024.06.23 Language PEFT Prefix Tuning

맨 위로 이동 ↑

Text2Image

[22' NIPS] Imagen: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding  2024.04.06 Generation Diffusion Text2Image

[22'] DALL-E 2: Hierarchical Text-Conditional Image Generation with CLIP Latents  2024.04.05 Generation Diffusion Text2Image

[22' ICLR-WS] Video Diffusion Models  2024.04.03 Generation Diffusion Text2Image Video

[22' PMLR] GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models  2024.04.02 Generation Diffusion Text2Image

맨 위로 이동 ↑

Foundation Model

[24'] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models  2024.09.10 Multimodal Few-shot Learning Foundation Model

[23' ICML] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models  2024.09.10 Multimodal Foundation Model

[23' NIPS] KOSMOS-1: Language Is Not All You Need: Aligning Perception with Language Models  2024.07.01 Multimodal Chain-of-Thought Foundation Model In-context Learning

[22' NIPS] Flamingo: a Visual Language Model for Few-Shot Learning  2024.05.27 Multimodal Foundation Model In-context Learning

맨 위로 이동 ↑

Referring Segmentation

[24'] LaSagnA: Language-based Segmentation Assistant for Complex Queries  2024.07.03 Multimodal Referring Segmentation

[24' CVPR] AnyRef: Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception  2024.07.03 Multimodal Referring Segmentation Visual Grounding

[24' CVPR] GSVA: Generalized Segmentation via Multimodal Large Language Models  2024.06.21 Multimodal Referring Segmentation

[23' CVPR] GRES: Generalized Referring Expression Segmentation  2024.06.21 Vision Referring Segmentation

맨 위로 이동 ↑

Prefix Tuning

[24' CVPR] VIVL: Towards Better Vision-Inspired Vision-Language Models  2024.06.25 Multimodal Adapter Prefix Tuning

[23'] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model  2024.06.25 Multimodal Joint Training Prefix Tuning

[Summary] Brief Summary of Parameter-Efficient Fine-Tuning for Language Models  2024.06.23 Language Adapter Tuning LoRA PEFT Prefix Tuning Prompt Tuning Summary

[24' ICLR] LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention  2024.06.23 Language PEFT Prefix Tuning

맨 위로 이동 ↑

Semantic Segmentation

[24' ECCV] SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference  2024.10.07 Vision Semantic Segmentation Training-free

[22' ECCV] MaskCLIP: Extract Free Dense Labels from CLIP  2024.07.10 Vision Contrastive Learning Semantic Segmentation Training-free

[24' ICML] Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation  2024.07.09 Vision Contrastive Learning Semantic Segmentation

[23' CVPR] ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation  2024.07.08 Vision Contrastive Learning Semantic Segmentation

맨 위로 이동 ↑

Training-free

[23' EMNLP Findings] Text Augmented Spatial-aware Zero-shot Referring Image Segmentation  2024.10.07 Vision Referring Image Segmentation Training-free

[23' CVPR] ZS-RS: Zero-Shot Referring Image Segmentation With Global-Local Context Features  2024.10.07 Vision Referring Image Segmentation Training-free

[24' ECCV] SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference  2024.10.07 Vision Semantic Segmentation Training-free

[22' ECCV] MaskCLIP: Extract Free Dense Labels from CLIP  2024.07.10 Vision Contrastive Learning Semantic Segmentation Training-free

맨 위로 이동 ↑

Reasoning Segmentation

[24' CVPR] PixelLM: Pixel Reasoning with Large Multimodal Model  2024.06.21 Multimodal Reasoning Segmentation

[23'] LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model  2024.06.02 Multimodal Reasoning Segmentation

[23'] LISA: Reasoning Segmentation via Large Language Model  2024.05.29 Multimodal Reasoning Segmentation

맨 위로 이동 ↑

Benchmark

[24' ECCV] BLINK: Multimodal Large Language Models Can See but Not Perceive  2024.10.13 Multimodal Benchmark Perception

[24' CVPR] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models  2024.07.30 Multimodal Benchmark Hallucination

[24'] MMStar: Are We on the Right Way for Evaluating Large Vision-Language Models?  2024.06.26 Multimodal Benchmark Bias

맨 위로 이동 ↑

Detection

[24' CVPR] PerceptionGPT: Effectively Fusing Visual Perception into LLM  2024.07.02 Multimodal Analysis Detection Segmentation Visual Perception

[24' ICLR] Ferret: Refer and Ground Anything Anywhere at Any Granularity  2024.07.01 Multimodal Detection Visual Grounding

[23'] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic  2024.06.26 Multimodal Chain-of-Thought Detection Visual Grounding

맨 위로 이동 ↑

Multimodal

[24'] M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models  2024.07.03 Medical Multimodal Segmentation

[23' PMLR] Med-Flamingo: a Multimodal Medical Few-shot Learner  2024.06.26 Medical In-context Learning Multimodal

[23' NIPS] LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day  2024.06.26 Medical Dataset Multimodal

맨 위로 이동 ↑

Decoding

[24' ICLR-WS] Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models  2024.08.14 Multimodal Decoding Hallucination

[24'] VDGD: Mitigating LVLM Hallucinations in Cognitive Prompts by Bridging the Visual Perception Gap  2024.08.13 Multimodal Decoding Hallucination

[24'] Not All Layers of LLMs Are Necessary During Inference  2024.08.13 Language Decoding Interpretability

맨 위로 이동 ↑

SAM

[23'] Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation  2024.05.06 Medical Fine-tuning SAM Segmentation

[Summary] SAM(Segment Anything) Models in Medical Domain  2024.05.04 Medical SAM Segmentation Summary

맨 위로 이동 ↑

PPO

[24' ICML] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study  2024.06.11 Language Analysis DPO PPO RLHF

[19’] RLHF: Fine-Tuning Language Models from Human Preferences  2024.06.10 Language PPO RLHF

맨 위로 이동 ↑

DPO

[24' ICML] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study  2024.06.11 Language Analysis DPO PPO RLHF

[23' NIPS] Direct Preference Optimization: Your Language Model is Secretly a Reward Model  2024.06.11 Language DPO RLHF

맨 위로 이동 ↑

Universal Segmentation

[24' CVPR] Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation  2024.06.21 Medical Prompt Tuning Segmentation Universal Segmentation

[24' CVPR] Universal Segmentation at Arbitrary Granularity with Language Instruction  2024.06.18 Vision Summary Universal Segmentation

맨 위로 이동 ↑

Prompt Tuning

[Summary] Brief Summary of Parameter-Efficient Fine-Tuning for Language Models  2024.06.23 Language Adapter Tuning LoRA PEFT Prefix Tuning Prompt Tuning Summary

[24' CVPR] Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation  2024.06.21 Medical Prompt Tuning Segmentation Universal Segmentation

맨 위로 이동 ↑

LoRA

[Summary] Brief Summary of Parameter-Efficient Fine-Tuning for Language Models  2024.06.23 Language Adapter Tuning LoRA PEFT Prefix Tuning Prompt Tuning Summary

[22' ICLR] LoRA: Low-Rank Adaptation of Large Language Models  2024.06.23 Language LoRA PEFT

맨 위로 이동 ↑

Bias

[24'] MMStar: Are We on the Right Way for Evaluating Large Vision-Language Models?  2024.06.26 Multimodal Benchmark Bias

[24' CVPR] REAL: The Neglected Tails in Vision-Language Models  2024.06.25 Multimodal Bias RAG

맨 위로 이동 ↑

Dataset

[21'] PubMedCLIP: Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?  2024.06.27 Medical Contrastive Learning Dataset

[23' NIPS] LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day  2024.06.26 Medical Dataset Multimodal

맨 위로 이동 ↑

Visual Perception

[24' CVPR] Osprey: Pixel Understanding with Visual Instruction Tuning  2024.07.05 Multimodal Instruction Tuning Visual Encoder Visual Perception

[24' CVPR] PerceptionGPT: Effectively Fusing Visual Perception into LLM  2024.07.02 Multimodal Analysis Detection Segmentation Visual Perception

맨 위로 이동 ↑

ViT

[23' CVPR] Teaching Matters: Investigating the Role of Supervision in Vision Transformers  2024.07.08 Vision Analysis ViT

[21' ICCV] Understanding Robustness of Transformers for Image Classification  2024.07.08 Vision Analysis ViT

맨 위로 이동 ↑

Evaluation

[24' ICML] Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations  2024.08.17 Multimodal Evaluation

[24' ICLR] Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning  2024.08.13 Multimodal Evaluation In-Context Learning

맨 위로 이동 ↑

Referring Image Segmentation

[23' EMNLP Findings] Text Augmented Spatial-aware Zero-shot Referring Image Segmentation  2024.10.07 Vision Referring Image Segmentation Training-free

[23' CVPR] ZS-RS: Zero-Shot Referring Image Segmentation With Global-Local Context Features  2024.10.07 Vision Referring Image Segmentation Training-free

맨 위로 이동 ↑

HTML5

[HTML5] HTML 총정리  2024.03.04 Web HTML5

맨 위로 이동 ↑

AR

[CS236n] 2. Autoregressive Models  2024.03.08 CS236n AR Generation

맨 위로 이동 ↑

MLE

[CS236n] 3. Maximum Likelihood Learning  2024.03.09 CS236n Generation MLE

맨 위로 이동 ↑

VAE

[CS236n] 4. Latent Variable Models  2024.03.10 CS236n Generation VAE

맨 위로 이동 ↑

FBM

[CS236n] 5. Normalizing Flow Models  2024.03.15 CS236n FBM Generation

맨 위로 이동 ↑

GAN

[CS236n] 6. Generative Adversarial Networks  2024.03.16 CS236n GAN Generation

맨 위로 이동 ↑

EBM

[CS236n] 7. Energy Based Models  2024.03.17 CS236n EBM Generation

맨 위로 이동 ↑

Video

[22' ICLR-WS] Video Diffusion Models  2024.04.03 Generation Diffusion Text2Image Video

맨 위로 이동 ↑

SBM

[21' ICLR] Score-Based Generative Modeling through Stochastic Differential Equations  2024.04.13 Generation Diffusion SBM

맨 위로 이동 ↑

Fine-tuning

[23'] Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation  2024.05.06 Medical Fine-tuning SAM Segmentation

맨 위로 이동 ↑

Multi-task Learning

[21' ICML] VL-T5: Unifying Vision-and-Language Tasks via Text Generation  2024.05.15 Multimodal Multi-task Learning

맨 위로 이동 ↑

Sioyek

[Sioyek] Sioyek 사용기  2024.06.02 Application Sioyek

맨 위로 이동 ↑

Meta-Prompting

[24'] Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding  2024.06.03 Language Meta-Prompting

맨 위로 이동 ↑

RLAIF

[23'] RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback  2024.06.11 Language RLAIF RLHF

맨 위로 이동 ↑

Few-Shot Segmentation

[24' CVPR] LLaFS: When Large Language Models Meet Few-Shot Segmentation  2024.06.11 Multimodal Few-Shot Segmentation In-context Learning

맨 위로 이동 ↑

Self-Rewarding

[24' ICML] Self-Rewarding Language Models  2024.06.17 Language RLHF Self-Rewarding

맨 위로 이동 ↑

Segmentation Dataset

[24' CVPR] GLaMM: Pixel Grounding Large Multimodal Model  2024.06.21 Multimodal Segmentation Dataset Visual Grounding

맨 위로 이동 ↑

Adapter Tuning

[Summary] Brief Summary of Parameter-Efficient Fine-Tuning for Language Models  2024.06.23 Language Adapter Tuning LoRA PEFT Prefix Tuning Prompt Tuning Summary

맨 위로 이동 ↑

Joint Training

[23'] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model  2024.06.25 Multimodal Joint Training Prefix Tuning

맨 위로 이동 ↑

Localization

[24' CVPR] PixelLLM: Pixel Aligned Language Models  2024.06.25 Multimodal Localization

맨 위로 이동 ↑

RAG

[24' CVPR] REAL: The Neglected Tails in Vision-Language Models  2024.06.25 Multimodal Bias RAG

맨 위로 이동 ↑

Local Attention

[21' ICCV] GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-Efficient Medical Image Recognition  2024.06.27 Medical Contrastive Learning Local Attention

맨 위로 이동 ↑

Panoptic Segmentation

[24' CVPR] GROUNDHOG: Grounding Large Language Models to Holistic Segmentation  2024.07.01 Multimodal Panoptic Segmentation Segmentation Visual Grounding

맨 위로 이동 ↑

Self-supervised Learning

[22' CVPR] Splicing ViT Features for Semantic Appearance Transfer  2024.07.08 Vision Analysis Self-supervised Learning Style Transfer

맨 위로 이동 ↑

Style Transfer

[22' CVPR] Splicing ViT Features for Semantic Appearance Transfer  2024.07.08 Vision Analysis Self-supervised Learning Style Transfer

맨 위로 이동 ↑

In-Context Learning

[24' ICLR] Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning  2024.08.13 Multimodal Evaluation In-Context Learning

맨 위로 이동 ↑

Few-shot Learning

[24'] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models  2024.09.10 Multimodal Few-shot Learning Foundation Model

맨 위로 이동 ↑

Unified Segmentation

[24'] PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model  2024.10.07 Multimodal Unified Segmentation

맨 위로 이동 ↑

Pruning

[24'] Quadratic Is Not What You Need For Multimodal Large Language Models  2024.10.11 Multimodal Efficiency Pruning

맨 위로 이동 ↑

Perception

[24' ECCV] BLINK: Multimodal Large Language Models Can See but Not Perceive  2024.10.13 Multimodal Benchmark Perception

맨 위로 이동 ↑

Alignment

[24' ACL] Cross-Modal Projection in Multimodal LLMs Doesn’t Really Project Visual Attributes to Textual Space  2024.10.14 Multimodal Alignment

맨 위로 이동 ↑