ML
11.3k articles
Arxiv
1h
151
Image Credit: Arxiv
SIMCOPILOT: Evaluating Large Language Models for Copilot-Style Code Generation
- SIMCOPILOT is a benchmark introduced to evaluate large language models (LLMs) in assisting with coding tasks.
- The benchmark focuses on completion and infill tasks for Java and Python codebases of varying sizes and complexities.
- The evaluation environment of SIMCOPILOT addresses factors such as task-specific performance, contextual understanding, and variable scope sensitivity often overlooked by existing benchmarks.
- Evaluations across different domains reveal insights into LLM strengths and challenges in maintaining logical consistency within complex code structures, indicating a shift towards more intelligent software development partners.
Read Full Article
9 Likes
Arxiv
1h
50
Image Credit: Arxiv
Temporal Restoration and Spatial Rewiring for Source-Free Multivariate Time Series Domain Adaptation
- Source-Free Domain Adaptation (SFDA) aims to adapt pre-trained models without accessing source data to preserve data privacy.
- Existing SFDA methods struggle with multivariate time series (MTS) due to neglecting spatial correlations inherent in MTS data.
- Temporal Restoration and Spatial Rewiring (TERSE) is proposed as a concise SFDA method tailored for MTS data, considering spatial correlations.
- TERSE consists of a spatial-temporal feature encoder, temporal restoration, and spatial rewiring tasks to transfer dependencies across domains effectively.
Read Full Article
3 Likes
Arxiv
1h
97
Image Credit: Arxiv
Fairness in Federated Learning: Fairness for Whom?
- Fairness in federated learning is a rapidly growing research area focusing on formal definitions and algorithmic interventions but often overlooks sociotechnical contexts.
- Existing approaches in fairness for federated learning optimize system level metrics and ignore harms throughout the FL lifecycle and their impact on diverse stakeholders.
- Critical analysis of the literature exposes recurring issues like framing fairness only through server-client architecture, simulation-context mismatches, and lack of multi-stakeholder alignment.
- A harm-centered framework is proposed to connect fairness definitions to risks and stakeholder vulnerabilities, advocating for more thorough and accountable fairness research in federated learning.
Read Full Article
5 Likes
Arxiv
1h
88
Image Credit: Arxiv
CellCLAT: Preserving Topology and Trimming Redundancy in Self-Supervised Cellular Contrastive Learning
- Self-supervised topological deep learning (TDL) is an emerging field with potential for modeling higher-order interactions in cellular complexes to derive representations of unlabeled graphs.
- Cellular complexes have more expressive power compared to simplicial complexes, but self-supervised learning in this domain faces challenges like extrinsic structural constraints and intrinsic semantic redundancy.
- CellCLAT (Cellular Complex Contrastive Learning with Adaptive Trimming) is introduced to address these challenges by preserving cellular topology and reducing informational redundancy through parameter perturbation-based augmentation and cellular trimming scheduler.
- CellCLAT shows significant improvements over existing self-supervised graph learning methods according to theoretical justifications and empirical validations.
Read Full Article
5 Likes
Arxiv
1h
9
Image Credit: Arxiv
Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning
- Model quantization reduces the bit-width of weights and activations, improving memory efficiency and inference speed in diffusion models.
- Challenges in achieving 4-bit quantization have been identified, including handling asymmetric activation distributions, temporal complexity in the denoising process during fine-tuning, and misalignment between fine-tuning loss and quantization error.
- To overcome these challenges, a mixup-sign floating-point quantization (MSFP) framework is proposed, introducing unsigned FP quantization, timestep-aware LoRA (TALoRA), and denoising-factor loss alignment (DFA) for precise and stable fine-tuning.
- Through extensive experiments, superior performance in 4-bit FP quantization for diffusion models has been achieved, surpassing existing post-training quantization fine-tuning methods in 4-bit integer quantization.
Read Full Article
Like
Arxiv
1h
312
Image Credit: Arxiv
Relevance-driven Input Dropout: an Explanation-guided Regularization Technique
- Overfitting is a common issue in Machine Learning models, even among state-of-the-art models, leading to reduced generalization and a significant performance gap between training and testing sets.
- To address overfitting, various techniques like dropout, data augmentation, and weight decay are used for regularization.
- A new data augmentation method called Relevance-driven Input Dropout (RelDrop) is proposed, which selectively occludes the most relevant regions of the input to improve model generalization through informed regularization.
- Experiments on benchmark datasets show that RelDrop enhances robustness towards occlusion, encourages models to use more features in prediction, and improves generalization performance during inference.
Read Full Article
18 Likes
Arxiv
1h
53
Image Credit: Arxiv
SOSBENCH: Benchmarking Safety Alignment on Scientific Knowledge
- A new benchmark called SOSBench has been introduced to assess the safety alignment of large language models (LLMs) in handling high-risk scientific domains.
- SOSBench consists of 3,000 prompts derived from real-world regulations and laws in six hazardous scientific fields: chemistry, biology, medicine, pharmacology, physics, and psychology.
- Evaluation of advanced models using SOSBench revealed alarming rates of harmful responses, indicating deficiencies in safety alignment and raising concerns about the responsible deployment of powerful LLMs.
Read Full Article
3 Likes
Arxiv
1h
170
Image Credit: Arxiv
Learning Where to Learn: Training Distribution Selection for Provable OOD Performance
- Out-of-distribution (OOD) generalization is a challenge in machine learning, where models trained on one data distribution often perform poorly on shifted domains.
- A study focuses on designing training data distributions to enhance average-case OOD performance.
- The research introduces algorithmic strategies to minimize OOD error, such as bilevel optimization and theoretical upper bound minimization.
- Experimental evaluation shows significant improvements in OOD accuracy compared to standard empirical risk minimization, emphasizing the importance of distribution-aware training for robust OOD generalization.
Read Full Article
10 Likes
Arxiv
1h
63
Image Credit: Arxiv
Apprenticeship learning with prior beliefs using inverse optimization
- The relationship between inverse reinforcement learning (IRL) and inverse optimization (IO) for Markov decision processes (MDPs) is explored in this work.
- The study incorporates prior beliefs on the cost function's structure into IRL and apprenticeship learning (AL) problems.
- The convex-analytic view of the AL formalism is identified as a relaxation of the framework, with AL being a special case when the regularization term is absent.
- The AL problem in the suboptimal expert setting is formulated as a regularized min-max problem, utilizing stochastic mirror descent (SMD) to solve it and establish convergence bounds.
Read Full Article
3 Likes
Arxiv
1h
309
Image Credit: Arxiv
Efficient Diffusion Models for Symmetric Manifolds
- Introduction of a framework for efficient diffusion models for symmetric-space Riemannian manifolds.
- Proposal of a new diffusion model for symmetric manifolds with spatially-varying covariance to bypass heat kernel computations.
- Training algorithm minimizes an efficient objective derived via Ito's Lemma, leading to reduced computational complexity.
- Empirical results show improved training speed and sample quality on synthetic datasets on various symmetric manifolds.
Read Full Article
18 Likes
Arxiv
1h
305
Image Credit: Arxiv
PrivATE: Differentially Private Confidence Intervals for Average Treatment Effects
- PrivATE is a novel machine learning framework for computing confidence intervals (CIs) for the average treatment effect (ATE) under differential privacy.
- The framework focuses on deriving valid privacy-preserving CIs for the ATE from observational data in sensitive settings such as medicine.
- PrivATE consists of three steps: estimating a differentially private ATE, estimating the differentially private variance, and constructing CIs while considering uncertainty from both estimation and privatization steps.
- This framework is model agnostic, doubly robust, and ensures valid CIs, demonstrated through synthetic and real-world medical datasets, marking a significant advancement in valid CIs for ATE under differential privacy.
Read Full Article
18 Likes
Arxiv
1h
113
Image Credit: Arxiv
PreGenie: An Agentic Framework for High-quality Visual Presentation Generation
- Visual presentations are essential for effective communication, but automated generation using deep learning has faced challenges like disorganized layouts and inaccurate text summarization.
- To overcome these limitations, a new agentic and modular framework called PreGenie has been introduced.
- PreGenie leverages multimodal large language models (MLLMs) to create high-quality visual presentations in two stages: Analysis and Initial Generation, and Review and Re-generation.
- Experiments show that PreGenie excels in aesthetics, content consistency, and alignment with human design preferences compared to existing models.
Read Full Article
6 Likes
Arxiv
1h
309
Image Credit: Arxiv
What happens when generative AI models train recursively on each others' generated outputs?
- The internet contains AI-generated content and serves as a training data source for generative AI models.
- Future generative AI models might be trained on outputs generated by other models.
- Studying the effects of models training on each other's outputs is crucial due to society's increasing reliance on genAI tools.
- Data-mediated interactions between models can introduce novel concepts to improve performance but may also lead to performance homogenization.
Read Full Article
18 Likes
Arxiv
1h
220
Image Credit: Arxiv
Incentivizing Permissionless Distributed Learning of LLMs
- An incentive system called Gauntlet has been developed for distributed deep learning of foundational models where peers are rewarded for contributions.
- Gauntlet has been deployed on the bittensor blockchain and used to train a 1.2B LLM with completely permissionless contributions of pseudo-gradients.
- The system can be applied to any synchronous distributed training scheme that relies on aggregating updates or pseudo-gradients.
- The project involves a mechanism for filtering peer uptime and reliability, an OpenSkill rating system, and a unique computation mechanism to ensure peer contributions are distinct.
Read Full Article
13 Likes
Arxiv
1h
72
Image Credit: Arxiv
Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling
- LrcSSM is a nonlinear recurrent model designed for efficient sequence modeling, capable of processing long sequences quickly.
- The model achieves parallel processing of full sequences with a single prefix-scan, leading to optimal time and memory complexity.
- LrcSSM provides a formal gradient-stability guarantee that sets it apart from other systems like Liquid-S4 and Mamba, making it reliable for training.
- In comparison to quadratic-attention Transformers, LrcSSM shows superior performance on long-range forecasting tasks, outperforming models like LRU, S5, and Mamba.
Read Full Article
4 Likes
For uninterrupted reading, download the app