OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
OptiMer enables flexible continual pre-training by decoupling data mixture ratio selection from training through post-hoc Bayesian optimization of distribution vectors extracted from individual dataset models. (1 upvotes on HuggingFace)
Abstract
OptiMer enables flexible continual pre-training by decoupling data mixture ratio selection from training through post-hoc Bayesian optimization of distribution vectors extracted from individual dataset models.
AI-generated summary
Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per dataset, extract each model's distribution vector, which represents the parameter shift induced by that dataset, and search for optimal composition weights post-hoc via Bayesian optimization. Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15-35 times lower search cost. Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.
View arXiv page View PDF GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2603.28858
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2603.28858 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2603.28858 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2603.28858 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
researchpaperarxiv
Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models - WSJ
<a href="https://news.google.com/rss/articles/CBMiuANBVV95cUxPS1dHbVBVRkY0cWFQLXZyQjJxZlo3cmlOdFdTU0RMTjlsSzI0R214NndpUy1Ka0JrX3RRMUtjQWcya05QY0MxNlNETldUUS02allHeFN4WFRCeUx4OTJxWmVKZ1RNemxLalVTRVd4amtYOEVPdFdCcktWSkhDZnF5SG9DbXpNcTBiRTU5UnBQbV9LekV3UEtqVFB3UGhYS3BGc3RQMDkyY1dfbmJ3QllPa1V5T08wd3RXTDdscHJINUdWV1hHTTBSVG9lbzVLc1Z0VlFHQlpuLWZJZHY4UEFDLU03YzZaZHBlNFc4dWhOTVh3aXJNSGV3UGFjaGFFMkFLcWdwOFg1YUE4eU5aUldycWVpbGQtc3U2MVFyRWUtWTB6N1lxVi1ZUWNhaUk2SURaUHJJUHg5VXFra1hEbTRaY0Nncm9ZaUZkbG0wMVZnUkRTWDZNeEVHRlV6eXhYQ3l1d2luYTZkc1JmRnBocmFHVHMtNkd5YU9jZXdCOE9faGN0dVdIR2tZOXRDSG9VYndjWVZCbzVDUkJCbVJfVFVIV2l0Vl85Zm90enRwbVd1WEtMRzJwdG1mSQ?oc=5" target="_blank">Exclusive | Caltech Researchers Claim Radical Compression of High-Fidelity AI Models</a> <font color="#6f6f6f">WSJ</font>
Jamie Dimon, office-work champion, vows his anti-remote culture ‘would crush you.’ The economy’s top talent begs to differ
JPMorgan’s CEO thinks in-person work makes companies more competitive. Most of the research says another thing about “the entitled non-complier.”
6 critical mistakes that undermine cyber resilience (and how to fix them)
Silos are the enemy of business resilience. As IT leaders, we’ve all felt the pain: the backup administrator, SOC analyst, and endpoint engineer operating in separate worlds—often meeting for the first time in the chaos of a live cyberattack. The result? Delayed responses, missed signals, and greater impact on the business. The N-able 2026 State of the SOC Report leaves no doubt. In just one year, 18% of all security alerts came from network and perimeter exploits—risks many endpoint-only teams never saw coming. Even scarier? 50% of attacks completely bypass endpoint controls. You can’t afford to be siloed. Here’s where most organizations go wrong—and the six crucial steps you need to take to align our teams, tools, and processes for true business resilience. Mistake 1: Unclear roles and r
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Research Papers
Jamie Dimon, office-work champion, vows his anti-remote culture ‘would crush you.’ The economy’s top talent begs to differ
JPMorgan’s CEO thinks in-person work makes companies more competitive. Most of the research says another thing about “the entitled non-complier.”
오라클, 전 세계 최대 3만 명 감원…기업 지원·제품 로드맵 차질 우려
오라클이 대규모 감원에 착수했다. 이번 구조조정은 오라클 역사상 최대 규모의 인력 감축이 될 가능성이 있다. 미국, 인도, 캐나다, 멕시코, 우루과이의 직원들은 현지 시간 기준 31일 오전 6시경 ‘오라클 리더십(Oracle Leadership)’ 명의의 해고 통보 이메일을 받았으며, 인사부나 직속 상사로부터 사전 안내는 없었다. 슬랙, 줌, VPN, 출입 배지 등 기업 시스템 접근 권한도 거의 동시에 차단됐다. 오라클은 영향을 받은 직원들에게 조직 개편과 운영 효율화를 위한 조치라고 설명했다. 컨설팅 기업 그레이하운드 리서치(Greyhound Research)의 수석 애널리스트 산치트 비르 고지아는 오라클 ERP, 오라클 클라우드 인프라(OCI), 넷스위트(NetSuite)를 운영하는 CIO들에게 가장 시급한 우려로 운영 안정성 저하를 지목했다. 그는 “조용히 누적되는 불균형, 지연되는 에스컬레이션 대응, 축소된 백엔드 전문성, 증가하는 팀 간 인수인계, 표준 운영 절차를 벗어난 사고 발생 시 신뢰 저하가 나타날 수 있다”고 분석했다. 이어 “기업 고객은 단순히 코드 개발 속도만을 구매하는 것이 아니라 지원의 연속성, 릴리스 규율, 품질 보증, 통합 안정성, 문제 발생 시 책임성을 함께 고려한다”고 설명했다. 또한 CIO들에게 오라클 계정 팀을 상대로 전담 지원 인력의 지속적인 제공 여부, 최근 60~90일 동안 변경된 제품 조직에 대한 명확성, 향후 두 분기 동안의 릴리스 이행 여부를 확인할 것을 권고했다. 그는 “답변이 모호하다면 그 자체가 하나의 신호”라고 언급했다. 감원 규모와 대상 조직 온라인 직장
UVA researchers use AI to speed up drug development - WVIR
<a href="https://news.google.com/rss/articles/CBMiiwFBVV95cUxOaHF0M0pSdUx0OUp4UHd4a0VnVllZVWtFZ0F6U0I2azlPejJLLTduTmdtZWFCYWhLRWRQSjRXTkxaWlJiV1ozc1JERnFqemtLczJmOEh3d0luZTlNdFNNcDlRdjdobU50RDd0Tk5NRkdqSU5HbGo0RVEzSTdoVThFeWxhRHFzUWpaX3FF0gGfAUFVX3lxTE1jbWpYOWZEWGtJd25vRGg0Nll3VFRzNGdoT01YYmt4YVZ1RHV5dVB3TVN0UVdGVDNHbDFKZnBlODlyQkZSWFFjZ2NDRWVvS05kXzJPOVNpT0xtZ3g5UjM2MF8wWmhPdGkwU1hGYTJzOTlreTJjNzVlaFdHVm9mNUxjOXdQVmR3cVE1ZlhrRmpMbWZpU1FFUEx0UVZXVlBBTQ?oc=5" target="_blank">UVA researchers use AI to speed up drug development</a> <font color="#6f6f6f">WVIR</font>
Illinois Tech computer science researcher honored by IEEE Chicago Section - EurekAlert!
<a href="https://news.google.com/rss/articles/CBMiXEFVX3lxTE13OVpWMEk1Z3hlMkR2bHNBQ2dkazFwb3VqN3hCa29GWGJvSVlPa00zd2xUakRmYXFqQmc5OWU0eGl4a21FMDAwWUN2Q3p0M3FrbXBkNV8zN0cxaG1s?oc=5" target="_blank">Illinois Tech computer science researcher honored by IEEE Chicago Section</a> <font color="#6f6f6f">EurekAlert!</font>
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!