Field Log · English Edition

Training and Evaluating Multilingual and Korean Embeddings (8/14)

Korean and English query-document pairs aligned in a shared vector space and evaluated by retrieval direction

Key conclusions

Multilingual capability means handling several languages individually; cross-lingual capability means matching a query in one language directly to a document in another. Do not collapse the two into one average score.

Korean retrieval has distinct hard cases: particles and endings, spacing variation, Korean-English code mixing, product names and abbreviations, and numbers with units. A translated benchmark is not enough.

Parallel text aligns language spaces, but it does not teach every form of retrieval relevance. Mix monolingual retrieval, bitext, and cross-lingual question-answer data.

Sampling in proportion to raw data volume lets high-resource languages such as English dominate training.

Choose a model with a quality–latency–tokenizer-cost matrix that separates ko→ko, ko→en, en→ko, and mixed-language queries.

This article extends the domain-adaptation methods from the previous part (Korean) to Korean and multilingual settings.

1. Four Retrieval Directions

Query	Document	Example	Evaluation slice
Korean	Korean	An internal Korean FAQ	ko→ko
Korean	English	Search English API docs with a Korean question	ko→en
English	Korean	An overseas team searches Korean policy documents	en→ko
Mixed	Mixed	`kubectl 롤백 방법` (“how to roll back kubectl”)	code-mixed

A cross-lingual model can perform well on ko→en while remaining weak on domain jargon in ko→ko. Report a matrix by direction.

2. Signals for Learning a Shared Space

Parallel Sentences

"환불은 14일 이내 가능합니다."
↔ "Refunds are available within 14 days."

This signal aligns meaning across languages, but its primary objective is sentence-level translation similarity.

Cross-lingual Query–Passage Pairs

query_ko: "인증서 갱신 주기는?"
           ("How often must certificates be renewed?")
passage_en: "Certificates must be rotated every 90 days."

This directly teaches the asymmetry found in retrieval.

Multilingual NLI and STS

Natural-language inference and semantic-textual-similarity data teach entailment, contradiction, and semantic similarity. Those targets are related to—but not identical to—answer-bearing relevance.

Comparable Documents

Wikipedia interlanguage links and localized manuals for the same product may share a topic without being exact sentence translations.

Do not treat every signal as the same kind of positive. Give each one a task tag and an appropriate weight.

3. Language Sampling

If language (l) has (n_l) examples:

p(l) ∝ n_l^α

α=1: preserve the raw distribution; high-resource languages dominate
α=0: sample languages uniformly; small noisy datasets may be oversampled
0<α<1: trade off the two extremes

Account for quality and task diversity per language as well. Row count alone should not determine the sampling policy.

4. Batch Composition

Monolingual Batches

These expose fine-grained hard negatives within the same language.

Cross-lingual Batches

When translations or equivalent meanings appear in different languages, these batches align the shared space. An unlabeled translation can become a false negative.

Mixed-language Batches

These resemble the production corpus distribution, but language-identity shortcuts and false negatives become harder to control.

A useful ablation:

A language-homogeneous batches
B 50% homogeneous + 50% cross-lingual
C fully mixed + duplicate/translation masking

5. Diagnosing Korean Tokenization

If a tokenizer fragments Korean too aggressively, the effective context becomes shorter under the same token limit.

fertility(text) = number_of_tokens / number_of_whitespace_words

Measure at least these slices:

ordinary prose
sentences with long particle or ending sequences
spacing errors
English product names followed by Korean particles
URLs, paths, and code
dates, versions, and units
Hanja and abbreviations

tokenizer_report:
  korean_general_p50: 1.8
  korean_technical_p50: 2.6
  english_technical_p50: 1.3
  truncation_rate_ko: 0.14
  truncation_rate_en: 0.05

The same 512 tokens do not represent the same amount of information in every language.

6. Korean Minimal Pairs

Construct pairs with high lexical overlap but different meanings.

"관리자만 삭제할 수 있다."  Only administrators may delete it.
"관리자는 삭제할 수 없다."  Administrators may not delete it.

"2026년 3월 이전 버전"       A version before March 2026
"2026년 3월 이후 버전"       A version after March 2026

"10MB 이하"                  10 MB or less
"10MB 이상"                  10 MB or more

"승인 없이 배포"             Deploy without approval
"승인 후 배포"               Deploy after approval

Each pair is close under topical similarity, yet the second item can be a strong negative for policy retrieval. Match the evaluation label to the task.

7. Spacing and Orthographic Variation

머신 러닝 / 머신러닝                    machine learning
로그 인 / 로그인                        login
Kubernetes / 쿠버네티스 / k8s
v3.2 / 3.2 버전 / version 3.2
GPU 24 GB / 24GB GPU

Use these as positive robustness cases, but do not mark every superficially different entity spelling as equivalent.

8. The Translation-Benchmark Trap

If an English query set is machine-translated into Korean:

the translated sentences are often more formal and regular than real user queries;
ambiguity in the English original may disappear or increase;
Korean local entities, culture, and workplace vocabulary are absent;
the source benchmark may already be present in model training data;
one translator’s style may favor a particular model.

Use translated sets to broaden coverage, and maintain a separate native-query set.

9. How to Use MIRACL and MMTEB

MIRACL provides a common retrieval dataset across 18 languages for comparing multilingual retrievers. MMTEB describes more than 500 tasks spanning over 1,000 languages and extends evaluation across multilingual, code, long-document, and instruction-following settings.

Public benchmarks serve a different role from native and in-domain evaluation:

public multilingual benchmark → broad transfer / shortlist
native Korean benchmark       → actual language phenomena
in-domain Korean benchmark    → product decision

Treat the three layers as complements, not substitutes.

10. A Cross-lingual Metric Matrix

               doc ko   doc en   doc mixed
query ko        nDCG     nDCG      nDCG
query en        nDCG     nDCG      nDCG
query mixed     nDCG     nDCG      nDCG

Add diagnostics beyond relevance:

language bias@k: does the model prefer documents in the query language regardless of relevance?
translation consistency: how much do the top-k results overlap for translated versions of a query?
bilingual hubness: do documents in one language appear for an unusually large number of queries?
code-switch robustness
tokenizer latency and tokens per input

11. Translation Consistency Does Not Require Exact Identity

The top-k results for an original query (q_{ko}) and its translation (q_{en}) can differ because each language may have a different relevant document.

Evaluate:

top-k overlap
rank correlation on shared candidates
relevance metric for each direction
evidence-set equivalence

Low document-ID overlap can still be a success when both result sets support the same facts.

12. Multilingual Hard Negatives

Examples by type:

the same entity under a different policy version
nearly identical translations with different numbers
the same product name under different national regulations
Japanese or Chinese documents that share keywords with a Korean query
the same code token explained differently in another language
a translated duplicate that is actually positive

Remove the last type from the negative set. This requires both a cross-lingual teacher and bilingual human auditing.

13. Building a Model Shortlist

Start from model families and verify their operational characteristics.

Family	What to check
multilingual E5	query/passage prefixes, context length
BGE-M3	dense, sparse, and multi-vector retrieval; 8K input
mGTE	multilingual long-context efficiency
jina-embeddings-v3	task LoRA, Matryoshka representation dimensions
Qwen3 Embedding	LLM backbone, instructions, 0.6B/4B/8B sizes
EmbeddingGemma	sub-500M/on-device use, supported languages

This is not a ranking. Compare only models whose licenses, hardware requirements, official prompts, and Korean evaluation results meet your constraints.

14. Constructing a Korean Evaluation Set

korean_eval:
  queries: 2500
  sources:
    native_real: 1200
    native_curated: 700
    translated: 300
    adversarial_minimal_pairs: 300
  directions:
    ko_ko: 0.65
    ko_en: 0.15
    en_ko: 0.10
    mixed: 0.10
  slices:
    - spacing_variation
    - entity_transliteration
    - negation
    - number_version_date
    - honorific_ellipsis
    - code_mixed
    - long_document
    - no_answer

Set the proportions from real traffic rather than copying this example unchanged.

15. Fair Serving Comparisons

If Korean fertility is higher, identical tokens per second can still produce fewer texts per second.

report both:
  tokens/second
  texts/second by language and length
  p95 query latency
  document indexing hours

For API models, include request overhead and rate limits. For local models, include batching behavior and GPU memory.

16. A Release Gate

multilingual_gate:
  ko_ko_ndcg_at_10: ">= baseline + 0.02"
  ko_en_recall_at_20: ">= baseline"
  en_ko_recall_at_20: ">= baseline"
  number_negation_slice: ">= baseline"
  translation_consistency_drop: "<= 0.03"
  korean_p95_ms: "<= 35"
  license: approved

Do not let an average multilingual gain conceal a Korean regression.

Check Your Understanding

What is the difference between multilingual and cross-lingual capability?
Why is parallel-sentence data alone insufficient for learning answer-bearing retrieval?
Why can α=0 be risky even when the goal is language-balanced sampling?
Why should a translated Korean benchmark be kept separate from a native-query set?
When can two cross-lingual top-k result lists both be successful despite containing different document IDs?

The next part (Korean) compares chunk-before-embed, late chunking, contextual document embeddings, situated embeddings, and multi-vector representations.