yasbd-lib vs PySBD: two philosophies of sentence boundary detection

Sentence boundary detection sounds boring. Split on . ? !, done, right? Anyone who has tried knows otherwise. Abbreviations, decimals, URLs, nested quotes, ellipsis, legal citations, biomedical jargon—each one turns "split text" into a language-specific puzzle. Two Python libraries tackle this problem with different philosophies. pysbd has been the go-to since 2020 with 22 languages, ported from Ruby's pragmatic segmenter [1]. yasbd-lib is newer, covers 39 languages, and takes a different architectural approach. This is the difference between protecting boundaries and finding them. Architecture: Mutation vs. Pointers To understand how these engines behave on large datasets, we have to look at how they treat input strings. PySBD: The Transformation Pipeline PySBD operates as a multi-stage transformation pipeline [1]. It treats text as a mutable object that must be modified before it can be split [1]. To prevent punctuation within abbreviations, numbers, or URLs from triggering false splits, PySBD applies regular expressions to replace characters with placeholder tokens [1]. flowchart TD A[Raw Input Text] --> B["Replace . with {} / Mask URLs"] B --> C[Run Rule Engine] C --> D[Split on Splitting Marks] D --> E[Reverse Replacement / Restore Text] E --> F[Extract Text Segments] Enter fullscreen mode Exit fullscreen mode The structural consequence: the original string layout is transformed during processing. Because the text is modified mid-flight, calculating exact character offsets (spans) relative to the original uncleaned text requires a post-processing reconstruction step [1]. If you enable text cleaning (clean=True), PySBD raises an error when requesting character spans because it cannot guarantee coordinate matching after modification [1]. yasbd-lib: The Query Planning Approach yasbd-lib treats text as immutable [2]. It does not modify the raw string [2]. Instead, its architecture resembles a database query planner—generating candidate coordinate arrays and using language-specific filters to narrow down boundary slices [2]. flowchart LR A[Raw Input String] --> B[Pass 1: Aggressive Candidate Identification] B --> C[Pass 2: Modular Filter Elimination] C --> D[Project Slices / Return Index Pointers] Enter fullscreen mode Exit fullscreen mode By evolving integer pointers rather than altering text strings, yasbd-lib maintains context of the source layout throughout processing [2]. Token spans are tracked as a first-class structural signal during parsing rather than reconstructed afterward [2]. Deep-Dive Feature Breakdown 1. Spans and Character Offsets Because PySBD does not natively track indices during its transformation phase, calculating character offsets requires a post-processing step that searches the original document to locate each sentence [1]. The PySBD Reconstruction Step PySBD reconstructs spans by scanning the original text for each sentence (source): def sentences_with_char_spans(self, sentences): sent_spans = [] prior_end_char_idx = 0 for sent in sentences: for match in re.finditer('{0}\s*'.format(re.escape(sent)), self.original_text): match_str = match.group() match_start_idx, match_end_idx = match.span() if match_end_idx > prior_end_char_idx: sent_spans.append( TextSpan(match_str, match_start_idx, match_end_idx)) prior_end_char_idx = match_end_idx break return sent_spans Enter fullscreen mode Exit fullscreen mode The Trade-off: This reconstruction performs repeated searches over the original document. In pathological cases—such as documents with many repeated sentences—this can approach quadratic behavior. For typical use cases, the overhead is manageable, but it does add runtime cost on longer texts. The yasbd-lib Approach In yasbd-lib, spans are produced natively during boundary detection [2]. It emits boundary allocations dynamically, avoiding lookback overhead [2]. The library also provides an adapter layer for migrating from PySBD [2]: from yasbd.utils.pysbd_adapter import Segmenter seg = Segmenter(language="ja") res = seg.segment('田中さんは「準備は完了しました」そう言って部屋を出た。U.S.A.の経済政策 is complex.') print(res) # ['田中さんは「準備は完了しました」そう言って部屋を出た。', 'U.S.A.の経済政策 is complex.'] Enter fullscreen mode Exit fullscreen mode 2. Memory and Streaming PySBD processes text as complete string buffers [1]. yasbd-lib provides abstractions for memory-constrained environments through lazy evaluation via ParagraphStream and StreamCleaner [2]: from yasbd.utils.cleaner import StreamCleaner from yasbd import BoundaryDetector cleaner = StreamCleaner("Hello world. This is messy.") detector = BoundaryDetector(lang="en") sentences = list(detector.segment(cleaner)) print(sentences) # ['Hello world.', 'This is messy.'] Enter fullscreen mode Exit fullscreen mode 3. Resource Management Under the hood of the BoundaryDetector pipeline, yasbd-lib manages execution rules using a 5-entry LRU cache (_MAX_CACHED_RULES = 5) [2]. When using automatic language identification (lang="auto"), if confidence drops below the threshold (_MIN_CONFIDENCE = 0.8), the module logs an informational message rather than masking the failure [2]: # From boundary_detector.py if lang == "auto": lang, confidence = classify_language(snippet) if confidence \s]+))?)+\s*|\s*)\/?>", '') Enter fullscreen mode Exit fullscreen mode When processing unfinished HTML attributes, this regex can cause the segmenter to hang indefinitely [11]. A simplified fix was proposed in the same issue, but it remains unreviewed and unmerged. Both issues stem from the same root cause: regex patterns with nested quantifiers in the transformation pipeline [10, 11]. The project has no active maintainer to review or merge fixes [9]. yasbd-lib was built in response to this situation, offering a drop-in adapter for PySBD to fix edge cases without heavy refactoring [2, 9]. Benchmark Comparisons The architectural differences influence accuracy and performance. Feature PySBD yasbd-lib Maintenance Unmaintained (as of 2025) [9] Actively maintained [2] Known Issues Infinite loop on numbered references [10]; catastrophic backtracking in HTML cleaner [11] No known catastrophic backtracking issues Approach Monolithic transformation pipeline [1] Modular immutable pipeline [2] State Handling String mutation with placeholder tokens [1] Pointer-based operations [2] Language Profiles 22 Languages [1] 39 Languages [2] English Golden Score 77 / 92 (83.7%) [2] 91 / 92 (98.9%) [2] Framework Adapters Native API [1] spaCy v3+ integration [2] Benchmark Note: The English Golden Score is measured on the project's expanded golden corpus of 92 evaluation cases [2]. The original PySBD corpus contained 48 cases; the expanded set removes ambiguous examples and adds coverage for abbreviation chains, contiguous terminators, and other edge cases. Full methodology and test cases are available in the benchmarks directory. Edge Case Behavior Consider how both engines handle challenging inputs: Input with numbered references (Issue #79): "..[111 111 111 111 111 111 111 111 111 111]" [10] PySBD: Can enter an infinite loop due to catastrophic backtracking in NUMBERED_REFERENCE_REGEX. This was reported in October 2020 and remains unresolved [10]. yasbd-lib: The two-pass boundary detection approach avoids complex regex substitutions, preventing this class of issue [2]. Input with unfinished HTML (Issue #92): " B{Using legacy spaCy v2?} B -- Yes --> C[Consider PySBD with caution] B -- No --> D{Need active maintenance?} D -- Yes --> E[Use yasbd-lib] D -- No --> F[Use yasbd-lib for accuracy gains] Enter fullscreen mode Exit fullscreen mode Consider PySBD only if: Absolute legacy lock-in: You are maintaining an existing pipeline tied to spaCy v2 or older deployments that cannot be migrated. Be aware that the project is unmaintained and has known unresolved issues, including infinite loops with numbered references and catastrophic backtracking with certain HTML inputs [9, 10, 11]. Use yasbd-lib if: Active maintenance: The project is actively maintained with a clear contribution path [2]. Performance at scale: Benchmark results on the Sherlock Holmes text (594k characters) show yasbd completing in approximately 1.6 seconds compared to 13.3 seconds for PySBD on the same hardware [2]. (These results are from the project's benchmark suite; your mileage may vary based on hardware and Python version.) Character span accuracy: Native span tracking may be preferable for downstream tasks like NER training or RAG indexing [2]. Non-standard inputs: The modular design handles raw markdown, chat logs, and multilingual text [2]. Custom language rules: The declarative profile system simplifies adding new languages or domain-specific abbreviations [2]. Migration path: The included PySBD adapter allows incremental migration without rewriting your entire pipeline [2]. No catastrophic backtracking: The pointer-based architecture avoids the regex issues that plague PySBD's transformation pipeline [2, 10, 11]. References [1] PySBD GitHub Repository [2] yasbd-lib GitHub Repository [3] PySBD Issue #108: Examples of modifying sentence segmentation rules [4] yasbd-lib Issue #20: Help Add More Languages to Yasbd [5] yasbd-lib Issue #198: Core Languages Locked at 39 [6] yasbd-lib Language Template [7] yasbd-lib Benchmarks [8] PySBD Paper: arXiv:2010.09657 [9] PySBD Issue #135: Archive Project [10] PySBD Issue #79: Infinite loop? [11] PySBD Issue #92: Catastrophic backtracking in HTMLTagRule