medkit.text.segmentation.sentence_tokenizer#

Classes:

SentenceTokenizer([output_label, ...])

Sentence segmentation annotator based on end punctuation rules
class SentenceTokenizer(output_label='SENTENCE', punct_chars=('.', ';', '?', '!'), keep_punct=False, split_on_newlines=True, uid=None)[source]#

Sentence segmentation annotator based on end punctuation rules

Instantiate the sentence tokenizer

Parameters

  • output_label (str, Optional) – The output label of the created annotations.

  • punct_chars (Tuple[str], Optional) – The set of characters corresponding to end punctuations.

  • keep_punct (bool, Optional) – If True, the end punctuations are kept in the detected sentence. If False, the sentence text does not include the end punctuations.

  • split_on_newlines (bool) – Whether to consider that newlines characters are sentence boundaries or not.

  • uid (str, Optional) – Identifier of the tokenizer

Methods:

run(segments)

Return sentences detected in segments.
run(segments)[source]#

Return sentences detected in segments.

Parameters

segments (List[Segment]) – List of segments into which to look for sentences

Return type

List[Segment]

Returns

List[Segments] – Sentences segments found in segments