medkit.text.preprocessing.duplicate_finder#

Classes:

DuplicateFinder(output_label[, ...])

Detect duplicated chunks of text across a collection of text documents, relying on the duptextfinder library.

DuplicationAttribute(value[, source_doc_id, ...])

Attribute indicating if some text is a duplicate of some other text in another document
class DuplicateFinder(output_label, segments_to_output='dup', min_duplicate_length=5, fingerprint_type='word', fingerprint_length=2, date_metadata_key=None, case_sensitive=True, allow_multiline=True, orf=1)[source]#

Detect duplicated chunks of text across a collection of text documents, relying on the duptextfinder library.

When a duplicated chunk of text is found, a segment is created on the newest document covering the span that is duplicated. A DuplicationAttribute having “is_duplicate” as label and True as value is attached to the segment. It can later be propagated to the entities created from the duplicate segments.

The attribute also holds the id of the source document from which the text was copied, the spans of the text in the source document, and optionally the date of the source document if provided.

Optionally, segments can also be created for non-duplicate zones to make it easier to process only those parts of the documents. For these segments, the attribute value is False and the source, spans and date fields are None.

NB: better performance may be achieved by installing the ncls python package, which will then be used by duptextfinder library.

Parameters

  • output_label (str) – Label of created segments

  • segments_to_output (Literal['dup', 'nondup', 'both']) – Type of segments to create: only duplicate segments (“dup”), only non-duplicate segments (“nondup”), or both (“both”)

  • min_duplicate_length (int) – Minimum length of duplicated segments, in characters (shorter segments will be discarded)

  • fingerprint_type (Literal['char', 'word']) – Base unit to use for fingerprinting (either “char” or “word”)

  • fingerprint_length (int) – Number of chars or words in each fingerprint. If fingerprint_type is set to “char”, this should be the same value as min_duplicate_length. If fingerprint_type is set to “word”, this should be around the average word size multiplied by min_duplicate_length

  • date_metadata_key (Optional[str]) – Key to use to retrieve the date of each document from their metadata dicts. When provided, this is used to determine which document should be the source of a duplicate (the older) and which document should be the recipient (the newer). If None, the order of the documents in the collection will be used.

  • case_sensitive (bool) – Whether duplication detection should be case-sensitive or not

  • allow_multiline (bool) – Whether detected duplicates can span across multiline lines, or each line should be handled separately

  • orf (int) – Step size when building fingerprints, cf the duptextfinder documentation

Methods:

run(collections)

Find duplicates in each collection of documents
run(collections)[source]#

Find duplicates in each collection of documents

For each duplicate found, a Segment object with a DuplicationAttribute will be created and attached to the document that is the recipient of the duplication (ie not the source document).

class DuplicationAttribute(value, source_doc_id=None, source_spans=None, source_doc_date=None, metadata=None, uid=None)[source]#

Attribute indicating if some text is a duplicate of some other text in another document

Variables

  • uid (str) – Identifier of the attribute

  • label (str) – The attribute label, always set to DuplicationAttribute.LABEL

  • value (Optional[Any]) – True if the segment or entity to which the attribute belongs is a duplicate of the part of another document, False otherwise.

  • source_doc_id (str) – Identifier of the document from which the text was copied

  • source_spans (List[medkit.core.text.span.AnySpan]) – Spans of the duplicated text in the source document

  • source_doc_date (Optional[Any]) – Date of the source document, if known

Attributes:

LABEL

Label used for all TNM attributes
LABEL: ClassVar[str] = 'is_duplicate'#

Label used for all TNM attributes