Context Detection

In this tutorial, we will use rule-based operations to attach additional contextual information to entities, such has:

• the section in which the entity is located;

• is the entity negated;

• whether it appears as part of an hypothesis;

• whether it is related to the patient or part of their family’s medical history.

Let’s start by loading a document:

from pathlib import Path
from medkit.core.text import TextDocument

doc = TextDocument.from_file(Path("../data/mtsamplesfr/1.txt"))
print(doc.text)

Section detection

medkit provides a SectionTokenizer operation that takes input segments containing full document texts and splits them into sections, returning a segment for each section.

The section tokenizer is configured with a list of trigger terms signaling the beginning of a section and corresponding section names. medkit provides a default list of sections, but it is missing some sections featured in our document, so we will manually define our own section rules:

from medkit.text.segmentation import SectionTokenizer

# Give a definition of the sections we may encounter
# with the section name and corresponding triggers
sections_definition = {
    "current_drugs": ["MÉDICAMENTS ACTUELS"],
    "clinical_exam": ["EXAMEN DES SYSTÈMES", "EXAMEN PHYSIQUE"],
    "allergies": ["ALLERGIES"],
    "antecedents": ["ANTÉCÉDENTS DE LA MALADIE ACTUELLE", "ANTÉCÉDENTS MÉDICAUX"],
    "family_history": ["ANTÉCÉDENTS FAMILIAUX"],
    "life_style": ["MODE DE VIE"]
}
section_tokenizer = SectionTokenizer(sections_definition, output_label="section")

# The section tokenizer takes a list of segments as input
# and returns a list of segments for each section, with
# a "section" attribute containing the section name
section_segs = section_tokenizer.run([doc.raw_segment])
for section_seg in section_segs:
    section_attr = section_seg.attrs.get(label="section")[0]
    print("section", section_attr.value)
    print(section_seg.text, end="\n\n\n")

Sentence splitting

We have covered sentence splitting previously, and will reuse the same code, with a little addition: we want the section information to be propagated onto the sentences, i.e. we want to be able to tell in which section a sentence belongs.

For this, we will use the attrs_to_copy init parameter. It takes a list of labels that we want to copy from the input segments to the new sentences segments created by the operation. Here, we will use it to copy the “section” attribute of the section segments (which has the section name as value):

from medkit.text.segmentation import SentenceTokenizer

sentence_tokenizer = SentenceTokenizer(
    output_label="sentence",
    keep_punct=True,
    split_on_newlines=True,
    # Copy the "section" attribute
    attrs_to_copy=["section"],
)

# Run the sentence tokenizer on the section segments,
# not on the full text
sentence_segs = sentence_tokenizer.run(section_segs)

for sentence_seg in sentence_segs:
    # Retrieve the copied section attribute
    section_attr = sentence_seg.attrs.get(label="section")[0]
    print("section:", section_attr.value)
    print(sentence_seg.text, end="\n\n")

Family history detection

In this document, we have a section dedicated to family medical history, but this is not always the case. To handle this, medkit provides a FamilyDetector operation based on regular expressions. It is somewhat similar to RegexpMatcher encountered previously, but instead of returning entities, it attaches attributes to the segments it receives, with a boolean value indicating whether it mentions family history.

Like most rule-based operations, FamilyDetector comes with predefined rules that will be used by default if none is provided. For the sake of learning, we will manually create a few rules:

from medkit.text.context import FamilyDetector, FamilyDetectorRule

family_rule_1 = FamilyDetectorRule(
    # Pattern to search inside each input segment.
    # If the pattern is found, the segment will be flagged
    # as being related to family history
    regexp=r"\bfamille\b",
    # Optional exclusions patterns: if found,
    # the segment won't be flagged
    # (Exclusion regexps are also supported for RegexpMatcher)
    exclusion_regexps=[r"\bavec\s+la\s+famille\b"],
    # The regexp will be used with a case-insensitivity flag
    case_sensitive=False,
    # Special chars in the input text will be converted
    # to equivalent ASCII char before runing the regexp on it
    unicode_sensitive=False,
)

family_rule_2 = FamilyDetectorRule(
    regexp=r"\bantecedents\s+familiaux\b",
    case_sensitive=False,
    unicode_sensitive=False,
)

family_detector = FamilyDetector(rules=[family_rule_1, family_rule_2], output_label="family")
# The family detector doesn't return anything but instead adds an attribute to each
# segment with a boolean value indicating if description of family history was detected or not
family_detector.run(sentence_segs)

# Print sentences detected as being related to family history
for sentence_seg in sentence_segs:
    # Retrieve the attribute created by the family detector
    family_attr = sentence_seg.attrs.get(label="family")[0]
    # Only print sentences about family history
    if family_attr.value:
        print(sentence_seg.text)

As with all rule-based operations, FamilyDetector provides the load_rules() and save_rules() methods to facilitate their persistence to a YAML file.

Negation detection

Detecting family history works best at the sentence level. However, for negation and hypothesis, it is better to split sentences into smaller chunks, as the scope of negation and hypothesis can be very limited. For this purpose, medkit provides a SyntagmaTokenizer operation.

from medkit.text.segmentation import SyntagmaTokenizer

# Here we will use the default settings of SyntagmaTokenizer,
# but you can specify your own separator patterns
syntagma_tokenizer = SyntagmaTokenizer(
    output_label="syntagma",
    # We want to keep the section and family history information
    # at the syntagma level
    attrs_to_copy=["section", "family"],
)
# The syntagma tokenizer expects sentence segments as input
syntagma_segs = syntagma_tokenizer.run(sentence_segs)

for syntagma_seg in syntagma_segs:
    print(syntagma_seg.text)

As you can see, a few sentences were split into smaller parts. We can now run a NegationDetector instance on the syntagmata (using the default rules).

from medkit.text.context import NegationDetector, NegationDetectorRule

# NegationDetectorRule objects have the same structure as FamilyDetectorRule
# Here we will use the default rules
negation_detector = NegationDetector(output_label="negation")
negation_detector.run(syntagma_segs)

# Display negated syntagmas
for syntagma_seg in syntagma_segs:
    negation_attr = syntagma_seg.attrs.get(label="negation")[0]
    if negation_attr.value:
        print(syntagma_seg.text)

Hypothesis detection

medkit also provides HypothesisDetector, which is very similar to NegationDetector, except it also uses a list of conjugated verb forms in addition to the list of rules. By default, verbs at conditional and future tenses indicate the presence of an hypothesis. This can be configured alongside the list of verbs.

from medkit.text.context import HypothesisDetector

hypothesis_detector = HypothesisDetector(output_label="hypothesis")
hypothesis_detector.run(syntagma_segs)

# Display hypothesis syntagmas
for syntagma_seg in syntagma_segs:
    hypothesis_attr = syntagma_seg.attrs.get(label="hypothesis")[0]
    if hypothesis_attr.value:
        print(syntagma_seg.text)

As you can see, no hypothesis was detected in this document.

Warning

The default settings (rules and verbs) of HypothesisDetector are NOT exhaustive and may not yield satisfactory results. If you plan on using HypothesisDetector, please consider specifying your own set of rules and conjugated verbs that are specifically tailored to your data.

Passing context information to matched entities

Now that we have gathered all this contextual information, we want to propagate it to the entities that we will find in the document. This can be done using the attrs_to_copy mechanism that we have already seen, which is available to all NER operations:

from medkit.text.ner.hf_entity_matcher import HFEntityMatcher

# Create a matcher using a pretrained HuggingFace model
drbert_matcher = HFEntityMatcher(
    model="medkit/DrBERT-CASM2",
    attrs_to_copy=["section", "family", "hypothesis", "negation"],
)
# Run the matcher on the appropriate input segments
# and add the entities found back to the document
entities = drbert_matcher.run(syntagma_segs)
for entity in entities:
    doc.anns.add(entity)

# Print all entities with their contextual attributes
for entity in doc.anns.entities:
    print(entity.label, ":", entity.text)
    section_attr = entity.attrs.get(label="section")[0]
    print("section:", section_attr.value)
    family_attr = entity.attrs.get(label="family")[0]
    print("family:", family_attr.value)
    negation_attr = entity.attrs.get(label="negation")[0]
    print("negation:", negation_attr.value)
    hypothesis_attr = entity.attrs.get(label="hypothesis")[0]
    print("hypothesis:", hypothesis_attr.value)
    print()

problem : Thrombocytose essentielle
section: head
family: False
negation: False
hypothesis: False

problem : thrombocytose essentielle
section: antecedents
family: False
negation: False
hypothesis: False

test : nombre
section: antecedents
family: False
negation: False
hypothesis: False

test : plaquettes
section: antecedents
family: False
negation: False
hypothesis: False

treatment : Hydrea
section: antecedents
family: False
negation: False
hypothesis: False

test : biopsie de moelle osseuse
section: antecedents
family: False
negation: False
hypothesis: False

problem : thrombocytose essentielle
section: antecedents
family: False
negation: False
hypothesis: False

problem : mutation JAK-2
section: antecedents
family: False
negation: False
hypothesis: False

treatment : Hydrea
section: antecedents
family: False
negation: False
hypothesis: False

problem : polyarthrite rhumatoïde
section: antecedents
family: False
negation: False
hypothesis: False

test : d
section: antecedents
family: False
negation: False
hypothesis: False

test : énergie
section: antecedents
family: False
negation: False
hypothesis: False

test : statut de performance ECOG
section: antecedents
family: False
negation: False
hypothesis: False

problem : fièvre
section: antecedents
family: False
negation: True
hypothesis: False

problem : frissons
section: antecedents
family: False
negation: True
hypothesis: False

problem : sueurs nocturnes
section: antecedents
family: False
negation: True
hypothesis: False

problem : adénopathie
section: antecedents
family: False
negation: True
hypothesis: False

problem : nausées
section: antecedents
family: False
negation: True
hypothesis: False

problem : vomissements
section: antecedents
family: False
negation: True
hypothesis: False

treatment : vitamine D
section: current_drugs
family: False
negation: False
hypothesis: False

treatment : aspirine
section: current_drugs
family: False
negation: False
hypothesis: False

treatment : vitamine C
section: current_drugs
family: False
negation: False
hypothesis: False

problem : allergie médicamenteuse
section: allergies
family: False
negation: True
hypothesis: False

test : EXAMEN DES SYSTÈMES
section: clinical_exam
family: False
negation: False
hypothesis: False

treatment : Appendicectomie
section: antecedents
family: False
negation: False
hypothesis: False

treatment : Amygdalectomie
section: antecedents
family: False
negation: False
hypothesis: False

treatment : adénoïdectomie
section: antecedents
family: False
negation: False
hypothesis: False

treatment : Chirurgie bilatérale de la cataracte
section: antecedents
family: False
negation: False
hypothesis: False

problem : tabagisme
section: life_style
family: False
negation: False
hypothesis: False

problem : tumeur solide
section: family_history
family: True
negation: False
hypothesis: False

problem : hémopathies malignes
section: family_history
family: True
negation: True
hypothesis: False

test : EXAMEN PHYSIQUE
section: clinical_exam
family: False
negation: False
hypothesis: False

problem : pèse
section: clinical_exam
family: False
negation: False
hypothesis: False

Let’s visualize this in context with displacy:

from spacy import displacy
from medkit.text.spacy.displacy_utils import medkit_doc_to_displacy

# Define a custom formatter that will also display some context flags
# ex: "disorder[fn]" for an entity with label "disorder" and
# family and negation attributes set to True
def _custom_formatter(entity):
    label = entity.label
    flags = []
    
    family_attr = entity.attrs.get(label="family")[0]
    if family_attr.value:
        flags.append("f")
    negation_attr = entity.attrs.get(label="negation")[0]
    if negation_attr.value:
        flags.append("n")
    hypothesis_attr = entity.attrs.get(label="hypothesis")[0]
    if hypothesis_attr.value:
        flags.append("h")

    if flags:
        label += "[" + "".join(flags) + "]"
    
    return label

# Pass the formatter to medkit_doc_to_displacy()
displacy_data = medkit_doc_to_displacy(doc, entity_formatter=_custom_formatter)
displacy.render(docs=displacy_data, manual=True, style="ent")

PLAINTE PRINCIPALE :
Thrombocytose essentielleproblem .

ANTÉCÉDENTS DE LA MALADIE ACTUELLE :
C’est un M. de 64 ans que je suis pour une thrombocytose essentielleproblem . Il a été initialement diagnostiqué lorsqu’il a vu un hématologue pour la première fois le 09/07/07. A cette époque, son nombretest de plaquettestest était de 1 240 000. Il a d’abord commencé à prendre de l’ Hydreatreatment 1000 mg par jour. Le 07/11/07, il a subi une biopsie de moelle osseusetest , qui a montré une thrombocytose essentielleproblem . Il était positif pour la mutation JAK-2problem . Le 11/06/07, ses plaquettes étaient à 766 000. Sa dose actuelle d’ Hydreatreatment est maintenant de 1500 mg les lundis et vendredis et de 1000 mg tous les autres jours. Il a déménagé à ABCD en décembre 2009 pour tenter d’améliorer la polyarthrite rhumatoïdeproblem de sa femme. Dans l’ensemble, il se porte bien. Il a un bon niveau dtest ‘ énergietest et son statut de performance ECOGtest est de 0. Absence de fièvreproblem[n] , frissonsproblem[n] ou sueurs nocturnesproblem[n] . Pas d’ adénopathieproblem[n] . Pas de nauséesproblem[n] ni de vomissementsproblem[n] . Aucun changement dans les habitudes intestinales ou vésicales.

MÉDICAMENTS ACTUELS :
Hydrea 1500 mg les lundis et vendredis et 1000 mg les autres jours de la semaine, Mecir 1cp/j, vitamine Dtreatment 1/j, aspirinetreatment 80 mg 1/j et vitamine Ctreatment 1/j

ALLERGIES :
Aucune allergie médicamenteuseproblem[n] connue.

EXAMEN DES SYSTÈMEStest  
Correspondant à l’histoire de la maladie. Pas d’autre signes.

ANTÉCÉDENTS MÉDICAUX :
1. Appendicectomietreatment .
2. Amygdalectomietreatment et une adénoïdectomietreatment .
3. Chirurgie bilatérale de la cataractetreatment .
4. HTA.

MODE DE VIE :
Il a des antécédents de tabagismeproblem qu’il a arrêté à l’âge de 37 ans. Il consomme une boisson alcoolisée par jour. Il est marié. Il est directeur de laboratoire à la retraite.

ANTÉCÉDENTS FAMILIAUX :
Antécédents de tumeur solideproblem[f] dans sa famille mais aucun d’ hémopathies malignesproblem[fn] .

EXAMEN PHYSIQUEtest  :
Le patient pèseproblem 85.7 kg.

Adding context attributes retrospectively

What if we already have some entities that we imported from another source, and we want to attach the resulting contextual information obtained with medkit? In that case, one can copy attributes retrospectively using the AttributeDuplicator operation.

Wrapping it up

In this tutorial, we have seen how medkit can facilitate detection of contextual information with built-in and customizable rule-based detectors.

These detectors can be run on segments of different granularity, including as sentences or syntagmas, with their results stored as attributes.

In order to propagate these contextual attributes from the outermost segments down to the entities matched, we use the attrs_to_copy operation init parameter.