Comparative evaluation of drug recognition methods#
This pipeline compares two tools (NER1 and NER2) for recognizing drug names in clinical texts, compares their performance, and outputs two texts annotated with the tool evaluated as the best performer.
Overview of the pipeline#
Data preparation#
Download two clinical texts, with drug entites manually annotated.
import os
import tarfile
import tempfile
from pathlib import Path
# path to local data
extract_to = Path(tempfile.mkdtemp())
data_tarfile = Path.cwd() / "data.tar.gz"
# download and extract
tarfile.open(name=data_tarfile, mode="r|gz").extractall(extract_to)
data_dir = extract_to / "data"
print(f"Data dir: {data_dir}")
Data dir: /tmp/tmpv84inbxl/data
Read text documents with medkit
from medkit.core.text import TextDocument
doc_dir = data_dir / "mtsamplesen" / "annotated_doc"
docs = TextDocument.from_dir(path=doc_dir, pattern='[A-Z0-9].txt', encoding='utf-8')
print(docs[0].text)
DISCHARGE DIAGNOSES:
1. Gram-negative rod bacteremia, final identification and susceptibilities still pending.
2. History of congenital genitourinary abnormalities with multiple surgeries before the 5th grade.
3. History of urinary tract infections of pyelonephritis.
OPERATIONS PERFORMED: Chest x-ray July 24, 2007, that was normal. Transesophageal echocardiogram July 27, 2007, that was normal. No evidence of vegetations. CT scan of the abdomen and pelvis July 27, 2007, that revealed multiple small cysts in the liver, the largest measuring 9 mm. There were 2-3 additional tiny cysts in the right lobe. The remainder of the CT scan was normal.
HISTORY OF PRESENT ILLNESS: Briefly, the patient is a 26-year-old white female with a history of fevers. For further details of the admission, please see the previously dictated history and physical.
HOSPITAL COURSE: Gram-negative rod bacteremia. The patient was admitted to the hospital with suspicion of endocarditis given the fact that she had fever, septicemia, and Osler nodes on her fingers. The patient had a transthoracic echocardiogram as an outpatient, which was equivocal, but a transesophageal echocardiogram here in the hospital was normal with no evidence of vegetations. The microbiology laboratory stated that the Gram-negative rod appeared to be anaerobic, thus raising the possibility of organisms like bacteroides. The patient does have a history of congenital genitourinary abnormalities which were surgically corrected before the fifth grade. We did a CT scan of the abdomen and pelvis, which only showed some benign appearing cysts in the liver. There was nothing remarkable as far as her kidneys, ureters, or bladder were concerned. I spoke with Dr. Leclerc of infectious diseases, and Dr. Leclerc asked me to talk to the patient about any contact with animals, given the fact that we have had a recent outbreak of tularemia here in Utah. Much to my surprise, the patient told me that she had multiple pet rats at home, which she was constantly in contact with. I ordered tularemia and leptospirosis serologies on the advice of Dr. Leclerc, and as of the day after discharge, the results of the microbiology still are not back yet. The patient, however, appeared to be responding well to levofloxacin. I gave her a 2-week course of 750 mg a day of levofloxacin, and I have instructed her to follow up with Dr. Leclerc in the meantime. Hopefully by then we will have a final identification and susceptibility on the organism and the tularemia and leptospirosis serologies will return. A thought of ours was to add doxycycline, but again the patient clinically appeared to be responding to the levofloxacin. In addition, I told the patient that it would be my recommendation to get rid of the rats. I told her that if indeed the rats were carriers of infection and she received a zoonotic infection from exposure to the rats, that she could be in ongoing continuing danger and her children could also potentially be exposed to a potentially lethal infection. I told her very clearly that she should, indeed, get rid of the animals. The patient seemed reluctant to do so at first, but I believe with some coercion from her family, that she finally came to the realization that this was a recommendation worth following.
DISPOSITION
DISCHARGE INSTRUCTIONS: Activity is as tolerated. Diet is as tolerated.
MEDICATIONS: Levaquin 750 mg daily x14 days.
Followup is with Dr. Leclerc of infectious diseases. I gave the patient the phone number to call on Monday for an appointment. Additional followup is also with Dr. Leclerc, her primary care physician. Please note that 40 minutes was spent in the discharge.
Pipeline definition#
Create and run a three-step doc pipeline that:
Split sentences in texts
Run PII detection for deidentification
Recognize drug entities with NER1: a dictionnary-based approach named UMLSMatcher
Recognize drug entities with NER2: a Transformer-based approach, see https://huggingface.co/samrawal/bert-large-uncased_med-ner
Sentence tokenizer#
from medkit.text.segmentation import SentenceTokenizer
# By default, SentenceTokenizer will use a list of punctuation chars to detect sentences.
sentence_tokenizer = SentenceTokenizer(
# Label of the segments created and returned by the operation
output_label="sentence",
# Keep the punctuation character inside the sentence segments
keep_punct=True,
# Also split on newline chars, not just punctuation characters
split_on_newlines=True,
)
PII detector#
from medkit.text.deid import PIIDetector
pii_detector = PIIDetector(name="deid")
Collecting en-core-web-lg==3.8.0
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl (400.7 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/400.7 MB ? eta -:--:--
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/400.7 MB 266.8 MB/s eta 0:00:02
━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.1/400.7 MB 180.0 MB/s eta 0:00:02
━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━ 133.7/400.7 MB 221.4 MB/s eta 0:00:02
━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 202.6/400.7 MB 251.6 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 273.2/400.7 MB 266.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 326.9/400.7 MB 314.0 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 372.5/400.7 MB 291.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 400.6/400.7 MB 299.4 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 400.7/400.7 MB 41.3 MB/s eta 0:00:00
?25h
Installing collected packages: en-core-web-lg
Successfully installed en-core-web-lg-3.8.0
✔ Download and installation successful
You can now load the package via spacy.load('en_core_web_lg')
⚠ Restart to reload dependencies
If you are in a Jupyter or Colab notebook, you may need to restart Python in
order to load all the package's dependencies. You can do this by selecting the
'Restart kernel' or 'Restart runtime' option.
Dictionnary-based drug recognizer#
import shutil
from medkit.text.ner import UMLSMatcher
umls_data_dir = data_dir / "UMLS" / "2023AB" / "META"
umls_cache_dir = Path.cwd() / ".umls_cache"
shutil.rmtree(umls_cache_dir, ignore_errors=True)
umls_matcher = UMLSMatcher(
# Directory containing the UMLS files with terms and concepts
umls_dir=umls_data_dir,
# Language to use (English)
language="ENG",
# Where to store the temp term database of the matcher
cache_dir=umls_cache_dir,
# Semantic groups to consider
semgroups=["CHEM"],
# Don't be case-sensitive
lowercase=True,
# Convert special chars to ASCII before matching
normalize_unicode=True,
name="NER1"
)
/home/docs/checkouts/readthedocs.org/user_builds/medkit/envs/latest/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
0%| | 0.00/13.9k [00:00<?, ?B/s]
100%|██████████| 13.9k/13.9k [00:00<00:00, 13.2MB/s]
Transformer-based drug recognizer#
from medkit.text.ner.hf_entity_matcher import HFEntityMatcher
# an alternate model: "Clinical-AI-Apollo/Medical-NER"
bert_matcher = HFEntityMatcher(
model="samrawal/bert-large-uncased_med-ner", name="NER2"
)
Device set to use cpu
Pipeline assembly#
from medkit.core import DocPipeline, Pipeline, PipelineStep
pipeline = Pipeline(
steps=[
PipelineStep(sentence_tokenizer, input_keys=["full_text"], output_keys=["sentence"]),
PipelineStep(pii_detector, input_keys=["sentence"], output_keys=["sentence_"]),
PipelineStep(umls_matcher, input_keys=["sentence_"], output_keys=["ner1_drug"]),
PipelineStep(bert_matcher, input_keys=["sentence_"], output_keys=["ner2_drug"]),
],
input_keys=["full_text"],
output_keys=["sentence_", "ner1_drug", "ner2_drug"],
)
doc_pipeline = DocPipeline(pipeline=pipeline)
doc_pipeline.run(docs)
Performance evaluation#
from medkit.io.brat import BratInputConverter
# Load text with annotations in medkit (our ground truth)
brat_converter = BratInputConverter()
ref_docs = brat_converter.load(doc_dir)
# Display selected drug annotations
for ann in ref_docs[0].anns.get(label="Drug"):
print(f"{ann.text} in {ann.spans}")
Levaquin in [Span(start=3391, end=3399)]
doxycycline in [Span(start=2587, end=2598)]
levofloxacin in [Span(start=2262, end=2274)]
levofloxacin in [Span(start=2322, end=2334)]
levofloxacin in [Span(start=2666, end=2678)]
## Compute some stats
print(f"Number of documents: {len(docs)}")
for i, doc in enumerate(docs):
print(f"Document {doc.uid}:")
# On annotations made by NER1 and NER2
sentence_nb = len(doc.anns.get(label="sentence"))
print(f"\t{sentence_nb} sentences,")
ner1_drug_nb = len(doc.anns.get(label="chemical"))
print(f"\t{ner1_drug_nb} drugs found with NER1,")
ner2_drug_nb = len(doc.anns.get(label="m"))
print(f"\t{ner2_drug_nb} drugs found with NER2,")
# On the manual annotation (our ground truth)
gt_nb = len(ref_docs[i].anns.get(label="Drug"))
print(f"\t{gt_nb} drugs manually annotated.")
Number of documents: 2
Document 7383a7b6-cd0f-11ef-bb67-0242ac110002:
0 sentences,
0 drugs found with NER1,
0 drugs found with NER2,
5 drugs manually annotated.
Document 7383ad4c-cd0f-11ef-bb67-0242ac110002:
0 sentences,
1 drugs found with NER1,
2 drugs found with NER2,
15 drugs manually annotated.
## Evaluate performance metrics of the NER1 and NER2 tools
from medkit.text.metrics.ner import SeqEvalEvaluator
import pandas as pd
def results_to_df(_results, _title):
results_list = list(_results.items())
arranged_results = {"Entities": ['P', 'R', 'F1']}
accuracy = round(results_list[4][1], 2)
for i in range(5, len(results_list), 4):
key = results_list[i][0][:-10]
arranged_results[key] = [round(results_list[n][1], 2) for n in [i, i + 1, i + 2]]
df = pd.DataFrame(arranged_results, index=[f"{_title} (acc={accuracy})", '', '']).T
return df
predicted_entities1=[]
predicted_entities2=[]
dfs = []
for doc in docs:
predicted_entities1.append(doc.anns.get(label="chemical"))
predicted_entities2.append(doc.anns.get(label="m"))
# Annotations of NER1 are labelled as 'chemical', NER2 as 'm', but as 'Drug' in the ground truth
# The following dic enables remappings various labels of the same type of entites
remapping= {"chemical": "Drug", "m": "Drug"}
evaluator = SeqEvalEvaluator(return_metrics_by_label=True, average='weighted', labels_remapping=remapping)
# eval of NER2
results1 = evaluator.compute(ref_docs, predicted_entities1)
dfs.append(results_to_df(_results=results1, _title="NER1"))
#print(results_to_df(_results=results1, _title="umls_matcher"))
# eval of NER2
results2 = evaluator.compute(ref_docs, predicted_entities2)
dfs.append(results_to_df(_results=results2, _title="NER2"))
print(pd.concat(dfs, axis=1))
NER1 (acc=0.98) NER2 (acc=0.98)
Entities P R F1 P R F1
Drug 1.0 0.05 0.1 0.5 0.05 0.09
## Read new unannotated documents
## Write annotations of tool NER2 in the brat format
from medkit.io.brat import BratOutputConverter
in_path = data_dir / "mtsamplesen" / "unannotated_doc"
# reload raw documents
final_docs = TextDocument.from_dir(
path=Path(in_path),
pattern='[A-Z0-9].txt',
encoding='utf-8',
)
# simplified pipeline, with only the best NER tool (NER2)
pipeline2 = Pipeline(
steps=[
PipelineStep(
sentence_tokenizer,
input_keys=["full_text"],
output_keys=["sentence"],
),
PipelineStep(
pii_detector,
input_keys=["sentence"],
output_keys=["sentence_"],
),
PipelineStep(
bert_matcher,
input_keys=["sentence_"],
output_keys=["ner2_drug"],
),
],
input_keys=["full_text"],
output_keys=["ner2_drug"],
)
doc_pipeline2 = DocPipeline(pipeline=pipeline2)
doc_pipeline2.run(final_docs)
# filter annotations to keep only drug annotations
# sensitive information can also be removed here
output_docs = [
TextDocument(text=doc.text, anns=doc.anns.get(label="m"))
for doc in final_docs
]
# Define Output Converter with default params,
# transfer all annotations and attributes
brat_output_converter = BratOutputConverter()
out_path = data_dir / "mtsamplesen" / "ner2_out"
# save the annotation with the best tool (considering F1 only) in `out_path`
brat_output_converter.save(
output_docs,
dir_path=out_path,
doc_names=["ner2_6", "ner2_7"],
)
Annotations of the discharge summary 6.txt, displayed with Brat
Annotations of the discharge summary 7.txt (partial view), displayed with Brat
