python – How to remove dataframe rows with empty objects

Here is a way to do what your question asks based on the annotation column which contains values ​​that look like lists but are actually strings:

df = df[df.annotation.apply(eval).str.len() > 0]

Column annotation from sample input (taken from df.head(15) in your question):

                                    annotation
0             ['dad with recent heart attcak']
1                ['mom with "thyroid disease']
2                           ['chest pressure']
3         ['intermittent episodes', 'episode']
4     ['felt as if he were going to pass out']
5                                           []
6       ['adderall', 'adderrall', 'adderrall']
7                                           []
8                                           []
9   ['palpitations', 'heart beating/pounding']
10                           ['3-4 months of']
11                                    ['17yo']
12                                       ['M']

Column annotation of result:

                                    annotation
0             ['dad with recent heart attcak']
1                ['mom with "thyroid disease']
2                           ['chest pressure']
3         ['intermittent episodes', 'episode']
4     ['felt as if he were going to pass out']
6       ['adderall', 'adderrall', 'adderrall']
9   ['palpitations', 'heart beating/pounding']
10                           ['3-4 months of']
11                                    ['17yo']
12                                       ['M']

Full test code:

import pandas as pd

df = pd.DataFrame({'id': {0: '00016_000', 1: '00016_001', 2: '00016_002', 3: '00016_003', 4: '00016_004', 5: '00016_005', 6: '00016_006', 7: '00016_007', 8: '00016_008', 9: '00016_009', 10: '00016_010', 11: '00016_011', 12: '00016_012'}, 'case_num': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 0}, 'pn_num': {0: 16, 1: 16, 2: 16, 3: 16, 4: 16, 5: 16, 6: 16, 7: 16, 8: 16, 9: 16, 10: 16, 11: 16, 12: 16}, 'feature_num': {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12}, 'annotation': {0: "['dad with recent heart attcak']", 1: '['mom with "thyroid disease']', 2: "['chest pressure']", 3: "['intermittent episodes', 'episode']", 4: "['felt as if he were going to pass out']", 5: '[]', 6: "['adderall', 'adderrall', 'adderrall']", 7: '[]', 8: '[]', 9: "['palpitations', 'heart beating/pounding']", 10: "['3-4 months of']", 11: "['17yo']", 12: "['M']"}, 'location': {0: "['696 724']", 1: "['668 693']", 2: "['203 217']", 3: "['70 91', '176 183']", 4: "['222 258']", 5: '[]', 6: "['321 329', '404 413', '652 661']", 7: '[]', 8: '[]', 9: "['26 38', '96 118']", 10: "['56 69']", 11: "['5 9']", 12: "['10 11']"}, 'feature_text': {0: 'Family-history-of-MI-OR-Family-history-of-myocardial-infarction', 1: 'Family-history-of-thyroid-disorder', 2: 'Chest-pressure', 3: 'Intermittent-symptoms', 4: 'Lightheaded', 5: 'No-hair-changes-OR-no-nail-changes-OR-no-temperature-intolerance', 6: 'Adderall-use', 7: 'Shortness-of-breath', 8: 'Caffeine-use', 9: 'heart-pounding-OR-heart-racing', 10: 'Few-months-duration', 11: '17-year', 12: 'Male'}, 'pn_history': {0: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 1: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 2: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 3: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 4: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 5: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 6: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 7: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 8: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 9: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 10: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 11: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms', 12: 'HPI: 17yo M presents with palpitations. Patient reports 3-4 months of intermittent episodes of "heart beating/pounding out of my chest." 2 days ago during a soccer game had an episode, but this time had chest pressure and felt as if he were going to pass out (did not lose conciousness). Of note patient endorses abusing adderall, primarily to study (1-3 times per week). Before recent soccer game, took adderrall night before and morning of game. Denies shortness of breath, diaphoresis, fevers, chills, headache, fatigue, changes in sleep, changes in vision/hearing, abdominal paun, changes in bowel or urinary habits. rnPMHx: nonernRx: uses friends adderrallrnFHx: mom with "thyroid disease," dad with recent heart attcakrnAll: nonernImmunizations: up to daternSHx: Freshmen in college. Endorses 3-4 drinks 3 nights / week (on weekends), denies tabacco, endorses trying marijuana. Sexually active with girlfriend x 1 year, uses condoms'}})

print(df.annotation.to_frame())
df = df[df.annotation.apply(eval).str.len() > 0]
print(df.annotation.to_frame())

Leave a Comment