Programmatic Access

EDirect connects to Entrez through the Entrez Programming Utilities (EUtils) interface. It supports searching by indexed terms, looking up precomputed neighbors or links, filtering results by date or category, and downloading record summaries or reports.

Navigation programs (esearch, elink, efilter, and efetch) communicate by means of a small structured message, which can be passed invisibly between operations with a Unix pipe. The message includes the current database, so it does not need to be given as an argument after the first step.

Accessory programs (nquire, transmute, and xtract) can help eliminate the need for writing custom software to answer ad hoc questions. Queries can move effortlessly between EDirect programs and Unix utilities or scripts to perform actions that cannot be accomplished entirely within Entrez.

EDirect programs are designed to work on large sets of data. They handle many technical details behind the scenes (avoiding the learning curve normally required for EUtils programming). There is no need to use a script to loop over records in small groups, or write code to retry a query after a transient network or server failure, or add a time delay between requests. All of those features are already built into the system.

Unix programs are run by typing the name of the program and then supplying any required or optional arguments on the command line. Argument names are letters or words that start with a dash ("‑") character.

Each program has a ‑help command that prints detailed information about available arguments.

Running xtract ‑unix will print a reference sheet of useful Unix commands and constructs. Additional documentation can be obtained by typing "man" followed by a Unix command name.

Navigation Functions

Esearch performs a new Entrez search using terms in indexed fields. It requires a ‑db argument for the database name and uses ‑query for the search terms. For PubMed, without field qualifiers, the server uses automatic term mapping to compose a search strategy by translating the supplied query:

  esearch -db pubmed -query "selective serotonin reuptake inhibitor"

Search terms can also be qualified with a bracketed field name to match within the specified index:

  esearch -db nuccore -query "insulin [PROT] AND rodents [ORGN]"

Elink looks up precomputed neighbors within a database, or follows annotated links to records in other databases:

  elink -related

  elink -target gene

Elink also connects to the NIH Open Citation Collection service (2) to find publications that cite the selected PubMed articles, or to retrieve the reference lists of PubMed records:

  elink -cited

  elink -cites

Efilter limits the results of a previous query, with shortcuts that can also be used in esearch:

  efilter -molecule genomic -location chloroplast -country sweden -mindate 1985

Efetch downloads selected records or reports in a style designated by ‑format:

  efetch -format abstract

Einfo can report the fields and links that are indexed for each database:

  einfo -db protein -fields

This will return a table of field abbreviations and names indexed for proteins:

  ACCN    Accession
  ALL     All Fields
  ASSM    Assembly
  AUTH    Author
  BRD     Breed
  CULT    Cultivar
  DIV     Division
  ECNO    EC/RN Number
  FILT    Filter
  FKEY    Feature key
  …

Constructing Multi-Line or Multi-Step Queries

A single command can be continued on the next line by typing the Unix backslash ("\") escape character immediately before pressing the Return key. Pay careful attention to the exact spacing if the line break is within a quoted argument:

  esearch -db biosample -query "package mims metagenome/environmental, \
  water version 6 0 [PROP] AND ncbi [FILT] AND biosample sra [FILT]"

EDirect allows individual operations to be described separately, combining them into a multi-step query by using the Unix vertical bar ("|") pipe symbol:

  esearch -db pubmed -query "tn3 transposition immunity" | efetch -format apa

The vertical bar also allows query steps to be placed on separate lines:

  esearch -db pubmed -query "raynaud disease AND fish oil" |
  efetch -format medline

In most modern versions of Unix the vertical bar pipe symbol also allows the query to continue on the next line, without the need for an additional backslash. That convention will be followed for the remainder of this document.

Accessory Programs

Nquire retrieves data from remote servers with URLs constructed from command line arguments:

  nquire -get https://icite.od.nih.gov api/pubs -pmids 2539356 |

Transmute converts a concatenated stream of JSON objects or other structured formats into XML:

  transmute -j2x -nest depth |

Xtract uses waypoints to navigate complex XML hierarchies, and selects data values by field name:

  xtract -pattern data -element cited_by |

The resulting output can be post-processed by Unix utilities or scripts:

  fmt -w 1 | sort -V | uniq

Mixed-Content Records

The ‑mixed flag is needed when processing XML or HTML records that contain embedded markup instructions. Using this argument, transmute ‑format can reformat the records for easier reading, and xtract ‑verify can confirm that all opening and closing tags are properly balanced:

  efetch -db pmc -id 4305238 -format xml |
  transmute -mixed -format | xtract -mixed -verify

Retrieving PubMed Reports

Piping PubMed query results to efetch and specifying the "abstract" format:

  esearch -db pubmed -query "lycopene cyclase" |
  efetch -format abstract

returns a set of reports that can be read by a person:

  …
  85. PLoS One. 2013;8(3):e58144. doi: 10.1371/journal.pone.0058144. Epub …

  Levels of lycopene β-cyclase 1 modulate carotenoid gene expression and
  accumulation in Daucus carota.

  Moreno JC(1), Pizarro L, Fuentes P, Handford M, Cifuentes V, Stange C.

  Author information:
  (1)Departamento de Biología, Facultad de Ciencias, Universidad de Chile,
  Santiago, Chile.

  Plant carotenoids are synthesized and accumulated in plastids through a
  highly regulated pathway. Lycopene β-cyclase (LCYB) is a key enzyme
  involved directly in the synthesis of α-carotene and β-carotene through
  …

If "medline" format is used instead:

  esearch -db pubmed -query "lycopene cyclase" |
  efetch -format medline

the output can be entered into common bibliographic management software packages:

  …
  PMID- 23555569
  OWN - NLM
  STAT- MEDLINE
  DA  - 20130404
  DCOM- 20130930
  LR  - 20131121
  IS  - 1932-6203 (Electronic)
  IS  - 1932-6203 (Linking)
  VI  - 8
  IP  - 3
  DP  - 2013
  TI  - Levels of lycopene beta-cyclase 1 modulate carotenoid gene expression
        and accumulation in Daucus carota.
  PG  - e58144
  LID - 10.1371/journal.pone.0058144 [doi]
  AB  - Plant carotenoids are synthesized and accumulated in plastids
        through a highly regulated pathway. Lycopene beta-cyclase (LCYB) is a
        key enzyme involved directly in the synthesis of alpha-carotene and
        …

Retrieving Sequence Reports

Nucleotide and protein records can be downloaded in FASTA format:

  esearch -db protein -query "lycopene cyclase" |
  efetch -format fasta

which consists of a definition line followed by the sequence:

  …
  >gi|735882|gb|AAA81880.1| lycopene cyclase [Arabidopsis thaliana]
  MDTLLKTPNKLDFFIPQFHGFERLCSNNPYPSRVRLGVKKRAIKIVSSVVSGSAALLDLVPETKKENLDF
  ELPLYDTSKSQVVDLAIVGGGPAGLAVAQQVSEAGLSVCSIDPSPKLIWPNNYGVWVDEFEAMDLLDCLD
  TTWSGAVVYVDEGVKKDLSRPYGRVNRKQLKSKMLQKCITNGVKFHQSKVTNVVHEEANSTVVCSDGVKI
  QASVVLDATGFSRCLVQYDKPYNPGYQVAYGIIAEVDGHPFDVDKMVFMDWRDKHLDSYPELKERNSKIP
  TFLYAMPFSSNRIFLEETSLVARPGLRMEDIQERMAARLKHLGINVKRIEEDERCVIPMGGPLPVLPQRV
  VGIGGTAGMVHPSTGYMVARTLAAAPIVANAIVRYLGSPSSNSLRGDQLSAEVWRDLWPIERRRQREFFC
  FGMDILLKLDLDATRRFFDAFFDLQPHYWHGFLSSRLFLPELLVFGLSLFSHASNTSRLEIMTKGTVPLA
  KMINNLVQDRD
  …

Sequence records can also be obtained as GenBank or GenPept flatfiles:

  esearch -db protein -query "lycopene cyclase" |
  efetch -format gp

which have features annotating particular regions of the sequence:

  …
  LOCUS       AAA81880                 501 aa            linear   PLN …
  DEFINITION  lycopene cyclase [Arabidopsis thaliana].
  ACCESSION   AAA81880
  VERSION     AAA81880.1  GI:735882
  DBSOURCE    locus ATHLYC accession L40176.1
  KEYWORDS    .
  SOURCE      Arabidopsis thaliana (thale cress)
    ORGANISM  Arabidopsis thaliana
              Eukaryota; Viridiplantae; Streptophyta; Embryophyta;
              Tracheophyta; Spermatophyta; Magnoliophyta; eudicotyledons;
              Brassicales; Brassicaceae; Camelineae; Arabidopsis.
  REFERENCE   1  (residues 1 to 501)
    AUTHORS   Scolnik,P.A. and Bartley,G.E.
    TITLE     Nucleotide sequence of lycopene cyclase (GenBank L40176) from
              Arabidopsis (PGR95-019)
    JOURNAL   Plant Physiol. 108 (3), 1343 (1995)
  …
  FEATURES             Location/Qualifiers
       source          1..501
                       /organism="Arabidopsis thaliana"
                       /db_xref="taxon:3702"
       Protein         1..501
                       /product="lycopene cyclase"
       transit_peptide 1..80
       mat_peptide     81..501
                       /product="lycopene cyclase"
       CDS             1..501
                       /gene="LYC"
                       /coded_by="L40176.1:2..1507"
  ORIGIN
          1 mdtllktpnk ldffipqfhg ferlcsnnpy psrvrlgvkk raikivssvv sgsaalldlv
         61 petkkenldf elplydtsks qvvdlaivgg gpaglavaqq vseaglsvcs idpspkliwp
        121 nnygvwvdef eamdlldcld ttwsgavvyv degvkkdlsr pygrvnrkql kskmlqkcit
        181 ngvkfhqskv tnvvheeans tvvcsdgvki qasvvldatg fsrclvqydk pynpgyqvay
        241 giiaevdghp fdvdkmvfmd wrdkhldsyp elkernskip tflyampfss nrifleetsl
        301 varpglrmed iqermaarlk hlginvkrie edercvipmg gplpvlpqrv vgiggtagmv
        361 hpstgymvar tlaaapivan aivrylgsps snslrgdqls aevwrdlwpi errrqreffc
        421 fgmdillkld ldatrrffda ffdlqphywh gflssrlflp ellvfglslf shasntsrle
        481 imtkgtvpla kminnlvqdr d
  //
  …

Restricting Query Results

The current results can be refined by further term searching in Entrez:

  esearch -db pubmed -query "opsin gene conversion" |
  elink -related |
  efilter -query "tetrachromacy"

Limiting by Date

Results can also be filtered by date. For example, the following statements:

  efilter -days 60 -datetype PDAT

  efilter -mindate 2000

  efilter -maxdate 1985

  efilter -mindate 1990 -maxdate 1999

restrict results to articles published in the previous two months, since the beginning of 2000, through the end of 1985, or in the 1990s, respectively. YYYY/MM and YYYY/MM/DD date formats are also accepted.

Fetch by Identifier

Efetch and elink can take a list of numeric identifiers or accessions in an ‑id argument:

  efetch -db pubmed -id 7252148,1937004 -format xml

  efetch -db nuccore -id 1121073309 -format acc

  efetch -db protein -id 3OQZ_a -format fasta

  efetch -db bioproject -id PRJNA257197 -format docsum

  efetch -db pmc -id PMC209839 -format medline

  elink -db pubmed -id 2539356 -cites

without the need for a preceding esearch command.

Non-integer accessions will be looked up with an internal search, using the appropriate field for the database:

  esearch -db bioproject -query "PRJNA257197 [PRJA]" |
  efetch -format uid | …

Most databases use the [ACCN] field for identifier lookup, but there are a few exceptions:

  annotinfo      [ASAC]
  assembly       [ASAC]
  bioproject     [PRJA]
  books          [AID]
  clinvar        [VACC]
  gds            [ALL]
  genome         [PRJA]
  geoprofiles    [NAME]
  gtr            [GTRACC]
  mesh           [MHUI]
  nuccore        [ACCN] or [PACC]
  pcsubstance    [SRID]
  snp            [RS] or [SS]

(For ‑db pmc it merely removes any "PMC" prefix from the integer identifier.)

For backward compatibility, esummary is a shortcut for esearch ‑format docsum:

  esummary -db bioproject -id PRJNA257197

  esummary -db sra -id SRR5437876

Reading Large Lists of Identifiers

Efetch and elink can also read a large list of identifiers or accessions piped in through stdin:

  cat "file_of_identifiers.txt" |
  efetch -db pubmed -format docsum

or from a file indicated by an ‑input argument:

  efetch -input "file_of_identifiers.txt" -db pubmed -format docsum

As mentioned above, there is no need to use a script to split the identifiers into smaller groups or add a time delay between individual requests, since that functionality is already built into EDirect.

Processing Groups of Terms

The join‑into‑groups‑of script combines text containing words, numbers, or identifiers into lines of comma-separated items:

  join-into-groups-of 20 |

Splitting text into one element per line is done by piping through word‑at‑a‑time for prose or accn‑at‑a‑time for accessions. These functions also lower-case the output:

  echo "$abstract_in_variable" |
  word-at-a-time |
  while read term
  do
    echo "$term"
  done |

The sort‑uniq‑count‑rank function then sorts the individual terms, counts the number of times each one appears in the list, and finally resorts the terms based on their frequency of occurrence:

  sort-uniq-count-rank

A faster version of accn‑at‑a‑time runs without lower-case transformation:

  tr -cs a-zA-Z0-9_. '\n' |

Qualifying Queries by Indexed Field

Query terms in esearch or efilter can be qualified by entering an indexed field abbreviation in brackets. Boolean operators and parentheses can also be used in the query expression for more complex searches.

Commonly used fields for PubMed queries include:

  [AFFL]    Affiliation           [LANG]    Language
  [ALL]     All Fields            [MAJR]    MeSH Major Topic
  [AUTH]    Author                [SUBH]    MeSH Subheading
  [FAUT]    Author - First        [MESH]    MeSH Terms
  [LAUT]    Author - Last         [PTYP]    Publication Type
  [CRDT]    Date - Create         [WORD]    Text Word
  [PDAT]    Date - Publication    [TITL]    Title
  [FILT]    Filter                [TIAB]    Title/Abstract
  [JOUR]    Journal               [UID]     UID

and a qualified query looks like:

  "Tager HS [AUTH] AND glucagon [TIAB]"

Filters that limit search results to subsets of PubMed include:

  humans [MESH]
  pharmacokinetics [MESH]
  chemically induced [SUBH]
  all child [FILT]
  english [FILT]
  freetext [FILT]
  has abstract [FILT]
  historical article [FILT]
  randomized controlled trial [FILT]
  clinical trial, phase ii [PTYP]
  review [PTYP]

Sequence databases are indexed with a different set of search fields, including:

  [ACCN]    Accession       [MLWT]    Molecular Weight
  [ALL]     All Fields      [ORGN]    Organism
  [AUTH]    Author          [PACC]    Primary Accession
  [GPRJ]    BioProject      [PROP]    Properties
  [BIOS]    BioSample       [PROT]    Protein Name
  [ECNO]    EC/RN Number    [SQID]    SeqID String
  [FKEY]    Feature key     [SLEN]    Sequence Length
  [FILT]    Filter          [SUBS]    Substance Name
  [GENE]    Gene Name       [WORD]    Text Word
  [JOUR]    Journal         [TITL]    Title
  [KYWD]    Keyword         [UID]     UID

and a sample query in the protein database is:

  "alcohol dehydrogenase [PROT] NOT (bacteria [ORGN] OR fungi [ORGN])"

Additional examples of subset filters in sequence databases are:

  mammalia [ORGN]
  mammalia [ORGN:noexp]
  txid40674 [ORGN]
  cds [FKEY]
  lacz [GENE]
  beta galactosidase [PROT]
  protein snp [FILT]
  reviewed [FILT]
  country united kingdom glasgow [TEXT]
  biomol genomic [PROP]
  dbxref flybase [PROP]
  gbdiv phg [PROP]
  phylogenetic study [PROP]
  sequence from mitochondrion [PROP]
  src cultivar [PROP]
  srcdb refseq validated [PROP]
  150:200 [SLEN]

(The calculated molecular weight (MLWT) field is only indexed for proteins and structures, not nucleotides.)

See efilter ‑help for a list of filter shortcuts available for several Entrez databases.

Examining Intermediate Results

EDirect navigation functions produce a custom XML message with the relevant fields (database, web environment, query key, and record count) that can be read by the next command in the pipeline. EDirect may store intermediate results on the Entrez history server or instantiate them in the XML message.

The results of each step in a query can be examined to confirm expected behavior before adding the next step. The Count field in the ENTREZ_DIRECT object contains the number of records returned by the previous step. A good measure of query success is a reasonable (non‑zero) count value. For example:

  esearch -db protein -query "tryptophan synthase alpha chain [PROT]" |
  efilter -query "28000:30000 [MLWT]" | tee /dev/tty |
  elink -target structure -log |
  efilter -query "0:2 [RESO]"

produces:

  <ENTREZ_DIRECT>
    <Db>protein</Db>
    <WebEnv>MCID_6a56bdcecaf7c3511b0de8e1</WebEnv>
    <QueryKey>2</QueryKey>
    <Count>17617</Count>
    <Step>2</Step>
    <Elapsed>1</Elapsed>
  </ENTREZ_DIRECT>
  <ENTREZ_DIRECT>
    <Db>structure</Db>
    <Count>57</Count>
    <Step>4</Step>
    <Id>5822</Id>
    <Id>11701</Id>
    …
    <Id>211309</Id>
    <Id>255950</Id>
    <Elapsed>203</Elapsed>
  </ENTREZ_DIRECT>

The first message shows the molecular weight range filter returning 17,617 proteins. (Piping through tee /dev/tty prints the result on the terminal and also pipes it to the next step.) The second message shows the final result of 57 protein structures having the desired (X‑ray crystallographic) atomic position resolution.

(When saving to history, the QueryKey value may differ from Step because the elink command splits its query into smaller chunks to avoid server truncation limits and timeout errors.)

Combining Independent Queries

Independent esearch/elink/efilter/efetch pipelines can be combined by passing the separate UID results to the intersect‑uid‑lists script. (The output of both pipelines must be in the same target database.)

(The history server's "#" convention is no longer used for this example. Intermediate results are now saved in Unix shell script variables, which will be discussed in a later section of this document.)

For example, the query:

  amyl=$(
    esearch -db protein -query "amyloid* [PROT]" -log | tee /dev/tty |
    elink -target pubmed |
    efetch -format uid
  ) 

  apoe=$(
    esearch -db gene -query "apo* [GENE]" -log | tee /dev/tty |
    elink -target pubmed |
    efetch -format uid
  )

  both=$( intersect-uid-lists <( echo "$amyl" ) <( echo "$apoe" ) )

  echo "$both" |
  efetch -db pubmed -format docsum |
  xtract -mixed -pattern DocumentSummary -element Id Title |
  cat -v

uses truncation searching (entering the first few letters of a word followed by an asterisk) for the two main concepts, and return titles of papers with links to both amyloid protein sequences and apolipoprotein gene records:

  23962925    Genome analysis reveals insights into physiology and …
  23959870    Low levels of copper disrupt brain amyloid-β homeostasis …
  23371554    Genomic diversity and evolution of the head crest in the …
  23251661    Novel genetic loci identified for the pathophysiology of …
  …

Format Customization

By default, the ‑pattern argument divides the results into rows, while placement of data into columns is controlled by ‑element, to create a tab-delimited table.

Formatting commands allow extensive customization of the output. The line break between ‑pattern rows is changed with ‑ret, while the tab character between ‑element columns is modified by ‑tab.

Multiple instances of the same element are distinguished using ‑sep, which controls their separation independently of the ‑tab command. The following query:

  efetch -db pubmed -id 6271474,24178092 -format docsum |
  xtract -pattern DocumentSummary -sep "|" -element Id PubDate Name

returns a tab-delimited table with individual author names separated by vertical bars:

  6271474     1981        Casadaban MJ|Chou J|Lemaux P|Tu CP|Cohen SN
  24178092    1994 Dec    Garber ED|Ruddat M

The ‑sep value also applies to distinct ‑element arguments that are grouped with commas. This can be used to keep data from multiple related fields in the same column:

  -sep " " -element Initials,LastName

Groups of fields are preceded by the ‑pfx value and followed by the ‑sfx value, both of which are initially empty.

The ‑def command sets a default placeholder to be printed when none of the comma-separated fields in an ‑element clause are present:

  -def "-" -sep " " -element Year,Month,MedlineDate

Limit by Parent

An ‑element argument can use the parent / child construct to limit selection when items can only be disambiguated by position, not by name. In this case, it prevents the display of additional PMIDs that might be present inside CommentsCorrections objects deeper within the MedlineCitation container:

  xtract -pattern PubmedArticle -element MedlineCitation/PMID

Element Variants

Derivatives of ‑element were initially created to avoid having to write post-processing scripts just to perform trivial modifications or integer calculations on extracted data. Other variants were added for content normalization, report formatting, or index generation. The commands are in several categories:

  Positional:       -first, -last, -even, -odd, -backward

  Numeric:          -num, -len, -inc, -dec, -mod, -bin, -hex, -bit

  Statistics:       -sum, -acc, -min, -max, -dev, -med

  Averages:         -avg, -geo, -hrm, -rms

  Logarithms:       -sqt, -lge, -lg2, -log

  Character:        -encode, -upper, -title, -mirror, -alpha, -alnum

  String:           -basic, -plain, -simple, -prose

  Text:             -words, -pairs, -letters, -split, -order, -reverse, -pentamers

  Citation:         -year, -month, -date, -auth, -initials, -page, -author, -journal

  Sequence:         -revcomp, -fasta, -ncbi2na, -molwt, -pept, -nucleic

  Translation:      -cds2prot, -gcode, -frame

  Coordinate:       -0-based, -1-based, -ucsc-based

  Variation:        -hgvs

  Frequency:        -histogram

  Expression:       -reg, -exp, -replace

  Substitution:     -transform, -translate

  Indexing:         -indexer, -aliases, -classify

  Miscellaneous:    -doi, -wct, -trim, -pad, -mask, -accession, -numeric

The original ‑element prefix shortcuts, "#" and "%", are redirected to ‑num and ‑len, respectively.

See xtract ‑help for a brief description of each command.

Substring Extraction

Following an element name with square brackets allows subset selection by numeric range (with a colon between the character start and stop positions), or permits removal of leading and trailing text (with a vertical bar [or caret] separating the required [or optional] interior prefix and suffix endpoints):

  -author Initials[1:1] -numeric "ArticleId[PMC|]" -upper "Accession[^.]"

XML Repackaging

Repackaging commands (‑wrp, ‑enc, and ‑pkg) wrap extracted data values with bracketed XML tags given only the object name. For example, "‑wrp Word" issues the following formatting instructions:

  -pfx "<Word>" -sep "</Word><Word>" -sfx "</Word>"

It also sets an internal flag to ensure that data values containing encoded symbols (ampersands, angle brackets, apostrophes, and quotation marks) remain properly encoded inside the new XML.

Combining ‑wrp commands and ‑element variants can break up complex tasks into a series of simpler operations. No custom reading or writing code is needed, since the data remains in XML format.

Generating Attributes

Additional commands (‑tag, ‑att, ‑atr, ‑cls, ‑slf, and ‑end) allow generation of XML tags with attributes. The following will produce regular and self-closing XML objects, respectively:

  -tag Item -att type journal -cls -element Source -end Item

  <Item type="journal">J Bacteriol</Item>

  -tag Item -att type journal -atr name Source -slf

  <Item type="journal" name="J Bacteriol" />

Exploration Control

Exploration commands control the order in which XML record contents are examined, by separately presenting each instance of the chosen subregion. This limits what subsequent commands "see" at any one time, and can allow related fields in an object to be kept together in the output.

Unlike the simpler DocumentSummary format, records retrieved as PubmedArticle XML:

  efetch -db pubmed -id 1413997 -format xml |

have authors with separate fields for last name and initials:

  <Author>
    <LastName>Mortimer</LastName>
    <Initials>RK</Initials>
  </Author>

Without being given any guidance about context, an ‑element command on initials and last names:

  xtract -pattern PubmedArticle -element Initials LastName

will explore the current record for each argument in turn, printing all initials followed by all last names:

  RK    CR    JS    Mortimer    Contopoulou    King

Inserting a ‑block command adds another exploration layer between ‑pattern and ‑element , which redirects data exploration to present the authors one at a time:

  xtract -pattern PubmedArticle -block Author -element Initials LastName

Each time through the loop, the ‑element command only sees the current author's values. This restores the correct association of initials and last names in the output:

  RK    Mortimer    CR    Contopoulou    JS    King

Grouping the two author subfields with a comma, and adjusting the ‑sep and ‑tab values:

  xtract -pattern PubmedArticle -block Author \
    -sep " " -tab ", " -element Initials,LastName

produces a more traditional formatting of author names:

  RK Mortimer, CR Contopoulou, JS King

Sequential Exploration

Multiple ‑block statements can be used in a single xtract to explore different areas of the XML. This limits element extraction to the desired subregions, and allows disambiguation of fields with identical names but with different parents. For example:

  efetch -db pubmed -id 6092233,4640931,4296474 -format xml |
  xtract -pattern PubmedArticle -element MedlineCitation/PMID \
    -block PubDate -sep " " -element Year,Month,MedlineDate \
    -block AuthorList -num Author -sep "/" -element LastName |
  sort-table -k 3,3n -k 4,4f

generates a table that allows easy parsing of author last names, and sorts the results by author count:

  4296474    1968 Apr        1    Friedmann
  4640931    1972 Dec        2    Tager/Steiner
  6092233    1984 Jul-Aug    3    Calderon/Contopoulou/Mortimer

Like ‑element arguments, the individual ‑block statements are executed sequentially, in order of appearance.

Note also that the PubDate object can exist either in a structured form:

  <PubDate>
    <Year>1968</Year>
    <Month>Apr</Month>
    <Day>25</Day>
  </PubDate>

(with the Day field frequently absent), or in a string form:

  <PubDate>
    <MedlineDate>1984 Jul-Aug</MedlineDate>
  </PubDate>

but would not contain a mixture of both types, so the directive:

  -sep " " -element Year,Month,MedlineDate

will only contribute a single column to the output.

Nested Exploration

Exploration command names (‑group, ‑block, and ‑subset) are assigned to a precedence hierarchy:

  -pattern > -group > -block > -subset > -element

and are combined in ranked order to control object iteration at progressively deeper levels in the XML data structure. Each command argument acts as a "nested for-loop" control variable, retaining information about the context, or state of exploration, at its level.

(Hypothetical) census data would need several nested loops to visit each unique address in context:

  -pattern State -group City -block Street -subset Number -element Resident

A nucleotide or protein sequence record can have multiple features. Each feature can have multiple qualifiers. And every qualifier has separate name and value nodes. Exploring this natural data hierarchy, with ‑pattern for the sequence record, ‑group for the feature, and ‑block for the qualifier:

  efetch -db nuccore -id NM_021486.4 -format gbc |
  xtract -pattern INSDSeq -element INSDSeq_accession-version \
    -group INSDFeature -deq "\n\t" -element INSDFeature_key \
      -block INSDQualifier -deq "\n\t\t" \
        -element INSDQualifier_name INSDQualifier_value

keeps qualifiers, such as gene and product, associated with their parent features, and keeps qualifier names and values together on the same line:

  NM_021486.4
      source
                organism       Mus musculus
                mol_type       mRNA
      gene
                gene           Bco1
      CDS
                gene           Bco1
                product        beta,beta-carotene 15,15'-dioxygenase isoform 1
                protein_id     NP_067461.2
                translation    MEIIFGQNKKEQLEPVQAKVTGSIPAWLQGTLLRNGPGM … 
                …

Saving Data in Variables

A value can be recorded in a variable and used wherever needed. Variables are created by a hyphen followed by a name consisting of a string of capital letters or digits (e.g., ‑KEY). Variable values are retrieved by placing an ampersand before the variable name (e.g., "&KEY") in an ‑element statement:

  efetch -db nuccore -id NM_021486.4 -format gbc |
  xtract -pattern INSDSeq -element INSDSeq_accession-version \
    -group INSDFeature -KEY INSDFeature_key \
      -block INSDQualifier -deq "\n\t" \
        -element "&KEY" INSDQualifier_name INSDQualifier_value

This prints the feature key on each line before the qualifier name and value, even though the feature key is now outside of the visibility scope (which is the current qualifier):

  NM_021486.4
      source    organism       Mus musculus
      source    mol_type       mRNA
      gene      gene           Bco1
      CDS       gene           Bco1
      CDS       product        beta,beta-carotene 15,15'-dioxygenase isoform 1
      CDS       protein_id     NP_067461.2
      CDS       translation    MEIIFGQNKKEQLEPVQAKVTGSIPAWLQGTLLRNGPGM … 
      …

Variables can be (re)initialized with an explicit literal value inside parentheses:

  -block Author -sep " " -tab "" -element "&COM" Initials,LastName -COM "(, )"

They can also be used as the first argument in a conditional statement:

  -CHR Chromosome -block GenomicInfoType -if "&CHR" -differs-from ChrLoc

A double-hyphen (e.g., ‑‑KYWDS) appends a value to the variable.

A variable can also save the modified data resulting from an ‑element variant operation. This can allow multiple sequential transitions within a single xtract command:

  -END -sum "Start,Length" -MID -avg "Start,&END"

All variables are reset when the next record is processed.

Conditional Execution

Conditional processing commands (‑if and ‑ unless) restrict object exploration by data content. They check to see if the named field is within the scope, and may be used in conjunction with string, numeric, or object constraints to require an additional match by value. Use ‑and and ‑or to build compound tests. Streaming input through ‑select removes records that do not satisfy the condition:

  esearch -db pubmed -query "Havran W [AUTH]" |
  efetch -format xml |
  xtract -pattern PubmedArticle -select Language -equals eng |
  xtract -pattern PubmedArticle -block Author -if LastName -is-not Havran \
    -sep ", " -tab "\n" -author LastName,Initials[1:1] |
  sort-uniq-count-rank

This limits the results to papers written in English and prints a table of the most frequent collaborators, using a range to keep only the first initial so that variants like "Berg, CM" and "Berg, C" are combined:

  35    Witherden, D
  15    Boismenu, R
  12    Jameson, J
  10    Allison, J
  10    Fitch, F
  …

Numeric constraints can compare the integer values of two fields. This can be used to find genes that are encoded on the minus strand of a particular chromosome:

  -if ChrLoc -equals X -and ChrStart -gt ChrStop

Object constraints will compare the string values of two named fields, and can look for internal inconsistencies between fields whose contents should (in most cases) be identical:

  -if Chromosome -differs-from ChrLoc

The ‑position command restricts presentation of objects by relative location or index number:

  -block Author -position last -sep ", " -element LastName,Initials

The ‑else command can supply alternative ‑element or ‑lbl instructions to be run if the condition is not satisfied:

  -if Strand -contains "-" -lbl "minus strand" -else -lbl "plus strand"

Parallel ‑if and ‑unless statements can be used to provide a more complex response to alternative conditions that include nested explorations.

Copying XML Objects

The ‑element "*" construct prints the entirety of the current XML container, including all XML tags. (A period or a percent sign writes ASN.1 or JSON, respectively. Tag names with leading, trailing, or internal underscores control item-specific formats, such as unquoted values or unnamed brackets.)

Automatic Format Conversion

Xtract can now detect and convert input data in JSON, text ASN.1, and GenBank/GenPept flatfile formats, into XML. Explicit transmute or shortcut commands are only needed to view the intermediate XML's field names or override the default conversion settings.

Text Modification

For custom editing, target and replacement strings are set with ‑reg and ‑exp, respectively. Regular expression text matching and substitution is then performed on an element with a ‑replace command:

  xtract  …  -reg "-" -exp "." -replace Phone

Value Substitution

External values can be loaded by reading a two-column precomputed file or ad hoc conversion table with ‑transform. Lookup is then requested by applying ‑translate to an element:

  xtract -transform accn-to-uid.txt  …  -translate Accession

  xtract -transform <( echo -e "Genomic\t1\nCoding\t2\nProtein\t3\n" )  …

Multi-Step Transformations

Although xtract provides ‑element derivatives to do simple data manipulation, more complex tasks may be broken up into a series of simpler transformations, or "processing chains".

BioSample document summaries:

  efetch -db biosample -id SAMN38051082 -format docsum |

store preferred qualifier names in a "harmonized_name" XML attribute:

  <Attribute harmonized_name="strain">BALB/c</Attribute>
  <Attribute harmonized_name="isolate">Mtb infected Spleen MZB-2</Attribute>
  <Attribute harmonized_name="geo_loc_name">Singapore</Attribute>

Piping the data to a first xtract command, and using the "@" sign to select the attribute:

  xtract -rec BioSampleInfo -pattern DocumentSummary \
    -wrp Accession -element Accession \
    -group Attribute -if @harmonized_name \
      -TAG -lower @harmonized_name -wrp "&TAG" -element Attribute |

generates an intermediate form, with XML tag names taken from the original XML attributes:

  <BioSampleInfo>
    <Accession>SAMN38051082</Accession>
    <strain>BALB/c</strain>
    <isolate>Mtb infected Spleen MZB-2</isolate>
    <geo_loc_name>Singapore</geo_loc_name>
    …

Desired fields can then be selected by name in a second xtract command:

  xtract -pattern BioSampleInfo -def "-" -first Accession \
    geo_loc_name strain isolate

Parsing XML Elements

GFF3 can be normalized with gff‑sort, and then converted to XML with tbl2xml, which gets field names from the remaining arguments (or, with ‑header, from the first row of the input data):

  tbl2xml -rec Rec SeqID Source Type Start End Score Strand Phase Attributes |

The Attributes element consists of multiple clauses that are delimited by semicolons:

  <Attributes>ID=gene-XXXX_016554;Name=XXXX_016554; … </Attributes>

Piping to xtract creates a container, and uses ‑with and ‑split to parse the Attribute field:

  xtract -pattern Rec -pkg Context -wrp Item -with ";" -split Attributes |

into individual "tag = value" elements within the new package:

   <Context>
    <Item>ID=gene-XXXX_016554</Item>
    <Item>Name=XXXX_016554</Item>
    …

A second xtract can now use element suffix and prefix removal, respectively, to isolate the tag and value. The tag is saved in a variable to become the name of a new XML object in which the value is stored:

  xtract -pattern Context -pkg Fields -block Item \
    -TAG "Item[|=]" -wrp "&TAG" -element "Item[=|]" |

The final XML can be queried by field names that were extracted from the original Attributes clauses:

  <Fields>
    <ID>gene-XXXX_016554</ID>
    <Name>XXXX_016554</Name>
    …

XML Namespaces

Namespace prefixes are followed by a colon, while a leading colon matches any prefix:

  nquire -url https://webservice.wikipathways.org getPathway -pwId WP455 |
  xtract -pattern "ns1:getPathwayResponse" -decode ":gpml" |

The embedded Graphical Pathway Markup Language object can then be processed:

  xtract -pattern Pathway -block Xref \
    -if @Database -equals "Entrez Gene" -tab "\n" -element @ID

Record Filtering

The filter‑records script can extract an arbitrary range of text records using ‑min and ‑max:

  filter-records -pattern "<PubmedArticle>" -min 13 -max 24

It also has arguments to ‑require or ‑exclude specific text content.

Sorting or Partitioning Sets of Records

To sort XML records by a particular field, pass the element name to xtract ‑sort‑fwd or ‑sort‑rev. Stable sorting, which preserves the relative order of records with identical sort key values, is provided by the ‑stable‑sort‑fwd and ‑stable‑sort‑rev commands:

  xtract -pattern PubmedArticle -sort-rev MedlineCitation/PMID

To partition XML records, give the number of records per output file to xtract ‑split‑by‑num:

  xtract -set PubmedArticleSet -pattern PubmedArticle \
    -split-by-num 100 -prefix subset -suffix xml

or use the maximum byte size for each output file as the argument to xtract ‑split‑by‑len:

  xtract -set PubmedArticleSet -pattern PubmedArticle \
    -split-by-len 1000000 -prefix subset -suffix xml

or make individual files by passing the unique identifier name to xtract ‑split‑by‑id:

  xtract -mixed -set pmc-articleset -pattern article \
    -split-by-id article-id -prefix PMC -suffix xml

Numbered output file names will be of the form "subset001.xml", "subset002.xml", etc.

To distribute text records into separate files, use split‑records with ‑by‑num or ‑by‑len.

Viewing an XML Hierarchy

Piping a PubmedArticle XML object to xtract ‑outline:

  esearch -db pubmed -query "Cozzarelli NR [AUTH] AND Casadaban MJ [AUTH]" |
  efetch -format xml | xtract -outline 3

will give an indented overview of the XML hierarchy, optionally limited to a specified object depth:

  PubmedArticle
    MedlineCitation
      PMID
      DateCompleted
      DateRevised
      Article
      MedlineJournalInfo
      ChemicalList
      CitationSubset
      MeshHeadingList
    PubmedData
    …

Using xtract ‑synopsis or ‑contour will show the full paths to all nodes or just the terminal (leaf) nodes, respectively. Piping those results to sort‑uniq‑count will produce a table of unique paths.

Use a caret ("^") as an element argument prefix to print its absolute depth in the XML hierarchy:

  xtract -pattern PubmedArticle -element "^PMID"

Code Nesting Comparison

Sketching with indented pseudo code can clarify relative nesting levels. The extraction command:

  xtract -pattern PubmedArticle \
    -block Author -element Initials,LastName \
    -block MeshHeading \
      -if QualifierName \
        -element DescriptorName \
        -subset QualifierName -element QualifierName

where the rank of the argument name controls the nesting depth, could be represented as a computer program in pseudo code by:

  for pat = each PubmedArticle {
    for blk = each pat.Author {
      print blk.Initials blk.LastName
    }
    for blk = each pat.MeSHTerm {
      if blk.Qual is present {
        print blk.MeshName
        for sbs = each blk.Qual {
          print sbs.QualName
        }
      }
    }
  }

where the brace indentation count controls the nesting depth.

Extra arguments are held in reserve to provide additional levels of organization, should the need arise in the future for processing complex, deeply-nested XML data. The full set of exploration commands below ‑pattern, in order of rank, are:

  -path
    -division
      -group
        -branch
          -block
            -section
              -subset
                -unit

Starting xtract exploration with ‑block, and expanding with ‑group and ‑subset, leaves additional level names that can be used wherever needed without having to redesign the entire command.

Author Exploration

What's in a name? That which we call an author, by any other name, may be a consortium, investigator, or editor:

  <PubmedArticle>
    <MedlineCitation>
      <PMID>99999999</PMID>
      <Article>
        <AuthorList>
          <Author>
            <LastName>Tinker</LastName>
          </Author>
          <Author>
            <LastName>Evers</LastName>
          </Author>
          <Author>
            <LastName>Chance</LastName>
          </Author>
          <Author>
            <CollectiveName>FlyBase Consortium</CollectiveName>
          </Author>
        </AuthorList>
      </Article>
      <InvestigatorList>
        <Investigator>
          <LastName>Alpher</LastName>
        </Investigator>
        <Investigator>
          <LastName>Bethe</LastName>
        </Investigator>
        <Investigator>
          <LastName>Gamow</LastName>
        </Investigator>
      </InvestigatorList>
    </MedlineCitation>
  </PubmedArticle>

Within the record, ‑element exploration on last name:

  xtract -pattern PubmedArticle -element LastName

prints each last name, but does not match the consortium:

  Tinker    Evers    Chance    Alpher    Bethe    Gamow

Limiting to the author list:

  xtract -pattern PubmedArticle -block AuthorList -element LastName

excludes the investigators:

  Tinker    Evers    Chance

Using ‑num on each type of object:

  xtract -pattern PubmedArticle -num Author Investigator LastName CollectiveName

displays the various object counts:

  4    3    6    1

Date Selection

Dates come in all shapes and sizes:

  <PubmedArticle>
    <MedlineCitation>
      <PMID>99999999</PMID>
      <DateCompleted>
        <Year>2011</Year>
      </DateCompleted>
      <DateRevised>
        <Year>2012</Year>
      </DateRevised>
      <Article>
        <Journal>
          <JournalIssue>
            <PubDate>
              <Year>2013</Year>
            </PubDate>
          </JournalIssue>
        </Journal>
        <ArticleDate>
          <Year>2014</Year>
        </ArticleDate>
      </Article>
    </MedlineCitation>
    <PubmedData>
      <History>
        <PubMedPubDate PubStatus="received">
          <Year>2015</Year>
        </PubMedPubDate>
        <PubMedPubDate PubStatus="accepted">
          <Year>2016</Year>
        </PubMedPubDate>
        <PubMedPubDate PubStatus="entrez">
          <Year>2017</Year>
        </PubMedPubDate>
        <PubMedPubDate PubStatus="pubmed">
          <Year>2018</Year>
        </PubMedPubDate>
        <PubMedPubDate PubStatus="medline">
          <Year>2019</Year>
        </PubMedPubDate>
      </History>
    </PubmedData>
  </PubmedArticle>

Within the record, ‑element exploration on the year:

  xtract -pattern PubmedArticle -element Year

finds and prints all nine instances:

  2011    2012    2013    2014    2015    2016    2017    2018    2019

Using ‑block to limit the scope:

  xtract -pattern PubmedArticle -block History -element Year

prints only the five years within the History object:

  2015    2016    2017    2018    2019

Inserting a conditional statement to limit element selection to a date with a specific attribute:

  xtract -pattern PubmedArticle -block History \
    -if @PubStatus -equals "pubmed" -element Year

surprisingly still prints all five years within History:

  2015    2016    2017    2018    2019

This is because the ‑if command uses the same exploration logic as ‑element, but is designed to declare success if it finds a match anywhere within the current scope. There is indeed a "pubmed" attribute within History, in one of its five PubMedPubDate child objects, so the test succeeds. Thus, ‑element is given free rein to do its own exploration in History, and prints all five years.

The solution is to explore the individual PubMedPubDate objects within History:

  xtract -pattern PubmedArticle -block History \
    -subset PubMedPubDate \
      -if @PubStatus -equals "pubmed" -element Year

This visits each PubMedPubDate separately, with the ‑if test matching only the indicated date type, thus returning only the desired year:

  2018

PMID Extraction

Because of the presence of a CommentsCorrections object:

  <PubmedArticle>
    <MedlineCitation>
      <PMID>99999999</PMID>
      <CommentsCorrectionsList>
        <CommentsCorrections RefType="ErratumFor">
          <PMID>88888888</PMID>
        </CommentsCorrections>
      </CommentsCorrectionsList>
    </MedlineCitation>
  </PubmedArticle>

attempting to print the record's PubMed Identifier:

  xtract -pattern PubmedArticle -element PMID

also returns the PMID of the comment:

  99999999    88888888

Using an exploration command cannot exclude the second instance, because it would need a parent node unique to the first element, and the chain of parents to the first PMID:

  PubmedArticle/MedlineCitation

is a subset of the chain of parents to the second PMID:

  PubmedArticle/MedlineCitation/CommentsCorrectionList/CommentsCorrections

Although ‑first PMID will work in this particular case, the more general solution is to limit by subpath with the parent / child construct:

  xtract -pattern PubmedArticle -element MedlineCitation/PMID

That would work even if the order of objects were reversed.

JSON Arrays

Human β‑globin information from a Scripps Research data integration project (4):

  nquire -get https://mygene.info/v3 gene 3043 | json2xml |

contains a multi-dimensional JavaScript Object Notation array of exon coordinates:

  "position": [
    [ 5225463, 5225726 ],
    [ 5226576, 5226799 ],
    [ 5226929, 5227071 ]
  ],
  "strand": -1,

Conversion to XML assigns distinct tag names to each level with the json2xml ‑nest element default:

  <position>
    <position_E>5225463</position_E>
    <position_E>5225726</position_E>
  </position>
  …

Heterogeneous Data

A query for the human green-sensitive opsin gene:

  nquire -get https://mygene.info/v3/gene/2652 | json2xml |

returns data containing a heterogeneous mixture of objects in the pathway section:

  <pathway>
    <reactome>
      <id>R-HSA-162582</id>
      <name>Signal Transduction</name>
    </reactome>
    …
    <wikipathways>
      <id>WP455</id>
      <name>GPCRs, Class A Rhodopsin-like</name>
    </wikipathways>
  </pathway>

The parent / star construct is used to visit the individual components of a parent object without needing to explicitly specify their names. For printing, the name of a child object is indicated by a question mark ("?"), while a tilde ("~") would return the anonymous object's value:

  xtract -pattern opt -group "pathway/*" \
    -pfc "\n" -element "?,name,id"

This displays a table of pathway database references:

  reactome        Signal Transduction                R-HSA-162582
  reactome        Disease                            R-HSA-1643685
  …
  reactome        Diseases of the neuronal system    R-HSA-9675143
  wikipathways    GPCRs, Class A Rhodopsin-like      WP455

Exhaustive Exploration

PubMed Central full-text records consist of recursive "sec" objects that contain section title and paragraph text elements. A record obtained in XML format:

  esearch -db pmc -query "Kitts PA [AUTH] AND type strains [TITL]" |
  efetch -format xml |

is first processed by xtract ‑mask, to remove complex formula and table mark-up instructions:

  xtract -mixed -pattern article -mask "table-wrap,alternatives,inline-formula,\
  disp-formula,pub-history,related-article,list-item,fig" -element "*" |

Extracting the title and abstract fields near the front of the document is straightforward:

  xtract -mixed -rec PMCData -pattern article \
    -division front \
      -wrp Title -prose title-group/article-title \
      -wrp Abstract -prose abstract/p \

The full text is obtained by exhaustive exploration using the parent / double star construct:

    -division "body/**" \

Multiple ‑if clauses use a question mark to test the child object's name and select only paragraph and title elements. A caret returns the nesting depth of the element:

      -if "?" -equals "p" -pkg Paragraph \
        -wrp Level -element "^" -wrp Text -prose p \
      -if "?" -equals "title" -pkg Section \
        -wrp Level -element "^" -wrp Title -prose sec/title

Like ‑else, the multiple ‑if convention can only be used for element extraction operations. Multiple explorations below the double star would require the use of individual ‑group "*" commands.

  <PMCData>
    <Title>Collection and curation of prokaryotic genome assemblies … </Title>
    <Abstract>The public sequence databases are entrusted with the … </Abstract>
    …
    <Section>
      <Level>5</Level>
      <Title>Assemblies not used as types</Title>
    </Section>
    <Paragraph>
      <Level>5</Level>
      <Text>NCBI evaluates all assemblies, including type assemblies … </Text>
    </Paragraph>
    …

GenBank Filtering

The most recent GenBank virus release file can be downloaded from NCBI servers:

  nquire -lst ftp.ncbi.nlm.nih.gov genbank |
  grep "^gbvrl" | grep ".seq.gz" | sort -V |
  tail -n 1 | skip-if-file-exists |
  nquire -dwn ftp.ncbi.nlm.nih.gov genbank

GenBank flatfile records can be selected by organism name or taxon identifier, or by presence or absence of an arbitrary text string, with filter-genbank:

  gunzip -c *.seq.gz | filter-genbank -taxid 11292 |

The full set of filter‑genbank arguments are ‑accession, ‑accessions, ‑taxid, ‑taxids, ‑organism, ‑truncate, ‑exclude, ‑require, ‑min, and ‑max.

While GenBank format can be read directly by xtract, explicit conversion to INSDSeq XML with gbf2xml may be up to three times faster for large sets of records:

  gbf2xml |

Feature location intervals and underlying sequences of individual coding regions are then obtained by:

  xtract -insd CDS gene product feat_location sub_sequence

Table Operations

Tab-delimited tables can be piped through filter‑columns for numeric and substring matching:

  filter-columns '10 <= $2 && $2 <= 30 && $5 ~ peptide'

and sent to print‑columns for flexible modification of text and numbers in the final output:

  print-columns '$1, $2+1, $3+$4-1, "\042" $5 "\042", tolower($6), total += $2'

Both scripts are front-ends to the awk data manipulation utility. In awk, a dollar sign followed by a digit indicates an input data column, not a Unix command-line argument position.

In order to prevent misinterpretation of awk dollar signs by the Unix shell interpreter, both scripts require their argument to be inside apostrophes instead of double quotation marks. The standard NF and NR awk variables can be used, as can YR (for year) and DT (for date in YYYY‑MM‑DD format).

Along with sort‑table, these scripts allow novice users to do useful things on tabular data without first having to know arcane setup details (such as how to specify the tab character as the column separator).

The align‑columns script has several arguments to reformat tables and make them easier to interpret.

The intersect‑uids, combine‑uids, and exclude‑uids scripts merge unique identifier files with Boolean operations (AND, OR, and NOT, respectively). Use compare‑uids to see UID differences.

Sequence Qualifiers

The NCBI data model for sequence records (5) is based on the central dogma of molecular biology. Sequences, including genomic DNA, messenger RNAs, and protein products, are "instantiated" with the actual sequence letters, and are assigned accession numbers for reference.

Features contain information about the biology of a region, including the transformations involved in gene expression. Qualifiers store specific details about a feature, such as the name of the gene, genetic code used for protein translation, or accession of the product sequence.

A gene feature indicates the location of a heritable region of nucleic acid that confers a measurable phenotype. An mRNA feature on genomic DNA represents the exonic and untranslated regions that remain after message transcription and intron splicing. A coding region (CDS) feature has a product reference to the translated protein sequence record:

Since messenger RNA sequences are not always submitted with a genomic region, CDS features (which model the travel of ribosomes on transcript molecules) are traditionally annotated on the genomic sequence, with locations that encode the exonic intervals.

A qualifier can be dynamically generated from underlying data for the convenience of the user. Thus, the sequence of a mature peptide may be extracted from the mat_peptide feature's location on the precursor protein and displayed in a /peptide qualifier, even if a mature peptide is not instantiated.

As a convenience for exploring sequence records, the xtract ‑insd helper function generates the appropriate nested extraction commands from feature and qualifier names on the command line. (The computed qualifiers feat_location, feat_intervals, and sub_sequence are also supported.)

Snail Venom Peptide Sequences

A search for cone snail venom mature peptides:

  esearch -db pubmed -query "conotoxin" |
  elink -target protein |
  efilter -query "mat_peptide [FKEY]" |
  efetch -format gpc |
  xtract -insd complete mat_peptide "%peptide" product mol_wt peptide |

uses the xtract ‑insd function to print the accession number, mature peptide length, product name, calculated molecular weight, and amino acid sequence for a sample of neurotoxic peptides:

  AAN78128.1    12    alpha-conotoxin ImI    1357    GCCSDPRCAWRC
  ADB65789.1    20    conotoxin Cal 16       2134    LEMQGCVCNANAKFCCGEGR
  ADB65788.1    20    conotoxin Cal 16       2134    LEMQGCVCNANAKFCCGEGR
  AGO59814.1    32    del13b conotoxin       3462    DCPTSCPTTCANGWECCKGYPCVRQHCSGCNH
  AAO33169.1    16    alpha-conotoxin GIC    1615    GCCSHPACAGNNQHIC
  AAN78279.1    21    conotoxin Vx-II        2252    WIDPSHYCCCGGGCTDDCVNC
  AAF23167.1    31    BeTX toxin             3433    CRAEGTYCENDSQCCLNECCWGGCGHPCRHP
  ABW16858.1    15    marmophin              1915    DWEYHAHPKPNSFWT
  …

Piping the results to a series of Unix commands and EDirect scripts:

  grep -i conotoxin |
  filter-columns '10 <= $2 && $2 <= 30' |
  sort-table -u -k 5 |
  sort-table -k 2,2n |
  align-columns -

filters by product name, limits the results to a specified range of peptide lengths, removes redundant sequences, sorts the table by peptide length, and aligns the columns for cleaner printing:

  AAN78127.1    12    alpha-conotoxin ImII     1515    ACCSDRRCRWRC
  AAN78128.1    12    alpha-conotoxin ImI      1357    GCCSDPRCAWRC
  ADB43130.1    15    conotoxin Cal 1a         1750    KCCKRHHGCHPCGRK
  ADB43131.1    15    conotoxin Cal 1b         1708    LCCKRHHGCHPCGRT
  AAO33169.1    16    alpha-conotoxin GIC      1615    GCCSHPACAGNNQHIC
  ADB43128.1    16    conotoxin Cal 5.1        1829    DPAPCCQHPIETCCRR
  AAD31913.1    18    alpha A conotoxin Tx2    2010    PECCSHPACNVDHPEICR
  ADB43129.1    18    conotoxin Cal 5.2        2008    MIQRSQCCAVKKNCCHVG
  ADB65789.1    20    conotoxin Cal 16         2134    LEMQGCVCNANAKFCCGEGR
  ADD97803.1    20    conotoxin Cal 1.2        2206    AGCCPTIMYKTGACRTNRCR
  AAD31912.1    21    alpha A conotoxin Tx1    2304    PECCSDPRCNSSHPELCGGRR
  AAN78279.1    21    conotoxin Vx-II          2252    WIDPSHYCCCGGGCTDDCVNC
  ADB43125.1    22    conotoxin Cal 14.2       2157    GCPADCPNTCDSSNKCSPGFPG
  ADD97802.1    23    conotoxin Cal 6.4        2514    GCWLCLGPNACCRGSVCHDYCPR
  …

The xtract ‑insdx variant:

  esearch -db protein -query "conotoxin" |
  efilter -query "mat_peptide [FKEY]" |
  efetch -format gpc |
  xtract -insdx complete mat_peptide "%peptide" product mol_wt peptide |
  xtract -pattern Rec -select product -contains conotoxin |
  xtract -pattern Rec -sort mol_wt

saves the output table directly as XML, with the XML tag names taken from the original qualifier names:

  … 
  <Rec>
    <accession>AAO33169.1</accession>
    <feature_key>mat_peptide</feature_key>
    <peptide_Len>16</peptide_Len>
    <product>alpha-conotoxin GIC</product>
    <mol_wt>1615</mol_wt>
    <peptide>GCCSHPACAGNNQHIC</peptide>
  </Rec>
  <Rec>
    <accession>AIC77099.1</accession>
    <feature_key>mat_peptide</feature_key>
    <peptide_Len>16</peptide_Len>
    <product>conotoxin Im1.2</product>
    <mol_wt>1669</mol_wt>
    <peptide>GCCSHPACNVNNPHIC</peptide>
  </Rec>
  …

Qualifier names with prefix shortcuts "#" and "%" are modified to use "_Num" and "_Len" suffixes, respectively.

Missing Qualifiers

For records where a particular qualifier is missing:

  esearch -db protein -query "RAG1 [GENE] AND Mus musculus [ORGN]" |
  efetch -format gpc |
  xtract -insd source organism strain |
  sort-table -u -k 2,3

a dash is inserted as a placeholder:

  P15919.2       Mus musculus               -
  AAO61776.1     Mus musculus               129/Sv
  NP_033045.2    Mus musculus               C57BL/6
  EDL27655.1     Mus musculus               mixed
  BAD69530.1     Mus musculus castaneus     -
  BAD69531.1     Mus musculus domesticus    BALB/c
  BAD69532.1     Mus musculus molossinus    MOA

Recursive Taxonomy Data

Certain XML objects returned by efetch are recursively defined, including Taxon in ‑db taxonomy and Gene-commentary in ‑db gene. Thus, they can contain nested objects with the same XML tag.

Retrieving a set of taxonomy records:

  efetch -db taxonomy -id 9615,9606 -format xml

produces XML with nested Taxon objects (marked below with line references) for each rank in the taxonomic lineage:

    <TaxaSet>
1     <Taxon>
        <TaxId>9606</TaxId>
        <ScientificName>Homo sapiens</ScientificName>
        …
        <LineageEx>
2         <Taxon>
            <TaxId>131567</TaxId>
            <ScientificName>cellular organisms</ScientificName>
            <Rank>no rank</Rank>
3         </Taxon>
4         <Taxon>
            <TaxId>2759</TaxId>
            <ScientificName>Eukaryota</ScientificName>
            <Rank>superkingdom</Rank>
5         </Taxon>
          …
        </LineageEx>
        …
6     </Taxon>
7     <Taxon>
        <TaxId>9615</TaxId>
        <ScientificName>Canis lupus familiaris</ScientificName>
        …
8     </Taxon>
    </TaxaSet>

Xtract tracks XML object nesting to determine that the <Taxon> start tag on line 1 is closed by the </Taxon> stop tag on line 6, and not by the first </Taxon> encountered on line 3.

To accommodate recursively-defined data, entry by ‑element to an internal object is blocked when its name matches the current exploration container.

The star / child construct bypasses the search constraint:

  efetch -db taxonomy -id 9606,7227 -format xml |
  xtract -pattern Taxon -block "*/Taxon" \
    -tab "\n" -element TaxId,ScientificName

to allow controlled descent to the next level:

  131567    cellular organisms
  2759      Eukaryota

Using double star / child recursively visits every object regardless of depth, and can flatten a complex structure into a linear set of elements in a single step:

  efetch -db taxonomy -id 9606 -format xml |
  xtract -pattern Taxon \
    -first TaxId -tab "\n" -element ScientificName \
    -block "**/Taxon" -if Rank -is-not "no rank" -and Rank -excludes "root" \
      -tab "\n" -element Rank,ScientificName

This prints all of the individual internal lineage nodes:

  9606          Homo sapiens
  domain        Eukaryota
  clade         Opisthokonta
  kingdom       Metazoa
  clade         Eumetazoa
  clade         Bilateria
  clade         Deuterostomia
  phylum        Chordata
  subphylum     Craniata
  clade         Vertebrata
  clade         Gnathostomata
  clade         Teleostomi
  clade         Euteleostomi
  superclass    Sarcopterygii
  clade         Dipnotetrapodomorpha
  clade         Tetrapoda
  clade         Amniota
  class         Mammalia
  …

Genes in a Region

Records for protein-coding genes on the human X chromosome are retrieved by running:

  esearch -db gene -query "Homo sapiens [ORGN] AND X [CHR]" |
  efilter -status alive -type coding | efetch -format docsum |

Gene names and chromosomal positions are extracted by piping the records to:

  xtract -pattern DocumentSummary -NAME Name -DESC Description \
    -block GenomicInfoType -if ChrLoc -equals X \
      -min ChrStart,ChrStop -element "&NAME" "&DESC" |

Exploring each GenomicInfoType is needed because of pseudoautosomal regions at the ends of the X and Y chromosomes:

  …
  <GenomicInfo>
    <GenomicInfoType>
      <ChrLoc>X</ChrLoc>
      <ChrAccVer>NC_000023.11</ChrAccVer>
      <ChrStart>155997630</ChrStart>
      <ChrStop>156013016</ChrStop>
      <ExonCount>14</ExonCount>
    </GenomicInfoType>
    <GenomicInfoType>
      <ChrLoc>Y</ChrLoc>
      <ChrAccVer>NC_000024.10</ChrAccVer>
      <ChrStart>57184150</ChrStart>
      <ChrStop>57199536</ChrStop>
      <ExonCount>14</ExonCount>
    </GenomicInfoType>
  </GenomicInfo>
  …

Without limiting to chromosome X, the copy of IL9R near the "q" telomere of chromosome Y would be erroneously placed with genes that are near the X chromosome centromere, shown here in between SPIN2A and ZXDB:

  …
  57121860    FAAH2     fatty acid amide hydrolase 2
  57133042    SPIN2A    spindlin family member 2A
  57184150    IL9R      interleukin 9 receptor
  57592010    ZXDB      zinc finger X-linked duplicated B
  …

The ‑if statement eliminates coordinates from pseudoautosomal gene copies present on Y chromosome telomeres. Results can now be sorted by position, and then filtered and partitioned:

  sort-table -k 1,1n | cut -f 2- |
  grep -v pseudogene | grep -v uncharacterized | grep -v hypothetical |
  between-two-genes AMER1 FAAH2

to produce an ordered table of known genes located between two markers flanking the centromere:

  FAAH2      fatty acid amide hydrolase 2
  SPIN2A     spindlin family member 2A
  ZXDB       zinc finger X-linked duplicated B
  NLRP2B     NLR family pyrin domain containing 2B
  ZXDA       zinc finger X-linked duplicated A
  SPIN4      spindlin family member 4
  ARHGEF9    Cdc42 guanine nucleotide exchange factor 9
  AMER1      APC membrane recruitment protein 1

SNP-Modified Product Pairs

Single nucleotide polymorphisms can represent different substitutions at the same position, but variation records do not explicitly match a specific CDS modification to its altered protein product:

  efetch -db snp -id 11549407 -format docsum |

The hgvs2spdi script converts 1‑based HGVS (6) data ("NM_000518.5:c.118C>T") into 0‑based SPDI (7) format ("NM_000518.5:167:C:T"). For SNPs on cDNA transcripts the position is CDS-relative, and the script retrieves the GenBank record in order to calculate the absolute sequence offset:

  snp2hgvs | hgvs2spdi | spdi2tbl | tbl2prod

The tbl2prod step translates the coding region locations (after nucleotide substitution), and sorts them with protein sequences (after residue replacement) to produce adjacent matching CDS/protein pairs:

  rs11549407    NM_000518.5:167:C:T    MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWT*R …
  rs11549407    NP_000509.1:39:Q:*     MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWT*R …
  rs11549407    NM_000518.5:167:C:G    MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTER …
  rs11549407    NP_000509.1:39:Q:E     MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTER …
  rs11549407    NM_000518.5:167:C:A    MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTKR …
  rs11549407    NP_000509.1:39:Q:K     MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTKR …
  …

Reverse Complementation

A GenBank sequence can be converted to FASTA, reverse-complemented, and printed as FASTA with:

  efetch -db nuccore -id U00096 -format gb |
  gbf2fsa | transmute -revcomp | transmute -fasta -width 50

Sequence Editing

The pBR322 cloning vector is a circular plasmid with unique restriction sites in two antibiotic resistance genes. The transmute ‑replace function introduces a second BamHI restriction enzyme recognition site (based on a site-directed mutagenesis experiment) by modifying two bases in the rop gene:

  efetch -db nuccore -id J01749 -format fasta |
  transmute -replace -offset 1907 -delete GG -insert TC |
  …

Pattern Searching

The modified sequence from above is then passed to transmute ‑search, which takes a list of sequence patterns (with optional labels) and uses a finite-state algorithm to simultaneously search for all patterns:

  …
  transmute -search -circular GGATCC:BamHI GAATTC:EcoRI CTGCAG:PstI |
  align-columns -g 4 -a rl

The (0‑based) starting positions and restriction enzyme names for each match are printed in a table:

   374    BamHI
  1904    BamHI
  3606    PstI
  4358    EcoRI

The disambiguate‑nucleotides and systematic‑mutations scripts can generate all possible single-base substitutions in a pattern for use in a relaxed-stringency search.

Six-Frame Translation

Protein translation uses a finite-state machine to slide a triplet-codon window along the sequence. State values are offsets into an amino acid lookup table. The next-state and reverse-complement transition tables, and an amino acid table for every genetic code, are all precomputed. A nucleotide sequence can then be translated in all six possible reading frames by an extremely fast loop of indexed lookups:

  efetch -db nuccore -id U54469.1 -format fasta |
  transmute -cds2prot -gcode 1 -all |

This produces regions of translated protein separated by asterisks representing stop codons:

  >U54469.1-1+
  RLLGFYNISQ*QAFPELPCSTIDSCLWPPKSQT*LKN*IIRIIIKPSNLR …
  >U54469.1-2+
  GCLGFITSVSDRHFQSCPVQQSIAAFGHQNPKLN*RIK*FE**LSPVTYA …
  …

The FASTAs are then converted to XML, and xtract ‑pept splits the translated frames at each stop codon (asterisk), sequence gap (hyphen), or ambiguous translation ("X") character:

  fsa2xml |
  xtract -rec FASTA -pattern FASTA \
    -group "FASTA/*" -element "*" \
    -group FASTA -wrp FRAG -pept Seq

That appends individual peptide fragments to the original fields and creates 6 new XML records:

  …
  <FRAG>rllgfynisq</FRAG>
  <FRAG>qafpelpcstidsclwppksqt</FRAG>
  <FRAG>lkn</FRAG>
  …

Feature Locations

A table of coding region locations and gene names can be saved directly as XML with xtract ‑insdx:

  efetch -db nuccore -id NC_000011 -format gb -style master |
  xtract -insdx CDS gene feat_location > cds_loc.xml

The human β‑globin coding region location is then retrieved and stored in a Unix shell variable with:

  loc=$( xtract -input cds_loc.xml -pattern Rec \
           -if gene -equals HBB -element feat_location )

Location intervals are shown in biological order, where start is greater than stop on the minus strand:

  5227021..5226930,5226799..5226577,5225726..5225598

Sequence Transformations

The cascading effects of a genomic SNP can be reproduced with transmute functions: ‑replace applies the substitution, ‑extract uses the location intervals from above to isolate the altered coding sequence, and ‑cds2prot translates the modified CDS into protein with the designated genetic code:

  efetch -db nuccore -id NC_000011 -format fasta |
  transmute -replace -offset 5226773 -delete G -insert A |
  transmute -extract -1-based "$loc" |
  transmute -cds2prot -gcode 1 -frame 0 -every -trim

The genomic G to A transition on human chromosome 11 corresponds to the C to T substitution on the minus-strand-encoded β‑globin mRNA in the earlier SNP example:

  MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWT*R …

Gene Positions

An understanding of sequence coordinate conventions is necessary in order to use gene positions to retrieve the corresponding chromosome subregion with efetch or with the UCSC browser.

Sequence records displayed in GenBank or GenPept formats use a "one‑based" coordinate system, with sequence position numbers starting at "1":

    1 catgccattc gttgagttgg aaacaaactt gccggctagc cgcatacccg cggggctgga
   61 gaaccggctg tgtgcggcca cagccaccat cctggacaaa cccgaagacg tgagtgaggg
  121 tcggcgagaa cttgtgggct agggtcggac ctcccaatga cccgttccca tccccaggga
  181 ccccactccc ctggtaacct ctgaccttcc gtgtcctatc ctcccttcct agatcccttc
  …

Under this convention, positions refer to the sequence letters themselves:

  C   A   T   G   C   C   A   T   T   C
  1   2   3   4   5   6   7   8   9  10

and the position of the last base or residue is equal to the length of the sequence. The ATG initiation codon above is at positions 2 through 4, inclusive.

For computer programs, however, using "zero‑based" coordinates can simplify the arithmetic used for calculations on sequence positions. The ATG codon in the 0‑based representation is at positions 1 through 3. (The UCSC browser uses a hybrid, half-open representation, where the start position is 0‑based and the stop position is 1‑based.)

Software at NCBI will typically convert positions to 0‑based coordinates upon input, perform whatever calculations are desired, and then convert the results to a 1‑based representation for display. These transformations are done by simply subtracting 1 from the 1‑based value or adding 1 to the 0‑based value.

Coordinate Conversions

Retrieving the docsum for a particular gene:

  esearch -db gene -query "BRCA2 [GENE] AND human [ORGN]" |
  efetch -format docsum |

returns the chromosomal position of that gene in "0‑based" coordinates:

  …
  <GenomicInfoType>
    <ChrLoc>13</ChrLoc>
    <ChrAccVer>NC_000013.11</ChrAccVer>
    <ChrStart>32315479</ChrStart>
    <ChrStop>32399671</ChrStop>
    <ExonCount>27</ExonCount>
  </GenomicInfoType>
  …

Piping the document summary to an xtract command using ‑element:

  xtract -pattern GenomicInfoType -element ChrAccVer ChrStart ChrStop

obtains the accession and 0‑based coordinate values:

  NC_000013.11    32315479    32399671

Efetch has ‑seq_start and ‑seq_stop arguments to retrieve a gene segment, but these expect the sequence subrange to be in 1‑based coordinates.

To address this problem, two additional efetch arguments, ‑chr_start and ‑chr_stop, were created to allow direct use of the 0‑based coordinates:

  efetch -db nuccore -format gb -id NC_000013.11 \
    -chr_start 32315479 -chr_stop 32399671

Xtract numeric extraction commands can also assist with coordinate conversion. Using xtract ‑inc:

  xtract -pattern GenomicInfoType -element ChrAccVer -inc ChrStart ChrStop

obtains the accession and 0‑based coordinates, then increments the positions to produce 1‑based values:

  NC_000013.11    32315480    32399672

EDirect knows the policies for sequence positions in all relevant Entrez databases (e.g., gene, snp, dbvar), and provides additional shortcuts for converting these to other conventions. For example:

  xtract -pattern GenomicInfoType -element ChrAccVer -1-based ChrStart ChrStop

understands that gene docsum ChrStart and ChrStop fields are 0‑based, sees that the desired output is 1‑based, and translates the command to convert coordinates internally using the ‑inc logic. Similarly:

  -element ChrAccVer -ucsc-based ChrStart ChrStop

leaves the 0‑based start value unchanged but increments the original stop value to produce the half-open form that can be passed to the UCSC browser:

  NC_000013.11    32315479    32399672

Gene Sequence

Genes encoded on the minus strand of a sequence:

  esearch -db gene -query "DDT [GENE] AND mouse [ORGN]" |
  efetch -format docsum |
  xtract -pattern GenomicInfoType -element ChrAccVer ChrStart ChrStop |

have coordinates ("0‑based" in docsums) where the start position is greater than the stop:

  NC_000076.6    75773373    75771232

These values can be read into Unix variables by a "while" loop:

  while IFS=$'\t' read acn str stp
  do
    efetch -db nuccore -format gb \
      -id "$acn" -chr_start "$str" -chr_stop "$stp"
  done

The variables can then be used to obtain the reverse-complemented subregion in GenBank format:

  LOCUS       NC_000076               2142 bp    DNA     linear   CON 08-AUG-2019
  DEFINITION  Mus musculus strain C57BL/6J chromosome 10, GRCm38.p6 C57BL/6J.
  ACCESSION   NC_000076 REGION: complement(75771233..75773374)
  …
       gene            1..2142
                       /gene="Ddt"
       mRNA            join(1..159,462..637,1869..2142)
                       /gene="Ddt"
                       /product="D-dopachrome tautomerase"
                       /transcript_id="NM_010027.1"
       CDS             join(52..159,462..637,1869..1941)
                       /gene="Ddt"
                       /codon_start=1
                       /product="D-dopachrome decarboxylase"
                       /protein_id="NP_034157.1"
                       /translation="MPFVELETNLPASRIPAGLENRLCAATATILDKPEDRVSVTIRP
                       GMTLLMNKSTEPCAHLLVSSIGVVGTAEQNRTHSASFFKFLTEELSLDQDRIVIRFFP
                       …

The reverse complement of a plus-strand sequence range can be selected with efetch ‑revcomp.

Local Record Cache

EDirect can now preload 40 million live PubMed records onto an inexpensive external 1 TB solid-state drive, storing them in indexed archive files. For example, PMID 2539356 would be entry 9356 in:

  /pubmed/Archive/00/02/000253.archive

Each archive contains up to 10,000 individually-compressed source data records in a specific range of identifiers. Preceding those concatenated, independent blocks is a binary array of the cumulative block lengths. This allows rapid random access to any or all of the compressed records in the archive.

The local archive is a completely self-contained turnkey product, with no need to download, configure, and maintain complicated third-party database software.

Set an environment variable in your configuration file(s) to reference a section of your external drive:

  export EDIRECT_LOCAL_ARCHIVE=/Volumes/external_drive_name/

Then run archive‑pubmed to download the PubMed release files and save each record on the drive. The initial download may take several hours, depending on your network connection, with initial archiving taking another few hours. Subsequent updates are incremental, and should finish in minutes.

Fetching a set of 143,173 PubMed records from NCBI's network service takes a bit over 30 minutes:

  esearch -db pubmed -query "PNAS [JOUR]" -pub abstract | efetch -format xml

Searching the local indices (see next section) with xsearch, and then using xfetch to retrieve and decompress the XML records from the local archive, runs in just under 6 seconds:

  xsearch -db pubmed -query "PNAS [JOUR] AND has abstract [PROP]" | xfetch

This is more than 300 times faster than using the EUtils servers, and avoids network reliability issues.

Piping to xfetch ‑stream returns the original compressed records, for a roughly four-fold size reduction of data to send over a network. This would be followed by decompression on the client with gunzip ‑c.

Alternatively, using xfetch ‑turbo precedes each record with an object containing its size in bytes:

  <NEXT_RECORD_SIZE>4027</NEXT_RECORD_SIZE>

Piping these enhanced records to xtract ‑turbo approximately doubles the speed of the (rate-limiting) record partitioning step, which otherwise uses the Boyer-Moore-Horspool fast string search algorithm.

Running xfetch ‑db pubmed ‑all retrieves the entire database by piping each ".archive" file through:

  dd ibs=8000 skip=10 2> /dev/null | gunzip -c 2> /dev/null

Local Search Index

A similar strategy was used to create a local information retrieval system suitable for large data mining queries. Run archive‑pubmed ‑index to populate retrieval index files from records stored in the local archive. The initial indexing will also take a few hours. To build a small subset for evaluation, run:

  archive-pubmed -index -custom "J Exp Med [JOUR] AND 2000:2026 [PDAT]"

For PubMed titles and primary abstracts, the indexing process deletes hyphens after specific prefixes, removes accents and diacritical marks, splits words at punctuation characters, corrects encoding artifacts, and spells out Greek letters for easier searching on scientific terms. It then prepares inverted indices with term positions, and uses them to build distributed term lists and postings files.

For example, the term list that includes "cancer" in the title or abstract would be located at:

  /pubmed/Postings/TIAB/c/a/n/c/canc.TIAB.trm

A query on cancer thus only needs to load a very small subset of the total index. The software supports expression evaluation, wildcard truncation, phrase queries, proximity searches, and partial matches.

The xinfo, xsearch, xlink, and xfilter scripts provide access to the local search system.

Names of indexed fields, all terms for a given field, and terms plus record counts, are shown by:

  xinfo -fields

  xinfo -terms TITL

  xinfo -totals PROP

Terms are truncated with a trailing asterisk, and can be expanded to show individual postings counts:

  xinfo -count "catabolite repress*"

  xinfo -counts "catabolite repress*"

Query evaluation includes Boolean operations and parenthetical expressions:

  xsearch -query "(literacy AND numeracy) NOT (adolescent OR child)"

Adjacent words in title or abstract fields are treated as a contiguous phrase:

  xsearch -query "selective serotonin reuptake inhibitor [TITL]"

Each plus sign will replace a single word inside a phrase, and runs of tildes indicate the maximum distance between sequential phrases:

  xsearch -query "vitamin c + + common cold"

  xsearch -query "vitamin c ~ ~ common cold"

Ranked partial term matching is available in any field with ‑match:

  xsearch -match "tn3 transposition immunity [PAIR]" | just-top-hits 1

An exact substring match, without special processing of Boolean operators or indexed field names, can be obtained with ‑title (on the article title) or ‑exact (on the title or abstract):

  xsearch -title "Genetic Control of Biochemical Reactions in Neurospora."

MeSH identifier code, MeSH hierarchy key, and year of publication are also indexed, and MESH field queries are supported by internally mapping to the appropriate CODE or TREE entries:

  xsearch -db pubmed -query "C14.907.617.812* [TREE] AND 2015:2019 [YEAR]"

PMIDs processed through an external source can be reintroduced to a local query pipeline with xfilter:

  | xfilter -db pubmed -query "Monoamine Oxidase [MESH] AND Deficiency [SUBH]" |

Data Analysis

All query commands return a structured message containing the database name and a list of UIDs, which can be piped directly to xfetch to retrieve the uncompressed records. For example:

  xsearch -db pubmed -query "selective serotonin ~ ~ ~ reuptake inhibit*" |
  xfetch |
  xtract -pattern PubmedArticle -num AuthorList/Author |
  sort-uniq-count -n |
  reorder-columns 2 1 |
  align-columns -g 4 -a lr

performs a proximity search with dynamic wildcard expansion (matching phrases like "selective serotonin and norepinephrine reuptake inhibitors") and fetches 16,088 PubMed records from the local archive. It then prints a frequency table of the number of papers per number of authors (a consortium is treated as a single author):

  0      56
  1    1494
  2    2144
  3    2228
  …

Data Visualization

The cumulative size of PubMed can be calculated with a running sum of the annual record counts:

  xinfo -db pubmed -totals YEAR |
  print-columns '$2, $1, total += $1' |
  print-columns '$1, log($2)/log(10), log($3)/log(10)' |
  filter-columns '$1 >= 1800 && $1 < YR' |
  xy-plot annual-and-cumulative.png

Exponential growth over time will appear as a roughly linear curve on a semi-logarithmic graph:

The sharp jump after World War II was caused by several factors, including the release of declassified papers, a policy of expanding biomedical research in postwar America, and the introduction of computers that could keep up with the indexing of articles from a broader range of subjects.

Natural Language Processing

NLM's Biomedical Text Mining Group performs computational analysis to extract chemical, disease, and gene references from article contents (8). NLM indexing of PubMed records assigns Gene Reference into Function (GeneRIF) mappings (9).

Running archive‑nlmnlp ‑index periodically (monthly) will automatically refresh any out-of-date support files and then index the connections in CHEM, DISZ, GENE, GRIF, GSYN, and PREF fields:

  xinfo -terms DISZ | grep -i Raynaud

  xinfo -counts "Raynaud* [DISZ]"

  xinfo -query "Raynaud Disease [DISZ]"

Following Citation Links

Running archive‑nihocc ‑index will download the latest NIH Open Citation Collection monthly release and build CITED and CITES indices, the local equivalent of elink ‑cited and ‑cites commands.

Citation links are retrieved by piping one or more PMIDs to xlink ‑target:

  xsearch -db pubmed -query "Haselkorn R* [AUTH]" |
  xlink -target CITED |

This returns PMIDs for 8,356 articles that cite the original 225 papers. The results are then restricted to a range of recent years, and those records are fetched. The xtract ‑histogram shortcut builds a journal frequency table from the subsequent articles:

  xfilter -query "2020:2025 [YEAR]" | xfetch |
  xtract -pattern PubmedArticle -histogram Journal/ISOAbbreviation |
  sort-table -nr | head -n 10

The archive‑pids ‑index command reads the PubMed local archive's incremental inverted index files, and builds a PMCID index that allows xlink to return PubMed Central identifiers from PMIDs:

  xlink -db pubmed -id 12372140 -target PMCID | xfetch

The xlink ‑ranked flag produces the same ranked output format as xsearch ‑match.

Additional Experimental Archives

Running archive‑pmc ‑index downloads PMC release files, and collects primary author names, citation details, section titles, and full-text paragraphs. It then converts them to the more tractable PMCInfo object, and builds an archive from those derived records.

Similarly, archive‑taxonomy ‑index archives novel records assembled from NCBI taxonomy data tables retrieved from the FTP site.

A new database is ready for use once its autonomous archiving script has finished all downloading, conversion, validation, caching, indexing, inversion, collection, merging, and posting steps.

Archive Scripts

The first step is to create a one-line archive script that sends command-line arguments to xbuild. A sample database indexed by peptide sequence fragments is built with archive-peptide, which runs:

  xbuild -db peptide -project peptide "$@"

A primary database has the same values for the ‑db and ‑project arguments. It may import structured records, optionally caching simpler derived records, or it may assemble records from tabular data.

Secondary projects add new indexed fields or links to primary databases. The archive‑nihocc script:

  xbuild -db pubmed -project nihocc "$@"

adds reciprocal citation reference links to the pubmed database, but does not create its own records.

Project Directories

A new database or project next requires adding a project directory within the edirect / extern folder. The edirect / extern / peptide subfolder contains several helper scripts. By convention, their names are composed of a formal step in the build process, a hyphen, and the project name:

  download-peptide
  populate-peptide
  index-peptide

A single configuration file named for the project is also present:

  peptide.ini

Secondary project files go in an edirect / extern / {database}-{project} subfolder.

Configuration Files

Project-specific configuration files guide the build process. The peptide.ini file provides sections for archiving and posting, as well as information used by xfetch to retrieve the derived records:

  [info]
  db=peptide
  project=peptide

  [archive]
  name=PeptInfo
  index=PeptInfo/UID

  [fetch]
  name=PeptInfo
  set=PeptInfoSet

In contrast, the nihocc.ini configuration file adds link flags in the merge and posting sections:

  [info]
  db=pubmed
  project=nihocc

  [merge]
  link=true

  [posting]
  fields="CITED CITES"
  link=true

while pubmed.ini has a links section with destination databases for xlink ‑target arguments:

  …
  [links]
  CITED=pubmed
  CITES=pubmed
  PMCID=pmc

"Download" Helper

The actual helper files needed for any given project depends on the source of the records, and on any accessory data files that must be retrieved separately. For each archive step, the helper file is "sourced" by an intermediate script, which executes it in the context of properly initialized variables.

An abridged version of the download‑peptide file is shown below:

  FilterByDivision() {

    dir="$1"

    if [ "$dir" = "archaea" ]
    then
      grep "wp_"
    elif [ "$dir" = "bacteria" ] || [ "$dir" = "plasmid" ]
    then
      grep "."
    else
      grep -v "wp_"
    fi
  }

  if [ -d "${sourceBase}" ]
  then
    cd "${sourceBase}"

    if [ -n "$custom" ]
    then
      nquire -lst ftp.ncbi.nih.gov/refseq/release "$custom" < /dev/null |
      grep gpff | FilterByDivision "$custom" | sort -V | skip-if-file-exists |
      while read fl
      do
        echo "$fl" | nquire -asp ftp.ncbi.nih.gov/refseq/release "$custom"
      done
    fi
  fi

For this example, a RefSeq division folder is passed by archive‑peptide using the ‑custom argument:

  archive-peptide -custom "viral"

In the real download-peptide file, the custom variable contains one or more division abbreviations:

  archive-peptide -custom "ARC FUN PRO VRL"

Individual build steps can be executed with ‑step, or bypassed with ‑skip:

  archive-peptide -step IDX
  archive-peptide -index -skip "SET CHK DAT DWN GEN POP RES IDX COL MRG PST CLR"

For the full download‑peptide helper file, running just the download step without a ‑custom argument prints a table displaying the division abbreviation, actual subfolder name, number of files, and total number of bytes, for each taxonomic division (formatted by align‑columns ‑h 2 ‑g 4 ‑a llrm):

  ARC    archaea                   9      1,956,632,000
  BCT    bacteria                825    181,043,837,778
  FUN    fungi                    36      4,029,672,534
  INV    invertebrate             93      6,206,662,948
  MAM    vertebrate_mammalian    121      5,892,072,442
  MIT    mitochondrion             1        113,495,942
  PLN    plant                    67      4,368,670,110
  PRO    protozoa                  7        786,600,954
  PSM    plasmid                   9      1,388,949,422
  PST    plastid                   3        512,697,988
  VRL    viral                     1        227,714,284
  VRT    vertebrate_other        178     10,748,398,867

The counts refer to data files run through the FilterByDivision function. This favors the non-redundant "WP" protein sequence files for archaea and bacteria, which were introduced for prokaryotes in 2013.

"Populate" Helper

Unless the source records are perfect for your intended use, you can design a more convenient structure containing just the information you want.

The populate‑peptide script originally converted the downloaded GenPept flatfiles to INSDSeq XML. The current record number, which was obtained by using a plus sign ("+") as an xtract ‑element argument, then became the integer access key for caching the derived record:

  base=${fl%.gz}
  if [ ! -f "${sentinelsBase}/$base.snt" ] && [ -s "$fl" ]
  then

    gunzip -c "$fl" | gbf2xml |
    xtract -rec PeptInfo -pattern INSDSeq \
      -wrp UID -element "+" \
      …
      -wrp Sequence -element INSDSeq_sequence |
    transmute -format |
    rchive -gzip -archive -db "$dbase" -index UID -pattern PeptInfo

    touch "${sentinelsBase}/$base.snt"
  fi

Each protein record is converted into a simple PeptInfo structure with a unique identifier value:

  <PeptInfo>
    <UID>302720946</UID>
    <Accession>WP_003047541</Accession>
    <Organism div="BCT" taxid="271">Thermus aquaticus</Organism>
    <Product>type I DNA topoisomerase</Product>
    <Sequence>mpkkpktqgaahlgeggpkaearattlvvvespakarsiqkmlgp … </Sequence>
    …

(The current code calls a direct conversion function to bypass the INSDSeq intermediate. It also adds the previous maximum UID value, which allows separate divisions to be processed at different times.)

"Index" Helper

Given a protein sequence, ‑element will show it as a polypeptide, ‑letters will split it into individual amino acids, and ‑pentamers will create overlapping oligopeptides (residues 1‑5, 2‑6, 3‑7, etc.):

  -element      mpkkpktqgaahlgeggpkaearattlvvvespakarsiqk …
  -letters      m p k k p k t q g a a h l g e g g p k a e …
  -pentamers    mpkkp pkkpk kkpkt kpktq pktqg ktqga tqgaa …

The ‑letters option resembles words in a sentence, which is the starting point for building the positional indices that support phrase and proximity searches. But a standard phrase search will drop candidates that have even a single mismatch to the query sequence.

A better approach for finding the most similar sequences is to use xsearch ‑match on the pentamers. Indexing short overlapping peptides would make more efficient use of the local archive's directory hierarchy. The choice of 5‑mers was based on results from NCBI's early SEQR indexed search experiment (10). For convenience, when peptide fragments are indexed using the PENT field, ‑match will internally split the protein query sequence into overlapping pentamers before running the search:

  xsearch -db peptide -match "rlgrdtadmiqlikefdaqgvavrfiddgistdgdmgqmv [PENT]"

The result is a set of record keys ranked by the number of unique overlapping fragments matched.

For primary databases, index helpers embed an xtract command in a Unix "HERE" document. This is then passed as an argument for execution within an incremental indexing function. (Secondary projects do not need this invalidated index repopulation mechanism.)

A simplified excerpt of the index‑peptide script:

  read -r -d '' idxtxt <<- EOS
  xtract -set IdxDocumentSet -rec IdxDocument \
    -pattern PeptInfo -UID PeptInfo/UID \
      -wrp IdxUid -element "&UID" -clr -rst -tab "" \
      -group PeptInfo -pkg IdxSearchFields \
        …
        -block Product -wrp PROD -indexer Product \
        -block Organism -wrp ORGN -element Organism -rst \
          -wrp DIV -element @div \
        -block Sequence  …  -wrp PENT -pentamers Sequence
  EOS

  IncrementalIndex "${idxtxt}"

will split the overlapping pentamers into separate terms for regular indexing:

  …
  <IdxDocument>
    <IdxUid>302720946</IdxUid>
    <IdxSearchFields>
      <UID>0302720946</UID>
      <ACCN-WP>003047541</ACCN-WP>
      <PROD pos="3">dna</PROD>
      <PROD pos="2">i</PROD>
      <PROD pos="4">topoisomerase</PROD>
      <PROD pos="1">type</PROD>
      <ORGN>Thermus aquaticus</ORGN>
      <DIV>BCT</DIV>
      <SLEN>833</SLEN>
      <MLWT>93298</MLWT>
      <PENT>mpkkp</PENT>
      <PENT>pkkpk</PENT>
      …

Text fields can use ‑indexer to create positional indices, but it also removes stop words, which would discard valid pentapeptides like "apply", "every", and "never". Those attributes will likely be decoupled by adding new configuration options to allow finer adjustment of field-specific behavior.

Later build steps, starting with index inversion, are standardized, and do not need external helper files.

Importing Data Tables

Large data tables are processed by programs written in the Go programming language. They are compiled and executed on-the-fly by helpers calling "go run". The central loop in prep‑nihocc.go, simplified for brevity, and without the usual reality checks and periodic output buffer flushing, is:

  var bldr strings.Builder

  wrtr := bufio.NewWriter(os.Stdout)
  scanr := bufio.NewScanner(os.Stdin)

  for scanr.Scan() {

      line := scanr.Text()
      if line == "citing,referenced" {
          continue
      }

      cols := strings.Split(line, ",")
      fst, scd := cols[0], cols[1]

      pdFst, pdScd := padNumericID(fst), padNumericID(scd)

      bldr.WriteString(fst + "\tCITED\t" + pdScd + "\n")
      bldr.WriteString(scd + "\tCITES\t" + pdFst + "\n")

      txt := bldr.String()
      wrtr.WriteString(txt[:])
      bldr.Reset()
  }

  wrtr.Flush()

Archive Configuration File

More flexible drive management uses an environment variable that points to a configuration file

  export EDIRECT_LOCAL_CONFIG="${HOME}/.edirectrc"

(For quick evaluation, you can instead place an edirect.ini configuration file into the edirect folder.)

The configuration file has one section per database, with entries pointing to solid-state drives:

  [pubmed]

An ARCHIVE entry is required, and defaults to having all folders in the absence of other entries, but it is primarily for the Archive, Data, and Posting folders needed for record retrieval and indexed queries:

  ARCHIVE=/Volumes/archive/

If set, a WORKING entry will move Extras, Invert, Merged, Scratch, and Source folders, used for building the archive and search indices but not for active queries, to another drive:

  WORKING=/Volumes/builder-pool-pm/

(Scratch prevents secondary projects from colliding with primary incremental inverted index files.)

Independently, POSTING and SOURCES can move their respective folders to other drives. The source file repository can even use a large, rotating hard disk, since it only needs sequential streaming access:

  POSTING=/Volumes/pm-posting/
  SOURCES=/Volumes/common-sources-1/

An optional [default] section applies to any databases that do not have their own named sections.

With this configuration mechanism, each separate database can point to its own set of unique drives.

Solid-State Drive Preparation

To initialize a solid-state drive for hosting the local archive on a Mac, log into an admin account, run Disk Utility, choose View → Show All Devices, select the top-level external drive, and press the Erase icon. Set the Scheme popup to GUID Partition Map, and APFS will appear as a format choice. Set the Format popup to APFS, enter the desired name for the volume, and click the Erase button.

To finish the drive configuration, disable Spotlight indexing on the drive with:

  sudo mdutil -i off "${EDIRECT_LOCAL_ARCHIVE}"
  sudo mdutil -E "${EDIRECT_LOCAL_ARCHIVE}"

and disable FSEvents logging with:

  sudo touch "${EDIRECT_LOCAL_ARCHIVE}/.fseventsd/no_log"

Also exclude the drive from being backed up by Time Machine or scanned by a virus checker, and, in Apple → System Settings → Privacy & Security → Full Disk Access, turn on the Terminal slide switch.

Einfo Data

Einfo field data contains status flags for several term list index properties:

  <Field>
    <Name>ALL</Name>
    <FullName>All Fields</FullName>
    <Description>All terms from all searchable fields</Description>
    <TermCount>280005319</TermCount>
    <IsDate>N</IsDate>
    <IsNumerical>N</IsNumerical>
    <SingleToken>N</SingleToken>
    <Hierarchy>N</Hierarchy>
    <IsHidden>N</IsHidden>
    <IsTruncatable>Y</IsTruncatable>
    <IsRangable>N</IsRangable>
  </Field>

Additional Elink Options

Elink has several additional modes that can be specified with the ‑cmd argument. When not using the default "neighbor" command ("neighbor_history" has been retired), elink will return an eLinkResult XML object, with the links for each UID presented in separate blocks. For example, the "neighbor" command:

  esearch -db pubmed -query "Hoffmann PC [AUTH] AND dopamine [MAJR]" |
  elink -related -cmd neighbor |
  xtract -pattern LinkSetDb -element Id

will show the original PMID in the first column and related article PMIDs in subsequent columns:

  1504781    11754494    3815119    1684029    14614914    12128255    …
  1684029    3815119     1504781    8097798    17161385    14755628    …
  2572612    2903614     6152036    2905789    9483560     1352865     …
  …

The "acheck" command returns all available link names for each record:

  esearch -db pubmed -query "Federhen S [AUTH]" |
  elink -cmd acheck |
  xtract -pattern LinkSet -tab "\n" -element IdLinkSet/Id \
    -block LinkInfo -tab "\n" -element LinkName

printing each on its own line:

  25510495
  pubmed_images
  pubmed_pmc
  pubmed_pmc_local
  pubmed_pmc_refs
  pubmed_pubmed
  pubmed_pubmed_citedin
  …

The "prlinks" command can obtain the URL reference to the publisher web page for an article. The Unix "xargs" command calls elink separately for each identifier:

  epost -db pubmed -id 22966225,19880848 |
  efetch -format uid |
  xargs -n 1 elink -db pubmed -cmd prlinks -id |
  xtract -pattern LinkSet -first Id -element ObjUrl/Url

Nquire Arguments

The nquire program uses command-line arguments to obtain data from RESTful, CGI, or FTP servers. Queries are built up from command-line arguments. Paths can be separated into components, which are combined with slashes. Remaining arguments (starting with a dash) are tag/value pairs, with multiple values between tags combined with commas.

For example, a POST request (using ‑url):

  nquire -url https://eutils.ncbi.nlm.nih.gov/entrez/eutils/ espell.fcgi \
    -db pubmed -term "vitamen" |
  xtract -pattern eSpellResult -element CorrectedQuery

returns the corrected spelling of "vitamen":

  vitamin

while a GET query:

  nquire -get https://eutils.ncbi.nlm.nih.gov/entrez/eutils/ einfo.fcgi -db pubmed |
  xtract -pattern DbInfo -block Field -element Name FullName -deq "\n"

returns information about the fields indexed for the PubMed database:

  ALL     All Fields
  UID     UID
  FILT    Filter
  TITL    Title
  MESH    MeSH Terms
  MAJR    MeSH Major Topic
  …

and an FTP request:

  nquire -ftp ftp.ncbi.nlm.nih.gov pub/gdp ideogram_9606_GCF_000001305.14_850_V1 |
  grep acen | cut -f 1,2,6,7 | awk '/^X\t/'

returns data with the (estimated) sequence coordinates of the human X chromosome centromere (here showing where the p and q arms meet):

  X    p    58100001    61000000
  X    q    61000001    63800000

Nquire can also produce a list of files in an FTP server directory:

  nquire -lst ftp://nlmpubs.nlm.nih.gov online/mesh/MESH_FILES/xmlmesh

or a list of FTP file names preceded by a column with the file sizes:

  nquire -dir ftp.ncbi.nlm.nih.gov gene/DATA

Finally, nquire can download FTP files to the local disk:

  nquire -dwn ftp.nlm.nih.gov online/mesh/MESH_FILES/xmlmesh desc2026.zip

If Aspera Connect is installed, the nquire ‑asp command will provide faster retrieval from NCBI servers:

  nquire -asp ftp.ncbi.nlm.nih.gov pubmed baseline pubmed26n0001.xml.gz

Without Aspera Connect, nquire ‑asp defaults to using the ‑dwn logic.

Announcement Mailing List

NCBI posts general announcements regarding the E-utilities to the utilities-announce announcement mailing list. This mailing list is an announcement list only; individual subscribers may not send mail to the list. Also, the list of subscribers is private and is not shared or used in any other way except for providing announcements to list members. The list receives about one posting per month. Please subscribe at the above link.

Documentation

EDirect navigation functions call the URL-based Entrez Programming Utilities:

  https://www.ncbi.nlm.nih.gov/books/NBK25501

NCBI database resources are described by:

  https://www.ncbi.nlm.nih.gov/pubmed/37994677

Information on how to obtain an API Key is described in this NCBI blogpost:

  https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities

An introduction to shell scripting for non-programmers is at:

  https://missing.csail.mit.edu/2020/shell-tools/

An article on the Go programming language, written by its creators, is at:

  https://cacm.acm.org/research/the-go-programming-language-and-environment/

and transcripts of talks on design philosophy and retrospective experience of Go are at:

  https://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html

  https://commandcenter.blogspot.com/2024/01/what-we-got-right-what-we-got-wrong.html

Instructions for downloading and installing the Go compiler are at:

  https://golang.org/doc/install#download

Additional NCBI website and data usage policy and disclaimer information is located at:

  https://www.ncbi.nlm.nih.gov/home/about/policies/

Public Domain Notice

A copy of the NCBI Public Domain Notice, which applies to EDirect, is shown below:

  
                            PUBLIC DOMAIN NOTICE
               National Center for Biotechnology Information

  This software/database is a "United States Government Work" under the
  terms of the United States Copyright Act. It was written as part of
  the author's official duties as a United States Government employee and
  thus cannot be copyrighted. This software/database is freely available
  to the public for use. The National Library of Medicine and the U.S.
  Government have not placed any restriction on its use or reproduction.

  Although all reasonable efforts have been taken to ensure the accuracy
  and reliability of the software and data, the NLM and the U.S.
  Government do not and cannot warrant the performance or results that
  may be obtained by using this software or data. The NLM and the U.S.
  Government disclaim all warranties, express or implied, including
  warranties of performance, merchantability or fitness for any particular
  purpose.

  Please cite the author in any work or product based on this material.
  

Getting Help

Please refer to the PubMed and Entrez help documents for more information about search queries, database indexing, field limitations and database content.

Suggestions, comments, and questions specifically relating to the EUtility programs may be sent to vog.hin.mln.ibcn@seitilitue.