[pcre-dev] [Bug 1841] New: Pcre using a lot of cpu and time …

Góra strony
Delete this message
Autor: admin
Data:  
Dla: pcre-dev
Temat: [pcre-dev] [Bug 1841] New: Pcre using a lot of cpu and time to match
https://bugs.exim.org/show_bug.cgi?id=1841

            Bug ID: 1841
           Summary: Pcre using a lot of cpu and time to match
           Product: PCRE
           Version: 8.34
          Hardware: x86-64
                OS: Windows
            Status: NEW
          Severity: bug
          Priority: medium
         Component: Code
          Assignee: ph10@???
          Reporter: pankaj-g@???
                CC: pcre-dev@???


Created attachment 890
--> https://bugs.exim.org/attachment.cgi?id=890&action=edit
Very Sleepy Data

Hi,

We are using PCRE is our application to match regex against log files.

A log file of size 352 MB (369,827,143 bytes), 1,819,451 lines take close to
5hrs to match with our application.

Used Very Sleepy(http://www.codersnotes.com/sleepy/) to check what was hogging
the cpu and why it was taking so much time.

And from the data of 60minutes (3600 sec) we can see that pcre_exec took
59.4minutes(3565sec) totally.

the regex used is
/(consider|minute|accord|evident|practice|intend|concern|commit|issue|approach|establish|utter|conduct|engage|obtain|scarce|straight|stock|apparent|fancy|concept|court|appoint|passage|vain|coast|project|commission|constant|circumstances|constitute|affect|institute|render|appeal|theory|range|campaign|league|labor|confer|grant|dwell|entertain|contract|earnest|yield|wander|insist|knight|convince|inspire|convention|skill|harry|financial|novel|furnish|compel|venture|territory|temper|bent|intimate|undertake|majority|assert|crew|chamber|humble|keen|liberal|despair|tide|attitude|justify|flag|merit|manifest|notion|scale|formal|persist|contempt|tour|plead|weigh|distinction|inclined|exert|oppress|contend|stake|toil|perish|disposition|rail|cardinal|boast|advocate|bestow|allege|notwithstanding|lofty|multitude|steep|heed|modest|apt|esteem|credible|provoke|tread|ascertain|fare|perpetual|decree|contrive|elaborate|substantial|frontier|facile|cite|warrant|sob|rider|dense|afflict|flourish|ordain|pious|vex|gravity|suspended|conspicuous|retort|bolt|assent|purse|plus|sanction|proceeding|exalt|siege|malice|extravagant|wax|throng|venerate|assail|sublime|exploit|exertion|kindle|endow|imposed|humiliate|suffrage|ensue|brook|gale|muse|satire|intrigue|indication|dispatch|cower|wont|canon|impel|latitude|vacate|undertaking|slay|predecessor|delicacy|forsake|beseech|philosophical|grove|frustrate|illustrious|pomp|entreat|impart|propriety|consecrate|proceeds|fathom|objective|clad|partisan|faction|contrived|venerable|restrained|besiege|manifestation|rebuke|insurgent|rhetoric|scrupulous|ratify|stump|discreet|imposing|wistful|mortify|ripple|premise|subside|adverse|caprice|muster|comprehensive|accede|fervent|cohere|tribunal|austere|recovering|stratum|conscientious|arbitrary|exasperate|conjure|ominous|edifice|elude|pervade|foster|admonish|repeal|retiring|incidental|acquiesce|slew|usurp|sentinel|precision|depose|wanton|odium|precept|deference|fray|candid|enduring|impertinent|bland|insinuate|nominal|suppliant|languid|rave|monetary|headlong|infallible|coax|explicate|gaunt|morbid|ranging|pacify|pastoral|dogged|aide|appease|stipulate|recourse|constrained|bate|aversion|conceit|loath|rampart|extort|tarry|perpetrate|decorum|luxuriant|cant|enjoin|avarice|disconcert|symmetry|capitulate|arbitrate|cleave|append|visage|horde|parable|chastise|foil|veritable|grapple|gentry|projection|prowess|dingy|semblance|tout|fortitude|asunder|rout|staid|beguile|purport|deprave|bequeath|enigma|assiduous|vassal|quail|outskirts|bulwark|swerve|gird|betrothed|prospective|advert|peremptory|rudiment|deduce|halting|ignominy|ideology|pallid|chagrin|obtrude|audacious|construe|ford|repast|stint|fresco|dutiful|hew|parity|affable|interminable|pillage|foreboding|rend|livelihood|deign|capricious|stupendous|chaff|innate|reverie|wrangle|crevice|ostensible|craven|vestige|plumb|reticent|propensity|chide|espouse|raiment|intrepid|seemly|allay|fitful|erode|unaffected|canto|docile|patronize|teem|estrange|spat|warble|mien|sate|constituency|patrician|parry|practitioner|ravel|infest|actuate|surly|convalesce|demoralize|devolve|alacrity|waive|unwonted|seethe|scrutinize|diffident|execrate|implacable|pique|mite|encumber|uncouth|petulant|expiate|cavalier|banter|bluster|debase|retainer|subjugate|extol|fraught|august|fissure|knoll|callous|inculcate|nettle|blanch|inscrutable|tenacious|thrall|exigency|disconsolate|impetus|imposition|auspices|sonorous|exploitation|bane|dint|ignominious|amicable|onset|conservatory|zenith|voluble|yeoman|levity|rapt|sultry|pinion|axiom|descry|retinue|functionary|imbibe|diversified|maraud|grudging|partiality|philology|wry|caucus|permeate|propitious|salient|propitiate|excise|betoken|palatable|upbraid|renegade|hoary|pedantic|coy|troth|encroachment|belie|armada|succor|imperturbable|irresolute|knack|unseemly|accentuate|divulge|brawn|burnish|palpitate|promiscuous|dissemble|flotilla|invective|hermitage|despoil|sully|malevolent|irksome|prattle|subaltern|welt|wreak|tenable|inimitable|depredation|amalgamate|immutable|proxy|dote|reactionary|rationalism|endue|discriminating|brooch|disembark|trappings|abet|clandestine|distend|glib|pucker|rejoinder|spangle|blighted|nicety|aggrieve|vestment|urbane|defray|spectral|munificent|dictum|scabbard|adulterate|beleaguer|gripe|remission|exorbitant|invocation|cajole|inclusive|interdict|abase|obviate|hurtle|unanimity|mettle|interpolate|surreptitious|dissimulate|ruse|specious|revulsion|hale|palliate|obtuse|querulous|vagary|incipient|obdurate|grovel|refractory|dregs|ascendancy|supercilious|pundit|commiserate|alcove|assay|parochial|conjugal|abjure|frieze|ornate|inflammatory|machination|mendicant|meander|bullion|diffidence|makeshift|husbandry|podium|dearth|granary|whet|imposture|diadem|fallow|hubbub|dispassionate|harrowing|askance|lancet|rankle|ramify|gainsay|polity|credence|indemnify|ingratiate|declivity|importunate|whittle|repine|flay|larder|threadbare|grisly|untoward|idiosyncrasy|quip|blatant|stanch|incongruity|perfidious|platitude|revelry|delve|extenuate|polemic|enrapture|virtuoso|glower|mundane|fatuous|incorrigible|postulate|vociferous|purvey|baleful|gibe|dyspeptic|prude|luminary|amenable|willful|overbearing|dais|automate|enervate|wheedle|gusto|bouillon|omniscient|apostate|carrion|emolument|ungainly|impiety|decadence|homily|avocation|circumvent|syllogism|collation|haggle|waylay|savant|cohort|unction|adjure|acrimony|clarion|turbid|cupidity|disaffected|preternatural|eschew|expatiate|didactic|sinuous|rancor|puissant|homespun|embroil|pathological|resonant|libretto|flail|bandy|gratis|upshot|aphorism|redoubtable|corpulent|benighted|sententious|cabal|paraphernalia|vitiate|adulation|quaff|unassuming|libertine|maul|adage|expostulation|tawdry|trite|hireling|ensconce|egregious|cogent|incisive|errant|sedulous|incandescent|derelict|entomology|execrable|sluice|moot|evanescent|vat|dapper|asperity|flair|circumspect|inimical|apropos|gruel|gentility|disapprobation|cameo|gouge|oratorio|inclement|scintilla|confluence|squalor|stricture|emblazon|augury|abut|banal|congeal|pilfer|malcontent|sublimate|eugenic|lineament|firebrand|fiasco|foolhardy|retrench|ulterior|equable|inured|invidious|unmitigated|concomitant|cozen|phlegmatic|dormer|pontifical|disport|apologist|abeyance|enclave|improvident|disquisition|categorical|placate|redolent|felicitous|gusty|natty|pacifist|buxom|heyday|herculean|burgeon|crone|prognosticate|lout|simper|iniquitous|rile|sentient|garish|readjustment|erstwhile|aquiline|bilious|vilify|nuance|gawk|refectory|palatial|mincing|trenchant|emboss|proletarian|careen|debacle|sycophant|crabbed|archetype|cryptic|penchant|bauble|mountebank|fawning|hummock|apotheosis|discretionary|pithy|comport|checkered|ambrosia|factious|disgorge|filch|wraith|demonstrable|pertinacious|emend|laggard|waffle|loquacious|venial|peon|effulgence|fanfare|dilettante|pusillanimous|ingrained|quagmire|mannered|squeamish|proclivity|miserly|vapid|mercurial|perspicuous|nonplus|enamor|hackneyed|spate|pedagogue|acme|masticate|sinecure|indite|emetic|temporize|unimpeachable|genesis|mordant|smattering|suavity|stentorian|junket|appurtenance|nostrum|immure|astringent|unfaltering|tutelage|testator|elysian|fulminate|fractious|pummel|manumit|unexceptionable|triumvirate|sybarite|jibe|magisterial|roseate|obloquy|hoodwink|striate|arrogate|rarefied|chary|credo|superannuated|impolitic|aspersion|abysmal|poignancy|stilted|effete|provender|endemic|jocund|procedural|rakish|skittish|peroration|nonentity|abstemious|viscid|doggerel|sleight|rubric|plenitude|rebus|wizened|whorl|fracas|iconoclast|saturnine|madrigal|discursive|zealot|moribund|modicum|connotation|adventitious|recondite|zephyr|countermand|captious|cognate|forebear|cadaverous|foist|dotage|nexus|choleric|garble|bucolic|denouement|animus|overweening|tyro|preen|largesse|retentive|unconscionable|badinage|insensate|sherbet|beatific|bemuse|microcosm|factitious|gestate|traduce|sextant|coiffure|malleable|rococo|fructify|nihilist|ellipsis|accolade|codicil|roil|grandiloquent|inconsequential|effervescence|stultify|tureen|pellucid|euphony|apocryphal|apocryphal|veracious|pendulous|exegesis|effluvium|apposite|viscous|misanthrope|vintner|halcyon|anthropomorphic|turgid|malaise|polemical|gadfly|atavism|contusion|parsimonious|dulcet|reprise|anodyne|bemused)/


This is one scenario, our general regex can have more then 1000 elements in OR
to match the log file.

We are using 8.34 in our application, we can not upgrade our library because
its is a multi-platform application that will take huge amount of time and a
lot of approvals as well.

Is there a work around or anything we can do to minimize this time and cpu
usage?

Let me know if you want to check the log file as well.

Regard,
Pankaj

--
You are receiving this mail because:
You are on the CC list for the bug.