fƠVdZdZddlmZddlmZddlZddlZddlZddl Z dZ ddl Z e rdZndZdZd ZeZej(ej+d ej,ej(ej+d ej,d ee<ej(eej,ej(eej,d ee<dd lmZGd deZGddZGddZy#e $r' ddl Z n#e $r ddlZ n #e $rdZ YnwxYwYnwxYwYwxYw)aBBeautiful Soup bonus library: Unicode, Dammit This library converts a bytestream to Unicode through any means necessary. It is heavily based on code from Mark Pilgrim's Universal Feed Parser. It works best on XML and HTML, but it does not rewrite the XML or HTML to reflect a new encoding; that's the tree builder's job. MIT)codepoint2name) defaultdictNcTt|trytj|dS)Nencoding) isinstancestrchardet_moduledetectss S/var/lib/jenkins/workspace/mettalog/venv/lib/python3.12/site-packages/bs4/dammit.pychardet_dammitr+s% a $$Q' 33cyNr s rrr0srz$^\s*<\?.*encoding=['"](.*?)['"].*\?>z0<\s*meta[^>]+charset\s*=\s*["']?([^>]*?)[ /;'">]ascii)htmlxml)html5ceZdZdZdZe\ZZZddddddZe jd Z e jd Z e d Ze d Ze d Ze ddZe ddZe dZy)EntitySubstitutionzFThe ability to substitute XML or HTML entities for certain characters.ci}i}t}tt}ttjD]\}}|j dr|dd}n|}||vr|||<|||<t |dk(rt|dkr|dvrNt |dkDrtd|Drot |dk(r|j|||dj|t}|D]T}||} | s|j|d j| D cgc]} | d c} } |j|d | d Vt|jD]} | D]} |j| d d j|z}ttjD]\}}t|}|||<||tj |fScc} w)uInitialize variables used by this class to manage the plethora of HTML5 named entities. This function returns a 3-tuple containing two dictionaries and a regular expression: unicode_to_name - A mapping of Unicode strings like "⦨" to entity names like "angmsdaa". When a single Unicode string has multiple entity names, we try to choose the most commonly-used name. name_to_unicode: A mapping of entity names like "angmsdaa" to Unicode strings like "⦨". named_entity_re: A regular expression matching (almost) any Unicode string that corresponds to an HTML5 named entity. ;Nz<>&c38K|]}t|dkyw)rN)ord).0xs r z?EntitySubstitution._populate_class_variables..s)J1#a&3,)Jsrz(?![z])z(%s)|)setrsortedritemsendswithlenr alladdjoinlistvaluesrchrrecompile)unicode_to_namename_to_unicodeshort_entities long_entities_by_first_charactername_with_semicolon charactername particlesshort long_versionsr"ignore long_entities long_entity re_definition codepoints r_populate_class_variablesz,EntitySubstitution._populate_class_variablesFs$+6s+;(.4U[[].C * #++C0*3B/* ?*(1% *.OI &I!#I(<U* 9~!c)J )J&J9~"""9-01>BB9M}/DDE #E1!A$!>? eV<=$""B"I"I"KLM,  k* -M)!44  $N$8$8$:;OItII)-OI & <M1JJJ+"?s G aposquotampltgt)'"&<>z&([<>]|&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;))z([<>&])c`|jj|jd}d|zS)zpUsed with a regular expression to substitute the appropriate HTML entity for a special character string.r&%s;)CHARACTER_TO_HTML_ENTITYgetgroupclsmatchobjentitys r_substitute_html_entityz*EntitySubstitution._substitute_html_entitys---11(..2CDrcH|j|jd}d|zS)zoUsed with a regular expression to substitute the appropriate XML entity for a special character string.rrN)CHARACTER_TO_XML_ENTITYrQrRs r_substitute_xml_entityz)EntitySubstitution._substitute_xml_entitys',,X^^A->?rcTd}d|vrd|vrd}|jd|}nd}||z|zS)a*Make a value into a quoted XML attribute, possibly escaping it. Most strings will be quoted using double quotes. Bob's Bar -> "Bob's Bar" If a string contains double quotes, it will be quoted using single quotes. Welcome to "my bar" -> 'Welcome to "my bar"' If a string contains both single and double quotes, the double quotes will be escaped, and the string will be quoted using double quotes. Welcome to "Bob's Bar" -> "Welcome to "Bob's bar" rIrHz")replace)selfvalue quote_with replace_withs rquoted_attribute_valuez)EntitySubstitution.quoted_attribute_valuesE& %<e| (  c<8! E!J..rcx|jj|j|}|r|j|}|S)a Substitute XML entities for special XML characters. :param value: A string to be substituted. The less-than sign will become <, the greater-than sign will become >, and any ampersands will become &. If you want ampersands that appear to be part of an entity definition to be left alone, use substitute_xml_containing_entities() instead. :param make_quoted_attribute: If True, then the string will be quoted, as befits an attribute value. )AMPERSAND_OR_BRACKETsubrYr`rSr]make_quoted_attributes rsubstitute_xmlz!EntitySubstitution.substitute_xmls?((,,  & &/ !..u5E rcx|jj|j|}|r|j|}|S)aSubstitute XML entities for special XML characters. :param value: A string to be substituted. The less-than sign will become <, the greater-than sign will become >, and any ampersands that are not part of an entity defition will become &. :param make_quoted_attribute: If True, then the string will be quoted, as befits an attribute value. )BARE_AMPERSAND_OR_BRACKETrcrYr`rds r"substitute_xml_containing_entitiesz5EntitySubstitution.substitute_xml_containing_entitiess?--11  & &/ !..u5E rcN|jj|j|S)aReplace certain Unicode characters with named HTML entities. This differs from data.encode(encoding, 'xmlcharrefreplace') in that the goal is to make the result more readable (to those with ASCII displays) rather than to recover from errors. There's absolutely nothing wrong with a UTF-8 string containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that character with "é" will make it more readable to some people. :param s: A Unicode string. )CHARACTER_TO_HTML_ENTITY_RErcrV)rSr s rsubstitute_htmlz"EntitySubstitution.substitute_html+s(..22  ' ', ,rN)F)__name__ __module__ __qualname____doc__rBrOHTML_ENTITY_TO_CHARACTERrkrXr1r2rhrb classmethodrVrYr`rfrirlrrrrrCsPuKn%>$?"7       !+ ,/!0&2::i0  "/"/H(*/*,,rrcVeZdZdZ ddZdZedZedZ ed dZ y) EncodingDetectoraLSuggests a number of possible encodings for a bytestring. Order of precedence: 1. Encodings you specifically tell EncodingDetector to try first (the known_definite_encodings argument to the constructor). 2. An encoding determined by sniffing the document's byte-order mark. 3. Encodings you specifically tell EncodingDetector to try if byte-order mark sniffing fails (the user_encodings argument to the constructor). 4. An encoding declared within the bytestring itself, either in an XML declaration (if the bytestring is to be interpreted as an XML document), or in a tag (if the bytestring is to be interpreted as an HTML document.) 5. An encoding detected through textual analysis by chardet, cchardet, or a similar external library. 4. UTF-8. 5. Windows-1252. NcDt|xsg|_|r|xj|z c_|xsg|_|xsg}t|Dcgc]}|j c}|_d|_||_d|_|j|\|_ |_ ycc}w)aConstructor. :param markup: Some markup in an unknown encoding. :param known_definite_encodings: When determining the encoding of `markup`, these encodings will be tried first, in order. In HTML terms, this corresponds to the "known definite encoding" step defined here: https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding :param user_encodings: These encodings will be tried after the `known_definite_encodings` have been tried and failed, and after an attempt to sniff the encoding by looking at a byte order mark has failed. In HTML terms, this corresponds to the step "user has explicitly instructed the user agent to override the document's character encoding", defined here: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding :param override_encodings: A deprecated alias for known_definite_encodings. Any encodings here will be tried immediately after the encodings in known_definite_encodings. :param is_html: If True, this markup is considered to be HTML. Otherwise it's assumed to be XML. :param exclude_encodings: These encodings will not be tried, even if they otherwise would be. N) r.known_definite_encodingsuser_encodingsr&lowerexclude_encodingschardet_encodingis_htmldeclared_encodingstrip_byte_order_markmarkupsniffed_encoding)r\r~rvr{ryrwoverride_encodingsr"s r__init__zEncodingDetector.__init__XsD)--E-K(L%   ) )-? ? ),2-3!$9J%KAaggi%K!L $ !%.2-G-G-O* T* &LsBcr|5|j}||jvry||vr|j|yy)zShould we even bother to try this encoding? :param encoding: Name of an encoding. :param tried: Encodings that have already been tried. This will be modified as a side effect. FT)rxryr,)r\rtrieds r_usablezEncodingDetector._usablesA  ~~'H4111u$ (#rc#Kt}|jD]}|j||s||j|j|r|j|jD]}|j||s||j +|j |j|j|_|j|j |r|j |jt|j|_ |j|j|r|jdD]}|j||s|yw)zmYield a number of encodings that might work for this markup. :yield: A sequence of strings. N)utf-8 windows-1252) r&rvrrrwr|find_declared_encodingr~r{rzr)r\res r encodingszEncodingDetector.encodingss' ..A||Au%/ <<--u 5'' '$$A||Au%%  ! ! )%)%@%@ T\\&+D " <<.. 6(( (  ($24;;$?D ! <<--u 5'' '+A||Au%+s,EAECEEcDd}t|tr||fSt|dk\r|dddk(r|dddk7r d}|dd}||fSt|dk\r|dddk(r|dddk7r d}|dd}||fS|dd d k(r d }|d d}||fS|ddd k(r d }|dd}||fS|dddk(rd}|dd}||fS)zIf a byte-order mark is present, strip it and return the encoding it implies. :param data: Some markup. :return: A 2-tuple (modified data, implied encoding) Nszzutf-16beszutf-16lesrszutf-32beszutf-32le)rr r*)rSdatars rr}z&EncodingDetector.strip_byte_order_marks' dC > ! INbq[!81I+!H8DX~$i1n48{#:1Qi:-!H8DX~"1X (H8DX~ "1X, ,!H8DX~"1X, ,!H8DX~rc|rt|x}}n#d}tdtt|dz}t|trt t}n t t }|d}|d}d} |j||} | s|r|j||} | | jd} | r2t| tr| jd d } | jSy) aGiven a document, tries to find its declared encoding. An XML encoding is declared at the beginning of the document. An HTML encoding is declared in a tag, hopefully near the beginning of the document. :param markup: Some markup. :param is_html: If True, this markup is considered to be HTML. Otherwise it's assumed to be XML. :param search_entire_document: Since an encoding is supposed to declared near the beginning of the document, most of the time it's only necessary to search a few kilobytes of data. Set this to True to force this method to search the entire document. iig?rrN)endposrrr[) r*maxintrbytes encoding_resr searchgroupsdecoderx) rSr~r{search_entire_document xml_endpos html_endposresxml_rehtml_rer|declared_encoding_matchs rrz'EncodingDetector.find_declared_encodings "'*6{ 2JJdCF d(:$; > @ C  +U3$5$<$dddd?dd@ddAddBddCddDddEd!dFd#dGd%dHd'dId)d*d+dJd-d*d.d*idKdLdMdLdNdOdPdOdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dad*dbdcdddedfddgdhididjdkdldmdndodpdqdrdsdGdtdudvdwdxdydzd{d|dhd}dd~dddTddddddiddddddddddRdd?ddddydddddddddd*ddddddddiddddddddddddddddddddddddddddddddddiddddddRddddddddddddeddœddǓddɓddɓddɓddɓddɓddɓiddГddjddӓddӓddӓddӓddؓddؓddؓddؓddddޓdddddddddddddddddddd Ziddddddddddddddddddddddddddddddddddid d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*id+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLidMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdnidodpdqdrdsdtdudvdwdxdydzd{d|d}d~ddddddddddddddddddiddddddddddddddddgddddddddddddddddddiddddddddddddddddddÓdĐdœdƐdǓdȐdɓdʐd˓d̐d͓dΐdϓdАdѓdҐdӓdԐdՐd֐dלZgdآZeddZeddZe dސd܄Zy( UnicodeDammitzA class for detecting the encoding of a *ML document and converting it to a Unicode string. If the source encoding is windows-1252, can replace MS smart quotes with their HTML or XML equivalents.z mac-romanz shift-jis) macintoshzx-sjis)rz iso-8859-1z iso-8859-2NFc||_g|_d|_||_t j t |_t|||||||_ t|ts|dk(r||_ t||_ d|_y|jj|_ d}|jjD]-} |jj}|j!| }|-n|sX|jjD]?} | dk7r|j!| d}||jj#dd|_n||_ |sd|_yy)a2Constructor. :param markup: A bytestring representing markup in an unknown encoding. :param known_definite_encodings: When determining the encoding of `markup`, these encodings will be tried first, in order. In HTML terms, this corresponds to the "known definite encoding" step defined here: https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding :param user_encodings: These encodings will be tried after the `known_definite_encodings` have been tried and failed, and after an attempt to sniff the encoding by looking at a byte order mark has failed. In HTML terms, this corresponds to the step "user has explicitly instructed the user agent to override the document's character encoding", defined here: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding :param override_encodings: A deprecated alias for known_definite_encodings. Any encodings here will be tried immediately after the encodings in known_definite_encodings. :param smart_quotes_to: By default, Microsoft smart quotes will, like all other characters, be converted to Unicode characters. Setting this to 'ascii' will convert them to ASCII quotes instead. Setting it to 'xml' will convert them to XML entity references, and setting it to 'html' will convert them to HTML entity references. :param is_html: If True, this markup is considered to be HTML. Otherwise it's assumed to be XML. :param exclude_encodings: These encodings will not be considered, even if the sniffing code thinks they might make sense. Fr$Nrr[zSSome characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.T)smart_quotes_totried_encodingscontains_replacement_charactersr{logging getLoggerrmlogrtdetectorrr r~unicode_markuporiginal_encodingr _convert_fromwarning) r\r~rvrr{ryrwrurs rrzUnicodeDammit.__init__sPL /!/4, $$X.( ,g7H .  fc "fl DK"%f+D %)D " mm**  //H]]))F""8,A} 0 !MM33w&**8Y?A=HH$$C<@D84 %)D "rc|jd}|jdk(r+|jj|j }|S|j j|}t |tk(r}|jdk(r7dj |dj zdj z}|Sdj |dj zdj z}|S|j }|S)z[Changes a MS smart quote character to an XML or HTML entity, or an ASCII character.rrrz&#xrrJr)rQrMS_CHARS_TO_ASCIIrPencodeMS_CHARStypetuple)r\matchorigrcs r _sub_ms_charzUnicodeDammit._sub_ms_charus{{1~   7 *((,,T299;C --##D)CCyE!''50,,.3q6==?:SZZ\IC  **,Q83::<GC jjl rc|j|}|r||f|jvry|jj||f|j}|jA||j vr3d}t j|}|j|j|} |j|||}||_||_ |jS#t$r }Yd}~yd}~wwxYw)z|Attempt to convert the markup to the proposed encoding. :param proposed: The name of a character encoding. Ns([-])) find_codecrappendr~rENCODINGS_WITH_SMART_QUOTESr1r2rcr _to_unicoder Exception)r\proposederrorsr~smart_quotes_resmart_quotes_compiledrrs rrzUnicodeDammit._convert_froms ??8,Hf-1E1EE ##Xv$67  ,D<<<.O$&JJ$? !*..t/@/@&IF   6:ADK%-D " {{   s!C CCct|||S)z}Given a string and its encoding, decodes the string into Unicode. :param encoding: The name of an encoding. )r )r\rrrs rrzUnicodeDammit._to_unicodes 46**rcH|jsy|jjS)zhIf the markup is an HTML document, returns the encoding declared _within_ the document. N)r{rr|)r\s rdeclared_html_encodingz$UnicodeDammit.declared_html_encodings ||}}...rcN|j|jj||xsf|xr!|j|jddxs?|xr!|j|jddxs|xr|j xs|}|r|j Sy)zConvert the name of a character set to a codec name. :param charset: The name of a character set. :return: The name of a codec. -r$_N)_codecCHARSET_ALIASESrPr[rx)r\charsetr]s rrzUnicodeDammit.find_codecs T1155gwGHDt{{7??3+CDEt{{7??3+DE.w}}  ;;= rcn|s|Sd} tj||}|S#ttf$rY|SwxYwr)codecslookup LookupError ValueError)r\rcodecs rrzUnicodeDammit._codecsJN  MM' "E Z(    s !44)euro20AC )sbquo201A)fnof192)bdquo201E)hellip2026)dagger2020)Dagger2021)circ2C6)permil2030)Scaron160)lsaquo2039)OElig152?)z#x17D17D)lsquo2018)rsquo2019)ldquo201C)rdquo201D)bull2022)ndash2013)mdash2014)tilde2DC)trade2122)scaron161)rsaquo203A)oelig153)z#x17E17E)Yumlr$)EUR,fz,,z...+z++^%SrKOEZrrHrrrIrr*rrrz--r ~r!z(TM)r"r r#rLr$oer%r&zr'Y!cGBP$YENr%z..r$z(th)z<<z(R)oz+-23)rHacuterP1z>>z1/4z1/2z3/4AAECEIDNOUbBaaerin/y) rs€s‚sƒs„s…s†s‡sˆs‰sŠs‹sŒsŽs‘s’s“s”s•s–s—s˜s™sšs›sœsžsŸs s¡s¢s£s¤s¥s¦s§s¨s©sªs«s¬s­s®s¯s°s±s²s³s´sµs¶s·s¸s¹sºs»s¼s½s¾s¿sÀsÁsÂsÃsÄsÅsÆsÇsÈsÉsÊsËsÌsÍsÎsÏsÐsÑsÒsÓsÔsÕsÖs×sØsÙsÚsÛsÜsÝsÞsßsàsâsãsäsåsæsçsèsésêsësìsísîsïsðsñsòsósôsõsös÷søsùsúsûsüsýsþ)))rr r)r rr)rr rrrrc|jddjdvr td|jdvr tdg}d}d}|t|kr||}t |t s t |}||jk\r7||jkr(|jD]\}} } ||k\s || ks|| z }nTnR|dk\rH||jvr:|j||||j|j||d z }|}n|d z }|t|kr|dk(r|S|j||d d j|S) aFix characters from one encoding embedded in some other encoding. Currently the only situation supported is Windows-1252 (or its subset ISO-8859-1), embedded in UTF-8. :param in_bytes: A bytestring that you suspect contains characters from multiple encodings. Note that this _must_ be a bytestring. If you've already converted the document to Unicode, you're too late. :param main_encoding: The primary encoding of `in_bytes`. :param embedded_encoding: The encoding that was used to embed characters in the main document. :return: A bytestring in which `embedded_encoding` characters have been converted to their `main_encoding` equivalents. rr)r windows_1252zPWindows-1252 and ISO-8859-1 are the only currently supported embedded encodings.)utf8rz4UTF-8 is the only currently supported main encoding.rrrNr) r[rxNotImplementedErrorr*rrr FIRST_MULTIBYTE_MARKERLAST_MULTIBYTE_MARKERMULTIBYTE_MARKERS_AND_SIZESWINDOWS_1252_TO_UTF8rr-) rSin_bytes main_encodingembedded_encoding byte_chunks chunk_startposbytestartendsizes r detwinglezUnicodeDammit.detwingles&  $ $S# . 4 4 6?, ,%&' '    (9 9%FH H  CM!C=DdC(4y222C555),(G(G$E3u}t )H$#*B*B"B""8K#<=""3#;#;D#ABq! q3CM!4 ! O   x 5 6xx $$r)strict)r-r)rmrnrorprrrrrrrrrrrrr2r1r/r0rrr=rrrrr sm%0!,.O#  9;!%u $V*p">+// ()((*((* ( + ( + (+((+(*(+()(()(( !("+***)**)**+))%?(HHB %B #B  #B  # B $ B % B  #B  $B  #B  #B  #B  #B  $B  #B  #B #!B " ##B $ #%B & #'B ( #)B * #+B , #-B . #/B 0 $1B 2 #3B 4 &5B 6 #7B 8 #9B : $;B < #=B > #?B @ #AB B #CB D #EB F #GB H %IB J #KB N %OB P #QB R #SB T $UB V "WB X &YB Z $[B \ #]B ^ #_B ` %aB b #cB d #eB f $gB h #iB j #kB l .mB n #oB p #qB r #sB t #uB v #wB x &yB z ${B | %}B ~ %B @ %AB B #CB D #EB F #GB H #IB J #KB L #MB N #OB P $QB R #SB T #UB V #WB X #YB Z #[B \ #]B ^ #_B ` #aB b #cB d #eB f #gB h #iB j #kB l #mB n #oB p #qB r #sB t #uB v #wB x #yB z #{B | #}B ~ #B @ #AB B #CB D #EB F #GB H #IB J #KB L #MB N #OB P $QB R #SB T #UB V #WB X #YB Z #[B \ #]B ^ #_B ` #aB b #cB d #eB f #gB h #iB j #kB l #mB n #oB p #qB rCB R{  {  {  {{   {  {  {  {  {{  {  {{  {  {{  {{  {  { !{ " #{ $ %{ & '{ ( ){ * {+{ , -{ . {/{ 0 1{ 2 {3{ 4 {5{ 6 {7{ 8 {9{ : {;{ < {={ > {?{ @ {A{ B {C{ D {E{ F {G{ H {I{ J {K{ L {M{ N {O{ P {Q{ R {S{ T {U{ V {W{ X {Y{ Z {[{ \ {]{ ^ {_{ ` {a{ b {c{ d {e{ f {g{ h {i{ j {k{ l {m{ n {o{ p {q{ r {s{ t {u{ v {w{ x {y{ z {{{ | {}{ ~ {{ @ {A{ B {C{ D {E{ F {G{ H {I{ J {K{ L {M{ N {O{ P {Q{ R {S{ T {U{ V {W{ X {Y{ Z {[{ \ {]{ ^ {_{ ` {a{ b {c{ d {e{ f {g{ h {i{ j {k{ l {m{ n {o{ p {q{ r {s{ t {u{ v {w{ x {y{ z w{{ | {}{ ~ {{ @ {A{ B {C{ D {E{ F {G{ H {I{ J {K{ L {M{ N {O{ P {Q{ R {S{ T {U{ V {W{ X {Y{ Z {[{ \ {]{ ^ {_{ ` {a{ b {c{ d {e{ f {g{ h {i{ j {k{ l {m{ n {o{ pu{ z#  9;A>7;A>/5$2@%@%rr)rp __license__ html.entitiesr collectionsrrr1rstringr cchardet ImportErrorchardetcharset_normalizerr xml_encoding html_metadictrr2rrrrr robjectrrtrrrrrKsH (#  "%4  ; C v RZZ ((1244 8 BJJ|**73RTT : U RZZ 244 ( BJJ|RTT * S  w,w,tKKZ|%|%[ " " ) " " 7 "!N " " "sYC<<D(DD(D" DD"DD"DD"D(!D""D('D(