B .(ไbึใ@sBddlmZddlZddlmZddlmZddlmZdd„Z ej   dd gกd d „ƒZ d d „Z dd„Zdd„Zej   ddddddgกdd„ƒZej   ddgกdd„ƒZej   ddddgกdd „ƒZd!d"„Zej   d#d$gกd%d&„ƒZd'd(„Zej   d)d d*d+id*d,igfgกd-d.„ƒZej   d)d d+d/d0œd*d,igfgกd1d2„ƒZdS)3้)ฺunicode_literalsN)ฺVocab)ฺ Tokenizer)ฺ ensure_pathcCs|dƒ}t|ƒdkst‚dS)Nฺr)ฺlenฺAssertionError)ฺ tokenizerฺtokensฉr ๚S/home/dcms/DCMS/lib/python3.7/site-packages/spacy/tests/tokenizer/test_tokenizer.pyฺtest_tokenizer_handles_no_word sr ฺtextZloremcCs||ƒ}|dj|kst‚dS)Nr)rr)r rr r r r ฺ"test_tokenizer_handles_single_wordsrcCshd}||ƒ}t|ƒdkst‚|djdks.t‚|djdks@t‚|djdksRt‚|djdksdt‚dS) Nz Lorem, ipsum.้rฺLorem้๚,้Zipsum)rrr)r rr r r r ฺtest_tokenizer_handles_punctsrcCs d}||ƒ}t|ƒdkst‚dS)NzLorem, (ipsum).้)rr)r rr r r r ฺ#test_tokenizer_handles_punct_bracessrcCsZddg}d}||ƒ}|dj|krVt|ƒdks2t‚|djdksDt‚|djdksVt‚dS) NฺhuZbnzLorem ipsum: 1984.r้r้Z1984)Zlang_rrr)r ฺ exceptionsrr r r r ฺtest_tokenizer_handles_digits%srz google.comz python.orgzspacy.ioz explosion.aizhttp://www.google.comcCs||ƒ}t|ƒdkst‚dS)Nr)rr)r rr r r r ฺtest_tokenizer_keep_urls0srz NASDAQ:GOOGcCs||ƒ}t|ƒdkst‚dS)Nr)rr)r rr r r r ฺtest_tokenizer_colons9srzhello123@example.comzhi+there@gmail.itzmatt@explosion.aicCs||ƒ}t|ƒdkst‚dS)Nr)rr)r rr r r r ฺtest_tokenizer_keeps_email?srcCs d}||ƒ}t|ƒdkst‚dS)NaำLorem ipsum dolor sit amet, consectetur adipiscing elit Cras egestas orci non porttitor maximus. Maecenas quis odio id dolor rhoncus dignissim. Curabitur sed velit at orci ultrices sagittis. Nulla commodo euismod arcu eget vulputate. Phasellus tincidunt, augue quis porta finibus, massa sapien consectetur augue, non lacinia enim nibh eget ipsum. Vestibulum in bibendum mauris. "Nullam porta fringilla enim, a dictum orci consequat in." Mauris nec malesuada justo.r)rr)r rr r r r ฺ test_tokenizer_handles_long_textGsr ฺ file_namezsun.txtcCsLttƒj|}|jddd ก}t|ƒdks0t‚||ƒ}t|ƒdksHt‚dS)Nฺrฺutf8)ฺencodingr้d)rฺ__file__ฺparentฺopenฺreadrr)r r!ฺlocrr r r r ฺ$test_tokenizer_handle_text_from_fileUs r+cCs@d}d}||ƒ}||ƒ}|djdks*t‚|djdkss*        (