B /`{F@sddlmZmZmZmZmZmZmZddlZ ddl m Z ddl m Z ddlmZddlmZddlmZddlmZmZmZmZdd lmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(Z Gd d d eZ)Gd d d eZ*dS))AnyDictListTextTupleOptional NamedTupleN)DOCS_URL_TRAINING_DATA_NLU) TrainingData)Message)Token) Component) TOKENS_NAMES ENTITY_ATTRIBUTE_CONFIDENCE_TYPE ENTITY_ATTRIBUTE_CONFIDENCE_ROLE!ENTITY_ATTRIBUTE_CONFIDENCE_GROUP)TEXTINTENTENTITIESENTITY_ATTRIBUTE_VALUEENTITY_ATTRIBUTE_STARTENTITY_ATTRIBUTE_END EXTRACTORENTITY_ATTRIBUTE_TYPEENTITY_ATTRIBUTE_GROUPENTITY_ATTRIBUTE_ROLE NO_ENTITY_TAGSPLIT_ENTITIES_BY_COMMA%SPLIT_ENTITIES_BY_COMMA_DEFAULT_VALUE*SINGLE_ENTITY_ALLOWED_INTERLEAVING_CHARSETc@sBeZdZUdZeed<eeefed<eeefed<eed<dS) EntityTagSpeczhsz>EntityExtractor.filter_irrelevant_entities..r()r:r;r()r;r)filter_irrelevant_entitiescs z*EntityExtractor.filter_irrelevant_entities)r/texttokensr,cCsdd|D}dd|D}|t|kr.cSsg|] }|jqSr()end)r<rBr(r(r)r=sszWInvalid entity {} in example '{}': entities must span whole tokens. Wrong entity start.zUInvalid entity {} in example '{}': entities must span whole tokens. Wrong entity end.)rformat ValueErrorrindex)r/r?r@offsetsZendsmessagerArCr(r(r) find_entityns  zEntityExtractor.find_entity)entity_examplesr,c Csg}x~|D]v}g}x6|tgD]&}|t}|r<||jkr ||q W|j}||t<|t|t||j |j |j dq W|S)zFilters out untrainable entity annotations. Creates a copy of entity_examples in which entities that have `extractor` set to something other than self.name (e.g. 'CRFEntityExtractor') are removed. )r?dataoutput_propertiestimefeatures) r3rrr-r1rLcopyr rrMrNrO)r.rKfilteredrIr+entZ extractorrLr(r(r)filter_trainable_entitiess"    z)EntityExtractor.filter_trainable_entitiesN)r?r@tagsr8 confidencesr,c Csddlmmm}g}t}t}t} d} xt|D]\} } t|t| } | tkrdt}| j } q8t|t | }| |}t|t | }| |}| |kp||k}| | r || ko|j| | ko|j| | k}|tkp|j| | k}|p|p|}| }| | } n|| kp|}| }|rNtt|| ||| | |}||ndt|| | || r| j |dt<|dk rt||| n(tt|| ||| | |}|||} |}| j } q8Wx&|D]}||t|t|t<qW|S)ajConvert predictions into entities. Args: text: The text message. tokens: Message tokens without CLS token. tags: Predicted tags. split_entities_config: config for handling splitting a list of entities confidences: Confidences of predicted tags. Returns: Entities. rN)Zrasa.nlu.utils.bilou_utilsZnlur6 bilou_utilsr enumerater* get_tag_forrrCrZtag_without_prefixrZbilou_prefix_from_tagZLASTZINSIDEZUNIT_create_new_entitylistkeysr1_check_is_single_entityr_update_confidence_valuesrr)r?r@rTr8rUrWr+Zlast_entity_tagZ last_role_tagZlast_group_taglast_token_endidxrBcurrent_entity_tagZcurrent_group_tagZcurrent_role_tagZgroup_or_role_changedZnew_bilou_tag_startsZnew_unigram_bilou_tag_startsZ new_tag_foundr/r(r(r)!convert_predictions_into_entitiess                  z1EntityExtractor.convert_predictions_into_entities)r+rUr`cCst|dt|t||dt<t|dkrPt|dt|t||dt<t|dkr~t|dt|t||dt<dS)NrV)minrrrrrr)r+rUr`r(r(r)r^/s     z)EntityExtractor._update_confidence_values)r?rBr_r8rac Csb|j|dkrdS|j|dk}|||j}ttdd|}|t}|||} |o`|o`| S)NrDTcSs|tkr dSdS)NTF)r)charr(r(r)`s z9EntityExtractor._check_is_single_entity..)rAallfilterrr3) r?rBr_r8raZtokens_within_rangeZinterleaving_textZ!tokens_separated_by_allowed_charsr9Zsplit_current_entity_typer(r(r)r]Cs  z'EntityExtractor._check_is_single_entity)rTr!r`r,cCs||kr|||StS)aGet the value of the given tag name from the list of tags. Args: tags: Mapping of tag name to list of tags; tag_name: The tag name of interest. idx: The index position of the tag. Returns: The tag value. )r)rTr!r`r(r(r)rYss  zEntityExtractor.get_tag_for) tag_names entity_tag group_tagrole_tagrBr`rUr,cCst|t|jt|ji}|dk r,|t||t<t|kr\|tkr\||t<|dk r\|t||t<t |kr|tkr||t <|dk r|t ||t <|S)apCreate a new entity. Args: tag_names: The tag names to include in the entity. entity_tag: The entity type value. group_tag: The entity group value. role_tag: The entity role value. token: The token. confidence: The confidence value. Returns: Created entity. N) rrrArrCrrrrrr)rirjrkrlrBr`rUr/r(r(r)rZs(    z"EntityExtractor._create_new_entity) training_datar,c Csx|jD]}dd|tD}dd|ttD}dd|ttD}x|D]\}}||ksn||krVdd|tD}dd|ttD}tjjjj d|td|t d |d |d t d PqVWqWd S)a2Check if entities are correctly annotated in the training data. If the start and end values of an entity do not match any start and end values of the respected token, we define an entity as misaligned and log a warning. Args: training_data: The training data. cSsg|]}|t|tfqSr()rr)r<r/r(r(r)r=szDEntityExtractor.check_correct_entity_annotations..cSsg|] }|jqSr()rA)r<tr(r(r)r=scSsg|] }|jqSr()rC)r<rnr(r(r)r=scSs"g|]}|t|t|tfqSr()rrr)r<r/r(r(r)r=scSsg|]}|j|j|jfqSr()rArCr?)r<rnr(r(r)r=sz)Misaligned entity annotation in message 'z' with intent 'z3'. Make sure the start and end values of entities (z3) in the training data match the token boundaries (z). Common causes: 1) entities include trailing whitespaces or punctuation 2) the tokenizer gives an unexpected result, due to languages such as Chinese that don't use whitespace for word separation)ZdocsN) rKr3rrrr5Zsharedr6ioZ raise_warningrr ) rmZexampleZentity_boundariesZtoken_start_positionsZtoken_end_positionsZ entity_startZ entity_endZ entities_reprZ tokens_reprr(r(r) check_correct_entity_annotationss  ( z0EntityExtractor.check_correct_entity_annotations)NN)N)r"r#r$r%rrrrr0r2boolr7 staticmethodr[setr>r rr'rJr rSrfloatrbr^r]rYrZr rpr(r(r(r)r*+s@  4, *"$(r*)+typingrrrrrrrZrasa.shared.utils.ior5Zrasa.shared.constantsr Z+rasa.shared.nlu.training_data.training_datar Z%rasa.shared.nlu.training_data.messager Zrasa.nlu.tokenizers.tokenizerr Zrasa.nlu.componentsr Zrasa.nlu.constantsrrrrZrasa.shared.nlu.constantsrrrrrrrrrrrrrrZrasa.utils.train_utilsr r*r(r(r(r)s$     @