B )`k2@s2dZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZdd lmZd Zydd lmZdd lmZWnek rd ZYnXe jZe dd ddZdZdZdZdZdZdZ dZ!ddZ"ddZ#ddZ$ddZ%d d!Z&d"d#Z'Gd$d%d%e(Z)dS)&zCloud TPU Client.)absolute_import)division)print_functionN)flags)futures)request) HTTPErrorT) discovery)clientFruntime_oom_exitz,Exit the script when the TPU runtime is OOM.ZKUBE_GOOGLE_CLOUD_TPU_ENDPOINTS,ZTPU_NAMEZTPU_API_DISCOVERY_URLzhttp://metadata.google.internalZ8470Zzhttp://{}:8475/requestversioncCs tjS)zA wrapper function around datetime.datetime.utcnow. This function is created for unit testing purpose. It's not easy to do StubOutWithMock with datetime.datetime package. Returns: datetime.datetime )datetimeutcnowrrR/home/dcms/DCMS/lib/python3.7/site-packages/tensorflow/python/tpu/client/client.py_utcnow7s rcCs tjtS)N)osenvironget#_DISCOVERY_SERVICE_URL_ENV_VARIABLErrrr_environment_discovery_urlCsrcCs0tjdt|fddid}t|}t|S)Nz%s/computeMetadata/v1/%szMetadata-FlavorZGoogle)headers)rRequest_GCE_METADATA_ENDPOINTurlopen_as_textread)pathreqresprrr_request_compute_metadataGs    r!ccsjxd|dD]V}d}||r,||d}|d}|d}t}t|dkrV|d}||dVq WdS)z'Yields a dict with ip address and port.r zgrpc://:r) ipAddressportN)split startswith_DEFAULT_ENDPOINT_PORTlen)Z endpointsZendpointZ grpc_prefixparts ip_addressr%rrr%_environment_var_to_network_endpointsOs   r,cCs2|r|Sx$ttgD]}|tjkrtj|SqWdS)N)_GKE_ENV_VARIABLE_DEFAULT_ENV_VARIABLErr)tpuerrr _get_tpu_name`s  r1cCst|tr|dS|S)Nzutf-8) isinstancebytesdecode)srrrrjs  rc@seZdZdZd1ddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd2d,d-Zd3d/d0ZdS)4ClientzClient for working with the Cloud TPU API. This client is intended to be used for resolving tpu name to ip addresses. It's recommended to use this library as a contextlib to utilize all functionality. NdefaultcCst|tr2|stdt|dkr*td|d}t|}|dkrJtdt||_|jd |_ ||_ d|_ d|_ d|_ d|_|j r|dkr||_ |rt||_ n td|_ |rt||_ ntd }|d d |_ tp||_dS) Nz#At least one TPU must be specified.r"z>Using multiple TPUs in a single session is not yet implementedrz(Please provide a TPU Name to connect to.zgrpc://r7zproject/project-idz instance/zone/)r2list ValueErrorr)NotImplementedErrorr1r_tpur'_use_api_service _credentials_project_zone_discovery_urlr!r&r)selfr/zoneproject credentialsserviceZ discovery_urlZ zone_pathrrr__init__ys8      zClient.__init__cCsd|S)z&Return the structured Symptom message.z Symptom: r)rDmsgrrr _symptom_msgszClient._symptom_msgcCs|}|sdSxtt|D]h}|ddkr,q|ddd}tj|d}t|}|tjtdkrt | d |j d SqWdS) z)Check if a runtime OOM event is reported.FZ symptomTypeZ OUT_OF_MEMORYZ createTime.rz%Y-%m-%dT%H:%M:%S)secondsza recent runtime OOM has occured ~{} seconds ago. The model script will terminate automatically. To prevent future OOM events, please consider reducing the model size. To disable this behavior, set flag --runtime_oom_exit=false when starting the script.T) symptomsreversedr&rstrptimer timedelta_OOM_EVENT_COOL_TIME_SECloggingwarningrKformatrM)rDrNZsymptomZoom_datetime_strZ oom_datetimeZ time_diffrrr _oom_events    zClient._oom_eventcCsj|jr |jStstd|j}|dks.|dkr8tj}|jrTtj dd||jddStj dd|ddSdS) aCreates a new Cloud TPU API object. This works around an issue where the underlying HTTP connection sometimes times out when the script has been running for too long. Other methods in this object call this method to get a new API object whenever they need to communicate with the Cloud API. Raises: RuntimeError: If the dependent Python packages are missing. Returns: A Google Cloud TPU API object. z_Missing runtime dependency on the Google API client. Run `pip install cloud-tpu-client` to fix.Nr7r/Zv1F)rGZdiscoveryServiceUrlcache_discovery)rGrW) r?_GOOGLE_API_CLIENT_INSTALLED RuntimeErrorr@r ZGoogleCredentialsZget_application_defaultrCr build)rDrGrrr _tpu_services  zClient._tpu_servicecCsd|j|j|jfS)z)Returns the full Cloud name for this TPU.z!projects/%s/locations/%s/nodes/%s)rArBr=)rDrrr _full_nameszClient._full_namec Csf|}y$|j|d}|Stk r`}ztd|j |fWdd}~XYnXdS)z:Returns the TPU metadata object from the TPU Get API call.)namezCould not lookup TPU metadata from name '%s'. Please doublecheck the tpu argument in the TPUClusterResolver constructor. Exception: %sN) r[Zprojects locationsZnodesrr\execute Exceptionr;r=)rDrHrr0rrr_fetch_cloud_tpu_metadatasz Client._fetch_cloud_tpu_metadatacCs|jr|}||SdS)N)r>rbr)rDkeymetadatarrr_get_tpu_propertys zClient._get_tpu_propertycCs d|_dS)NT)_open)rDrrr __enter__szClient.__enter__cCs ~~~dS)Nr)rDtypevalue tracebackrrr__exit__szClient.__exit__cCs.|}|r|dkrdStjr*|r*dSdS)zReturns true if the TPU is in a state where training should eventually resume. If false the TPU is in a unrecoverable state and should be recreated. )Z TERMINATEDZ PREEMPTEDFT)stateFLAGSr rV)rDrlrrr recoverables  zClient.recoverablecCs |dS)z%Return Cloud TPU Symptoms of the TPU.rN)re)rDrrrrN szClient.symptomscCs |dS)zReturn state of the TPU.rl)re)rDrrrrlsz Client.statecCs |dS)zReturn health of the TPU.health)re)rDrrrrosz Client.healthc Cs|jst|dd}y,t|}t|}t| }| dSt k r}z|j }|dkrldS|Wdd}~XYnX| dS)z"Return runtime version of the TPU.rr$ZcurrentVersioniNZtensorflowVersion)r>_VERSION_SWITCHER_ENDPOINTrUnetwork_endpointsrrrjsonloadsrrrcodere)rDurlrr Zversion_detailsr0 status_coderrrruntime_versions   zClient.runtime_versioncCs |dS)z#Return accelerator type of the TPU.ZacceleratorType)re)rDrrraccelerator_type,szClient.accelerator_typecCs|jS)zPReturn if the Cloud TPU API is available, if not certain features will not work.)r>)rDrrr api_available0szClient.api_availablecCs|jS)zFReturn the name of the tpu, or the ip address if name is not provided.)r=)rDrrrr]4sz Client.namecCstdS)zNReturn the local ip address of the Google Cloud VM the workload is running on.z instance/network-interfaces/0/ip)r!)rDrrr get_local_ip8szClient.get_local_ipcCsj|jstt|jS|}|ddkrBtd|j|dfd|krR|dS|d|ddgSdS) zReturn a list of tpu endpoints.rlZREADYz&TPU "%s" is not yet ready; state: "%s"ZnetworkEndpointsr$r%)r$r%N)r>r:r,r=rbrrY)rDresponserrrrq<szClient.network_endpointscCsxt|}xV|dkrbtd|||t||krVtd|t|qWtd|dS)a?Wait for TPU to become healthy or raise error if timeout reached. Args: timeout_s (int): The timeout in seconds for waiting TPU to become healthy. interval (int): The interval in seconds to poll the TPU for health. Raises: RuntimeError: If the TPU doesn't become healthy by the timeout. ZHEALTHYzFWaiting for TPU "%s" with state "%s" and health "%s" to become healthyz0Timed out waiting for TPU "%s" to become healthyzTPU "%s" is healthy.N)timerorSrTr]rlrYsleep)rDZ timeout_sintervaltimeoutrrrwait_for_healthyJs zClient.wait_for_healthyalwaysc s\fdd}|}tjt|d,}|||}x|D]}|r:|q:WWdQRXdS)aConfigure TPU software version. Args: version (string): Version of software to configure the TPU with. restart_type (string): Restart behaviour when switching versions, defaults to always restart. Options are 'always', 'ifNeeded'. c s|d}td|}tj|dd}yt|WnNtk r}z0|j}|dkrftdntd|Wdd}~XYnXdS) zConfigure individual TPU worker. Args: worker: A dict with the field ipAddress where the configure request will be sent. r$z/{}?restartType={})dataizTensorflow version {} is not available on Cloud TPU, try a previous nightly version or refer to https://cloud.google.com/tpu/docs/release-notes for the latest official version.zFailed to configure worker {}N)rprUrrrrrtr`)Zworkerr+rurr0rv) restart_typeversionrrconfigure_workerks  z6Client.configure_tpu_version..configure_worker) max_workersN)rqrZThreadPoolExecutorr)mapresult)rDrrrworkersexecutorresultsrr)rrrconfigure_tpu_versionas   zClient.configure_tpu_version)NNNr7NN)r|r})r)__name__ __module__ __qualname____doc__rIrKrVr[r\rbrergrkrnrNrlrorwrxryr]rzrqrrrrrrr6ps6 &$   r6)*r __future__rrrrrrrSrr~ZabslrZ concurrentrZsix.moves.urllibrZsix.moves.urllib.errorrrXZgoogleapiclientr Z oauth2clientr ImportErrorrmZ DEFINE_boolr-Z_ENDPOINTS_SEPARATORr.rrr(rRrprrr!r,r1robjectr6rrrrsH