B `m@sdZddlmZddlmZddlmZddlZddlZddlZddl m m Z ddl m mZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZd ddddddddddddddddddd d!d"gZd#ZGd$ddeZ dAd%dZ!dBd&dZ"dCd'dZ#dDd(dZ$dEd)dZ%d*dZ&d+dZ'd,dZ(dFd-d.Z)dGd/d Z*d0d Z+ej, dfd1dZ-d2dZ.dHd3d4Z/dId6dZ0dJd8dZ1dKd9d!Z2d:dZ3d;dZ4dZ6d?d"Z7dLd@dZ8dS)Mz>Internal utility functions for implementing TransitionKernels.)absolute_import)division)print_functionN)distribution_util) dtype_util) prefer_static)tensorshape_util)value_and_gradient)control_flow_util) deprecationchoose"enable_store_parameters_in_resultsleft_justified_expand_dims_likeleft_justified_expand_dims_toleft_justified_broadcast_likeleft_justified_broadcast_toindex_remapping_gather is_list_likeis_namedtuple_likemake_innermost_gettermake_innermost_setter make_namemaybe_call_fn_and_gradsprepare_state_partsPrettyNamedTupleMixinsafe_sumSEED_CTOR_ARG_DEPRECATION_MSGset_docsmart_for_loop strip_seeds trace_scan)warn_if_parameters_are_not_simple_tensorszSeeding `tfp.mcmc.TransitionKernel` instances by constructor argument is deprecated. Use the `seed` argument to `tfp.mcmc.sample_chain` or directly on `one_step`. The legacy behavior is still supported and should be through 2020-09-20.c@seZdZdZdZddZdS)rz2Mixin adding a nicer `__repr__` for `namedtuple`s.cCs*dt|jddd|DS)Nz{}( {} )z, css*|]"\}}d|t|ddVqdS)z {}={} z N)formatreprreplace).0kvr"r"_/home/dcms/DCMS/lib/python3.7/site-packages/tensorflow_probability/python/mcmc/internal/util.py Nsz1PrettyNamedTupleMixin.__repr__..)r$type__name__join_asdictitems)selfr"r"r*__repr__Ks zPrettyNamedTupleMixin.__repr__N)r- __module__ __qualname____doc__ __slots__r2r"r"r"r*rGsc Cs*t|p dt|t|SQRXdS)z5Right pads `x` with `rank(reference) - rank(x)` ones.rN)tf name_scoperrrank)x referencenamer"r"r*rRsc Csrt|p dZtj|tjd}t|t|d}tjt|tj |gtjdgdd}t ||SQRXdS)z*Right pads `x` with `rank - rank(x)` ones.r)dtyper)shaper=)axisN) r7r8convert_to_tensorint32rmaximumr9concatr>ZonesZreshape)r:r9r<Z expand_ndimsZ expand_shaper"r"r*rXsc Cs*t|p dt|t|SQRXdS)zABroadcasts `x` to shape of reference, in a left-justified manner.rN)r7r8rrr>)r:r;r<r"r"r*rdsc Cs2t|p dtt|t||SQRXdS)z4Broadcasts `x` to shape, in a left-justified manner.rN)r7r8 broadcast_torrsize)r:r>r<r"r"r*rjscs2t|}|r|n|g}fdd|D}||fS)zHCalls c2t on each element or the entirety if not iterable; returns list.csg|]}tj|dqS))r=r<)r7r@)r'r:)r=r<r"r* vsz'prepare_state_parts..)r)Zstate_or_state_partr=r< is_multipartZ state_partsr")r=r<r*rqs  cCst|ttfS)z4Helper which returns `True` if input is `list`-like.) isinstancetuplelist)r:r"r"r*r{scCs8yx|jD]}t||}q WdStk r2dSXdS)zFHelper which returns `True` if input is `collections.namedtuple`-like.TFN)_fieldsgetattrAttributeError)r:fn_r"r"r*rs  cCs(|dk r |n|}|dk r$|d|7}|S)z:Helper which makes a `str` name; useful for tf.name_scope.NrOr")Z super_nameZdefault_super_nameZsub_namer<r"r"r*rs c sTfddtpd0t|s.||Sfddt||DSQRXdS)zHHelper to `choose` which expand_dims `is_accepted` and applies tf.where.csR||kr |St|dd}|dk r:|ddddd}tjt||||dS) zWraps `tf.where`.r<N/:r)r<)rL rpartitionrsplitr7wherer)proposedcurrentr<) is_acceptedr"r*_wheres  z!_choose_base_case.._wherer cs2g|]*\}}t|r$t||dn||qS))r<)rr )r'pc)rZrYr<r"r*rFsz%_choose_base_case..N)r7r8rzip)rYrWrXr<r")rZrYr<r*_choose_base_cases   r^c Cst|p dt|s(t||||dSt|t|sRtdt|jt|ji}x.|j D]$}t |t ||t |||d||<q^Wt|f|SQRXdS)z=Helper which expand_dims `is_accepted` then applies tf.where.r )r<zCType of `proposed` ({}) must be identical to type of `current` ({})N) r7r8rr^rHr, TypeErrorr$r-rKr rL)rYrWrXr<r0rNr"r"r*r s cCs,t|s |St|fdd|DS)NcSs&i|]\}}|dkrt|ng|qS)seed)r)r'rNZfvr"r"r* szstrip_seeds..)rr,r/r0)objr"r"r*rsc Cs|t|p ddt|s td|s,td|dj}t|}ttj ||tj ||j d}t |||SQRXdS)aElementwise adds list members, replacing non-finite results with alt_value. Typically the `alt_value` is chosen so the `MetropolisHastings` `TransitionKernel` always rejects the proposal. Args: x: Python `list` of `Tensors` to elementwise add. alt_value: Python scalar used to replace any elementwise sums which would otherwise be non-finite. name: Python `str` name prefixed to Ops created by this function. Default value: `None` (i.e., "safe_sum"). Returns: safe_sum: `Tensor` representing the elementwise sum of list of `Tensor`s `x` or `alt_value` where sums are non-finite. Raises: TypeError: if `x` is not list-like. ValueError: if `x` is empty. rzExpected list input.zInput should not be empty.r)r=N)r7r8rr_ ValueErrorr>Zadd_nrVmath is_finiteZconstantr=r set_shape)r:Z alt_valuer<Zin_shaper"r"r*rs  " csfdd}|S)z7Decorator to programmatically set a function docstring.cs |_|S)N)r5)func)valuer"r*_docszset_doc.._docr")rhrir")rhr*rs c st|p ddd}tr(tng|d|dkrf}|dkrftrfddD||d}|dk r||d }||fSt|rt|tkr·fd d fd dtt|D}nt\}}||fSQRXdS) z$Helper to `maybe_call_fn_and_grads`.Zvalue_and_gradientscs.fddt|r&fdd|DS|S)Ncs|dkr dStj|dS)N)r<)r7r@)x_)r<r"r*szB_value_and_gradients.._convert_to_tensor..csg|] }|qSr"r")r'rj)cttr"r*rFszD_value_and_gradients.._convert_to_tensor..)r)r:r<r")rlr<r*_convert_to_tensors z0_value_and_gradients.._convert_to_tensorZfn_argNcSsg|] }d|qS)rr")r'r:r"r"r*rFsz(_value_and_gradients..Z fn_resultZfn_gradcsfddS)z6Needed to prevent `cell-var-from-loop` pylint warning.cs&d|gddS)NrSr")r:)rN fn_arg_listir"r*rkz8_value_and_gradients..fn_slice..r")ro)rNrn)ror*fn_slicesz&_value_and_gradients..fn_slicecs"g|]}t||dqS)rS)tfp_math_value_and_gradients)r'ro)rnrqr"r*rFs)r7r8rrJexecuting_eagerlylenrangerr)rNrnresultgradsr<rmrOr")rNrnrqr*_value_and_gradientss&    rxTc Cst|p dt|r t|n|g}t||||\}}tddt|rL|n|gDs`tdt|t|krxtd|rt dd|Drtd ||||fSQRXdS) zCCalls `fn` and computes the gradient of the result wrt `args_list`.rcss|]}t|jVqdS)N)rZ is_floatingr=)r'rr"r"r*r+&sz*maybe_call_fn_and_grads..z8Function result must be a `Tensor` with `float` `dtype`.z>Function args must be in one-to-one correspondence with grads.css|]}|dkVqdS)Nr")r'gr"r"r*r+-sz:Encountered `None` gradient. fn_arg_list: {} grads: {}N) r7r8rrJrxallr_rtrcanyr$)rNrnrvrwZcheck_non_none_gradsr<r"r"r*rs  c st|p dt}|dks8ts8ttr~tjtj dtj fddfddt dg||dd dS|}xt |D] }|}qW|SQRXdS) aConstruct a for loop, preferring a python loop if `n` is staticaly known. Given `loop_num_iter` and `body_fn`, return an op corresponding to executing `body_fn` `loop_num_iter` times, feeding previous outputs of `body_fn` into the next iteration. If `loop_num_iter` is statically known, the op is constructed via python for loop, and otherwise a `tf.while_loop` is used. Args: loop_num_iter: `Integer` `Tensor` representing the number of loop iterations. body_fn: Callable to be executed `loop_num_iter` times. initial_loop_vars: Listlike object of `Tensors` to be passed in to `body_fn`'s first execution. parallel_iterations: The number of iterations allowed to run in parallel. It must be a positive integer. See `tf.while_loop` for more details. Default value: `10`. name: Python `str` name prefixed to Ops created by this function. Default value: `None` (i.e., "smart_for_loop"). Returns: result: `Tensor` representing applying `body_fn` iteratively `n` times. rN)r=cs|kS)Nr")roargs) loop_num_iterr"r*rkVrpz smart_for_loop..cs|dgt|S)NrS)rJ)ror~)body_fnr"r*rkWrpr)condbody loop_varsparallel_iterationsrS) r7r8get_static_valuersr ZGraphOrParentsInXlaContexttf1Zget_default_graphcastrA while_loopnpru)rrZinitial_loop_varsrr<Zloop_num_iter_rvrOr")rrr*r4s       c st|p djttN}|jdkrDtsD|ddtj dd|}tj |dd}t |tj |j|jddd |d \dkrtn|rd |tj fd d|} fd dfdd} tjfdd| d|d| f|d\} } } } tj dd| } trHdnfdd}tj || } | | fSQRXWdQRXdS)aA simplified version of `tf.scan` that has configurable tracing. This function repeatedly calls `loop_fn(state, elem)`, where `state` is the `initial_state` during the first iteration, and the return value of `loop_fn` for every iteration thereafter. `elem` is a slice of `elements` along the first dimension, accessed in order. Additionally, it calls `trace_fn` on the return value of `loop_fn`. The `Tensor`s in return values of `trace_fn` are stacked and returned from this function, such that the first dimension of those `Tensor`s matches the size of `elems`. Args: loop_fn: A callable that takes in a `Tensor` or a nested collection of `Tensor`s with the same structure as `initial_state`, a slice of `elems` and returns the same structure as `initial_state`. initial_state: A `Tensor` or a nested collection of `Tensor`s passed to `loop_fn` in the first iteration. elems: A `Tensor` that is split along the first dimension and each element of which is passed to `loop_fn`. trace_fn: A callable that takes in the return value of `loop_fn` and returns a `Tensor` or a nested collection of `Tensor`s. trace_criterion_fn: Optional callable that takes in the return value of `loop_fn` and returns a boolean `Tensor` indicating whether to trace it. If `None`, all steps are traced. Default value: `None`. static_trace_allocation_size: Optional Python `int` size of trace to allocate statically. This should be an upper bound on the number of steps traced and is used only when the length cannot be statically inferred (for example, if a `trace_criterion_fn` is specified). It is primarily intended for contexts where static shapes are required, such as in XLA-compiled code. Default value: `None`. parallel_iterations: Passed to the internal `tf.while_loop`. name: Name scope used in this function. Default: 'trace_scan'. Returns: final_state: The final return value of `loop_fn`. trace: The same structure as the return value of `trace_fn`, but with each `Tensor` being a stack of the corresponding `Tensors` in the return value of `trace_fn` for each slice of `elems`. r NcSs|jS)N)Zdevice)opr"r"r*rkrpztrace_scan..cSstj|ddS)N initial_state)r<)r7r@)r:r"r"r*rkrpelems)r<rS)rE element_shape)TrFcstj|j|jdS)N)rE dynamic_sizer)r7 TensorArrayr=r>)r:)r initial_sizer"r*rkscstjfdd||S)Ncs ||S)N)write)tar:)num_steps_tracedr"r*rkrpz4trace_scan..trace_one_step..)r7nest map_structure)r trace_arraysstate)trace_fn)rr*trace_one_steps z"trace_scan..trace_one_stepcsZ|}|tr$ndfddfdd\|dfS)NTcsdfS)NrSr"r")rrrrr"r*rks z+trace_scan.._body..csfS)Nr"r")rrr"r*rkrprS)readrr)rorrrelem) elems_arrayloop_fntrace_criterion_fnr)rrrr*_bodys  ztrace_scan.._bodycs|kS)Nr")rorO)lengthr"r*rkrpr)rrrrcSs|S)N)stack)r:r"r"r*rkrpcs t||jdd|S)NrS)rrfZ concatenater>)r:) static_lengthr"r*_merge_static_lengthsz(trace_scan.._merge_static_length)r7r8rZvariable_scopeZget_variable_scopeZcaching_devicersZset_caching_devicerrr@rZsize0rr=r>ZunstackZ is_tensorrZ TensorShape)rrrrrZstatic_trace_allocation_sizerr<vsrrrOZ final_stateZ stacked_tracerr") rrrrrrrrrr*r as@0          cstfdd}|S)aWraps a setter so it applies to the inner-most results in `kernel_results`. The wrapped setter unwraps `kernel_results` and applies `setter` to the first results without an `inner_results` attribute. Args: setter: A callable that takes the kernel results as well as some `*args` and `**kwargs` and returns a modified copy of those kernel results. Returns: new_setter: A wrapped `setter`. csVg}xt|dr"|||j}qW|f||}xt|D]}|j|d}q>W|S)zWrapped setter. inner_results)r)hasattrappendrreversed_replace)kernel_resultsr~kwargs results_stackZnew_kernel_resultsZ outer_results)setterr"r* _new_setters    z*make_innermost_setter.._new_setter) functoolswraps)rrr")rr*rscstfdd}|S)anWraps a getter so it applies to the inner-most results in `kernel_results`. The wrapped getter unwraps `kernel_results` and returns the return value of `getter` called with the first results without an `inner_results` attribute. Args: getter: A callable that takes Kernel results and returns some value. Returns: new_getter: A wrapped `getter`. cs4g}xt|dr"|||j}qW|f||S)zWrapped getter.r)rrr)rr~rr)getterr"r* _new_getters    z*make_innermost_getter.._new_getter)rr)rrr")rr*rs  cCsvg}x,t|dr0d|jkr0|||jd}qWdd}t|drN||i}x"t|D]}||d|i}|}qXW|S)aEnables the `store_parameters_in_results` parameter in a chain of kernels. This is a temporary utility for use during the transition period of the parameter storage methods. Args: kernel: A TransitionKernel. Returns: kernel: The same kernel, but recreated with `store_parameters_in_results` recursively set to `True` in its parameters and its inner kernels (as appropriate). parametersZ inner_kernelc SsF|j}||d|kr$d|d<tt|f|SQRXdS)NZstore_parameters_in_resultsT)rcopyupdater Zsilencer,)kernelrZnew_parametersr"r"r*_recreate_kernel#s    z._recreate_kernel)rrrr)rZ kernel_stackrZ outer_kernelr"r"r*r s   cCs8t|rtdd|DSt|tjp6t|jtjkS)NcSsg|] }t|qSr")_is_tensor_like)r'r[r"r"r*rF7sz#_is_tensor_like..) rr{rHr7ZTensorrarrayr=object)paramr"r"r*r5srcCs2x,|D] \}}t|s td|q WdS)NaY`{}` is not a `tf.Tensor`, Python number, or Numpy array. If this parameter is mutable (e.g., a `tf.Variable`), then the behavior implied by `store_parameters_in_results` will silently change on 2019-08-01. Please consult the docstring for `store_parameters_in_results` details and use `store_parameters_in_results=True` to silence this warning.)r0rwarningswarnr$) params_dict param_namerr"r"r*r!;s c Cst|tj|dd}tj|dd}t|j}t|j}ttj|tjd}ttj|tjd}|dkr~td|dkrtd|dkrtd||krtd |||d krtd ||dk r|d krtd ||dk r||||krtd ||||t j ||dd}|||}t j |||d d} t | } t j | d|d t |||d | |dgdd} t || d|} tj| | |d |d d} t j | |d |dSQRXdS)adGather values from `axis` of `params` using `indices_axis` of `indices`. The shape of `indices` must broadcast to that of `params` when their `indices_axis` and `axis` (respectively) are aligned: ```python # params.shape: [p[0], ..., ..., p[axis], ..., ..., p[rank(params)] - 1]) # indices.shape: [i[0], ..., i[indices_axis], ..., i[rank(indices)] - 1]) ``` In particular, `params` must have at least as many leading dimensions as `indices` (`axis >= indices_axis`), and at least as many trailing dimensions (`rank(params) - axis >= rank(indices) - indices_axis`). The `result` has the same shape as `params`, except that the dimension of size `p[axis]` is replaced by one of size `i[indices_axis]`: ```python # result.shape: [p[0], ..., ..., i[indices_axis], ..., ..., p[rank(params) - 1]] ``` In the case where `rank(params) == 5`, `rank(indices) == 3`, `axis = 2`, and `indices_axis = 1`, the result is given by ```python # alignment is: v axis # params.shape == [p[0], p[1], p[2], p[3], p[4]] # indices.shape == [i[0], i[1], i[2]] # ^ indices_axis result[i, j, k, l, m] = params[i, j, indices[j, k, l], l, m] ``` Args: params: `N-D` `Tensor` (`N > 0`) from which to gather values. Number of dimensions must be known statically. indices: `Tensor` with values in `{0, ..., params.shape[axis] - 1}`, whose shape broadcasts to that of `params` as described above. axis: Python `int` axis of `params` from which to gather. indices_axis: Python `int` axis of `indices` to align with the `axis` over which `params` is gathered. name: String name for scoping created ops. Returns: `Tensor` composed of elements of `params`. Raises: ValueError: If shape/rank requirements are not met. params)r<indices)r=NzoRank of `params`, must be known statically. This is due to tf.gather not accepting a `Tensor` for `batch_dims`.zd`axis` must be known statically. This is due to tf.gather not accepting a `Tensor` for `batch_dims`.zl`indices_axis` must be known statically. This is due to tf.gather not accepting a `Tensor` for `batch_dims`.z3`indices_axis` should be <= `axis`, but was {} > {}rSz*Rank of params should be `> 0`, but was {}z+Rank of indices should be `> 0`, but was {}za`rank(params) - axis` ({} - {}) must be >= `rank(indices) - indices_axis` ({} - {}), but was not.)Z source_idxZdest_idxr)r?)Z batch_dimsr?)r7r8r@rr9r>rint64rcr$ dist_utilZmove_dimensionrrCrDZgather)rrr?Z indices_axisr<Z params_ndimsZ indices_ndimsZtransposed_indicesZbroadcast_indices_ndimsZtransposed_paramsZtransposed_params_shapeZ result_shapeZbroadcast_indicesZresult_tr"r"r*rHsn8          )N)N)N)N)NN)N)N)NNN)NNTN)r}N)NNr}N)rrr)9r5 __future__rrrrrnumpyrZtensorflow.compat.v1compatZv1rZtensorflow.compat.v2Zv2r7Z&tensorflow_probability.python.internalrrrrrZ+tensorflow_probability.python.math.gradientr rrZtensorflow.python.opsr Ztensorflow.python.utilr __all__rrrrrrrrrrrr^r rinfrrrxrrr rrr rr!rr"r"r"r*s                 ! .  0 n %