作用：将变量初始化为给定的常量，初始化一切所提供的值。
作用：将变量设置为全0；也可以简写为tf.Zeros()
作用：将变量设置为全1；可简写为tf.Ones()
作用：将变量初始化为满足正太分布的随机值，主要参数（正太分布的均值和标准差），用所给的均值和标准差初始化均匀分布。
@tf_export("initializers.truncated_normal", "truncated_normal_initializer")
class TruncatedNormal(Initializer):
"""Initializer that generates a truncated normal distribution.
These values are similar to values from a `random_normal_initializer`
except that values more than two standard deviations from the mean
are discarded and re-drawn. This is the recommended initializer for
neural network weights and filters.
Args:
mean: a python scalar or a scalar tensor. Mean of the random values
to generate. 一个python标量或一个标量张量。要生成的随机值的均值
stddev: a python scalar or a scalar tensor. Standard deviation of the
random values to generate.一个python标量或一个标量张量。要生成的随机值的标准偏差。
seed: A Python integer. Used to create random seeds. See
`tf.set_random_seed`
for behavior.一个Python整数。用于创建随机种子。查看 tf.set_random_seed 行为。
dtype: The data type. Only floating point types are supported.数据类型。只支持浮点类型。
"""
def __init__(self, mean=0.0, stddev=1.0, seed=None, dtype=dtypes.float32):
self.mean = mean
self.stddev = stddev
self.seed = seed
self.dtype = _assert_float_dtype(dtypes.as_dtype(dtype))
def __call__(self, shape, dtype=None, partition_info=None):
if dtype is None:
dtype = self.dtype
return random_ops.truncated_normal(
shape, self.mean, self.stddev, dtype, seed=self.seed)
def get_config(self):
return {
"mean": self.mean,
"stddev": self.stddev,
"seed": self.seed,
"dtype": self.dtype.name
}
举例：bert中初始化token_type_embeddings、embedding_table时，假设token_type_embeddings服从正态分布
def embedding_postprocessor(input_tensor,
use_token_type=False,
token_type_ids=None,
token_type_vocab_size=16,
token_type_embedding_name="token_type_embeddings",
use_position_embeddings=True,
position_embedding_name="position_embeddings",
initializer_range=0.02,
max_position_embeddings=512,
dropout_prob=0.1):
...
if use_token_type:
if token_type_ids is None:
raise ValueError("`token_type_ids` must be specified if"
"`use_token_type` is True.")
token_type_table = tf.get_variable(
name=token_type_embedding_name,
shape=[token_type_vocab_size, width],
initializer=create_initializer(initializer_range))
...
def create_initializer(initializer_range=0.02):
"""Creates a `truncated_normal_initializer` with the given range."""
return tf.truncated_normal_initializer(stddev=initializer_range)
作用：从a到b均匀初始化，将变量初始化为满足均匀分布的随机值，主要参数（最大值，最小值）。
作用：将变量初始化为满足均匀分布但不影响输出数量级的随机值
原文：https://www.cnblogs.com/nxf-rabbit75/p/12093343.html