"Fossies" - the Fresh Open Source Software Archive

Member "tensorflow-2.6.0/tensorflow/lite/kernels/hashtable.md" (9 Aug 2021, 5311 Bytes) of package /linux/misc/tensorflow-2.6.0.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming markdown format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field.

How to use TF Lookup ops in TFLite

The objective of this file is to provide examples to demonstrate how to use TF Lookup ops in TFLite.

Supported Tensorflow Lookup ops in TFLite

Here is the supported status of TensorFlow Lookup ops.

TF Python lookup ops Supported status
tf.lookup.StaticHashTable Supported only with tensor initializers.

Supported mapping type: string → int64, int64 → string

tf.lookup.Hashtable Supported only with tensor initializers.

Supported mapping type: string → int64, int64 → string

tf.lookup.index_to_string_table_from_tensor Supported.
tf.lookup.index_table_from_tensor Supported natively when num_oov_buckets=0 and dtype=dtypes.string.

For the oov concept, you will need a Flex delegate.

tf.lookup.StaticVocabularyTable Supported but you will need a Flex delegate.

Use tf.index_table_from_tensor or tf.index_to_string_table_from_tensor instead if possible if you don’t want to use Flex delegate.

tf.lookup.experimental.DenseHashTable

tf.contrib.lookup.MutableHashTable

tf.contrib.lookup.MutableDenseHashTable

Not supported yet.
tf.lookup.IdTableWithHashBuckets Supported but you need a Flex delegate.

Python Sample code

Here, you can find the Python sample code:

int64_values = tf.constant([1, 2, 3], dtype=tf.int64)
string_values = tf.constant(['bar', 'foo', 'baz'], dtype=tf.string)

initializer = tf.lookup.KeyValueTensorInitializer(string_values, int64_values)
table = tf.lookup.StaticHashTable(initializer, 4)

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  input_string_tensor = tf.compat.v1.placeholder(tf.string, shape=[1])
  out_int64_tensor = table.lookup(input_string_tensor)
with open('/tmp/vocab.file', 'r') as f:
  words = f.read().splitlines()

string_values = tf.constant(words, dtype=tf.string)

initializer = tf.lookup.KeyValueTensorInitializer(string_values, int64_values)
table = tf.lookup.StaticHashTable(initializer, 4)

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  input_string_tensor = tf.placeholder(tf.string, shape=[1])
  out_int64_tensor = table.lookup(input_string_tensor)
UNK_ID = -1
vocab = tf.constant(["emerson", "lake", "palmer"])
vocab_table = tf.lookup.index_table_from_tensor(vocab, default_value=UNK_ID)

input_tensor = tf.compat.v1.placeholder(tf.string, shape=[5])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)
with open('/tmp/vocab.file', 'r') as f:
  words = f.read().splitlines()

UNK_ID = -1
vocab = tf.constant(words)
vocab_table = tf.lookup.index_table_from_tensor(vocab, default_value=UNK_ID)

input_tensor = tf.compat.v1.placeholder(tf.string, shape=[5])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)
UNK_WORD = "unknown"
vocab = tf.constant(["emerson", "lake", "palmer"])
vocab_table = tf.lookup.index_to_string_table_from_tensor(vocab, default_value=UNK_WORD)

input_tensor = tf.compat.v1.placeholder(tf.int64, shape=[1])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)
with open('/tmp/vocab.file', 'r') as f:
  words = f.read().splitlines()

UNK_WORD = "unknown"
vocab = tf.constant(words)
vocab_table = tf.lookup.index_to_string_table_from_tensor(vocab, default_value=UNK_WORD)

input_tensor = tf.compat.v1.placeholder(tf.int64, shape=[1])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)

How to Include Hashtable ops in your TFLite.

Currently, hashtable ops are now a part of the TFLite builtin op set. You don't need to add hashtable ops manually.