Package io.aiven.commons.kafka.connector.source

Class NativeSourceData<K extends Comparable<K>>

java.lang.Object
io.aiven.commons.kafka.connector.source.NativeSourceData<K>
Type Parameters:
K - the key type for the native object.
All Implemented Interfaces:
AutoCloseable
public abstract class NativeSourceData<K extends Comparable<K>> extends Object implements AutoCloseable
A data source for native info from a source.

  • Constructor Details

    • NativeSourceData

      protected NativeSourceData(SourceCommonConfig sourceConfig, OffsetManager offsetManager)
      Constructor
      Parameters:
      sourceConfig - the source configuration for the native source.
      offsetManager - The offset manager from the kafka task.

  • Method Details

    • getSourceName

      public abstract String getSourceName()
      Gets the name for the source type.
      Returns:
      the common name for the data source. For example "AWS S3 Storage" or "AMQP Stream source".

    • getNativeItemIterator

      protected abstract Iterator<? extends AbstractSourceNativeInfo<K,?>> getNativeItemIterator(K startFrom)
      Get an iterator of Native objects from the underlying storage layer. The implementation must return the native objects in a repeatable order based on the key. In addition, the underlying storage must be able to start from a specific previously returned key.
      Parameters:
      startFrom - the native key to start from. May be null ot indicate start at the beginning.
      Returns:
      An iterator of native objects. May be empty but not null.

    • createOffsetManagerEntry

      public abstract OffsetManager.OffsetManagerEntry createOffsetManagerEntry(Map<String,Object> data)
      Creates an offset manager entry using the data in the map.
      Parameters:
      data - the data to create the offset manager from.
      Returns:
      a valid offset manager entry.

    • createOffsetManagerEntry

      protected abstract OffsetManager.OffsetManagerEntry createOffsetManagerEntry(Context context)
      Creates an offset manager entry from a context. The OffsetManagerEntry implementation must meet the contract: 
      
 K key = ...
 OffsetManagerEntry entry = createOffsetManagerEntry(context);
 OffsetManagerEntry emtry2 = createOffsetManagerEntry(entry1.getProperties());
 entry2.getProperties() is element for element equal to entry1.getProperties()

 also

 OffsetManagerKey k = entry.getManagerKey()
 OffsetManagerKey k2 = entry2.getManagerKey()
 k2.partitionMap() is element for element equal to k.partitionMap()
      Parameters:
      context - the context to create the offset manager from.
      Returns:
      a valid offset manager.

    • getNativeKeySerde

      protected abstract Optional<NativeSourceData.KeySerde<K>> getNativeKeySerde()
      Returns a KeySerde for the native keys. If native key serialization to String is not supported the method must return an empty Optional.
      Returns:
      an Optional KeySerde for this system.

    • getOffsetManagerKey

      protected abstract OffsetManager.OffsetManagerKey getOffsetManagerKey(K nativeKey)
      Creates an offset manager key for the native key. The OffsetManagerKey implementation must meet the contract: 
      
 K key = ...
 OffsetManagerKey k = getOffsetManagerKey(key);
 OffsetManagerKey k2 = getOffsetManagerKey(key);
 k2.partitionMap() is element for element equal to k.partitionMap()
      Parameters:
      nativeKey - THe native key to create an offset manager key for.
      Returns:
      An offset manager key.

    • close

      public void close() throws Exception
      Specified by:
      close in interface AutoCloseable
      Throws:
      Exception