gaborgsomogyi

Flink State Serialization and Deserialization: TypeInformation and Serializers

Table of Contents

1. Overview
2. TypeInformation: The Foundation
3. TypeSerializer: The Workhorse
4. The Complete Flow
5. Detailed Example Walkthrough
6. State Evolution and Migration
7. Architecture Insights

gaborgsomogyi

Tasks

• Tokens must be tracked per service, only updated tokens must be sent to task managers
• Dynamic token initialization from user code must be added

  package org.apache.flink.api.common.security;

  /**
   * Manager for obtaining delegation tokens dynamically at runtime.
   * This allows user code to request tokens for services not known at cluster startup.

gaborgsomogyi

Which version affected?

$ gpg --version
gpg (GnuPG) 2.4.8

What kind of issue is one facing with?

Simple hanging!

gaborgsomogyi

Dependency management in python

Python is not able to handle 2 different versions from the same package.

Dependency handling issue

When Pip <20.3 finds an unresolvable dependency (resolves a dependecy to 2+ versions) then would not halt the installation process. It’d successfully continue the process by installing the first matching dependency in the list of conflicts.

gaborgsomogyi

    @SuppressWarnings("unchecked")
    public static void setEnv(String key, String value) {
        try {
            Map<String, String> env = System.getenv();
            Class<?> cl = env.getClass();
            Field field = cl.getDeclaredField("m");
            field.setAccessible(true);
            Map<String, String> writableEnv = (Map<String, String>) field.get(env);
 writableEnv.put(key, value);

gaborgsomogyi

Flink version 1.16.0

Note1: types are java compatible
Note2: all strings are in UTF format

UTF format String

• short length
• byte [length]

gaborgsomogyi

Prerequisites:

• Deploy Flink k8s operator
• Install cmctl

Double check that the appropriate certificate is created

$ kubectl get cert       
NAME                          READY   SECRET                AGE
flink-operator-serving-cert   True    webhook-server-cert   91m

gaborgsomogyi

Prerequisites:

• Install any container system like docker/Virtualbox/hyperkit
• Install minikube
• mkdir -p ${HOME}/share
• chmod 777 ${HOME}/share
• minikube start --mount-string="${HOME}/share:/share" --mount
• kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=default:default
• kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Clone the necessary repos:

gaborgsomogyi

sun.net.www.protocol.http.NegotiateAuthentication caches per host (and no port information in the key) whether negotiation is supported or not (for instance SPNEGO). Worth to mention the cache is static (god knows why). This is so far so good but can lead to issues in unit tests. For example:

• Developer writes a positive and negative test case for an SPNEGO protected server
• Normally the order of test cases are not defined
• Negative test case executes where SPNEGO turned off and expect 401
• NegotiateAuthentication stores false in a map
• Negative test case pass
• Positive test case executes where SPNEGO turned on and expect 200
• NegotiateAuthentication gives base isSupported=false from cache

gaborgsomogyi

Modify krb5.conf

$ cat /etc/krb5.conf
[libdefaults]
	udp_preference_limit=1
	default_realm = VPC.CLOUDERA.COM
	forwardable = true

[realms]
	VPC.CLOUDERA.COM = {