kubernets---kubernetes-client的使用--java中提交argo工作流

云存储云计算 专栏收录该内容
85 篇文章 5 订阅

转载请注明出处:java中提交argo工作流

背景

argo是一个建立在k8s框架之上的工作流调度工具。 详见

工作流workflow任务调度工具argo

需求是通过api来进行argo的调度脚本提交,不通过shell的手动提交方式。

方案思路

argo的提交 通过argo sumbit命令如下:

argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml

一般在linux的shell环境中提交。

在argo的官网上和github中翻了很久没找到相关api和客户端可以调用触发的文档说明。
官网
github

然后转换下思路。

argo调度服务其实也是k8s服务的一种,通过k8s的命令也能进行提交如下:

kubectl create -f https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml

而k8s本身是有其他语言进行操作的客户端包的。

kubernetes客户端库

如果想要编写应用程序来使用 Kubernetes REST API,您并不需要自己实现 API 调用或者请求(或者响应)的类型,使用与您正在使用的编程语言对应的客户端库即可。

客户端库通常已经帮您处理了一些常见的任务,例如身份验证。如果 API 客户端运行在 Kubernetes 集群内,那么大多数的客户端库能够发现并使用 Kubernetes 服务账号来进行身份验证,或者通过 kubeconfig file 文件读取验证信息和 API Server 地址。

详情参考以下链接:
如何在多种编程语言中通过客户端库来使用 Kubernetes API

kubernetes官方版java客户端

kubernetes社区版维护的java客户端fabric8io

用例

我们使用的yaml文件进行测试实验如下:
hello-k8s.yaml

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
  namespace: default
spec:
  restartPolicy: OnFailure
  containers:
  - name: hello
    image: "ubuntu"
    command: ["/bin/echo","hello”,”world"]

hello-argo.yaml

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-parameters-
  namespace: default
spec:
  # invoke the whalesay template with
  # "hello world" as the argument
  # to the message parameter
  entrypoint: whalesay
  arguments:
    parameters:
    - name: message
      value: hello

  templates:
  - name: whalesay
    inputs:
      parameters:
      - name: message       #parameter declaration
    container:
      # run cowsay with that message input parameter as args
      image: docker/whalesay
      command: [echo, "{{inputs.parameters.message}}"]

argo中提交使用

argo submit hello-argo.yaml -p message="goodbye"

发现message参数会自动传入 输出为: goodbye

但是kubectl命令模式则无法传参实现自动替换

kubectl create -f  hello-argo.yaml

实际上需要对hello.yaml文件作物理替换如下:

sed -i "s/hello/goodbye/" hello-argo.yaml
kubectl create -f  hello-argo.yaml

所以 我们使用java来调用时,也需要读文件 替换文件的内容。

开发使用

引入maven包

<!--kubernetes-client 中包括了 对应版本的kubernetes-model依赖 -->
<dependency>
    <groupId>io.fabric8</groupId>
    <artifactId>kubernetes-client</artifactId>
    <version>4.1.1</version><!-- 对应k8s的1.10.6版本-->
</dependency>

初始化client

KubernetesClient client =new DefaultKubernetesClient();

这种方式会按以下优先级去加载配置:
1、系统属性
2、Environment variables
3、Kube配置文件
4、已经挂载的服务帐户令牌&证书
推荐使用系统属性
以下系统属性 & 环境变量 可以用于配置:

kubernetes.master/KUBERNETES_MASTER
kubernetes.api.version/KUBERNETES_API_VERSION
kubernetes.oapi.version/KUBERNETES_OAPI_VERSION
kubernetes.trust.certificates/KUBERNETES_TRUST_CERTIFICATES
kubernetes.certs.ca.file/KUBERNETES_CERTS_CA_FILE
kubernetes.certs.ca.data/KUBERNETES_CERTS_CA_DATA
kubernetes.certs.client.file/KUBERNETES_CERTS_CLIENT_FILE
kubernetes.certs.client.data/KUBERNETES_CERTS_CLIENT_DATA
kubernetes.certs.client.key.file/KUBERNETES_CERTS_CLIENT_KEY_FILE
kubernetes.certs.client.key.data/KUBERNETES_CERTS_CLIENT_KEY_DATA
kubernetes.certs.client.key.algo/KUBERNETES_CERTS_CLIENT_KEY_ALGO
kubernetes.certs.client.key.passphrase / KUBERNETES_CERTS_CLIENT_KEY_PASSPHRASE
kubernetes.auth.basic.username/KUBERNETES_AUTH_BASIC_USERNAME
kubernetes.auth.basic.password/KUBERNETES_AUTH_BASIC_PASSWORD
kubernetes.auth.tryKubeConfig/KUBERNETES_AUTH_TRYKUBECONFIG
kubernetes.auth.tryServiceAccount/KUBERNETES_AUTH_TRYSERVICEACCOUNT
kubernetes.auth.token/KUBERNETES_AUTH_TOKEN
kubernetes.watch.reconnectInterval/KUBERNETES_WATCH_RECONNECTINTERVAL
kubernetes.watch.reconnectLimit/KUBERNETES_WATCH_RECONNECTLIMIT
kubernetes.user.agent/KUBERNETES_USER_AGENT
kubernetes.tls.versions/KUBERNETES_TLS_VERSIONS
kubernetes.truststore.file/KUBERNETES_TRUSTSTORE_FILE
kubernetes.truststore.passphrase/KUBERNETES_TRUSTSTORE_PASSPHRASE
kubernetes.keystore.file/KUBERNETES_KEYSTORE_FILE
kubernetes.keystore.passphrase/KUBERNETES_KEYSTORE_PASSPHRASE

通过authtoken

   ConfigBuilder builder = new ConfigBuilder();
    if (authToken !=null ) {
      builder.withOauthToken(authToken);
    }
    Config config = builder.build();
    final KubernetesClient client = new DefaultKubernetesClient(config)

或者通过master url连接
先查询master url

[zzq@localhost ~]$ kubectl cluster-info
Kubernetes master is running at https://xxxx:8080

然后在java代码中

Config config = new ConfigBuilder().withMasterUrl("http://x.x.x.x:8080").build();
KubernetesClient client = new DefaultKubernetesClient(config);//使用默认的就足够了

在pod外部运行推荐通过MasterUrl的方式
在pod中运行操作会自动获取

			KubernetesClient client = new DefaultKubernetesClient();
				System.out.println("自动获取k8s配置");
				if(!"auto".equals(masterurl)) {
				Config config = new ConfigBuilder().withMasterUrl(masterurl).build();
				client = new DefaultKubernetesClient(config);
				System.out.println("自定义k8s配置:"+masterurl);
				}

关于在pod中访问k8s,参考
https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#accessing-the-api-from-a-pod

更多参数和client 初始化的源码参考
https://raw.githubusercontent.com/fabric8io/kubernetes-client/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java

获取资源操作类

client.namespaces()	
client.services()
client.pods()		
client.customResources()
client.storage()
client.network()

资源的CURD


创建:
Service service = client.services().inNamespace(namespace).create(service);

更新:
Namespace namespace = client.namespaces().withName(name).get();
 //update resources
client.namespaces().createOrReplace(namespace);

查询:
ServiceList services = client.services().inNamespace("default").list();
Service service = client.services().inNamespace("default").withName("myservice").get()
     
删除:
client.services().inNamespace("default").withName("myservice").delete();

DSL内联方式操作


client.services().inNamespace("default").withName("myservice").edit()
                     .editMetadata()
                       .addToLabels("another", "label")
                     .endMetadata()
                     .done();


从外部文件中加载资源

有些情况下,你希望从外部源读取资源,而不是使用客户端的DSL来定义资源。比如我们有一些比较复杂的yml,不希望重新构建一遍。 对于这些情况,客户端允许你从以下位置加载资源:
一个文件 (支持 java.io.File 和 java.lang.String)
url
输入流
一旦资源被加载,你就可以对它进行创建。

例如让我们从yml文件中读取一个Pod,并创建它:

Pod refreshed = client.load('/path/to/a/pod.yml').fromServer().get();
Boolean deleted = client.load('/workspace/pod.yml').delete();
LogWatch handle = client.load('/workspace/pod.yml').watchLog(System.out);

测试通过的完整示例代码

提交普通yaml

ArgoController.java

package com.biologic.api;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.ClassPathResource;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

import com.biologic.api.service.LogService;

import io.fabric8.kubernetes.api.model.DoneablePod;
import io.fabric8.kubernetes.api.model.HasMetadata;
import io.fabric8.kubernetes.api.model.Pod;
import io.fabric8.kubernetes.api.model.PodList;
import io.fabric8.kubernetes.client.Config;
import io.fabric8.kubernetes.client.ConfigBuilder;
import io.fabric8.kubernetes.client.DefaultKubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.dsl.NonNamespaceOperation;
import io.fabric8.kubernetes.client.dsl.PodResource;

@Controller
@RequestMapping(path = "/report/generation/argo")
public class ArgoController {

	public final static String SECRET_TOKEN_ARGO = "123";

	@Value("${env}")
	private String env;
	
	
	@Value("${kubernetes.master}")
	private String masterurl;

	@Autowired
	private LogService logService;

	@PostMapping(value = "/{chip}/quality-check")
	@ResponseBody
	public Object quality_check(@RequestParam(value = "token", required = true) String token,
			@PathVariable("chip") String chip) {
		if (!token.equals(SECRET_TOKEN_ARGO)) {
			return "unauthorized";
		}
		logService.printK8sNow("当前环境 " + env);
		
	    String namespace = "default";
	    if ("sit".equals(env)) {
	      namespace = "sit";
	    }

	    
	
			try {
				ClassPathResource yamlresource = new ClassPathResource("hello.yaml");
				InputStream input=yamlresource.getInputStream();
				StringBuffer sb = new StringBuffer();
				readToBuffer(sb, input);
				String fileContent = sb.toString();
				String fileContent2 = fileContent.replaceFirst("value: hello", "value: " + chip);
				System.out.println(fileContent2);
				InputStream stream = new ByteArrayInputStream(fileContent2.getBytes());
				Config config = new ConfigBuilder().withMasterUrl(masterurl).build();
				KubernetesClient client = new DefaultKubernetesClient(config);
				namespace = client.getNamespace();
				 List<HasMetadata> resources = client.load(stream).get();
			      if (resources.isEmpty()) {
			        System.err.println("No resources loaded from file: " +yamlresource.getPath());
			        return "No resources loaded from file: " +yamlresource.getPath();
			      }
			      HasMetadata resource = resources.get(0);
			      if (resource instanceof Pod){
			        Pod pod = (Pod) resource;
			        System.out.println("Creating pod in namespace " + pod.getMetadata().getNamespace());
			        NonNamespaceOperation<Pod, PodList, DoneablePod, PodResource<Pod, DoneablePod>> pods = client.pods().inNamespace(namespace);
			        Pod result = pods.create(pod);
			        System.out.println("Created pod " + result.getMetadata().getName());
			      } else {
			        System.err.println("Loaded resource is not a Pod! " + resource);
			      }
				return fileContent2;
			} catch (IOException e) {
				e.printStackTrace();
				System.out.println("argo未找到配置文件");
			}

		return "ok";
	}

	/**
	 * 将文本文件中的内容读入到buffer中
	 * 
	 * @param buffer
	 *            buffer
	 * @param filePath
	 *            文件路径
	 * @throws IOException
	 *             异常
	 * @author cn.outofmemory
	 * @date 2013-1-7
	 */
	private void readToBuffer(StringBuffer buffer, InputStream input) throws IOException {
		String line; // 用来保存每行读取的内容
		BufferedReader reader = new BufferedReader(new InputStreamReader(input));
		line = reader.readLine(); // 读取第一行
		while (line != null) { // 如果 line 为空说明读完了
			buffer.append(line); // 将读到的内容添加到 buffer 中
			buffer.append("\n"); // 添加换行符
			line = reader.readLine(); // 读取下一行
		}
		reader.close();
		input.close();
	}

}

触发api的方式为:

curl -d "token=123" http://localhost:9999/report/generation/argo/123/quality-check

成功提交创建pod如图所示

这里name为hello-world,下次再创建会报重名错误。

建议使用随机数替换一下后缀如下:
hello.yaml

name: hello-world-random

ArgoController

				String fileContent2 = fileContent.replaceFirst("-random", "- " + Math.random());

提交argo的yaml

argo类型的则要麻烦一些,因为defaultClient不支持argo类型的。
会报错No resource type found for:argoproj.io/v1alpha1

com.fasterxml.jackson.databind.JsonMappingException: No resource type found for:argoproj.io/v1alpha1#Workflow

我们需要进行自定义资源才能提交argo类型的yaml。

首先需要查看集群中已有的crd。
使用命令

kubectl  get crd
kubectl  get crd |grep argo

java中查询已有的crd代码如下:

	 try {
				      if (!client.supportsApiPath("/apis/apiextensions.k8s.io/v1beta1") && !client.supportsApiPath("/apis/apiextensions.k8s.io/v1")) {
				        System.out.println("WARNING this cluster does not support the API Group apiextensions.k8s.io");
				        return "fail";
				      }
				      CustomResourceDefinitionList list = client.customResourceDefinitions().list();
				      if (list == null) {
				        System.out.println("ERROR no list returned!");
				        return "fail";
				      }
				      List<CustomResourceDefinition> items = list.getItems();
				      for (CustomResourceDefinition item : items) {
				        System.out.println("CustomResourceDefinition " + item.getMetadata().getName() + " has version: " + item.getApiVersion());
				      }
				    } catch (KubernetesClientException e) {
				      System.out.println("Failed: " + e);
				      e.printStackTrace();
				    }

结果输出如下:

CustomResourceDefinition alertmanagers.monitoring.coreos.com has version: apiextensions/v1beta1
CustomResourceDefinition backups.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition backupstoragelocations.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition deletebackuprequests.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition downloadrequests.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition elasticsearchclusters.enterprises.upmc.com has version: apiextensions/v1beta1
CustomResourceDefinition podvolumebackups.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition podvolumerestores.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition prometheuses.monitoring.coreos.com has version: apiextensions/v1beta1
CustomResourceDefinition prometheusrules.monitoring.coreos.com has version: apiextensions/v1beta1
CustomResourceDefinition resticrepositories.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition restores.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition schedules.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition servicemonitors.monitoring.coreos.com has version: apiextensions/v1beta1
CustomResourceDefinition volumesnapshotlocations.ark.heptio.com has version: apiextensions/v1beta1
CustomResourceDefinition workflows.argoproj.io has version: apiextensions/v1beta1

找到argo的crd名称为workflows.argoproj.io

初始化几个类

ArgoCluster.java

package com.biologic.entity;

import java.util.ArrayList;
import java.util.Map;

import io.fabric8.kubernetes.client.CustomResource;

public class ArgoCluster extends CustomResource{
	private ArgoSpec spec;
	private Map<String,Object> status;


	  @Override
	  public String toString() {
	    return "Argo{" +
	        "apiVersion='" + getApiVersion() + '\'' +
	        ", metadata=" + getMetadata() +
	        ", spec=" + spec +
	        '}';
	  }

	  public ArgoSpec getSpec() {
	    return spec;
	  }

	  public void setSpec(ArgoSpec spec) {
	    this.spec = spec;
	  }

	public Map<String, Object> getStatus() {
		return status;
	}

	public void setStatus(Map<String, Object> status) {
		this.status = status;
	}


}

ArgoSpec.java

package com.biologic.entity;



import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;

import io.fabric8.kubernetes.api.model.KubernetesResource;


@JsonDeserialize(
	    using = JsonDeserializer.None.class
	)
public class ArgoSpec implements KubernetesResource {
	
	private Object entrypoint;
	  private Object arguments;
	  private Object templates;
	  private Object volumes;
	  
	public Object getEntrypoint() {
		return entrypoint;
	}
	public void setEntrypoint(Object entrypoint) {
		this.entrypoint = entrypoint;
	}
	public Object getArguments() {
		return arguments;
	}
	public void setArguments(Object arguments) {
		this.arguments = arguments;
	}
	public Object getTemplates() {
		return templates;
	}
	public void setTemplates(Object templates) {
		this.templates = templates;
	}
	public Object getVolumes() {
		return volumes;
	}
	public void setVolumes(Object volumes) {
		this.volumes = volumes;
	}

}

ArgoList.java

package com.biologic.entity;

import io.fabric8.kubernetes.client.CustomResourceList;

public class ArgoList extends CustomResourceList<ArgoCluster>{

}

DoneableArgo.java

package com.biologic.entity;

import io.fabric8.kubernetes.api.builder.Function;
import io.fabric8.kubernetes.client.CustomResourceDoneable;

public class DoneableArgo extends CustomResourceDoneable<ArgoCluster>{

	public DoneableArgo(ArgoCluster resource, Function<ArgoCluster, ArgoCluster> function) {
		super(resource, function);
	}

}

调用crd的关键代码

Config config = new ConfigBuilder().withMasterUrl(masterurl).build();
				KubernetesClient client = new DefaultKubernetesClient(config);
				CustomResourceDefinition argoCRD = null;
				
				 try {
				      if (!client.supportsApiPath("/apis/apiextensions.k8s.io/v1beta1") && !client.supportsApiPath("/apis/apiextensions.k8s.io/v1")) {
				        System.out.println("WARNING this cluster does not support the API Group apiextensions.k8s.io");
				        return "fail";
				      }
				      CustomResourceDefinitionList list = client.customResourceDefinitions().list();
				      if (list == null) {
				        System.out.println("ERROR no list returned!");
				        return "fail";
				      }
				      List<CustomResourceDefinition> items = list.getItems();
				      for (CustomResourceDefinition item : items) {
				        System.out.println("CustomResourceDefinition " + item.getMetadata().getName() + " has version: " + item.getApiVersion());
				        if (ARGO_CRD_NAME.equals(item.getMetadata().getName())) {
				        	argoCRD = item;
				          }
				      }
				    } catch (KubernetesClientException e) {
				      System.out.println("Failed: " + e);
				      e.printStackTrace();
				    }
				 
				 
				 if (argoCRD != null) {
				        System.out.println("Found CRD: " + argoCRD.getMetadata().getSelfLink());
				      } else {
				    	  return "fail";  
				      }

				 MixedOperation<ArgoCluster, ArgoList, DoneableArgo, Resource<ArgoCluster, DoneableArgo>> argoClient = client.customResources(argoCRD, ArgoCluster.class, ArgoList.class, DoneableArgo.class);

				
				
				CustomResourceList<ArgoCluster> argoList =argoClient.list();
			      List<ArgoCluster> items = argoList.getItems();
			      System.out.println("  found " + items.size() + " argo");
			      for (ArgoCluster item : items) {
			        System.out.println("    " + item);
			      }
			      
			      
				 ArgoCluster createArgo= argoClient.load(stream).get();
				 ArgoCluster finishpod=argoClient.create(createArgo);
				 String podname=finishpod.getMetadata().getName();
				  Pod pod = client.pods().inNamespace(namespace).withName(podname).get();
			      if (pod !=null){
			        System.out.println("Creating pod in namespace " + pod.getMetadata().getNamespace());
			        System.out.println("Created pod " + podname);
			      } else {
			        System.err.println("not found a Pod! " + podname);
			      }

完整代码

package com.biologic.api;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.ClassPathResource;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

import com.biologic.api.service.LogService;
import com.biologic.entity.ArgoCluster;
import com.biologic.entity.ArgoList;
import com.biologic.entity.DoneableArgo;

import io.fabric8.kubernetes.api.model.DoneablePod;
import io.fabric8.kubernetes.api.model.HasMetadata;
import io.fabric8.kubernetes.api.model.Pod;
import io.fabric8.kubernetes.api.model.PodList;
import io.fabric8.kubernetes.api.model.apiextensions.CustomResourceDefinition;
import io.fabric8.kubernetes.api.model.apiextensions.CustomResourceDefinitionList;
import io.fabric8.kubernetes.client.AppsAPIGroupClient;
import io.fabric8.kubernetes.client.AutoAdaptableKubernetesClient;
import io.fabric8.kubernetes.client.AutoscalingAPIGroupClient;
import io.fabric8.kubernetes.client.BatchAPIGroupClient;
import io.fabric8.kubernetes.client.Config;
import io.fabric8.kubernetes.client.ConfigBuilder;
import io.fabric8.kubernetes.client.CustomResourceList;
import io.fabric8.kubernetes.client.DefaultKubernetesClient;
import io.fabric8.kubernetes.client.ExtensionsAPIGroupClient;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientException;
import io.fabric8.kubernetes.client.dsl.MixedOperation;
import io.fabric8.kubernetes.client.dsl.NonNamespaceOperation;
import io.fabric8.kubernetes.client.dsl.PodResource;
import io.fabric8.kubernetes.client.dsl.Resource;

@Controller
@RequestMapping(path = "/report/generation/argo")
public class ArgoController {

	public final static String SECRET_TOKEN_ARGO = "uiuda";

	private static final String ARGO_CRD_NAME = "workflows.argoproj.io";

	@Value("${env}")
	private String env;
	
	
	@Value("${kubernetes.master}")
	private String masterurl;

	@Autowired
	private LogService logService;

	@PostMapping(value = "/{chip}/quality-check")
	@ResponseBody
	public Object quality_check(@RequestParam(value = "token", required = true) String token,
			@PathVariable("chip") String chip) {
		if (!token.equals(SECRET_TOKEN_ARGO)) {
			return "unauthorized";
		}
		logService.printK8sNow("当前环境 " + env);
		
	    String namespace = "default";
	    if ("sit".equals(env)) {
	      namespace = "sit";
	    }

	    
	
			try {
				ClassPathResource yamlresource = new ClassPathResource("hello.yaml");
				InputStream input=yamlresource.getInputStream();
				StringBuffer sb = new StringBuffer();
				readToBuffer(sb, input);
				String fileContent = sb.toString();
				String fileContent2 = fileContent.replaceFirst("value: hello", "value: '" + chip +"'");
				String fileContent3 = fileContent2.replaceFirst("-random", "-" + chip);
				String fileContent4 = fileContent3.replaceFirst("namespace: default", "namespace: " + namespace);
				System.out.println(fileContent4);
				InputStream stream = new ByteArrayInputStream(fileContent4.getBytes());
				Config config = new ConfigBuilder().withMasterUrl(masterurl).build();
				KubernetesClient client = new DefaultKubernetesClient(config);
				CustomResourceDefinition argoCRD = null;
				
				 try {
				      if (!client.supportsApiPath("/apis/apiextensions.k8s.io/v1beta1") && !client.supportsApiPath("/apis/apiextensions.k8s.io/v1")) {
				        System.out.println("WARNING this cluster does not support the API Group apiextensions.k8s.io");
				        return "fail";
				      }
				      CustomResourceDefinitionList list = client.customResourceDefinitions().list();
				      if (list == null) {
				        System.out.println("ERROR no list returned!");
				        return "fail";
				      }
				      List<CustomResourceDefinition> items = list.getItems();
				      for (CustomResourceDefinition item : items) {
				        System.out.println("CustomResourceDefinition " + item.getMetadata().getName() + " has version: " + item.getApiVersion());
				        if (ARGO_CRD_NAME.equals(item.getMetadata().getName())) {
				        	argoCRD = item;
				          }
				      }
				    } catch (KubernetesClientException e) {
				      System.out.println("Failed: " + e);
				      e.printStackTrace();
				    }
				 
				 
				 if (argoCRD != null) {
				        System.out.println("Found CRD: " + argoCRD.getMetadata().getSelfLink());
				      } else {
				    	  return "fail";  
				      }

				 MixedOperation<ArgoCluster, ArgoList, DoneableArgo, Resource<ArgoCluster, DoneableArgo>> argoClient = client.customResources(argoCRD, ArgoCluster.class, ArgoList.class, DoneableArgo.class);

				
				
				CustomResourceList<ArgoCluster> argoList =argoClient.list();
			      List<ArgoCluster> items = argoList.getItems();
			      System.out.println("  found " + items.size() + " argo");
			      for (ArgoCluster item : items) {
			        System.out.println("    " + item);
			      }
			      
			      
				 ArgoCluster createArgo= argoClient.load(stream).get();
				 ArgoCluster finishpod=argoClient.create(createArgo);
				 String podname=finishpod.getMetadata().getName();
				  Pod pod = client.pods().inNamespace(namespace).withName(podname).get();
			      if (pod !=null){
			        System.out.println("Creating pod in namespace " + pod.getMetadata().getNamespace());
			        System.out.println("Created pod " + podname);
			      } else {
			        System.err.println("not found a Pod! " + podname);
			      }
				return fileContent2;
			} catch (IOException e) {
				e.printStackTrace();
				System.out.println("argo未找到配置文件");
			}

		return "ok";
	}

	/**
	 * 将文本文件中的内容读入到buffer中
	 * 
	 * @param buffer
	 *            buffer
	 * @param filePath
	 *            文件路径
	 * @throws IOException
	 *             异常
	 * @author cn.outofmemory
	 * @date 2013-1-7
	 */
	private void readToBuffer(StringBuffer buffer, InputStream input) throws IOException {
		String line; // 用来保存每行读取的内容
		BufferedReader reader = new BufferedReader(new InputStreamReader(input));
		line = reader.readLine(); // 读取第一行
		while (line != null) { // 如果 line 为空说明读完了
			buffer.append(line); // 将读到的内容添加到 buffer 中
			buffer.append("\n"); // 添加换行符
			line = reader.readLine(); // 读取下一行
		}
		reader.close();
		input.close();
	}

}

可能遇到的问题—Forbidden!Configured service account doesn’t have access和User “system:serviceaccount:default:default” cannot get path “/”

解决方案 一、增加role

当集群是基于RBAC(Role-Based Access Control,基于角色的访问控制)时,默认账户有很多的限制,比如不能支持组件,输出等等。
执行以下命令授权默认的服务账户在默认的namespace中拥有admin的权限
我们在本地提交argo submit之前会先运行语句如下:

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

同理,在pod中也需要拿到权限。

pod拿到权限的方法之一就是通过role。
role分为两种 一种是命名空间的role,使用 rolebinding进行绑定。一种是集群的role,使用clusterrolebinding绑定。
我们可以根据自己的需求进行role的绑定。
因为我们这里会访问api的根目录,所以需要比较大的权限,使用clusterrolebinding绑定。

步骤如下:

首先创建一个report-api的pod专用的serviceaccount。
使用命令

vi  report-api-account.yaml

输入内容如下:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: report-api

使用如下命令创建

kubectl create -f report-api-account.yaml

然后给default命名空间下的report-api赋予cluster-admin的角色。
在任意有kubectl命令行的环境中运行

kubectl create clusterrolebinding report-api-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=default:report-api

输出如下:

zhangxiaofans-MacBook-Pro:platform joe$ kubectl create clusterrolebinding report-api-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=default:report-api
clusterrolebinding.rbac.authorization.k8s.io "report-api-on-cluster-admin" created

最后在创建pod的yaml的depolyment中加入参数

serviceAccount: report-api

完成的depolyment.yaml示例如下:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: report-api
spec:
  selector:
    matchLabels:
      app: report-api
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: report-api
    spec:
      serviceAccount: report-api
      containers:
        - name: report-api
          image: 123/vpc/java8:latest
          imagePullPolicy: Always
          command: ["/bin/sh","-c"] 
          args: ["java -jar /jar/report-api-1.0.0-SNAPSHOT.jar --spring.profiles.active=beta"] 
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
          ports:
            - containerPort: 9988
          volumeMounts:
            - name: workdir
              mountPath: /jar
          env:
            - name: USERDB_USERNAME_BETA
              value: _USERDB_USERNAME_BETA_
            - name: USERDB_PASSWORD_BETA
              value: _USERDB_PASSWORD_BETA_
      initContainers:
        - name: pull-lib
          image: anigeo/awscli:latest
          command: ["/bin/sh","-c"] 
          args: ["ls"]
          env:
            - name: AWS_DEFAULT_REGION
              value: cn-southwest-2
          volumeMounts:
            - name: workdir
              mountPath: /jar
      volumes:
        - name: workdir
          emptyDir: {}

参考链接。

clusterrolebinding的相关介绍

https://bugzilla.redhat.com/show_bug.cgi?id=1392767

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/rbd/deploy/rbac/rolebinding.yaml

https://stackoverflow.com/questions/47973570/kubernetes-log-user-systemserviceaccountdefaultdefault-cannot-get-services

https://github.com/fabric8io/fabric8/issues/6840

解决方案 二、传递token

pod拿到权限的方法之二就是通过token,拿到最高权限。

生成token的方法参考

https://jimmysong.io/kubernetes-handbook/guide/auth-with-kubeconfig-or-token.html

关于更多serviceaccount和token方面的理解可以参考

https://www.sharpcode.cn/devops/kubernetes-authn-authz/

https://tonybai.com/2017/03/03/access-api-server-from-a-pod-through-serviceaccount/

https://www.troyying.xyz/index.php/IT/8.html

https://docops.ca.com/ca-apm/10-7/cn/kubernetes-458828513.html

使用如下:

config = new ConfigBuilder().withMasterUrl(master)
                    .withTrustCerts(true)
                    .withNamespace(namespace)
                    .withOauthToken(cluster.getOauthToken())
                    .withUsername(cluster.getUsername())
                    .withPassword(cluster.getPassword())
                    .removeFromTlsVersions(TlsVersion.TLS_1_0)
                    .removeFromTlsVersions(TlsVersion.TLS_1_1)
                    .removeFromTlsVersions(TlsVersion.TLS_1_2)
                    .withRequestTimeout(REQUEST_TIMEOUT)
                    .withConnectionTimeout(CONNECTION_TIMEOUT)
                    .build();

可能遇到的问题–com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field “status” (class com.biologic.entity.ArgoCluster), not marked as ignorable (4 known properties: “spec”, “kind”, “apiVersion”, “metadata”])

原因 ArgoCluster 没有status字段。

在ArgoCluster 中增加status字段即可。

status的字段类型需要打断点查看如下图:

参考链接:

https://raw.githubusercontent.com/argoproj/argo/v2.2.0/manifests/install.yaml

Kubernetes CRDs 自定义资源

https://argoproj.github.io/docs/argo/docs/rest-api.html

https://github.com/kubernetes-client/java/blob/master/kubernetes/docs/CustomObjectsApi.md

https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/CRDExample.java

https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/ListCustomResourceDefinitions.java

https://juejin.im/post/5c108bb8e51d450c5a47c67f

https://blog.csdn.net/jiangpeng_xu/article/details/83688990

参考使用fabric8io的例子

https://github.com/fabric8io/kubernetes-client

https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/CreatePod.java

https://blog.csdn.net/jiangpeng_xu/article/details/83688990

https://www.helplib.com/GitHub/article_128598

其他更多fabric8io使用例子

https://github.com/fabric8io/kubernetes-client/tree/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples

可能遇到的问题 – the server could not find the requested resource

使用io.fabric8 client API 创建部署报错:“the server could not find the requested resource”

问题:
在使用部署文件进行部署时,发现总是报“the server could not find the requested resource”错误,后来发现原来是部署文件中声明的 apiVersion头的问题,

对于kubernetes 1.8.2 api, 创建部署文件时,使用

apiVersion:apps/v1beta2

apiVersion:apps/v1beta1

均会报上述错误。

public void testCreateDeploymentWithFile(){
		
		String sfile = "E:\\deployment.yaml";
		File file = new File(sfile);
		
		FileInputStream fis = null;
		try {
			fis = new FileInputStream(file);
			List<HasMetadata> result = client.load(fis).createOrReplace();
			System.out.println("result:" + result);
		} catch (Exception e) {
			e.printStackTrace();
		} finally{
			if(fis != null){
				try {
					fis.close();
				} catch (Exception e2) {
					// TODO: handle exception
				}
			}
		}		
	}

解决:
将apiVersion值换成

apiVersion: extensions/v1beta1

问题解决。

替换方式可以使用下面代码的简单替换方式:

String fileContent2 = fileContent.replaceFirst("^apiVersion:.*\r\nkind: Deployment",
                       "apiVersion: extensions/v1beta1\r\nkind: Deployment");

注意:
若部署文件中包含服务(service)部署描述,则服务的apiVersion不能为extensions/v1beta1, 应为v1,即:

apiVersion:v1

可能遇到的问题–NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper DefaultKubernetesClient

报错如下:

Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectWriter.forType(Lcom/fasterxml/jackson/databind/JavaType;)Lcom/fasterxml/jackson/databind/ObjectWriter

原因

spring boot有多个json数据转换包所导致,由于spring boot自带的是jackson的包,不需要在maven依赖里面重复添加此类转换包,添加会导致上面出现的异常。

解决方法
在kubernetes-client包中排除多余的jackson包
如下:

		<dependency>
			<groupId>io.fabric8</groupId>
			<artifactId>kubernetes-client</artifactId>
			<version>4.1.1</version><!-- 对应k8s的1.10.6版本 -->
			<exclusions>
				<exclusion>
					<groupId>com.fasterxml.jackson.core</groupId>
					<artifactId>jackson-databind</artifactId>
				</exclusion>
				<exclusion>
					<groupId>com.fasterxml.jackson.core</groupId>
					<artifactId>jackson-core</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

可能遇到的问题–tryServiceAccount WARN Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]

原因
缺少k8s环境的认证信息。

解决方式
该客户端需要运行在配置有k8sconfig的环境下运行,或者k8s的pod内。

可能遇到的问题–io.fabric8.kubernetes.client.KubernetesClientException: Namespace not specified. But operation requires namespace

yaml文件中需要指定namespace

参考:

1、Support complete app deployment with multiple pods, services using yaml file #906. https://github.com/fabric8io/kubernetes-client/issues/906

2、how to load yaml file #170. https://github.com/kubernetes-client/java/issues/170

转载请注明出处:java中提交argo工作流

  • 2
    点赞
  • 0
    评论
  • 12
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

打赏
文章很值,打赏犒劳作者一下
相关推荐
©️2020 CSDN 皮肤主题: Age of Ai 设计师:meimeiellie 返回首页

打赏

张小凡vip

你的鼓励将是我创作的最大动力

¥2 ¥4 ¥6 ¥10 ¥20
输入1-500的整数
余额支付 (余额:-- )
扫码支付
扫码支付:¥2
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值