14:00:51 #startmeeting clover 14:00:51 Meeting started Mon Jan 22 14:00:51 2018 UTC. The chair is s3wong. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:51 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:00:51 The meeting name has been set to 'clover' 14:01:03 #info liangqi 14:01:21 #info yapeng 14:01:26 #info enriquetaso 14:01:43 #info earrage 14:02:16 #link Agenda: https://wiki.opnfv.org/display/meetings/Clover 14:02:46 #topic Status / Update of Fraser tasks 14:03:26 The first one is pod upgrade from k8s 1.7 to 1.9 14:03:51 Though Xuan isn't on the channel now, he very quickly got this work done 14:04:05 https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2018-January/019870.html 14:04:28 Thanks for Xuan for promptly getting this done 14:05:22 I believe enriquetaso already tried running clearwater on 1.9 pod? 14:05:33 i've test continer4nfv's patch for 1.9 k8s installation 14:05:58 liangqi: good! 14:05:58 yes, It works on most of the cases 14:06:12 #link https://gerrit.opnfv.org/gerrit/#/c/50467/ 14:07:22 And that gets to the second task from two weeks ago: Helm charts for Clearwater 14:07:40 As liangqi linked above, enriquetaso already has an initial patch out 14:08:03 I have to update the patch 50467 but the patch works for me 14:08:36 enriquetaso: excellent progress 14:09:00 enriquetaso i'll test your patch tomorrow 14:09:14 Thanks liangqi ! 14:09:31 i've test old patch set, it works for me 14:10:06 dose not test the latest one with helm charts support 14:10:13 #link https://github.com/Metaswitch/clearwater-docker/pull/85 14:10:50 could you access to Ellis webserver liangqi? 14:11:07 enriquetaso: nice 14:11:19 I really appreciate your help liangqi :) 14:11:43 yes, i tried access the webserver 14:12:05 but did not make a ims call yet 14:12:42 ok :) 14:13:21 Great progress on the Helm chart front; and that brings us to the third item: Istio Installation 14:13:33 yapeng: anything to update? 14:13:57 The task is done. I submit a patch last week, it is merged: https://gerrit.opnfv.org/gerrit/#/c/50793/ 14:14:09 thanks ruijing code reviewed. 14:15:06 yapeng: excellent 14:16:32 With these two things done (Clearwater Helm charts and Istio installation) --- the question now is how to link them together 14:17:06 Question #1 is how to inject the Envoy sidecar? 14:17:27 #link https://istio.io/docs/setup/kubernetes/sidecar-injection.html 14:18:22 do we want to use automatic inject? 14:19:03 On a regular docker-clearwater deployment (assuming you use the specs generated by k8s-gen), you have 13 pods when deploying Clearwater 14:19:09 yapeng: good question 14:19:37 The two options (as stated by the link above) are: manual injection or automatic injection 14:19:49 in the current installation patch, i installed manual injection option. 14:20:46 If we go with automatic injection, then when we set up the k8s cluster, we need to enable "initializer" option 14:21:24 yapeng: of the 13 pods off of Clearwater, there are some that I am NOT sure would require Envoy 14:22:01 For example: there are both the Cassandra and the etcd pods --- they are probably NOT candidates for being part of the mesh 14:22:21 Even for astaire, I am not sure 14:22:51 Bono, Homer, Homestead, Ralf, and Sprout --- sure 14:23:33 Another requirement for injection (manual or automatic) is that the service port name needs to adhere to the format 14:23:57 [-suffix] 14:25:08 enriquetaso: for example, the service spec generated by k8s-gen has all the ports with name that are the service port number, which needs to be modified 14:25:40 s3wong: good points. i saw 4 requirements in pod spec.. did not quite follow all of them yet. 14:27:21 yapeng, enriquetaso: assuming we use manual injection, can we invoke istioctl from Helm? 14:28:39 I haven't tried it yet, s3wong . What should be change inside the spec? I'm trying to find this information in the link above, is it there? 14:29:23 enriquetaso: two ways 14:29:59 enriquetaso: we can invoke "istioctl kube-inject" whenever we would have done kubectl apply -f ... 14:30:51 enriquetaso: Or... in the pod spec for each target pod type, we need to add a new field to spec: containers: 14:31:04 enriquetaso: as well as initContainers 14:31:23 s3wong, enriquetaso: i think it is possible, i saw some discussion in the helm discussion: https://github.com/kubernetes/helm/issues/2596 14:31:35 enriquetaso: See the section "Example" under the https://istio.io/docs/setup/kubernetes/sidecar-injection.html 14:31:52 yapeng: oh, good 14:32:23 aah, ok. I'll try the manual injection s3wong 14:33:00 yapeng: I am not much of an Helm user --- but I believe you do "helm install" instead of "kubectl apply", right? 14:33:51 enriquetaso: or follow yapeng 's link above --- seems like the Helm folks are looking at this problem as well 14:34:14 enriquetaso: that said, we still need to name the service ports differently in the Helm charts :-) 14:34:22 s3wong: me neither. i don't know the answer yet. i am trying to dig into helm discussion about istio injection. 14:35:15 yapeng, enriquetaso: worse case, we perform manual injection by directly adding it to template (I mean, istioctl kube-inject basically just changes that pod spec anyway) 14:35:43 cool 14:35:44 But I believe as per the reason above, we are finalizing on NOT doing automatic injection, right? 14:35:48 s3wong: agreed. 14:36:29 agreed. 14:36:33 Would be strange to run an Envoy sidecar in Cassandra cluster --- which Clearwater is already creating three pods by default 14:37:12 Alright, that brings us to the Clearwater / Istio integration investigation 14:37:57 I have been cheating --- I have set up testing on GKE. Hence realizing all the above problems (though to be fair, they are fairly well documented) 14:39:03 Will update folks on Slack as I find out more. So far I do have an Clearwater deployment with Istio 14:39:44 With all the great work done already, we should move forward to the next set of unclaimed tasks 14:39:45 s3wong, in GKE? 14:40:17 yapeng: yep, I am skipping k8s and Istio installation by going directly on GKE 14:40:59 yapeng: the somewhat hairy part was the static IP that needs to bind to Bono, as GKE UI is quite unfriendly 14:42:01 yapeng: at this point of my testing, clearwater-live-test is still not really passing on a non-Istio setup, at the Istio setup, at least the pods aren't crashing 14:42:49 yapeng: you and Amit both signed up for this investigation also, will ping you guy on Slack on more technical discussion as I move forward 14:43:06 #topic Next set of tasks 14:43:09 s3wong, sure 14:44:39 Folks are making good progress, and I believe we need to start looking into running CI jobs 14:45:15 Of course, none of the patches at this point is going to the Clover repo, so it isn't urgent yet 14:46:39 Which gets me thinking --- since some Helm templates will be changed as part of sidecar injection, should that go to container4nfv, or should that be a clover item? 14:47:43 enriquetaso: your Helm charts are for more generic Clearwater installation on k8s, should we assume we need to always include Istio? 14:48:28 (particularly when them going to the clearwater-docker repo) 14:49:36 Something to think about 14:49:51 I have to double-check with my mentors... As you wrote, my charts are for more generic installation. However, I think I have to include Istio 14:50:26 enriquetaso: is it possible to combine two templates during time? 14:50:58 enriquetaso: anyway, something to think about --- how you would like to structure your Helm charts such that you have a generic piece and an istio addition 14:51:02 Not sure... this is my first time usign Helm... I have to try it out today 14:51:13 enriquetaso: as yapeng said, Helm community is looking at this problem also 14:52:10 And CI jobs also bring us to the second additional task: CI means automated unit / functional tests 14:52:31 How do we verify? What are the test cases? 14:53:45 For the Istio / Clearwater front, for functional testing, I need to have something to trace if the traffic followed Mixer policy or proper load balancing 14:54:53 Envoy has Opentracing support, and at least for GKE, the standard Istio installation template includes Zipkin (as a tracer) 14:55:07 https://console.cloud.google.com/launcher/config?templateurl=https:%2F%2Fraw.githubusercontent.com%2Fistio%2Fistio%2Fmaster%2Finstall%2Fgcp%2Fdeployment_manager%2Fistio-cluster.jinja&pli=1 14:55:49 Whether we use Zipkin or Jaegar is a task for investigation 14:56:00 Good points. 14:56:56 And of course, once Clearwater works with Envoy, we need to do some performance comparison 14:57:03 That is another task 14:58:39 For now, I will open these on JIRA, will give it to folks who are interested 14:59:27 Anything else to discuss? 15:00:44 not from me, i think we have good handful tasks :) 15:00:52 Cool, Thanks enriquetaso, yapeng, and xuan_jia for the excellent progress. Talk to you guys either on Slack or in two weeks 15:01:02 #endmeeting