14:00:51 <s3wong> #startmeeting clover 14:00:51 <collabot> Meeting started Mon Jan 22 14:00:51 2018 UTC. The chair is s3wong. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:51 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic. 14:00:51 <collabot> The meeting name has been set to 'clover' 14:01:03 <liangqi> #info liangqi 14:01:21 <yapeng> #info yapeng 14:01:26 <enriquetaso> #info enriquetaso 14:01:43 <earrage> #info earrage 14:02:16 <s3wong> #link Agenda: https://wiki.opnfv.org/display/meetings/Clover 14:02:46 <s3wong> #topic Status / Update of Fraser tasks 14:03:26 <s3wong> The first one is pod upgrade from k8s 1.7 to 1.9 14:03:51 <s3wong> Though Xuan isn't on the channel now, he very quickly got this work done 14:04:05 <s3wong> https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2018-January/019870.html 14:04:28 <s3wong> Thanks for Xuan for promptly getting this done 14:05:22 <s3wong> I believe enriquetaso already tried running clearwater on 1.9 pod? 14:05:33 <liangqi> i've test continer4nfv's patch for 1.9 k8s installation 14:05:58 <s3wong> liangqi: good! 14:05:58 <enriquetaso> yes, It works on most of the cases 14:06:12 <liangqi> #link https://gerrit.opnfv.org/gerrit/#/c/50467/ 14:07:22 <s3wong> And that gets to the second task from two weeks ago: Helm charts for Clearwater 14:07:40 <s3wong> As liangqi linked above, enriquetaso already has an initial patch out 14:08:03 <enriquetaso> I have to update the patch 50467 but the patch works for me 14:08:36 <s3wong> enriquetaso: excellent progress 14:09:00 <liangqi> enriquetaso i'll test your patch tomorrow 14:09:14 <enriquetaso> Thanks liangqi ! 14:09:31 <liangqi> i've test old patch set, it works for me 14:10:06 <liangqi> dose not test the latest one with helm charts support 14:10:13 <enriquetaso> #link https://github.com/Metaswitch/clearwater-docker/pull/85 14:10:50 <enriquetaso> could you access to Ellis webserver liangqi? 14:11:07 <s3wong> enriquetaso: nice 14:11:19 <enriquetaso> I really appreciate your help liangqi :) 14:11:43 <liangqi> yes, i tried access the webserver 14:12:05 <liangqi> but did not make a ims call yet 14:12:42 <enriquetaso> ok :) 14:13:21 <s3wong> Great progress on the Helm chart front; and that brings us to the third item: Istio Installation 14:13:33 <s3wong> yapeng: anything to update? 14:13:57 <yapeng> The task is done. I submit a patch last week, it is merged: https://gerrit.opnfv.org/gerrit/#/c/50793/ 14:14:09 <yapeng> thanks ruijing code reviewed. 14:15:06 <s3wong> yapeng: excellent 14:16:32 <s3wong> With these two things done (Clearwater Helm charts and Istio installation) --- the question now is how to link them together 14:17:06 <s3wong> Question #1 is how to inject the Envoy sidecar? 14:17:27 <s3wong> #link https://istio.io/docs/setup/kubernetes/sidecar-injection.html 14:18:22 <yapeng> do we want to use automatic inject? 14:19:03 <s3wong> On a regular docker-clearwater deployment (assuming you use the specs generated by k8s-gen), you have 13 pods when deploying Clearwater 14:19:09 <s3wong> yapeng: good question 14:19:37 <s3wong> The two options (as stated by the link above) are: manual injection or automatic injection 14:19:49 <yapeng> in the current installation patch, i installed manual injection option. 14:20:46 <s3wong> If we go with automatic injection, then when we set up the k8s cluster, we need to enable "initializer" option 14:21:24 <s3wong> yapeng: of the 13 pods off of Clearwater, there are some that I am NOT sure would require Envoy 14:22:01 <s3wong> For example: there are both the Cassandra and the etcd pods --- they are probably NOT candidates for being part of the mesh 14:22:21 <s3wong> Even for astaire, I am not sure 14:22:51 <s3wong> Bono, Homer, Homestead, Ralf, and Sprout --- sure 14:23:33 <s3wong> Another requirement for injection (manual or automatic) is that the service port name needs to adhere to the format 14:23:57 <s3wong> <protocol>[-suffix] 14:25:08 <s3wong> enriquetaso: for example, the service spec generated by k8s-gen has all the ports with name that are the service port number, which needs to be modified 14:25:40 <yapeng> s3wong: good points. i saw 4 requirements in pod spec.. did not quite follow all of them yet. 14:27:21 <s3wong> yapeng, enriquetaso: assuming we use manual injection, can we invoke istioctl from Helm? 14:28:39 <enriquetaso> I haven't tried it yet, s3wong . What should be change inside the spec? I'm trying to find this information in the link above, is it there? 14:29:23 <s3wong> enriquetaso: two ways 14:29:59 <s3wong> enriquetaso: we can invoke "istioctl kube-inject" whenever we would have done kubectl apply -f ... 14:30:51 <s3wong> enriquetaso: Or... in the pod spec for each target pod type, we need to add a new field to spec: containers: 14:31:04 <s3wong> enriquetaso: as well as initContainers 14:31:23 <yapeng> s3wong, enriquetaso: i think it is possible, i saw some discussion in the helm discussion: https://github.com/kubernetes/helm/issues/2596 14:31:35 <s3wong> enriquetaso: See the section "Example" under the https://istio.io/docs/setup/kubernetes/sidecar-injection.html 14:31:52 <s3wong> yapeng: oh, good 14:32:23 <enriquetaso> aah, ok. I'll try the manual injection s3wong 14:33:00 <s3wong> yapeng: I am not much of an Helm user --- but I believe you do "helm install" instead of "kubectl apply", right? 14:33:51 <s3wong> enriquetaso: or follow yapeng 's link above --- seems like the Helm folks are looking at this problem as well 14:34:14 <s3wong> enriquetaso: that said, we still need to name the service ports differently in the Helm charts :-) 14:34:22 <yapeng> s3wong: me neither. i don't know the answer yet. i am trying to dig into helm discussion about istio injection. 14:35:15 <s3wong> yapeng, enriquetaso: worse case, we perform manual injection by directly adding it to template (I mean, istioctl kube-inject basically just changes that pod spec anyway) 14:35:43 <enriquetaso> cool 14:35:44 <s3wong> But I believe as per the reason above, we are finalizing on NOT doing automatic injection, right? 14:35:48 <yapeng> s3wong: agreed. 14:36:29 <enriquetaso> agreed. 14:36:33 <s3wong> Would be strange to run an Envoy sidecar in Cassandra cluster --- which Clearwater is already creating three pods by default 14:37:12 <s3wong> Alright, that brings us to the Clearwater / Istio integration investigation 14:37:57 <s3wong> I have been cheating --- I have set up testing on GKE. Hence realizing all the above problems (though to be fair, they are fairly well documented) 14:39:03 <s3wong> Will update folks on Slack as I find out more. So far I do have an Clearwater deployment with Istio 14:39:44 <s3wong> With all the great work done already, we should move forward to the next set of unclaimed tasks 14:39:45 <yapeng> s3wong, in GKE? 14:40:17 <s3wong> yapeng: yep, I am skipping k8s and Istio installation by going directly on GKE 14:40:59 <s3wong> yapeng: the somewhat hairy part was the static IP that needs to bind to Bono, as GKE UI is quite unfriendly 14:42:01 <s3wong> yapeng: at this point of my testing, clearwater-live-test is still not really passing on a non-Istio setup, at the Istio setup, at least the pods aren't crashing 14:42:49 <s3wong> yapeng: you and Amit both signed up for this investigation also, will ping you guy on Slack on more technical discussion as I move forward 14:43:06 <s3wong> #topic Next set of tasks 14:43:09 <yapeng> s3wong, sure 14:44:39 <s3wong> Folks are making good progress, and I believe we need to start looking into running CI jobs 14:45:15 <s3wong> Of course, none of the patches at this point is going to the Clover repo, so it isn't urgent yet 14:46:39 <s3wong> Which gets me thinking --- since some Helm templates will be changed as part of sidecar injection, should that go to container4nfv, or should that be a clover item? 14:47:43 <s3wong> enriquetaso: your Helm charts are for more generic Clearwater installation on k8s, should we assume we need to always include Istio? 14:48:28 <s3wong> (particularly when them going to the clearwater-docker repo) 14:49:36 <s3wong> Something to think about 14:49:51 <enriquetaso> I have to double-check with my mentors... As you wrote, my charts are for more generic installation. However, I think I have to include Istio 14:50:26 <s3wong> enriquetaso: is it possible to combine two templates during <helm install> time? 14:50:58 <s3wong> enriquetaso: anyway, something to think about --- how you would like to structure your Helm charts such that you have a generic piece and an istio addition 14:51:02 <enriquetaso> Not sure... this is my first time usign Helm... I have to try it out today 14:51:13 <s3wong> enriquetaso: as yapeng said, Helm community is looking at this problem also 14:52:10 <s3wong> And CI jobs also bring us to the second additional task: CI means automated unit / functional tests 14:52:31 <s3wong> How do we verify? What are the test cases? 14:53:45 <s3wong> For the Istio / Clearwater front, for functional testing, I need to have something to trace if the traffic followed Mixer policy or proper load balancing 14:54:53 <s3wong> Envoy has Opentracing support, and at least for GKE, the standard Istio installation template includes Zipkin (as a tracer) 14:55:07 <s3wong> https://console.cloud.google.com/launcher/config?templateurl=https:%2F%2Fraw.githubusercontent.com%2Fistio%2Fistio%2Fmaster%2Finstall%2Fgcp%2Fdeployment_manager%2Fistio-cluster.jinja&pli=1 14:55:49 <s3wong> Whether we use Zipkin or Jaegar is a task for investigation 14:56:00 <yapeng> Good points. 14:56:56 <s3wong> And of course, once Clearwater works with Envoy, we need to do some performance comparison 14:57:03 <s3wong> That is another task 14:58:39 <s3wong> For now, I will open these on JIRA, will give it to folks who are interested 14:59:27 <s3wong> Anything else to discuss? 15:00:44 <yapeng> not from me, i think we have good handful tasks :) 15:00:52 <s3wong> Cool, Thanks enriquetaso, yapeng, and xuan_jia for the excellent progress. Talk to you guys either on Slack or in two weeks 15:01:02 <s3wong> #endmeeting