08:00:37 <morgan_orange> #startmeeting Functest weekly meeting June 28th
08:00:37 <collabot`> Meeting started Tue Jun 28 08:00:37 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:37 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:37 <collabot`> The meeting name has been set to 'functest_weekly_meeting_june_28th'
08:00:42 <morgan_orange> #ttopic call role
08:00:42 <boucherv_orange> hi
08:00:47 <morgan_orange> #topic call role
08:00:50 <lhinds> #info lhinds
08:00:51 <morgan_orange> #info Morgan Richomme
08:00:55 <juhak> #info Juha Kosonen
08:00:57 <JuhaHaapa> #info Juha Haapavirta
08:00:59 <SerenaFeng> #info SerenaFeng
08:00:59 <viktor_t> #info Viktor Tikkanen
08:01:00 <boucherv_orange> #info Valentin Boucher
08:01:06 <raghavendrachari> #info Raghavendrachari K
08:01:06 <CG_Nokia> #info CG_Nokia (Colum Gaynor)
08:01:44 <jose_lausuch> #info Jose Lausuch
08:01:57 <morgan_orange> #info agenda https://wiki.opnfv.org/display/functest/Functest+Meeting
08:02:05 <morgan_orange> #topic action point follow up
08:02:18 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-06-14-08.00.html
08:02:28 <morgan_orange> #info AP OK, presentation done
08:02:40 <morgan_orange> #topic Feedback from berlin
08:02:53 <jose_lausuch> only 1 ap :)
08:02:59 <morgan_orange> if all the participants can share their feedback on 2 or 3 info line
08:03:43 <morgan_orange> #info exhausting but rich summit: Functest now well known in the community
08:04:02 <CG_Nokia> Applause!
08:04:04 <morgan_orange> #info new contacts: tacker, Okinawa lab, domino project
08:04:07 <viktor_t> #info Face to face meetings are efficient...
08:04:09 <lhinds> #info lhinds: great summit, enjoyed breakouts, meeting functest folk, security scanning got a lot of interest and others wanting to contribute.
08:04:37 <morgan_orange> #info 2 hot topics were orchestration and VNF on boarding
08:05:00 <morgan_orange> #info Functest is Mano Agnistic but we may be involved in VNF onboard topic for D release
08:05:09 <JuhaHaapa> #info functest had a strong precense in the summit.  There was for instance Morgan's excellent presentation in the beginning
08:05:18 <SerenaFeng> #info SerenaFeng: really happy to meet all of you face to face
08:05:39 <jose_lausuch> #info we managed to become even more visible and grab the attention of others. On the other hand, we got some feedback about things we could do/improve, like make the feature test integration a bit clearer.
08:05:55 <jose_lausuch> #info meeting people personally makes things easier later on
08:06:19 <boucherv_orange> #info Really happy to meet a part of functest team .. Otherwise, I have noticed a strong interest of functest project in different presentation.
08:06:58 <morgan_orange> jose_lausuch: yep that is why we must think to the fature project integration through template as suggested by SerenaFeng for D release. For Colorado we will still do some stuff manually but a tempalte an a easy integration journey has to be prepared for the future
08:07:25 <morgan_orange> raghavendrachari: viktor_t your feedback on the summit?
08:08:21 <jose_lausuch> #info feedback from others: we need to improve our jenkins landing page.
08:08:23 <raghavendrachari> #info good knowledge sharing, it was nice to meet functest team , lot of presentations on tesing,
08:08:43 <morgan_orange> jose_lausuch: that is why I add a topic on jenkins for today :)
08:08:57 <jose_lausuch> awesome
08:09:25 <morgan_orange> ok if no additional comments or questions on berlin, I suggest to move to next topic
08:09:37 <morgan_orange> #topic Colorado Status & Sprint #9 follow-up
08:09:54 <morgan_orange> #info feature freeze planned on the 7 of July
08:10:32 <morgan_orange> #info main remaining task regarding this date: finalize security_scan integration, refactoring of ODL, troubleshooting
08:10:50 <jose_lausuch> mmmm
08:10:53 <jose_lausuch> nope
08:10:56 <morgan_orange> and of course documentation but thanks to CG_Nokia this time we are not late..
08:10:57 <jose_lausuch> feature freeze is this week
08:11:06 <jose_lausuch> test implementation freeze on 7th july
08:11:15 <morgan_orange> oups you are right
08:11:41 <lhinds> how you handle feature freezes, create a release branch?
08:11:44 <morgan_orange> #info correction: feature freeze this week, test implementation freeze 7/7
08:12:09 <jose_lausuch> #link https://wiki.opnfv.org/display/SWREL/Colorado?preview=%2F2926065%2F6823197%2Fcolorado+r7.pdf
08:12:41 <jose_lausuch> #link https://wiki.opnfv.org/display/SWREL/Colorado
08:12:53 <morgan_orange> but anyway we do not develop new features on our side so the milestone we are dealing with is more the one of the 7th of July I think
08:13:04 <jose_lausuch> yes
08:13:05 <jose_lausuch> it is
08:13:16 <jose_lausuch> 1st of july is for feature integration in installers
08:13:26 <jose_lausuch> we are working on that for sdnvpn :)
08:13:34 <jose_lausuch> and new test cases coming for the 7th
08:14:09 <morgan_orange> we will have probably additional questions: domino project can do stand alone testing (policy check), if it makes tests on real SUT, integration to be planned, fastdatastacks is also finalizing the code
08:14:19 <morgan_orange> and of course maybe contrail, copper, ...
08:14:37 <morgan_orange> basically in the JIRA we should have now have mainly bug fixes + integration tasks
08:14:59 <morgan_orange> so lhinds for the freeze it is still....a bit manual
08:15:21 <morgan_orange> we just will not accept big patch introducing new features
08:15:29 <morgan_orange> until we got the colorado branch
08:15:44 <lhinds> morgan_orange: got it, thanks
08:15:46 <morgan_orange> but it makes no sens to branch too early, otherwise we will keep on cherry picking
08:16:32 <JuhaHaapa> odl will still drag behind. I'm not able to get it refactored for initial release
08:16:33 <morgan_orange> #info sdnvpn, sfc, sfc onos, domino, fastdatastack and maybe moon, copper, ovno to be integrated
08:17:15 <jose_lausuch> morgan_orange: no, not branching to early please... it will be a chaos, still some work to do :)
08:17:27 <jose_lausuch> still big commits to come
08:17:36 <morgan_orange> jose_lausuch: yes I think it is end of August..we will keep on play on master
08:17:47 <morgan_orange> we just have to be "reasonable"
08:18:15 <morgan_orange> #info Sprint 9 ends this week
08:18:24 <morgan_orange> i suggest to create a Sprint 1à until the 7/7
08:19:07 <morgan_orange> any objection for this short 2 weeks sprint
08:19:28 <jose_lausuch> no
08:19:30 <viktor_t> no...
08:19:31 <jose_lausuch> it makes sense
08:19:34 <morgan_orange> #action morgan_orange prepare Sprint 10 until test integration freeze
08:19:38 <SerenaFeng> nope,
08:20:20 <morgan_orange> #action morgan_orange precise during the release meeting that today there are still not real news on some projects (copper, moon, ovno,..) if nothing before the 7th of July, we will not accept integration patch
08:20:41 <morgan_orange> I assume the delcaration + the scripts shall be available (even if not fully finalized)
08:21:33 <morgan_orange> in the list of internal tasks for functest, I forgot the work on ELK to get the dashboard
08:21:48 <morgan_orange> but SerenaFeng already started working on this aspect
08:22:21 <morgan_orange> maybe focus on some open tasks at the moment
08:22:32 <morgan_orange> lhinds: last news from security_scan
08:22:41 <lhinds> morgan_orange: ok..
08:23:19 <jose_lausuch> morgan_orange: I think we got some info from copper
08:23:27 <lhinds> Jenkins is still not reaching my test as yet..hitting failures first. Won't go into details as I don't know it too well, but would be good if someone could take a look
08:23:31 <jose_lausuch> we need to provide some info to them to help Bryan integrate the testcases
08:23:40 <morgan_orange> jose_lausuch: ok
08:24:23 <lhinds> secondly: a few folk want to contribute, so I think it makes sense to move us into our own repo, and then be cloned in by functest.
08:24:35 <lhinds> plan was to take this to the TSC for approval first.
08:24:43 <morgan_orange> lhinds: make sense but you want to do it now or after colorado?
08:24:51 <lhinds> but consider all the 'security is important' messages at the summit, i think they will approve :)
08:25:10 <lhinds> morgan_orange: I am open here, maybe I put the request in and get it set up.
08:25:21 <lhinds> and then we can move when it makes life easier?
08:25:36 <lhinds> is there any blowback from moving before Colorado?
08:25:45 <morgan_orange> OK, currently the last status on apex => heacheck KO with odl scenarios, feature tests Ko with no SDn and still high tempest error rate
08:26:11 <jose_lausuch> lhinds: I dont think there is an issue with moving to your own repo, the question is, will it be ready this week?
08:26:20 <jose_lausuch> you have to talk to healpdesk to create that repo
08:26:46 <morgan_orange> lhinds: no strong opinion, it is just that the feature freeze is today so in theory, no new repo could be created for Colorado
08:27:31 <lhinds> jose_lausuch: understand, makes sense that I get the approval underway, infra tickets etc..and then we do the change over later
08:27:50 <lhinds> morgan_orange: I mean
08:27:58 <lhinds> wrong highlight nick
08:28:10 <morgan_orange> if it is just a question of repo, I imagine we can keep it as it is for colorado and move everything in security_scan after colorado but it is up to you, I understand the need to get a dedicated repo (it may lead to some confusion)
08:28:21 <lhinds> I will talk to them and see what they say later today
08:28:42 <morgan_orange> #action lhinds clarify need for a dedicated security repo for Colorado
08:29:09 <lhinds> one other question...
08:29:36 <lhinds> with feature freeze coming, I need to continue work on other installers, other scan types...
08:30:00 <lhinds> do I just keep to local branch, until a D master branch is ready?
08:30:08 <morgan_orange> in theory no..or at least no for colorado but then we will need a experimental branch
08:30:09 <lhinds> I should know this, but I don't yet :P
08:30:19 <lhinds> experimental..ok
08:30:21 <lhinds> understand
08:30:26 <lhinds> good for me
08:30:53 <morgan_orange> we do not have experimental branch, I may ask this afternoon how release manager plan to manage that
08:31:37 <morgan_orange> Colorado branch is planned for end of August but shall we have a D branch already for work not included in Colorado (so not possible to push in Master until we branch...)
08:31:53 <lhinds> morgan_orange: understand
08:32:02 <morgan_orange> #action morgan_orange ask release manager how to deal with work on non C features
08:32:15 <morgan_orange> ok so let's come back to the current status
08:32:26 <morgan_orange> #link http://testresults.opnfv.org/reporting/functest/release/master/index-status-joid.html
08:32:31 <lhinds> so for security tests...how can we insure a good run ?
08:33:28 <lhinds> I would like to see it working, or debug accordingly.
08:34:01 <morgan_orange> lhinds: I think we shall secure apex run, nosdn scenario in progress at the moment. we should be able to troubleshoot and run only security_scan to fix integration issue
08:34:02 <lhinds> do we change the test order perhaps?
08:34:19 <morgan_orange> you can run security_scan test only... if you have CI time
08:34:42 <morgan_orange> the easies way will be to have a nosdn stable env and troubleshoot directly from the container
08:34:54 <lhinds> trozet mentioned this as a possible, run it a silo'ed job
08:35:07 <lhinds> I can chat to him later about that, if that's what we want to do?
08:35:46 <morgan_orange> we have also to troubleshoot the odl scenarios that currently lead to odl issue, we must ask for time on POD to be able to finalize secruity_scan integration + troubleshoot ODL + test future next ODL suite
08:36:10 <viktor_t> I was struggling with odl in apex
08:36:17 <viktor_t> pre-Boron doesn't solve neutron object creation problems
08:36:25 <viktor_t> null exception logs are still written to karaf.log...
08:36:50 <jose_lausuch> viktor_t: what pre-boron version are you usiung?
08:37:08 <viktor_t> 1.3.0.boron-daily-v201606060053
08:37:36 <morgan_orange> viktor_t: would it be possible to use also Nokia lab to validate security_scan?
08:37:49 <viktor_t> I suppose...
08:38:08 <viktor_t> The problem is that healthcheck fails
08:38:26 <morgan_orange> you have the same errors that the one on LF POD1
08:38:32 <morgan_orange> at least it is consistant...
08:38:38 <viktor_t> yes
08:38:43 <jose_lausuch> viktor_t: at what point does it fail?
08:38:46 <jose_lausuch> creating what
08:38:50 <morgan_orange> https://build.opnfv.org/ci/job/functest-apex-apex-daily-master-daily-master/47/console
08:39:01 <morgan_orange> 4. Create Routers...
08:39:01 <morgan_orange> Port c063a450-d437-40a5-bfc2-cb6180f28f85 cannot be deleted directly via the port API: has device owner network:router_gateway.
08:39:01 <morgan_orange> Neutron server returns request_ids: ['req-12fefc15-6b22-4bd4-bc30-78f4a516635b']
08:39:02 <viktor_t> subnets or routers this time
08:39:21 <morgan_orange> there is no 3...
08:39:29 <morgan_orange> 2016-06-27 15:34:15,615 - healtcheck - INFO -  1. Create Networks...
08:39:29 <morgan_orange> 2016-06-27 15:34:19,714 - healtcheck - INFO -  2. Create subnets...
08:39:29 <morgan_orange> 2016-06-27 15:34:22,684 - healtcheck - INFO -  4. Create Routers...
08:39:39 <viktor_t> it fails in different points every time
08:40:24 <morgan_orange> ok so security scan can be tested only on an apex/nosdn scenario...
08:40:41 <viktor_t> vs. https://build.opnfv.org/ci/job/functest-apex-nokia-pod1-daily-master/55/console
08:41:48 <morgan_orange> healthcheck tests are supposed to be run on apex, not sure it is otherwise it will not be deployed in bare metal
08:42:06 <morgan_orange> but as far as I remembe rtim disable the tests...
08:42:19 <jose_lausuch> #ap jose_lausuch create step 3 in healtcheck
08:42:20 <jose_lausuch> :)
08:42:22 <ollivier> Uhm... @Morgan could it be related to a floating IP issue?
08:42:54 <jose_lausuch> 2016-06-28 06:40:23,149 - healtcheck - DEBUG -  router 'opnfv-router1' gateway set to 28620a29-c1ba-4f17-a190-330b613fedcc. Port ba48eced-6dbc-4f3a-b001-0f57df812e49 cannot be deleted directly via the port API: has device owner network:router_gateway. Neutron server returns request_ids: ['req-76617d34-8526-4be8-a48f-f2f0c0547cf6']
08:42:57 <jose_lausuch> strange error
08:43:26 <viktor_t> port creation problem?
08:43:40 <ollivier> remove
08:43:42 <morgan_orange> we have similar errors in Tempest suite...post error..
08:44:04 <raghavendrachari> i think tenant has no access
08:44:29 <JuhaHaapa> for to delete you have to (manually) change the owner first.
08:44:34 <lhinds> sounds like a device_owner issue
08:44:45 <morgan_orange> at least we do not have this error with compass, fuel, joid / odl_l2 scenarios
08:44:50 <JuhaHaapa> ...so can not be done from within the testcase
08:45:01 <jose_lausuch> let's not try the solve the problem now, who takes the ap to troubleshoot? I can help but if its an odl issue, I'm not an expert :)
08:45:18 <viktor_t> see builds 54...57, failures are in different steps
08:45:20 <morgan_orange> I think we shall report it to apex as well
08:45:32 <morgan_orange> not sure we are supposed to troubleshoot all the scenarios of everybody...
08:45:34 <ollivier> not sure it's related to ODL
08:45:58 <jose_lausuch> +1
08:46:01 <jose_lausuch> let's report it to apex
08:46:12 <morgan_orange> or better to the odl scenario owner in apex
08:46:37 <morgan_orange> #action morgan_orange report apex/odl issues on scenario owners
08:47:10 <morgan_orange> we have also some issues in compass, fuel and joid...that needs some troubleshooting
08:47:29 <morgan_orange> see status reporting
08:47:50 <morgan_orange> it will be probably our main tasks during summer time...
08:48:05 <viktor_t> =during holiday season :)
08:48:26 <morgan_orange> today there are 3 compass and 1 fuel scenarios OK from Functest perspective
08:49:12 <morgan_orange> #action viktor_t report tempest errors to compass and joid scenario owner
08:49:30 <morgan_orange> ok enough for the status for today, let's keep some minutes for jenkins
08:49:45 <morgan_orange> #topic Functest jenkins status / weekly/daily jobs
08:50:03 <morgan_orange> jose_lausuch:  could you info the feedback you got from the jenkins functest home page during the summit
08:50:08 <jose_lausuch> so our jenkins landing page is not so clear
08:50:14 <jose_lausuch> if you want to find jobs for all the installers
08:50:34 <jose_lausuch> https://build.opnfv.org/ci/view/functest/
08:50:39 <morgan_orange> #link https://build.opnfv.org/ci/view/functest/
08:50:51 <jose_lausuch> apex is not there for example
08:50:55 <morgan_orange> #info feedback from berlin; functest jenkins kanding page not clear
08:51:00 <jose_lausuch> it should be in barametal section
08:51:23 <morgan_orange> May-meimei: are you there, can I action you on this task (I think there is already a Jira)
08:51:46 <morgan_orange> #action May-meimei add Apex in bare metal section of Functest jenkins landing page
08:52:49 <morgan_orange> except apex which is a bit different (as they are running their 3 scenarios (nosdn, odl_l2, odll3 in their daily job) what could we do
08:53:00 <morgan_orange> in the current page, I can see fuel, compass and joid
08:53:08 <morgan_orange> the Barematel section corresponds to the CI POD
08:53:19 <morgan_orange> Non CI POD are listed below
08:53:40 <jose_lausuch> ja
08:53:42 <morgan_orange> so main poitn will be to see apex as we see the 3 other installers
08:53:53 <jose_lausuch> so it looks good, except missing apex
08:54:05 <morgan_orange> ok
08:54:18 <morgan_orange> May-meimei: can you manage it?
08:54:19 <jose_lausuch> the problem now is taht we dont know on what pod it is executed
08:54:29 <jose_lausuch> unless you click in the job and look for it
08:54:59 <morgan_orange> yes but it was a choice to consider POD as non specific resources
08:55:07 <jose_lausuch> lf-pod2 is iddle
08:55:20 <May-meimei> morgan_orange: about the result, yes?
08:55:34 <jose_lausuch> only 1 run on fuel https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-suite-master/
08:55:48 <morgan_orange> May-meimei: about putting apex in the bare metal list on the jenkins functest landing page
08:56:03 <May-meimei> morgan_orange: ok
08:56:05 <jose_lausuch> I mean
08:56:09 <jose_lausuch> if we have 2 pods / installer
08:56:15 <jose_lausuch> how do we reflec that in the landing page?
08:56:58 <morgan_orange> we should have more run...and less time between two runs
08:57:33 <morgan_orange> we must have 2 CI PODs per installer => it shoudl trigger jobs accordingly, shall it be reflected on functest main page?
08:58:18 <morgan_orange> I think it make sense to have 4 jobs (one per installer) for the daily job, 4 for the weekly and 4 for the custom (run one test for troubleshooting)
08:58:20 <jose_lausuch> It would be nice
08:59:44 <morgan_orange> would it not be confusing to have 2 lines per installer if we have 2 CI PODs?
09:00:00 <morgan_orange> because if we want to have weekly / daily / custom the list will be very long ...
09:00:23 <morgan_orange> do we want to test installer X with scenario Y on POD Z, or installer X with scenario Y
09:00:32 <morgan_orange> at the end POD information is suefull but not a constriant
09:01:03 <jose_lausuch> ya
09:01:04 <morgan_orange> we should have at least one pod in stable state (so today with nosdn scenario)
09:01:07 <jose_lausuch> its difficult to say
09:01:23 <jose_lausuch> but if we have 2 pods/installer, if you click on joid daily for example
09:01:25 <jose_lausuch> what do you get?
09:01:27 <jose_lausuch> both pods?
09:01:30 <morgan_orange> yes
09:01:37 <morgan_orange> in theory
09:01:49 <morgan_orange> but pretty sure that today we have only intel pod5
09:01:52 <jose_lausuch> ok
09:02:01 <jose_lausuch> for joid?
09:02:10 <morgan_orange> yes
09:02:21 <jose_lausuch> and intel-pod6 as well
09:02:26 <jose_lausuch> https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-master/54/
09:02:28 <morgan_orange> and we must also create the weekly jobs
09:02:29 <jose_lausuch> where it always fails
09:02:42 <jose_lausuch> (not always)
09:02:45 <jose_lausuch> yes
09:02:53 <morgan_orange> ok it should be visible on the dashboard...
09:03:24 <morgan_orange> I will maybe try to reconfigure my old one until ELK is operational :)
09:03:47 <jose_lausuch> :)
09:03:52 <jose_lausuch> need to leave to another meeting
09:03:54 <morgan_orange> how do you see the triggering of the weekly jobs?
09:04:01 <morgan_orange> ok we are alsready late sorry
09:04:12 <jose_lausuch> on saturday evenings :D
09:04:21 <jose_lausuch> weekly-beer job
09:04:37 <morgan_orange> yes probably the simple way
09:04:41 <morgan_orange> #topic documentation
09:04:52 <morgan_orange> CG_Nokia: made a great update of the documentation
09:04:55 <morgan_orange> I want to merge
09:04:58 <morgan_orange> any objection?
09:05:05 <jose_lausuch> morgan_orange: wait, let me review first
09:05:10 <jose_lausuch> wait for my +2 please
09:05:12 <CG_Nokia> OK with me if you are happy with patch set #4
09:05:19 <jose_lausuch> it will come soon if everything is fine with me :)
09:05:20 <morgan_orange> during the summit we were asked also to provide a dev guide
09:05:27 <jose_lausuch> #info agreed to use a different commit for the user guide
09:05:32 <CG_Nokia> Working on userguide at moment... gerrit review soon I hope!
09:05:38 <morgan_orange> so update of the dev guide will be needed, SerenaFeng you may start on the section dedicated to the API
09:05:52 <morgan_orange> thanks CG_Nokia
09:05:59 <jose_lausuch> I also need to work on the dev guide
09:06:01 <morgan_orange> #info wait for Jose +2 before merging config guide
09:06:11 <morgan_orange> #info user guide in progress
09:06:18 <morgan_orange> #info dev guide to be refactored
09:06:28 <morgan_orange> #action SerenaFeng update dev guide APi section
09:06:29 <SerenaFeng> morgan_orange add testAPI capture in dev guide?
09:06:54 <morgan_orange> yes please
09:06:57 <morgan_orange> #link https://git.opnfv.org/cgit/functest/tree/docs/devguide
09:06:57 <SerenaFeng> Ok, I will do it
09:07:05 <morgan_orange> and maybe also a section on dahboard
09:07:26 <morgan_orange> #topic AoB
09:07:35 <SerenaFeng> Ok
09:07:41 <morgan_orange> any other topic you want to share?
09:07:57 <CG_Nokia> Backroom boys say thanks for the nice  OPNFV T-Shirt from Berlin!
09:08:32 <morgan_orange> ok see you next week
09:08:37 <morgan_orange> #endmeeting