08:00:31 <morgan_orange> #startmeeting Functest weekly meeting September 20th Colorado-2
08:00:31 <collabot> Meeting started Tue Sep 20 08:00:31 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:31 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:31 <collabot> The meeting name has been set to 'functest_weekly_meeting_september_20th_colorado_2'
08:00:37 <morgan_orange> #topic call role
08:00:43 <ollivier> #info ollivier
08:00:47 <viktor_t> #info Viktor Tikkanen
08:00:47 <morgan_orange> #info morgan_orange
08:00:58 <JuhaHaapa> #info Juha Haapavirta
08:01:04 <CG_Nokia> #info CG_Nokia (Colum Gaynor)
08:01:07 <morgan_orange> #info agenda https://wiki.opnfv.org/display/functest/Functest+Meeting
08:01:16 <morgan_orange> anything you want to add to the agenda for today?
08:01:16 <SerenaFeng> #info SerenaFeng
08:01:34 <morgan_orange> I will speak admin & organiation in the AoB
08:01:36 <jose_lausuch> #info Jose Lausuch
08:01:50 <morgan_orange> #topic Action point follow-up
08:01:53 <lhinds> #info Luke Hinds
08:01:59 <morgan_orange> #info AP1: morgan_orange sync with apex/onos
08:02:05 <morgan_orange> #info mail sent, discussions during the last release meeting, paex contacted onos integration team
08:02:10 <morgan_orange> #info AP2: morgan_orange wait for merge before enabling it in reporting
08:02:15 <morgan_orange> #info done
08:02:20 <morgan_orange> #info AP3: raghav review yardstick
08:02:25 <morgan_orange> #info done
08:02:30 <morgan_orange> #info AP4: morgan_orange check arm ref and add them in the doc review for CristinaPauna
08:02:41 <morgan_orange> #info done section on ARm results added in release note
08:02:46 <morgan_orange> #info AP5: morgan_orange prepare pool for Functest meetup in Lannion
08:02:51 <morgan_orange> #link https://framadate.org/sK56myIDT6KGeRNv
08:03:04 <morgan_orange> #info see next topic
08:03:09 <morgan_orange> #info AP6: morgan_orange contact bitergia / ELK
08:03:20 <morgan_orange> #info first contact, possible meeting in Barcelona, first exchanges proposed during the test weekly meeting
08:03:29 <morgan_orange> any comment on the action point?
08:03:42 <jose_lausuch> nope
08:03:50 <morgan_orange> #topic Colorado Status
08:04:04 <morgan_orange> #info Daily Release meeting now
08:04:26 <morgan_orange> #info last Friday Release manager say we were still on track for the 22nd (so tomorrow)
08:04:50 <jose_lausuch> tomorrow?
08:04:53 <jose_lausuch> thursday
08:04:56 <jose_lausuch> :)
08:05:04 <CristinaPauna> #info Cristina Pauna (Enea)
08:05:06 <morgan_orange> #info from Functest perspective, the status is green as most of the scenarios have a high score and the errors are known
08:05:08 <SerenaFeng> the day after tomorrow
08:05:17 <morgan_orange> yep
08:05:39 <morgan_orange> #info release note: http://artifacts.opnfv.org/functest/colorado/docs/release-notes/index.html
08:06:25 <morgan_orange> #info note the mention to Arno is due to the label on master (see explanation from fdegir - several options possible new tag or use a variable)
08:06:58 <jose_lausuch> how to use that variable?
08:07:40 <morgan_orange> nobody answers to fdegir I think it would be like an env variable by opnfvdocs instead of relying on last tag on master
08:08:10 <morgan_orange> #action morgan_orange answer to fdegir for the tag/master triggering doc issue for "old" projects
08:08:11 <jose_lausuch> ok
08:08:34 <morgan_orange> regarding the scenarios, do you see problems
08:09:02 <jose_lausuch> problems that are not reported?
08:09:05 <morgan_orange> apex/onos is a regression egarding brahmaputra but people have been contacted
08:09:32 <morgan_orange> there are still some issues in some scenarios with tempest
08:09:43 <morgan_orange> but I think the issues are captured in the release note
08:10:09 <jose_lausuch> yes
08:10:21 <jose_lausuch> I think the release notes captures everything we have observed
08:10:23 <jose_lausuch> so it should be ko
08:10:25 <jose_lausuch> ok
08:10:25 <morgan_orange> viktor_t: you are in line with this statement?
08:10:39 <viktor_t> yes
08:10:42 <viktor_t> BTW, will we empty blacklist for D release?
08:10:45 <jose_lausuch> I will also update the sdnvpn project release notes since we disabled parser for bgpvpn
08:11:11 <raghav> #info raghavendrachari
08:11:13 <jose_lausuch> viktor_t: we wanted 100% working for smoke for Colorado
08:11:23 <morgan_orange> I think it makes sense to empty black list for new release
08:11:23 <SerenaFeng> so about that, why disable parser from bgpvpn?
08:11:26 <jose_lausuch> so... I dont want to say it :)
08:11:57 <jose_lausuch> SerenaFeng: due to a performance issue we have in that scenario, it is captured in the release notes, we also disabled rally
08:12:14 <SerenaFeng> ok
08:12:16 <jose_lausuch> SerenaFeng: it is not the best solution, but it is to be re-enabled for Colorado 2.0
08:12:27 <jose_lausuch> SerenaFeng: ugly workaround :)
08:12:54 <morgan_orange> as a reminder the scenario owner is master on board regarding the integration of additional feature tests
08:13:16 <SerenaFeng> ok, I just need to know is it because of parser problem. if it is, I need to ask Parser people to fix it
08:13:25 <jose_lausuch> yes, but there should be a good reason behind and documented
08:13:31 <morgan_orange> if he/she wants to focus on its feature, he/she may ask to remove some tests, it is captured in the release note, the highest the scoring his the more features are tested
08:13:55 <jose_lausuch> SerenaFeng: it isn't because of parser, it's due to the bgpvpn extension causes some troubles that we can't reproduce locally and think will be fixed with ODL Boron
08:14:11 <morgan_orange> #info scenarios OK: globally good scoring + errors documented in release note
08:14:20 <morgan_orange> let's review the Open JIRAs
08:14:26 <SerenaFeng> ok, I understank, thank you
08:14:47 <morgan_orange> in the release note we mentioned 419, 446, 450, 454, 460 and 462
08:15:11 <morgan_orange> if you look at https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=59
08:15:26 <morgan_orange> 419 is the issue on docker we did not reproduce (empty tag)
08:15:50 <morgan_orange> 446 is on odl-sfc cleanup (as far as I know, scenarios are even not run so not cleaned)
08:15:51 <jose_lausuch> I dont have logs to see how I can solve it
08:16:05 <morgan_orange> 450 it was a bug
08:16:14 <morgan_orange> not sure it was reproducible
08:16:15 <ollivier> 462 is not related to functest
08:16:33 <jose_lausuch> ollivier: right
08:16:40 <jose_lausuch> it's related to the deployment
08:16:43 <jose_lausuch> I will close it
08:16:46 <CristinaPauna> we saw this 462 on ARM as well
08:16:52 <jose_lausuch> is it tok?
08:17:18 <ollivier> I think this is an ODL issue
08:17:25 <jose_lausuch> yes
08:17:33 <jose_lausuch> we are reproducing that in our labs
08:17:34 <jose_lausuch> but you are right
08:17:42 <morgan_orange> so I suggets we clean this one and invite the reporter to open a bug upstream
08:17:42 <ollivier> please send me karaf.log
08:17:48 <jose_lausuch> functest shows the appropriate output, if it fails, it fails
08:18:00 <jose_lausuch> I'll close it
08:18:08 <morgan_orange> ok summary
08:18:28 <morgan_orange> 419, 450 will be close (not reproducible)
08:18:39 <morgan_orange> 462 closed and reported upstream
08:19:05 <ollivier> #agreed
08:19:56 <morgan_orange> jose_lausuch: for 446, we keep it open even if we have no more regular run on odl_l2-sfc?
08:20:32 <jose_lausuch> morgan_orange: we can move that to colorado 2.0
08:20:39 <jose_lausuch> is that ok?
08:20:47 <morgan_orange> for 454 it is also a failure on cleanup  so for colorado 2.0
08:20:51 <jose_lausuch> I think it is not an urgent think of colorado 1.0
08:21:11 <morgan_orange> 460 it is assigned to Juha
08:21:18 <morgan_orange> any idea of the status?
08:21:34 <morgan_orange> #info 419,450 will be closed (not reproducible)
08:21:41 <morgan_orange> #info 462 closed and reported upstream
08:21:56 <jose_lausuch> I just closed 450 with a message that I think it solves the issue
08:22:09 <morgan_orange> #info 446 and 454 set to Colorado 2.0
08:22:20 <morgan_orange> #action juha give feedback on 460
08:23:07 <morgan_orange> SerenaFeng: I did not mention 465 and 474 in the release note (kibana enhancement), are you OK with that
08:23:26 <morgan_orange> and of course 434 and 436 (doc review will also not been considered for release note)
08:23:52 <morgan_orange> #info Jira status pretty clear: ready for release
08:24:03 <morgan_orange> any other ocmment/remark for Colorado?
08:24:32 <SerenaFeng> ok, it is D Release work
08:24:46 <morgan_orange> hello juhak we had a quesiton on the status of https://jira.opnfv.org/browse/FUNCTEST-460
08:25:41 <morgan_orange> #topic update on Kibana dashboard
08:25:50 <morgan_orange> SerenaFeng: you want to give an update?
08:26:02 <morgan_orange> #info dashboard picture changed in release note (Kibana base)
08:26:03 <juhak> morgan_orange: that's done, I'll close the case
08:26:18 <morgan_orange> juhak: OK I will remove the JIRA from the release note
08:26:25 <SerenaFeng> I am working on refactoring the current code structure
08:26:29 <morgan_orange> #action juhak close JIRA 460
08:26:37 <morgan_orange> #action morgan_orange update release note / JIRAs
08:27:08 <SerenaFeng> as to the display content we want to refactor, I'd like to do it after the discuss with bitgia people
08:27:08 <morgan_orange> #link http://artifacts.opnfv.org/functest/colorado/docs/userguide/_images/FunctestDashboardColorado.png
08:27:40 <morgan_orange> we contacted bitergia people who are in charge of the reporting for git/gerrit/irc/mailing list and soon jenkins
08:27:54 <SerenaFeng> yes, I get the email
08:27:56 <morgan_orange> they moved to ELK (presentation during the Summit in Berlin)
08:28:17 <morgan_orange> the guy will confirm if he can attend the test weekly meeting on Thursday
08:28:21 <SerenaFeng> and I think due to my English level, a face-to-face meeting is a good idea
08:28:32 <morgan_orange> if so would you be OK to prepare some slides on the status on Functest side?
08:28:47 <SerenaFeng> because I am afraid during the test meeting, maybe I cannot catch up with his idea
08:28:52 <SerenaFeng> and sorry about that
08:29:15 <morgan_orange> SerenaFeng: do not be sorry our Chinese is very very poor...
08:29:44 <morgan_orange> #action SerenaFeng prepare some slide on Kibana dashboard to share with bitergia people and the community
08:29:45 <jose_lausuch> my Chinese is None :)
08:30:02 <morgan_orange> I hope jesus could join (he is based in Madrid)
08:30:09 <morgan_orange> for an F2F meeting in barcelona
08:30:13 <jose_lausuch> action SerenaFeng teach Chinese to morgan_orange and jose_lausuch
08:30:15 <SerenaFeng> will the slides be used on this Thur.?
08:30:31 <SerenaFeng> teaching English no problem
08:30:46 <morgan_orange> possibly, it is just a status, no need to do something nice
08:30:56 <SerenaFeng> Teaching Chinese no problem
08:31:15 <morgan_orange> we shall plan a session during the next summit in Beijing..
08:32:03 <SerenaFeng> ok, slides, I will try
08:32:19 <SerenaFeng> Welcome to Beijing
08:32:21 <morgan_orange> The guy from bitergia shared a link with the new dashboard, it looks great and apparenly they are interested by the tests
08:32:39 <morgan_orange> SerenaFeng: next year..for the OPNFV Summit we will be in Beijing
08:32:42 <SerenaFeng> I can treat you some authentic delicious Chinese food
08:32:48 <SerenaFeng> :0
08:32:51 <SerenaFeng> :)
08:32:54 <morgan_orange> you will be our guide
08:33:12 <morgan_orange> any question on ELK?
08:33:27 <morgan_orange> #topic Discussion on test criteria and exit condition
08:33:55 <morgan_orange> #info discussion in gerrit about a bug that was not a bug but showed that we were not fully aligned in the different test cases on the exit conditions
08:34:14 <morgan_orange> I think we should agree and try to harmonize a little bit
08:34:28 <morgan_orange> exit -1 only allowed from main
08:34:47 <ollivier> #agreed
08:35:00 <jose_lausuch> agree
08:35:09 <jose_lausuch> we need to do some refactor for Danube
08:35:16 <morgan_orange> shall we consider if test OK but pushing data to the database fails we also consider that the test fails as pushing is part of the test
08:35:30 <jose_lausuch> because I have detected some scripts (including some I wrote) that use "exit" in functions, not in main
08:35:44 <morgan_orange> in the case of ARM, their POD was not declared so the jenkins job was -1, which was normal because pushing data was part of the test...
08:35:52 <SerenaFeng> vping, I do it a lot :)
08:36:02 <jose_lausuch> that is something to discuss
08:36:06 <SerenaFeng> because it is very convenient to do it, sorry
08:36:12 <morgan_orange> #action all Danube review the exit conditions...no more exit -1 out of the main...
08:36:14 <ollivier> I think we exit -1 if push to db arg is set even if tests are ok
08:36:22 <jose_lausuch> if a user runs the test, the push-to-db will not be used and the test works
08:36:39 <jose_lausuch> so my opinion is that if push-to-db fails don't mark the test failed
08:37:03 <morgan_orange> hmm from a humane perspective I would say yes, but from a CI i would say no
08:37:18 <jose_lausuch> because it could be due to DB offline, something else's problem than functest
08:37:23 <morgan_orange> what we want is test + push to DB => if somethign is wrong we must report it
08:37:33 <ollivier> So arg should be unset
08:37:40 <morgan_orange> DB offline is a possible issue it is thus interesting to know it
08:37:57 <jose_lausuch> but we will see it in jenkins console
08:38:36 <jose_lausuch> its good feedback for us, but a false negative for the scenario/test owner
08:38:39 <morgan_orange> if we have blue ball...we could miss it...not possible to read all the consoles of all the tests...we will see it in the reporting page
08:38:43 <SerenaFeng> my opinion, just logger.error, when push to db failed
08:38:55 <ollivier> it's amazing
08:38:57 <jose_lausuch> yes, I would say so too
08:39:01 <CristinaPauna> I think functest's purpose is to test functionality of the stack, this push-to-db is just infrastructure
08:39:14 <jose_lausuch> I constantly check jenkins logs and look for "error" :)
08:40:21 <morgan_orange> ollivier: what is amazing?
08:41:19 <ollivier> morgan_orange: if u ask for a report which failed, the script should exit in error
08:42:03 <ciprian-barbu> guys, there might be a bit of a trap in releng as well, when manually triggering the job you can opt to push results to DB
08:42:25 <SerenaFeng> can we distinguish test failure from push-to-db failure from the reporting?
08:42:37 <ciprian-barbu> that person will have to know that setting it for a non-CI pod will potentially break functest
08:42:55 <SerenaFeng> I mean exit with different error code
08:43:53 <morgan_orange> ollivier: the issue is that the exit -1 is fine for us (what we want failed so we exit -1) but misleading for the scenario owner (their tests were OK) and can be considered as a false negative for them
08:44:51 <jose_lausuch> and if the reporting fails because DB is offline for example, only vping will be executed and will block all the rest
08:44:55 <ciprian-barbu> I think there are two sides of this storry, first is the case where the user tries to push results for a non-CI pod, second is failure to push results (for different reasons) for a CI POD
08:45:00 <SerenaFeng> so I suggest, exit -1 for test failure, exit -2 for push-to-db failure,
08:45:26 <jose_lausuch> ciprian-barbu: a regular user can't push results to db
08:45:52 <ciprian-barbu> jose_lausuch: I meant the job, ofc
08:46:16 <SerenaFeng> only -1 will block the rest tests, or else the rest tests will be continued
08:46:28 <ollivier> jose_lausuch:  we can if we modify the db url
08:47:51 <jose_lausuch> ollivier: ok
08:49:07 <morgan_orange> ok I think I got the different views, it is really depending if we have a dev view or integrator view
08:49:26 <morgan_orange> let's mature it and decide next week, not rush and could be linked to next topic on refactoring
08:49:57 <morgan_orange> I suggest we close the JIRA anyway and abandon the change on gerrit, the fix (good one) consisted in declaring the POD in the DB
08:50:25 <morgan_orange> are you OK with that?
08:50:36 <morgan_orange> #topic Danube: Discussion on framework refactoring / SNAPS
08:50:42 <jose_lausuch> ok
08:50:43 <morgan_orange> as said it could be connected
08:50:44 <jose_lausuch> yes
08:51:05 <morgan_orange> would it make sense to create a class with the exceptions (including DB_not_available)
08:51:12 <morgan_orange> to clean a little bit our exit conditions?
08:51:38 <ciprian-barbu> morgan_orange: sounds like a good idea if you were to ask me
08:52:08 <morgan_orange> if we consider a more Object oriented approach of each test case, it shall be possible to create a list of exceptions and have a better management that wild exit in the middle of the code...
08:53:06 <morgan_orange> I shared some weeks ago this figure https://wiki.opnfv.org/download/attachments/2925495/Functest-feature-project-interfaces.png?version=1&modificationDate=1473091940000&api=v2
08:53:16 <morgan_orange> it was my first contribution to the reflexion for the refactoring
08:53:39 <morgan_orange> I think we should also imagine the APIs we need to declare the tests and run them
08:53:59 <morgan_orange> we have an API for test result reporting we should have a more global approach
08:54:12 <morgan_orange> it would be also interesting to discuss with the other test project
08:54:48 <morgan_orange> we could imagine for instance yardstick cally Functest APi to deploy a VNF and run their perfo tests
08:55:05 <morgan_orange> and we could imagien also calling Yardstick APi from our docker to run perfo tests
08:55:25 <morgan_orange> it is still a bit fuzzy but I think there is a risk to create // framework that will do the same
08:55:44 <morgan_orange> so we need a better view on the API we coudl offer to scenario owners and other test projects
08:56:07 <morgan_orange> What do you thin?
08:56:09 <morgan_orange> Did I smoke too much?
08:56:47 <morgan_orange> regading SNAP integration, we exchange with Steve from Cable labs
08:57:25 <morgan_orange> main scenario consider SNAPS as an third party library - coexistence with functest, openstack_utils  for D, E and let darwinain selection happens
08:57:52 <morgan_orange> steve ask if we have objection to consider as an upstream lib for Functest
08:58:13 <morgan_orange> I do not see objection but jose_lausuch you said that LF wanted to limit that for legal issues
08:58:50 <morgan_orange> OK I am bit lonely - probably smoke too much -
08:58:54 <jose_lausuch> sorry, parallel meeting
08:59:14 <morgan_orange> we are already late
08:59:18 <morgan_orange> #topic meetup
08:59:24 <SerenaFeng> sorry, I need to digest your idea first
08:59:30 <jose_lausuch> I agree, what I was thinking is having SNAPS as another way to do things than openstack utils
08:59:31 <jose_lausuch> sorry
08:59:40 <morgan_orange> #info not lots of answers yet for a possible meetup
08:59:54 * jose_lausuch pending for approval :)
09:00:17 <morgan_orange> #info meeting in Barcelona (Openstack Summit) but another functest meetup is still possible
09:00:36 <morgan_orange> #link https://framadate.org/sK56myIDT6KGeRNv
09:00:43 <morgan_orange> #info any news from Nokia side?
09:01:21 <morgan_orange> #topic AoB
09:01:51 <morgan_orange> #info I exchanged with the TSC. I will launch an internal election for Functest PTL
09:02:17 <morgan_orange> #info all committers can vote, need to send the info by mail and plan a vote on IRC
09:02:37 <morgan_orange> #info the idea is to organize such elections every 2 release and limit the mandate to 2
09:02:51 <jose_lausuch> limit the mandate to 2?
09:02:52 <morgan_orange> #info I planned to be candidate for this first one but will no more eligible for the next one
09:02:52 <jose_lausuch> ah ok
09:03:02 <jose_lausuch> so a PTL can be PTL only during 2 releases
09:03:08 <morgan_orange> yes
09:03:20 <morgan_orange> so new ideas, new energy, less smoke..
09:03:35 <morgan_orange> any objection?
09:03:43 <SerenaFeng> 2 release = 1 year?
09:04:02 <morgan_orange> yes
09:04:13 <morgan_orange> you think it is too short?
09:04:31 <morgan_orange> #action morgan_orange send mail to initiate PTL election for Functest
09:05:03 <jose_lausuch> I think its ok
09:05:04 <morgan_orange> #action all brainstorm on D refactoring API definitions, exceptions,...
09:05:19 <morgan_orange> any other topic you would like to share in the AoB
09:05:49 <SerenaFeng> nope
09:05:51 <morgan_orange> #link https://wiki.opnfv.org/display/functest/Functextnexttaks
09:06:27 <morgan_orange> OK thanks
09:06:44 <morgan_orange> enjoy last Colorado days and first Danube days
09:06:50 <morgan_orange> #endmeeting