08:00:19 <morgan_orange> #startmeeting Functest weekly meeting July 5th
08:00:19 <collabot`> Meeting started Tue Jul  5 08:00:19 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:19 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:19 <collabot`> The meeting name has been set to 'functest_weekly_meeting_july_5th'
08:00:24 <morgan_orange> #topic call role
08:00:29 <morgan_orange> #info Morgan Richomme
08:00:32 <lhinds> #info Luke Hinds
08:00:38 <morgan_orange> #chair jose_lausuch
08:00:38 <collabot`> Current chairs: jose_lausuch morgan_orange
08:00:38 <juhak> #info Juha Kosonen
08:00:41 <viktor_t> #info Viktor Tikkanen
08:00:41 <raghavendrachari> #info raghavendrachari
08:00:43 <ollivier> #info ollivier
08:00:52 <morgan_orange> hello everybody
08:01:02 <morgan_orange> #agenda https://wiki.opnfv.org/display/functest/Functest+Meeting
08:01:06 <raghavendrachari> hi good day
08:01:13 <morgan_orange> #info agenda https://wiki.opnfv.org/display/functest/Functest+Meeting
08:01:23 <morgan_orange> any point you want to add on the agenda
08:01:45 <morgan_orange> I will leave at 10h45 today, José will chair the end of the meeting
08:01:56 <jose_lausuch> $info Jose Lausuch
08:02:02 <morgan_orange> in AoB I would like to share some info on intern program
08:02:03 <jose_lausuch> #info Jose Lausuch
08:02:04 <jose_lausuch> ok
08:02:24 <morgan_orange> if you have topics, and want to mentor some interns => https://wiki.opnfv.org/display/DEV/Intern-projects-page
08:02:45 <morgan_orange> #topic action point follow up
08:02:55 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-06-28-08.00.html
08:03:16 <morgan_orange> #AP1: lhinds I think secruity repo has beeen clarified
08:03:44 <lhinds> morgan_orange: 12/07 it will go to the TSC, but yes, its a given it will go ahead
08:04:03 <lhinds> just had to get them to agree between themselves
08:04:28 <morgan_orange> ok so security tests will be hosted in dedicated security repo, just minor changes to be planned regarding integration in Functest
08:04:38 <jose_lausuch> lhinds: question
08:04:49 <jose_lausuch> when you said "run the tests in functest or standalone"
08:04:56 <jose_lausuch> how do you run the standalone?
08:05:13 <jose_lausuch> can they be run without the functest framework?
08:05:31 <lhinds> setuptools install > run the tool
08:05:38 <SerenaFeng_> #info SerenaFeng
08:05:58 <JuhaHaapa> #info Juha Haapavirta
08:06:23 <morgan_orange> my view: standalone means here out of the CI but you still need a ONFV SUT
08:06:31 <jose_lausuch> lhinds: ok,
08:06:33 <morgan_orange> because in some cases standalone means no need of OPNFV...
08:06:35 <jose_lausuch> ya that was my question
08:06:42 <morgan_orange> parser, ..
08:06:58 <morgan_orange> #info AP2: apex bare metal now in functest menu  https://build.opnfv.org/ci/view/functest/
08:07:14 <lhinds> quite a number of people want to run the tool on its own, so that was the thinking behind having it standalone
08:07:35 <morgan_orange> #info we suggest to create 3 sub menus for daily / weekly/ commit gating (can be discussed later in the jenkins topic)
08:08:03 <morgan_orange> #AP3 Sprint 10 just started this morning mainly focus on feature integration
08:08:11 <jose_lausuch> and bugfixing
08:08:18 <morgan_orange> #info AP3 Sprint 10 just started this morning mainly focus on feature integration and bug fixing
08:09:00 <morgan_orange> #info AP4: I indicated during last week release meeting that we got most of the info regarding feature integration and will consider the 7th of July as last date to create Jira for that
08:09:26 <morgan_orange> #info AP5: question on how to deal Dxxx activity sent by mail, no feedback so far
08:09:45 <jose_lausuch> I think David was on vacation yesterday
08:09:54 <morgan_orange> Independance Day...
08:09:55 <jose_lausuch> and for that matter, we should bring it to Releng
08:10:12 <morgan_orange> the community was in copy of the mail
08:10:17 <jose_lausuch> the aliens are coming...
08:10:37 <morgan_orange> one more time...but US army will save us...as usual
08:10:58 <jose_lausuch> thank god we can count on them :D
08:11:13 <morgan_orange> #info AP6 report issues to scenario owners for apex/compass/joid
08:11:22 <morgan_orange> I should do the same for fuel...
08:11:26 <viktor_t> #info Analysis (speculations) regarding tempest errors in Joid/Compass are sent by e-mail to scenario owners.
08:11:34 <viktor_t> #info APEX-194 and APEX-195 Jira tickets are created.
08:11:35 <jose_lausuch> although we had blue balls in Jenkins for Fuel jobs
08:11:44 <jose_lausuch> but we should report tempest errors
08:11:55 <morgan_orange> yep
08:12:07 <viktor_t> I think Fuel has reached even 100%
08:12:38 <viktor_t> at least once
08:12:40 <morgan_orange> #info AP7 SerenaFeng started work on doc (dedicated topic planned, as great improvement from Column + feedback from bryan_att)
08:12:48 <jose_lausuch> 2016-07-04 23:05:29,012 - run_tempest - INFO - Results: {'timestart': '2016-07-0422:49:54.724808', 'duration': 932, 'tests': 165, 'errors': '', 'failures': 0}
08:12:49 <jose_lausuch> awesome
08:13:09 <morgan_orange> good indication for others...it is possible! :)
08:13:36 <morgan_orange> #info AP8 done already mentioned by viktor_t
08:14:03 <morgan_orange> #topic trust Indicator
08:14:11 <morgan_orange> we discussed it during the summit
08:14:36 <morgan_orange> we created this indicator to consider the repeatability of the success criteria for long duration tests
08:14:46 <morgan_orange> we created a field in result data model
08:14:59 <morgan_orange> but as mentioned by SerenaFeng we were losing the history
08:15:18 <morgan_orange> #link https://jira.opnfv.org/browse/FUNCTEST-309
08:15:45 <morgan_orange> SerenaFeng suggested to add this history in addition of the current value of the trust indicator
08:16:08 <morgan_orange> trust indicator will look like { current: 0.6, /* current trust_indicator value, will be used for all cases */ histories: [("2016-06-30 12:02:21", 0.2)], /* only used for long duration cases
08:16:33 <jose_lausuch> well, we will have weekly jobs soon
08:16:34 <morgan_orange> any objection/comments?
08:16:43 <jose_lausuch> they will already run long duration cases
08:16:58 <jose_lausuch> no objections
08:17:02 <morgan_orange> jose_lausuch: yes but probably not all the tests all the time
08:17:10 <morgan_orange> that is part of my answer to Fatih
08:17:42 <SerenaFeng_> and, As to keep the last 10 or 20 values
08:17:52 <morgan_orange> even if we have weekly jobs, it does not mean that we will runn all the long duration tests (otherwise it will be very long) this mechanism is there to help us to run long duration tests
08:18:11 <jose_lausuch> ok
08:18:18 <morgan_orange> yep last open point was about the history, how many values shall we keep
08:18:24 <morgan_orange> I think 10 are enough
08:18:30 <morgan_orange> if not we will adapt later
08:18:35 <SerenaFeng_> will each time we run a Case, we will push a new result record to DB?
08:18:44 <morgan_orange> but it should already give a good indication on the evolution of the test
08:19:03 <morgan_orange> each time we run a case from CI yes
08:19:14 <morgan_orange> but a case = f (pod, scenarios, installer, version)
08:19:43 <morgan_orange> when we need history we need history for a case assuming we know all the parameters
08:20:03 <SerenaFeng_> in this case, I don't think we need to worry about the maximum number of histories
08:20:23 <SerenaFeng_> As to keep the last 10 or 20 values, since each time we run a Case, we will push a new result record to DB, and the update will operate on the new record, so the maximum history number will be 0.2/0.05=4, so I don't think we need to worry about that.
08:20:51 <morgan_orange> the history is usefull in the case you did not run the case...if you run systematically it is useless
08:21:01 <morgan_orange> ok so do not worry :)
08:21:04 <SerenaFeng_> yes
08:21:17 <morgan_orange> it is true that 10 CI weekly loop without running, it will be suspicious..
08:21:27 <SerenaFeng_> so, only when -0.05 happened, we will update ti
08:21:42 <morgan_orange> #topic Web portal consuming test API
08:21:52 <morgan_orange> raghavendrachari:  any update?
08:22:10 <morgan_orange> #link https://jira.opnfv.org/browse/FUNCTEST-26
08:22:14 <raghavendrachari> yeah displaying testcases pending , will complete eod
08:22:23 <morgan_orange> some snapshot are attached in the Jira
08:22:40 <morgan_orange> the idea is to do something even simpler that the  swagger documentation allows to do
08:22:47 <morgan_orange> just a form and a button to click
08:22:58 <raghavendrachari> ok
08:23:08 <morgan_orange> you should however plan a basic/auth to avoid robot
08:23:30 <morgan_orange> any comment/remark?
08:23:44 <morgan_orange> we will host it on testresults.opnfv.org
08:23:47 <raghavendrachari> ok , where can i check authnetication ?
08:23:49 <raghavendrachari> any db ?
08:24:56 <morgan_orange> up to you, if you want we can just add a nginx basic/auth or you use a param in DB or in a file or you use LF authentication module if any
08:25:14 <raghavendrachari> ok
08:26:10 <morgan_orange> #topic ODL refactoring
08:26:20 <morgan_orange> ollivier: any update?
08:26:26 <ollivier> #info the modifications we have done for days on Robotframework files have been backported to ODL.
08:26:38 <ollivier> Then 010__networks.robot, 020__subnets.robot and 030__ports.robot were removed from the OPNFV functest repository.
08:26:51 <ollivier> There was no reason to keep on forking these test cases from the upstream project.
08:27:03 <ollivier> #link https://git.opendaylight.org/gerrit/#/c/40948/
08:27:23 <morgan_orange> good practice..upstream, upstream, upstream :)
08:27:56 <ollivier> we could also move the delete operation test cases currently hosted in the OPNFV Functest project to ODL.
08:28:04 <ollivier> The question was asked to the ODL community
08:28:18 <morgan_orange> makes sense for me
08:28:47 <morgan_orange> any issue regarding the planning to finalize the refactoring and get a success criteria which would make more sense that the previosu test suite?
08:29:47 <ollivier> We already have applied some patches to ensure delete operation testing
08:29:47 <jose_lausuch> nope
08:30:08 <ollivier> #link https://gerrit.opnfv.org/gerrit/#/c/15827/
08:30:45 <ollivier> The next (last?) action regarding this topic is to replace start_tests.sh by a python script.
08:30:50 <morgan_orange> ok
08:31:05 <ollivier> It must strongly rely on the Robotframework library.
08:31:15 <morgan_orange> ok
08:31:17 <ollivier> If you agree, I'll take the action.
08:31:32 <ollivier> #action ollivier write the python script relying on the Robotframework library
08:31:46 <morgan_orange> note the last odl results reported to DB http://testresults.opnfv.org/test/api/v1/results?case=odl&last=10 are now quiete old and will lead to a poor scoring on odl scenraio
08:31:59 <morgan_orange> I can remove temporarily (idem for doctor )
08:32:15 <morgan_orange> ollivier: you are master on bord of the ODL ship :)
08:32:29 <ollivier> morgan_orange: ok :)
08:32:39 <AntonySilvester> @jose :  https://gerrit.opnfv.org/gerrit/16311    ---> please let me know, when this piece of code will be merged  ?
08:32:39 <collabot`> AntonySilvester: Error: "jose" is not a valid command.
08:32:45 <morgan_orange> but very good point to have this connection with odl testing community
08:32:58 <morgan_orange> #topic COlorado status
08:33:30 <morgan_orange> AntonySilvester: status for odl sfc => I just merged
08:34:23 <morgan_orange> you probably need to declare onosfc in the feature section of https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
08:34:42 <morgan_orange> with constarints on scenario onos and sfc
08:35:06 <morgan_orange> on security scan, I showed some exchanges between lhinds and jose_lausuch this morning
08:35:09 <AntonySilvester> ok sure. will update on that
08:35:50 <morgan_orange> last apex run https://build.opnfv.org/ci/view/functest/job/functest-apex-apex-daily-master-daily-master/lastFailedBuild/consoleText
08:35:59 <jose_lausuch> AntonySilvester: sure, this is ready to merge I think
08:36:09 <morgan_orange> jose_lausuch: I merged :)
08:36:34 <morgan_orange> 2 patches have been created...I think lhinds was affraid of the famous ollivier -1 :)
08:36:54 <jose_lausuch> morgan_orange: lhinds: I tried to merge one
08:37:09 <jose_lausuch> https://gerrit.opnfv.org/gerrit/#/c/16339/
08:37:11 <jose_lausuch> but cant merge
08:37:28 <morgan_orange> second one https://gerrit.opnfv.org/gerrit/16337
08:37:36 <AntonySilvester> morgan has already merged. thanks :)
08:38:12 <morgan_orange> jose_lausuch: that is because of the dependency towards 16337...
08:38:32 <ollivier> jose_lausuch: AntonySilvester: about #16311. You should have removed osbolete code
08:40:02 <lhinds> i think i need to rebase
08:40:08 <morgan_orange> yep old code in # is not a flake8 violation but itis not useful
08:40:37 <morgan_orange> BTW I asked LF to configure jenkins to give -1 on flake8 violation
08:40:51 <ollivier> great.
08:41:08 <morgan_orange> any other comment/remark on Colorado activities?
08:41:17 <SerenaFeng_> I think we need to add trailing space check in jenkins
08:41:24 <ollivier> It will be great to announce git reviews in irc too.
08:41:28 <ollivier> #agreed
08:41:34 <SerenaFeng_> always forget to do that check, awkward
08:41:41 <morgan_orange> IRC review on IRC is also an old request we have
08:42:04 <morgan_orange> basically we created opnfv-functest a long time ago and said we will use it when the gerrit robot will be setup...
08:42:09 <morgan_orange> it is not yet the case
08:42:20 <morgan_orange> #action morgan_orange ask the status of gerrit robot in opnfv-functest
08:42:31 <SerenaFeng_> is opnfv-functest in use now?
08:42:46 <morgan_orange> I proposed we move to opnfv-functest (we got a remark that it was easy to find us here rather than on testperf...)
08:42:57 <morgan_orange> opnfv-functest does exist but was not used
08:43:01 <morgan_orange> I suggest to switch
08:43:23 <morgan_orange> #action morgan_orange document change from opnfv-testperf to opnfv-functest
08:44:10 <AntonySilvester> opnfv-functest sound meaningful and easy to remember
08:44:25 <morgan_orange> yes that is why..next meeting will be there
08:44:32 <AntonySilvester> okay
08:44:35 <morgan_orange> assuming that functest is now well known in the community
08:44:38 <morgan_orange> it is time for me to leave, jose_lausuch I let you the topic on jenkins (mainly review Fatih's proposal) and on Doc (discussion on the how tos and the last version of userguide)
08:44:42 <morgan_orange> #topic Jenkins
08:44:46 <morgan_orange> sorry I have to go
08:44:57 <morgan_orange> #chair viktor_t
08:44:57 <collabot`> Current chairs: jose_lausuch morgan_orange viktor_t
08:45:48 <jose_lausuch> ok
08:45:52 <jose_lausuch> I was in a call sorry
08:46:16 <jose_lausuch> #info proposal to have installer weekly jjobs
08:46:27 <jose_lausuch> #info   https://wiki.opnfv.org/display/INF/CI+Evolution
08:47:11 <jose_lausuch> #info Morgan created a patch to activate functest weekly jobs already, that will be triggered by he (still non existing) installer weekly jobs https://gerrit.opnfv.org/gerrit/#/c/16245/
08:48:23 <jose_lausuch> #info the idea is also to provide a weekly deployment
08:48:50 <jose_lausuch> #info any new patch in functest will be gated and run thorugh that deployment to validate the patch
08:49:01 <jose_lausuch> at least that it doesn't destroy the natural execution of things
08:49:11 <jose_lausuch> any comments on that? do you think its a good idea?
08:49:20 <jose_lausuch> I have bad experiences breaking the CI before :)
08:49:44 <JuhaHaapa> What installer would it use?
08:49:58 <JuhaHaapa> Anyway an excellent idea
08:50:23 <jose_lausuch> we don't know yet, we should select one
08:50:34 <jose_lausuch> the idea is not to verify the installer, its to verify functest basic operations
08:50:48 <jose_lausuch> like creating the container, running prepare env, and the healthceck for example
08:50:59 <JuhaHaapa> to get the maximum # of error, use every one in turn :)
08:51:31 <jose_lausuch> ya, but if it's a jenkins gate to verify a patch, it could take too long
08:51:37 <jose_lausuch> I would say we dont care which installer we use
08:51:42 <jose_lausuch> the one that works that week
08:51:47 <jose_lausuch> a stable build
08:51:51 <viktor_t> so you will recheck until you'll get the right installer/scenario :)
08:51:53 <jose_lausuch> where we can run our verification on
08:52:06 <jose_lausuch> viktor_t: ?
08:52:14 <jose_lausuch> I think it should be a nosfn-nofeature
08:52:18 <viktor_t> just kidding...
08:52:22 <jose_lausuch> :)
08:52:52 <jose_lausuch> I hope we can have that feature soon
08:52:56 <jose_lausuch> it would be great
08:53:31 <jose_lausuch> ok
08:53:32 <jose_lausuch> moving on
08:53:36 <jose_lausuch> #topic Documentation
08:53:52 <jose_lausuch> #info big patch from Colum, already merge. However, a few things to fix
08:53:54 <jose_lausuch> for example
08:54:02 <jose_lausuch> lhinds: can you provide a description for the security testing?
08:54:35 <lhinds> jose_lausuch: will put it on my list for today
08:54:37 <jose_lausuch> can I AP you?
08:54:39 <jose_lausuch> ok
08:54:44 <JuhaHaapa> there were some TODO which can/should be merged separately
08:54:46 <jose_lausuch> its not urgent though
08:55:06 <jose_lausuch> #AP lhinds update userguide security-scan description
08:55:18 <jose_lausuch> we also need Doctor description
08:55:38 <jose_lausuch> #AP jose_lausuch morgan_orange ask Doctor folks to provide a brief description for our user guide
08:55:47 <JuhaHaapa> ...and ocl
08:55:56 <jose_lausuch> yes, but we disabled ocl
08:56:04 <jose_lausuch> does anyone know the status of contrail ?
08:56:21 <jose_lausuch> is it already running in CI ? the test freeze is end of this week, and we don't haev anything yet
08:57:45 <jose_lausuch> #AP jose_lausuch morgan_orange check status of OCL, but for now it is out of scope in functest (not running OCL test in CI)
08:57:59 <jose_lausuch> we have 3 minutes for AoB
08:58:08 <jose_lausuch> SerenaFeng I'm checking your dockerfile
08:58:55 <jose_lausuch> #info SerenaFeng created  a new dockerfile to run the TestAPI service https://gerrit.opnfv.org/gerrit/#/c/16349/
08:59:19 <jose_lausuch> let's use ubuntu 14.04 for now, as we do in Functest (we now it works stable)
08:59:43 <jose_lausuch> any other business?
09:00:09 <viktor_t> not here...
09:00:20 <jose_lausuch> ok, we can close then
09:00:50 <jose_lausuch> back to work then :)
09:00:51 <jose_lausuch> #endmeeting