08:00:12 <morgan_orange> #startmeeting Functest weekly meeting August 16th
08:00:12 <collabot> Meeting started Tue Aug 16 08:00:12 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:12 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:12 <collabot> The meeting name has been set to 'functest_weekly_meeting_august_16th'
08:00:16 <morgan_orange> #topic call roll
08:00:19 <morgan_orange> #info Morgan
08:00:26 <viktor_t> #info Viktor Tikkanen
08:00:28 <viktor_t> (must leave already at 8.30 UTC)
08:00:42 <juhak> #info Juha Kosonen
08:00:48 <lhinds> #info Luke Hinds
08:01:01 <morgan_orange> #topic Action point follow-up
08:01:09 <morgan_orange> #info AP morgan_orange check with Domino project: sync done, tests run but skipped on joid (domaino waiting method to get IP)
08:01:15 <morgan_orange> #info AP morgan_orange reference this page from Brahamaputra page only: done
08:01:31 <morgan_orange> #info AP morgan_orange for Coloradao put the URl to swagger: done
08:01:37 <morgan_orange> #info AP morgan_orange contact bryan_att to sync on this issue: done copper ok on joid still issue on apex - JIRA updated
08:01:47 <morgan_orange> #info AP all formalize the condition for tempest smoke suite case exclusion (may initiate a wiki page for that) - not done but question valid as several scenarios no fully OK and exlusion list for onos submitted as a patch
08:02:12 <morgan_orange> #action morgan_orange viktor_t create a wiki page to explain how to justify case exclusion for Tempest
08:02:20 <morgan_orange> #info AP morgan_orange ask for x2 on storage capability for the testresult VM: done
08:02:26 <morgan_orange> #info AP morgan_orange see if we should buy (OPNFV has money..) an elasticsearch licence: done elascrtic search cloud licence to be purchased
08:02:33 <morgan_orange> #info AP morgan_orange contact onos-sfc to discuss this option: mail sent, no feedback received yet
08:02:40 <morgan_orange> #info AP SerenaFeng contact Doctor team to support integration: done tests run manually, troubleshooting in progress on doctor side
08:02:47 <morgan_orange> #info AP CG_Nokia organize webex to discuss remaining topic on documentation: done
08:02:54 <morgan_orange> we will discuss doc after
08:03:00 <morgan_orange> #info AP morgan_orange contact releng team to see if we could have weekly run on Sunday to give a chance to vIMS, Tempest full and Rally full: done but no chance due to resource management, long duration tests can be run on Community labs
08:03:07 <morgan_orange> #info AP morgan_orange initiate mail for bitergia related to test results: ne feedback in the elasticsearch thread
08:03:17 <morgan_orange> #info AP morgan_orange invite hongbo to present Dovetail update to Functest team: will be done during the test weekly meeting
08:03:23 <CG_Nokia> #info CG_Nokia
08:03:43 <morgan_orange> #info AP all brainstorm on intern proposals: I sent some proposals mail to CENGN => dedicated topic after
08:03:44 <MatthewLi> #info Jun Li
08:03:47 <JuhaHaapa> #info Juha Haapavirta
08:03:58 <morgan_orange> that is all for the last week action point, any comment, question?
08:04:04 <viktor_t> wow
08:04:39 <morgan_orange> #topic Feature projects integration status
08:04:52 <morgan_orange> #info integration of feature projects on track
08:05:06 <morgan_orange> #info new scenario with moon, multisites,
08:05:14 <morgan_orange> #info parser added also to testcases.yaml
08:05:29 <morgan_orange> #link http://testresults.opnfv.org/reporting/functest/release/master/index-status-joid.html
08:06:46 <May-meimei> morgan_orange: why all the status are fail?
08:07:10 <morgan_orange> May-meimei: all the status are failed because we are very (too?) strict
08:07:27 <morgan_orange> in fact we expect that all the test criteria are OK at least 4 times in a raw
08:07:42 <morgan_orange> we just got our first blue balls (i.e. all criteria OK) very recently
08:08:16 <morgan_orange> so if for instance you have a scenario with 1 Tempest error (on 165) and all the other tests are OK, the scenario is declared as failed...
08:08:23 <May-meimei> morgan_orange: oh! got it , we should pass 4 times
08:08:36 <morgan_orange> yes, moreover we consider the feature projects to be part of the criteria
08:09:08 <morgan_orange> so for ex at the moment all the apex scenarios could not be green as doctor is always failed...of course we could exlude it until it becomes OK
08:09:39 <morgan_orange> for compass for instance, it is just a question of time, as first blue balls are there, so wait and see but most of the scenarios must turn green
08:10:10 <morgan_orange> #info scenarios still seen as red/failed in reporting page because criteria are strict (internal + feature) and must be Ok at least 4 times in a raw
08:10:23 <morgan_orange> #info we got end of last week our first blue balls in Jenkins
08:10:41 <morgan_orange> #info some scenarios should be green before the end of the week
08:11:17 <morgan_orange> #info note that feature projects are also part of the scenario score, so if a feature project is part of all scenarios for an installer, it may prevent the scenario to be green
08:11:33 <morgan_orange> #info it is possible to exclude feature projects from score calculation
08:11:59 <morgan_orange> we could decide to include feature if and only if it has been Ok at least one time
08:12:25 <morgan_orange> we do not have really rules
08:12:47 <morgan_orange> for instance this morning, there were "unknown" test failed on fuel
08:13:06 <morgan_orange> these unknown correspond to parser that has been included in testcases.yaml
08:13:30 <morgan_orange> parser, multisite do not push results into DB
08:13:49 <morgan_orange> until we do not have results into DB, reporting will be failed
08:14:22 <morgan_orange> any view on the procedure to adopt to consider feature projet for the scenario scoring?
08:15:19 <morgan_orange> #info Documentation for colorado
08:15:43 <morgan_orange> #info user guide review in progress (wait for Cedric final review on ODL section)
08:16:07 <morgan_orange> #info merge will be done afterwards, and feature projects will be contacted to provide the URL of their own user guide as reference
08:16:27 <morgan_orange> #action morgan_orange merge userguide patch + call feature proejct to complete user guide with their own reference
08:16:35 <morgan_orange> #info config guide in progress
08:16:59 <morgan_orange> JuhaHaapa: I made a review this morning, I just had a question on the section related to credentials
08:17:13 <morgan_orange> and a question related to the focus on triple-o
08:17:30 <morgan_orange> #info config guide shall be ready this week
08:17:55 <JuhaHaapa> morgan_orange: ok, I will look into it. I changed so much it was hard to get everything right first time
08:18:51 <morgan_orange> JuhaHaapa: sure ..documentation is always ..difficult
08:19:14 <JuhaHaapa> I was not 100% sure it was ok to change it so much. Requires a lot from reviewers also
08:19:19 <morgan_orange> Info doc project PTL confirmed we can keep our structure (no need to rename the directories/files)
08:19:28 <morgan_orange> #action all review config guide
08:19:39 <morgan_orange> #link https://gerrit.opnfv.org/gerrit/#/c/18479/
08:20:08 <morgan_orange> #info release note to be initiated thsi week to start capturing the status of the scenarios
08:20:15 <JuhaHaapa> all comments welcome..
08:20:18 <morgan_orange> note there is a patch for onos to exclude Tempest cases
08:20:52 <morgan_orange> #info patch in progros to exclude Tempest cases from onos scenario
08:20:56 <morgan_orange> #link https://gerrit.opnfv.org/gerrit/#/c/18645/
08:21:00 <viktor_t> is it so that metedata service is still not supported by onos?
08:21:45 <morgan_orange> I think so but not sure, I assumed they used a local drive as a workaround
08:22:13 <morgan_orange> but if the issue is just medata service not supported (which was the case in brahmaputra), it is fine, we just have to clearly capture it
08:22:30 <morgan_orange> #info issue could be the non support of medata service (to be confirmed)
08:23:21 <morgan_orange> #info documentation user/config/dev guide target => end of this week, release note for the release date 2 days
08:23:32 <morgan_orange> #info Colorado scenarios
08:23:50 <morgan_orange> #info first scenarios Ok with compass, fuel and joid
08:24:20 <morgan_orange> #info some issues with some feature projects + some tempest/rally issues on some scenarios
08:24:44 <morgan_orange> #info globla improvement regarding Tempest (untill last week there were no 0 error, it is no more the case now)
08:25:13 <viktor_t> For Apex: it seems that https://jira.opnfv.org/browse/APEX-115 problem is present again in Colorado (mail sent to Apex developers)
08:26:10 <morgan_orange> viktor_t: yep, on apex/nosdn, a recent patch lead to 0 error on tempest (https://build.opnfv.org/ci/view/functest/job/functest-apex-apex-daily-master-daily-master/245/console)
08:26:17 <morgan_orange> #info but still some with heat on rally
08:26:55 <morgan_orange> juhak: I think you already discussed this point with apex team, right?
08:27:29 <juhak> morgan_orange: that's right, https://jira.opnfv.org/projects/APEX/issues/APEX-161?filter=allopenissues
08:27:52 <juhak> in general: in order to see what is the actual problem an access to rally report files (json) would be needed
08:28:02 <juhak> console log reveals only rally scenario (nova, heat...) that failed
08:28:16 <morgan_orange> #info improvement on apex (due to patch after great analysis doe by michael) but still issue on rally (heat)
08:28:25 <morgan_orange> Juhak you have access to lf-pod1?
08:28:39 <morgan_orange> you mean that json are not pushed as artifact?
08:28:49 <morgan_orange> hmm I will have a look at that
08:28:50 <juhak> yes I should that
08:29:07 <morgan_orange> #action morgan_orange see why json rally result files are not pushed anymore to artifact
08:29:34 <morgan_orange> #action juhak see json locally on lf-pod1 to troubleshoot rally issue with apex scenario
08:29:46 <juhak> ok
08:30:50 <morgan_orange> #info regarding dashboard, LF foundation accepted to purcharsed an elastic search licence. I think it is a cloud licence, so the Data will be pushed from test results to this cloud
08:31:01 <morgan_orange> SerenaFeng: could you sync with Juraj on this topic?
08:31:21 <SerenaFeng> noope
08:31:50 <SerenaFeng> is it settled, I didn't see any email of this issue
08:32:32 <morgan_orange> I forwarded the last mail I got
08:32:55 <morgan_orange> mail from Ray on the licence
08:33:13 <morgan_orange> #action morgan_orange SerenaFeng sync with Juraj and peopel dealing with ElasticSearch licence
08:33:16 <morgan_orange> we can do it offline
08:33:30 <SerenaFeng> ok
08:34:00 <morgan_orange> #info first lxd and multisite tests run (new scenario)
08:34:16 <morgan_orange> #info multisite OK
08:34:18 <morgan_orange> #link https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-master/lastSuccessfulBuild/console
08:34:29 <morgan_orange> May-meimei: do you know if it is planned to push the results into the DB
08:34:36 <morgan_orange> I created a multisite testcase for that
08:34:55 <May-meimei> morgan_orange: I saw your email
08:35:04 <morgan_orange> as results sound OK, pushing the results allow to automatically be included in the reporting
08:35:07 <May-meimei> just one line script to push the result
08:35:08 <May-meimei> ?
08:35:09 <morgan_orange> at the moment we do not see the scenario
08:35:20 <morgan_orange> maybe 4 or 5 :)
08:35:34 <May-meimei> ok! will do it
08:35:51 <morgan_orange> you can see some examples in https://git.opnfv.org/cgit/functest/tree/testcases/features
08:35:55 <morgan_orange> https://git.opnfv.org/cgit/functest/tree/testcases/features/doctor.py
08:36:09 <morgan_orange> L77=>84
08:36:17 <May-meimei> ok!
08:36:27 <morgan_orange> instead of doctor, doctor-notification => multisite/multisite
08:37:45 <SerenaFeng> I suggest we add a report flag, to prevent pushing meaningless results to db, which is not created by ci test
08:38:23 <morgan_orange> yes, report flag is done fot that, by default we do not report anything, we report results from CI PODs
08:38:35 <morgan_orange> BTW, any run of parser test suite?
08:38:42 <SerenaFeng> if args.report:
08:39:10 <SerenaFeng> reference promise.py or parser.py
08:39:28 <SerenaFeng> I think I have run parser
08:39:33 <morgan_orange> #action SerenaFeng morgan_orange add report flag to avoid meaningless results or 500 because source is not referenced
08:39:56 <SerenaFeng> but I didn't see it run through ci board
08:40:14 <morgan_orange> OK
08:40:25 <SerenaFeng> #link https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/343/console
08:40:46 <morgan_orange> soon then
08:40:54 <SerenaFeng> ok
08:41:11 <SerenaFeng> and I have a question here
08:41:29 <SerenaFeng> according to code structure, parser belong to vnf tier
08:41:38 <SerenaFeng> but this tier is defined weekly
08:41:58 <SerenaFeng> for now I move parser to feature tier in testcases.yaml
08:42:26 <SerenaFeng> I think maybe we should attach daily/weekly flag to each testcase not tier
08:43:16 <morgan_orange> to be discussed with Jose, we defined tiers to group cases by category and also behavior
08:43:26 <SerenaFeng> ok
08:43:27 <morgan_orange> but I understand your point
08:43:37 <morgan_orange> do you know the duration of parser tests?
08:44:10 <morgan_orange> we should probably formalize a little bit for D to say when a case is part of feature, enhanced or vnf..
08:44:11 <SerenaFeng> for normal condition, very quickly, 2-3minutes
08:44:36 <morgan_orange> vnf has been created for cases that deal with VNF and last more than 10 minutes
08:44:47 <morgan_orange> so we cannord afford to run it all the time (usually 30 minutes)
08:45:18 <SerenaFeng> okey, I will disscuss this issue with parser people later
08:45:25 <morgan_orange> #action create rules for D release to decide to which category a case belong to
08:45:47 <morgan_orange> it means also that the user guide should be probably amended, as parser was pushed to vnf section...
08:45:48 <morgan_orange> :)
08:46:17 <morgan_orange> otherwise is the issue linked to the testapi docker tag fixed?
08:46:36 <SerenaFeng> nope
08:46:40 <SerenaFeng> no respond yet
08:47:20 <morgan_orange> I saw a mail from fdegir asking what you were expecting (he has the creds)
08:47:27 <SerenaFeng> I remember it is yesterday I send a email again, but no respond
08:47:46 <morgan_orange> you can contact him on releng chan
08:48:03 <morgan_orange> only 15 minutes left, let move to last topics
08:48:03 <SerenaFeng> did that email include me?
08:48:07 * fdegir has eyes everywhere
08:48:13 <morgan_orange> SerenaFeng: yes
08:48:30 <fdegir> how may I help you SerenaFeng ?
08:48:52 <fdegir> we can chat once your meeting is over SerenaFeng
08:48:58 <morgan_orange> #topic Date for Functest Branching Colorado
08:49:02 <SerenaFeng> okey, thank you
08:49:16 <morgan_orange> it is time to branch/tag ...and cherry pick
08:49:31 <morgan_orange> I know that yardstick planned to branch on the 18th
08:49:48 <morgan_orange> it was decided to have a window to do that from the 15 to  the ...
08:49:57 <SerenaFeng> 22nd
08:50:08 <SerenaFeng> I remember the deadline is 22nd
08:50:21 <morgan_orange> I would suggest to do it on Friday (19th)
08:50:33 <morgan_orange> or on Monday (22nd)
08:51:41 <morgan_orange> we can be reasonably at the end as we clone lots of repos..so it makes sense that they branched first...
08:52:08 <morgan_orange> any objection to branch/tag on the 19th?
08:52:30 <morgan_orange> let's say 13 UTC
08:53:04 <SerenaFeng> nope from my side
08:53:12 <morgan_orange> #vote branch/tag Functest Colorado on the 19th 13 UTC
08:53:31 <morgan_orange> newby with vote commande
08:53:37 <morgan_orange> #endvote
08:53:46 <morgan_orange> ok let's collect the objection...
08:53:57 <morgan_orange> if no, I will indicate this date to release manager
08:54:17 <morgan_orange> #info Functest will branch/tag Colorado on the 19th 13UTC
08:54:27 <morgan_orange> #topic intern project proposals
08:54:38 <lhinds> morgan_orange: can we do a small slot on sec?
08:54:53 <morgan_orange> #undo
08:54:53 <collabot> Removing item from minutes: <MeetBot.ircmeeting.items.Topic object at 0x2f3b7d0>
08:54:55 <morgan_orange> sure lhinds
08:54:58 <lhinds> thx!
08:55:05 <morgan_orange> #topic Security
08:55:10 <lhinds> those issues are resolved now
08:55:23 <lhinds> it was a problem with the FDIO repo that Apex was using
08:55:31 <lhinds> I will update Jira
08:55:55 <lhinds> Also I wanted to raise changing functest to use the new repo we have
08:56:07 <lhinds> so instead of runnning from functest/testcases
08:56:18 <lhinds> it will do a git clone like the other projects
08:56:29 <morgan_orange> ok no problem, it is better to do it before we branch :)
08:56:37 <lhinds> agree!
08:56:45 <morgan_orange> #info main issues fixed, JIRA will be updated
08:56:47 <lhinds> I will do a patch today
08:57:01 <morgan_orange> #info Security repo will be used (instead of sub dire in Functest)
08:57:07 <lhinds> that's all!
08:57:12 <lhinds> thanks
08:57:14 <morgan_orange> #action lhinds modify exec_test to point to security repo
08:57:35 <morgan_orange> for Colorado, it will then be supported only on apex, right?
08:57:57 <morgan_orange> I will have a look at apex run, when re run, I will re enable it on the reporting
08:58:11 <morgan_orange> #action morgan_orange re-enable security_scan when ready
08:58:21 <morgan_orange> we will prbably also have to look at the artifact issue
08:58:40 <lhinds> morgan_orange: colarado will be apex, and d release we hope to have fuel
08:58:43 <morgan_orange> I believe it was OK, but if no rally json push, I assume that the html page of the security report are also not pushed
08:58:50 <lhinds> and it should run fine now
08:59:07 <morgan_orange> #info security scean on apex for Colorado, other installer for D release
08:59:27 <morgan_orange> #info check artifact management to be sure that security reporting pages are propelery pushed to artifact
08:59:51 <morgan_orange> #topic intern project proposals
09:00:09 <morgan_orange> #link https://wiki.opnfv.org/display/DEV/Intern-projects-page
09:00:16 <morgan_orange> #info proposals made by mail
09:00:45 <morgan_orange> #info mainly new simple VNF test cases (can reused the onos list of elementary VNF for inspiration)
09:01:02 <morgan_orange> any proposal welcom
09:01:15 <morgan_orange> let's say end of August to submit 3 or 4 proposals for Functest
09:01:37 <morgan_orange> feel free to react on the mail (you were in copy) and make suggestions
09:01:49 <morgan_orange> #topic Resources for Plugfest/Meetup
09:02:00 <morgan_orange> we are already late but just few words on that
09:02:24 <morgan_orange> for the plugfest (beginning of Decemember in Boston): does anyone already know he may participate?
09:02:37 <morgan_orange> no one from Nokia?
09:03:02 <juhak> not yet known
09:03:24 <morgan_orange> I think that we need to have a representative fro Functest, unfortunately we do not have lots of US contributors...
09:03:51 <JuhaHaapa> not yet know for me either. Best guess is "h**l no - no money" :)
09:03:59 <morgan_orange> #info search functest rrepresentative for plugfest (registration already open, especially for company aiming to send hardware)
09:04:37 <morgan_orange> #info for the next meetup, my suggestion will be to do it either in barcelona (during the OpenStack summit) or in France (beginning of October)
09:05:06 <morgan_orange> same question on the participation to the OpenStack Summit end of October...
09:05:25 <morgan_orange> but if no money to fly to barcelona, I assume no money to go to France...
09:05:41 <SerenaFeng> I prefer barcelona, in that case maybe I will be there
09:06:18 <morgan_orange> ok we will decide it beginning of September
09:06:19 <juhak> I vote for Barcelona too
09:06:26 <morgan_orange> #topic AoB
09:06:33 <morgan_orange> any other topic you want to raise
09:07:17 <morgan_orange> no
09:07:36 <morgan_orange> ok have a good week, branching on Friday!
09:07:39 <morgan_orange> #endmeeting