08:00:11 <jose_lausuch> #startmeeting Functest weekly meeting July 4th 2017
08:00:11 <collabot`> Meeting started Tue Jul  4 08:00:11 2017 UTC.  The chair is jose_lausuch. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:11 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:00:11 <collabot`> The meeting name has been set to 'functest_weekly_meeting_july_4th_2017'
08:00:16 <jose_lausuch> #topic role call
08:00:20 <jose_lausuch> #info Jose Lausuch
08:00:24 <morgan_orange> #info Morgan Richomme
08:00:29 <juhak> #info Juha Kosonen
08:00:44 <LindaWang> #info Linda Wang
08:01:01 <SerenaFeng> #info SerenaFeng
08:01:47 <jose_lausuch> #chair morgan_orange
08:01:47 <collabot`> Current chairs: jose_lausuch morgan_orange
08:01:53 <jose_lausuch> #info agenda for today: https://wiki.opnfv.org/display/functest/Functest+5.+Meeting#Functest5.Meeting-04/07(8UTC)
08:02:03 <jose_lausuch> Hi Martin_nokia and welcome to the team
08:02:20 <Martin_nokia> Hi Jose and everyone, thanks for the warm welcome!
08:02:32 <SerenaFeng> welcome
08:02:39 <jose_lausuch> #info Martin_nokia joins Functest, to start with rally/tempest needs
08:02:51 <LindaWang> Martin_nokia: Welcome!
08:02:58 <morgan_orange> Tervetuloa
08:03:01 <jose_lausuch> Martin_nokia: we use to have IRC only meetings
08:03:13 <boucherv> #info Valentin Boucher
08:03:16 <jose_lausuch> #topic Action Point follow-up
08:03:31 <jose_lausuch> #info AP jose_lausuch check problem with docker image build in danube
08:03:36 <jose_lausuch> #info Done. https://review.openstack.org/#/c/478134/ is merged and the docker image builds normally. Before, I was doing it manually.
08:03:45 <jose_lausuch> #info AP morgan_orange sync with JackChan_ for next steps on landing page
08:04:04 <jose_lausuch> morgan_orange: I guess it's done
08:04:09 <morgan_orange> #info done mails excha,nged last week. catalog is almost OK, dockerization in progress
08:04:27 <jose_lausuch> ok, let's then start with the first topic for today
08:04:28 <jose_lausuch> #topic Alpine or not Alpine: what are the problems/is missing?
08:04:36 <morgan_orange> #info for the landing page we need to add that at the end of a CI run. Must check with SerenaFeng if it is OK to use the test APi to play with scenario object
08:04:59 <morgan_orange> I think this topic is linked to the next one on refstack/Rally/Tempest
08:05:11 <jose_lausuch> alpine?
08:05:23 <morgan_orange> ollivier: already cleaned alot the docker but there are still issues with these tests for a full alpine support
08:05:37 <morgan_orange> ollivier: can you elaborate a little bit on that?
08:05:42 <jose_lausuch> ollivier: can you tell what's missing to switch to alpine?
08:06:51 <jose_lausuch> mmmm
08:06:58 <jose_lausuch> maybe he is not in...
08:07:06 <morgan_orange> ok maybe we could come back to this topic once ollivier would be there
08:07:10 <jose_lausuch> well, my view is that there has been good progress
08:08:08 <jose_lausuch> but swtiching to another OS could cause new issues and it's maybe a bit too late for them, as installers and release managers rely on functest framework being stable
08:08:39 <jose_lausuch> and we are still doing some framework changes, that are not disrubptive , but that looks like a big change
08:08:53 <jose_lausuch> I would be ok to delay that to the next release
08:08:55 <morgan_orange> I think for Euphrates we could have both - we could consider alpine as experiemental if it is there
08:09:02 <jose_lausuch> in the meantime, we can produce a dokcer image and do tests locally
08:09:12 <jose_lausuch> or even have a job in CI using that image
08:09:20 <jose_lausuch> but not for official E-release
08:09:28 <jose_lausuch> yes, something like that
08:09:49 <jose_lausuch> but I'd like to hear Cedric's opinion on this
08:10:01 <LindaWang> Yes, apline is used, more than one images will be needed to run test cases, right?
08:10:02 <jose_lausuch> maybe we can move on and talk about it once he is back
08:10:16 <jose_lausuch> LindaWang: more than one images?
08:10:46 <LindaWang> I guess there is one core image, and then other images will be built based on that.
08:10:55 <LindaWang> But I am not sure.
08:11:20 <LindaWang> If there is a core image.
08:12:05 <jose_lausuch> well, that is the idea, but that is docker slicing topic
08:12:09 <LindaWang> Is it related to docker slicing?
08:12:13 <LindaWang> ok
08:13:09 <SerenaFeng> this topic is about selecting os alpine/ubuntu
08:13:28 <jose_lausuch> yes
08:13:39 <LindaWang> Got it
08:13:40 <jose_lausuch> whats your opinion?
08:13:48 <jose_lausuch> SerenaFeng: LindaWang juhak ?
08:14:33 <SerenaFeng> I am just worry about how many people and how much people know about alpine
08:14:37 <ollivier> sorry I was afk
08:15:06 <jose_lausuch> ollivier: we are discussing about the switch to alpine (see previous messages)
08:15:10 <SerenaFeng> if it is not very difficult for people to learn and use it
08:15:38 <ollivier> To fully switch to Alpine, we must stop running scripts which execute apt or yum
08:15:40 <SerenaFeng> I agree to use it
08:16:00 <jose_lausuch> SerenaFeng: yes, but the idea of functest is to become a product, a package, the idea is that you don't even need to enter the container and undestand the OS. For running the tests cases, you can just call the docker command
08:16:10 <jose_lausuch> ollivier: good point
08:16:21 <SerenaFeng> in that case, it is ok
08:16:24 <jose_lausuch> and there are some test cases that still do that (domino)
08:16:38 <SerenaFeng> and I agree we can switch to alpine next release
08:16:42 <ollivier> They remain 3 scripts in Dockerfile
08:16:49 <jose_lausuch> we can have experimental alpine in E-release
08:16:59 <SerenaFeng> for this release we can have some experimental
08:17:02 <ollivier> I think we must install only packages and python packages in Dockerfile
08:17:06 <jose_lausuch> and release the Ubuntu container, and also have Alpine as experimental (also documented)
08:17:24 <ollivier> Then we could stop running third party scripts
08:17:38 <ollivier> And configurations should be in charge of the TestCases
08:17:43 <ollivier> instead of Dockerfile
08:18:10 <ollivier> If there are conflicts between tempest versions we must setup virtualenvs
08:18:17 <jose_lausuch> LindaWang got rid of setup_env from refstack, that is one step further
08:18:29 <ollivier> I have planned to write that in the wiki to share with you
08:18:36 <SerenaFeng> can you list out how many changes left to full use alpine?
08:18:37 <jose_lausuch> ok
08:18:49 <jose_lausuch> ollivier: can I AP you on that?
08:19:01 <jose_lausuch> and we can follow up next week or the other?
08:19:05 <ollivier> I can share an incomplete alpine docker for testing purpose without promise, refstack, rally and sfc
08:19:15 <SerenaFeng> there will be only one tempest left
08:19:24 <SerenaFeng> so not conflicts
08:19:36 <jose_lausuch> ok
08:19:37 <ollivier> sure. please add the action points
08:20:10 <jose_lausuch> #action ollivier create a wiki and list all the remaining issues that prevent us from fully using Alpine
08:20:11 <jose_lausuch> thank you
08:20:30 <jose_lausuch> do we all agree on releasing for Euprates the current Ubuntu based image, and provide Alpine as experimental?
08:20:33 <ollivier> yes I wil try to list what a third party should provide to be integrated
08:20:40 <morgan_orange> #agree
08:20:42 <jose_lausuch> and move completelly to Alpine in F-release?
08:20:51 <boucherv> #agree
08:20:59 <SerenaFeng> and I think it makes sense that we run a fully support alpine for a period, before switching to it
08:21:02 <SerenaFeng> #agree
08:21:08 <juhak> #agree
08:21:09 <jose_lausuch> ok
08:21:14 <ollivier> The question is: do we remove some testcase (promise) to help switching?
08:21:43 <jose_lausuch> #info Agreed to release the current Ubuntu based image for Euprates along with the experimental Alpine
08:21:52 <SerenaFeng> from my perspective, if it is supported now, it shouldn't be removed
08:21:53 <boucherv> With a dedicated CI/pod for the test of this container ?
08:21:55 <jose_lausuch> I think we can remove that from Alpine, and whenever there is support for it, we can add it
08:22:01 <ollivier> do we disable incomplete package refstack?
08:22:07 <LindaWang> ollivier: Do you think there are still some issues with the proposal of"got rid of setup_env from refstack
08:22:39 <jose_lausuch> we should disable everthing that is not supported (because of apt-get/yum, or specific ubuntu things, ...) and then see how much effort is to support that
08:22:49 <jose_lausuch> and start adding the test cases step by step into Alpine
08:22:58 <jose_lausuch> but starting with something at least
08:22:58 <ollivier> I must check your update. I think refstack should be installed in a dedicated virtualenv if there are conflict between tempest versions
08:23:04 <SerenaFeng> agree, that also makes sense
08:23:14 <LindaWang> ok
08:23:49 <jose_lausuch> would it make sense to create a new Dockerfile for Alpine? /docker/alpine/Dockerfile ?
08:24:00 <ollivier> yes
08:24:09 <ollivier> I must publish it.
08:24:12 <jose_lausuch> ok
08:24:31 <jose_lausuch> #action ollivier publish Alpine Dockerfile in /docker/alpine/Dockerfile
08:24:33 <ollivier> but we could publish 2 containers... One light and a full.
08:24:36 <jose_lausuch> or just Dockerfile.alpine
08:24:57 <ollivier> a new dir seems better.
08:25:08 <jose_lausuch> ok
08:25:16 <jose_lausuch> good
08:25:18 <jose_lausuch> thanks
08:25:21 <jose_lausuch> next topic?
08:25:25 <jose_lausuch> there are many topics today :)
08:25:32 <SerenaFeng> why two containers?
08:25:57 <SerenaFeng> will both of them be used in experimental?
08:26:12 <ollivier> The first alpine publish only contains Functest (all testcases host by Functest) without third parties
08:26:52 <ollivier> It's much difficult to integrate everything as we are fixing lots of projects
08:27:11 <ollivier> (see securityscanning etc.)
08:27:54 <jose_lausuch> sounds good
08:28:04 <jose_lausuch> in the end I guess we will release the full one, in F-release
08:28:11 <jose_lausuch> #topic Refstack/Rally/tempest
08:28:33 <jose_lausuch> #info Currently, we are running tempest test cases from Rally (rally verify start)
08:29:12 <jose_lausuch> #info Rally needs to have a copy of the repo somewhere to install the verify environment
08:29:46 <jose_lausuch> rally provides nice tools like auto generation of tempest.conf and other things
08:29:49 <jose_lausuch> juhak: am I right?
08:29:59 <juhak> yes, that's right
08:30:33 <jose_lausuch> I tried to install rally-verify with the tempest package installer (no git repo) and it complained
08:30:38 <ollivier> but it's redundant regarding the tempest testcase.
08:30:46 <jose_lausuch> and I talked to the Rally team, and it's the only way
08:30:55 <jose_lausuch> if you don't provide the tempest repo, it will download it anyway
08:31:02 <jose_lausuch> (which goes against our functest offline feature)
08:31:05 <jose_lausuch> so, what do we do?
08:31:11 <ollivier> Even if we remove -s and stop running tempest?
08:31:15 <jose_lausuch> ollivier: what do you mean it's redundant?
08:31:23 <LindaWang> Which is the only way?
08:31:27 <LindaWang> Why not run tempest by tempest cli directly?
08:31:30 <ollivier> tempest is ran via its Testcase or am I wrong?
08:31:32 <jose_lausuch> that is anotjher option
08:31:46 <LindaWang> Just use tempest.conf generated by rally
08:31:47 <ollivier> and we run it again via rally
08:31:48 <jose_lausuch> ollivier: yes, but if you look into the code of Tempest, you will see some rally commands
08:32:44 <jose_lausuch> options are:
08:32:58 <jose_lausuch> 1) keep cloning tempest repo and leave it as it is (it has worked well so far)
08:33:12 <jose_lausuch> 2) switch to testr (pure tempest) without Rally (no need to clone tempest repo)
08:33:30 <jose_lausuch> 2) requires extra work
08:34:11 <jose_lausuch> 2) means modifying the logic of Tempest Testcase in Functest
08:34:22 <SerenaFeng> yes, at lease tempest workspace and tempest.conf need to be generated manually
08:34:26 <ollivier> If 1). Do we need to install tempest by package?
08:34:44 <jose_lausuch> ollivier: I think it's not necessary
08:34:55 <jose_lausuch> but not 100% sure
08:35:08 <ollivier> If yes we can avoid the conflicts
08:35:29 <LindaWang> ollivier:  I am sure it is needed.
08:35:51 <SerenaFeng> I don't remember tempest need to be installed
08:36:14 <jose_lausuch> before including tempest as a python package, we were cloning the repo and doing pip install in the repo
08:36:21 <jose_lausuch> so I imagine its necessary, yes
08:36:30 <LindaWang> If we did not install tempest, the step of creating verifier will fail for sure, complaining that:
08:36:30 <LindaWang> Failed to set up verifier 'opnfv-tempest': Missed package(s) for system-wide installation found. Please install 'os-testr', 'testrepository'.
08:36:52 <jose_lausuch> ok
08:36:55 <jose_lausuch> thanks
08:36:58 <jose_lausuch> then?
08:37:04 <jose_lausuch> what are the best options?
08:37:15 <jose_lausuch> juhak:  do you know if there is any other way to generate tempest.conf without rally?
08:37:20 <ollivier> If we can't rely one the official release, we can simply replace the line by -e git+http... in thirdparty-requirements.txt
08:37:46 <ollivier> a working dir will be available in /root
08:38:05 <ollivier> and it will be installed in PYTHON PATH
08:38:23 <ollivier> I was afraid of conflicts between two versions of tempest
08:38:26 <juhak> jose_lausuch: not at the moment, need to check
08:38:58 <LindaWang> ollivier:  I am confused why two versions of tempest are installed?
08:39:39 <ollivier> One via thirdparty-requirements.txt and the second one when executing scripts (git clone...)
08:39:59 <LindaWang> In thirdparty-requirements.txt, why is tempest>=15.0.0?
08:40:10 <ollivier> This is the official OpenStack requirements
08:40:24 <LindaWang> Actually tempest 16.0.0 is installed.
08:40:40 <ollivier> Yes via upper-constraints
08:41:15 <SerenaFeng> but in Functest, currently we leverage version 15.0.0
08:42:09 <jose_lausuch> I think 16 is targeting Pike
08:42:12 <ollivier> cat upper-constraints.txt |grep tempest
08:42:12 <ollivier> tempest===14.0.0
08:42:22 <jose_lausuch> and in OPNFV we are testing Ocata
08:43:46 <ollivier> ?? must be checked
08:44:44 <jose_lausuch> https://releases.openstack.org/ocata/
08:44:48 <jose_lausuch> search for tempest
08:44:59 <ollivier> I'm directly reading the requirements project
08:45:49 <ollivier> https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?stable/ocata
08:46:04 <ollivier> tempest===16.0.0
08:46:31 <jose_lausuch> yes, but wouldn't it make sense to force 15.0.0 since we are testing Ocata? (sorry, not very familiar with upper constraints)
08:47:32 <ollivier> I must add a section on the wiki how we could handle requirements as well.
08:47:53 <ollivier> global requirements, upper constaints...
08:48:07 <jose_lausuch> what do we do then? keep 16.0.0?
08:48:13 <SerenaFeng> well according to this page, if we are testing ocata, it should be version 16
08:48:21 <jose_lausuch> yes, taht's confusing
08:48:25 <jose_lausuch> (to me at least)
08:48:37 <jose_lausuch> cause here https://releases.openstack.org/ocata/#ocata-tempest  it says 14 or 15
08:48:38 <ollivier> Maybe my url is buggy. please open an action point
08:49:22 <jose_lausuch> #action ollivier check tempest version to be used to run against Ocata
08:49:35 <ollivier> bug me https://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt?h=stable/ocata
08:49:56 <ollivier> tempest===14.0.0
08:50:13 <ollivier> I must fix tox.ini and dockerfile...
08:50:30 <jose_lausuch> is it still ok to use 15? even though it says 14 here?
08:50:41 <jose_lausuch> what version should we use in the end?
08:51:44 <ollivier> And do we need the package or simply a clone dir?
08:51:52 <jose_lausuch> let's discuss online, we are running out of time and boucherv had a topic
08:51:59 <jose_lausuch> we still need to clone dir as well
08:52:03 <jose_lausuch> for rally-verify
08:52:27 <jose_lausuch> if we change to testr and not use rally, we don't need to clone, but that is another story, we need to fix current issue
08:52:34 <jose_lausuch> so we need the clone as well
08:52:37 <jose_lausuch> #topic Logger
08:52:40 <jose_lausuch> boucherv: please
08:52:45 <ollivier> I will give you a patch if you want to test without the offical package
08:52:52 <jose_lausuch> ollivier: sure, thanks
08:53:21 <boucherv> Yes, for the logger part it's really strange to don't show the log of testcase
08:53:45 <jose_lausuch> but now Jenkins output is readable
08:53:47 <boucherv> I understand for the CI but not for the manual use of functest
08:53:54 <jose_lausuch> if you go to danueb jobs, it's impossible to find something
08:54:07 <jose_lausuch> we could add a flag to show logs on the screen
08:54:08 <boucherv> Yes, but not in master
08:54:23 <boucherv> During my developement of the new cloudify_ims release
08:54:31 <jose_lausuch> testcase run clearwater_ims --show-logs or something like that
08:54:42 <jose_lausuch> we decided to mute all logs in master
08:54:45 <jose_lausuch> they are in the log files
08:55:02 <ollivier> be free to change functest/ci/logging.ini
08:55:10 <jose_lausuch> you can have another window with tail -f ~/functes/results/functest.log
08:55:20 <jose_lausuch> what about adding a flag to the script?
08:55:23 <boucherv> It's strange for end user because when It run the test no log like it's freeze ?
08:55:29 <ollivier> you can turn replace wconsole by console for testing purpose
08:55:31 <jose_lausuch> I would like to keep the logs muted on the screen by default
08:56:06 <boucherv> ollivier: I know the logging.ini file but for end user ?
08:56:41 <ollivier> When running by hand or by CI?
08:56:50 <boucherv> if the show log arg work it's good but in think in the help the arg is not display
08:56:54 <boucherv> by hand
08:57:13 <ollivier> The why not tweaking logging.ini for that?
08:57:47 <boucherv> When someone who does not know functest wants to run it for the first time ..
08:57:58 <jose_lausuch> #info option 1) tweak logging.ini when manual testing/troubleshooting: replace wconsole by console
08:58:11 <jose_lausuch> #info option 2) tail -f ~/functest/results/functest.log
08:59:32 <ollivier> #agree
09:00:17 <boucherv> For example in vnf testcase, the console show only "tenant creation" and 20 minutes after test PASS
09:00:22 <jose_lausuch> I think boucherv concern makes also sense, for end users
09:00:34 <jose_lausuch> we could add a flag in run_tests or something
09:00:59 <boucherv> I think about the number of mail that I will receive if it does not work ...
09:01:20 <ollivier> Yes but functest.log must be attached
09:02:29 <jose_lausuch> I'm lost about the emails
09:02:44 <jose_lausuch> we can follow up this topic next week
09:03:21 <jose_lausuch> or discuss offline
09:03:27 <jose_lausuch> thank you all
09:03:31 <jose_lausuch> some topics moved to next week
09:03:33 <jose_lausuch> #endmeeting