20:29:35 <dfarrell07> #startmeeting Lithium Test Strategy
20:29:35 <odl_meetbot> Meeting started Mon Sep 29 20:29:35 2014 UTC.  The chair is dfarrell07. Information about MeetBot at http://ci.openstack.org/meetbot.html.
20:29:35 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:29:35 <odl_meetbot> The meeting name has been set to 'lithium_test_strategy'
20:30:37 <dfarrell07> #info LuisGomez presenting
20:37:27 <dfarrell07> #info Working with Linux Foundation to get better test infra
20:38:22 <dfarrell07> #info Karaf introduced huge changes in test strategy, came late in release
20:38:42 <dfarrell07> #info Need to work with projects eairly in reelase to let them know what we expect (context of Karaf)
20:39:03 <dfarrell07> #info Lack of resources to write system tests in int team, need projects to take some of this work
20:39:41 <dfarrell07> #info TODO for Li, need to get projects to test their features more, can't do it all in int team
20:40:02 <dfarrell07> #info Int team with help of course, will provide VMs for example with help of LF
20:40:22 <dfarrell07> #info More training, more info, more wikis for helping projects write system tests
20:41:06 <dfarrell07> #info Used to build three distros, now much more complex (Karaf). Need to build based on features
20:41:28 <dfarrell07> #info All the more important that projects write systems tests so we can reuse them for multi-feature distros
20:42:06 <dfarrell07> #info Continue working on stability and perf test, that's very much in-scope for int team
20:42:36 <dfarrell07> #info Make sure projects know about milestones, what they need done by them, context of system tests
20:43:15 <dfarrell07> #info Projects don't know much about how to use Robot FW, need to teach them. Appoint int folks to talk with projects to make sure they are on same page?
20:43:57 <dfarrell07> #info Need folks to do this work system test ambassador, vols needed
20:44:21 <dfarrell07> #info Question: What is recommendation for Li w/ regard to how to do system tests?
20:45:25 <dfarrell07> #info Answer: We'll provide more info via wikis, calls, TWS, Int team call. We'll put people in place to ask as well. Major diff is that projects need to own their system tests, can no longer rely on int team.
20:45:47 <dfarrell07> #info Need way more time after code freeze to write system tests
20:47:32 <dfarrell07> #info Suggestion: Need more docs on installing ODL, all the things.
20:48:06 <dfarrell07> #info All wikis have old HOWTOs on deploying ODL, but it's not Karaf
20:48:33 * dfarrell07 sees this causing all kinds of issues very soon^^
20:48:43 <dfarrell07> expect tons of IRC and ask.odl questions
20:50:07 <dfarrell07> #info Question: Need to do a better job of communicating code freeze timetable
20:50:28 <dfarrell07> #info Code freeze didn't really happen, and int team only had one week between "freeze" and first RC
20:50:47 <dfarrell07> #info Short testing window sucks for projects as well as int team
20:51:38 <dfarrell07> #info Suggestion: Make tests part of code freeze
20:52:32 <dfarrell07> #info Suggestion: Don't let system tests come after features, do them with the code (this is a well Good Thing(TM), imho)
20:54:36 <dfarrell07> #info Need to "police" projects to make sure they are builidng syste/int tests as they go. Need to sync up with release managment folks to make this happen.
20:57:48 <dfarrell07> #info New test strategy with Li, punting system tests to projects so they get run early/fail early, would greatly reduce long long it takes to find fails as the int team
20:59:44 <dfarrell07> #info Doing away with the silo system we use now, where every project is their on silo, is going to help clean up test infra
21:00:50 <dfarrell07> #info Question: To teams need to come back and review system tests to make sure they are are still okay, still good coverage?
21:02:13 <dfarrell07> #info Answer: Core projects seem to have a good understanding of our test coverage, we know what they are doing and vice versa, but can't do the same for every project
21:03:36 <dfarrell07> #info Question: Should we move int tests out of int repo, into project repos?
21:03:44 <dfarrell07> #info LuisGomez likes the idea
21:04:17 <dfarrell07> #info Int team wants to move to hosting tests, letting projects own/build/maintain them. Moving to projects hosting tests would help this.
21:04:45 <dfarrell07> #info TODO: Move system test code out of int repo, into project repos
21:05:49 <dfarrell07> #info When there are new tests that are not in our repo, we need to be told about that
21:07:04 <dfarrell07> #info Pushback: Int should be about testing project X with project Y. That's different than tests of project X that depend on standing up controller, for example.
21:07:49 <dfarrell07> #info Difference between testing multi-feature/multi-project and projects testing their own feature, may depend on other parts of codebase
21:08:39 <dfarrell07> #info Int team would like to let it be known that we have tons of tooling and such that makes it easy to write system tests
21:09:22 <dfarrell07> #info Suggestion: Two types of tests should be called "project" and "multi-project"
21:11:01 <dfarrell07> #info Third catagory could be "Interop test", which would work with different low-level hardware for example
21:11:49 <dfarrell07> #info "Integration tests" is a way overloaded term
21:15:35 <dfarrell07> #info All types of tests apply to both single-project and multi-project tests
21:24:01 <gzhao> #info Luis notes we need to form a team and focus on performance
21:24:40 <gzhao> #denial notes performance should be part of integration.
21:24:45 <dfarrell07> #info lots of perf talk, but I was talking so no notes atm
21:25:17 <dfarrell07> #info I'll do an unconf on perf, watch out for it
21:28:02 <gzhao> #todo: release management need to have process and status call from beginning of the simultaneously release.
21:28:21 <dfarrell07> #endmeeting