#opendaylight-devforum5: Lithium Test Strategy

Meeting summary

1. 1. LuisGomez presenting (dfarrell07, 20:30:37)
   2. Working with Linux Foundation to get better test infra (dfarrell07, 20:37:27)
   3. Karaf introduced huge changes in test strategy, came late in release (dfarrell07, 20:38:22)
   4. Need to work with projects eairly in reelase to let them know what we expect (context of Karaf) (dfarrell07, 20:38:42)
   5. Lack of resources to write system tests in int team, need projects to take some of this work (dfarrell07, 20:39:03)
   6. TODO for Li, need to get projects to test their features more, can't do it all in int team (dfarrell07, 20:39:41)
   7. Int team with help of course, will provide VMs for example with help of LF (dfarrell07, 20:40:02)
   8. More training, more info, more wikis for helping projects write system tests (dfarrell07, 20:40:22)
   9. Used to build three distros, now much more complex (Karaf). Need to build based on features (dfarrell07, 20:41:06)
   10. All the more important that projects write systems tests so we can reuse them for multi-feature distros (dfarrell07, 20:41:28)
   11. Continue working on stability and perf test, that's very much in-scope for int team (dfarrell07, 20:42:06)
   12. Make sure projects know about milestones, what they need done by them, context of system tests (dfarrell07, 20:42:36)
   13. Projects don't know much about how to use Robot FW, need to teach them. Appoint int folks to talk with projects to make sure they are on same page? (dfarrell07, 20:43:15)
   14. Need folks to do this work system test ambassador, vols needed (dfarrell07, 20:43:57)
   15. Question: What is recommendation for Li w/ regard to how to do system tests? (dfarrell07, 20:44:21)
   16. Answer: We'll provide more info via wikis, calls, TWS, Int team call. We'll put people in place to ask as well. Major diff is that projects need to own their system tests, can no longer rely on int team. (dfarrell07, 20:45:25)
   17. Need way more time after code freeze to write system tests (dfarrell07, 20:45:47)
   18. Suggestion: Need more docs on installing ODL, all the things. (dfarrell07, 20:47:32)
   19. All wikis have old HOWTOs on deploying ODL, but it's not Karaf (dfarrell07, 20:48:06)
   20. Question: Need to do a better job of communicating code freeze timetable (dfarrell07, 20:50:07)
   21. Code freeze didn't really happen, and int team only had one week between "freeze" and first RC (dfarrell07, 20:50:28)
   22. Short testing window sucks for projects as well as int team (dfarrell07, 20:50:47)
   23. Suggestion: Make tests part of code freeze (dfarrell07, 20:51:38)
   24. Suggestion: Don't let system tests come after features, do them with the code (this is a well Good Thing(TM), imho) (dfarrell07, 20:52:32)
   25. Need to "police" projects to make sure they are builidng syste/int tests as they go. Need to sync up with release managment folks to make this happen. (dfarrell07, 20:54:36)
   26. New test strategy with Li, punting system tests to projects so they get run early/fail early, would greatly reduce long long it takes to find fails as the int team (dfarrell07, 20:57:48)
   27. Doing away with the silo system we use now, where every project is their on silo, is going to help clean up test infra (dfarrell07, 20:59:44)
   28. Question: To teams need to come back and review system tests to make sure they are are still okay, still good coverage? (dfarrell07, 21:00:50)
   29. Answer: Core projects seem to have a good understanding of our test coverage, we know what they are doing and vice versa, but can't do the same for every project (dfarrell07, 21:02:13)
   30. Question: Should we move int tests out of int repo, into project repos? (dfarrell07, 21:03:36)
   31. LuisGomez likes the idea (dfarrell07, 21:03:44)
   32. Int team wants to move to hosting tests, letting projects own/build/maintain them. Moving to projects hosting tests would help this. (dfarrell07, 21:04:17)
   33. TODO: Move system test code out of int repo, into project repos (dfarrell07, 21:04:45)
   34. When there are new tests that are not in our repo, we need to be told about that (dfarrell07, 21:05:49)
   35. Pushback: Int should be about testing project X with project Y. That's different than tests of project X that depend on standing up controller, for example. (dfarrell07, 21:07:04)
   36. Difference between testing multi-feature/multi-project and projects testing their own feature, may depend on other parts of codebase (dfarrell07, 21:07:49)
   37. Int team would like to let it be known that we have tons of tooling and such that makes it easy to write system tests (dfarrell07, 21:08:39)
   38. Suggestion: Two types of tests should be called "project" and "multi-project" (dfarrell07, 21:09:22)
   39. Third catagory could be "Interop test", which would work with different low-level hardware for example (dfarrell07, 21:11:01)
   40. "Integration tests" is a way overloaded term (dfarrell07, 21:11:49)
   41. All types of tests apply to both single-project and multi-project tests (dfarrell07, 21:15:35)
   42. Luis notes we need to form a team and focus on performance (gzhao, 21:24:01)
   43. lots of perf talk, but I was talking so no notes atm (dfarrell07, 21:24:45)
   44. I'll do an unconf on perf, watch out for it (dfarrell07, 21:25:17)

Action items

1. (none)

People present (lines said)

1. dfarrell07 (47)
2. gzhao (3)
3. odl_meetbot (3)