#opendaylight-meeting: tws

Meeting started by colindixon at 17:02:57 UTC (full logs).

Meeting summary

  1. agenda bashing (colindixon, 17:03:05)
    1. today’s meeting is Carol (catohornet) presenting how to set up your own system tests (colindixon, 17:03:52)
    2. catohornet presents Getting Started with Robot Framework (tbachman, 17:04:28)
    3. Slides will be uploaded - LuisGomez ... if you have a copy, could you up load and link to minutes please ? (alagalah, 17:04:36)
    4. What is Robot Framework: keyword driven automation framework using Python or Java as its backend (tbachman, 17:05:08)
    5. keywrods make it easier for test writers, as they aren’t required to know Java or Python (tbachman, 17:05:30)
    6. keyword use means you don't have to touch java or python if you don't want, or create keywords using python/java (alagalah, 17:05:45)
    7. The framework provides a simple library API for creating customized test libs (tbachman, 17:05:49)
    8. The framework also provides a CLI and XML based outputs for integration into existing build infrastructure (tbachman, 17:06:14)
    9. Report files are created as a result of every pybot execution: report.html, log.html, output.html (tbachman, 17:06:43)
    10. The framework is implemented with pytyhon, runs also on Jython (JVM) and IronPython (.NET); can be extended natively using Pyton or Java; other languages supported via a remote interface (tbachman, 17:07:27)
    11. it’s open source as well, under the Apache 2. license (tbachman, 17:07:38)
    12. The framework has a RobotFramework Core (standard libs, reporting), an IDE (Robot IDE, or RIDE) (tbachman, 17:08:27)
    13. git clone https://git.opendaylight.org/gerrit/integration to clone the repo (tbachman, 17:09:19)
    14. there are two directories: CSIT, which contains the test suites, resource files, etc.; and Tools (test scripts specifically written taht test features) (tbachman, 17:09:56)
    15. tbachman asks if RIDE is included in the rpo (tbachman, 17:11:12)
    16. catohornet says it’s not in the repo, but she can provide a link to download and run it — it doesn’t have to run from the integration repo (tbachman, 17:11:41)
    17. LuisGomez says the integration/test/csit/suites contains the test suites available (tbachman, 17:13:29)
    18. LuisGomez says instead of using 20 keywords to do something, you can create a new keyword that comprises those 20 keywords (tbachman, 17:14:06)
    19. LuisGomez says there’s a variables dir that contains global variables for the test (tbachman, 17:14:23)
    20. LuisGomez says every project that has system tests has a subdirectory in the integration/test/csit/suites directory (tbachman, 17:15:29)
    21. catohornet says, as an example, the 010Restconf_OK is an individual test suite (tbachman, 17:16:12)
    22. catohornet shows that this file has test setup, teardown, libraries, resources, etc. These are required sections in your test suite (tbachman, 17:16:41)
    23. catohornet says that test suites have test cases (tbachman, 17:17:31)
    24. You can use ctrl-space when selecting a keyword to get the definition of the keyword/variable (tbachman, 17:18:12)
    25. jmedved asks where you get the logs from — the response or from something else (tbachman, 17:19:00)
    26. LuisGomez says the log file is just for logging a particular variable; these are robot logs, not ODL logs (tbachman, 17:19:39)
    27. LuisGomez says you get a variable that contains the REST response (tbachman, 17:20:06)
    28. tbachman asks if RIDE can run on a machine that’s not hosting the integration enviornment (tbachman, 17:22:19)
    29. catohornet says yes - you can point it to the machine hosting the integration environment (tbachman, 17:22:36)
    30. LuisGomez says you can run robot using RIDE or using commands (tbachman, 17:23:13)
    31. LuisGomez says RIDE provides features that help debugging of test cases (tbachman, 17:23:29)
    32. LuisGomez says the robot framework provides reporting, which isn’t available when using simple python scripting (tbachman, 17:24:38)
    33. LuisGomez says RIDE provides some realtime information as well (tbachman, 17:25:58)
    34. Etka points out the more information you put in your log messages, the more meaningful the robot framework log report will be (tbachman, 17:26:55)
    35. Etka says using RIDE, it takes care of any issues with spacing/formatting — you just create the keywords and link them to your test (tbachman, 17:28:01)
    36. LuisGomez says the integration group is working to publish more information on the keywords they’ve already defined (tbachman, 17:28:43)
    37. colindixon asks if the integration team is available to help out scanning through tests to make sure they’re being done correctly (tbachman, 17:30:29)
    38. LuisGomez says yes — contact the integration group via email, and they will work on providing answers (tbachman, 17:30:44)
    39. alagalah says folks might want to create a topic called [ROBOT] when providing these questions to the integration mailing lists (tbachman, 17:31:54)
    40. LuisGomez says that keeping the tests in the integration project means that code reviews will help people write better system tests that leverage the exisitng libraries and keywords (colindixon, 17:32:03)
    41. to open RIDE from the command line, use “ride.py &” (ampersand to background the process) (tbachman, 17:33:09)
    42. LuisGomez says projects should create suites or folders for the functionality they want to test (tbachman, 17:34:02)
    43. LuisGomez says test writers should take into consideration the time it takes to run the tests — robot tests can get run in integration due to code merges, which increases the integration build/test time (tbachman, 17:35:32)
    44. LuisGomez says that within a folder, you have an entire suite (tbachman, 17:36:22)
    45. LuisGomez says that the automation they use today uses a test plan. (tbachman, 17:37:23)
    46. the directory structure seems to be integration/test/csit/suites/<project>/<project-specific-test>/<robot/ride-files> (colindixon, 17:37:39)
    47. LuisGomez says the test plan file defines what gets run with merge jobs (tbachman, 17:38:41)
    48. test plans then run suites of tests integration/test/testplans/<project>-<karaf-feature>.txt (colindixon, 17:38:56)
    49. LuisGomez says you can also create two test plans (tbachman, 17:38:56)
    50. LuisGomez says the robot reports are automatically imported to jenkins, b/c there’s a jenkins plugin for that (tbachman, 17:40:59)
    51. Uri asks if there’s anything that documents what coverage is for the tests (tbachman, 17:41:14)
    52. LuisGomez says that’s decided by the test writers (tbachman, 17:41:25)
    53. colindixon asks if there’s an equivalent code coverage for system test (tbachman, 17:41:54)
    54. LuisGomez says it’s not an easy one, as the test writers are usually the same ones writing the functionality in the project (tbachman, 17:42:06)
    55. LuisGomez says that system test is also “outside the box”, so not able to map to lines of code (tbachman, 17:42:26)
    56. LuisGomez says the only information they have today is only the number of test cases (tbachman, 17:42:46)
    57. LuisGomez says the integration team is also looking into other ways to define and report this type of coverage (tbachman, 17:43:24)
    58. colindixon says teh requirement for Lithium is to have at least one system test per user-facing feature (tbachman, 17:43:39)
    59. colindixon says codifying this in robot should be a goal here, to help identify breakage sooner (tbachman, 17:44:29)
    60. LuisGomez says don’t make complicated test cases in the beginning — start simple (tbachman, 17:45:52)
    61. jmedved asks for the archetype project, which builds its own distribution — how can it put tests in the integration project? (tbachman, 17:46:30)
    62. LuisGomez says the way this works in CI is after a merge, you call a distribution job that automatically creates a distribution with your change; the system tests are then run against this distribution (tbachman, 17:47:00)
    63. LuisGomez says the local distribution can also be used for testing after verify jobs, if needed (tbachman, 17:47:21)
    64. colindixon asks if there are keywords to load features (tbachman, 17:47:56)
    65. LuisGomez says in the jenkins automation jobs, the jenkins configuration has a variable called BUNDLEURL which is the URL used for fetching the artifact from nexus, and CONTROLLERFEATURES defines the features to be brought in for the jenkins test; you can also use TESTOPTIONS and TESTPLAN to pass in as robot test options (tbachman, 17:49:54)
    66. LuisGomez shows lots of parameters for the l2switch-csit-1node-cds-switch-all-master jenkins file that let you specify what distribution to test, etc. (colindixon, 17:50:27)
    67. flaviof asks if they factor in the amount of time it takes for features to load before testing (tbachman, 17:50:43)
    68. LuisGomez says there’s no magic answer for determining when the controller is ready, but they have a script that tests restconf/modules until it gets a 200 response code (tbachman, 17:51:54)
    69. hideyuki says the sleep time may not be enough time — for example, VTN tests require more time (tbachman, 17:52:57)
    70. LuisGomez says the integration team would like to get this input from projects, so they can understand how much time is needed and why (tbachman, 17:53:28)
    71. LuisGomez says for example, when running compatible-all, there’s no problem; but when adding vtn-manager, it may take more time (tbachman, 17:53:50)
    72. flaviof asks if there’s something that’s part of the existing library for this (tbachman, 17:54:10)
    73. LuisGomez says they don’t use the robot framework to launch the controller at this moment — that’s done as part of the jenkins job (tbachman, 17:54:59)
    74. colindixon says getting more folks familiar with the framework should take the load off the integration team (~4 people full time, vs. ~200 devs across projects) (tbachman, 17:56:24)
    75. flaviof says that having a common script for this will help across projects — they can all use it, and we can all use and test the same infra (tbachman, 17:56:58)
    76. A test case file can contain other configuration information to ease the processing and speed results such as specifying variables, path to external keywords or python functions or other configuration data (tbachman, 17:58:40)
    77. The recommended test suite contents include: setup, settings, resource file, keywords, documentation, log, tags, test case steps, and tear down (tbachman, 17:59:15)
    78. https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Using_Robot_Framework#Best_Practices test case best practices (colindixon, 17:59:27)
    79. https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Test_Code_Guidelines that section links to here (which has more detail) (colindixon, 17:59:45)
    80. catohornet says that test cases should be written to test one function — doing several different things with a pass/fail result makes it difficult to understand what’s being tested (tbachman, 18:00:57)
    81. LuisGomez says that the teardown step is important, as without it, it can cause problems with subsequent tests (tbachman, 18:01:29)
    82. colindixon asks if we can periodically nuke the test environment (tbachman, 18:01:53)
    83. colindixon says an example of “nuke the environment” is a hard reset of the controller (tbachman, 18:02:16)
    84. LuisGomez says today we don’t have to do this because of the dynamic JJB environment (tbachman, 18:02:44)
    85. colindixon asks if it makes sense to do a robot happy hour (will take that to the mailing lists as well) (tbachman, 18:05:55)
    86. catohornet and LuisGomez say that’s a great idea (tbachman, 18:06:06)
    87. ACTION: catohornet to send mail about setting up a time for robot happy hour, maybe 7a pacific for a bit? (colindixon, 18:07:24)


Meeting ended at 18:07:46 UTC (full logs).

Action items

  1. catohornet to send mail about setting up a time for robot happy hour, maybe 7a pacific for a bit?


People present (lines said)

  1. tbachman (92)
  2. colindixon (17)
  3. odl_meetbot (6)
  4. alagalah (6)
  5. hideyuki (1)
  6. regXboi (0)
  7. phrobb- (0)


Generated by MeetBot 0.1.4.