17:02:57 <colindixon> #startmeeting tws
17:02:57 <odl_meetbot> Meeting started Mon Mar 16 17:02:57 2015 UTC.  The chair is colindixon. Information about MeetBot at http://ci.openstack.org/meetbot.html.
17:02:57 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:57 <odl_meetbot> The meeting name has been set to 'tws'
17:03:05 <colindixon> #topic agenda bashing
17:03:51 <tbachman> #topic Robot Framework
17:03:52 <colindixon> #info today’s meeting is Carol (catohornet) presenting how to set up your own system tests
17:04:03 <colindixon> #chair tbachman phrobb- alagalah regXboi
17:04:05 <odl_meetbot> Current chairs: alagalah colindixon phrobb- regXboi tbachman
17:04:28 <tbachman> #info catohornet presents Getting Started with Robot Framework
17:04:36 <alagalah> #info Slides will be uploaded - LuisGomez ... if you have a copy, could you up load and link to minutes please ?
17:04:48 <colindixon> alagalah: +100 to that
17:05:08 <tbachman> #info What is Robot Framework: keyword driven automation framework using Python or Java as its backend
17:05:30 <tbachman> #info keywrods make it easier for test writers, as they aren’t required to know Java or Python
17:05:45 <alagalah> #info keyword use means you don't have to touch java or python if you don't want, or create keywords using python/java
17:05:49 <tbachman> #info The framework provides a simple library API for creating customized test libs
17:06:08 * alagalah Leaving it to the master, tbachman :) ...
17:06:14 <tbachman> #info The framework also provides a CLI and XML based outputs for integration into existing build infrastructure
17:06:23 <tbachman> alagalah:  ;)
17:06:43 <tbachman> #info Report files are created as a result of every pybot execution: report.html, log.html, output.html
17:07:27 <tbachman> #info The framework is implemented with pytyhon, runs also on Jython (JVM) and IronPython (.NET); can be extended natively using Pyton or Java; other languages supported via a remote interface
17:07:38 <tbachman> #info it’s open source as well, under the Apache 2. license
17:08:27 <tbachman> #info The framework has a RobotFramework Core (standard libs, reporting), an IDE (Robot IDE, or RIDE)
17:09:19 <tbachman> #info git clone https://git.opendaylight.org/gerrit/integration to clone the repo
17:09:56 <tbachman> #info there are two directories: CSIT, which contains the test suites, resource files, etc.; and Tools (test scripts specifically written taht test features)
17:11:12 <tbachman> #info tbachman asks if RIDE is included in the rpo
17:11:41 <tbachman> #info catohornet says it’s not in the repo, but she can provide a link to download and run it — it doesn’t have to run from the integration repo
17:13:29 <tbachman> #info LuisGomez says the integration/test/csit/suites contains the test suites available
17:14:06 <tbachman> #info LuisGomez says instead of using 20 keywords to do something, you can create a new keyword that comprises those 20 keywords
17:14:23 <tbachman> #info LuisGomez says there’s a variables dir that contains global variables for the test
17:15:29 <tbachman> #info LuisGomez says every project that has system tests has a subdirectory in the integration/test/csit/suites directory
17:16:12 <tbachman> #info catohornet says, as an example, the 010Restconf_OK is an individual test suite
17:16:41 <tbachman> #info catohornet shows that this file has test setup, teardown, libraries, resources, etc. These are required sections in your test suite
17:17:31 <tbachman> #info catohornet says that test suites have test cases
17:18:12 <tbachman> #info You can use ctrl-space when selecting a keyword to get the definition of the keyword/variable
17:19:00 <tbachman> #info jmedved asks where you get the logs from — the response or from something else
17:19:39 <tbachman> #info LuisGomez says the log file is just for logging a particular variable; these are robot logs, not ODL logs
17:20:06 <tbachman> #info LuisGomez says you get a variable that contains the REST response
17:22:19 <tbachman> #info tbachman asks if RIDE can run on a machine that’s not hosting the integration enviornment
17:22:36 <tbachman> #info catohornet says yes - you can point it to the machine hosting the integration environment
17:23:13 <tbachman> #info LuisGomez says you can run robot using RIDE or using commands
17:23:29 <tbachman> #info LuisGomez says RIDE provides features that help debugging of test cases
17:24:38 <tbachman> #info LuisGomez says the robot framework provides reporting, which isn’t available when using simple python scripting
17:25:58 <tbachman> #info LuisGomez says RIDE provides some realtime information as well
17:26:55 <tbachman> #info Etka points out the more information you put in your log messages, the more meaningful the robot framework log report will be
17:28:01 <tbachman> #info Etka says using RIDE, it takes care of any issues with spacing/formatting — you just create the keywords and link them to your test
17:28:43 <tbachman> #info LuisGomez says the integration group is working to publish more information on the keywords they’ve already defined
17:30:29 <tbachman> #info colindixon asks if the integration team is available to help out scanning through tests to make sure they’re being done correctly
17:30:44 <tbachman> #info LuisGomez says yes — contact the integration group via email, and they will work on providing answers
17:30:56 <colindixon> tbachman: thanks for all of this, you rock
17:31:00 <tbachman> colindixon: np!
17:31:54 <tbachman> #info alagalah says folks might want to create a topic called [ROBOT] when providing these questions to the integration mailing lists
17:32:03 <colindixon> #info LuisGomez says that keeping the tests in the integration project means that code reviews will help people write better system tests that leverage the exisitng libraries and keywords
17:32:33 <tbachman> #info to open ride on the command line, use ride.py &
17:32:41 <tbachman> #undo
17:32:41 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x18d7710>
17:33:09 <tbachman> #info to open RIDE from the command line, use “ride.py &” (ampersand to background the process)
17:34:02 <tbachman> #info LuisGomez says projects should create suites or folders for the functionality they want to test
17:35:32 <tbachman> #info LuisGomez says test writers should take into consideration the time it takes to run the tests — robot tests can get run in integration due to code merges, which increases the integration build/test time
17:36:22 <tbachman> #info LuisGomez says that within a folder, you have an entire suite
17:37:23 <tbachman> #info LuisGomez says that the automation they use today uses a test plan.
17:37:39 <colindixon> #info the directory structure seems to be integration/test/csit/suites/<project>/<project-specific-test>/<robot/ride-files>
17:38:15 <tbachman> #info LuisGomez says the test plan defines what gets run with merge jobs
17:38:30 <tbachman> #undo
17:38:30 <odl_meetbot> Removing item from minutes: <MeetBot.ircmeeting.items.Info object at 0x19ccbd0>
17:38:41 <tbachman> #info LuisGomez says the test plan file defines what gets run with merge jobs
17:38:56 <colindixon> #Info test plans then run suites of tests integration/test/testplans/<project>-<karaf-feature>.txt
17:38:56 <tbachman> #info LuisGomez says you can also create two test plans
17:39:03 <tbachman> colindixon: thx!
17:39:12 <colindixon> tbachman: trying :-)
17:40:59 <tbachman> #info LuisGomez says the robot reports are automatically imported to jenkins, b/c there’s a jenkins plugin for that
17:41:14 <tbachman> #info Uri asks if there’s anything that documents what coverage is for the tests
17:41:25 <tbachman> #info LuisGomez says that’s decided by the test writers
17:41:54 <tbachman> #info colindixon asks if there’s an equivalent code coverage for system test
17:42:06 <tbachman> #info LuisGomez says it’s not an easy one, as the test writers are usually the same ones writing the functionality in the project
17:42:26 <tbachman> #info LuisGomez says that system test is also “outside the box”, so not able to map to lines of code
17:42:46 <tbachman> #info LuisGomez says the only information they have today is only the number of test cases
17:43:24 <tbachman> #info LuisGomez says the integration team is also looking into other ways to define and report this type of coverage
17:43:39 <tbachman> #info colindixon says teh requirement for Lithium is to have at least one system test per user-facing feature
17:44:29 <tbachman> #info colindixon says codifying this in robot should be a goal here, to help identify breakage sooner
17:44:55 <alagalah> colindixon: It also means committers don't have to keep the test env up to date etc etc
17:45:02 <alagalah> colindixon: consistency thru centralisation
17:45:24 <colindixon> alagalah: +1000
17:45:49 * alagalah <---- poster child for simple things :)
17:45:52 <tbachman> #info LuisGomez says don’t make complicated test cases in the beginning — start simple
17:45:57 <tbachman> alagalah: ;)
17:46:30 <tbachman> #info jmedved asks for the archetype project, which builds its own distribution — how can it put tests in the integration project?
17:47:00 <tbachman> #info LuisGomez says the way this works in CI is after a merge, you call a distribution job that automatically creates a distribution with your change; the system tests are then run against this distribution
17:47:21 <tbachman> #info LuisGomez says the local distribution can also be used for testing after verify jobs, if needed
17:47:56 <tbachman> #info colindixon asks if there are keywords to load features
17:49:54 <tbachman> #info LuisGomez says in the jenkins automation jobs, the jenkins configuration has a variable called BUNDLEURL which is the URL used for fetching the artifact from nexus, and CONTROLLERFEATURES defines the features to be brought in for the jenkins test; you can also use TESTOPTIONS and TESTPLAN to pass in as robot test options
17:50:27 <colindixon> #info LuisGomez shows lots of parameters for the l2switch-csit-1node-cds-switch-all-master jenkins file that let you specify what distribution to test, etc.
17:50:43 <tbachman> #info flaviof  asks if they factor in the amount of time it takes for features to load before testing
17:51:54 <tbachman> #info LuisGomez says there’s no magic answer for determining when the controller is ready, but they have a script that tests restconf/modules until it gets a 200 response code
17:52:57 <tbachman> #info hideyuki says the sleep time may not be enough time — for example, VTN tests require more time
17:53:28 <tbachman> #info LuisGomez says the integration team would like to get this input from projects, so they can understand how much time is needed and why
17:53:50 <tbachman> #info LuisGomez says for example, when running compatible-all, there’s no problem; but when adding vtn-manager, it may take more time
17:54:10 <tbachman> #info flaviof asks if there’s something that’s part of the existing library for this
17:54:59 <tbachman> #info LuisGomez says they don’t use the robot framework to launch the controller at this moment — that’s done as part of the jenkins job
17:56:24 <tbachman> #info colindixon says getting more folks familiar with the framework should take the load off the integration team (~4 people full time, vs. ~200 devs across projects)
17:56:58 <tbachman> #info flaviof says that having a common script for this will help across projects — they can all use it, and we can all use and test the same infra
17:58:40 <tbachman> #info A test case file can contain other configuration information to ease the processing and speed results such as specifying variables, path to external keywords or python functions or other configuration data
17:59:15 <tbachman> #info The recommended test suite contents include: setup, settings, resource file, keywords, documentation, log, tags, test case steps, and tear down
17:59:27 <colindixon> #link https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Using_Robot_Framework#Best_Practices test case best practices
17:59:36 <tbachman> colindixon: always has the right links :)
17:59:45 <colindixon> #link https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Test_Code_Guidelines that section links to here (which has more detail)
17:59:56 <colindixon> tbachman: colindixon is good at searching for keywords on the wiki :p
18:00:00 <tbachman> lol
18:00:52 <hideyuki> Before ending the meeting, would you someone share the status of cutting a release candidate of Helium Stable Release 3?
18:00:57 <tbachman> #info catohornet says that test cases should be written to test one function — doing several different things with a pass/fail result makes it difficult to understand what’s being tested
18:01:29 <tbachman> #info LuisGomez says that the teardown step is important, as without it, it can cause problems with subsequent tests
18:01:53 <tbachman> #info colindixon asks if we can periodically nuke the test environment
18:02:16 <tbachman> #info colindixon says an example of “nuke the environment” is a hard reset of the controller
18:02:44 <tbachman> #info LuisGomez says today we don’t have to do this because of the dynamic JJB environment
18:05:38 * tbachman applauds
18:05:55 <tbachman> #info colindixon asks if it makes sense to do a robot happy hour (will take that to the mailing lists as well)
18:06:06 <tbachman> #info catohornet and LuisGomez say that’s a great idea
18:07:11 * tbachman lets colindixon do the action
18:07:24 <colindixon> #action catohornet to send mail about setting up a time for robot happy hour, maybe 7a pacific for a bit?
18:07:37 <tbachman> endmeeting?
18:07:45 <colindixon> tbachman: works for me
18:07:46 <tbachman> #endmeeting