1. 27 May, 2016 1 commit
    • Olivier Berthier's avatar
      framework: Add aborting option when a monitored error is detected · 21031482
      Olivier Berthier authored
      This adds a policy which advises when the user should reboot the system
      to avoid noisy test results due to system becoming unstable, for
      instance, and therefore continues testing successfully.
      
      To do this, a new module is proposed. A class Monitoring is used for
      managing the monitoring rules. Two types of rules, MonitoringFile and
      MonitoringLinuxDmesg, derived from the abstract class MonitoringBase,
      have been implemented. The first allow to track a pattern on standard
      files or locked files. The second, derived from dmesg.LinuxDmesg, will
      track a pattern on the dmesg.
      
      The monitoring rules must be defined in piglit.conf at the section
      monitored-errors. If one of the regex is found, Piglit will raise a
      PiglitAbort exception, stop the test execution -terminating test thread
      pool- and exit with code 3.
      
      Then test execution resume, after rebooting the system or not, is done
      like usually with command line parameter "resume".
      
      To call it, use command line parameter: --abort-on-monitored-error
      
      This option implies --no-concurrent
      
      This include also a set of unit tests for this module.
      Reviewed-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      21031482
  2. 10 May, 2016 1 commit
  3. 03 May, 2016 4 commits
  4. 17 Apr, 2016 1 commit
  5. 08 Feb, 2016 5 commits
  6. 11 Nov, 2015 1 commit
  7. 29 Jun, 2015 1 commit
  8. 19 May, 2015 1 commit
  9. 05 May, 2015 1 commit
    • Dylan Baker's avatar
      framework: write an incomplete status before running tests · 8ce57882
      Dylan Baker authored
      This sets the file based backends (both the json and junit) to write out
      an incomplete status before starting the test. (for junit this is a
      'fail' with a message of 'incomplete', junit is a fairly limited
      format). When the test is completed this incomplete status is
      overwritten with the final status.
      
      This patch also adds a new mode to the 'summary console' option, '-i',
      which will list all tests with an incomplete status.
      
      This will allow a developer to see what tests failed when a system
      crashed, or what tests were running when they aborted piglit.
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      Reviewed-by: Mark Janes's avatarMark Janes <mark.a.janes@intel.com>
      8ce57882
  10. 01 Apr, 2015 1 commit
    • Dylan Baker's avatar
      profile.py: Don't allow accidental reassignments of TestDict keys · dfd8d8f9
      Dylan Baker authored
      A common error in all.py is that two different tests are assigned to the
      same key value in profile.test_list. This change prevents a key from
      being reassigned to a new value on accident, but provides a mechanism to
      allow reassignment, using a context manager. This allows tests like the
      image_load_store tests in quick.py to be overwritten, but requires the
      author to mark that they know that they are overwriting tests, and that
      it's intentional.
      
      The error message will compare the tests by command, if the two tests
      have different commands then the commands will be printed for easy
      visual comparison. This helps to identify whether a test is being added
      twice (in a loop for example) or if two different tests are being
      assigned the same name.
      
      This also adds some tests for TestDict.
      
      v2: - remove some rebase artifacts, this makes the code a little simpler
            and a little cleaner
      v3: - rebase on master, includes grouptools separator changes
      v4: - Use a incremented value instead of True false for the context
            manager, this allows the allow_reassignment contextmanager to be
            nested.  (Ilia)
      v5: - Add comparison methods to the Test class
          - Use said comparisons to make the error reporting more verbose
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu> (v4)
      dfd8d8f9
  11. 26 Mar, 2015 1 commit
  12. 17 Mar, 2015 1 commit
  13. 09 Mar, 2015 1 commit
    • Dylan Baker's avatar
      profile.py: Support automatic naming of tests with string arguments · c08e24a2
      Dylan Baker authored
      I made a somewhat silly assumption when writing the group_manager, that
      all Test classes would use list arguments. While this assumption is true
      for tests that require an entire command as their first argument it
      isn't true of GleanTest, where only the name of the subtest is passed.
      In this case it will be a string, and if a name isn't passed as well
      then name will end up being " ".join()'ed, ie, "api" -> "a p i".
      
      This patch adds support to the group_manager to support this feature.
      
      I considered two other approaches here. The first was to say screw
      correctness, and just add a duplicate name to the handful of tests that
      use the group_manager. The second was to change GleanTest to take a 1
      element list. Both of these had downsides that I couldn't justify.
      
      This fixes a handful of glean tests that use the group_manager (fbo,
      api2, etc)
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      Tested-by: Jose Fonseca's avatarJose Fonseca <jfonseca@vmware.com>
      Tested-by: Michel Dänzer's avatarMichel Dänzer <michel.daenzer@amd.com>
      c08e24a2
  14. 08 Mar, 2015 1 commit
    • Jose Fonseca's avatar
      framework: Try to fix the exclusions of llvmpipe test list. · afa9c8b6
      Jose Fonseca authored
      The
      
        del profile.test_list[key]
      
      statement was silently failing because the TestDict's key lowering magic
      was not happening for deleted items.
      
      Unfortunately this still is not enough to fix the glean exclusions:
      somehow all glean test names are being munged with extra whitespace.
      For example:
      
        $ ./piglit-print-commands.py tests/llvmpipe.py | grep '^glean/p'
        warning: test glean/pointAtten does not exist
        warning: test glean/texCombine does not exist
        [...]
        glean/p o i n t a t t e n ::: bin/glean -o -v -v -v -t +pointAtten --quick
      Reviewed-by: Dylan Baker's avatarDylan Baker <baker.dylan.c@gmail.com>
      afa9c8b6
  15. 05 Mar, 2015 3 commits
    • Dylan Baker's avatar
      profile.py: Add kwargs to TestProfile.group_manager · f735cbfa
      Dylan Baker authored
      This adds the ability for the group_manager method to take additional
      keyword arguments that are passed to the underlying Test constructor.
      This allows groups that all need the same arguments (say platform
      requirements), but be passed to the constructor and be added by the
      adder.
      
      >>> prof = TestProfile()
      >>> with prof.group_manater(Test, 'group', required_platforms=['glx']) as g:
      ...     g(['glx-foobar'])
      >>> prof.test_list['group/glx-foobar'].required-platforms == ['glx']
      True
      f735cbfa
    • Dylan Baker's avatar
      profile.py: Remove TestProfile.tests · ac687d0a
      Dylan Baker authored
      This removes the tests attribute from TestProfile, which has been
      deprecated for some time. The problem with tests is that it uses a
      nested tree structure, which has to be flattened before tests can be
      run. This creates a significant amount of overhead.
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      ac687d0a
    • Dylan Baker's avatar
      profile.py: Add a context_manager to TestProfile for adding tests. · 569c7d15
      Dylan Baker authored
      This adds a very powerful method to the TestProfile class,
      group_manager. This group_manager method is a context_manager (it's used
      with the 'with' statement), and is passed a test class and a flattened
      group name (such as one returned by grouptools.join), and yields a
      callable that is used to add tests to the profile.
      
      This gives us a lot of advantages. First, it means that tests are added
      in a context, so if we need to define data structures for adding tests
      to a specific group (say a list of texture formats supported by a
      specific version of GL), that data structure is defined in a nested
      scope, so it cannot be accidentally be used in a different group.
      Second, it means not passing a group around anymore, in fact, it creates
      an abstraction on top of the group data structure so it doesn't matter
      how we're representing it. Third the function itself is more flexible
      than anything we've had before. It can either take an explicit name, or
      it can call ' '.join() on the arguments to the Test, and use that as a
      name, which can help make Test assignments much less verbose.
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      569c7d15
  16. 24 Feb, 2015 2 commits
    • Dylan Baker's avatar
      profile.py: Lower all test keys before getting and setting. · ea381355
      Dylan Baker authored
      One of the problems for piglit is that it is a cross platform system, and that
      it is useful to think about the file hierarchy and the group hierarchy as
      equivalent. This allows auto-discovery of tests by doing a file system search,
      which requires very little code and makes developers lives easy.
      
      This is very easy on Linux and other systems with sane file
      systems, because python has the same view of strings as these systems do of
      paths. a == a, but a != A. To avert problems with case sensitivity files
      generally are lower case, while in all.py they have mirrored the case of
      the functionality they test, whether that be a function name or an
      extension name.
      
      This is problematic because it means that if there are tests for
      ARB_ham_sandwich that are automatically discovered, and those that are
      added by all.py they will end up in two separate groups
      (ARB_ham_sandwhich and arb_ham_sandwich), and thus be spread across the
      html summary on sane systems.
      
      This patch adds another class, TestDict, which is, like Tree, derived
      from dict, and provides a __getitem__, __setitem__ and __delitem__
      method, which call str.lower() on the key input before actually getting,
      setting, or deleting the key/value pair. They key is also forced to be a
      string, while the value is forced to be a Test derived class.
      
      Tree is, in turn, made a subclass of TestDict, providing the same
      functionality for TestProfile.tests and TestProfile.test_list.
      
      Finally, this change necessitates bumping the results version, and
      adding all of the boilerplate that goes along with that.
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      ea381355
    • Dylan Baker's avatar
      profile.py: Use a Tree structure for Profile.tests · f302ebca
      Dylan Baker authored
      This change converts Profile.tests from a dict to a Tree, which is a new
      class that descends from dict, but automatically adds new branches
      whenever one is requested that doesn't exist.
      
      This solves a couple of problems, first, there is a lot of code in
      all.py to add new dictionaries to the tests tree, and that is annoying
      (In fact this ends up removing about 200 lines of code total). The
      second problem it solves is that some of the search-and-add functions in
      all.py add nodes to the tree, which are then relied on by functions in
      all.py. This is sloppy and annoying, and while it could be remedied by
      adding all of the missing branches to the tree that would result in a
      lot of LOC additions for no reason.
      
      This class will be removed by the end the series.
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      f302ebca
  17. 12 Feb, 2015 1 commit
  18. 24 Dec, 2014 1 commit
    • Dylan Baker's avatar
      framework: Cleanup imports · 2d34c2d5
      Dylan Baker authored
      Makes 2 changes, all of which should have absolutely zero effect
      
      1) breaks the imports into three newline separated groups: builtins,
      installed modules, local modules
      2) use print_function and absolute_import in all modules.
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      2d34c2d5
  19. 12 Dec, 2014 1 commit
  20. 04 Nov, 2014 1 commit
  21. 14 Oct, 2014 2 commits
  22. 25 Sep, 2014 1 commit
    • Dylan Baker's avatar
      framework: refactor the log module · a69d67aa
      Dylan Baker authored
      This refactors the log module away from a single class that does
      everything to having two classes. The first class LogFactory, is a state
      manager that returns BaseLog derived objects when it's get() method is
      called. However, it mainly acts as a shared state manager.
      
      The BaseLog derived classes provide three public methods; start(),
      log(), summary().
      
      This makes the interfaces much cleaner, just 3 public methods that are
      fairly obvious and clearly documented. It uses inheritance for more
      shared code, and is just generally much less complex than the previous
      implementation
      
      The use of a non-recursive lock mandates that the majority of the code
      be in un-locked protected methods, while the public method takes the
      lock, otherwise inheritance is nearly impossible.
      
      v2: - Fix summary line in the verbose logger to support using '\r'. This
            works by printing the summary at the end of start() and log()
          - add -v/--verbose option back (Ilia)
      v3: - Be more aggressive about locking
      v4: - be more granular with locking by using two locks, one for
            printing, one for manipulating the shared data.
          - Move some unit tests to the next patch that belong there.
      v5: - simplify the locking. This may have a performance impact, but
            makes the code much simpler. It should be possible to make the
            locking finer grained later
      v6: - Lock one additional command
      v7: - fix bug in Test. Test did not call log.start() so verbose log
            would print <status>: None
          - replace RLock with Lock. This requires a little bit of refactoring
            to make it work, since the lock cannot be called recursively
      v8: - remove trailing whitespace
      Signed-off-by: default avatarDylan Baker <dylanx.c.baker@intel.com>
      Reviewed-by: Ilia Mirkin's avatarIlia Mirkin <imirkin@alum.mit.edu>
      a69d67aa
  23. 29 Aug, 2014 2 commits
  24. 30 Jun, 2014 1 commit
  25. 24 Jun, 2014 1 commit
  26. 13 May, 2014 3 commits