Goals: 100% coverage of API functions via test code; 85% line coverage. We currently fall short of the API coverage. We need to monitor it and improve it. What does "API" mean?
"Coverage" usually means "invoked somehow while running the tests", including indirectly. It would be better if API functions were called directly from test code. In addition, we should really cover every method in the library, public or not. If a method is not covered, we can't be sure that it works. If every method is covered, then we need not distinguish API+implementation+override from others (i.e., easy to measure). On the other hand, we do not aspire to 100% line coverage, and therefore may decide not to require 100% implementation method coverage. Ideally, we would also "cover" public API constants, typedefs, and macros, but those are hard to measure. They would probably require static code analysis. If there is code in the library that is only used for tests, tools, CLDR, Unicode Tools, ... then we should move it out of the library. This will not only improve library code coverage, but also reduce the size of the library. Do not move UnicodeMap or Relation, because Mark wants to eventually make those public (may need to discuss these). Priorities for improving coverage where missing:
See the Trac query for tickets with the "coverage" keyword. We might use exceptions for hard-to-cover or somehow irrelevant methods. However, exception mechanisms are specific to each coverage tool (TODO: true?), and it might be more efficient in the end to try to cover as much as physically possible. JavaWe have a continuous-build bot that will automatically fail if new methods are added to the library without test coverage: http://bugs.icu-project.org/trac/build/icu4j-cov (Since 2016-apr-29, ICU ticket #12445.)
Therefore:
Download the report_html.zip from the latest build of the "ICU4J trunk coverage" build bot, unzip, and open its index.html file. Look for Missed Methods. Drill down into packages and classes. Snapshot from around ICU 57: http://icu-project.org/~yoshito/jacoco_57.1/#dn-j You can run the code coverage plug-in tool EclEmma which is based on JaCoCo. See the section 'Test Coverage' in Eclipse Setup for Java Developers. You can run the JaCoCo tool from ant: See the email “ICU4J method coverage” from Yoshito 2016-mar-17 about ant coverageJaCoCo failing with not-covered methods. TODO: JaCoCo reports non-existent constructors in static-only classes as not covered. What to do? (e.g., CollationLoader.ASCII) Test via reflection? Testing package private methods: We have been adding helper funcitons to icu4j/main/tests/framework/src/com/ibm/icu/dev/test/TestFmwk.java which use reflection to test common types of package private methods that are common throughout the code base.
Refactor ICU4J+CLDR+Unicode ToolsBefore you move or remove internal (e.g., impl or dev) or @internal ICU4J classes or methods, make sure that they are not used by CLDR and Unicode Tools. Normally, those projects use two pre-built jar files for ICU4J which makes their dependencies on ICU4J not visible. Change their project settings as follows, then refactor, then revert the project settings. Eclipse workspace with projects for ICU, CLDR, and Unicode Tools. Open Properties for the cldr-tools project.
Open Properties for the cldr-unittest and unicodetools projects, make the same changes. If you move classes from one ICU package to another, you need to update CLDR's icu4j.jar. See also https://sites.google.com/site/unicodetools/home/refactoring C/C++Go to the latest build of the "icu4c trunk coverage" build bot, scroll down to the "gcov" section. Open a source file link in a new tab. It shows line coverage but not method coverage. Might be tedious to track API and other methods. Does this merge the coverage from all test suites? |