Improved Results Analysis (Charting)

[fcohen]

This enhancement significantly enhances the way PushToTest TestMaker analysis test results and enables root-cause analysis and mitigation.

Click on the following link for a summary of the changes: http://downloads.pushtotest.com/tm5/Results_UI_Design.pdf

Step Results Analysis

TestMaker 5.2 and later records the time it takes to process the steps within each transaction. The transaction distribution charts then show the time each step takes within a transaction. For instance, the following TestScenario defines a transaction with three steps:

<usecase name="LoadTestCase">
<sequence name="MySequence" proportion="100">
<test>
  <run module="frankmodule" name="Log-in" testclass="franktest" method="runTest" langtype="jython"/>
  <run module="frankmodule" name="Select Pen" testclass="franktest" method="runTest" langtype="jython"/>
  <run module="frankmodule" name="Choose box" testclass="franktest" method="runTest" langtype="jython"/>
</test>
</sequence>
</usecase>

Also in the TestScenario is a new command to tell TestMaker to build a Transaction Distribution chart with Step Details.

<logs stepdetails="true"/>

The above TestScenario creates a Transaction Distribution chart showing each transaction as one vertical column with a comparison of the contribution of each value to a total across categories.

See the attached PNG for an example.

-Frank

[fcohen]

Transaction distribution chart showing the contribution of each step to a total across the transaction

[fcohen]

William wrote:

In testscenario we have different levels of transaction execution. As noted in the XML, we have a TestUseCase? that contains several Usecases (each with a name). Each usecase contains one or more sequences (with a name) and each sequence contains one or more run tags.

Right now, the XML contains the usecase name enclosing all the transactions tags. Sequences and runs may execute different transactions in the same period of time. Their names are not used.

Frank proposal can be summarized as: 1. Adding a name to each individual Run tag. Name is optional, and if not found a default "RunX" where X is the index of the run in the sequence, will be used. 2. All transaction tags will contain two additional attributes: The sequence name and the run (step) name. 3. The Tally will be require to parse and group by sequence and by step, and add to the summary that detail. 4. We will require to add chart options too, to indicate which charts to create and by which level. 5. I will add the comparison tool to be able to drill down the info.

Now, the chart Frank proposes may not be feasible. Luis Carlos, please clarify what conforms a transaction? Is a complete sequence a transaction, or each individual run tags are? Now, transactions may be hundreds or thousands, the chart may not be able to show all transactions dissected as the sample. A step comparison may be done (adding up all the time for each step and compare its importance).

[fcohen]

Relationship between test definition and proposed transaction/step logs

[fcohen]

Luis Carlos design document in PDF format

[fcohen]

William wrote:

Hello Frank. This is the list Luis Carlos put together plus his estimates. (See the document "Luis Carlos Design Document" attached to ticket 188) I'm still not sure about the chart. Your idea of bars each 10 seconds would clutter the chart. Luis Carlos idea was to create one chart per sequence. Each chart will contains at most 4 bars, each containing the segmented values of the steps. I see no use for that kind of chart.

I would say one chart per sequence and one bar per cvs/payload.

What do you think?

BTW. Those are estimates for Luis Carlos. If implemented by Max, it may take a little longer.

William

[fcohen]

I attached Luis Carlos' design document to this ticket. I have the following issues with the design:

1) Modifications to TXLog.xsd is good for me. I read the XSD as generating <step> child elements within each <transaction> entry. Please correct me if this is wrong.

2) The cumulative charts with each sample being up to 10 seconds does not work for me. Users would have too many charts on their hands. I am looking for a single Transaction Distribution chart showing all of the transactions for a given <testTime> period (for the testusecase.) Each <transaction> entry in the log appears as a vertical column in the single charge. Each column is made up of the <step> values. The horizontal axis of this one chart is the duration of the test in seconds, with zero at the left and the <testTime> value at the right.

3) I prefer my design for <logs stepdetails="true"/> over Luis Carlos' proposal for a <chart> element.

4) The Performance Comparison Utility needs to support this new <step> method in its existing transaction report.

5) The chart elements (vertical columns and step values within each column need to display the name of the transaction and the names of the step values.

It seems that Luis Carlos' estimates 14 hours for this work. We kicked around the idea of Max doing the work with Luis Carlos advising Max. Go for it.

-Frank

[fcohen]

I posted a new design for TestMaker Results Analysis (Charting) function that addresses the issues in Ticket 188 and other issues I am hearing from PushToTest customers and users. The design is in the form of an Excel spreadsheet and downloaded at downloads.pushtotest.com/tm5/Chart_Design_Ticket188.xls

William Martinez Pomares and I will get together to discuss the design in the next few days. Please send me your feedback, comments, and changes to fcohen@….

Thanks.

-Frank

[fcohen]

I had a design review meeting with William, Luis Carlos, Max, and Jimmy. Here is the email I sent after the meeting. -Frank

---

Hi Guys:

Thank you for the phone call today and your good questions and feedback. Attached is an updated design document (Excel format.) Here are the changes I made to the design.

1) I updated the Chart drop-down menu in the Graphical Interface Design worksheet to reflect the functions to save the chart settings to the testmaker.properties file and the current TestScenario document. Using Save As saves the chart settings to testmaker.properties. All changes using the - icon and the Chart Settings dialog save to the current TestScenario document.

2) The + and - icons in the Graphical Interface Design are to add a new chart to the current TestScenario and remove a chart from the TestScenario.

3) I moved the secondary axis settings (Distribution Line, and CPU/Memory/Net Utilization) to the Y axis.

4) As discussed, some chart settings will be disabled depending on the axis choice.

5) I removed the Pass/Fail Status from the X Axis.

6) Users may choose Distance or Average, Minimum, Maximum or Moving Average.

The new design document (Excel) is at: http://downloads.pushtotest.com/tm5/Chart_Design_Ticket188_v2.xls

I am looking forward to the Sunday Clothes version of the new Results system.

-Frank

[fcohen]

I realized that I did not incorporate the update Graphical User Interface elements. I added them to:

http://downloads.pushtotest.com/tm5/Chart_Design_Ticket188_v3.xls

-Frank

[fcohen]

Hi TestMaker Developers:

William, Luis Carlos, Luigi and I had a phone call to review the new design document. Download the design document at:

http://downloads.pushtotest.com/tm5/TM_Results_DesignDoc_fc_v2.pdf

Here were my questions:

Question: Will the Performance Comparison Utility (PCU) continue to function after the new Results Analysis package changes?

Answer: Yes, the PCU will continue to function as it does today.

Question: Will I be able to point the new results charting to an archived results directory?

Answer: Frank will develop a new user interface to select the source results.

Question: Does the new result factory deal with multiple results files?

Answer: Yes.

Question: How will the new Results Analysis package handle very large test result files? For instance, a test that runs for 24 hours.

Answer: William and Luis Carlos are considering a summary index function so the entire log does not reside in memory.

Question: What is the schedule for rest of the days in this week?

Answer: Luis Carlos started on the changes to the summary schema, design of the classes, will start with summary class and tally class.

Our next phone call status update meeting is on Thursday December 20 at 11 am Pacific time. Please let me know if you would like to participate.

-Frank

[fcohen]

TestMaker test scenarios generate a lot of useful data. TestMaker 5.0 introduced standard charts to visualize the data into actionable knowledge, including the Scalability Index, Transaction Distribution Charts, and Resource Monitor Charts. TestMaker 5.0 also introduces a result log archiving system and a Performance Comparison Utility to generate charts to compare results between operations of test scenarios.

TestMaker 5.2 will introduce new results analysis functions:

1) Transaction logs will include the "step" times for the operations within a

2) A new charting function will summarize logged results data in new and flexible ways.

TestMaker 5 creates a lot of data that can be put into a chart. Here are the base data series:

Test Scenario (name)

Test Node (name)

Concurrent User levels (CRs)

Data Index (Message size)

Transaction (Use Case)

Sequence name

Step (run) name

Pass/Fail status

I put together a new presentation to introduce you to the Enhanced Results Analysis functions. Download the presentation at http://downloads.pushtotest.com/tm5/PushToTest_ResultsAnalysis_20071220.pdf

-Frank

[fcohen]

Luis Carlos wrote:

About the idea of clicking in a summary graph and the system showing the individual node graphs, I think it's a nice idea, and I don't think of it as very far from our scope, because the system support it, and I had developed two engines that actually create the target (summary) graph and graphs for the rest of the nodes.

If you want, when we finish ticket 188 (Only summary graph), I can work in the POC of your Idea with these two engines that already support both options (summary and per node).

[fcohen]

We're making outstanding progress on building the Enhanced Results Analysis engine for PushToTest TestMaker. The internal work is almost done. I put together a user interface design for your review, feedback, and comments. Please download it in Adobe Acrobat format at:

http://downloads.pushtotest.com/tm5/Results_UI_Design.pdf

Thanks, in advance!

-Frank

[fcohen]

Working through the following issues:

Create a new chart, run a load test scenario, click the magnify icon on the chart, make some changes, choose save. At this point TM should ask the user if they want TM to rerun the results.<br> unfinished<br> <br> For a chart setting that generates tabbed charts, the vertical scale needs to be the same for all of the tabs.<br> unfinished<br> <br> When a chart setting includes Steps and the TestScenario &lt;log&gt; has step="false" or has no step attribute, the user needs to get a warning unfinished<br> Not yet<br> <br> CRL = Users. Please change this.<br> Done<br> <br> The drop-down menu for saving a chart definition to the current TestScenario or TestMaker was a bad idea. Please change this to buttons.<br> unfinished<br> <br> When I edit a chart definition and change its name then clicking Save creates a new chart definition.<br> unfinished<br> <br> <br> I don't see the progress/abort controls while TM updates a chart.<br> Does this work?<br> It is working. IN most of the test the processing takes to small amount of time, you won't see it.<br> <br> <br> I set up a chart with Duration Average -&gt; Node and clicked Usecase and CRL.<br> Question 1: should the checkboxes actually be radio buttons (meaning you can only select one?)<br> Yep, there is an error there. The user can only select a type.<br> We are not using radiobuttons. The current technique is that when there are series combination that don't work, the system will make uselectable the series that doesn't make sense<br> e.g. If you select the<br> - Y Duration Average<br> - X Node<br> - Series CRL.<br> Then the system will make USECASE, DATA INDEX uselectable.<br> <br> <br> Question 2: What should this report contain?<br> This report is known as the fastest testnetwork node. It is a comparison of nodes average transaction duration (excluding the target) in the usescases.<br> <br> I click the + icon from the controller panel, enter a chart name, Duration Average, Periods, click Steps, select Stacked Bars, click save, then click Magnify icon. I see the Graph Type is set to Bars. I can't get it to register for Stacked Bars.<br> unfinished

[fcohen]

Create a new chart, run a load test scenario, click the magnify icon on the chart, make some changes, choose save. At this point TM should ask the user if they want TM to rerun the results. unfinished

For a chart setting that generates tabbed charts, the vertical scale needs to be the same for all of the tabs. unfinished

When a chart setting includes Steps and the TestScenario <log> has step="false" or has no step attribute, the user needs to get a warning unfinished Not yet

CRL = Users. Please change this. Done

The drop-down menu for saving a chart definition to the current TestScenario or TestMaker was a bad idea. Please change this to buttons. unfinished

When I edit a chart definition and change its name then clicking Save creates a new chart definition. unfinished

I don't see the progress/abort controls while TM updates a chart. Does this work? It is working. IN most of the test the processing takes to small amount of time, you won't see it.

I set up a chart with Duration Average -> Node and clicked Usecase and CRL. Question 1: should the checkboxes actually be radio buttons (meaning you can only select one?) Yep, there is an error there. The user can only select a type. We are not using radiobuttons. The current technique is that when there are series combination that don't work, the system will make uselectable the series that doesn't make sense e.g. If you select the - Y Duration Average - X Node - Series CRL. Then the system will make USECASE, DATA INDEX uselectable.

Question 2: What should this report contain? This report is known as the fastest testnetwork node. It is a comparison of nodes average transaction duration (excluding the target) in the usescases.

I click the + icon from the controller panel, enter a chart name, Duration Average, Periods, click Steps, select Stacked Bars, click save, then click Magnify icon. I see the Graph Type is set to Bars. I can't get it to register for Stacked Bars. unfinished

[fcohen]

Frank Cohen wrote on Wednesday, May 28, 2008:

1) Click the "Remove chart" icon, TM removes the chart, the chart definition remains in the TestScenario.

2) Click the "Change chart settings" icon for a chart, nothing happens.

3) Click the "Add A New Results Chart" icon in the Controller Panel, "Results Analysis Chart Settings" dialog appears. However, the last selected Chart setting values are in the edit fields. TM should open the Chart Settings dialog and be ready to enter a new Chart.

4) I can't seem to get any chart to display CPU/Net/Mem resources. For instance, see the attached results directory archive. When I run this I see an empty chart.

<monitoring enablednodes="false" enabledtarget="true" targethost="http://localhost"/>

5) In the "Results Analysis Chart Settings" dialog the BTNUpdate JButton only needs to save the current chart to TestMaker.

6) In the "Results Analysis Chart Settings" dialog please rename the "TestMaker" label to "Chart Templates".

[fcohen]

Archive referenced in May 28 change requests comment

[fcohen]

We need a chart expansion function. Some charts do not have enough vertical space to display the results and the legend. See the attached chart for an example of the problem.

[fcohen]

Example of sizing issue from May 27 comment

[llara]

I want to add another issue for 188

Currently data index can not be 0. If data index is 0, testmaker will process tally incorrectly.

Reason: In the tally I needed a "null" value for the data index. Then I choose 0 for the new value. That is because I made the dataindex implementation using the users implementation (it can never be 0 users, I think).

Now thinking again, I realized that much of java developers start their indices in 0, then I will set the dataIndex "null" value in Integer.MIN_VALUE.

[fcohen]

Luis Carlos added:

- Use tab to move between graph definitions - Be able to set the data index name in the xml. the name will be shown in the graph

<dataindices>
  <dataindex value="0"  name="10k"/>
<dataindices>

[fcohen]

Waiting on some final tweaks I will make to the chart settings window. Then we're done! 9 months in the making! -Frank

[fcohen]

Still needs documentation, but this baby is done! Congratulations team. Well done! -Frank